S204: Inter-rater reliability

As the semester draws to a close, Jess wakes up in the middle of the night concerned not with completing her grading, but with measuring inter-rater reliability for coding schemes. Reliability can refer to the agreement among raters, or agreement among items. There’s a lot of online discourse right now around reliability, but most of it is focused around the former: agreement among items. Over here in the field of developmental science, we’re often coding human behavior through observations, including implementation fidelity. With scales like these, reliability is used to mean several different types of agreement among raters: Training coders to a reliability standard, “in-field” reliability between pairs of raters, “drift check” reliability as you continue to collect data over time, and then overall or average inter-rater reliability for all raters and observations. Only after assessing all of these can you begin to examine reliability designed to assess agreement among items.
Papers to read about establishing and measuring inter-rater reliability:
Hallgren, K. A. (2012). Computing inter-rater reliability for observational data: an overview and tutorial. Tutorials in Quantitative Methods for Psychology, 8(1), 23.
Bruton, A., Conway, J. H., & Holgate, S. T. (2000). Reliability: what is it, and how is it measured?. Physiotherapy, 86(2), 94-99.
Also mentioned
McNeish, D. (2018). Thanks coefficient alpha, we’ll take it from here. Psychological Methods, 23(3), 412.
Raykov, T., & Marcoulides, G. A. (2019). Thanks coefficient alpha, we still need you!. Educational and Psychological Measurement, 79(1), 200-210.
Edwards, A., Joyner, K., & Schatschneider, C. (2019, June 27). A Simulation Study on the Performance of Different Reliability Estimation Methods. PsyArXiv. https://doi.org/10.31234/osf.io/xzc52.
Connect with the podcast on twitter @within_between, or email us letters about developmental science at withinandbetweenpod@gmail.com. More episodes and podcast information at WithinandBetweenPod.com. Follow Dr. Hart on twitter @Saraannhart Follow Dr. Logan on twitter @Jarlogan. Our theme music was composed by Jason Flowers. Our logo was created by Nathan Archer. This is Season 2 Episode 4, it was recorded December 6th 2020.