International Journal of Qualitative Methods
We outline a process for using large coder teams (10 + coders) to code large-scale qualitative data sets. The process reflects experience recruiting and managing large teams of novice and trainee coders for 18 projects in the last decade, each engaging a coding team of 12 (minimum) to 54 (maximum) coders. We identify four unique challenges to large coder teams that are not presently discussed in the methodological literature: (1) recruiting and training coders, (2) providing coder compensation and incentives, (3) maintaining data quality and ensuring coding reliability at scale, and (4) building team cohesion and morale. For each challenge, we provide associated guidance. We conclude with a discussion of advantages and disadvantages of large coder teams for qualitative research and provide notes of caution for anyone considering hiring and/or managing large coder teams for research (whether in academia, government and non-profit sectors, or industry).
National Science Foundation
collaborative research, intercoder agreement, intercoder reliability, qualitative coding, qualitative data analysis, team-based coding, text analysis
Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial 4.0 License
Melissa Beresford, Amber Wutich, Margaret V. du Bray, Alissa Ruth, Rhian Stotts, Cindi SturtzSreetharan, and Alexandra Brewis. "Coding Qualitative Data at Scale: Guidance for Large Coder Teams Based on 18 Studies" International Journal of Qualitative Methods (2022). https://doi.org/10.1177/16094069221075860