When two or more researchers code the same qualitative data, they will not always agree. One coder may label an interview extract as "work-life conflict", while another labels it "workload". Inter-rater reliability is a way of measuring how consistently different coders apply the same coding scheme. It is commonly reported in content analysis and in some forms of qualitative and mixed-methods research, and examiners or journal reviewers may ask for it.
This post explains what inter-rater reliability is, when it is appropriate, the most common measures, how to calculate them with a worked example, and how to report your results.
What is inter-rater reliability?
Inter-rater reliability (also called inter-coder reliability or inter-coder agreement) is the extent to which independent coders, working with the same data and the same codebook, assign the same codes to the same units of data. High agreement suggests that your codes are clearly defined and that your coding is not simply a matter of one person's interpretation.
Some authors distinguish between agreement (how often coders assign exactly the same code) and reliability (agreement adjusted for chance). In practice, the terms are often used interchangeably, but it is good to be precise in your methodology chapter.
Is inter-rater reliability always appropriate?
Not always. Whether you should calculate it depends on your research approach:
- Often appropriate: qualitative content analysis, deductive or codebook-based coding, studies that count code frequencies, and large projects with several coders.
- Often not appropriate: interpretive approaches such as reflexive thematic analysis, interpretative phenomenological analysis and many forms of grounded theory. In these approaches, the researcher's interpretation is treated as a resource, and some methodologists argue that agreement statistics do not fit the underlying philosophy.
If you work in an interpretive tradition, you can still demonstrate rigour through other strategies, such as peer debriefing, an audit trail, reflexive memos, member checking and discussion of coding with a supervisor. Explain your choice clearly so examiners understand why you did or did not calculate inter-rater reliability.
Common measures of inter-rater reliability
Percentage agreement
The simplest measure: the number of coding decisions on which coders agree, divided by the total number of decisions. It is easy to understand but does not account for agreement that could occur by chance, so it tends to overestimate reliability, especially when there are few codes or one code is used very often.
Cohen's kappa
Cohen's kappa measures agreement between two coders for categorical codes, adjusting for chance agreement. The formula is:
kappa = (Po − Pe) / (1 − Pe)
where Po is the observed proportion of agreement and Pe is the proportion of agreement expected by chance.
Fleiss' kappa
An extension of the kappa approach for situations where more than two coders rate the data.
Krippendorff's alpha
A flexible measure that works with any number of coders, different types of data (nominal, ordinal, interval and ratio) and missing data. It is widely recommended in content analysis.
How to interpret the values
Kappa values range from below 0 (less agreement than chance) to 1 (perfect agreement). A frequently cited set of benchmarks from Landis and Koch (1977) is:
- Below 0: poor agreement
- 0.00 to 0.20: slight
- 0.21 to 0.40: fair
- 0.41 to 0.60: moderate
- 0.61 to 0.80: substantial
- 0.81 to 1.00: almost perfect
For Krippendorff's alpha, Krippendorff suggests relying on variables with alpha of at least 0.800, and considering values from 0.667 only for tentative conclusions. These benchmarks are guidelines, not universal rules, so cite the source you use.
A worked example of Cohen's kappa
Two coders independently decide whether each of 50 interview extracts reflects the code "job insecurity" (Yes or No).
- Both coders said Yes: 25 extracts
- Both coders said No: 17 extracts
- Coder A said Yes, Coder B said No: 5 extracts
- Coder A said No, Coder B said Yes: 3 extracts
Step 1: Calculate observed agreement (Po)
The coders agreed on 25 + 17 = 42 extracts. Po = 42 / 50 = 0.84, or 84% agreement.
Step 2: Calculate chance agreement (Pe)
Coder A said Yes to 30 extracts (0.60) and No to 20 (0.40). Coder B said Yes to 28 extracts (0.56) and No to 22 (0.44).
Pe = (0.60 × 0.56) + (0.40 × 0.44) = 0.336 + 0.176 = 0.512
Step 3: Calculate kappa
kappa = (0.84 − 0.512) / (1 − 0.512) = 0.328 / 0.488 ≈ 0.67
Although percentage agreement was 84%, the chance-corrected kappa is about 0.67, which falls in the "substantial" range using the Landis and Koch benchmarks. This shows why kappa is usually preferred over simple percentage agreement.
Calculating inter-rater reliability with software
- SPSS: arrange each coder's decisions in separate columns, then use Analyze > Descriptive Statistics > Crosstabs, click Statistics and tick Kappa. This gives Cohen's kappa for two coders.
- NVivo: the Coding Comparison query compares the coding of two users or user groups and reports percentage agreement and kappa.
- ATLAS.ti and MAXQDA: both include inter-coder agreement tools. Check the documentation for your version to see which coefficients are available.
- R: packages such as irr provide functions for kappa and Krippendorff's alpha.
A practical process for qualitative coding
- Develop a clear codebook with code names, definitions, inclusion and exclusion criteria, and examples.
- Train the coders using a small sample of data and discuss differences.
- Decide the unit of analysis, such as sentence, paragraph or whole response, before coding.
- Code a subset independently. Many studies double-code a portion of the data, often around 10% to 25%, though the appropriate amount depends on your study.
- Calculate agreement for each code and overall.
- Discuss and resolve disagreements, and refine code definitions where needed.
- Repeat until agreement is acceptable, then code the remaining data.
- Document every round in your audit trail.
How to report inter-rater reliability
In your methodology chapter, report who coded the data, how much of the data was double-coded, the measure you used and why, the values obtained (overall and, ideally, for each code), the benchmark you applied, and how disagreements were resolved. For example:
"Two researchers independently coded 20% of the transcripts using the final codebook. Inter-coder reliability was assessed using Cohen's kappa, which ranged from 0.68 to 0.85 across codes. Disagreements were discussed and resolved by consensus before the remaining transcripts were coded."
Replace the details with your own results.
Common mistakes to avoid
- Reporting only percentage agreement without chance correction.
- Calculating reliability on data the coders had already discussed.
- Reporting one overall value that hides very low agreement on some codes.
- Not defining the unit of analysis.
- Applying inter-rater statistics to an approach that is philosophically opposed to them without explanation.
Final thoughts
Inter-rater reliability shows that your coding scheme can be applied consistently by different people. Decide first whether it fits your methodological approach. If it does, build a clear codebook, double-code a meaningful subset of data, use a chance-corrected measure such as Cohen's kappa or Krippendorff's alpha, and report your process transparently. If it does not, explain why and demonstrate rigour through other well-documented strategies.