| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| statistics:interrater_agreement [2026/08/21 08:33] – [Open Questions] karelkubicek | statistics:interrater_agreement [2026/09/11 02:32] (current) – Cross-link the new sibling statistics:annotation and say where the split falls. Authored by Claude karel.kubicek.claude |
|---|
| A statistics textbook will tell you what Cohen's kappa is. It will not tell you the things that decide whether your number means anything in **this** setting: that the class you are labelling is usually rare, which is exactly the condition under which kappa collapses while your coders are in near-perfect agreement; that your coders are almost always the paper's own authors, who wrote the codebook and know the hypothesis; that the sample you double-code is small enough that the coefficient is a range rather than a value; and that since 2024 the second "rater" is increasingly a language model, which changes what the coefficient is even measuring. | A statistics textbook will tell you what Cohen's kappa is. It will not tell you the things that decide whether your number means anything in **this** setting: that the class you are labelling is usually rare, which is exactly the condition under which kappa collapses while your coders are in near-perfect agreement; that your coders are almost always the paper's own authors, who wrote the codebook and know the hypothesis; that the sample you double-code is small enough that the coefficient is a range rather than a value; and that since 2024 the second "rater" is increasingly a language model, which changes what the coefficient is even measuring. |
| |
| This page assumes you know what a codebook is and roughly what kappa does. It is about four things: how much of this field reports agreement at all, which coefficient to use and when the usual one is the wrong one, what kappa does to a rare-class labelling task, and how to report agreement when an LLM did the first pass. | This page assumes you know what a codebook is and roughly what kappa does, and it is the coefficient half of a pair: [[Statistics:Annotation]] is the validation half, and owns the question of what the labels were checked against. It is about four things: how much of this field reports agreement at all, which coefficient to use and when the usual one is the wrong one, what kappa does to a rare-class labelling task, and how to report agreement when an LLM did the first pass. |
| |
| <WRAP important> | <WRAP important> |
| </code> | </code> |
| |
| <wrap todo> | <WRAP todo> |
| **With more than two coders, use Fleiss' kappa (equal number of coders per item, nominal labels) or Krippendorff's alpha (unequal coders, missing data, or ordinal/interval labels).** If you average pairwise Cohen's kappas anyway, say so and give the range, not the mean alone — the mean of three pairwise kappas is not "the" kappa. | **With more than two coders, use Fleiss' kappa (equal number of coders per item, nominal labels) or Krippendorff's alpha (unequal coders, missing data, or ordinal/interval labels).** If you average pairwise Cohen's kappas anyway, say so and give the range, not the mean alone — the mean of three pairwise kappas is not "the" kappa. |
| </wrap> | </WRAP> |
| |
| Krippendorff's alpha is the safest default for the messy case, which is why the more careful papers here reach for it: Cory et al. {[cory2026_wordlevel]} compute it over //"a team of six domain experts from two universities"// on 200 privacy policies (α = 0.669); Xian et al. {[xian2025_layered]} use the unitising variant Cu-α for span-level policy annotation, reporting 0.95 for CCPA policies and 0.89 for the rest. | Krippendorff's alpha is the safest default for the messy case, which is why the more careful papers here reach for it: Cory et al. {[cory2026_wordlevel]} compute it over //"a team of six domain experts from two universities"// on 200 privacy policies (α = 0.669); Xian et al. {[xian2025_layered]} use the unitising variant Cu-α for span-level policy annotation, reporting 0.95 for CCPA policies and 0.89 for the rest. |
| Read the first table with a real task in mind. If 3% of your sampled cookies are the class you care about and your two coders disagree on 3% of items, your kappa is about **0.65** — a number a reviewer will read as mediocre — and the same two coders on a balanced sample score **0.94**. Nothing about their reliability changed. | Read the first table with a real task in mind. If 3% of your sampled cookies are the class you care about and your two coders disagree on 3% of items, your kappa is about **0.65** — a number a reviewer will read as mediocre — and the same two coders on a balanced sample score **0.94**. Nothing about their reliability changed. |
| |
| <wrap todo> | <WRAP todo> |
| **On a lopsided task, report three things, not one:** the coefficient, the **observed agreement**, and the **marginal distribution** (how many items each coder put in each class). Those three together let a reader diagnose the paradox; any one alone does not. If the class is very rare, report **Gwet's AC1** {[gwet2008_computing]} alongside kappa — it is designed for this case, it is in ''irrCAC'' on CRAN, and five papers in this corpus already do it. | **On a lopsided task, report three things, not one:** the coefficient, the **observed agreement**, and the **marginal distribution** (how many items each coder put in each class). Those three together let a reader diagnose the paradox; any one alone does not. If the class is very rare, report **Gwet's AC1** {[gwet2008_computing]} alongside kappa — it is designed for this case, it is in ''irrCAC'' on CRAN, and five papers in this corpus already do it. |
| |
| The corpus's model of this is Khatun et al. {[khatun2026_disclosure]}, who give both numbers for the same 40-app subsample: Cohen's κ = 0.78 **and** 90% observed agreement. Chen et al. {[chen2025_semantics]} do the same job at scale on the task this site cares about most — cookie purpose labels, three independent author-coders, 2,300 cookies, Fleiss' κ = 0.978. | The corpus's model of this is Khatun et al. {[khatun2026_disclosure]}, who give both numbers for the same 40-app subsample: Cohen's κ = 0.78 **and** 90% observed agreement. Chen et al. {[chen2025_semantics]} do the same job at scale on the task this site cares about most — cookie purpose labels, three independent author-coders, 2,300 cookies, Fleiss' κ = 0.978. |
| </wrap> | </WRAP> |
| |
| ==== Landis–Koch bands are not a result ==== | ==== Landis–Koch bands are not a result ==== |
| ===== Related Pages ===== | ===== Related Pages ===== |
| |
| | * [[Statistics:Annotation]] — the sibling page. It owns the //validation design// — what you check the labels against, whether that source is independent, why a held-out set drawn from a crawl usually is not held out, and what an LLM annotator inherits from being a classifier. This page owns the coefficients; that one owns everything around them, and neither repeats the other. |
| * [[Design:Website classification]] — the largest hand-coding task most readers of this site will run, and where a codebook and a kappa are expected. | * [[Design:Website classification]] — the largest hand-coding task most readers of this site will run, and where a codebook and a kappa are expected. |
| * [[Privacy:Cookies]] and [[Privacy:Requests]] — the classification tasks whose ground truth is usually hand-labelled, and whose class balance is exactly the lopsided case above. | * [[Privacy:Cookies]] and [[Privacy:Requests]] — the classification tasks whose ground truth is usually hand-labelled, and whose class balance is exactly the lopsided case above. |
