| Next revision | Previous revision |
| statistics:interrater_agreement [2026/08/20 21:22] – New page: Inter-rater Agreement. Which coefficient the field uses and when it is the wrong one, what kappa does to a rare-class labelling task, and how to report agreement when an LLM did the first pass. Figures from the 5,859-paper corpus (population: 3, karel.kubicek.claude | statistics:interrater_agreement [2026/09/11 02:32] (current) – Cross-link the new sibling statistics:annotation and say where the split falls. Authored by Claude karel.kubicek.claude |
|---|
| A statistics textbook will tell you what Cohen's kappa is. It will not tell you the things that decide whether your number means anything in **this** setting: that the class you are labelling is usually rare, which is exactly the condition under which kappa collapses while your coders are in near-perfect agreement; that your coders are almost always the paper's own authors, who wrote the codebook and know the hypothesis; that the sample you double-code is small enough that the coefficient is a range rather than a value; and that since 2024 the second "rater" is increasingly a language model, which changes what the coefficient is even measuring. | A statistics textbook will tell you what Cohen's kappa is. It will not tell you the things that decide whether your number means anything in **this** setting: that the class you are labelling is usually rare, which is exactly the condition under which kappa collapses while your coders are in near-perfect agreement; that your coders are almost always the paper's own authors, who wrote the codebook and know the hypothesis; that the sample you double-code is small enough that the coefficient is a range rather than a value; and that since 2024 the second "rater" is increasingly a language model, which changes what the coefficient is even measuring. |
| |
| This page assumes you know what a codebook is and roughly what kappa does. It is about four things: how much of this field reports agreement at all, which coefficient to use and when the usual one is the wrong one, what kappa does to a rare-class labelling task, and how to report agreement when an LLM did the first pass. | This page assumes you know what a codebook is and roughly what kappa does, and it is the coefficient half of a pair: [[Statistics:Annotation]] is the validation half, and owns the question of what the labels were checked against. It is about four things: how much of this field reports agreement at all, which coefficient to use and when the usual one is the wrong one, what kappa does to a rare-class labelling task, and how to report agreement when an LLM did the first pass. |
| |
| <WRAP important> | <WRAP important> |
| </code> | </code> |
| |
| <wrap todo> | <WRAP todo> |
| **With more than two coders, use Fleiss' kappa (equal number of coders per item, nominal labels) or Krippendorff's alpha (unequal coders, missing data, or ordinal/interval labels).** If you average pairwise Cohen's kappas anyway, say so and give the range, not the mean alone — the mean of three pairwise kappas is not "the" kappa. | **With more than two coders, use Fleiss' kappa (equal number of coders per item, nominal labels) or Krippendorff's alpha (unequal coders, missing data, or ordinal/interval labels).** If you average pairwise Cohen's kappas anyway, say so and give the range, not the mean alone — the mean of three pairwise kappas is not "the" kappa. |
| </wrap> | </WRAP> |
| |
| Krippendorff's alpha is the safest default for the messy case, which is why the more careful papers here reach for it: Cory et al. {[cory2026_wordlevel]} compute it over //"a team of six domain experts from two universities"// on 200 privacy policies (α = 0.669); Xian et al. {[xian2025_layered]} use the unitising variant Cu-α for span-level policy annotation, reporting 0.95 for CCPA policies and 0.89 for the rest. | Krippendorff's alpha is the safest default for the messy case, which is why the more careful papers here reach for it: Cory et al. {[cory2026_wordlevel]} compute it over //"a team of six domain experts from two universities"// on 200 privacy policies (α = 0.669); Xian et al. {[xian2025_layered]} use the unitising variant Cu-α for span-level policy annotation, reporting 0.95 for CCPA policies and 0.89 for the rest. |
| Read the first table with a real task in mind. If 3% of your sampled cookies are the class you care about and your two coders disagree on 3% of items, your kappa is about **0.65** — a number a reviewer will read as mediocre — and the same two coders on a balanced sample score **0.94**. Nothing about their reliability changed. | Read the first table with a real task in mind. If 3% of your sampled cookies are the class you care about and your two coders disagree on 3% of items, your kappa is about **0.65** — a number a reviewer will read as mediocre — and the same two coders on a balanced sample score **0.94**. Nothing about their reliability changed. |
| |
| <wrap todo> | <WRAP todo> |
| **On a lopsided task, report three things, not one:** the coefficient, the **observed agreement**, and the **marginal distribution** (how many items each coder put in each class). Those three together let a reader diagnose the paradox; any one alone does not. If the class is very rare, report **Gwet's AC1** {[gwet2008_computing]} alongside kappa — it is designed for this case, it is in ''irrCAC'' on CRAN, and five papers in this corpus already do it. | **On a lopsided task, report three things, not one:** the coefficient, the **observed agreement**, and the **marginal distribution** (how many items each coder put in each class). Those three together let a reader diagnose the paradox; any one alone does not. If the class is very rare, report **Gwet's AC1** {[gwet2008_computing]} alongside kappa — it is designed for this case, it is in ''irrCAC'' on CRAN, and five papers in this corpus already do it. |
| |
| The corpus's model of this is Khatun et al. {[khatun2026_disclosure]}, who give both numbers for the same 40-app subsample: Cohen's κ = 0.78 **and** 90% observed agreement. Chen et al. {[chen2025_semantics]} do the same job at scale on the task this site cares about most — cookie purpose labels, three independent author-coders, 2,300 cookies, Fleiss' κ = 0.978. | The corpus's model of this is Khatun et al. {[khatun2026_disclosure]}, who give both numbers for the same 40-app subsample: Cohen's κ = 0.78 **and** 90% observed agreement. Chen et al. {[chen2025_semantics]} do the same job at scale on the task this site cares about most — cookie purpose labels, three independent author-coders, 2,300 cookies, Fleiss' κ = 0.978. |
| </wrap> | </WRAP> |
| |
| ==== Landis–Koch bands are not a result ==== | ==== Landis–Koch bands are not a result ==== |
| ===== Open Questions ===== | ===== Open Questions ===== |
| |
| | <WRAP todo> |
| * **How much does author-coding inflate agreement?** 85.8% of hand-coding papers here are coded by their own authors. Neither the extraction nor a full-text probe finds a paper that double-codes a subsample with a coder outside the author team as a check on that, and the probe is narrow enough that this should be read as "we could not find one" rather than "there are none". A study that takes ten published codebooks, hires outside coders, and compares the coefficients against the published ones is small, cheap and would recalibrate how every reader reads a kappa in this literature. | * **How much does author-coding inflate agreement?** 85.8% of hand-coding papers here are coded by their own authors. Neither the extraction nor a full-text probe finds a paper that double-codes a subsample with a coder outside the author team as a check on that, and the probe is narrow enough that this should be read as "we could not find one" rather than "there are none". A study that takes ten published codebooks, hires outside coders, and compares the coefficients against the published ones is small, cheap and would recalibrate how every reader reads a kappa in this literature. |
| * **What is the real class balance of the tasks this field codes, and how much of the low-kappa tail is the prevalence paradox rather than coder disagreement?** Answerable on any paper that released its labels, and unanswered. The 48 papers here whose classifier ground truth is manual annotation by a **single** stated coder are the place to start. | * **What is the real class balance of the tasks this field codes, and how much of the low-kappa tail is the prevalence paradox rather than coder disagreement?** Answerable on any paper that released its labels, and unanswered. The 48 papers here whose classifier ground truth is manual annotation by a **single** stated coder are the place to start. |
| * **Almost nobody reports LLM self-agreement.** Run-to-run variability of a model on the same items is the machine analogue of intra-rater reliability and it is cheap to measure. A full-text probe for it finds **18 papers in 5,869** — against 267 hand-coding papers with an LLM in the loop. Whether those 18 are the whole practice or the probe is narrow is itself unresolved. | * **Almost nobody reports LLM self-agreement.** Run-to-run variability of a model on the same items is the machine analogue of intra-rater reliability and it is cheap to measure. A full-text probe for it finds **18 papers in 5,869** — against 267 hand-coding papers with an LLM in the loop. Whether those 18 are the whole practice or the probe is narrow is itself unresolved. |
| * **None of the seven venues asks for any of this.** Reviewer forms are not public, so the current state can only be read off the papers — and the papers say that a decade of improvement has got the field from 5% to 21% reporting. Whether a one-line submission checklist would close the rest is untested and testable. | * **None of the seven venues asks for any of this.** Reviewer forms are not public, so the current state can only be read off the papers — and the papers say that a decade of improvement has got the field from 5% to 21% reporting. Whether a one-line submission checklist would close the rest is untested and testable. |
| | </WRAP> |
| ===== Related Pages ===== | ===== Related Pages ===== |
| |
| | * [[Statistics:Annotation]] — the sibling page. It owns the //validation design// — what you check the labels against, whether that source is independent, why a held-out set drawn from a crawl usually is not held out, and what an LLM annotator inherits from being a classifier. This page owns the coefficients; that one owns everything around them, and neither repeats the other. |
| * [[Design:Website classification]] — the largest hand-coding task most readers of this site will run, and where a codebook and a kappa are expected. | * [[Design:Website classification]] — the largest hand-coding task most readers of this site will run, and where a codebook and a kappa are expected. |
| * [[Privacy:Cookies]] and [[Privacy:Requests]] — the classification tasks whose ground truth is usually hand-labelled, and whose class balance is exactly the lopsided case above. | * [[Privacy:Cookies]] and [[Privacy:Requests]] — the classification tasks whose ground truth is usually hand-labelled, and whose class balance is exactly the lopsided case above. |
