| Both sides previous revisionPrevious revision | |
| statistics:interrater_agreement [2026/08/21 14:50] – Boxes: <wrap> renders a span, use uppercase <WRAP>; one box per list, not per bullet karel.kubicek.claude | statistics:interrater_agreement [2026/09/11 02:32] (current) – Cross-link the new sibling statistics:annotation and say where the split falls. Authored by Claude karel.kubicek.claude |
|---|
| A statistics textbook will tell you what Cohen's kappa is. It will not tell you the things that decide whether your number means anything in **this** setting: that the class you are labelling is usually rare, which is exactly the condition under which kappa collapses while your coders are in near-perfect agreement; that your coders are almost always the paper's own authors, who wrote the codebook and know the hypothesis; that the sample you double-code is small enough that the coefficient is a range rather than a value; and that since 2024 the second "rater" is increasingly a language model, which changes what the coefficient is even measuring. | A statistics textbook will tell you what Cohen's kappa is. It will not tell you the things that decide whether your number means anything in **this** setting: that the class you are labelling is usually rare, which is exactly the condition under which kappa collapses while your coders are in near-perfect agreement; that your coders are almost always the paper's own authors, who wrote the codebook and know the hypothesis; that the sample you double-code is small enough that the coefficient is a range rather than a value; and that since 2024 the second "rater" is increasingly a language model, which changes what the coefficient is even measuring. |
| |
| This page assumes you know what a codebook is and roughly what kappa does. It is about four things: how much of this field reports agreement at all, which coefficient to use and when the usual one is the wrong one, what kappa does to a rare-class labelling task, and how to report agreement when an LLM did the first pass. | This page assumes you know what a codebook is and roughly what kappa does, and it is the coefficient half of a pair: [[Statistics:Annotation]] is the validation half, and owns the question of what the labels were checked against. It is about four things: how much of this field reports agreement at all, which coefficient to use and when the usual one is the wrong one, what kappa does to a rare-class labelling task, and how to report agreement when an LLM did the first pass. |
| |
| <WRAP important> | <WRAP important> |
| ===== Related Pages ===== | ===== Related Pages ===== |
| |
| | * [[Statistics:Annotation]] — the sibling page. It owns the //validation design// — what you check the labels against, whether that source is independent, why a held-out set drawn from a crawl usually is not held out, and what an LLM annotator inherits from being a classifier. This page owns the coefficients; that one owns everything around them, and neither repeats the other. |
| * [[Design:Website classification]] — the largest hand-coding task most readers of this site will run, and where a codebook and a kappa are expected. | * [[Design:Website classification]] — the largest hand-coding task most readers of this site will run, and where a codebook and a kappa are expected. |
| * [[Privacy:Cookies]] and [[Privacy:Requests]] — the classification tasks whose ground truth is usually hand-labelled, and whose class balance is exactly the lopsided case above. | * [[Privacy:Cookies]] and [[Privacy:Requests]] — the classification tasks whose ground truth is usually hand-labelled, and whose class balance is exactly the lopsided case above. |
