Measure The Web

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Inter-rater Agreement ======

Sooner or later a web measurement stops being automatic. You classify two hundred consent banners by hand to check the detector, or you read four hundred privacy policies, or you decide which of five thousand third-party domains are advertising. That hand-coded set becomes ground truth, and everything downstream — the classifier's reported accuracy, the "we found 41% of sites" headline — inherits whatever error is in it. A reviewer who is doing their job will ask how reliable it was, and the answer is a number.

A statistics textbook will tell you what Cohen's kappa is. It will not tell you the things that decide whether your number means anything in **this** setting: that the class you are labelling is usually rare, which is exactly the condition under which kappa collapses while your coders are in near-perfect agreement; that your coders are almost always the paper's own authors, who wrote the codebook and know the hypothesis; that the sample you double-code is small enough that the coefficient is a range rather than a value; and that since 2024 the second "rater" is increasingly a language model, which changes what the coefficient is even measuring.

This page assumes you know what a codebook is and roughly what kappa does, and it is the coefficient half of a pair: [[Statistics:Annotation]] is the validation half, and owns the question of what the labels were checked against. It is about four things: how much of this field reports agreement at all, which coefficient to use and when the usual one is the wrong one, what kappa does to a rare-class labelling task, and how to report agreement when an LLM did the first pass.

<WRAP important>
**Four papers in five that hand-code data report no reliability statistic at all. Where one is reported it is usually Cohen's kappa, a two-rater statistic, and a quarter of the papers using it had more than two raters. Almost nobody reports an interval.**

Of the 5,859 papers extracted from seven security, privacy and measurement venues (2010–2026), **3,318 hand-coded something**. Of those:

  * **512 (15.4%) name an agreement statistic.** A further 97 (2.9%) give an agreement *number* without saying what statistic it is. **2,709 (81.6%) report neither.**
  * **1,288 (38.8%) say how many people did the coding.** Of the 1,176 that state two or more coders — where an agreement statistic is even possible — only **494 (42.0%)** report one.
  * **2,848 (85.8%) were coded by the authors themselves.** Domain experts are next at 340 (10.2%); crowdworkers, students and hired annotators together account for 168 (5.1%).

The reporting is improving, and that is not an artefact of the corpus growing: the share of hand-coding papers naming a statistic goes **5.2% (2010–2014) → 9.9% → 17.2% → 21.3% (2025–2026, provisional)**, and the share stating a coder count goes **18.8% → 49.5%** over the same periods.

Which statistic, of the 512 that name one (multi-valued, so shares exceed 100%): **Cohen's kappa 252 (49.2%)**, **Krippendorff's alpha 84 (16.4%)**, **raw uncorrected agreement 80 (15.6%)**, **Fleiss' kappa 75 (14.6%)**. All four require folding — the field writes them 133 different ways, and ranking exact strings puts Cohen's kappa at 170 rather than 252, a 32.5% undercount.

Two things nobody does. **245 papers name Cohen's kappa and no multi-rater coefficient; of the 236 of those that state a coder count, 61 (25.8%) had more than two coders**, for whom Cohen's kappa is not defined without a further choice the papers do not state. And across **all 5,869 full texts, a probe requiring the string "95%" near the coefficient finds two papers that put an interval next to a kappa** — against 230 that write the phrase "Cohen's kappa". A kappa on 25 double-coded items has a 95% interval about a third of the coefficient's usable width; **of the 457 papers that name a statistic and also state a coded sample size, the median sample is 385 items and 100 (21.9%) code fewer than 50**.

And the currency point. The share of hand-coding papers with a language model somewhere in the annotation loop goes **2.3% (2023) → 11.1% (2024) → 22.1% (2025) → 40.6% (2026, provisional)**. Those papers report agreement at **34.1%**, against **13.8%** for the rest — LLM-assisted annotation is the part of this literature that reports reliability best — plausibly because the human coding is now explicitly a validation step, though that reading is an interpretation and not something the corpus tests. But what they are computing is human-versus-machine agreement, which is a different quantity from inter-rater reliability and does not license the same conclusion. See [[#When the Second Rater Is a Language Model]].
</WRAP>

===== What to Read First =====

  * **Hallgren 2012** {[hallgren2012_computing]} — //"Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial"//. Twelve pages, free, worked examples with code. If you read one thing, read this: it is the only item on this list that will get you from "I have two coders" to a defensible number in an afternoon.
  * **Krippendorff 2004** {[krippendorff2004_reliability]} — //"Reliability in Content Analysis: Some Common Misconceptions and Recommendations"//. The source of the α ≥ 0.800 / ≥ 0.667 thresholds this field quotes without attribution, and — more usefully — of the argument for why you should have decided the threshold before you saw the data. The book {[krippendorff2018_content]} is the long version.
  * **Feinstein & Cicchetti 1990** {[feinstein1990_paradoxes]} and its companion {[cicchetti1990_resolving]} — //"High agreement but low kappa"//, parts I and II. Part I is the problem this page's simulation reproduces; part II is what to do about it. **Two papers in the whole corpus name this paradox**, and it is the single most likely thing to be wrong with a kappa computed on web-measurement labels, where the positive class is usually rare.
  * **Landis & Koch 1977** {[landis1977_measurement]} — the source of "slight / fair / moderate / substantial / almost perfect". Read it to learn that its authors called the divisions //"clearly arbitrary"//. **67 papers here put one of those verbal bands within 120 characters of a coefficient; 7 cite the paper.**
  * **McHugh 2012** {[mchugh2012_interrater]} — a short clinical-methods review of kappa that is unusually direct about its failure modes and proposes stricter bands than Landis–Koch. Useful as the citation for "we treated 0.6 as inadequate".
  * **Gwet 2008** {[gwet2008_computing]} — AC1, the coefficient designed for the high-agreement/low-kappa case, with a variance estimator. **Five papers in the corpus use it.** If your labels are 95% one class, this is the literature you want.
  * **Tang, Bauer & Christin** {[tang2025_misuse]} (SOUPS 2025) — a systematic review of statistical reporting in an adjacent community. SOUPS is not one of this page's seven venues, which is why it is the external anchor; its finding that //"58% of papers lack information to verify whether an assertion is supported"// is the same shape as the reporting gap measured here.
  * **Pangakis, Wolken & Fasching 2023** {[pangakis2023_validation]} and **Törnberg 2024** {[tornberg2024_bestpractices]} — validation protocols for LLM annotation. Read these before you let a model do your first pass; Törnberg's is the more prescriptive.

===== How Much of the Field Reports Anything =====

The population for everything in this section is the **3,318 papers that hand-coded something** — that is, that have at least one ''humanAnnotation'' record. A paper with no manual coding has no reliability to report, and including it would manufacture silence. Papers are counted once each, never once per mention.

^ Reported ^ Papers ^ Share of 3,318 ^
| Names an agreement statistic | 512 | 15.4% |
| Gives an agreement number, names no statistic | 97 | 2.9% |
| Neither | 2,709 | 81.6% |

The 81.6% is not all a defect. Half of this coding is ''binary-verification'' — one person eyeballing a sample of the detector's output to confirm it did what it should — and a large share of that is genuinely single-coder work where reliability is not defined. What the extraction cannot separate is "single coder, correctly said so" from "several coders, silently". The count of coders is the field's real gap:

^ Coders ^ Papers ^ Share of the 1,288 stating a count ^
| 1 — no second coder, agreement not computable | 112 | 8.7% |
| 2 | 591 | 45.9% |
| 3 | 319 | 24.8% |
| 4–5 | 124 | 9.6% |
| 6–10 | 64 | 5.0% |
| more than 10 | 78 | 6.1% |

**2,030 of the 3,318 (61.2%) never say how many people coded.** Restrict to the 1,176 that state two or more — the papers for which a reliability statistic exists and is meaningful — and **42.0% report one**. That is the honest version of the headline: not "85% of hand-coding papers are unreliable", but "in three cases out of five the reader cannot tell whether reliability was even possible, and where it clearly was, more than half still do not report it."

==== The crawl subset is worse, not better ====

864 of the hand-coding papers also ran an automated web crawl — the population this site is about. They report agreement at **15.9%** (against 15.4% overall), state a coder count at **35.9%** (against 38.8%), and were coded by the authors themselves in **86.6%** of cases. The literature closest to this site's readership is not doing better than the average: the three differences are small and point in opposite directions, and this page does not test them.

==== Who codes, and why it matters here ====

^ Annotator type ^ Papers ^ Share of 3,318 ^
| authors | 2,848 | 85.8% |
| domain experts | 340 | 10.2% |
| ''not-stated'' //(sentinel)// | 177 | 5.3% |
| mixed | 109 | 3.3% |
| crowdworkers | 73 | 2.2% |
| students | 60 | 1.8% |
| hired annotators | 35 | 1.1% |
| legal experts | 5 | 0.2% |

//(Multi-valued: a paper with two coding tasks can appear twice.)//

Two authors of the same paper, working from a codebook they wrote, who have discussed the hypothesis over eight months, are not two independent raters in the sense the coefficient assumes. This does not make the number worthless — it is still evidence the codebook is applicable — but it is an **upper bound** on what an outside coder would achieve, and it should be presented that way. Nothing in the corpus lets you quantify the inflation. The extraction has no field for "was the second coder outside the author team", and a deliberately narrow full-text probe for it returns **zero papers in 5,869** — which is evidence that the practice is rare and is **not** evidence that it never happens. See [[#Open Questions]].

The corpus's best-documented case of taking this seriously is Coopamootoo & Groß {[coopamootoo2017_forgotten]}, who separate the author-coder from the trained outside coders: //"each unit was independently coded by two coders. The coding assignments were done asymmetrically, with one of the authors coding all units"//, and then recruit //"two local PhD students ... trialled on a sample of four units and trained in content analysis"// and compute the coefficient on that pair instead. It is also one of only two papers a full-text probe locates in 5,869 that publish an interval alongside the coefficient: **κ = .666, 95% CI [.630, .670]**.

===== Which Coefficient, and When the Usual One Is Wrong =====

^ Coefficient family (folded) ^ Papers ^ Share of the 512 naming one ^ Raters ^ Data ^
| Cohen's kappa {[cohen1960_coefficient]} | 252 | 49.2% | exactly 2 | nominal |
| Krippendorff's alpha {[krippendorff2004_reliability]} | 84 | 16.4% | any, missing data allowed | nominal, ordinal, interval, ratio |
| raw agreement, no chance correction | 80 | 15.6% | any | any |
| Fleiss' kappa {[fleiss1971_measuring]} | 75 | 14.6% | any, fixed number per item | nominal |
| not an agreement coefficient (F1, Pearson, accuracy) | 12 | 2.3% | — | — |
| kappa, variant never named | 10 | 2.0% | unknown | unknown |
| Kupper–Hafner | 9 | 1.8% | any | nominal |
| Gwet's AC1 / AC2 {[gwet2008_computing]} | 5 | 1.0% | any | nominal |
| intra-class correlation | 1 | 0.2% | any | continuous |

**418 (81.6%) name at least one chance-corrected coefficient; 77 (15.0%) name only an uncorrected agreement rate or something that is not an agreement statistic at all.** Reporting "the two coders agreed on 93%" and stopping is the second most common failure on this page after not reporting anything, and it is the one that most often survives review, because 93% sounds decisive.

==== Cohen's kappa with three coders ====

Cohen's kappa is defined for **exactly two** raters. **245 papers name it and no multi-rater coefficient. Of the 236 of those that state a coder count, 61 (25.8%) state more than two** — three is the usual case, but the corpus contains papers doing this with 4, 5, 15 and 30 coders. Skolka, Staicu & Pradel {[skolka2019_anything]} is a clean example of the ambiguity: five coders, and //"we compute Cohen's kappa for each pair of participants. The pairwise agreement ranges between 0.64 and 0.93, with an average of 0.81"//. That 61 is a flag for inspection, not an error count: Coopamootoo & Groß {[coopamootoo2017_forgotten]} are in it at three coders and are one of the most careful papers in the corpus, because their three coders are two separate arrangements and the coefficient is computed on the pair. Averaging pairwise Cohen's kappas is a defensible thing to do and it is almost certainly what these papers did, but it is **a different estimator from Fleiss' kappa**, and when the coders differ in skill the three pairwise values differ substantially:

<code>
C. Three coders of UNEQUAL skill: which pairwise kappa gets quoted?
   error rates 2% / 2% / 12%, balanced task, n = 1000, 200 repetitions

   lowest pairwise kappa   0.725
   mean pairwise kappa     0.794
   highest pairwise kappa  0.922   <- the number an author is tempted to quote
   spread (max - min)      0.196
</code>

<WRAP todo>
**With more than two coders, use Fleiss' kappa (equal number of coders per item, nominal labels) or Krippendorff's alpha (unequal coders, missing data, or ordinal/interval labels).** If you average pairwise Cohen's kappas anyway, say so and give the range, not the mean alone — the mean of three pairwise kappas is not "the" kappa.
</WRAP>

Krippendorff's alpha is the safest default for the messy case, which is why the more careful papers here reach for it: Cory et al. {[cory2026_wordlevel]} compute it over //"a team of six domain experts from two universities"// on 200 privacy policies (α = 0.669); Xian et al. {[xian2025_layered]} use the unitising variant Cu-α for span-level policy annotation, reporting 0.95 for CCPA policies and 0.89 for the rest.

==== The one thing that will surprise you: kappa on a rare class ====

Most web-measurement labelling tasks are lopsided. Third-party requests that are trackers, cookies that are advertising, banners that are non-compliant, sites that are phishing — the positive class is a minority, sometimes a small one. Chance-corrected coefficients are built to punish exactly that, and they punish it hard.

Kappa is (P<sub>o</sub> − P<sub>e</sub>) / (1 − P<sub>e</sub>). When one label covers almost everything, agreement by chance P<sub>e</sub> is already near 1, the denominator collapses, and kappa goes to zero **even though the coders disagreed on almost nothing**. Feinstein & Cicchetti {[feinstein1990_paradoxes]} named this in 1990; **two papers in this corpus mention it.**

The script below demonstrates it on synthetic data — the point is a property of the coefficient, not a fact about the web — and also measures how wide a kappa is at the sample sizes this field actually double-codes.

<file python kappa_on_skewed_labels.py>
#!/usr/bin/env python3
"""Why kappa collapses on a web-measurement labelling task, and how wide it is.

SYNTHETIC data, fixed seed. Nothing here is a measurement claim about the web;
the numbers are reproducible by running the file. numpy only (2.4.6 here).

Two facts that decide whether an inter-rater number means anything, both of
which bite precisely on the kind of task this field annotates -- a binary
verification pass over a crawl sample, where one class is rare.

  A. THE PREVALENCE PARADOX. Cohen's kappa is (Po - Pe) / (1 - Pe). When one
     label covers almost everything, chance agreement Pe is already near 1, so
     the denominator collapses and kappa is small even though the coders
     disagreed on almost nothing. Two coders who agree on 97% of a sample that
     is 97% negatives can score kappa near 0 -- and the same two coders on a
     balanced sample score high. The coders did not change. The base rate did.

  B. THE INTERVAL. Kappa on 50 items is not a number, it is a range. Most
     papers report it without one.

Run: python3 kappa_on_skewed_labels.py
"""

import numpy as np

RNG = np.random.default_rng(20260820)


def cohen_kappa(a, b):
    """Two raters, binary labels, no weighting."""
    n = len(a)
    po = np.mean(a == b)
    pe = sum(np.mean(a == k) * np.mean(b == k) for k in (0, 1))
    if pe == 1.0:
        return float("nan")  # both raters used one label only: kappa undefined
    return (po - pe) / (1 - pe)


def two_raters(n, prevalence, error_rate):
    """Rater A is the truth; rater B flips each item with probability
    `error_rate`. Rater skill is held CONSTANT across prevalences -- that is the
    whole point: only the base rate moves."""
    a = (RNG.random(n) < prevalence).astype(int)
    flip = RNG.random(n) < error_rate
    b = np.where(flip, 1 - a, a)
    return a, b


# ----------------------------------------------------------------- A
print("A. Same two coders, same 3% error rate, only the base rate changes")
print("   (n = 1000 items, 200 repetitions, mean over repetitions)\n")
print("   positive class   observed agreement   Cohen's kappa")
for prev in (0.50, 0.30, 0.15, 0.05, 0.03, 0.01):
    pos, ks = [], []
    for _ in range(200):
        a, b = two_raters(1000, prev, 0.03)
        pos.append(np.mean(a == b))
        ks.append(cohen_kappa(a, b))
    ks = np.array(ks)
    print(f"   {prev:>13.0%}   {np.mean(pos):>18.3f}   {np.nanmean(ks):>13.3f}")

print("\n   Observed agreement barely moves; kappa falls off a cliff. A paper")
print("   that reports only kappa on a rare-class task looks unreliable when it")
print("   is not; one that reports only observed agreement looks reliable when")
print("   it may not be. Report BOTH, plus the marginal distribution.\n")

# ----------------------------------------------------------------- B
print("B. How wide is a kappa? Balanced task, 5% error rate, 2000 repetitions")
print("   (Monte Carlo over 2000 simulated datasets)\n")
print("   double-coded n   median kappa   2.5th pct   97.5th pct   width")
for n in (25, 50, 100, 385, 1000):
    ks = np.array([cohen_kappa(*two_raters(n, 0.5, 0.05)) for _ in range(2000)])
    ks = ks[~np.isnan(ks)]
    lo, hi = np.percentile(ks, [2.5, 97.5])
    print(f"   {n:>14}   {np.median(ks):>12.3f}   {lo:>9.3f}   {hi:>10.3f}   {hi - lo:>5.3f}")

print("\n   385 is the median double-coded sample size in the corpus behind this")
print("   page; 22% of its papers double-code fewer than 50 items. At n = 25 the")
print("   95% interval is about a third of kappa's usable range and spans both")
print("   'substantial' and 'almost perfect', so a kappa from a sample that small")
print("   does not distinguish them. Quote an interval, not a Landis-Koch band.\n")

# ----------------------------------------------------------------- C
print("C. Three coders of UNEQUAL skill: which pairwise kappa gets quoted?")
print("   error rates 2% / 2% / 12%, balanced task, n = 1000, 200 repetitions\n")
trip = []
for _ in range(200):
    truth = (RNG.random(1000) < 0.5).astype(int)
    rs = [np.where(RNG.random(1000) < e, 1 - truth, truth) for e in (0.02, 0.02, 0.12)]
    pairwise = [cohen_kappa(rs[i], rs[j]) for i, j in ((0, 1), (0, 2), (1, 2))]
    trip.append((min(pairwise), float(np.mean(pairwise)), max(pairwise)))
trip = np.array(trip)
print(f"   lowest pairwise kappa   {trip[:, 0].mean():.3f}")
print(f"   mean pairwise kappa     {trip[:, 1].mean():.3f}")
print(f"   highest pairwise kappa  {trip[:, 2].mean():.3f}   <- the number an author is tempted to quote")
print(f"   spread (max - min)      {(trip[:, 2] - trip[:, 0]).mean():.3f}")
print("\n   One weaker coder is the normal case -- a student added late, an author")
print("   who coded the first pass. 'Cohen's kappa' is then three numbers, and")
print("   the corpus behind this page has 61 papers naming Cohen's kappa while")
print("   stating more than two annotators. With >2 coders the statistic is")
print("   Fleiss' kappa or Krippendorff's alpha; if you average pairwise Cohen's")
print("   kappas instead, say so, because it is a different estimator.")
</file>

Its output for parts A and B, verbatim except that the script's own interpretive paragraphs between the two tables are removed here (they are repeated in the prose around this box):

<code>
A. Same two coders, same 3% error rate, only the base rate changes
   (n = 1000 items, 200 repetitions, mean over repetitions)

   positive class   observed agreement   Cohen's kappa
             50%                0.970           0.940
             30%                0.970           0.930
             15%                0.971           0.891
              5%                0.970           0.753
              3%                0.970           0.649
              1%                0.970           0.378

B. How wide is a kappa? Balanced task, 5% error rate, 2000 repetitions
   (Monte Carlo over 2000 simulated datasets)

   double-coded n   median kappa   2.5th pct   97.5th pct   width
               25          0.918       0.679        1.000   0.321
               50          0.915       0.757        1.000   0.243
              100          0.900       0.808        0.980   0.172
              385          0.901       0.855        0.938   0.083
             1000          0.900       0.872        0.926   0.054
</code>

The 385 in that table is the median coded sample size across the **457 papers here that name an agreement statistic and also state one**; 100 of those 457 (21.9%) state fewer than 50 items.

Read the first table with a real task in mind. If 3% of your sampled cookies are the class you care about and your two coders disagree on 3% of items, your kappa is about **0.65** — a number a reviewer will read as mediocre — and the same two coders on a balanced sample score **0.94**. Nothing about their reliability changed.

<WRAP todo>
**On a lopsided task, report three things, not one:** the coefficient, the **observed agreement**, and the **marginal distribution** (how many items each coder put in each class). Those three together let a reader diagnose the paradox; any one alone does not. If the class is very rare, report **Gwet's AC1** {[gwet2008_computing]} alongside kappa — it is designed for this case, it is in ''irrCAC'' on CRAN, and five papers in this corpus already do it.

The corpus's model of this is Khatun et al. {[khatun2026_disclosure]}, who give both numbers for the same 40-app subsample: Cohen's κ = 0.78 **and** 90% observed agreement. Chen et al. {[chen2025_semantics]} do the same job at scale on the task this site cares about most — cookie purpose labels, three independent author-coders, 2,300 cookies, Fleiss' κ = 0.978.
</WRAP>

==== Landis–Koch bands are not a result ====

"κ = 0.72, indicating substantial agreement" is the standard sentence. **67 papers in this corpus put one of those verbal bands next to a coefficient and 7 cite Landis & Koch** {[landis1977_measurement]}, whose own paper describes the divisions as arbitrary and offered them for a specific clinical context. Zhou et al. {[zhou2025_secrets]} is a representative use: //"a Cohen's Kappa value of 0.91 ... indicating 'near perfect agreement' according to Landis and Koch"//.

The bands are not wrong so much as empty. They are not calibrated to your task's difficulty, its class balance or its consequences, and at the sample sizes above the interval routinely spans two of them. Krippendorff's thresholds {[krippendorff2004_reliability]} — α ≥ 0.800 for drawing conclusions, ≥ 0.667 for tentative ones — are at least explicitly a decision rule rather than a description, which is why they are the better citation if you need one. McHugh {[mchugh2012_interrater]} proposes stricter bands again and argues for choosing them by consequence.

**What actually helps a reader:** state the threshold you would have accepted **before** you computed the coefficient, and what you did when a category fell below it. Swart, den Hengst & Chen {[swart2025_detecting]} publish Fleiss' κ = 0.35 across 14 expert annotators on a linguistic-bias task and treat it as a finding about task difficulty rather than hiding it. A published low kappa with an explanation is worth more than an unpublished high one.

==== Where the reported values actually land ====

Of the 418 papers naming a chance-corrected coefficient, **387 report a value this page's parser could read**. Bucketing each paper by its **lowest** reported coefficient (papers commonly report one per category, or a range):

^ Landis–Koch band ^ Papers ^ Share of 387 ^
| 0.00–0.20 slight | 7 | 1.8% |
| 0.21–0.40 fair | 13 | 3.4% |
| 0.41–0.60 moderate | 28 | 7.2% |
| 0.61–0.80 substantial | 167 | 43.2% |
| 0.81–1.00 almost perfect | 172 | 44.4% |

Median 0.80, quartiles 0.70 and 0.86. **74 papers (19.1%) have at least one coefficient below 0.67**, Krippendorff's floor for even tentative conclusions — which is roughly one in five, and almost all of those publish the number anyway. That is the healthy behaviour. The unhealthy version is invisible here by construction: a paper that computed a low kappa and did not report it looks identical to a paper that never computed one.

===== When the Second Rater Is a Language Model =====

This is the part of the page that has changed since 2023, and it is the part where the corpus is thinnest, so treat the direction as solid and the levels as provisional.

Counting hand-coding papers with any language-model signal in their annotation — an LLM named in the annotation record, its quote, or the paper's tools:

^ Year ^ LLM signal / hand-coding papers ^ Share ^
| 2022 | 4 / 303 | 1.3% |
| 2023 | 9 / 394 | 2.3% |
| 2024 | 44 / 398 | 11.1% |
| 2025 //(provisional)// | 106 / 479 | 22.1% |
| 2026 //(provisional)// | 99 / 244 | 40.6% |

2025 and 2026 are under-represented by construction (see [[#Methodology and Limitations of These Figures]]), so the 40.6% is a share of a thin, non-random slice. The matcher also over-fires slightly on early years — it returns 1–2 papers per year in 2017, 2019, 2020 and 2021, which are false positives from words like //llama// and //Gemini// in unrelated senses, and the table above starts at 2022 for that reason rather than because the earlier rows are empty. The **shape** — a floor of noise until 2023, a step change in 2024 — survives that comfortably; the levels should not be quoted to a decimal place.

These papers report agreement statistics at **34.1%**, against **13.8%** for the rest of the hand-coding literature. The likeliest reason is structural rather than cultural: when a model does the first pass, the human coding //is// the validation, so a paper without a number has nothing to report. This is an interpretation of a difference between two groups, not a tested claim. Two shapes recur, and they are not the same thing:

  * **Humans validate the model.** Alam et al. {[alam2026_philter]}: //"Two experts independently validate all LLM-generated preliminary assessments by checking the extracted evidence and rationale against the original paper and the codebook"//, reporting 345/385 (89.6%) agreement. Cory et al. {[cory2026_wordlevel]} have six domain experts review LLM-generated annotations of 200 privacy policies.
  * **The model is treated as a rater alongside humans.** Sun, Vekaria & Nithyanand {[sun2026_suitability]} audit an LLM agent for dark patterns across 456 data-broker sites and compute //"Cohen's κ on binary (present/absent) labels for each dark pattern category, yielding κ = 51.9%"//, rising to 73.7% after calibration.

<WRAP important>
**Human–model agreement is a validity check on the model, not an inter-rater reliability of your coding.** A high kappa between one human and GPT-something tells you the model reproduces that human. It does not tell you the human was right, and it cannot — the model was trained on text written by people making the same distinctions, so agreement is partly circular. Reliability and validity come apart here in a way they do not with two humans.

The consequence for a paper: **you still need human–human agreement**, on a subsample, computed between people who did not see the model's output. Pangakis, Wolken & Fasching {[pangakis2023_validation]} make the argument at length; Törnberg {[tornberg2024_bestpractices]} turns it into a protocol; Baumann et al. {[baumann2025_llmhacking]} give the largest empirical demonstration of why it matters — replicating 37 annotation tasks from 21 published studies across 18 models, they find that ordinary variation in the annotation pipeline flips the conclusion for roughly one hypothesis in three (arXiv preprint, September 2025, not yet peer-reviewed); Ziems et al. {[ziems2024_transform]} give the scale at which LLM labels are and are not adequate substitutes. Gilardi, Alizadeh & Kubli {[gilardi2023_chatgpt]} is the paper that started the substitution argument and is worth reading for what it does **not** claim.
</WRAP>

Two mechanical traps, both visible in the corpus. First, **an LLM's labels are not deterministic**, so "agreement between the model and itself across runs" is a real and separate quantity — and unlike a human rater it is cheap to measure over many runs. A full-text probe finds **18 papers in 5,869** reporting anything of the kind, so it is rare rather than absent; the probe is narrow and that 18 is a lower bound. Second, **do not let the humans see the model's labels first.** Zhang et al. {[zhang2025_abusability]} are explicit about this: their two authors //"collaboratively annotated the IPV attacks in the selected validation set without seeing LLM responses"// before the comparison, and only then report κ = 0.96 on the 100 shortcuts two authors verified independently. A paper that does not say whether its coders saw the model's output has an anchoring problem it cannot rule out.

===== Methodology and Limitations of These Figures =====

  * **Population, always named.** Every figure here is of the **3,318 papers with at least one ''humanAnnotation'' record**, or of a stated subset of it (512 naming a statistic; 1,288 stating a coder count; 864 that also crawled). Never of 5,859. Papers are counted once each; the 3,318 papers carry 5,778 annotation records between them.
  * **Sentinels are silence, not values.** A null ''agreementMetric'' or ''annotatorCount'' is counted as not-reported and never as a category. The reporting-gap figures are therefore **bounds**: a paper that computed a kappa and left it out of the text is indistinguishable from one that never computed it.
  * **''agreementMetric'' is free text and was folded.** 133 distinct strings across the corpus — case, apostrophe glyph, the Greek letter versus the word, and several coefficients in one string. The fold is an ordered rule list with its **unmapped residue printed in full** (24 distinct strings, 29 mentions) on the provenance page. Ranking the raw strings instead puts Cohen's kappa at 170 papers rather than 252.
  * **The value parser is conservative and one-sided.** "0.68-0.96" is a range, "σ = 0.09" is a dispersion and "93%" is raw agreement, and each of those was silently corrupting the distribution until it was fixed. As a consequence a genuinely **negative** coefficient cannot be distinguished from a range separator and is never returned, so below-chance results are invisible in the value table.
  * **A cross-check on the extraction.** 252 papers fold to "Cohen's kappa"; an independent whitespace-collapsed full-text probe for the phrase finds 230. The two agree to within about 10%, with the extraction the more generous — it also catches "Cohen-Kappa" and inferred cases. Neither is exact; the ranking is what should be trusted.
  * **Quotes were spot-checked by hand.** 21 evidence quotes behind figures and named papers on this page were checked against ''paper.cols.txt''. Nine were verbatim-exact; the rest failed exact match on ellipsis elision or PDF hyphenation, and **all of them were then confirmed by locating the coefficient value and the surrounding phrase in the full text**. None was fabricated. Full list on the provenance page.
  * **Full-text probes are lower bounds.** The counts of Landis–Koch citations, kappa intervals and the paradox come from regex over 5,869 stored full texts with whitespace collapsed and soft hyphens rejoined. A paper phrasing something differently is missed. The probe script and its unedited output are on the provenance page.
  * **''humanAnnotation.sampleSize'' is the coded sample, not always the double-coded one**, and it is stated by 457 of the 512 papers naming a statistic (89.3%). A few records carry the whole labelled corpus. The median (385) is robust to that; the bucket table is not entirely, and the 22%-under-50 figure is if anything conservative.
  * **Venue coverage.** Seven venues — CCS, IMC, NDSS, PoPETs, USENIX Security, TheWebConf, IEEE S&P. **SOUPS, CHI and CSCW are absent**, and they matter more on this page than on most: they are where content analysis is a normal method and where reporting standards for it are highest. Every figure here is a claim about the seven venues. The wider community's practice is **probably** better — content analysis is a normal method at those venues and reporting standards for it are older there — but that is an inference from where the method is native, not something this corpus measures.
  * **2025 and 2026 are provisional.** CCS 2026 and IMC 2026 have not been held; IEEE S&P 2026 and TheWebConf 2026 are under-selected because selection screens on abstracts that are not yet indexed. Any trend row ending in 2026 is a share of a thin slice, which is why the LLM table is labelled rather than trimmed.
  * **Every external reference was fetched on 2026-08-20**, not recalled: 14 DOIs resolved through the Crossref and DataCite content-negotiation APIs, the PoPETs and USENIX landing pages for author lists, and the current PyPI and CRAN versions of every package named below.
  * The full query log, the fold's residue, the probe scripts and their unedited output, and the list of things that could not be established are on [[provenance:statistics:interrater_agreement]]. Corpus-level caveats are on [[Literature:Corpus]].

===== What to Report =====

  - **How many people coded, and who they were** — authors, students, hired annotators, domain experts. 61.2% of the hand-coding papers here do not say. This is one clause and it is the highest-value item on the list.
  - **How many items were double-coded**, as a count and as a share of the labelled set. The coefficient is only as precise as that number.
  - **The coefficient, named exactly**, with the variant. "Kappa" is not a name; 10 papers here stop there. If more than two coders, it is Fleiss' or Krippendorff's — or say explicitly that you averaged pairwise Cohen's kappas, and give the range.
  - **An interval or a bootstrap**, not a bare point estimate. **A probe finds two papers in 5,869 doing this** (a lower bound — the probe requires the literal string "95%" near the coefficient, so an SE or a 90% interval is missed). ''irrCAC'' (CRAN 1.4) and ''statsmodels.stats.inter_rater.cohens_kappa'' both give one; ''sklearn.metrics.cohen_kappa_score'' and the ''krippendorff'' PyPI package do not, so bootstrap it yourself.
  - **The observed agreement and the marginal distribution alongside the coefficient**, always, and unavoidably if the positive class is rare. See the simulation above.
  - **The threshold you set in advance** and what happened to categories that missed it. A low kappa reported with its explanation is a finding; a low kappa deleted is misconduct-adjacent.
  - **How disagreements were resolved** — discussion to consensus, a third adjudicator, majority vote — and whether the coefficient is from before or after that resolution. **435 papers in 5,869 describe some form of resolution** — that one is common practice, unlike everything else on this list — but a kappa computed **after** consensus discussion is not the same statistic as one computed before it, and almost none of them say which they report.
  - **The codebook**, released as an artifact. It costs nothing and it is the only thing that makes your labels reusable. See [[:Artifacts]].
  - **If an LLM did the first pass:** which model and version, the exact prompt, the temperature or sampling settings, the run-to-run agreement of the model with itself, and a human–human coefficient on a subsample coded **blind to the model's output**.

===== Open Questions =====

<WRAP todo>
  * **How much does author-coding inflate agreement?** 85.8% of hand-coding papers here are coded by their own authors. Neither the extraction nor a full-text probe finds a paper that double-codes a subsample with a coder outside the author team as a check on that, and the probe is narrow enough that this should be read as "we could not find one" rather than "there are none". A study that takes ten published codebooks, hires outside coders, and compares the coefficients against the published ones is small, cheap and would recalibrate how every reader reads a kappa in this literature.
  * **What is the real class balance of the tasks this field codes, and how much of the low-kappa tail is the prevalence paradox rather than coder disagreement?** Answerable on any paper that released its labels, and unanswered. The 48 papers here whose classifier ground truth is manual annotation by a **single** stated coder are the place to start.
  * **Do LLM-assisted annotations agree with humans better or worse than humans agree with each other, on web-measurement tasks specifically?** The general-purpose evidence {[gilardi2023_chatgpt]} {[ziems2024_transform]} is not about cookie purposes, dark patterns or policy spans. The corpus's 91 LLM-signal papers that name an agreement statistic (not all of them a chance-corrected coefficient) are enough to attempt a meta-analysis and nobody has.
  * **Almost nobody reports LLM self-agreement.** Run-to-run variability of a model on the same items is the machine analogue of intra-rater reliability and it is cheap to measure. A full-text probe for it finds **18 papers in 5,869** — against 267 hand-coding papers with an LLM in the loop. Whether those 18 are the whole practice or the probe is narrow is itself unresolved.
  * **None of the seven venues asks for any of this.** Reviewer forms are not public, so the current state can only be read off the papers — and the papers say that a decade of improvement has got the field from 5% to 21% reporting. Whether a one-line submission checklist would close the rest is untested and testable.
</WRAP>
===== Related Pages =====

  * [[Statistics:Annotation]] — the sibling page. It owns the //validation design// — what you check the labels against, whether that source is independent, why a held-out set drawn from a crawl usually is not held out, and what an LLM annotator inherits from being a classifier. This page owns the coefficients; that one owns everything around them, and neither repeats the other.
  * [[Design:Website classification]] — the largest hand-coding task most readers of this site will run, and where a codebook and a kappa are expected.
  * [[Privacy:Cookies]] and [[Privacy:Requests]] — the classification tasks whose ground truth is usually hand-labelled, and whose class balance is exactly the lopsided case above.
  * [[Statistics:Hypothesis testing]] — what happens after the labels exist, and the same lesson about the unit of analysis.
  * [[Statistics:Biases]] — an unreliable label set is a measurement-error problem before it is anything else.
  * [[Statistics:Study preregistration]] — where "we set the threshold in advance" becomes checkable rather than assertable.
  * [[Design:User studies]] — where open coding, thematic analysis and grounded theory actually happen; 8.5% of hand-coding papers here use open coding.
  * [[Practices:Ethics]] — hired annotators and crowdworkers are participants, with everything that follows.
  * [[:Artifacts]] — release the codebook and the per-item labels; nothing else makes the coefficient checkable.
  * [[provenance:statistics:interrater_agreement|Provenance of this page's figures]].

====== References ======

/*
To insert citations, you have to follow 3 steps:

  - Check whether the BibTex entry is in https://measuretheweb.org/literature/bibliography, if not, add it to the end. For example, ''LePochat2019_tranco'' is already present.
  - In the text, where you want to use the citation, use {[LePochat2019_tranco]} - it will be rendered as [1] or number given by order in the page.
  - Keep this section as it is to print the bibliography.
*/

<bibtex bibliography></bibtex>

~~DISCUSSION~~