Sooner or later a web measurement stops being automatic. You classify two hundred consent banners by hand to check the detector, or you read four hundred privacy policies, or you decide which of five thousand third-party domains are advertising. That hand-coded set becomes ground truth, and everything downstream — the classifier's reported accuracy, the “we found 41% of sites” headline — inherits whatever error is in it. A reviewer who is doing their job will ask how reliable it was, and the answer is a number.

A statistics textbook will tell you what Cohen's kappa is. It will not tell you the things that decide whether your number means anything in this setting: that the class you are labelling is usually rare, which is exactly the condition under which kappa collapses while your coders are in near-perfect agreement; that your coders are almost always the paper's own authors, who wrote the codebook and know the hypothesis; that the sample you double-code is small enough that the coefficient is a range rather than a value; and that since 2024 the second “rater” is increasingly a language model, which changes what the coefficient is even measuring.

This page assumes you know what a codebook is and roughly what kappa does. It is about four things: how much of this field reports agreement at all, which coefficient to use and when the usual one is the wrong one, what kappa does to a rare-class labelling task, and how to report agreement when an LLM did the first pass.

• Hallgren 2012 [1Hallgren, Kevin A. (2012): "Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial", Tutorials in Quantitative Methods for Psychology 8(1):23-34. (DOI)] — “Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial”. Twelve pages, free, worked examples with code. If you read one thing, read this: it is the only item on this list that will get you from “I have two coders” to a defensible number in an afternoon.
• Krippendorff 2004 [2Krippendorff, Klaus (2004): "Reliability in Content Analysis: Some Common Misconceptions and Recommendations", Human Communication Research 30(3):411-433. (DOI)] — “Reliability in Content Analysis: Some Common Misconceptions and Recommendations”. The source of the α ≥ 0.800 / ≥ 0.667 thresholds this field quotes without attribution, and — more usefully — of the argument for why you should have decided the threshold before you saw the data. The book [3Krippendorff, Klaus (2018): "Content Analysis: An Introduction to Its Methodology". SAGE Publications.] is the long version.
• Feinstein & Cicchetti 1990 [4Feinstein, Alvan R.; Cicchetti, Domenic V. (1990): "High Agreement but Low Kappa: I. The Problems of Two Paradoxes", Journal of Clinical Epidemiology 43(6):543-549. (DOI)] and its companion [5Cicchetti, Domenic V.; Feinstein, Alvan R. (1990): "High Agreement but Low Kappa: II. Resolving the Paradoxes", Journal of Clinical Epidemiology 43(6):551-558. (DOI)] — “High agreement but low kappa”, parts I and II. Part I is the problem this page's simulation reproduces; part II is what to do about it. Two papers in the whole corpus name this paradox, and it is the single most likely thing to be wrong with a kappa computed on web-measurement labels, where the positive class is usually rare.
• Landis & Koch 1977 [6Landis, J. Richard; Koch, Gary G. (1977): "The Measurement of Observer Agreement for Categorical Data", Biometrics 33(1):159-174. (DOI)] — the source of “slight / fair / moderate / substantial / almost perfect”. Read it to learn that its authors called the divisions “clearly arbitrary”. 67 papers here put one of those verbal bands within 120 characters of a coefficient; 7 cite the paper.
• McHugh 2012 [7McHugh, Mary L. (2012): "Interrater Reliability: The Kappa Statistic", Biochemia Medica 22(3):276-282. (DOI)] — a short clinical-methods review of kappa that is unusually direct about its failure modes and proposes stricter bands than Landis–Koch. Useful as the citation for “we treated 0.6 as inadequate”.
• Gwet 2008 [8Gwet, Kilem Li (2008): "Computing Inter-Rater Reliability and Its Variance in the Presence of High Agreement", British Journal of Mathematical and Statistical Psychology 61(1):29-48. (DOI)] — AC1, the coefficient designed for the high-agreement/low-kappa case, with a variance estimator. Five papers in the corpus use it. If your labels are 95% one class, this is the literature you want.
• Tang, Bauer & Christin [9Tang, Jenny; Bauer, Lujo; Christin, Nicolas (2025): "Misuse, Misreporting, Misinterpretation of Statistical Methods in Usable Privacy and Security Papers", in: Proceedings of the Twenty-First Symposium on Usable Privacy and Security (SOUPS), pp. 475-493. USENIX Association. (Link)] (SOUPS 2025) — a systematic review of statistical reporting in an adjacent community. SOUPS is not one of this page's seven venues, which is why it is the external anchor; its finding that “58% of papers lack information to verify whether an assertion is supported” is the same shape as the reporting gap measured here.
• Pangakis, Wolken & Fasching 2023 [10Pangakis, Nicholas; Wolken, Samuel; Fasching, Neil (2023): "Automated Annotation with Generative AI Requires Validation". arXiv:2306.00176. (Link)] and Törnberg 2024 [11Törnberg, Petter (2024): "Best Practices for Text Annotation with Large Language Models", Sociologica 18(2):67-85. (DOI)] — validation protocols for LLM annotation. Read these before you let a model do your first pass; Törnberg's is the more prescriptive.

The population for everything in this section is the 3,318 papers that hand-coded something — that is, that have at least one humanAnnotation record. A paper with no manual coding has no reliability to report, and including it would manufacture silence. Papers are counted once each, never once per mention.

The 81.6% is not all a defect. Half of this coding is binary-verification — one person eyeballing a sample of the detector's output to confirm it did what it should — and a large share of that is genuinely single-coder work where reliability is not defined. What the extraction cannot separate is “single coder, correctly said so” from “several coders, silently”. The count of coders is the field's real gap:

2,030 of the 3,318 (61.2%) never say how many people coded. Restrict to the 1,176 that state two or more — the papers for which a reliability statistic exists and is meaningful — and 42.0% report one. That is the honest version of the headline: not “85% of hand-coding papers are unreliable”, but “in three cases out of five the reader cannot tell whether reliability was even possible, and where it clearly was, more than half still do not report it.”

864 of the hand-coding papers also ran an automated web crawl — the population this site is about. They report agreement at 15.9% (against 15.4% overall), state a coder count at 35.9% (against 38.8%), and were coded by the authors themselves in 86.6% of cases. The literature closest to this site's readership is not doing better than the average: the three differences are small and point in opposite directions, and this page does not test them.

(Multi-valued: a paper with two coding tasks can appear twice.)

Two authors of the same paper, working from a codebook they wrote, who have discussed the hypothesis over eight months, are not two independent raters in the sense the coefficient assumes. This does not make the number worthless — it is still evidence the codebook is applicable — but it is an upper bound on what an outside coder would achieve, and it should be presented that way. Nothing in the corpus lets you quantify the inflation. The extraction has no field for “was the second coder outside the author team”, and a deliberately narrow full-text probe for it returns zero papers in 5,869 — which is evidence that the practice is rare and is not evidence that it never happens. See Open Questions.

The corpus's best-documented case of taking this seriously is Coopamootoo & Groß [12Coopamootoo, Kovila P. L.; Groß, Thomas (2017): "Why Privacy Is All But Forgotten: An Empirical Study of Privacy and Sharing Attitude", Proceedings on Privacy Enhancing Technologies 2017(4):97-118. (DOI)], who separate the author-coder from the trained outside coders: “each unit was independently coded by two coders. The coding assignments were done asymmetrically, with one of the authors coding all units”, and then recruit “two local PhD students … trialled on a sample of four units and trained in content analysis” and compute the coefficient on that pair instead. It is also one of only two papers a full-text probe locates in 5,869 that publish an interval alongside the coefficient: κ = .666, 95% CI [.630, .670].

418 (81.6%) name at least one chance-corrected coefficient; 77 (15.0%) name only an uncorrected agreement rate or something that is not an agreement statistic at all. Reporting “the two coders agreed on 93%” and stopping is the second most common failure on this page after not reporting anything, and it is the one that most often survives review, because 93% sounds decisive.

Cohen's kappa is defined for exactly two raters. 245 papers name it and no multi-rater coefficient. Of the 236 of those that state a coder count, 61 (25.8%) state more than two — three is the usual case, but the corpus contains papers doing this with 4, 5, 15 and 30 coders. Skolka, Staicu & Pradel [15Skolka, Philippe; Staicu, Cristian-Alexandru; Pradel, Michael (2019): "Anything to Hide? Studying Minified and Obfuscated Code in the Web", in: Proceedings of the ACM Web Conference. (DOI)] is a clean example of the ambiguity: five coders, and “we compute Cohen's kappa for each pair of participants. The pairwise agreement ranges between 0.64 and 0.93, with an average of 0.81”. That 61 is a flag for inspection, not an error count: Coopamootoo & Groß [12Coopamootoo, Kovila P. L.; Groß, Thomas (2017): "Why Privacy Is All But Forgotten: An Empirical Study of Privacy and Sharing Attitude", Proceedings on Privacy Enhancing Technologies 2017(4):97-118. (DOI)] are in it at three coders and are one of the most careful papers in the corpus, because their three coders are two separate arrangements and the coefficient is computed on the pair. Averaging pairwise Cohen's kappas is a defensible thing to do and it is almost certainly what these papers did, but it is a different estimator from Fleiss' kappa, and when the coders differ in skill the three pairwise values differ substantially:

C. Three coders of UNEQUAL skill: which pairwise kappa gets quoted?
   error rates 2% / 2% / 12%, balanced task, n = 1000, 200 repetitions

   lowest pairwise kappa   0.725
   mean pairwise kappa     0.794
   highest pairwise kappa  0.922   <- the number an author is tempted to quote
   spread (max - min)      0.196

With more than two coders, use Fleiss' kappa (equal number of coders per item, nominal labels) or Krippendorff's alpha (unequal coders, missing data, or ordinal/interval labels). If you average pairwise Cohen's kappas anyway, say so and give the range, not the mean alone — the mean of three pairwise kappas is not “the” kappa.

Krippendorff's alpha is the safest default for the messy case, which is why the more careful papers here reach for it: Cory et al. [16Cory, Thomas; Rieder, Wolf; Krämer, Julia; Raschke, Philip; Herbke, Patrick; Küpper, Axel (2026): "Word-level Annotation of GDPR Transparency Compliance in Privacy Policies using Large Language Models", Proceedings on Privacy Enhancing Technologies 2026(1):509-528. (DOI)] compute it over “a team of six domain experts from two universities” on 200 privacy policies (α = 0.669); Xian et al. [17Xian, Lu; Tran, Van Hong; Lee, Lauren; Kumar, Meera; Zhang, Yichen; Schaub, Florian (2025): "Layered, Overlapping, and Inconsistent: A Large-Scale Analysis of the Multiple Privacy Policies and Controls of U.S. Banks", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)] use the unitising variant Cu-α for span-level policy annotation, reporting 0.95 for CCPA policies and 0.89 for the rest.

Most web-measurement labelling tasks are lopsided. Third-party requests that are trackers, cookies that are advertising, banners that are non-compliant, sites that are phishing — the positive class is a minority, sometimes a small one. Chance-corrected coefficients are built to punish exactly that, and they punish it hard.

Kappa is (P_o − P_e) / (1 − P_e). When one label covers almost everything, agreement by chance P_e is already near 1, the denominator collapses, and kappa goes to zero even though the coders disagreed on almost nothing. Feinstein & Cicchetti [4Feinstein, Alvan R.; Cicchetti, Domenic V. (1990): "High Agreement but Low Kappa: I. The Problems of Two Paradoxes", Journal of Clinical Epidemiology 43(6):543-549. (DOI)] named this in 1990; two papers in this corpus mention it.

The script below demonstrates it on synthetic data — the point is a property of the coefficient, not a fact about the web — and also measures how wide a kappa is at the sample sizes this field actually double-codes.

kappa_on_skewed_labels.py

#!/usr/bin/env python3
"""Why kappa collapses on a web-measurement labelling task, and how wide it is.
 
SYNTHETIC data, fixed seed. Nothing here is a measurement claim about the web;
the numbers are reproducible by running the file. numpy only (2.4.6 here).
 
Two facts that decide whether an inter-rater number means anything, both of
which bite precisely on the kind of task this field annotates -- a binary
verification pass over a crawl sample, where one class is rare.
 
  A. THE PREVALENCE PARADOX. Cohen's kappa is (Po - Pe) / (1 - Pe). When one
     label covers almost everything, chance agreement Pe is already near 1, so
     the denominator collapses and kappa is small even though the coders
     disagreed on almost nothing. Two coders who agree on 97% of a sample that
     is 97% negatives can score kappa near 0 -- and the same two coders on a
     balanced sample score high. The coders did not change. The base rate did.
 
  B. THE INTERVAL. Kappa on 50 items is not a number, it is a range. Most
     papers report it without one.
 
Run: python3 kappa_on_skewed_labels.py
"""
 
import numpy as np
 
RNG = np.random.default_rng(20260820)
 
 
def cohen_kappa(a, b):
    """Two raters, binary labels, no weighting."""
    n = len(a)
    po = np.mean(a == b)
    pe = sum(np.mean(a == k) * np.mean(b == k) for k in (0, 1))
    if pe == 1.0:
        return float("nan")  # both raters used one label only: kappa undefined
    return (po - pe) / (1 - pe)
 
 
def two_raters(n, prevalence, error_rate):
    """Rater A is the truth; rater B flips each item with probability
    `error_rate`. Rater skill is held CONSTANT across prevalences -- that is the
    whole point: only the base rate moves."""
    a = (RNG.random(n) < prevalence).astype(int)
    flip = RNG.random(n) < error_rate
    b = np.where(flip, 1 - a, a)
    return a, b
 
 
# ----------------------------------------------------------------- A
print("A. Same two coders, same 3% error rate, only the base rate changes")
print("   (n = 1000 items, 200 repetitions, mean over repetitions)\n")
print("   positive class   observed agreement   Cohen's kappa")
for prev in (0.50, 0.30, 0.15, 0.05, 0.03, 0.01):
    pos, ks = [], []
    for _ in range(200):
        a, b = two_raters(1000, prev, 0.03)
        pos.append(np.mean(a == b))
        ks.append(cohen_kappa(a, b))
    ks = np.array(ks)
    print(f"   {prev:>13.0%}   {np.mean(pos):>18.3f}   {np.nanmean(ks):>13.3f}")
 
print("\n   Observed agreement barely moves; kappa falls off a cliff. A paper")
print("   that reports only kappa on a rare-class task looks unreliable when it")
print("   is not; one that reports only observed agreement looks reliable when")
print("   it may not be. Report BOTH, plus the marginal distribution.\n")
 
# ----------------------------------------------------------------- B
print("B. How wide is a kappa? Balanced task, 5% error rate, 2000 repetitions")
print("   (Monte Carlo over 2000 simulated datasets)\n")
print("   double-coded n   median kappa   2.5th pct   97.5th pct   width")
for n in (25, 50, 100, 385, 1000):
    ks = np.array([cohen_kappa(*two_raters(n, 0.5, 0.05)) for _ in range(2000)])
    ks = ks[~np.isnan(ks)]
    lo, hi = np.percentile(ks, [2.5, 97.5])
    print(f"   {n:>14}   {np.median(ks):>12.3f}   {lo:>9.3f}   {hi:>10.3f}   {hi - lo:>5.3f}")
 
print("\n   385 is the median double-coded sample size in the corpus behind this")
print("   page; 22% of its papers double-code fewer than 50 items. At n = 25 the")
print("   95% interval is about a third of kappa's usable range and spans both")
print("   'substantial' and 'almost perfect', so a kappa from a sample that small")
print("   does not distinguish them. Quote an interval, not a Landis-Koch band.\n")
 
# ----------------------------------------------------------------- C
print("C. Three coders of UNEQUAL skill: which pairwise kappa gets quoted?")
print("   error rates 2% / 2% / 12%, balanced task, n = 1000, 200 repetitions\n")
trip = []
for _ in range(200):
    truth = (RNG.random(1000) < 0.5).astype(int)
    rs = [np.where(RNG.random(1000) < e, 1 - truth, truth) for e in (0.02, 0.02, 0.12)]
    pairwise = [cohen_kappa(rs[i], rs[j]) for i, j in ((0, 1), (0, 2), (1, 2))]
    trip.append((min(pairwise), float(np.mean(pairwise)), max(pairwise)))
trip = np.array(trip)
print(f"   lowest pairwise kappa   {trip[:, 0].mean():.3f}")
print(f"   mean pairwise kappa     {trip[:, 1].mean():.3f}")
print(f"   highest pairwise kappa  {trip[:, 2].mean():.3f}   <- the number an author is tempted to quote")
print(f"   spread (max - min)      {(trip[:, 2] - trip[:, 0]).mean():.3f}")
print("\n   One weaker coder is the normal case -- a student added late, an author")
print("   who coded the first pass. 'Cohen's kappa' is then three numbers, and")
print("   the corpus behind this page has 61 papers naming Cohen's kappa while")
print("   stating more than two annotators. With >2 coders the statistic is")
print("   Fleiss' kappa or Krippendorff's alpha; if you average pairwise Cohen's")
print("   kappas instead, say so, because it is a different estimator.")

Its output for parts A and B, verbatim except that the script's own interpretive paragraphs between the two tables are removed here (they are repeated in the prose around this box):

A. Same two coders, same 3% error rate, only the base rate changes
   (n = 1000 items, 200 repetitions, mean over repetitions)

   positive class   observed agreement   Cohen's kappa
             50%                0.970           0.940
             30%                0.970           0.930
             15%                0.971           0.891
              5%                0.970           0.753
              3%                0.970           0.649
              1%                0.970           0.378

B. How wide is a kappa? Balanced task, 5% error rate, 2000 repetitions
   (Monte Carlo over 2000 simulated datasets)

   double-coded n   median kappa   2.5th pct   97.5th pct   width
               25          0.918       0.679        1.000   0.321
               50          0.915       0.757        1.000   0.243
              100          0.900       0.808        0.980   0.172
              385          0.901       0.855        0.938   0.083
             1000          0.900       0.872        0.926   0.054

The 385 in that table is the median coded sample size across the 457 papers here that name an agreement statistic and also state one; 100 of those 457 (21.9%) state fewer than 50 items.

Read the first table with a real task in mind. If 3% of your sampled cookies are the class you care about and your two coders disagree on 3% of items, your kappa is about 0.65 — a number a reviewer will read as mediocre — and the same two coders on a balanced sample score 0.94. Nothing about their reliability changed.

On a lopsided task, report three things, not one: the coefficient, the observed agreement, and the marginal distribution (how many items each coder put in each class). Those three together let a reader diagnose the paradox; any one alone does not. If the class is very rare, report Gwet's AC1 [8Gwet, Kilem Li (2008): "Computing Inter-Rater Reliability and Its Variance in the Presence of High Agreement", British Journal of Mathematical and Statistical Psychology 61(1):29-48. (DOI)] alongside kappa — it is designed for this case, it is in irrCAC on CRAN, and five papers in this corpus already do it. The corpus's model of this is Khatun et al. [18Khatun, Mst Eshita; Noureddine, Lamine; Bello, Sideeq; Ali-Gombe, Aisha (2026): "Disclosure Divergence: Measuring Privacy Policy and Data Safety Misalignment at Scale", Proceedings on Privacy Enhancing Technologies 2026(4):213-231. (DOI)], who give both numbers for the same 40-app subsample: Cohen's κ = 0.78 and 90% observed agreement. Chen et al. [19Chen, Baiqi; Lyu, Jiawei; Wu, Tingmin; Chhetri, Mohan Baruwal; Bai, Guangdong (2025): "Semantics-Aware Cookie Purpose Compliance", in: Proceedings of the ACM Web Conference. (DOI)] do the same job at scale on the task this site cares about most — cookie purpose labels, three independent author-coders, 2,300 cookies, Fleiss' κ = 0.978.

“κ = 0.72, indicating substantial agreement” is the standard sentence. 67 papers in this corpus put one of those verbal bands next to a coefficient and 7 cite Landis & Koch [6Landis, J. Richard; Koch, Gary G. (1977): "The Measurement of Observer Agreement for Categorical Data", Biometrics 33(1):159-174. (DOI)], whose own paper describes the divisions as arbitrary and offered them for a specific clinical context. Zhou et al. [20Zhou, Jiawei; Zhang, Zidong; Ying, Lingyun; Chai, Huajun; Cao, Jiuxin; Duan, Haixin (2025): "Hey, Your Secrets Leaked! Detecting and Characterizing Secret Leakage in the Wild", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)] is a representative use: “a Cohen's Kappa value of 0.91 … indicating 'near perfect agreement' according to Landis and Koch”.

The bands are not wrong so much as empty. They are not calibrated to your task's difficulty, its class balance or its consequences, and at the sample sizes above the interval routinely spans two of them. Krippendorff's thresholds [2Krippendorff, Klaus (2004): "Reliability in Content Analysis: Some Common Misconceptions and Recommendations", Human Communication Research 30(3):411-433. (DOI)] — α ≥ 0.800 for drawing conclusions, ≥ 0.667 for tentative ones — are at least explicitly a decision rule rather than a description, which is why they are the better citation if you need one. McHugh [7McHugh, Mary L. (2012): "Interrater Reliability: The Kappa Statistic", Biochemia Medica 22(3):276-282. (DOI)] proposes stricter bands again and argues for choosing them by consequence.

What actually helps a reader: state the threshold you would have accepted before you computed the coefficient, and what you did when a category fell below it. Swart, den Hengst & Chen [21Swart, Milena de; Hengst, Floris den; Chen, Jieying (2025): "Detecting Linguistic Bias in Government Documents Using Large Language Models", in: Proceedings of the ACM Web Conference. (DOI)] publish Fleiss' κ = 0.35 across 14 expert annotators on a linguistic-bias task and treat it as a finding about task difficulty rather than hiding it. A published low kappa with an explanation is worth more than an unpublished high one.

Of the 418 papers naming a chance-corrected coefficient, 387 report a value this page's parser could read. Bucketing each paper by its lowest reported coefficient (papers commonly report one per category, or a range):

Median 0.80, quartiles 0.70 and 0.86. 74 papers (19.1%) have at least one coefficient below 0.67, Krippendorff's floor for even tentative conclusions — which is roughly one in five, and almost all of those publish the number anyway. That is the healthy behaviour. The unhealthy version is invisible here by construction: a paper that computed a low kappa and did not report it looks identical to a paper that never computed one.

This is the part of the page that has changed since 2023, and it is the part where the corpus is thinnest, so treat the direction as solid and the levels as provisional.

Counting hand-coding papers with any language-model signal in their annotation — an LLM named in the annotation record, its quote, or the paper's tools:

2025 and 2026 are under-represented by construction (see Methodology and Limitations of These Figures), so the 40.6% is a share of a thin, non-random slice. The matcher also over-fires slightly on early years — it returns 1–2 papers per year in 2017, 2019, 2020 and 2021, which are false positives from words like llama and Gemini in unrelated senses, and the table above starts at 2022 for that reason rather than because the earlier rows are empty. The shape — a floor of noise until 2023, a step change in 2024 — survives that comfortably; the levels should not be quoted to a decimal place.

These papers report agreement statistics at 34.1%, against 13.8% for the rest of the hand-coding literature. The likeliest reason is structural rather than cultural: when a model does the first pass, the human coding is the validation, so a paper without a number has nothing to report. This is an interpretation of a difference between two groups, not a tested claim. Two shapes recur, and they are not the same thing:

• Humans validate the model. Alam et al. [22Alam, Mahbub; Rahman, Muhammad Lutfor; Paul, Sonjoy Kumar; Hays, Amy W.; Hussain, Aftab; Huq, Md Imanul; Saxena, Nitesh (2026): "SoK: PHILTER: Uncovering Security and Functional Gaps in AI-based Phishing Website Detection Literature via an LLM-based Reasoning Framework", in: Proceedings of the USENIX Security Symposium. (Link)]: “Two experts independently validate all LLM-generated preliminary assessments by checking the extracted evidence and rationale against the original paper and the codebook”, reporting 345/385 (89.6%) agreement. Cory et al. [16Cory, Thomas; Rieder, Wolf; Krämer, Julia; Raschke, Philip; Herbke, Patrick; Küpper, Axel (2026): "Word-level Annotation of GDPR Transparency Compliance in Privacy Policies using Large Language Models", Proceedings on Privacy Enhancing Technologies 2026(1):509-528. (DOI)] have six domain experts review LLM-generated annotations of 200 privacy policies.
• The model is treated as a rater alongside humans. Sun, Vekaria & Nithyanand [23Sun, Chen; Vekaria, Yash; Nithyanand, Rishab (2026): "On the Suitability of LLM-Driven Agents for Dark Pattern Audits", Proceedings on Privacy Enhancing Technologies 2026(4):927-946. (DOI)] audit an LLM agent for dark patterns across 456 data-broker sites and compute “Cohen's κ on binary (present/absent) labels for each dark pattern category, yielding κ = 51.9%”, rising to 73.7% after calibration.

Two mechanical traps, both visible in the corpus. First, an LLM's labels are not deterministic, so “agreement between the model and itself across runs” is a real and separate quantity — and unlike a human rater it is cheap to measure over many runs. A full-text probe finds 18 papers in 5,869 reporting anything of the kind, so it is rare rather than absent; the probe is narrow and that 18 is a lower bound. Second, do not let the humans see the model's labels first. Zhang et al. [27Zhang, Shirley; Chung, Paul; Vervelde, Jacob; Korapati, Nishant; Chatterjee, Rahul; Fawaz, Kassem (2025): "Abusability of Automation Apps in Intimate Partner Violence", in: Proceedings of the USENIX Security Symposium. (Link)] are explicit about this: their two authors “collaboratively annotated the IPV attacks in the selected validation set without seeing LLM responses” before the comparison, and only then report κ = 0.96 on the 100 shortcuts two authors verified independently. A paper that does not say whether its coders saw the model's output has an anchoring problem it cannot rule out.

• Population, always named. Every figure here is of the 3,318 papers with at least one humanAnnotation record, or of a stated subset of it (512 naming a statistic; 1,288 stating a coder count; 864 that also crawled). Never of 5,859. Papers are counted once each; the 3,318 papers carry 5,778 annotation records between them.
• Sentinels are silence, not values. A null agreementMetric or annotatorCount is counted as not-reported and never as a category. The reporting-gap figures are therefore bounds: a paper that computed a kappa and left it out of the text is indistinguishable from one that never computed it.
• agreementMetric is free text and was folded. 133 distinct strings across the corpus — case, apostrophe glyph, the Greek letter versus the word, and several coefficients in one string. The fold is an ordered rule list with its unmapped residue printed in full (24 distinct strings, 29 mentions) on the provenance page. Ranking the raw strings instead puts Cohen's kappa at 170 papers rather than 252.
• The value parser is conservative and one-sided. “0.68-0.96” is a range, “σ = 0.09” is a dispersion and “93%” is raw agreement, and each of those was silently corrupting the distribution until it was fixed. As a consequence a genuinely negative coefficient cannot be distinguished from a range separator and is never returned, so below-chance results are invisible in the value table.
• A cross-check on the extraction. 252 papers fold to “Cohen's kappa”; an independent whitespace-collapsed full-text probe for the phrase finds 230. The two agree to within about 10%, with the extraction the more generous — it also catches “Cohen-Kappa” and inferred cases. Neither is exact; the ranking is what should be trusted.
• Quotes were spot-checked by hand. 21 evidence quotes behind figures and named papers on this page were checked against paper.cols.txt. Nine were verbatim-exact; the rest failed exact match on ellipsis elision or PDF hyphenation, and all of them were then confirmed by locating the coefficient value and the surrounding phrase in the full text. None was fabricated. Full list on the provenance page.
• Full-text probes are lower bounds. The counts of Landis–Koch citations, kappa intervals and the paradox come from regex over 5,869 stored full texts with whitespace collapsed and soft hyphens rejoined. A paper phrasing something differently is missed. The probe script and its unedited output are on the provenance page.
• humanAnnotation.sampleSize is the coded sample, not always the double-coded one, and it is stated by 457 of the 512 papers naming a statistic (89.3%). A few records carry the whole labelled corpus. The median (385) is robust to that; the bucket table is not entirely, and the 22%-under-50 figure is if anything conservative.
• Venue coverage. Seven venues — CCS, IMC, NDSS, PoPETs, USENIX Security, TheWebConf, IEEE S&P. SOUPS, CHI and CSCW are absent, and they matter more on this page than on most: they are where content analysis is a normal method and where reporting standards for it are highest. Every figure here is a claim about the seven venues. The wider community's practice is probably better — content analysis is a normal method at those venues and reporting standards for it are older there — but that is an inference from where the method is native, not something this corpus measures.
• 2025 and 2026 are provisional. CCS 2026 and IMC 2026 have not been held; IEEE S&P 2026 and TheWebConf 2026 are under-selected because selection screens on abstracts that are not yet indexed. Any trend row ending in 2026 is a share of a thin slice, which is why the LLM table is labelled rather than trimmed.
• Every external reference was fetched on 2026-08-20, not recalled: 14 DOIs resolved through the Crossref and DataCite content-negotiation APIs, the PoPETs and USENIX landing pages for author lists, and the current PyPI and CRAN versions of every package named below.
• The full query log, the fold's residue, the probe scripts and their unedited output, and the list of things that could not be established are on interrater_agreement. Corpus-level caveats are on Corpus.

1. How many people coded, and who they were — authors, students, hired annotators, domain experts. 61.2% of the hand-coding papers here do not say. This is one clause and it is the highest-value item on the list.
2. How many items were double-coded, as a count and as a share of the labelled set. The coefficient is only as precise as that number.
3. The coefficient, named exactly, with the variant. “Kappa” is not a name; 10 papers here stop there. If more than two coders, it is Fleiss' or Krippendorff's — or say explicitly that you averaged pairwise Cohen's kappas, and give the range.
4. An interval or a bootstrap, not a bare point estimate. A probe finds two papers in 5,869 doing this (a lower bound — the probe requires the literal string “95%” near the coefficient, so an SE or a 90% interval is missed). irrCAC (CRAN 1.4) and statsmodels.stats.inter_rater.cohens_kappa both give one; sklearn.metrics.cohen_kappa_score and the krippendorff PyPI package do not, so bootstrap it yourself.
5. The observed agreement and the marginal distribution alongside the coefficient, always, and unavoidably if the positive class is rare. See the simulation above.
6. The threshold you set in advance and what happened to categories that missed it. A low kappa reported with its explanation is a finding; a low kappa deleted is misconduct-adjacent.
7. How disagreements were resolved — discussion to consensus, a third adjudicator, majority vote — and whether the coefficient is from before or after that resolution. 435 papers in 5,869 describe some form of resolution — that one is common practice, unlike everything else on this list — but a kappa computed after consensus discussion is not the same statistic as one computed before it, and almost none of them say which they report.
8. The codebook, released as an artifact. It costs nothing and it is the only thing that makes your labels reusable. See Artifacts.
9. If an LLM did the first pass: which model and version, the exact prompt, the temperature or sampling settings, the run-to-run agreement of the model with itself, and a human–human coefficient on a subsample coded blind to the model's output.

• How much does author-coding inflate agreement? 85.8% of hand-coding papers here are coded by their own authors. Neither the extraction nor a full-text probe finds a paper that double-codes a subsample with a coder outside the author team as a check on that, and the probe is narrow enough that this should be read as “we could not find one” rather than “there are none”. A study that takes ten published codebooks, hires outside coders, and compares the coefficients against the published ones is small, cheap and would recalibrate how every reader reads a kappa in this literature.
• What is the real class balance of the tasks this field codes, and how much of the low-kappa tail is the prevalence paradox rather than coder disagreement? Answerable on any paper that released its labels, and unanswered. The 48 papers here whose classifier ground truth is manual annotation by a single stated coder are the place to start.
• Do LLM-assisted annotations agree with humans better or worse than humans agree with each other, on web-measurement tasks specifically? The general-purpose evidence [26Gilardi, Fabrizio; Alizadeh, Meysam; Kubli, Maël (2023): "ChatGPT Outperforms Crowd Workers for Text-Annotation Tasks", Proceedings of the National Academy of Sciences 120(30):e2305016120. (DOI)] [25Ziems, Caleb; Held, William; Shaikh, Omar; Chen, Jiaao; Zhang, Zhehao; Yang, Diyi (2024): "Can Large Language Models Transform Computational Social Science?", Computational Linguistics 50(1):237-291. (DOI)] is not about cookie purposes, dark patterns or policy spans. The corpus's 91 LLM-signal papers that name an agreement statistic (not all of them a chance-corrected coefficient) are enough to attempt a meta-analysis and nobody has.
• Almost nobody reports LLM self-agreement. Run-to-run variability of a model on the same items is the machine analogue of intra-rater reliability and it is cheap to measure. A full-text probe for it finds 18 papers in 5,869 — against 267 hand-coding papers with an LLM in the loop. Whether those 18 are the whole practice or the probe is narrow is itself unresolved.
• None of the seven venues asks for any of this. Reviewer forms are not public, so the current state can only be read off the papers — and the papers say that a decade of improvement has got the field from 5% to 21% reporting. Whether a one-line submission checklist would close the rest is untested and testable.

• Website classification — the largest hand-coding task most readers of this site will run, and where a codebook and a kappa are expected.
• Cookies and Requests — the classification tasks whose ground truth is usually hand-labelled, and whose class balance is exactly the lopsided case above.
• Hypothesis testing — what happens after the labels exist, and the same lesson about the unit of analysis.
• Biases — an unreliable label set is a measurement-error problem before it is anything else.
• Study preregistration — where “we set the threshold in advance” becomes checkable rather than assertable.
• User studies — where open coding, thematic analysis and grounded theory actually happen; 8.5% of hand-coding papers here use open coding.
• Ethics — hired annotators and crowdworkers are participants, with everything that follows.
• Artifacts — release the codebook and the per-item labels; nothing else makes the coefficient checkable.
• Provenance of this page's figures.

[1]

Hallgren, Kevin A. (2012): "Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial", Tutorials in Quantitative Methods for Psychology 8(1):23-34. (DOI)

[2]

Krippendorff, Klaus (2004): "Reliability in Content Analysis: Some Common Misconceptions and Recommendations", Human Communication Research 30(3):411-433. (DOI)

[3]

Krippendorff, Klaus (2018): "Content Analysis: An Introduction to Its Methodology". SAGE Publications.

[4]

Feinstein, Alvan R.; Cicchetti, Domenic V. (1990): "High Agreement but Low Kappa: I. The Problems of Two Paradoxes", Journal of Clinical Epidemiology 43(6):543-549. (DOI)

[5]

Cicchetti, Domenic V.; Feinstein, Alvan R. (1990): "High Agreement but Low Kappa: II. Resolving the Paradoxes", Journal of Clinical Epidemiology 43(6):551-558. (DOI)

[6]

Landis, J. Richard; Koch, Gary G. (1977): "The Measurement of Observer Agreement for Categorical Data", Biometrics 33(1):159-174. (DOI)

[7]

McHugh, Mary L. (2012): "Interrater Reliability: The Kappa Statistic", Biochemia Medica 22(3):276-282. (DOI)

[8]

Gwet, Kilem Li (2008): "Computing Inter-Rater Reliability and Its Variance in the Presence of High Agreement", British Journal of Mathematical and Statistical Psychology 61(1):29-48. (DOI)

[9]

Tang, Jenny; Bauer, Lujo; Christin, Nicolas (2025): "Misuse, Misreporting, Misinterpretation of Statistical Methods in Usable Privacy and Security Papers", in: Proceedings of the Twenty-First Symposium on Usable Privacy and Security (SOUPS), pp. 475-493. USENIX Association. (Link)

[10]

Pangakis, Nicholas; Wolken, Samuel; Fasching, Neil (2023): "Automated Annotation with Generative AI Requires Validation". arXiv:2306.00176. (Link)

[11]

Törnberg, Petter (2024): "Best Practices for Text Annotation with Large Language Models", Sociologica 18(2):67-85. (DOI)

[12]

Coopamootoo, Kovila P. L.; Groß, Thomas (2017): "Why Privacy Is All But Forgotten: An Empirical Study of Privacy and Sharing Attitude", Proceedings on Privacy Enhancing Technologies 2017(4):97-118. (DOI)

[13]

Cohen, Jacob (1960): "A Coefficient of Agreement for Nominal Scales", Educational and Psychological Measurement 20(1):37-46. (DOI)

[14]

Fleiss, Joseph L. (1971): "Measuring Nominal Scale Agreement Among Many Raters", Psychological Bulletin 76(5):378-382. (DOI)

[15]

Skolka, Philippe; Staicu, Cristian-Alexandru; Pradel, Michael (2019): "Anything to Hide? Studying Minified and Obfuscated Code in the Web", in: Proceedings of the ACM Web Conference. (DOI)

[16]

Cory, Thomas; Rieder, Wolf; Krämer, Julia; Raschke, Philip; Herbke, Patrick; Küpper, Axel (2026): "Word-level Annotation of GDPR Transparency Compliance in Privacy Policies using Large Language Models", Proceedings on Privacy Enhancing Technologies 2026(1):509-528. (DOI)

[17]

Xian, Lu; Tran, Van Hong; Lee, Lauren; Kumar, Meera; Zhang, Yichen; Schaub, Florian (2025): "Layered, Overlapping, and Inconsistent: A Large-Scale Analysis of the Multiple Privacy Policies and Controls of U.S. Banks", in: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. (DOI)

[18]

Khatun, Mst Eshita; Noureddine, Lamine; Bello, Sideeq; Ali-Gombe, Aisha (2026): "Disclosure Divergence: Measuring Privacy Policy and Data Safety Misalignment at Scale", Proceedings on Privacy Enhancing Technologies 2026(4):213-231. (DOI)

[19]

Chen, Baiqi; Lyu, Jiawei; Wu, Tingmin; Chhetri, Mohan Baruwal; Bai, Guangdong (2025): "Semantics-Aware Cookie Purpose Compliance", in: Proceedings of the ACM Web Conference. (DOI)

[20]

Zhou, Jiawei; Zhang, Zidong; Ying, Lingyun; Chai, Huajun; Cao, Jiuxin; Duan, Haixin (2025): "Hey, Your Secrets Leaked! Detecting and Characterizing Secret Leakage in the Wild", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)

[21]

Swart, Milena de; Hengst, Floris den; Chen, Jieying (2025): "Detecting Linguistic Bias in Government Documents Using Large Language Models", in: Proceedings of the ACM Web Conference. (DOI)

[22]

Alam, Mahbub; Rahman, Muhammad Lutfor; Paul, Sonjoy Kumar; Hays, Amy W.; Hussain, Aftab; Huq, Md Imanul; Saxena, Nitesh (2026): "SoK: PHILTER: Uncovering Security and Functional Gaps in AI-based Phishing Website Detection Literature via an LLM-based Reasoning Framework", in: Proceedings of the USENIX Security Symposium. (Link)

[23]

Sun, Chen; Vekaria, Yash; Nithyanand, Rishab (2026): "On the Suitability of LLM-Driven Agents for Dark Pattern Audits", Proceedings on Privacy Enhancing Technologies 2026(4):927-946. (DOI)

[24]

Baumann, Joachim; Röttger, Paul; Urman, Aleksandra; Wendsjö, Albert; Plaza-del-Arco, Flor Miriam; Gruber, Johannes B.; Hovy, Dirk (2025): "Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation". arXiv:2509.08825. (Link)

[25]

Ziems, Caleb; Held, William; Shaikh, Omar; Chen, Jiaao; Zhang, Zhehao; Yang, Diyi (2024): "Can Large Language Models Transform Computational Social Science?", Computational Linguistics 50(1):237-291. (DOI)

[26]

Gilardi, Fabrizio; Alizadeh, Meysam; Kubli, Maël (2023): "ChatGPT Outperforms Crowd Workers for Text-Annotation Tasks", Proceedings of the National Academy of Sciences 120(30):e2305016120. (DOI)

[27]

Zhang, Shirley; Chung, Paul; Vervelde, Jacob; Korapati, Nishant; Chatterjee, Rahul; Fawaz, Kassem (2025): "Abusability of Automation Apps in Intimate Partner Violence", in: Proceedings of the USENIX Security Symposium. (Link)