The query log, folds, hand audits, quote checks and judgement calls behind annotation. Working notes, not prose: this page is for someone checking a number, not learning a topic. Corpus-level caveats — the venue scope, the selection funnel, what the extraction is and how good it is — are on Corpus and are not restated here.

Citations use the same keys and the same shared Bibliography as the content page. This page adds no bibliography entries of its own.

Two, and they are never mixed. Both are defined by the presence of a record, not by a keyword.

The LLM sub-population is CLASSIFIED papers with a classification[].method == “llm” tuple: 177, of which 175 have usedOrMentioned in {used, produced}. The 175 is used for every figure on the page; the 177 is printed alongside it in the report because Website classification uses the 177 for a different question and the two must not look like drift.

The grouped-split sub-population is web-platform papers (platforms includes web) whose classification carries held-out-test-set or cross-validation: 300.

All of these are in scripts/report_annotation.mjs, which prints them with the denominator attached. The section letters below are the section letters in its output.

Selectors on the same script for anything not in the tables:

node scripts/report_annotation.mjs                  # the whole audit trail
node scripts/report_annotation.mjs --wiki           # DokuWiki tables
node scripts/report_annotation.mjs --list llm       # the 175 papers
node scripts/report_annotation.mjs --list novalidation   # the 1,328
node scripts/report_annotation.mjs --quotes 'easylist'   # tuples + evidence quotes

This is the single judgement call most likely to be made differently by a reasonable person, so it is written out.

classification[].validation has six values. Four are validation. The other two are not the same thing:

• none-reported is treated as a real value: the paper is saying there was no check. It is not a sentinel and is counted.
• not-applicable is treated as a sentinel: the extraction could not find a validation step and did not think one was called for. It is reported as its own row and never folded into “validated”.

That produces two defensible headline figures and the page publishes both with the difference spelled out:

The roadmap row that queued this page said “42.4% of the 4,439 papers that classified anything report no validation”, which reads as the second claim and is the first. The gap is twelve points. The roadmap row was corrected in the same sitting.

What is not established: how many of the 462 all-not-applicable papers genuinely did not need a check. Nobody has read a sample. It is on the page's open-questions list because it is 10.4% of the population and it is the difference between the two figures above.

scripts/gt_fold.mjs. Folds classification[].groundTruthSource — 7,114 tuples, 6,223 distinct lower-cased strings — by what kind of authority the ground truth is, not by resource name. Nine families, ordered, first match wins, each tagged with whether the ground truth is independent of the thing being validated.

Order is load-bearing and the self-tests cover the cases that make it so:

• model-output runs first, but only fires when the model is the labeller. “manual review of LLM output” therefore falls through to human-annotation, because the human is the authority in that sentence.
• human-annotation runs before by-construction, so “authors generated and labeled 400 real and 400 synthetic routes” is human.
• curated-list and third-party-service run before existing-dataset, so “EasyList and EasyPrivacy labels” is a list and not a dataset.
• documentation-or-declaration runs before rule-or-tool-output, so “PHP function signatures and PHPDoc documentation” is documentation and not a detection signature. That ordering was introduced after the first version got it wrong, and the test is in the file.

Two regex narrowings were forced by a failing self-test and are recorded in the file:

• bare standard was removed from the documentation family — it captured “gold standard collection”.
• bare signature was narrowed to (detection|malware|attack|yara|av|virus|rule) signature — it captured “PHP function signatures”.
• \bprivacy (policy|label) is anchored on a word boundary so it does not fire on “EasyPrivacy labels”.

Self-tests: 21, all passing, run by node scripts/gt_fold.mjs.

The residue is 34.2% of tuples and it is printed in full below. Three things about it:

1. It is a long tail: 2,068 of the 2,206 distinct unmapped strings (93.7%) appear exactly once. Each names a resource or a procedure specific to one paper — “touchpal call tags”, “indegree median of the population”, “optical lens scanner labels”, “cryptoapi-bench”, “ecg demographic labels”.
2. A systematic 30-string sample (every 73rd line of the printed residue; stored literally in RESIDUE_SAMPLE in scripts/annotation_audit.mjs so it survives a corpus change, with a self-check that warns if a sampled string leaves the residue) was hand-classified into the family each string would have joined. It spreads across every family: 6 existing-dataset, 6 by-construction, 4 third-party-service, 3 human-annotation, 3 rule-or-tool-output, 2 curated-list, 2 execution-or-oracle, and 4 the string alone cannot place. So every family's share is a floor and none is systematically missing.
3. The paper-level residue share (48.8%) is much higher than the tuple-level one (34.2%) because a paper carries several tuples and needs only one unmapped string to appear. Both are printed; the page quotes both.

Deliberately not attempted: a proper-name patch family to catch LibriSpeech / TIMIT / MAWILab / DGArchive / MS MARCO, which are existing-dataset with no keyword. A first version used ^<token>( <word>)? (labels|categories|transcripts|…) and matched almost any two-word string ending in “labels”, including ecg demographic labels and packer detector labels, which belong elsewhere. It was removed rather than tuned: an over-broad rule that moves the residue into a family is worse than a residue that is printed.

scripts/annotation_audit.mjs holds all three as explicit per-paper maps with the deciding phrase. Running the file re-derives each probe and exits non-zero if a probe returns a paper the map does not cover, or a map entry the probe no longer returns. The maps therefore cannot go stale silently when the corpus grows.

(Four, in the file; the page treats the grouped-split one as part of the same discipline.)

Why each had to be an audit rather than a count:

• The title probe catches papers that merely mention an LLM. Seven of the 30 are attacks on LLMs or studies of them. Precision 70.0%; recall against the enum 13.1% (23 of 175). This is why the page's population is the enum.
• “temperature” has four other senses in this corpus — the colour temperature of a Tuya smart bulb, a weather API payload, the softmax / Gumbel-Softmax τ, a simulated-annealing schedule — plus two papers where the only hit is a cited paper's title. A raw probe would have published 55 (31.4%) where the answer is 41 (23.4%), 34% too high.
• “the prompt is in the appendix” matches four sentences that are about something else entirely (“moderating the exact prompt mentioned in the policy”, “we combine the above 5 parts to form a complete prompt”). One further paper matches because it says it is withholding its prompts — and those are attack prompts, not annotation prompts, so it is not a counter-example to the norm and the page says so.
• The grouped-split probe first included data leakage, which in this corpus almost always means privacy leakage rather than ML leakage. That term alone took the hit count from 52 to 99 on the wider population. It was removed.

The grouped-split figure is the weakest number on the page and is labelled as a lower bound in both directions there. It supports “rare and almost never stated”; it does not support “only seven papers did it”.

All run over data/fulltext/<year>/<venue>/<slug>/paper.cols.txt with whitespace collapsed and soft hyphens rejoined, which is non-optional: a line break inside a phrase silently undercounts. All 175 LLM papers and all 300 web held-out/CV papers have a stored full text — no coverage hole.

Tight-vs-loose was printed for every probe and the report throws if an audited or tight count exceeds its loose count:

node scripts/quote_check.mjs –method llm — all 255 evidence quotes on the LLM classification tuples carried by the 177 papers, against the rendering the extractor actually read, with a fallback to a second rendering of the PDF. (255 is a tuple count, not a paper count; an earlier draft of this line wrote “the 177 LLM classification tuples” and attached the wrong noun's number.)

The 18 were read. None is a fabrication. They are elisions (“…” in the stored quote), math notation the text layer mangles, or a paraphrase the extractor wrote around a real sentence. The two behind papers named on the content page were confirmed by hand:

• [1Xiong, Shenping; Wang, Xutong; Jin, Ze; Liu, Xinyu; Wang, Haoqiang; Chen, Zhen; Tan, Ru; Liu, Qixu (2026): "TGNN: Enhancing Pixel Tracking Detection via LLM-driven Annotation and GAT-powered Structural Representation", in: Proceedings of the ACM Web Conference. (DOI)] — stored quote “we employ the large language model Qwen3 as the underlying model Mλ for LLM-based labeling”, scored 27%. The paper says “we employ the large language model Qwen3 [49] as the underlying model 𝑀𝜆 for LLM-based labeling”. The extraction dropped the citation marker and the math italic. Present.
• [2Cory, Thomas; Rieder, Wolf; Krämer, Julia; Raschke, Philip; Herbke, Patrick; Küpper, Axel (2026): "Word-level Annotation of GDPR Transparency Compliance in Privacy Policies using Large Language Models", Proceedings on Privacy Enhancing Technologies 2026(1):509-528. (DOI)] — stored quote about a 99.7% accuracy, scored 40%. Present, but it is about the document-type filter step (is this text a privacy policy), not about the annotation. The page therefore does not quote the 99.7% and quotes the 200-policy expert gold set instead.

The bulk checker above traces the extraction's evidence.quote values, which is not the same set as the strings a human typed into the page, and check_page_numbers.mjs traces digits and cannot see a quotation at all. scripts/annotation_quotespans.mjs closes that gap: it pulls every //"…"// span out of the page source and requires each to be present in the report script's output, in the extraction's own free-text values, or verbatim in one of the cited papers — with a de-splicing fallback for two-column breaks. 13 distinct spans, 0 unaccounted, exit 0.

It was mutation-tested rather than read, and the first two versions were wrong:

• Non-overlapping three-word fragments failed on [3Munir, Shaoor; Siby, Sandra; Iqbal, Umar; Englehardt, Steven; Shafiq, Zubair; Troncoso, Carmela (2023): "CookieGraph: Understanding and Detecting First-Party Tracking Cookies", pp. 3490–3504. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)], where the splice happens to land on a fragment boundary. Replaced with sliding windows.
• The sliding windows then collapsed to 5 of 16 because the in-order cursor advanced past the whole matched window, making the next (two-word-overlapping) window unfindable. Advancing to the match start fixed it.
• With a 70% coverage threshold, changing “a ratio of 80% and 20%” to “70% and 30%” still passed — it killed only 6 of 11 windows. Every numeric token in a span is now additionally required to be present verbatim on a word boundary. That mutation, and a fabricated-but-plausible sentence, both now fail the guard; the real page passes.

The guard caught one real defect: the page quoted [2Cory, Thomas; Rieder, Wolf; Krämer, Julia; Raschke, Philip; Herbke, Patrick; Küpper, Axel (2026): "Word-level Annotation of GDPR Transparency Compliance in Privacy Policies using Large Language Models", Proceedings on Privacy Enhancing Technologies 2026(1):509-528. (DOI)] as “equally distributed among the six experts, with two experts randomly assigned to each pol[icy]”, which is a de-spliced reconstruction across two column breaks presented as one verbatim span. It was shortened to the contiguous “two experts randomly assigned to each pol[icy]” with the six experts stated in the page's own words.

A known limit, stated because the guard's pass is otherwise easy to over-read: a span that exists somewhere in the extraction is accepted, so it cannot tell a residue string from a folded one. That specific question is answered by the RESIDUE_SAMPLE self-check instead.

Four further quotes were located by hand outside the bulk check:

[5Arp, Daniel; Quiring, Erwin; Pendlebury, Feargus; Warnecke, Alexander; Pierazzi, Fabio; Wressnegger, Christian; Cavallaro, Lorenzo; Rieck, Konrad (2022): "Dos and Don'ts of Machine Learning in Computer Security", in: Proceedings of the USENIX Security Symposium. (Link)] is cited without a verbatim quote, on purpose. Its paper.cols.txt is heavily two-column-spliced (“conduct a study of 30 papers from top-tier security conferhowever, we argue…”), and a “57% present potential sampling bias” fragment in it is part of a figure's axis labels rather than a sentence. The structural facts the page asserts — 30 papers, the top-four security venues, ten years, the pitfall names — were each read directly out of the text and hold. A percentage was nearly quoted off that splice and was not.

pages/holdout_by_site.py. Standard library only, deterministic, three subcommands.

• selftest — passes. Checks (a) every Wilson endpoint against |p̂ − p| = z·√(p(1−p)/n) for six (k, n) pairs, skipping clamped endpoints; (b) that wilson(0, 50) is not zero-width; © that an abstention is scored as a recall miss and not as a negative prediction; (d) that the grouped split never puts a group on both sides and never empties a side.
• demo — runs in under a second, fixed seed 20260911. Precision 0.964 [0.940, 0.979] under a page-level split, 0.659 [0.604, 0.710] under a site-grouped split, non-overlapping. The output on the content page is unedited.
• score — was run end-to-end against a synthetic 1,000-row labels.csv with a 10.9% positive class and 74 abstentions, and reported them separately as designed.

The classifier in demo is 1-NN over a feature vector containing a memorisable site fingerprint and one weak generalisable signal. That is a choice, and a reasonable person could call it a straw man. The defence is in the docstring: a bag-of-words model over page HTML and a URL-pattern heuristic are memorisers with extra steps, and the effect size depends on how much of the label the site decides — here 1.0 against 0.6, which is if anything conservative for a “does this site do X” label.

• How many of the 462 all-not-applicable papers really needed no validation. 10.4% of the population, and the difference between the page's two headline figures. A hand audit of 50 would settle it; it was out of scope for this sitting and is on the page's open-questions list.
• The true rate of grouped splitting. The probe finds 7 of 300 and both error directions are open. Only a hand read of the 300 would close it.
• Whether papers report a residue / refusal count at all. There is no schema field for it and a full-text probe returns mostly discussions of JSON parsing rather than counts. The page says it is the cheapest missing statistic and does not put a number on it.
• Whether the held-out-test-set enum means what it says. See the [1Xiong, Shenping; Wang, Xutong; Jin, Ze; Liu, Xinyu; Wang, Haoqiang; Chen, Zhen; Tan, Ru; Liu, Qixu (2026): "TGNN: Enhancing Pixel Tracking Detection via LLM-driven Annotation and GAT-powered Structural Representation", in: Proceedings of the ACM Web Conference. (DOI)] case above: the one instance read closely was wrong. A hand audit of 30 of the 547 would put a bound on it and was out of scope for this sitting.
• Self-agreement of a model with itself across runs. Deliberately not re-measured here: Interrater agreement already has a figure (18 papers in 5,869) from a different probe over a different population, and publishing a second, looser number over the 175 would have created a contradiction between two pages for no gain. The page cites theirs.
• Whether the 58% run-to-run stability of classification.method was re-measured on this corpus. It was not; it comes from the extraction's own stability audit on the older run and is quoted as an order of magnitude, which is what the page says.
• Anything about SOUPS, CHI, CSCW, EuroS&P, ACSAC, RAID or AsiaCCS. Absent from the corpus. The first three matter most here and the page says so.

1. none-reported counted as a value, not-applicable as a sentinel. Both headline figures published. (Above.)
2. The 175 rather than the 177 as the LLM population, with the 177 printed beside it and the two-paper difference explained in a footnote on the content page, because Website classification publishes the 177 series.
3. Per-target and per-method tables published as rankings with a base-rate row, not as thirteen or nineteen precise percentages, because classification.method agrees ~58% run-to-run. Rows under ~40 papers are flagged on the page as counts dressed as rates.
4. The four-year buckets, not the per-year series, on the content page: the per-year series is noisy below n = 150 and the shape (a step around 2018, then flat) is identical in both. Both are in the report output.
5. The ground-truth fold classifies the kind of authority, not the resource. A resource-name fold would have duplicated req_fold.mjs, ip_fold.mjs and webcat_fold.mjs badly and answered a question three pages already answer better.
6. fingerprinting-script in the per-target table points at Fingerprinting. An earlier draft pointed it at Darkpatterns with a hand-wave; that was a mistake and was fixed before publication.
7. The page does not name a single paper as a negative example of a random split. [1Xiong, Shenping; Wang, Xutong; Jin, Ze; Liu, Xinyu; Wang, Haoqiang; Chen, Zhen; Tan, Ru; Liu, Qixu (2026): "TGNN: Enhancing Pixel Tracking Detection via LLM-driven Annotation and GAT-powered Structural Representation", in: Proceedings of the ACM Web Conference. (DOI)] is described as not saying whether the downstream split respects site boundaries, which is what the corpus figure is about; singling a paper out for criticism off one sentence would not be.
8. The base rate for the per-method and per-target tables was wrong in the first published version and was corrected on 2026-09-11. Each row asks whether the tuple for that method or target validates; the base row asked whether any tuple of the paper validates. Those are not the same test — a paper with several classifiers clears the second on one of them while appearing in several rows — so every “far below the base rate” sentence overstated its gap. The comparable denominator is the (paper, method) or (paper, target) cell: 58.2% and 60.1%, against the 70.1% first published. The row ordering is unaffected. The non-comparable figure is still printed in the report, marked as such, so the two cannot be confused again.
9. The extraction is wrong about [1Xiong, Shenping; Wang, Xutong; Jin, Ze; Liu, Xinyu; Wang, Haoqiang; Chen, Zhen; Tan, Ru; Liu, Qixu (2026): "TGNN: Enhancing Pixel Tracking Detection via LLM-driven Annotation and GAT-powered Structural Representation", in: Proceedings of the ACM Web Conference. (DOI)] and the page says so rather than inheriting it. The extraction records that paper's LLM tuple as validation: held-out-test-set. The paper evaluates its LLM labelling component “on the ground truth dataset” — the whole 20,000-request expert-validated set, F1 98.17% — with no held-out portion for the labelling step at all; the 80/20 split it describes belongs to the downstream graph model that consumes those labels. A first draft of the content page repeated the extraction and attributed the 80/20 split to the LLM component's validation; a reviewer caught it on 2026-09-11 by reading the paper. What this does to the 12.3% held-out-test-set figure is unknown: one such tuple was read closely and it was wrong, which is a sample of one and says nothing about the other 546. It is a worked example of the page's own instruction to read what a validation enum actually refers to — and of the fact that this extraction is a model's reading, not a ground truth.
10. The 30 candidate papers, 55 temperature candidates, 31 prompt candidates and 10 grouped-split candidates were read from the extracted full text, not from the PDFs. For the two papers whose characterisation is load-bearing on the page ([1Xiong, Shenping; Wang, Xutong; Jin, Ze; Liu, Xinyu; Wang, Haoqiang; Chen, Zhen; Tan, Ru; Liu, Qixu (2026): "TGNN: Enhancing Pixel Tracking Detection via LLM-driven Annotation and GAT-powered Structural Representation", in: Proceedings of the ACM Web Conference. (DOI)], [2Cory, Thomas; Rieder, Wolf; Krämer, Julia; Raschke, Philip; Herbke, Patrick; Küpper, Axel (2026): "Word-level Annotation of GDPR Transparency Compliance in Privacy Policies using Large Language Models", Proceedings on Privacy Enhancing Technologies 2026(1):509-528. (DOI)]) the deciding passages were re-read in context. The other candidates were classified from a 130-character window, which is enough to tell “colour temperature” from “temperature = 0” and not enough for anything subtler.

Every figure on the content page is in this script's output, with its denominator attached. Committed as scripts/report_annotation.mjs.

report_annotation.mjs

// Every figure on statistics:annotation, with its denominator.
//
//   node scripts/report_annotation.mjs            # the audit trail
//   node scripts/report_annotation.mjs --wiki     # DokuWiki tables
//   node scripts/report_annotation.mjs --list llm # the papers behind a row
//   node scripts/report_annotation.mjs --quotes <regex>
//
// The page is about producing labels and showing they are right, so it has two
// populations and they are NOT the same one:
//
//   CLASSIFIED  4,439 papers with >=1 classification[] record — "something
//               assigned labels", by any method including a person
//   ANNOTATED   3,318 papers with >=1 humanAnnotation[] record — "a person
//               coded items by hand"
//
// statistics:interrater_agreement owns ANNOTATED and the agreement
// coefficients. This page owns CLASSIFIED and `classification[].validation`,
// and the intersection where an LLM does the labelling. Figures that belong to
// the sibling page are printed here only as the base rate for a subgroup
// comparison, and are labelled as such.
//
// Rules from data/extract/README.md, enforced rather than remembered: name the
// denominator, never count a sentinel as an answer, count papers not tuples,
// fold free text and print the residue.
 
import fs from 'node:fs';
import path from 'node:path';
import { loadExtractions, dataRoot, pct, table, wikiTable, YEAR_BUCKETS } from './lib.mjs';
import { FAMILIES, foldGroundTruth } from './gt_fold.mjs';
import {
  PROBE_AUDIT, TEMPERATURE_AUDIT, PROMPT_AUDIT, GROUPED_SPLIT_AUDIT, GROUPED_SPLIT_RE, RESIDUE_SAMPLE,
} from './annotation_audit.mjs';
 
const argv = process.argv.slice(2);
const WIKI = argv.includes('--wiki');
const LIST = argv.includes('--list') ? argv[argv.indexOf('--list') + 1] : null;
const QUOTES = argv.includes('--quotes') ? argv[argv.indexOf('--quotes') + 1] : null;
 
const rows = loadExtractions();
const key = (p) => `${p.venue}/${p.year}/${p.slug}`;
const out = [];
const say = (s = '') => out.push(s);
const T = (h, r) => say(WIKI ? wikiTable(h, r) : table(h, r));
const hdr = (s) => { say(); say('='.repeat(72)); say(s); say('='.repeat(72)); };
 
// ---------------------------------------------------------------- populations
const CLASSIFIED = rows.filter((p) => (p.classification ?? []).length > 0);
const ANNOTATED = rows.filter((p) => (p.humanAnnotation ?? []).length > 0);
const cKeys = new Set(CLASSIFIED.map(key));
const aKeys = new Set(ANNOTATED.map(key));
const BOTH = [...cKeys].filter((k) => aKeys.has(k)).length;
 
// `validation` has six enum values. Four are validation; `none-reported` is the
// paper saying there was none, and `not-applicable` is the extraction declining
// to judge. Only the four are counted as an answer anywhere below.
const REAL_VALIDATION = new Set([
  'manual-validation', 'cross-validation', 'held-out-test-set', 'comparison-to-other-method',
]);
const VALIDATION_ORDER = [
  'manual-validation', 'none-reported', 'not-applicable',
  'comparison-to-other-method', 'held-out-test-set', 'cross-validation',
];
const USED = new Set(['used', 'produced']);
const validates = (p, pred = () => true) =>
  p.classification.some((c) => pred(c) && REAL_VALIDATION.has(c.validation));
 
hdr('A. POPULATIONS');
T(['Population', 'Papers', 'Share of 5,859'], [
  ['all extracted papers', rows.length, pct(rows.length, rows.length)],
  ['CLASSIFIED — >=1 classification[] record', CLASSIFIED.length, pct(CLASSIFIED.length, rows.length)],
  ['ANNOTATED — >=1 humanAnnotation[] record', ANNOTATED.length, pct(ANNOTATED.length, rows.length)],
  ['both', BOTH, pct(BOTH, rows.length)],
  ['CLASSIFIED only', cKeys.size - BOTH, pct(cKeys.size - BOTH, rows.length)],
  ['ANNOTATED only', aKeys.size - BOTH, pct(aKeys.size - BOTH, rows.length)],
  ['neither', rows.length - (cKeys.size + aKeys.size - BOTH), pct(rows.length - (cKeys.size + aKeys.size - BOTH), rows.length)],
]);
say();
say(`Of the ${CLASSIFIED.length} CLASSIFIED papers, ${BOTH} (${pct(BOTH, CLASSIFIED.length)}) also hand-coded`);
say('something. That overlap is the whole subject of the page: in this literature a');
say('hand-coded set is usually not the result, it is the thing the automatic labels');
say('are checked against.');
 
hdr('B. classification[].validation — MULTI-VALUED, ROWS OVERLAP');
say(`Population: the ${CLASSIFIED.length} CLASSIFIED papers. A paper is counted in a row if ANY`);
say('of its classification tuples carries that value, so the rows sum past 100%.');
say();
{
  const r = VALIDATION_ORDER.map((v) => {
    const n = CLASSIFIED.filter((p) => p.classification.some((c) => c.validation === v)).length;
    return [v, n, pct(n, CLASSIFIED.length)];
  });
  T(['validation', 'Papers', `Share of ${CLASSIFIED.length}`], r);
}
say();
say('The same field read at PAPER level, where the rows are exclusive. This is the');
say('number to quote: "42.4% report no validation" is the share of papers with at');
say('least one unvalidated classifier, not the share that validated nothing.');
say();
{
  const anyReal = CLASSIFIED.filter((p) => validates(p));
  const allNA = CLASSIFIED.filter((p) => p.classification.every((c) => c.validation === 'not-applicable'));
  const noneRep = CLASSIFIED.filter((p) => !validates(p) && p.classification.some((c) => c.validation === 'none-reported'));
  T(['Paper-level verdict', 'Papers', `Share of ${CLASSIFIED.length}`], [
    ['validates at least one classifier', anyReal.length, pct(anyReal.length, CLASSIFIED.length)],
    ['validates nothing — at least one tuple says none-reported', noneRep.length, pct(noneRep.length, CLASSIFIED.length)],
    ['validates nothing — every tuple is the not-applicable sentinel', allNA.length, pct(allNA.length, CLASSIFIED.length)],
    ['validates nothing, either way', noneRep.length + allNA.length, pct(noneRep.length + allNA.length, CLASSIFIED.length)],
  ]);
}
 
hdr('C. VALIDATION RATE BY WHAT DID THE CLASSIFYING');
say(`Population: the ${CLASSIFIED.length} CLASSIFIED papers, split by classification[].method.`);
say('A paper appears in every row whose method it used. The share is of papers');
say('using that method whose tuple FOR THAT METHOD carries one of the four real');
say('validation values — so a paper that validates its own model but not the');
say('blocklist it also used counts as validated in one row and not in the other.');
say('The ALL row is the base rate and must be read with every other row.');
say();
{
  const methods = [...new Set(CLASSIFIED.flatMap((p) => p.classification.map((c) => c.method)))];
  const r = methods
    .map((m) => {
      const pap = CLASSIFIED.filter((p) => p.classification.some((c) => c.method === m));
      const val = pap.filter((p) => validates(p, (c) => c.method === m));
      return [m, pap.length, val.length, pct(val.length, pap.length), (100 * val.length) / pap.length];
    })
    .sort((a, b) => b[4] - a[4])
    .map((x) => x.slice(0, 4));
  // LIKE-FOR-LIKE BASE RATE. Each row above asks "does the tuple FOR THIS
  // METHOD validate". Comparing that to "does ANY tuple of the paper validate"
  // (70.1%) is not like for like: a paper with three classifiers clears the
  // second test on one of them and is counted once in each of three rows. A
  // 2026-09-11 review found every "far below the base rate" sentence on the
  // page overstated for exactly this reason. The comparable denominator is the
  // (paper, method) CELL.
  let cells = 0;
  let cval = 0;
  for (const p of CLASSIFIED)
    for (const m of new Set(p.classification.map((c) => c.method))) {
      cells += 1;
      if (validates(p, (c) => c.method === m)) cval += 1;
    }
  const base = CLASSIFIED.filter((p) => validates(p));
  r.push(['**ALL (paper,method) CELLS — the like-for-like base rate**', cells, cval, pct(cval, cells)]);
  r.push(['*any tuple of the paper validates — NOT comparable to the rows*', CLASSIFIED.length, base.length, pct(base.length, CLASSIFIED.length)]);
  T(['method', 'Papers / cells', 'Validated', 'Share'], r);
}
 
hdr('C2. VALIDATION RATE BY WHAT WAS CLASSIFIED');
say(`Population: the ${CLASSIFIED.length} CLASSIFIED papers, split by classification[].target,`);
say('which is an enum. A paper appears in every row whose target it classified. The');
say('share is of papers classifying that target whose tuple FOR THAT TARGET carries');
say('one of the four real validation values. The rightmost column is the page on');
say('this wiki that a reader with that target would be reading.');
say();
{
  const PAGE = {
    'website-category': 'design:website_classification', cookie: 'privacy:cookies',
    'web-request': 'privacy:requests', 'ip-address': 'design:ip_classification',
    'mobile-app': 'design:mobile_and_app_measurement', javascript: 'privacy:javascript',
    'fingerprinting-script': 'privacy:fingerprinting', 'privacy-policy': 'privacy:policies',
    'consent-notice': 'privacy:consent', 'dark-pattern': 'privacy:darkpatterns',
    'website-popularity': 'design:website_selection', 'email-message': 'privacy:email_tracking',
  };
  const targets = [...new Set(CLASSIFIED.flatMap((p) => p.classification.map((c) => c.target)))];
  const r = targets
    .map((t) => {
      const pap = CLASSIFIED.filter((p) => p.classification.some((c) => c.target === t));
      const v = pap.filter((p) => validates(p, (c) => c.target === t));
      const g = pap.filter((p) => p.classification.some((c) => c.target === t && c.groundTruthSource));
      return [t, pap.length, pct(v.length, pap.length), pct(g.length, pap.length), PAGE[t] ?? '—', pap.length];
    })
    .sort((a, b) => b[5] - a[5])
    .map((x) => x.slice(0, 5));
  // Same like-for-like correction as section C, on (paper, target) cells.
  let cells = 0;
  let cval = 0;
  let cgt = 0;
  for (const p of CLASSIFIED)
    for (const t of new Set(p.classification.map((c) => c.target))) {
      cells += 1;
      if (validates(p, (c) => c.target === t)) cval += 1;
      if (p.classification.some((c) => c.target === t && c.groundTruthSource)) cgt += 1;
    }
  const base = CLASSIFIED.filter((p) => validates(p));
  const gb = CLASSIFIED.filter((p) => p.classification.some((c) => c.groundTruthSource));
  r.push(['**ALL (paper,target) CELLS — the like-for-like base rate**', cells, pct(cval, cells), pct(cgt, cells), '—']);
  r.push(['*any tuple of the paper — NOT comparable to the rows*', CLASSIFIED.length, pct(base.length, CLASSIFIED.length), pct(gb.length, CLASSIFIED.length), '—']);
  T(['target', 'Papers / cells', 'Validates', 'Names a ground truth', 'Page'], r);
}
 
hdr('D. HAS VALIDATION REPORTING IMPROVED? (no)');
say(`Population: the ${CLASSIFIED.length} CLASSIFIED papers, by publication year.`);
say('2025 and 2026 are provisional venue-years (see literature:corpus).');
say();
{
  const years = [...new Set(rows.map((p) => p.year))].sort();
  const r = years.map((y) => {
    const s = CLASSIFIED.filter((p) => p.year === y);
    const v = s.filter((p) => validates(p));
    const g = s.filter((p) => p.classification.some((c) => c.groundTruthSource));
    return [String(y) + (y >= 2025 ? '*' : ''), s.length, pct(v.length, s.length), pct(g.length, s.length)];
  });
  T(['Year', 'CLASSIFIED', 'Validates something', 'Names a ground truth'], r);
  say();
  say('The same thing in the shared four-year buckets, which is the form the page');
  say('publishes because the per-year series is noisy at n < 150.');
  say();
  T(['Years', 'CLASSIFIED', 'Validates something', 'Names a ground truth'],
    YEAR_BUCKETS.map(([name, fn]) => {
      const s2 = CLASSIFIED.filter((p) => fn(p.year));
      const v = s2.filter((p) => validates(p));
      const g = s2.filter((p) => p.classification.some((c) => c.groundTruthSource));
      return [name, s2.length, pct(v.length, s2.length), pct(g.length, s2.length)];
    }));
}
 
hdr('D2. WHICH KIND OF VALIDATION, OVER TIME');
say(`Population: the ${CLASSIFIED.length} CLASSIFIED papers, bucketed. Rows overlap (a paper can`);
say('carry several kinds). This is the "is the method current" question for the');
say('validation half of the page, and it has an answer: cross-validation is being');
say('abandoned, held-out test sets plateaued after 2018, and manual validation of a');
say('sample has been the dominant instrument throughout.');
say();
{
  const KINDS = [...VALIDATION_ORDER];
  T(['Years', 'CLASSIFIED', ...KINDS],
    YEAR_BUCKETS.map(([name, fn]) => {
      const s2 = CLASSIFIED.filter((p) => fn(p.year));
      return [name, s2.length,
        ...KINDS.map((k) => pct(s2.filter((p) => p.classification.some((c) => c.validation === k)).length, s2.length))];
    }));
  say();
  say('The none-reported jump in 2025-2026 (38.2% -> 51.5%) is the largest move in the');
  say('table and it lands entirely in the provisional slice, so it is NOT publishable');
  say('as a trend. It is flagged on the page as a thing to re-check when those');
  say('venue-years fill in, not as a finding.');
}
 
hdr('E. WHERE GROUND TRUTH COMES FROM (folded, residue printed)');
{
  const stated = CLASSIFIED.filter((p) => p.classification.some((c) => c.groundTruthSource));
  const tax = CLASSIFIED.filter((p) => p.classification.some((c) => c.taxonomy));
  say(`Of the ${CLASSIFIED.length} CLASSIFIED papers, ${stated.length} (${pct(stated.length, CLASSIFIED.length)}) name a groundTruthSource`);
  say(`on at least one tuple, and ${tax.length} (${pct(tax.length, CLASSIFIED.length)}) name a taxonomy.`);
  say();
  say('groundTruthSource is free text: 7,114 tuples carry one, with 6,223 distinct');
  say('lower-cased strings, so an exact-string ranking is meaningless. scripts/gt_fold.mjs');
  say('folds them by WHAT KIND of authority the ground truth is — not by resource name —');
  say('because that is what decides whether the check is independent of the classifier.');
  say('Counted by PAPER: a paper is in a family if any of its tuples folds there.');
  say();
  const famPapers = new Map();
  const residue = new Map();
  let residuePapers = new Set();
  for (const p of CLASSIFIED)
    for (const c of p.classification) {
      if (!c.groundTruthSource) continue;
      const f = foldGroundTruth(c.groundTruthSource);
      if (f === null) {
        residue.set(c.groundTruthSource.toLowerCase(), (residue.get(c.groundTruthSource.toLowerCase()) ?? 0) + 1);
        residuePapers.add(key(p));
      } else {
        if (!famPapers.has(f)) famPapers.set(f, new Set());
        famPapers.get(f).add(key(p));
      }
    }
  const seen = new Set();
  const r = [];
  for (const f of FAMILIES) {
    if (seen.has(f.key)) continue;
    seen.add(f.key);
    const n = famPapers.get(f.key)?.size ?? 0;
    r.push([f.key, f.label, f.independent ? 'yes' : 'NO', n, pct(n, stated.length)]);
  }
  r.sort((a, b) => b[3] - a[3]);
  r.push(['RESIDUE', 'unmapped — no keyword family fits', '—', residuePapers.size, pct(residuePapers.size, stated.length)]);
  T(['family', 'What the ground truth is', 'Independent of the classifier', 'Papers', `Share of ${stated.length}`], r);
  const resTuples = [...residue.values()].reduce((a, b) => a + b, 0);
  say();
  say(`RESIDUE: ${resTuples} of the 7,114 tuples (${pct(resTuples, 7114)}), ${residue.size} distinct strings, touching`);
  say(`${residuePapers.size} papers (${pct(residuePapers.size, stated.length)} of the ${stated.length}, high because a paper has several tuples). Printed in full`);
  say('by `node scripts/gt_fold.mjs --residue`.');
  say();
  const once = [...residue.values()].filter((n) => n === 1).length;
  say(`It is a LONG TAIL: ${once} of the ${residue.size} distinct strings (${pct(once, residue.size)}) appear exactly`);
  say('once. Each names a resource or a procedure specific to one paper, which no');
  say('keyword family can be expected to reach. The most frequent unmapped strings,');
  say('READ OUT OF THE MAP rather than retyped — retyping them is precisely the defect');
  say('a reviewer found in the previous version of this block on 2026-09-11:');
  say();
  T(['Unmapped string', 'Tuples'],
    [...residue.entries()].sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0])).slice(0, 8)
      .map(([str, n]) => [str, n]));
  say();
  say('The page quotes four of these verbatim. If this table changes, the page is');
  say('stale and check_page_numbers.mjs will not notice, because these are words.');
  say();
  say('A 30-string systematic sample (every 73rd line of the printed residue) was');
  say('hand-classified in scripts/annotation_audit.mjs into the family each string');
  say('WOULD have joined. It spreads across every family rather than piling into one:');
  say();
  {
    const rv = {};
    for (const v of Object.values(RESIDUE_SAMPLE)) rv[v] = (rv[v] ?? 0) + 1;
    T(['Family the sampled string would have joined', 'of 30'],
      Object.entries(rv).sort((a, b) => b[1] - a[1]));
  }
  say();
  say('So EVERY family above is a floor, not just one of them, and none is');
  say('systematically missing. ("unclear" means the string alone does not say.)');
  say('An earlier draft of this script asserted that the residue was dominated by');
  say('by-construction setups and gave three examples. All three in fact FOLD');
  say('correctly — two are gt_fold.mjs self-test fixtures — and a reviewer caught it');
  say('on 2026-09-11. The sample above is what replaced the claim.');
}
 
hdr('F. HOW THE HAND CODING WAS DONE (the sibling page owns the coefficients)');
say(`Population: the ${ANNOTATED.length} ANNOTATED papers. humanAnnotation[].codingApproach is an`);
say('enum; not-stated is a sentinel and is shown as its own row, never as an answer.');
say();
{
  const vals = [...new Set(ANNOTATED.flatMap((p) => p.humanAnnotation.map((a) => a.codingApproach)))];
  const r = vals
    .map((v) => {
      const n = ANNOTATED.filter((p) => p.humanAnnotation.some((a) => a.codingApproach === v)).length;
      return [v + (v === 'not-stated' ? ' (SENTINEL)' : ''), n, pct(n, ANNOTATED.length), n];
    })
    .sort((a, b) => b[3] - a[3])
    .map((x) => x.slice(0, 3));
  T(['codingApproach', 'Papers', `Share of ${ANNOTATED.length}`], r);
}
 
// ------------------------------------------------------------------- LLM half
const LLM = CLASSIFIED.filter((p) => p.classification.some((c) => c.method === 'llm' && USED.has(c.usedOrMentioned)));
const LLM_ANY = CLASSIFIED.filter((p) => p.classification.some((c) => c.method === 'llm'));
 
hdr('G. THE LLM AS ANNOTATOR');
say(`${LLM_ANY.length} papers carry a classification tuple with method == "llm"; ${LLM.length} of them`);
say('USED or PRODUCED the labels rather than only comparing against an LLM baseline.');
say(`The ${LLM.length} is this section's population. It is the same field and the same`);
say('definition design:website_classification uses for its per-target table, on');
say('purpose, so the two pages cannot drift.');
say();
{
  const years = [...new Set(rows.map((p) => p.year))].sort().filter((y) => y >= 2022);
  // BOTH series are printed because design:website_classification's per-year
  // table counts all 177 (it is asking "has an LLM arrived at this target"),
  // while this page counts the 175 that used the labels (it is asking "did
  // these papers validate their labels"). The two-paper gap is why that page
  // says 77 and 71 where this one says 76 and 70. Printing both here is what
  // stops a reviewer reading it as a drift.
  const r = years.map((y) => {
    const cls = CLASSIFIED.filter((p) => p.year === y).length;
    const n = LLM.filter((p) => p.year === y).length;
    const nAny = LLM_ANY.filter((p) => p.year === y).length;
    return [String(y) + (y >= 2025 ? '*' : ''), n, pct(n, cls), nAny, pct(nAny, cls), cls];
  });
  T(['Year', 'LLM labellers (used/produced, 175)', 'Share', 'Any llm tuple (177)', 'Share', 'CLASSIFIED that year'], r);
}
say();
{
  const alsoHuman = LLM.filter((p) => (p.humanAnnotation ?? []).length > 0);
  const agreement = LLM.filter((p) => (p.humanAnnotation ?? []).some((a) => a.agreementMetric));
  const anAgree = ANNOTATED.filter((p) => p.humanAnnotation.some((a) => a.agreementMetric));
  const manVal = LLM.filter((p) => p.classification.some((c) => c.method === 'llm' && USED.has(c.usedOrMentioned) && c.validation === 'manual-validation'));
  const heldOut = LLM.filter((p) => p.classification.some((c) => c.method === 'llm' && USED.has(c.usedOrMentioned) && c.validation === 'held-out-test-set'));
  const anyVal = LLM.filter((p) => validates(p, (c) => c.method === 'llm' && USED.has(c.usedOrMentioned)));
  const noneRep = LLM.filter((p) => p.classification.some((c) => c.method === 'llm' && USED.has(c.usedOrMentioned) && c.validation === 'none-reported'));
  const nothing = LLM.filter((p) => !validates(p, (c) => c.method === 'llm' && USED.has(c.usedOrMentioned)) && (p.humanAnnotation ?? []).length === 0);
  T(['Question about the LLM labels', 'Papers', `Share of ${LLM.length}`], [
    ['the LLM tuple carries some validation', anyVal.length, pct(anyVal.length, LLM.length)],
    ['…manual validation of a sample', manVal.length, pct(manVal.length, LLM.length)],
    ['…a held-out test set', heldOut.length, pct(heldOut.length, LLM.length)],
    ['the LLM tuple says none-reported', noneRep.length, pct(noneRep.length, LLM.length)],
    ['the paper also hand-codes something', alsoHuman.length, pct(alsoHuman.length, LLM.length)],
    ['the paper reports an agreement metric', agreement.length, pct(agreement.length, LLM.length)],
    ['NEITHER validation nor any hand coding', nothing.length, pct(nothing.length, LLM.length)],
  ]);
  say();
  say(`BASE RATE for the agreement row, which is the sibling page's population, not`);
  say(`this one's: ${anAgree.length} of ${ANNOTATED.length} ANNOTATED papers (${pct(anAgree.length, ANNOTATED.length)}) name an agreement`);
  say(`metric. ${LLM.length - alsoHuman.length} of the ${LLM.length} LLM papers hand-code nothing at all.`);
  say('metric. LIKE FOR LIKE, because a paper that hand-codes nothing cannot report a');
  say(`coefficient: ${agreement.length} of the ${alsoHuman.length} LLM papers that DO hand-code (${pct(agreement.length, alsoHuman.length)}) name one,`);
  say(`against the same ${pct(anAgree.length, ANNOTATED.length)}. The 33.7%-of-175 figure understates the gap and the`);
  say('page quotes the like-for-like one. The likeliest explanation is structural: when');
  say('a model does the first pass, the human coding IS the validation, so a paper');
  say('without a number has nothing to report. That is an interpretation, not a test.');
}
say();
{
  const gts = [];
  for (const p of LLM)
    for (const c of p.classification)
      if (c.method === 'llm' && USED.has(c.usedOrMentioned) && c.groundTruthSource) gts.push([key(p), c.groundTruthSource]);
  const fam = new Map();
  for (const [k, s] of gts) {
    const f = foldGroundTruth(s) ?? 'RESIDUE';
    if (!fam.has(f)) fam.set(f, new Set());
    fam.get(f).add(k);
  }
  const stated = new Set(gts.map(([k]) => k)).size;
  say(`Ground truth for the LLM labels: ${stated} of ${LLM.length} papers (${pct(stated, LLM.length)}) state one on an`);
  say('LLM tuple. Folded by the same rule as section E, counted by paper:');
  say();
  T(['family', 'Papers', `Share of ${stated}`],
    [...fam.entries()].sort((a, b) => b[1].size - a[1].size).map(([f, s]) => [f, s.size, pct(s.size, stated)]));
}
 
hdr('H. WHAT THE 175 SAY ABOUT HOW THEY RAN THE MODEL');
say('Full-text probes over the 175 LLM-labelling papers, whitespace collapsed and');
say('soft hyphens rejoined. A probe is a candidate set; the temperature row is the');
say('only one hand-audited, and it is the only one quoted on the page as a measured');
say('share. The rest are LOWER BOUNDS and are labelled as such.');
say();
{
  const norm = (s) => s.replace(/­/g, '').replace(/-\s*\n\s*/g, '').replace(/\s+/g, ' ');
  const texts = new Map();
  for (const p of LLM)
    texts.set(key(p), norm(fs.readFileSync(path.join(dataRoot(), 'fulltext', String(p.year), p.venue, p.slug, 'paper.cols.txt'), 'utf8')));
  const probe = (re) => [...texts.values()].filter((t) => re.test(t)).length;
  const PROMPT_LOOSE = /prompt/i;
  const PROMPT_PUB =
    /(full|exact|complete|verbatim) prompts?|prompts? (are|is|used are)? ?(listed|shown|given|provided|included|available|reproduced) in (the )?(appendix|artifact|repository|supplementary|table|figure|listing)|prompts? in (the )?appendix|see appendix[^.]{0,40}prompt|prompt templates? (in|are) /i;
  // 'top_p' and 'top-\u{1D45D}' (the PDF's math-italic p) both occur; a naive /top[- ]?p/
  // misses both and reported 23 where the answer is larger.
  const DECODE_OTHER = /top[-_ ]?[pk\u{1D45D}\u{1D458}]\b|top[-_ ]?[pk\u{1D45D}\u{1D458}][^a-z]|nucleus sampling|greedy decoding|max[_ ]tokens|num_return_sequences|frequency penalty|presence penalty/iu;
  const tempLoose = probe(/temperature/i);
  const tempValue = Object.values(TEMPERATURE_AUDIT).filter(([v]) => v === 'value').length;
  const pl = probe(PROMPT_LOOSE);
  const pp = probe(PROMPT_PUB);
  const promptPub = Object.values(PROMPT_AUDIT).filter(([v]) => v === 'published').length;
  if (promptPub > pp) throw new Error(`audited prompt ${promptPub} > loose probe ${pp}`);
  if (pp > pl) throw new Error(`tight prompt probe ${pp} > loose ${pl}`);
  if (tempValue > tempLoose) throw new Error(`audited temperature ${tempValue} > loose probe ${tempLoose}`);
  T(['Probe', 'Width', 'Papers', `Share of ${LLM.length}`], [
    ['the word "temperature" appears', 'loose', tempLoose, pct(tempLoose, LLM.length)],
    ['**states a numeric decoding temperature** (HAND-AUDITED)', 'audited', tempValue, pct(tempValue, LLM.length)],
    ['names another decoding setting (top-p / top-k / max_tokens)', 'lower bound', probe(DECODE_OTHER), pct(probe(DECODE_OTHER), LLM.length)],
    ['the word "prompt" appears', 'loose', pl, pct(pl, LLM.length)],
    ['the full text says a prompt is reproduced somewhere', 'loose', pp, pct(pp, LLM.length)],
    ['**a prompt really is reproduced** (HAND-AUDITED)', 'audited', promptPub, pct(promptPub, LLM.length)],
  ]);
  say();
  say('HOW MANY OF THE THREE a single paper supplies. The page\'s checklist asks for');
  say('six things from an LLM annotator; three of them are measurable here, and this');
  say('is the joint distribution rather than three marginals:');
  say();
  {
    const tempSet = new Set(Object.entries(TEMPERATURE_AUDIT).filter(([, v]) => v[0] === 'value').map(([k]) => k));
    const promSet = new Set(Object.entries(PROMPT_AUDIT).filter(([, v]) => v[0] === 'published').map(([k]) => k));
    const dist = [0, 0, 0, 0];
    for (const p of LLM) {
      let n = 0;
      if (tempSet.has(key(p))) n += 1;
      if (promSet.has(key(p))) n += 1;
      if (DECODE_OTHER.test(texts.get(key(p)))) n += 1;
      dist[n] += 1;
    }
    T(['Items supplied (of 3)', 'Papers', `Share of ${LLM.length}`],
      dist.map((n, i) => [i, n, pct(n, LLM.length)]));
    let cum = 0;
    let med = 0;
    for (let i = 0; i < 4; i += 1) { cum += dist[i]; if (cum >= LLM.length / 2) { med = i; break; } }
    say();
    say(`MEDIAN = ${med}. The median paper that lets a model assign labels supplies none of`);
    say('the three. Two papers of 175 supply all three.');
  }
  say();
  say('Temperature hand audit (scripts/annotation_audit.mjs, 55 candidates read in');
  say('context — the word has four other senses in this corpus):');
  const tv = {};
  for (const [, [v]] of Object.entries(TEMPERATURE_AUDIT)) tv[v] = (tv[v] ?? 0) + 1;
  T(['Verdict', 'Papers', `Share of the ${tempLoose} candidates`],
    Object.entries(tv).sort((a, b) => b[1] - a[1]).map(([v, n]) => [v, n, pct(n, tempLoose)]));
  say();
  say('Prompt hand audit (same script, 31 candidates). "declined" is one paper that');
  say('matched because it says it is deliberately WITHHOLDING its prompts, which is');
  say('the opposite of what the probe is for. 26 is a lower bound in the other');
  say('direction too: a prompt printed in an appendix figure with no sentence');
  say('pointing at it is invisible to any probe of this shape.');
  const pv = {};
  for (const [, [v]] of Object.entries(PROMPT_AUDIT)) pv[v] = (pv[v] ?? 0) + 1;
  T(['Verdict', 'Papers', `Share of the ${pp} candidates`],
    Object.entries(pv).sort((a, b) => b[1] - a[1]).map(([v, n]) => [v, n, pct(n, pp)]));
}
 
hdr('J. THE HELD-OUT SET, AND WHETHER IT IS HELD OUT');
say('A held-out test set drawn by shuffling rows is not held out when the rows come');
say('in groups — pages of one site share a template, a tag manager and a consent');
say('vendor, so a classifier that memorises the site scores against itself. The unit');
say('of independence on a crawl is the site, not the page.');
say();
{
  const HOW = rows.filter((p) => p.platforms.includes('web') &&
    p.classification?.some((c) => c.validation === 'held-out-test-set' || c.validation === 'cross-validation'));
  const norm = (s2) => s2.replace(/­/g, '').replace(/-\s*\n\s*/g, '').replace(/\s+/g, ' ');
  const gHits = HOW.filter((p) =>
    GROUPED_SPLIT_RE.test(norm(fs.readFileSync(path.join(dataRoot(), 'fulltext', String(p.year), p.venue, p.slug, 'paper.cols.txt'), 'utf8'))));
  const grouped = Object.values(GROUPED_SPLIT_AUDIT).filter(([v]) => v === 'grouped').length;
  T(['Question', 'Papers', 'Share'], [
    ['web-platform papers using a held-out test set or cross-validation', HOW.length, '—'],
    ['…whose full text contains ANY grouped-split language (probe)', gHits.length, pct(gHits.length, HOW.length)],
    ['…**that really split or evaluate across a group** (HAND-AUDITED)', grouped, pct(grouped, HOW.length)],
  ]);
  say();
  say('This is a LOWER BOUND in both directions. A paper that split by site without');
  say('writing a sentence about it is invisible; so is one phrasing it outside the');
  say('regex. What the number will support is "this is rare and almost never stated",');
  say('not "only seven papers did it". The regex, and the ten candidates with the');
  say('deciding phrase for each, are in scripts/annotation_audit.mjs.');
  const gv = {};
  for (const [, [v]] of Object.entries(GROUPED_SPLIT_AUDIT)) gv[v] = (gv[v] ?? 0) + 1;
  say();
  T(['Verdict', 'Papers', `Share of the ${gHits.length} candidates`],
    Object.entries(gv).sort((a, b) => b[1] - a[1]).map(([v, n]) => [v, n, pct(n, gHits.length)]));
}
 
hdr('I. THE TITLE PROBE THAT PUT THIS PAGE ON THE ROADMAP, AUDITED');
say('scripts/gap_probe_roadmap.mjs `llm_annotator` is a title+summary regex. The');
say('roadmap row says 30 papers. Every one was read; the verdicts are in');
say('scripts/annotation_audit.mjs with the deciding reason per paper.');
say();
{
  const v = {};
  for (const [, [x]] of Object.entries(PROBE_AUDIT)) v[x] = (v[x] ?? 0) + 1;
  const n = Object.keys(PROBE_AUDIT).length;
  T(['Verdict', 'Papers', `Share of ${n}`], Object.entries(v).sort((a, b) => b[1] - a[1]).map(([k2, c]) => [k2, c, pct(c, n)]));
  const llmKeys = new Set(LLM.map(key));
  const caught = Object.keys(PROBE_AUDIT).filter((k) => llmKeys.has(k)).length;
  say();
  say(`PRECISION: ${v.annotator} of ${n} candidates (${pct(v.annotator, n)}) actually point a model at their own`);
  say('data. The rest study LLMs or attacks on them, or use one as a generator.');
  say(`RECALL: the probe catches ${caught} of the ${LLM.length} papers the enum finds (${pct(caught, LLM.length)}).`);
  say('A title+summary probe is a candidate set. The enum is the population, and the');
  say('page uses the enum for every figure it publishes.');
}
 
hdr('Z. NUMBERS ON THE PAGE THAT DO NOT COME FROM THIS SCRIPT');
say('Each is sourced on the page itself and on provenance:statistics:annotation.');
say();
T(['Figure', 'Where it is from'], [
  ['0.99 — "published F1 scores of up to 0.99"', 'pendlebury2019_tesseract, abstract, verbatim in paper.cols.txt'],
  ['20,000 requests heuristic-labelled and expert-validated as ground truth', 'xiong2026_tgnn, appendix, verbatim'],
  ['98.17% F1 for the LLM labelling component, on the whole ground truth', 'xiong2026_tgnn §5.5, verbatim; the extraction records that tuple as held-out-test-set, which the paper does not support'],
  ['200 privacy policies, 6 experts, 2 per policy', 'cory2026_wordlevel, methodology, verbatim'],
  ['0.964 and 0.659, the demo\'s two precisions, and the 96.6% of a 3.4% class that is negative', 'pages/holdout_by_site.py demo output, quoted on the page'],
  ['illustrative arithmetic: a 1,000-item sample at 3.4% positives holds ~34', 'the page; 34/34, 30/34 and 25/34 precisions with 95% Wilson intervals 10, 22 and 29 points wide come from pages/holdout_by_site.py demo'],
  ['5,859 / 4,439 / 3,318 / 1,120 / 1,762', 'data/extract/OVERVIEW.md corpus populations'],
  ['15.4% agreement-metric base rate', 'computed here in G, and matches statistics:interrater_agreement'],
  ['252 papers folding to Cohen\'s kappa, 18 self-agreement papers, 34.1% vs 13.8%', 'statistics:interrater_agreement and its report script — NOT re-derived here; its LLM-signal matcher is full-text where this page\'s is the classification.method enum'],
  ['19.4% of 175 name a resolvable model', 'design:website_classification / scripts/report_llm_currency.mjs'],
  ['37.3% validation rate for topic classification', 'design:website_classification, population 330 not 4,439'],
  ['Wilson interval arithmetic in pages/holdout_by_site.py', 'the script itself; it prints its own self-test'],
  ['46.5% -> 75.9% artifact-link share, 2018 -> 2026', 'data/extract/OVERVIEW.md per-year table; a DIFFERENT field from artifacts.links, which gives 86.2% -> 93.0%'],
  ['6 distinct gpt-4 snapshots', 'design:website_classification, counted from OpenAI deprecations 2026-09-03'],
  ['58% run-to-run agreement of classification.method', 'data/extract/README.md stability audit (old run; order of magnitude)'],
]);
 
// ------------------------------------------------------------------ selectors
if (LIST === 'llm') { for (const p of LLM.sort((a, b) => a.year - b.year)) console.log(`${p.year}\t${p.venue}\t${p.title}`); process.exit(0); }
if (LIST === 'novalidation') {
  for (const p of CLASSIFIED.filter((q) => !validates(q)).sort((a, b) => a.year - b.year)) console.log(`${p.year}\t${p.venue}\t${p.title}`);
  process.exit(0);
}
if (QUOTES) {
  const re = new RegExp(QUOTES, 'i');
  for (const p of CLASSIFIED)
    for (const c of p.classification)
      if (re.test(c.resourceName ?? '') || re.test(c.groundTruthSource ?? ''))
        console.log(`${key(p)}\n  [${c.method}/${c.validation}] ${c.resourceName} <- ${c.groundTruthSource}\n  "${c.evidence.quote}" (${c.evidence.section})`);
  process.exit(0);
}
 
console.log(out.join('\n'));

node scripts/report_annotation.mjs, run 2026-09-11 against data/extract/run1.

report_annotation-output.txt

========================================================================
A. POPULATIONS
========================================================================
Population                                Papers  Share of 5,859
----------------------------------------  ------  --------------
all extracted papers                      5859    100.0%
CLASSIFIED — >=1 classification[] record  4439    75.8%
ANNOTATED — >=1 humanAnnotation[] record  3318    56.6%
both                                      2927    50.0%
CLASSIFIED only                           1512    25.8%
ANNOTATED only                            391     6.7%
neither                                   1029    17.6%
 
Of the 4439 CLASSIFIED papers, 2927 (65.9%) also hand-coded
something. That overlap is the whole subject of the page: in this literature a
hand-coded set is usually not the result, it is the thing the automatic labels
are checked against.
 
========================================================================
B. classification[].validation — MULTI-VALUED, ROWS OVERLAP
========================================================================
Population: the 4439 CLASSIFIED papers. A paper is counted in a row if ANY
of its classification tuples carries that value, so the rows sum past 100%.
 
validation                  Papers  Share of 4439
--------------------------  ------  -------------
manual-validation           2265    51.0%
none-reported               1881    42.4%
not-applicable              1160    26.1%
comparison-to-other-method  791     17.8%
held-out-test-set           547     12.3%
cross-validation            314     7.1%
 
The same field read at PAPER level, where the rows are exclusive. This is the
number to quote: "42.4% report no validation" is the share of papers with at
least one unvalidated classifier, not the share that validated nothing.
 
Paper-level verdict                                             Papers  Share of 4439
--------------------------------------------------------------  ------  -------------
validates at least one classifier                               3111    70.1%
validates nothing — at least one tuple says none-reported       866     19.5%
validates nothing — every tuple is the not-applicable sentinel  462     10.4%
validates nothing, either way                                   1328    29.9%
 
========================================================================
C. VALIDATION RATE BY WHAT DID THE CLASSIFYING
========================================================================
Population: the 4439 CLASSIFIED papers, split by classification[].method.
A paper appears in every row whose method it used. The share is of papers
using that method whose tuple FOR THAT METHOD carries one of the four real
validation values — so a paper that validates its own model but not the
blocklist it also used counts as validated in one row and not in the other.
The ALL row is the base rate and must be read with every other row.
 
method                                                           Papers / cells  Validated  Share
---------------------------------------------------------------  --------------  ---------  -----
supervised-ml                                                    1087            922        84.8%
llm                                                              177             148        83.6%
static-analysis                                                  472             361        76.5%
dynamic-analysis                                                 300             220        73.3%
graph-analysis                                                   154             102        66.2%
unsupervised-ml                                                  320             198        61.9%
manual-labelling                                                 1381            843        61.0%
regex-or-signature                                               310             171        55.2%
heuristic-rules                                                  2183            1198       54.9%
other                                                            394             205        52.0%
blocklist                                                        189             61         32.3%
third-party-service                                              643             195        30.3%
curated-database                                                 649             180        27.7%
**ALL (paper,method) CELLS — the like-for-like base rate**       8259            4804       58.2%
*any tuple of the paper validates — NOT comparable to the rows*  4439            3111       70.1%
 
========================================================================
C2. VALIDATION RATE BY WHAT WAS CLASSIFIED
========================================================================
Population: the 4439 CLASSIFIED papers, split by classification[].target,
which is an enum. A paper appears in every row whose target it classified. The
share is of papers classifying that target whose tuple FOR THAT TARGET carries
one of the four real validation values. The rightmost column is the page on
this wiki that a reader with that target would be reading.
 
target                                                      Papers / cells  Validates  Names a ground truth  Page
----------------------------------------------------------  --------------  ---------  --------------------  ---------------------------------
other                                                       2594            64.0%      69.3%                 —
vulnerability                                               883             71.5%      72.6%                 —
website-category                                            424             45.3%      57.3%                 design:website_classification
user-generated-text                                         419             71.6%      57.8%                 —
network-traffic                                             383             58.7%      58.2%                 —
domain                                                      351             41.0%      59.5%                 —
ip-address                                                  295             33.6%      43.1%                 design:ip_classification
mobile-app                                                  282             66.3%      75.5%                 design:mobile_and_app_measurement
web-request                                                 262             48.9%      51.1%                 privacy:requests
malware                                                     160             51.9%      78.1%                 —
privacy-policy                                              102             72.5%      70.6%                 privacy:policies
sdk-or-library                                              77              50.6%      64.9%                 —
email-message                                               54              50.0%      66.7%                 privacy:email_tracking
cookie                                                      53              32.1%      56.6%                 privacy:cookies
javascript                                                  44              63.6%      59.1%                 privacy:javascript
consent-notice                                              39              74.4%      61.5%                 privacy:consent
fingerprinting-script                                       32              59.4%      65.6%                 privacy:fingerprinting
website-popularity                                          16              25.0%      31.3%                 design:website_selection
dark-pattern                                                13              84.6%      76.9%                 privacy:darkpatterns
**ALL (paper,target) CELLS — the like-for-like base rate**  6483            60.1%      65.2%                 —
*any tuple of the paper — NOT comparable to the rows*       4439            70.1%      72.9%                 —
 
========================================================================
D. HAS VALIDATION REPORTING IMPROVED? (no)
========================================================================
Population: the 4439 CLASSIFIED papers, by publication year.
2025 and 2026 are provisional venue-years (see literature:corpus).
 
Year   CLASSIFIED  Validates something  Names a ground truth
-----  ----------  -------------------  --------------------
2010   77          63.6%                55.8%
2011   83          63.9%                71.1%
2012   108         55.6%                61.1%
2013   96          59.4%                68.8%
2014   126         69.0%                73.8%
2015   140         69.3%                63.6%
2016   144         66.0%                70.1%
2017   185         63.2%                72.4%
2018   192         71.4%                73.4%
2019   293         69.3%                74.1%
2020   315         75.6%                72.7%
2021   296         71.3%                73.0%
2022   412         74.5%                72.8%
2023   503         72.4%                75.0%
2024   529         72.0%                72.6%
2025*  607         69.2%                74.3%
2026*  333         70.6%                80.5%
 
The same thing in the shared four-year buckets, which is the form the page
publishes because the per-year series is noisy at n < 150.
 
Years       CLASSIFIED  Validates something  Names a ground truth
----------  ----------  -------------------  --------------------
2010–2013   364         60.2%                64.3%
2014–2017   595         66.6%                70.1%
2018–2021   1096        72.0%                73.3%
2022–2024   1444        72.9%                73.5%
2025–2026*  940         69.7%                76.5%
 
========================================================================
D2. WHICH KIND OF VALIDATION, OVER TIME
========================================================================
Population: the 4439 CLASSIFIED papers, bucketed. Rows overlap (a paper can
carry several kinds). This is the "is the method current" question for the
validation half of the page, and it has an answer: cross-validation is being
abandoned, held-out test sets plateaued after 2018, and manual validation of a
sample has been the dominant instrument throughout.
 
Years       CLASSIFIED  manual-validation  none-reported  not-applicable  comparison-to-other-method  held-out-test-set  cross-validation
----------  ----------  -----------------  -------------  --------------  --------------------------  -----------------  ----------------
2010–2013   364         40.4%              41.5%          29.9%           16.8%                       6.9%               7.7%
2014–2017   595         47.4%              43.5%          32.3%           17.6%                       10.9%              9.9%
2018–2021   1096        53.6%              39.8%          26.1%           16.7%                       12.5%              9.9%
2022–2024   1444        52.5%              38.2%          25.1%           17.0%                       13.8%              5.8%
2025–2026*  940         52.1%              51.5%          22.3%           21.0%                       12.9%              3.6%
 
The none-reported jump in 2025-2026 (38.2% -> 51.5%) is the largest move in the
table and it lands entirely in the provisional slice, so it is NOT publishable
as a trend. It is flagged on the page as a thing to re-check when those
venue-years fill in, not as a finding.
 
========================================================================
E. WHERE GROUND TRUTH COMES FROM (folded, residue printed)
========================================================================
Of the 4439 CLASSIFIED papers, 3234 (72.9%) name a groundTruthSource
on at least one tuple, and 2854 (64.3%) name a taxonomy.
 
groundTruthSource is free text: 7,114 tuples carry one, with 6,223 distinct
lower-cased strings, so an exact-string ranking is meaningless. scripts/gt_fold.mjs
folds them by WHAT KIND of authority the ground truth is — not by resource name —
because that is what decides whether the check is independent of the classifier.
Counted by PAPER: a paper is in a family if any of its tuples folds there.
 
family                        What the ground truth is                     Independent of the classifier  Papers  Share of 3234
----------------------------  -------------------------------------------  -----------------------------  ------  -------------
human-annotation              Human annotation of a sample                 yes                            1683    52.0%
by-construction               Known by construction (experimental design)  yes                            350     10.8%
existing-dataset              An existing labelled dataset or benchmark    yes                            287     8.9%
third-party-service           A third-party service’s labels               yes                            234     7.2%
curated-list                  A curated list or blocklist                  yes                            186     5.8%
documentation-or-declaration  Documentation, source code or a declaration  yes                            104     3.2%
execution-or-oracle           Execution, probing or a reproducible oracle  yes                            88      2.7%
rule-or-tool-output           Another automated method’s output            NO                             74      2.3%
model-output                  Another model’s output                       NO                             9       0.3%
RESIDUE                       unmapped — no keyword family fits            —                              1577    48.8%
 
RESIDUE: 2434 of the 7,114 tuples (34.2%), 2206 distinct strings, touching
1577 papers (48.8% of the 3234, high because a paper has several tuples). Printed in full
by `node scripts/gt_fold.mjs --residue`.
 
It is a LONG TAIL: 2068 of the 2206 distinct strings (93.7%) appear exactly
once. Each names a resource or a procedure specific to one paper, which no
keyword family can be expected to reach. The most frequent unmapped strings,
READ OUT OF THE MAP rather than retyped — retyping them is precisely the defect
a reviewer found in the previous version of this block on 2026-09-11:
 
Unmapped string                                   Tuples
------------------------------------------------  ------
optical lens scanner labels                       6
touchpal call tags                                6
eth-psd and ctd benign/phishing labels            5
facebook known fake/real labels                   5
fingerprint definitions from previous literature  5
fingerprint pattern labels                        5
indegree median of the population                 5
market-purchased iot devices                      5
 
The page quotes four of these verbatim. If this table changes, the page is
stale and check_page_numbers.mjs will not notice, because these are words.
 
A 30-string systematic sample (every 73rd line of the printed residue) was
hand-classified in scripts/annotation_audit.mjs into the family each string
WOULD have joined. It spreads across every family rather than piling into one:
 
Family the sampled string would have joined  of 30
-------------------------------------------  -----
by-construction                              6
existing-dataset                             6
unclear                                      4
third-party-service                          4
rule-or-tool-output                          3
human-annotation                             3
execution-or-oracle                          2
curated-list                                 2
 
So EVERY family above is a floor, not just one of them, and none is
systematically missing. ("unclear" means the string alone does not say.)
An earlier draft of this script asserted that the residue was dominated by
by-construction setups and gave three examples. All three in fact FOLD
correctly — two are gt_fold.mjs self-test fixtures — and a reviewer caught it
on 2026-09-11. The sample above is what replaced the claim.
 
========================================================================
F. HOW THE HAND CODING WAS DONE (the sibling page owns the coefficients)
========================================================================
Population: the 3318 ANNOTATED papers. humanAnnotation[].codingApproach is an
enum; not-stated is a sentinel and is shown as its own row, never as an answer.
 
codingApproach         Papers  Share of 3318
---------------------  ------  -------------
binary-verification    1672    50.4%
ad-hoc                 1384    41.7%
predefined-codebook    935     28.2%
open-coding            281     8.5%
thematic-analysis      134     4.0%
grounded-theory        41      1.2%
not-stated (SENTINEL)  20      0.6%
 
========================================================================
G. THE LLM AS ANNOTATOR
========================================================================
177 papers carry a classification tuple with method == "llm"; 175 of them
USED or PRODUCED the labels rather than only comparing against an LLM baseline.
The 175 is this section's population. It is the same field and the same
definition design:website_classification uses for its per-target table, on
purpose, so the two pages cannot drift.
 
Year   LLM labellers (used/produced, 175)  Share  Any llm tuple (177)  Share  CLASSIFIED that year
-----  ----------------------------------  -----  -------------------  -----  --------------------
2022   0                                   0.0%   0                    0.0%   412
2023   2                                   0.4%   2                    0.4%   503
2024   27                                  5.1%   27                   5.1%   529
2025*  76                                  12.5%  77                   12.7%  607
2026*  70                                  21.0%  71                   21.3%  333
 
Question about the LLM labels           Papers  Share of 175
--------------------------------------  ------  ------------
the LLM tuple carries some validation   146     83.4%
…manual validation of a sample          110     62.9%
…a held-out test set                    16      9.1%
the LLM tuple says none-reported        35      20.0%
the paper also hand-codes something     151     86.3%
the paper reports an agreement metric   59      33.7%
NEITHER validation nor any hand coding  16      9.1%
 
BASE RATE for the agreement row, which is the sibling page's population, not
this one's: 512 of 3318 ANNOTATED papers (15.4%) name an agreement
metric. 24 of the 175 LLM papers hand-code nothing at all.
metric. LIKE FOR LIKE, because a paper that hand-codes nothing cannot report a
coefficient: 59 of the 151 LLM papers that DO hand-code (39.1%) name one,
against the same 15.4%. The 33.7%-of-175 figure understates the gap and the
page quotes the like-for-like one. The likeliest explanation is structural: when
a model does the first pass, the human coding IS the validation, so a paper
without a number has nothing to report. That is an interpretation, not a test.
 
Ground truth for the LLM labels: 138 of 175 papers (78.9%) state one on an
LLM tuple. Folded by the same rule as section E, counted by paper:
 
family                        Papers  Share of 138
----------------------------  ------  ------------
human-annotation              115     83.3%
RESIDUE                       22      15.9%
existing-dataset              4       2.9%
documentation-or-declaration  2       1.4%
execution-or-oracle           2       1.4%
model-output                  2       1.4%
third-party-service           1       0.7%
rule-or-tool-output           1       0.7%
by-construction               1       0.7%
 
========================================================================
H. WHAT THE 175 SAY ABOUT HOW THEY RAN THE MODEL
========================================================================
Full-text probes over the 175 LLM-labelling papers, whitespace collapsed and
soft hyphens rejoined. A probe is a candidate set; the temperature row is the
only one hand-audited, and it is the only one quoted on the page as a measured
share. The rest are LOWER BOUNDS and are labelled as such.
 
Probe                                                        Width        Papers  Share of 175
-----------------------------------------------------------  -----------  ------  ------------
the word "temperature" appears                               loose        55      31.4%
**states a numeric decoding temperature** (HAND-AUDITED)     audited      41      23.4%
names another decoding setting (top-p / top-k / max_tokens)  lower bound  31      17.7%
the word "prompt" appears                                    loose        172     98.3%
the full text says a prompt is reproduced somewhere          loose        31      17.7%
**a prompt really is reproduced** (HAND-AUDITED)             audited      26      14.9%
 
HOW MANY OF THE THREE a single paper supplies. The page's checklist asks for
six things from an LLM annotator; three of them are measurable here, and this
is the joint distribution rather than three marginals:
 
Items supplied (of 3)  Papers  Share of 175
---------------------  ------  ------------
0                      106     60.6%
1                      42      24.0%
2                      25      14.3%
3                      2       1.1%
 
MEDIAN = 0. The median paper that lets a model assign labels supplies none of
the three. Two papers of 175 supply all three.
 
Temperature hand audit (scripts/annotation_audit.mjs, 55 candidates read in
context — the word has four other senses in this corpus):
Verdict    Papers  Share of the 55 candidates
---------  ------  --------------------------
value      41      74.5%
other      11      20.0%
default    2       3.6%
discussed  1       1.8%
 
Prompt hand audit (same script, 31 candidates). "declined" is one paper that
matched because it says it is deliberately WITHHOLDING its prompts, which is
the opposite of what the probe is for. 26 is a lower bound in the other
direction too: a prompt printed in an appendix figure with no sentence
pointing at it is invisible to any probe of this shape.
Verdict    Papers  Share of the 31 candidates
---------  ------  --------------------------
published  26      83.9%
other      4       12.9%
declined   1       3.2%
 
========================================================================
J. THE HELD-OUT SET, AND WHETHER IT IS HELD OUT
========================================================================
A held-out test set drawn by shuffling rows is not held out when the rows come
in groups — pages of one site share a template, a tag manager and a consent
vendor, so a classifier that memorises the site scores against itself. The unit
of independence on a crawl is the site, not the page.
 
Question                                                           Papers  Share
-----------------------------------------------------------------  ------  -----
web-platform papers using a held-out test set or cross-validation  300     —
…whose full text contains ANY grouped-split language (probe)       10      3.3%
…**that really split or evaluate across a group** (HAND-AUDITED)   7       2.3%
 
This is a LOWER BOUND in both directions. A paper that split by site without
writing a sentence about it is invisible; so is one phrasing it outside the
regex. What the number will support is "this is rare and almost never stated",
not "only seven papers did it". The regex, and the ten candidates with the
deciding phrase for each, are in scripts/annotation_audit.mjs.
 
Verdict  Papers  Share of the 10 candidates
-------  ------  --------------------------
grouped  7       70.0%
other    3       30.0%
 
========================================================================
I. THE TITLE PROBE THAT PUT THIS PAGE ON THE ROADMAP, AUDITED
========================================================================
scripts/gap_probe_roadmap.mjs `llm_annotator` is a title+summary regex. The
roadmap row says 30 papers. Every one was read; the verdicts are in
scripts/annotation_audit.mjs with the deciding reason per paper.
 
Verdict     Papers  Share of 30
----------  ------  -----------
annotator   21      70.0%
object      7       23.3%
instrument  2       6.7%
 
PRECISION: 21 of 30 candidates (70.0%) actually point a model at their own
data. The rest study LLMs or attacks on them, or use one as a generator.
RECALL: the probe catches 23 of the 175 papers the enum finds (13.1%).
A title+summary probe is a candidate set. The enum is the population, and the
page uses the enum for every figure it publishes.
 
========================================================================
Z. NUMBERS ON THE PAGE THAT DO NOT COME FROM THIS SCRIPT
========================================================================
Each is sourced on the page itself and on provenance:statistics:annotation.
 
Figure                                                                                      Where it is from
------------------------------------------------------------------------------------------  --------------------------------------------------------------------------------------------------------------------------------------------------------------------
0.99 — "published F1 scores of up to 0.99"                                                  pendlebury2019_tesseract, abstract, verbatim in paper.cols.txt
20,000 requests heuristic-labelled and expert-validated as ground truth                     xiong2026_tgnn, appendix, verbatim
98.17% F1 for the LLM labelling component, on the whole ground truth                        xiong2026_tgnn §5.5, verbatim; the extraction records that tuple as held-out-test-set, which the paper does not support
200 privacy policies, 6 experts, 2 per policy                                               cory2026_wordlevel, methodology, verbatim
0.964 and 0.659, the demo's two precisions, and the 96.6% of a 3.4% class that is negative  pages/holdout_by_site.py demo output, quoted on the page
illustrative arithmetic: a 1,000-item sample at 3.4% positives holds ~34                    the page; 34/34, 30/34 and 25/34 precisions with 95% Wilson intervals 10, 22 and 29 points wide come from pages/holdout_by_site.py demo
5,859 / 4,439 / 3,318 / 1,120 / 1,762                                                       data/extract/OVERVIEW.md corpus populations
15.4% agreement-metric base rate                                                            computed here in G, and matches statistics:interrater_agreement
252 papers folding to Cohen's kappa, 18 self-agreement papers, 34.1% vs 13.8%               statistics:interrater_agreement and its report script — NOT re-derived here; its LLM-signal matcher is full-text where this page's is the classification.method enum
19.4% of 175 name a resolvable model                                                        design:website_classification / scripts/report_llm_currency.mjs
37.3% validation rate for topic classification                                              design:website_classification, population 330 not 4,439
Wilson interval arithmetic in pages/holdout_by_site.py                                      the script itself; it prints its own self-test
46.5% -> 75.9% artifact-link share, 2018 -> 2026                                            data/extract/OVERVIEW.md per-year table; a DIFFERENT field from artifacts.links, which gives 86.2% -> 93.0%
6 distinct gpt-4 snapshots                                                                  design:website_classification, counted from OpenAI deprecations 2026-09-03
58% run-to-run agreement of classification.method                                           data/extract/README.md stability audit (old run; order of magnitude)

scripts/gt_fold.mjs. Nine ordered families, 21 self-tests, residue returned rather than absorbed.

gt_fold.mjs

// Fold for `classification[].groundTruthSource` — the free-text answer to
// "where did the labels you checked against come from?".
//
// 6,223 distinct lower-cased strings across 7,114 tuples, so an exact-string
// ranking is meaningless. This fold does NOT try to name the resource (that is
// what ip_fold / req_fold / webcat_fold do for their own targets). It answers a
// coarser question that no other page asks: **what KIND of thing was the
// ground truth**, because that is what decides whether the validation is
// independent of the classifier.
//
// Ordered families — first match wins, so the order is load-bearing:
//   - a string naming BOTH a service and a manual check ("manual review of
//     VirusTotal labels") is human-annotation, because the human is the
//     authority in that sentence;
//   - "gold standard" alone is existing-dataset, not human, because the string
//     does not say who made it.
//
// Every family is tagged with whether the ground truth is INDEPENDENT of the
// thing being validated. `model-output` is the interesting non-independent one.
//
//   node scripts/gt_fold.mjs            # self-test + residue over the corpus
//   node scripts/gt_fold.mjs --residue  # print every unmapped string
 
export const FAMILIES = [
  // 1. Another model produced the labels. Checked FIRST: "manual review of LLM
  //    output" is caught by rule 2 below only if this rule does not claim the
  //    string, so this rule requires the model to be the LABELLER, not the
  //    thing being checked.
  {
    key: 'model-output',
    label: 'Another model’s output',
    independent: false,
    re: /\b(llm|gpt|chatgpt|claude|gemini|llama|language model)[- ]?(generated|produced|assigned|labell?ed|annotat|judg|as[- ]a[- ]judge|output|vote|consensus|ensemble)|majority vote (across|of) (three )?(llm|gpt|model)|\bllm[- ]?(as[- ]?a?[- ]?judge|judge)|model[- ]generated (labels|annotations)|pseudo[- ]?labels?|self[- ]?labell?ed/i,
  },
  // 2. A person looked at the items. The dominant family by far.
  {
    key: 'human-annotation',
    label: 'Human annotation of a sample',
    independent: true,
    re: /\bmanual|by hand|hand[- ]?(label|annotat|code|check|verif|curat|pick|select)|\bannotat|\bcoder|\bcoded by|codebook|human (label|annotat|review|verif|evaluat|rating|judg|coding|inspect|expert|analys)|expert(s)? (label|annotat|review|judg|assess|analys|classif|verif|opinion|consensus|evaluat)|(author|researcher|student|volunteer|analyst|psychiatrist|lawyer|legal expert|domain expert)s?[’']? ?(label|annotat|review|judg|assess|classif|verif|inspect|coding|consensus|evaluat|examination|analys)|inspected? by|reviewed? by|verified by (two|three|the )|(two|three|four|five|six|\d+) (authors?|researchers?|experts?|annotators?|coders?|students?|volunteers?|reviewers?|analysts?)|cross[- ]?check|inter[- ]?(rater|annotator|coder)|crowdworker|mechanical turk|\bmturk\b|prolific|user study|participants?[’']? (annotation|label|response|decision|identit|report)|survey responses?|self[- ]report/i,
  },
  // 3. The label is known because the experimenter created the condition:
  //    the browser was told which site to load, the member set was chosen, the
  //    anomaly was injected, the traffic was generated. Independent of the
  //    classifier and perfectly reliable — and a statement about a lab, not
  //    about the web. Checked after human-annotation, so "authors generated
  //    and labelled 400 routes" stays human.
  {
    key: 'by-construction',
    label: 'Known by construction (experimental design)',
    independent: true,
    re: /\bknown (member|non-?member|monitored|unmonitored|malicious|benign|phishing|positive|negative|training|test|ground|device|speaker|troll|synthes|receiver)|experiment(al|-specific)? ?(label|condition|phase|trial|design|setup|role)|assigned (from|during|by) |during (trace|data|packet|browsing) collection|in collected traces|collected traces?\b|\bsynthetic|synthes[ie]|\bgenerated\b|\bcrafted\b|\binjected\b|\bseeded\b|\bsimulat|\bhoney(pot|client|vault)|\bdecoy\b|controlled (browsing|experiment|visit|environment|setting)|laborator(y|ies)|\blab\b|self-(collected|generated)|(our|the authors)[’']? own|ground[- ]truth by construction|(website|webpage|web-page|site|page|device|user|participant|speaker|circuit|workload|accelerator|ecu|handset|browser-instance|prompt|query|keystroke|typed)[- ]?(identit(y|ies)|labels?|urls?|identifiers?)|(monitored|unmonitored|member|non-?member|attack|no-attack|honest|malicious|benign|genuine|authentic|replayed|real|fake|positive|negative)[- ]and[- ]|one-vs\.?-all|\bpartition\b|predefined /i,
  },
  // 4. Another automated method produced the labels: a heuristic, a rule set, a
  //    parser, an earlier detector, "the proposed algorithm". Like
  //    `model-output` this is NOT independent of the thing being validated
  //    whenever the two share a signal, which is the reason the fold separates
  //    it from a human or a list at all.
  {
    key: 'rule-or-tool-output',
    re: /\bheuristics?\b|rule[- ](derived|based|set)|internal rules?|\bthresholds?\b|\bparsers?\b|\bdetectors?\b|\bclassifiers?\b|(the )?(proposed|our) (algorithm|method|system|tool|pipeline)|inferred by|automatically (labell?ed|derived|synchronized|assigned)|regexe?s?\b|(detection|malware|attack|yara|av|virus|rule) signatures?\b|static analysis|\bavclass|\bnDPI\b|\bbro\b|\bzeek\b|\bsnort\b|\bids\b alerts|sandbox (traffic|report)|propagated from/i,
    label: 'Another automated method’s output',
    independent: false,
  },
  // 6. A named blocklist / filter list / curated public list.
  {
    key: 'curated-list',
    label: 'A curated list or blocklist',
    independent: true,
    re: /easylist|easyprivacy|\bdisconnect\b|filter[- ]?list|block[- ]?list|black[- ]?list|allow[- ]?list|white[- ]?list|\bufopenthemall|adblock|ublock|\bwhotracks|tracker radar|\btranco\b|\balexa\b|\bumbrella\b|majestic|\bcrux\b|public suffix|\bcurlie\b|\bdmoz\b|\bodp\b|known (malicious|phishing|benign) (domains?|urls?|list)|\bphishtank|openphish|\bspamhaus|\bsurbl|\bcert\b.{0,12}list/i,
  },
  // 7. A third-party service or vendor API was the authority.
  {
    key: 'third-party-service',
    label: 'A third-party service’s labels',
    independent: true,
    re: /virustotal|\bvt\b.{0,10}(label|score|verdict)|safe ?browsing|\bav\b.{0,10}(label|vendor|engine)|antivirus|\bmcafee|trellix|symantec|broadcom|forti(guard|net)|trend ?micro|forcepoint|websense|\bcyren\b|webshrinker|similarweb|cloudflare (radar|categor)|maxmind|\bip2location|\bwhois\b|\bripe\b|peeringdb|\bcaida\b|\bapnic|\bshodan|\bcensys\b|google (play|cloud|topics|adwords)|app ?store (categor|metadata)|cookiepedia|\bdns ?db|farsight|third[- ]party (service|api|vendor|classif)/i,
  },
  // 8. A previously published labelled dataset, benchmark or corpus.
  {
    key: 'existing-dataset',
    label: 'An existing labelled dataset or benchmark',
    independent: true,
    re: /\bdataset|\bbenchmark|\bcorpus\b|gold[- ]?standard|ground[- ]?truth (set|data|corpus|collection|labels? from)|public(ly available)? (labels?|data)|prior (work|study|studies|literature)|published (labels?|data|list)|\bcve\b|\bnvd\b|\bcwe\b|bug ?report|\bmalrec|androzoo|\bimagenet|\bcifar|\bmnist|labelled? (collection|set)\b|training (set|data|labels)|reference (set|labels|data|collection)|existing (labels?|annotations?|classif)/i,
  },
  // 10. Documentation, source code, specifications, declared metadata.
  {
    key: 'documentation-or-declaration',
    label: 'Documentation, source code or a declaration',
    independent: true,
    re: /documentation|\bdocs?\b|source code|\bapi (signature|reference|spec)|specifications?\b|\brfc\b|\bmanifest|\bschema\b|\bprivacy (polic(y|ies)|label|nutrition)|data safety|terms of service|developer[- ]?(declared|provided|stated)|vendor (documentation|statement)|self[- ]declar|metadata\b|\breadme|code comments?|\bchangelog/i,
  },
  // 9. Running, probing or reproducing the thing settled the label.
  {
    key: 'execution-or-oracle',
    label: 'Execution, probing or a reproducible oracle',
    independent: true,
    re: /\b(dynamic|runtime) (analysis|execution|test)|execut(ed|ion|ing)|\bcrash|\bproof[- ]of[- ]concept|\bpoc\b|reproduc(ed|ible|tion)|exploit(ed|ation)? (confirm|verif|success)|\bfuzz|\btestbed|test (harness|suite|case)|(active|live|real)[- ](probing|testing|measurement|devices?)|instrument(ed|ation) (browser|run)|\boracle\b|ping|traceroute|response (validity|validation|test)|(verif|confirm)[a-z]* (on|against|with) (real|live|actual|physical)|simulat/i,
  },
];
 
export const FAMILY_BY_KEY = new Map(FAMILIES.map((f) => [f.key, f]));
 
// Returns the family key, or null when nothing matched (the residue).
export function foldGroundTruth(s) {
  if (s === null || s === undefined) return null;
  const t = String(s);
  if (!t.trim()) return null;
  for (const f of FAMILIES) if (f.re.test(t)) return f.key;
  return null;
}
 
// Self-tests. Each is a string that appeared in the corpus (or a near variant)
// whose family a reasonable person should agree on, including the ordering
// cases that make the rule list load-bearing.
export const TESTS = [
  ['manual review of VirusTotal labels', 'human-annotation'],
  ['VirusTotal AV labels', 'third-party-service'],
  ['majority vote across three LLM judgments', 'model-output'],
  ['manual labeling of a random 10% sample (n=397)', 'human-annotation'],
  ['EasyList and EasyPrivacy labels', 'curated-list'],
  ['dataset class labels', 'existing-dataset'],
  ['gold standard collection', 'existing-dataset'],
  ['reproduced hacks, verifier results, and researcher cross-checking', 'human-annotation'],
  ['subsystem API documentation, code comments, and manual code reviewing', 'human-annotation'],
  ['PHP function signatures and PHPDoc documentation', 'documentation-or-declaration'],
  ['confirmation on real Matter devices', 'execution-or-oracle'],
  ['known member and non-member samples', 'by-construction'],
  ['website labels assigned during trace collection', 'by-construction'],
  ['synthetically generated fake users', 'by-construction'],
  ['high-precision heuristic labels', 'rule-or-tool-output'],
  ['R1-R3 rule-derived labels', 'rule-or-tool-output'],
  ['four psychiatrists’ annotations using C-SSRS-derived labels', 'human-annotation'],
  ['Google Safe Browsing blacklist', 'curated-list'],
  ['three cybersecurity experts’ field annotations', 'human-annotation'],
  ['participant identities', 'human-annotation'],
  ['indegree median of the population', null],
];
 
if (import.meta.url === `file://${process.argv[1]}`) {
  const { loadExtractions, pct } = await import('./lib.mjs');
  let fails = 0;
  for (const [s, want] of TESTS) {
    const got = foldGroundTruth(s);
    if (got !== want) {
      fails += 1;
      console.log(`FAIL  ${JSON.stringify(s)}  want ${want}  got ${got}`);
    }
  }
  console.log(`self-tests: ${TESTS.length - fails}/${TESTS.length} pass`);
 
  const rows = loadExtractions();
  const strings = [];
  for (const p of rows)
    for (const c of p.classification ?? []) if (c.groundTruthSource) strings.push(c.groundTruthSource);
  const counts = new Map();
  const residue = new Map();
  for (const s of strings) {
    const k = foldGroundTruth(s);
    if (k === null) residue.set(s.toLowerCase(), (residue.get(s.toLowerCase()) ?? 0) + 1);
    else counts.set(k, (counts.get(k) ?? 0) + 1);
  }
  console.log(`\ntuples with a stated groundTruthSource: ${strings.length}`);
  for (const f of FAMILIES)
    console.log(`  ${f.key.padEnd(30)} ${String(counts.get(f.key) ?? 0).padStart(5)}  ${pct(counts.get(f.key) ?? 0, strings.length)}`);
  const resTotal = [...residue.values()].reduce((a, b) => a + b, 0);
  console.log(`  ${'RESIDUE (unmapped)'.padEnd(30)} ${String(resTotal).padStart(5)}  ${pct(resTotal, strings.length)}  (${residue.size} distinct)`);
  if (process.argv.includes('--residue'))
    for (const [s, n] of [...residue.entries()].sort((a, b) => b[1] - a[1])) console.log(`${String(n).padStart(4)}  ${s}`);
}

node scripts/gt_fold.mjs –residue. Every one of the 2,206 distinct unmapped strings with its tuple count, so the part the fold could not classify stays visible.

gt_fold-residue.txt

self-tests: 21/21 pass
 
tuples with a stated groundTruthSource: 7114
  model-output                      17  0.2%
  human-annotation                2779  39.1%
  by-construction                  581  8.2%
  rule-or-tool-output              106  1.5%
  curated-list                     266  3.7%
  third-party-service              307  4.3%
  existing-dataset                 418  5.9%
  documentation-or-declaration     113  1.6%
  execution-or-oracle               93  1.3%
  RESIDUE (unmapped)              2434  34.2%  (2206 distinct)
   6  touchpal call tags
   6  optical lens scanner labels
   5  participants’ runtime allow, obfuscate, or deny decisions
   5  indegree median of the population
   5  facebook known fake/real labels
   5  state grid on-site investigations
   5  volunteer-performed and labeled activities
   5  fingerprint definitions from previous literature
   5  fingerprint pattern labels
   5  market-purchased iot devices
   5  eth-psd and ctd benign/phishing labels
   4  campaign records and weibo account baselines
   4  actual new edges in the next temporal snapshot
   4  participants' binary acceptability responses
   4  configured ecu source identities
   4  shadow-model training and testing membership labels
   4  authors' artificially modified and aes-256-encrypted files
   4  prompt membership construction
   4  policies observed through public apis
   4  ecg demographic labels
   4  authors’ exploit and baseline models
   4  adversary data for training; other participants for testing
   3  human-segmented query corpora
   3  observer-defined labels from participant traces
   3  tweets collected from twitter, crowdturfing sites, and black-market sites
   3  query traces labelled with corresponding keywords
   3  editor labels of tipextract
   3  vpn providers' claimed vantage-point locations
   3  instagram posts with location check-ins
   3  cryptoapi-bench ground truth
   3  cryptoapi-bench
   3  auxiliary data labeled with and without the property
   3  geotagged tweet locations
   3  participant comfort labels for 1,040 email-meeting pairs
   3  can identifiers mapped to ecus
   3  source celebrity identity and target identity labels
   3  persistent http plaintext cookies
   3  original adjacency matrices
   3  one-year production event data
   3  reported results from respective defense studies
   3  tcp/ip-to-socket-to-application correlation using logcat
   3  collected event and sensor traces
   3  observed task accuracy from worker responses
   3  prompted words and typed character pairs
   3  25 onion services
   3  experimentally constructed matched and unmatched flow pairs
   3  ordinal-inscription-labelled entities
   3  online voting ground truth and industry-engine comparison
   3  paper's scanner-organization labels
   3  esc-50 categories
   3  librispeech reference transcripts
   3  human participant statement choices
   3  iteratively labeled pip and non-pip tweets
   3  traffic labeled by triggered iot event
   3  positive training trajectories and negative non-training trajectories
   2  labeled geant anomalies
   2  airmaestro rf signal analyzer
   2  public repositories and malware analysis reports
   2  seven evaluator assessments per hit
   2  later offline category rating of previously unknown urls
   2  experiment-specific target-stimulus labels
   2  comscore person identifiers
   2  observed allow and deny settings in lbe logs
   2  actual interaction labels derived from keyboard and mouse timings
   2  three independent reviewers
   2  handshakedataset and campus traces
   2  known tidigits speaker and digit labels
   2  iwpc genotype labels
   2  100 monitored webpages
   2  exact local mixing times computed by brute force
   2  participant-entered dictionary words
   2  server-side recurring-device cookies
   2  instrumented tor logs identifying correct splits
   2  hdfs and openstack labeled anomalies
   2  plata database [41]
   2  alipay labels assigned after long-term observation and hindsight
   2  historical bgp data with known attacks
   2  30 regular and 30 irregular video streams
   2  known printer of each fabricated key
   2  dgarchive magds and cleaned network bnxds
   2  sd09 intra-person and inter-person fingerprint pairs
   2  timestamped access-log results
   2  researcher-created paypal and irs phishing pages
   2  telegram pump-event records and coin announcements
   2  experienced javascript developers
   2  wdvc-2016-links labels
   2  salticus hourly crawls
   2  researchers prepared and labelled 100 screen content types
   2  operator feedback for 75 measured ass
   2  throughput classes derived from iperf measurements
   2  associated and non-associated channel/user traffic flows
   2  measured amcl particle-number classes
   2  measured route labels
   2  shadow-model in/out splits from the hold-out group
   2  fact-checking organizations
   2  qualitative analysis of participant interviews
   2  website loaded during each recorded current trace
   2  genuine-user and impostor-user samples
   2  survey-derived subject and bystander labels
   2  attribute-inference game with assigned sensitive value
   2  youtube100 video titles
   2  kialo edge labels
   2  stockx product-category labels
   2  known letters displayed on the victim screen
   2  open-source attack data and deployed attack scenarios
   2  certificate transparency logs
   2  labeled feature vectors from known camera states
   2  migrated and randomly selected non-migrated users
   2  interview transcripts
   2  labeling of silkroad2 and agora data
   2  neo-pi-3 questionnaire responses
   2  geneva traces and normal traffic
   2  human-curated hallucination labels
   2  labeled ssh connections by username
   2  publicly released labeled tor traffic traces
   2  labeled vpn traffic from android apps
   2  literature review of 133 plc-security papers
   2  shadow-model training and testing partitions
   2  compiler output for source-available contracts
   2  known shared versus non-shared bottlenecks in experiments
   2  timit phoneme labels
   2  timit word transcripts
   2  existing taxonomy pairs
   2  expert-established ground truth
   2  law-enforcement-seized wallet data
   2  previously reported vulnerable packages and sink locations
   2  existing literature
   2  known identities in a 1,000-identity gallery
   2  training-set membership and held-out same-distribution samples
   2  ms marco ground-truth answers
   2  harvard sentences transcription text
   2  network-operator confirmations and nine-month roa tracking
   2  publicly known vulnerabilities and unique vulnerabilities discovered by both tools
   2  experimentally constructed inputs and target labels
   2  original system prompt
   2  video identities within anonymity sets
   2  scientific and grey literature review
   2  known red team cell replacements
   1  physical maps from local as tests in strasbourg
   1  six url-validation steps
   1  geoip city comparison
   1  non-malicious dns/isp distributions and modeled malicious distributions
   1  domain reputation services
   1  source labels assigned to live calls
   1  authors' downloaded fraudulent websites
   1  cluster membership labels
   1  four hierarchical-cluster labels
   1  hard-coded vendor-specific domain knowledge for template validation; trouble-ticket comparison for event validation
   1  tcpdump payload examination
   1  five domain network-profile classes
   1  500 seed queries from milw0rm.com
   1  project honey pot ip list
   1  editorial labels of web pages
   1  phoneyc signatures
   1  15 editors labeling 7,303 users' sessions
   1  controllable users’ identities and forum login names
   1  httponly and non-httponly cookies from the initial website study
   1  proper orientations of 800 study c images
   1  clamav labels
   1  permission map constructed from api testing
   1  google insights for search
   1  user-selected phone bounding box in first video frame
   1  keyboard geometry and bluetooth-transmitted typed characters
   1  known source movies and broadcast segments
   1  real panasonic-42-a emi traces
   1  domain knowledge and random sampling of trading thread titles
   1  known volunteer ip addresses
   1  ases hosting records for more than 100 spam domains
   1  http first-hop resolution data
   1  known grid certificate sub-data set
   1  known survey respondent associated with each survey
   1  two malware feeds and services including malwaredomains and zeus tracker
   1  known c&c and benign signature-labeled connections
   1  324 confirmed legitimate online pharmacies
   1  six-month sample of spam email
   1  malware hashes, exploit urls, and hostname-ip clusters
   1  real-valued normalized visit counts
   1  known search-engine crawlers and as ownership
   1  editorial judgments of yahoo! news article pairs
   1  wepawet analysis results
   1  authors’ categorization of chatroulette screenshots
   1  comparison with known cryptographic reference implementations
   1  osvdb and vendor advisories
   1  threat explorer descriptions
   1  instrumented victim executable writing operation labels to cross-vm shared memory
   1  exploit pack code, live-instance traffic, and community knowledge
   1  virus scanner reports
   1  multiple online malware analysis services
   1  hotmail account reputation scores
   1  same eye from different samples
   1  operator email verification
   1  none; authors explicitly assumed no ground truth
   1  fraud department positive identifications; post-paid account sampling; approved-device database
   1  indirect validation using inference and validation paths
   1  twitter account status information: suspended accounts malicious, active accounts benign
   1  known attribute values in collected profiles
   1  lac obtained through standard platform apis
   1  gps location and phone-number prefix collected from mobileapp users
   1  nxdomains collected by dga discovery and dga modeling
   1  mozilla and google development-team security classifications
   1  recorded actions placing phones in known states
   1  recorded participant voices
   1  20 training photographs per user
   1  whether the user returned for session k+1
   1  retaining versus non-retaining url labels derived from training-user sessions
   1  url participation in retained and non-retained user sessions
   1  authors assigned words occurring at least 10 times to relevant categories
   1  explicit likes and dislikes observed via the facebook linkr app and facebook interface
   1  interactive advertising bureau definition
   1  conversion in the target window
   1  normal click logs
   1  wikipedia administrator status and supreme court justice/lawyer roles
   1  invocation results from a test app on a real android phone
   1  law-enforcement visual inspection
   1  daily tor exit-node consensus files
   1  nasa and noaa ground-truth labels
   1  human-labeled image set
   1  noaa city snowfall observations
   1  nasa modis greenery labels
   1  935 comments labeled by two assessors
   1  human-labeled sentiment on 149 main posts
   1  human examination of extracted concepts from 149 main posts
   1  consensus labels from 10 random samples per newly formed cluster
   1  observed human clicks; randomly sampled never-observed clicks
   1  real human trajectories with known target articles
   1  vac-ban status in the second crawl
   1  known interference during wired and vehicle experiments
   1  transmitter and usrp-emulated jammer experiments
   1  traffic-camera images and known transmitter vehicle
   1  facebook post application-field semantics
   1  wepawet classifications
   1  595 malware samples with known contingency plans
   1  ios maturity ratings for 1,464 cross-platform apps
   1  smith's privacy regulatory culture framework
   1  user-adjusted word importance constraints
   1  mypersonality demographic profiles and microsoft live profiles
   1  mypersonality religion free-text fields
   1  mypersonality political-view free-text fields
   1  f1-score on device-classification experiments
   1  known tds toolkit url patterns
   1  passivedns records and parking-service identifiers
   1  known emulated-phone identities
   1  generator-created original and feature-modified packet-sequence classes
   1  packet-sequence ambiguity sets
   1  malware behavior and cyber probe detections
   1  hs1 confidential student lists; hs2 and hs3 limited facebook-derived ground truth
   1  facebook search and friend-list data, plus school ground truth where available
   1  mailserver spam-filtering system
   1  brightmail and isp security-vendor labels
   1  mozilla firefox bug database
   1  user activity histories and abandonment rule
   1  http content-type and content-encoding headers
   1  zeroaccess dns telemetry and supernode data
   1  known vulnerabilities and deliberately removed checks
   1  known participant-entered pins
   1  six cryptographic security rules defined by the authors
   1  wikipedia profiles
   1  wikipedia page views from may and june 2011
   1  positive labels were observed target-user accesses; negative labels were randomly sampled workflows where the user did not issue an access.
   1  corresponding complete lcc snapshots
   1  domains known to be associated with mobile malware or mobile malware operators
   1  known app actions, selected conditions, stocks, and driven routes
   1  204 webmd conditions
   1  presence of selected keywords in forum messages
   1  malware genome project
   1  red team-created malware and non-malware apps
   1  red team-created applications and documented malicious flows
   1  authors' investigation of cross-platform vulnerabilities and public exploits
   1  http headers observed when running whatweb
   1  wepawet detections
   1  android permission categories and prior classification of information-relevant permissions
   1  facebook permission categories
   1  chrome permission warnings
   1  wot ratings
   1  appbrain spam labels
   1  human readers labeling whether description sentences imply permissions
   1  three non-author participants
   1  expert-labeled binary responses and customer complaints
   1  expert-labeled binary responses
   1  openwall wordlist
   1  unlabeled random facebook users for training; anomalous and normal ground-truth users for evaluation
   1  ground-truth labels for black-market, compromised, and colluding users
   1  1,015 infiltration seed chains
   1  comparison with human-labeled relative orders
   1  analyzed finspy samples and previously reported server responses
   1  analyzed rcs and fsbspy samples
   1  professional judges' isolation and conditional relevance labels
   1  pairs of sessions labeled by same-person membership
   1  formalized posix standards
   1  gold set of 327 cookies
   1  authentication-token testing by submitting requests with cookie subsets
   1  <dc,p,v> malware-package matching
   1  comparison across simultaneous crawls on separate machines
   1  identified cookie values and http traffic
   1  automated canvas-fingerprinting detection
   1  decrypted provider exploit descriptors and reverse engineering
   1  publicly available ghostery tracker database
   1  developers verifying source-code-derived labels
   1  microsoft-confirmed bad feed and security essentials labels
   1  known webpage gpu memory dumps
   1  observed dns, tcp/ip, and http responses in isp traces
   1  comparison of asntrace and asnreal
   1  direct heartbleed measurements
   1  heartbleed-induced reissue classification
   1  verified vulnerable flows
   1  presence of dnskey records and secure delegations
   1  openemr-supported operations and author assumptions
   1  expert-labeled disclosures divided into training and testing sets
   1  tracker behaviors summarized by roesner et al. [34]
   1  known real accounts that accepted or rejected friend requests from known fakes
   1  cbl signatures
   1  800notes and ftc complaint data
   1  randomized treatment assignment
   1  proportions observed in 94 individuals
   1  exit survey data
   1  same incident and non-victim organization labels
   1  google code jam contestant identities
   1  later human-written seandroid policy and consultation with experienced policy analysts
   1  nmap service fingerprints
   1  vendor-selected marketplace labels from agora and evolution
   1  malicia signed samples with family labels
   1  zone-h-verified defacements and supposedly undefaced websites from popularity lists
   1  authors' list of ip addresses related to anonymization services
   1  known routes and true device locations
   1  true recorded routes and final road segments
   1  known prefix hijacking attacks
   1  65 remaining trustmanager implementations
   1  known chromium exploits, designed unit tests, and vendor confirmation
   1  initial privacy-related seeds and verb/possessive phrase patterns
   1  microsoft security advisories
   1  majority-vote labels from 1,500 removed apps and top-ranked apps as non-spam proxies
   1  one author's judgments, with link-following and google searches for unclear cases
   1  derived from documented primitives and implementations
   1  public social-media profiles and contextual evidence
   1  known tidigits speaker genders
   1  tidigits digit labels
   1  rfcs and bcps
   1  chinese pinyin list and oxford english dictionary
   1  dates in the range from 1900 to 2099
   1  coca tagged n-grams and specialized word lists
   1  wordnet synsets
   1  three crowd responses per ad or landing page
   1  fraudulent access-token attack procedure from wang et al.
   1  related measurement project [19]
   1  http traffic analyses, reverse engineering, and idle-device experiment
   1  triggered activity traces recorded as ground truth
   1  matching hashes in nsrl rds
   1  tcpdump timestamps
   1  webpage timestamps and mouse-event logs
   1  known websites loaded during experiments
   1  participant-typed key transitions
   1  repeated alphabetic-key presses
   1  participant-typed words
   1  top 65 injection libraries and rogue dom elements
   1  studies [20]–[26]
   1  affiliate-program cookie names and structures
   1  popshops merchant data
   1  appbrain advertisement-library listings
   1  captured dns, http, dom, and redirect-chain data
   1  trusted recursive resolvers, as/rdns matching, and https certificates
   1  anti-virus vendor classifications
   1  known darknet and switch unused /24 blocks
   1  conficker destination-pattern and scanning-strategy characteristics
   1  known dodo-telstra, bell-dery, and libyan disruption events
   1  dns response packet headers
   1  referrer domain compared with ad display domain
   1  operator labels from the search engine
   1  reverse-engineered zeus and sality behavior; cross-verification with industry contacts
   1  18 known active crawlers in logged gameover zeus traffic
   1  security appliance alerts and operator incident reports
   1  known doppelgänger bots and randomly selected twitter accounts
   1  previously labeled victim-impersonator and avatar-avatar pairs
   1  known ix ip addresses and prefixes
   1  facebook relationship statuses and zimbio celebrity dating histories
   1  tagged faces and reference photographs
   1  controlled latency baseline vectors
   1  webmail provider spam labels; vetted set l
   1  leaked campaign spreadsheets recording incentivized forum posts
   1  prerecorded reference library of machine-angle recordings
   1  100 confirmed bars and 300 legitimate buckets
   1  known providers with and without customer certificate management
   1  synchronized measurements on separate machines
   1  caller and callee phone logs, android database files, ott application logs, and cdrs
   1  foreground set of hidden-service pages and background set of public web pages
   1  domains appearing on uribl after registration
   1  domains captured by the authors' spam trap
   1  original library sdk binaries and profiles
   1  known equivalent and different acfgs
   1  software vendors' websites and internet
   1  comscore panel software-installation data
   1  seneviratne et al. curated library list
   1  ground truth from four network operators and ixp-published address information
   1  irr records and technical support websites for 30 ases
   1  middlebox classification interface and t-mobile data-usage counter
   1  reprobing selected /24 pairs
   1  bgp announcements collected by 40 monitors
   1  dns hostnames for level 3 and teliasonera interfaces
   1  as number and provider name in hostnames
   1  ntp packet fields in tcpdump traces
   1  highest-performing algorithm on each graph snapshot
   1  conversations with operations personnel and maintenance records
   1  known pii available on controlled test devices
   1  known pii values
   1  known fetched webpage
   1  50 messages posted at random ucsc campus locations
   1  nine yaks posted from the centers of nine ucsc colleges
   1  publicly linked cross-domain accounts and acknowledged identities
   1  known corresponding authors across domains
   1  known covertcast streams versus actual youtube and webcam streams
   1  correct category associated with each beacon
   1  participants' interview responses
   1  dividendrippler-published gateway wallets and gateway confirmations
   1  gateway-published cold wallets and confirmations from bitstamp and ripplefox
   1  three gateways: bitstamp, ripplefox, and snapswap
   1  instrumented tor cell logs and page-loading procedure
   1  instrumented tor logs identifying page boundaries
   1  noise and real cell labels from constructed traces
   1  real-world website and mobile-application examples and privacy-incident reports
   1  pre-built deep learning network
   1  reverse-engineered dga outputs
   1  llvm source-level information linked through dwarf/pdb
   1  guess numbers produced by password-guessing models and minguess
   1  message transmission counts over several minutes
   1  timing analysis, wordpress version, and per-domain exploit urls
   1  previously published cookie-tracking taxonomy, augmented by the authors
   1  domain research, ownership, archived behavior, context, and modern behavior
   1  registry modifications, contacted domains, process names, certificates, and installation preconditions
   1  ppi forums, publisher websites, and internet archive
   1  case/control disease labels
   1  held-out attribute links of test users
   1  training users' known attributes
   1  wikipedia editor hoax flags, deletion, and acquittal outcomes
   1  2,692 hoaxes passing patrol versus randomly sampled correctly flagged hoaxes
   1  960 wrongly flagged articles versus matched deleted hoaxes
   1  author-created lists of private information for authenticated accounts
   1  experimentally constructed matching and mismatching checksum challenges
   1  opencorporates company directory
   1  herdprotect publisher records
   1  direct attack testing of client implementations
   1  100,000 unrelated messages
   1  100,000 test emails
   1  normal log entries for training; labeled test logs
   1  normal log-key sequences
   1  sym reports of exploits detected at scale
   1  html login-form inspection and visual similarity
   1  avast classification
   1  asip vulnerability descriptions
   1  developer descriptions, websites, and hosted source-code links
   1  semantic-versioning rules applied to version strings
   1  compilation provenance: same source function versus different functions
   1  http return values and request errors
   1  inspection of companies' public policies
   1  user-specific grid profiles
   1  instagram api followee lists; users counted as related when they mutually follow
   1  googlebot certificate-error observations within the prior thirty days
   1  team cymru registry
   1  timing of inv messages and public stratum-server dns names
   1  previous-month origin as
   1  three mobile operators' labeled cellular and non-cellular ip subnet lists
   1  public bgp data
   1  bing shutdown decisions under internal policies
   1  random sample of 464 doxes
   1  encrypted prices from probing ad-campaign a1
   1  published operator information, irr records, and private communication
   1  english dictionary
   1  openintel a records
   1  slow-start throughput relative to access-link capacity
   1  nsrl rds 2.52 hashes and publishers
   1  robovic-detected phone numbers
   1  collector network status consensuses
   1  sanitized fingerprint lookup in collector
   1  cymon threat reports
   1  abuseipdb threat reports
   1  dshield reported malicious activities and attacks
   1  captured browser and interception-product handshakes
   1  fanboy's social blocking list
   1  wps attributes, vendor ie signatures, mdns fingerprints, and controlled-device testing
   1  1,417 ground-truth mac addresses
   1  29 controlled ios devices and apple ouis
   1  paired fingerprinted and non-fingerprinted versions
   1  induced privacy and sharing attitude conditions
   1  privacy and sharing attitude conditions
   1  appbrain sdk inclusion statistics
   1  participant interest and demographic questionnaires
   1  parsed sets of open ports
   1  lbnl security staff investigation and incident database
   1  retire.js vulnerability knowledge base
   1  iana list of 256 common uri schemes
   1  authors' known target and malicious print experiments
   1  authors' target and malicious print labels
   1  authors' raman measurements of three filament classes
   1  strings from the original mirai source-code release
   1  67 c2 domains and ips extracted from binaries
   1  heuristically identified sockpuppets and matched ordinary users
   1  heuristically identified sockpuppet pairs and matched ordinary accounts
   1  runtime xposed trace of 30 installed apps
   1  labels provided by linkedin's anti-abuse team
   1  participant confirmation of the displayed twitter profile
   1  crowdsourcing tasks defining deceptive question and answer groups
   1  rfcs, w3c, iana, and blogs
   1  editor labeling of tripadvisor tip candidates
   1  useful tripadvisor city-guide tips
   1  hashtag-based political self-expression
   1  cdc and american cancer society symptom and drug lists
   1  queries beginning with self-diagnosis phrases
   1  known labels and population statistics
   1  underlying exploit-kit identity
   1  participants' post-experiment familiar/unfamiliar responses
   1  listing suspension state in the google maps snapshot
   1  device settings
   1  labeled posts for buy, sell, currency-exchange, and other categories
   1  authors constructed associated and non-associated flow pairs
   1  942 accounts attributed by 23 recruited fraud workers
   1  human fraud-worker confirmations
   1  942 attributed fraud accounts
   1  open-source smartapp dfas
   1  original versus transformed image space
   1  curated sensitive-validation seed groups
   1  typed text
   1  volunteers' typed content
   1  http-log-derived exposed and unexposed session labels
   1  browser-emitted cache-access logs
   1  prevalent script source domains in crawled javascript
   1  inspection of eight third-party push libraries
   1  known transmitting ecu for each can frame
   1  entering and exiting times recorded separately by four users
   1  non-noisy cluster labels
   1  red-team campaign credit and external knowledge
   1  actual subsequent security event in held-out sequences
   1  actual hospital age records
   1  legitimate-call rtt measurements; redirected calls for evaluation
   1  osv definitions v1–v4
   1  known original cards and researcher-created clones
   1  seed binaries selected by yara rules
   1  giotsas et al.'s dictionary of blackholing communities
   1  experimental communication data and androlizer information
   1  mandatory root-cause field in sev reports
   1  lumen data from more than 60,000 android apps
   1  hphosts ats file
   1  icmp responsiveness survey data
   1  device-activity logs
   1  known-unmodified university-ip captures
   1  expected http headers and server-observed requests
   1  routerinfo ip-address and introducer fields
   1  capacity fields in collected routerinfos
   1  list of verified and inferred blackholing communities from previous work
   1  dns names and reverse dns names
   1  bgp routes and isp peering-link information
   1  service activity duration and observed service behavior
   1  12,447 interior prefixes of major us isp networks
   1  operator- and website-based validation data for 15 ixps
   1  know your meme image galleries, entries, and tags
   1  kym tags
   1  network trouble tickets
   1  network-operator validation and router configurations
   1  publicly documented amazon aws and microsoft azure server locations
   1  encoded serial numbers, ttls, query state, and authoritative expectations
   1  duckduckgo search for “public dns” on 2018-01-15
   1  abuseipdb
   1  access.watch
   1  all.s5h.net
   1  dnsbl.beetjevreemd.nl
   1  alphabay crawl data
   1  domain name in the final url
   1  power-trace fragments tagged by key activity
   1  labeled power traces of english text
   1  labeled power trace of 24 incoming calls
   1  disclosed training identities and corresponding held-out test sequences
   1  known identities of split user traces
   1  google code jam programs with known single authors
   1  known authorship of original and forged programs
   1  hashed username and device id pairs
   1  joinmarket transaction structure and prior algorithms
   1  coppa requirements and ftc guidance
   1  issue and patch descriptions in module bug trackers and version-control systems
   1  national vulnerability database entries
   1  exchange interactions, publicized founder addresses, and scraped mining-pool tags
   1  reported cves for each library
   1  etherscan verified solidity source files compiled with solidity compiler versions
   1  56 exploited-in-the-wild and 300 not-exploited-in-the-wild vulnerabilities
   1  known exponent bits and control-flow sequences from training decryptions
   1  training snippets grouped by known bval and bit position
   1  774 known dark jargons and corresponding hypernyms
   1  benign training files and embedded-message test files
   1  evaluation files with append and invisible-secrets embeddings
   1  known versions of darkcomet and njrat
   1  rat-hole behavior
   1  smartthings online-store category and source-code definition block
   1  2092 preferences classified as sensitive by panelists
   1  aggregated tracker and advertiser host list
   1  recovery/relapse labels
   1  palo alto networks malware verdicts
   1  twitter accounts suspended six months after data collection
   1  twitter-suspended accounts
   1  anchor nodes labelled with location information
   1  game-bot bans and rmt website records
   1  three assessors per document
   1  training-set location distribution
   1  human-labeled formula dag structures
   1  12 adchoices templates
   1  6,320 ads collected by hussain et al.
   1  wechat security-team labels from behavior-based detection and user reports
   1  existing behavior-based sybil detection system and user reports
   1  malicious powershell script databases
   1  union of true vulnerabilities detected by ilf, unif, maian, and contractfuzzer
   1  509 malware samples and benign web applications
   1  d1 members and d4 non-members
   1  participant-selected desired privacy settings
   1  participant friend-post sharing preferences
   1  physical ground truth from contact, force-sensitive, scheduled, and device-control sensors
   1  debian security team advisories
   1  video identity labels for 50 selected youtube videos
   1  labeled member versus non-member examples from training and test points
   1  pci dss and asv guidelines; 17 implemented testing rules
   1  route views bgp updates
   1  public exploits for 27 vulnerabilities
   1  gsm arena and gadgets360 device prices
   1  ghostery
   1  pi-hole
   1  google analysis and user reports
   1  whether three targeted attacks guessed the pair within 1,000 guesses
   1  passwords chosen by the same user as context
   1  dns, tcp syn, and other packet traces
   1  membership labels for evaluation; labels unknown during attack training
   1  independent coding of 75 chrome help-forum threads
   1  known drone and moving-iot-device experiments
   1  records labeled for both the main task and the inferred property
   1  benign audit runs for noise-reduction models
   1  user-supplied repeat-submission responses and submission ip addresses
   1  raw-data fingerprints from 795 devices
   1  porn and non-porn picture sets
   1  50 obfuscated qr-code images and 50 images without codes
   1  shared promotional or explicit content
   1  known displayed websites
   1  11 known foreground windows
   1  experimental attack validation
   1  reverse engineered intel slice function
   1  expected responses defined from 3gpp standards
   1  3gpp standards review and carrier interviews
   1  targeted mbaas sdk api definitions
   1  cloud-server response messages and key formats
   1  responses to zero-data-leakage probing requests
   1  shared identifiers, ancestors, hosting, iocs, domain aliases, and proxies
   1  known mining-tool binaries
   1  self-attacks and observed ntp packet-size distributions
   1  self-attacks
   1  w3c html 5.2 standard
   1  rir-assigned entity or asn information
   1  ixp membership, hybrid connectivity, and public reachability
   1  as ownership of interfaces in alias sets
   1  sampled encrypted and unencrypted traffic
   1  midar and mercator alias clusters
   1  itdk router alias clusters
   1  photodna database of known child abuse material
   1  tineye image database
   1  five years of nanog operator mailing-list threads
   1  230 labeled ases: 23 serial hijackers and 217 legitimate ases
   1  at least 4 of 70 malware scanners
   1  polisis web-version output
   1  endpoint process names and executable sha-256 hashes
   1  sslbl watch list
   1  goodput values of bbr and cubic from 640 lan experiments
   1  wadi and swat attack periods
   1  gnutls 3.5.12 anchor implementation
   1  commercial broadband market database
   1  daily opendns top-one-million lists
   1  curated threat list collected april 3 and july 30, 2018
   1  1704 known codesys f/fbs
   1  i/o memory maps extracted from codesys trg files
   1  survey of prior iot safety and security literature and app use cases
   1  reverse-engineered hajime attack templates
   1  publisher information matched against known pup publishers
   1  training-set membership from shadow-model data splits
   1  online database of malicious ip addresses
   1  foursquare venue categories
   1  five undergraduate volunteers corrected ocr output
   1  experimental voice condition labels
   1  browser screenshots, web-server logs, and ajax messages
   1  class patterns, logcat tags, and correlated app parameters
   1  known user inputs during attack experiments
   1  concrete exploitation of vulnerabilities in 100 apps
   1  crowdsourced ratings from three turkers per tweet
   1  git blame assignments
   1  authors' reverse engineering
   1  observed device models and os versions
   1  known non-targetable dummy records and authors' active-account pii
   1  united states board on geographic names gazetteer
   1  survey respondents' allow/deny choices
   1  survey respondents' selected comfort-changing factors
   1  survey respondents' dollar amounts
   1  android runtime permissions and decoded network transmissions
   1  reverse-engineered circumvention implementations
   1  modified droidbench applications
   1  known advertising-company url patterns
   1  physical ecu access and disconnection
   1  isolated real ecus
   1  client discussion and consultant inspection of flagged applications
   1  devices, firmware, online reports, and vendor responses
   1  contemporaneous bgp routability data
   1  nsrl and shadowserver known-software repositories
   1  publicly available npm advisories
   1  same-source versus different-source method pairs
   1  walletexplorer address tags
   1  etherscan address tags
   1  hand-collected zcash exchange tags
   1  shapeshift api responses
   1  list of 124 mobile advertising and analytics libraries provided in [41]
   1  android dangerous-permission list
   1  fixmystreet forum categories
   1  median value of each success measure for each k
   1  party-labeled speeches and letters from 20 democrats and 19 republicans
   1  style keywords in product descriptions
   1  seed websites
   1  wikipedia editor citation need labels
   1  cookie ids extracted from set-cookie headers
   1  known-location assessment task
   1  automatic labels obtained by analyzing wikidata rollback-tool use
   1  fixed expressions in biographies, birthday tweets, and curated organization lists
   1  rosenberg self-esteem scale scores
   1  three workers per comment with majority voting
   1  published authentic domains, seizure notices, address verifier, and surface-site ownership
   1  whether a user eventually logged a weight at or below the goal
   1  whether users were certified at the end of the course
   1  hashtags, crisis-lexicon, and author-added disaster phrases
   1  crisis-lexicon matches at unigram and bigram level
   1  pattern matching in reddit posts and personachat descriptions
   1  imdb character and actor or actress attributes
   1  twitter rest api deletion-status queries
   1  media bias/fact check ratings for 1,540 media-source domains
   1  brazilian fact-checking agency review and prior fact-checking labels
   1  fact-checking pages returned by google images
   1  majority voting by two android experts and cmu privacygrade
   1  two android experts and cmu privacygrade
   1  ground-truth and predicted heart-rate sequences
   1  url origin mapped to cryptojacking family names
   1  clicked-link provenance cross-referenced with advertising-domain lists
   1  crowdsourced pairwise judgments of path naturalness
   1  ad network policies and public opinion
   1  ad-related domain lists
   1  exactly-one-link article-paper pairs as positive; random article-paper pairs as negative
   1  site reputability tiers used as article labels
   1  yelp not recommended reviews, duplicate reviews, crowd labels, and trusted benign reviews
   1  four-turker majority vote over five-category judgments
   1  bike repair records
   1  participant button-box responses: “need to search” versus other responses
   1  sample examination by the authors
   1  observed wikipedia post-block trajectories
   1  official openstreetmap tag page
   1  app-brain, privacygrade, and common library classifications
   1  top 100 popular app names, excluding 10 common words
   1  number of apps released by each developer
   1  developer app counts and average installs
   1  five published fake-news publisher lists and further verification from snopes.com
   1  law-enforcement confirmation of sha1 hashes and visual inspection
   1  gcc's compilation option -fdump-class-hierarchy; rtti for mysqld, mysqlbinlog and mysqlpump
   1  239 previously identified eeh bugs
   1  empirical observations of identified eeh bugs
   1  sdk appearance parameters and mobile-app descriptions
   1  exploit-path verification in a virtual machine; vendor confirmations and assigned cves
   1  recorded real-sign and projected-phantom videos
   1  555 million unique passwords previously leaked in data breaches
   1  630,034 english dictionary words
   1  comparison with captcha ground-truth labels
   1  participant-entered pins, passwords, patterns, and picture-password inputs
   1  existing data sources that accurately pinpoint rov ass
   1  opc ua security recommendations
   1  nrc word-emotion association lexicon
   1  xcode application with all capabilities enabled
   1  completion-handler responses, entitlement error messages, and method-name review
   1  dynamic hints and non-hint ui information
   1  vulnerability reports assigned cvss scores
   1  known government domain formats and country-code extensions
   1  certificate chains retrieved from target hosts
   1  public ip ranges published by cloud and cdn providers
   1  ev policy oids encoded in certificates
   1  iperf 3.7 throughput measurements
   1  measured throughput
   1  operator confirmations, dns services, and comparison with igreedy
   1  igreedy census data from april 2017
   1  rrdp http user-agent strings and observed synchronization intervals
   1  1,194 hate, 16,025 offensive, and 20,499 neither twitter labels from davidson et al.
   1  allsides media bias ratings
   1  iana port assignments and known vpn protocols
   1  böttger et al. [6]
   1  labeled data from late 2019
   1  training and validation sets derived from bgp observations
   1  information found online
   1  apn network-identifier patterns and gsma tac data
   1  apn information
   1  public religious and medical wordlists plus an author-created political wordlist
   1  distinctive cdn error responses and successful origin access
   1  functional-clustering labels
   1  known browser user-agent strings and platform/provider substrings
   1  known proxy endpoints and common non-proxy endpoints
   1  known webpage corresponding to each collected dns trace
   1  http status and requested fully qualified domain name
   1  labeled browser and non-browser traffic
   1  clinvar archive
   1  labeled set
   1  recorded speech labels
   1  200 password samples and 2200 negative samples
   1  paired acceleration and speech spectrograms
   1  iclab and greatfire censorship measurement data
   1  vulnerability-involved functions and memory-operation analysis
   1  record-pair identity labels: same person versus different persons
   1  timestamped triggered events
   1  recorded event timestamps
   1  500 associated and 2,000 non-associated training pairs; 200 associated and 4,000 non-associated test pairs
   1  linux accept/drop behavior and observable dpi alarms
   1  inspection of linux tcp implementation and kfree_skb/tcp_drop paths
   1  known edges added by the anonymization mechanisms
   1  benign training graphs and labeled attack graphs
   1  majority vote of seven anti-malware tools
   1  authors' benign/malicious labels
   1  nine packer classes
   1  active poisoning experiments
   1  benign enterprise instances and malware-triggered malicious instances
   1  tencent habo reports
   1  brave ad block engine filters
   1  experiments a and b
   1  continuous bids discretized using mean and standard deviation
   1  comparison of cookie instances across two crawls
   1  nomoats stack-trace labels
   1  library-purpose labeling by authors
   1  surveyed allow/deny preferences
   1  surveyed privacy-preference feature vectors
   1  android-defined dangerous-permission classification
   1  1,000 valid policies and 1,000 invalid web pages
   1  ico gdpr checklist requirements
   1  pi-hole default, firebog, moaab, and stopad lists
   1  device settings and packaging; md5 and sha1 hashes of pii
   1  closed captions around detected activations
   1  controlled idle and wake-word activation traffic
   1  crowdflower workers
   1  boxer bin identification database and extracted card features
   1  400 screen images and 400 non-screen images
   1  autoencoder trained only on real samples
   1  comparison of chipsec reports from original and pruned images
   1  sampled production data and referenced-object existence checks
   1  known real ardupilot control-semantic bugs
   1  symbol information in target firmware samples
   1  subject identity labels from 90 participants
   1  legitimate-user versus other-user data
   1  legitimate-user templates from registration
   1  reported cves and corresponding patched chakracore versions
   1  mcu datasheets
   1  controlled dns responses and cache-probing timing measurements
   1  30 controlled chinese vantage points probing upstream resolver caches
   1  two independent domain-knowledge labelers, with expert tie-breaking
   1  stopbadware clearinghouse search
   1  ground-truth target locations from trajectories
   1  train-group membership and test-group non-membership
   1  apwg and organization's phishing url feeds
   1  750 human-labeled threads
   1  370 ips-relevant threads
   1  libscout package-name matching and code-similarity profiles
   1  python-based ground-truth implementation
   1  20 previously reported memory bugs
   1  authors' collected live-human and replay attack samples; asvspoof labels
   1  asvspoof evaluation set labels
   1  north american numbering plan administrator information
   1  human-interpretable packet fields
   1  user check-in histories
   1  hourly crawl observations of future price information
   1  click information aggregated on the following day
   1  impression information aggregated on the following day
   1  correct answers in collected attention-check questions
   1  presence of the fingerprint2 library
   1  clinical report anchor-word labels
   1  original reports written by doctors
   1  answer-connected edges as positives; ten randomly sampled negative edges
   1  two co-authors independently coded 116 questions and 32 statements in study 1
   1  aggregated actively maintained ip banlists
   1  kodi team repository and xbmc foundation banned-addon designation
   1  exact addon-id search results containing xbmc or kodi
   1  expert-labelled factoid and non-factoid inputs
   1  matching vendor names across markets
   1  users who default within one month versus users who do not
   1  basketball-reference.com, pro-football-reference.com, and the @unitedstates project
   1  prior classification of alphabay listings into ten b2b cybercrime product categories
   1  tracking-service response feedback
   1  emoji tags provided with each sticker
   1  100 chinese and english gambling-related keywords
   1  chargeback reports from card issuer banks and label propagation
   1  chargeback reports from card issuer banks; device and card information used to mark similar transactions
   1  public ieee oui database
   1  wikipedia pages of popular entities via distant supervision
   1  human grading on a three-point relevance scale
   1  matched and unmatched ehr–eligibility-criterion pairs
   1  query-document pairs as positives and anchor-document pairs as negatives
   1  neural-ranker ndcg reward
   1  labeled training examples and guessed labels from unlabeled reviews
   1  existing taxonomy edges and procedurally sampled negatives
   1  pod-affiliated posts and follower-count-matched control posts
   1  pattern examples and counter-examples labeled by crowd workers
   1  independent verification by two co-authors of 510 test samples
   1  observed client codenames, propagated replacement transactions, and block messages
   1  web3_clientversion responses and peer-discovery messages
   1  packet protocol/application identification
   1  credential-stuffing susceptibility derived from password pairs
   1  76 publicly collected descriptions: 36 industry, 30 media, and 10 academic
   1  formally verified pkcs#1-v1.5 standard implementation
   1  site:<url> search results
   1  snyk vulnerability labels
   1  ossfuzz bug-introducing commit and regression marker
   1  x.509, ca, and browser implementation requirements
   1  industrial partner's ct monitor
   1  known source and non-source transactions in controlled testnet experiment
   1  real test set labels
   1  tum-crypto
   1  authors' experimental activity labels
   1  2018 fda draft guidance recommendations
   1  locally stored lj speech audio files
   1  rules collected from prior papers and nist/ietf documents
   1  dma-related debug symbols in elf files
   1  mcu data-sheets and instrumented emulators
   1  successful download of the public play webpage by package name
   1  known register values set to 0 and 1
   1  known aes-dom architecture and recovered snapshots
   1  ieee oui database
   1  company websites, government websites, annual reports, regulators, itu, commsupdate, world bank, and imf
   1  orbis corporate ownership structures and sovereign ownership filter
   1  fingerprints labelled with generating applications
   1  allsides and media bias/fact check categorizations
   1  participant recruitment group labels: aso workers and regular users
   1  worker-promotion and cross-device installation criteria; regular-app installation and review-count criteria
   1  coinbase transaction wallet addresses
   1  transaction input relationships within blocks
   1  default authoritative ttls
   1  rpc responses, exceptions, and performance effects
   1  owner-provided labels in the system model; volunteered posts treated as non-damaging
   1  collected audible and inaudible voice-command samples
   1  given labels for supervised fare
   1  herrmann et al. website-access traces
   1  actual sensor readings
   1  apple security updates
   1  captured traffic and known library domains
   1  ccpa, coppa, and gdpr requirements
   1  known eye-chart letters displayed at 11 scales
   1  eight researcher-created shadow-attack exploits
   1  user-defined recipients and app descriptions
   1  existing attack reports, advisories, and blogs
   1  experimental and wild-capture signatures
   1  cloudflare and akamai hosting identities
   1  jsdynamic-groundtruth
   1  quic and tcp trace labels
   1  retire.js library classification
   1  webxray same-entity list
   1  ipinfo cellular-ip labels
   1  500 non-proxy, 50 proxy, and 500 relayed connections
   1  time-stamped ground truth of keys typed by participants
   1  participant-typed words and text
   1  threat models based on research literature, other resources, and authors’ expertise
   1  wong and henderson file-format assessments
   1  orbis nace rev. 2 codes
   1  observed export timing, format, and completeness
   1  source snapshots paired with corresponding binaries
   1  recursive replacement and lookup in the .com idn list
   1  presence in the verisign .com idn list
   1  unicode confusable table
   1  inductive coding of interview data
   1  four likert-scale privacy-concern questions
   1  pairwise file-path overlap
   1  automated labels and human-reviewed account labels
   1  high precision human-labeled data
   1  google exact-match searches and gpt-2 training-data matching
   1  604 samples confirmed as memorized
   1  shared attack transactions and identical bot bytecode
   1  participant likert-scale and management-decision responses
   1  open-coded interview transcripts
   1  mac-address oui records and publicly available manufacturer information
   1  pre-defined library signature checks
   1  normal-operation mmio profiles
   1  front and end check codes
   1  40 sampled model files from four encodings
   1  experimental handshakes with and without a leading zero byte
   1  hibp database of breached passwords
   1  chainalysis service labels for known bitcoin service wallets
   1  grams darknet-market database
   1  exodus-privacy's detected tracking or advertising libraries
   1  erc20 standard
   1  controlled spoofing and non-spoofing experiments
   1  controlled adaptive-spoofing and non-spoofing experiments
   1  code reuse relations between vulnerable software
   1  mozilla bugzilla disclosure reports and audit details
   1  ca audit documents and operational features
   1  samples consistently labeled benign by all av products
   1  characteristic function names and frequent data-section strings
   1  5% masked entries of each language's hyperlink adjacency matrix
   1  article language labels
   1  400 randomly sampled labeled videos
   1  runtime ui screenshots and app exploration
   1  fashionstyle14 professional style labels
   1  fancy nmf-derived style labels
   1  gdpr article 13 clauses and sentence-classification results
   1  prior research and icann-accredited registrars list
   1  aosp allow and neverallow rules
   1  known tpr platforms and public ip ranges
   1  tweets mentioning cybersecurity events
   1  top popular-domain lists and major cdn services
   1  control node dns responses and subsequent certificate/http validation
   1  control-node webpage and network responses
   1  fingerprintjs function list
   1  http error codes and displayed text
   1  aiwen ip threat labels
   1  malicious traffic-flow reports
   1  confirmation through related disclosure processes
   1  android code sources, sinks, and output handlers
   1  apk signing schemes and supported android versions
   1  108,530 labeled addresses from walletexplorer, bitcoin-otc, satoshidice, and ransomware reports
   1  transaction-sequence patterns specified for cerber, pony, skidmap, and glupteba
   1  darpa red-team ground truth
   1  preliminary crawl of 21,000 websites
   1  decoded cmp id in consent strings
   1  global vendor list version 168
   1  survey ratings of high-level concepts
   1  keywords extracted from low-rated comments and li et al.'s android-app keywords
   1  at least three participants per person, using mean q1 score
   1  four recipient devices and seven messaging apps
   1  runtime logs, type-based analysis, and error-condition checks
   1  928 registry cookies plus 10,000 randomly sampled cookies
   1  automatic opt-out-tool registries
   1  criteria evaluated by englehardt et al.
   1  crawler server ips, locations, browser/os versions, and visited urls
   1  malware and benign traffic labels
   1  comparison of original and modified runtime headers
   1  known attack targets in evaluated attacks
   1  500 cloaked phishing websites and 500 benign websites
   1  google geolocation api responses
   1  youtube copyright-detection results
   1  majority vote of seven selected anti-virus products
   1  open-source list of offensive words and phrases
   1  lastline security-operator labels
   1  qualitative coding of interview transcripts
   1  signals collected at known antenna and screen coordinates
   1  comparison with successful and failed iemi attacks
   1  clean and backdoored shadow models
   1  known poisoned and clean models
   1  clean and triggered samples
   1  clean and poisoned models
   1  appliance verdicts corrected and validated by the company's it department
   1  ground-truth labels and calibration p-values
   1  known reddit troll accounts
   1  40 academic and industry artifacts
   1  isp information retrieved by ipinfo
   1  flashbots bundle labels and mined-block data
   1  lending-platform flash-loan events
   1  existing meta tcp instrumentation
   1  publicly listed ip addresses for cisco, siemens, and microsoft
   1  at least five malware detection engines
   1  mirai, gafgyt, and daddyl33t protocol profiles
   1  iqbal et al. fingerprinter list
   1  ihr rpki and irr status fields
   1  open-source application repositories
   1  dns queries for www.google.com, alternative domains, and a controlled domain
   1  random responsiveness probes within prefix subranges
   1  repeated responsiveness scans
   1  tcp handshake features
   1  icmpv6 fragmentation and pmtu behavior
   1  sbl records
   1  sbl records without sufficient keyword matches
   1  mba subscription information
   1  us ofac sdn and uk sanctions lists
   1  roa records from the rpki repository
   1  reported manufacturer labels from real android devices
   1  experimentally scripted regular and anomalous office activities
   1  experimentally scripted attack scenarios
   1  controlled device-tampering scenarios
   1  experimental traffic labeled malicious during mining and benign when devices were not mining
   1  bid values from multiple bidders
   1  obfuscated and non-obfuscated personas
   1  preset fan speeds and correctly labeled mmwave samples
   1  certificate transparency logs and isp-case comparison
   1  potguide and government marijuana dispensary lists; local legality checks
   1  balanced training-set pairs of similar and different devices
   1  coalition against stalkerware definition
   1  class signatures and exodus privacy tracker information
   1  class signatures in the cocoapods repository
   1  x-ray database of tracker companies and tracking domains
   1  activity labels derived from selected youtube video titles
   1  blocking configurations and statistical difference from control
   1  held-out passive-party features and ground-truth labels
   1  fp-inspector classification of fingerprinting and non-fingerprinting scripts
   1  patched tor and shadow traces
   1  cms enforcement tracker summaries and violated gdpr articles
   1  dpo questionnaire responses
   1  authors’ identified respawned-cookie owners
   1  authors’ market-sector categorization
   1  user account posting the code
   1  controlled associated and non-associated trace pairs
   1  android permission definitions
   1  control experiments with publicly known takedown, developer-blocked, device-targeted, and carrier-targeted apps
   1  exodus privacy list of 400 trackers
   1  http status 418, requests containing 'wp-' or 'access.log'
   1  purchased-follower samples from two accounts
   1  reference-image identity
   1  sc09 speaker and 0-9 labels
   1  cmp cookie-purpose declarations
   1  cmp cookie-purpose labels
   1  github verified creator designation
   1  reactos source-code inspection
   1  expert inspection of candidate rpc entry points
   1  usb pin-pad keylogs synchronized with video
   1  automatic network-flow labeling using android framework instrumentation
   1  automatic app-level network-flow labeling
   1  docker scan vulnerability database
   1  anchore vulnerability database
   1  reverse-image-search verification of photo sources
   1  seven scientific papers and seven online resources
   1  32,093 vulnerabilities with functional-exploit evidence
   1  42,195 exploitdb pocs with file extensions
   1  bugtraq exploit pages and other exploitation-evidence scrapes
   1  ibm x-force exchange and tenable nessus labels
   1  screenshots labelled to indicate whether they require user credentials
   1  370 covid-19-related keywords in 15 languages
   1  known-tracker list
   1  two independent reviewers per paper, with dispute resolution
   1  clean calibration distribution
   1  chainalysis-provided ground-truth labels for 60 tp and 60 fp clusters
   1  chainalysis-labeled tp and fp clusters
   1  chainalysis reactor tags
   1  phone models used in the experiments
   1  marketplace plugin files and md5 hash comparisons
   1  participant diary responses
   1  dns-resolved destination domains observed at the tor exit relay; pseudonymized with keyed hmac-sha3-256
   1  pseudonymized exit-relay domain labels
   1  known valid, invalid, and manipulated signature documents
   1  logged javascript calls and property accesses
   1  unicode emoji v11.0
   1  2021 allsides media bias chart
   1  classification proposed by bastys et al.
   1  dynamic validation using 10-second delay within 1 million characters
   1  combined vulnerabilities reported by at least one tool
   1  transmitted random bit strings
   1  known red or green experimental traffic-light state and correctly recognized bdd100k instances
   1  known red or green experimental traffic-light state
   1  safe-driving, eco-driving, and auto-insurance rules
   1  vpa platform policy phrases
   1  ccpa/gdpr requirements and exploratory analysis of 18 policies
   1  urlhaus
   1  malwareworld
   1  greynoise
   1  experimentally constructed target and non-target samples
   1  cache profiles for n target-user states
   1  victim, other-user, and public-resource traces
   1  categories provided by the extension store
   1  1,343 validated origin hijackings and 2,365 rpki-derived legitimate moas conflicts
   1  route origin authorizations
   1  routing registry prefix-ownership records
   1  bgpmon-reported events
   1  speech commands labels
   1  dnssec, tlsa, and starttls certificate consistency checks
   1  user-assigned sensitivity labels from survey 2
   1  ja3er.com fingerprint database
   1  user-agents returned by the ja3 database
   1  known video service providers
   1  known video titles
   1  nine domain experts
   1  expert-curated topics
   1  paired real celebrity and deepfake images
   1  department of health & human services policy dates
   1  wikipedia moderator-verified sockpuppet groups and temporally constructed evasion pairs
   1  spot-checking parsed results
   1  wikidata properties, wikipedia categories, and page pronouns
   1  wikipedia category names and country lists
   1  majority vote over six fact-checking organizations' labels
   1  penn discourse treebank, wordnet, and causal constructions
   1  sentiment, causal-language, and core-value coding
   1  mobsf tracking-sdk identification
   1  nvram parameters and web-server configuration files
   1  six collection hashtags
   1  literal scores used as distant supervision
   1  device information from manufacturers, suppliers, users, and product descriptions
   1  sentiment140 sentiment labels
   1  debate.org votes, using “who made more convincing arguments?” as criterion
   1  weakly supervised seed attribute sets
   1  rfc8617 recommendations
   1  train/test membership labels
   1  nyt ground-truth document labels
   1  twitter social media data used to train the models
   1  18 purchased cpe devices and aggregate wan/bssid observations
   1  ieee-assigned oui records
   1  euphony malware-family information
   1  stockx target-consumer labels
   1  top-20 percentile of resale premium
   1  trojannn, badnet, cleanlabel, and weight-pruned models
   1  open-source attack data
   1  normal running-mode sensor data
   1  test-set ground-truth labels and human task outputs
   1  iana root zone database
   1  game logs and match outcomes
   1  seven existing pocs and reimplemented pocs
   1  published framework literature
   1  idle traffic with inferred periodicity
   1  manufacturer disclosures for two case studies
   1  ccadb, mozilla, apple, and microsoft trust stores
   1  mozilla, apple, and microsoft trust stores
   1  greynoise actor reputation labels
   1  rtp payload type headers
   1  webrtc per-second frame-height metrics
   1  webrtc application-level qoe metrics
   1  three initial labels from existing studies plus iteratively discovered activities
   1  third-party domain reputation and scam databases
   1  clamav database
   1  filescan.io database
   1  inquest database
   1  cert.pl mwdb database
   1  yoroi yomi database
   1  wechat and baidu key-validation apis
   1  snmpv3 vendor labels
   1  service-banner vendor information
   1  authoritative irrs, bgp, and rpki
   1  isp-disclosed throttling policies
   1  steven black host list
   1  fido alliance mds entries
   1  instrumented llvm 14 pass logs
   1  rich-header compiler ids
   1  ad-blocking lists compiled by ren et al.
   1  exodus tracker list
   1  ip addresses and peer announcements
   1  provider peer ip addresses
   1  snyk vulnerability database
   1  codeql queries written for target cwes
   1  qax threat intelligence
   1  360 security threat intelligence
   1  dns delegation, historical dns, ip, as, geolocation, certificate, http, and pdns comparisons
   1  known txt-record categories
   1  security-operator log analysis and examination of retsina/baseline detections
   1  academic literature and official guidelines
   1  signup attempts on 160 ground-truth domains
   1  participants' categorization during activities
   1  rsa signature verification and successful modulus factorization
   1  observed signatures associated with each rsa public key
   1  password matches in the rockyou wordlist
   1  systematically derived from allowed protocol message sequences
   1  thorough code review of all raw detection results
   1  known evcms seed candidates
   1  known can-node positions and identifiers
   1  menn predictions from validation and test folds
   1  validation against 41 public resolvers
   1  audiomnist speech-digit labels and collected victim labels
   1  other speakers' labeled mmwave data; victim data unlabeled during adaptation
   1  known tosyn-mutated tokens
   1  features provided by the bot and fraud detection/defense platform
   1  1,000 em samples from four finger-pressing statuses
   1  open-ended survey answers
   1  provider confirmation, dns resolution, server banners, open ports, and community comments
   1  known decoding strategies and hyperparameters configured in evaluated models
   1  controlled app-launch activity labels
   1  controlled website-visit labels
   1  controlled command labels
   1  controlled foreground-app labels
   1  publicly available vulnerability list also used by dependabot
   1  10 google assistant voices and other sound sources
   1  dns-resolver-based isav measurements
   1  149 abnormal and 851 normal ipv6 addresses
   1  file-encryption checks and file hashes
   1  authors' binary persona labels
   1  authors' binary video labels
   1  six tested ml frameworks
   1  2,081 phishing website reports and 2,000 official dapp websites
   1  chainabuse txphish reports and official web3 project websites
   1  host-app error messages and successful invocation feedback
   1  categories of documented apis
   1  android apis accessing permission-protected resources
   1  original and de-identified image pairs; celeba attributes
   1  200 celeba identities
   1  celeba's 40 binary attribute labels and face++ attributes
   1  decoded satellite id and beam id fields
   1  table 2 cloud-provider mx domains
   1  brightspace, courseleaf, blackboard, and canvas provider domains
   1  zoom, webex, adobe connect, skype-for-business, and microsoft authentication records
   1  known experimental camera states
   1  camera and non-camera traffic flows
   1  ios touch api data
   1  app used by the participant
   1  known app types as positive samples and unseen app types as negative samples
   1  cal. civil code 1798.140 categories
   1  online searches using crunchbase, netify, and other resources
   1  known original/edited pairs and provenance manipulations
   1  participant step-1 policy responses and step-2 human decisions
   1  audiomnist labels
   1  captured traffic, portal html, and experiments on test accounts
   1  protocol websites, social media, public repositories, and google
   1  intent-filter inspection, installation, and test-url loading
   1  libradar fingerprints augmented with exodus and online intelligence
   1  controlled webpage invoking device-sensor javascript apis
   1  known identifiers and sensitive data, including hashed values
   1  names or encoded representations of automatically visited websites
   1  operator websites, internet routing registry records, nlnog, and one step dictionaries
   1  nist national vulnerability database
   1  student-labelled image sets
   1  ofac list of sanctioned addresses
   1  relay reports and builder-to-proposer payments
   1  kontur population-density data
   1  decrypted volte messages from an attacker-controlled ue
   1  known comfort noise frame sizes and retransmission frequencies
   1  digital element mobile carrier database
   1  bgp community validation for 11 ases
   1  irr and as operator websites
   1  shadow-model configurations
   1  justdomains domain list
   1  university password history database
   1  university password-composition policies
   1  timestamped it security account-lock and password-reset logs
   1  co identifiers in comcast and spectrum hostnames
   1  known clean and poisoned models
   1  pii keyword rules in appendix b
   1  iotlineup.com device categories
   1  denied nodes in the crawled domain graph
   1  graph paths from benign to denied nodes
   1  shortest-path flow through domains
   1  python-captured key press and release events
   1  encoded secret bit
   1  qualitative extraction from 10 empirical works
   1  documented uui values and cross-device/session comparisons
   1  dns response fingerprints
   1  nmap os fingerprints
   1  known commercial device models
   1  iab tcf vendor list and external mobile cmp resources
   1  gdpr-derived criteria catalog
   1  professional security audits of smart contracts
   1  iab europe cmp list
   1  gdprhub, enforcementtracker and privacyaffairs repositories
   1  ip intelligence feeds and dns/rdns observations
   1  coingecko token names and addresses
   1  coinmarketcap and coingecko token lists
   1  five research-team members coding incident descriptions
   1  paper's semi-automated ground truth
   1  research-team coding of free-response answers
   1  observed traffic leakage, blocking, or successful tunnel protection during experiments
   1  table 1's 11 certified sdks
   1  non-minified function names, grouped and condensed into 189 labels
   1  inductive coding by five research-team members
   1  majority voting results of snopes.com, factcheck.org, and truthorfiction.com
   1  rips list of 205 critical apis
   1  phpggc repository of known gadget chains
   1  red-team configuration files and darpa ground-truth files
   1  red-team attack campaigns
   1  wikipedia article description-presence labels
   1  automatic labels derived from invoked api access-control sensitivity
   1  experiment-defined app and activity labels
   1  experiments with one, two, and three simultaneous victims
   1  experiment-defined app labels
   1  pandataset and gptplagiarismdataset
   1  msrp evaluation set
   1  github code-search hit counts and matching snippets
   1  facebook groups deemed non-recommendable; regular-expression matching for breastfeeding groups
   1  regular-expression terms matching group names
   1  none reported
   1  sequences of deresistor and geneva traces
   1  known sender and receiver locations during measurement collection
   1  wikiart labels and 13 digital genre labels
   1  20 original artworks from each artist
   1  abuseipdb reports
   1  openimage (v6)
   1  content experts and subject matter experts
   1  irdb and remote central forums
   1  known email, phone, pin, and password formats
   1  rpki route origin authorizations
   1  known spammer ip addresses
   1  tradingview permanently_suspended status
   1  cisco talos reputation classification
   1  web-app public apis and reconstructed malware sessions
   1  693 exodus tracker signatures plus 32 externally reported trackers
   1  five country-specific app-store ratings
   1  50 representative comments from each topic and a 5,000-comment pilot set
   1  two attack categories for training and one unknown attack for testing
   1  airdrop, ens, and debank side-channel data
   1  known black and white pixels
   1  top and bottom 20% of schools ranked by post-minus-pre attribute change
   1  participants' recorded commands and configuration modifications
   1  opensecrets political entities
   1  iterative analysis by a bilingual political scientist
   1  labeled set of 150 ads
   1  ten volunteers opening and closing evaluated applications
   1  ten volunteers walking at preset distances
   1  573 evidenced and 573 non-evidenced responses
   1  navigator.useragent strings
   1  finorg session tags and user-agent strings
   1  240 scanner runs and 159 human browsing sessions
   1  experiment-defined audible and inaudible recordings
   1  california department of justice registered data-broker list
   1  cisa catalog, extended through comment-term snowballing
   1  chromium and firefox issue-tracker fields, comments, and advisories
   1  service names and identified parent owners
   1  orlis and atvhunter ground truths; source-code checking for as2
   1  package names, signing certificates, process names, and log tags
   1  third-party, official, and mutated official qr codes
   1  populist party tags
   1  publicly available company-name lists
   1  issuer comparison with certificate transparency logs
   1  crowdsourced recognition of visual distortions by 18 users
   1  known sql operations and key-value updates
   1  majority vote by three journalism degree holders
   1  shadow gradients labeled by property-ratio bins
   1  publicly available security reports verified and addressed by developers
   1  origin as numbers from route announcements
   1  app installation counts and play market categories
   1  devices labeled by malware or pua encounters
   1  arin registration data and fcc provider registration data
   1  formal detection rules for nine vulnerability types
   1  fingerprinting functions from academic studies and privacy solutions
   1  known ransom note samples
   1  sqli microbenchmark vulnerable and parameterised samples
   1  organizations identified by greynoise
   1  semi-structured interviews and twitter account data
   1  root-servers.org published root-server sites
   1  unique evm log event hashes
   1  high-confidence labels identified by consistency checks
   1  public vulnerability databases and android security bulletins
   1  last android version and device chipset
   1  voice-command and idle-state labels from controlled invocations
   1  known command, skill, and streaming-command identities
   1  authors selected and invoked commands, skills, and streaming services
   1  authors' audit of blockscout and sourcify implementations
   1  resolver websites and service descriptions
   1  5,554 existing and 5,554 new beatleader users
   1  bdd100k and carla object-class labels
   1  benign context profiles
   1  cross-monitor deduplicated union of searched certificate sets
   1  controlled sybil attacks and no-attack experiments
   1  deployed rules' event and command parts
   1  16 previous studies and 6 cves
   1  independent review of 1,016 resource clusters
   1  chrome telemetry of users’ past permission-prompt actions
   1  reported accounts from twitter, scamsniffer, etherscan, cryptoscamdb, and chainabuse; accounts with gifted-in nfts labeled drainers
   1  connection with phishing attackers and possession of suspicious nfts
   1  labeled drainer and regular-user accounts
   1  same labeled accounts as drainclog
   1  csrf scanner-derived token patterns
   1  five-participant pilot recordings
   1  known typed keys and controller coordinates
   1  known keystroke sequences and observed acoustic transitions
   1  labeled traffic from 100 android apps
   1  labeled normal traffic from android apps
   1  packet-size frequency distribution
   1  controlled client and onion-service sessions with endpoint ground truth
   1  controlled correlated and non-correlated flow pairs
   1  roa objects from rpki
   1  rpki validation status of route objects
   1  rir transfer logs and isp-as relationships
   1  zap alert types and risk/confidence labels
   1  fortune global report
   1  developer confirmation and stack-trace analysis
   1  observed funded and failed campaigns
   1  ground-truth object in evaluation scenarios
   1  virus total classification service
   1  high-confidence normal and malicious sets selected by density and distance
   1  existing session-level cyberbullying or neutral labels
   1  wikipedia editing community
   1  uc berkeley guide to fake news
   1  reddit mental-health communities and non-mental-health reddit content
   1  known deserialization vulnerabilities and cves
   1  functional models of known arithmetic operations
   1  policylint ontology
   1  lfw identity labels
   1  lfw images and attacker labels
   1  labeled privacy-notice sentences and privacy-policy notices
   1  3,672 labeled privacy-notice sentences with bio labels
   1  mbfc labels and factual-reporting and credibility ratings
   1  groundtruth appraisal reviews and 1,000 random reviews
   1  hatebase.org hate-word list
   1  google's static classification provided with the chrome browser
   1  affiliated security vendors
   1  official provider websites and dns-response inspection
   1  14 pii types from a nist report
   1  reports by reputable blockchain auditing communities
   1  independent auditor reviews and consensus
   1  known responses recorded in the behavior repository
   1  known cdn response headers
   1  npm package advisories
   1  cookieblock functionality description
   1  matched and unmatched entry-exit flow pairs in dcf
   1  9,210 abnormal calls in collected transactions
   1  standardized prompt-sharing formats
   1  platform tags and keyword matching
   1  three labelers independently labeled 400 sampled prompts
   1  pairwise levenshtein similarity among 1,405 prompts
   1  definitions from the computer security resource center or wikipedia
   1  raw traffic data accompanied by labels
   1  adwords verticals
   1  apt group labels from public apt attribution reports
   1  historical vulnerability combinations from github issues
   1  cloudflare's 150 content categories
   1  repeated transmission of exploiting packets
   1  1,000 labeled packets per scenario
   1  http responses, side-channel signals, file access, and netflow records
   1  user-marked false alarms
   1  calculated layer-operation overhead
   1  interview transcripts and open comments
   1  oplis scores and german population norm table
   1  modified wamr translator labels
   1  recorded timing data for five wasm instructions
   1  similar vulnerability assessments in cvss examples
   1  ores damaging labels
   1  172,000 ordered and disordered sentence triplets
   1  ores damaging and non-damaging revisions
   1  insecure code fragments specified for the aes, sql injection, and os command injection tasks
   1  klees et al. guidelines and implicit community guidelines
   1  dasp vulnerability categories and smartbugs curated labels
   1  meta-moderated political advertisements
   1  authors' semantic dictionary
   1  czds zone snapshots and rdap timestamps
   1  rdap-reported registration timestamps
   1  ipv6 hitlist responsive addresses and inferred prefix-boundary changes
   1  gns3 router experiments
   1  snmpv3 vendor labels and gns3 fingerprints
   1  snmpv3-labeled router measurements
   1  apple app store review and human interface design guidelines
   1  netify and blocada classifications
   1  3gpp cause codes and antenna-vendor sub-cause descriptions
   1  exploit-db database
   1  ipinfo database
   1  systematic review of five standards and discovered vulnerabilities
   1  pre-recorded reference sounds
   1  autonomous-system and residential-netblock data
   1  source address, destination-ip count, packet rate, and one-hour expiration
   1  juliet bad/good programs
   1  original and predicted gender labels
   1  device-specific fingerprint pairs
   1  known poisoned and unpoisoned fingerprint pairs
   1  authorized and unauthorized graph construction
   1  snli plus domain-expert validation and rectification
   1  known accessed websites and videos
   1  known bandwidth authority ipv4 and ipv6 addresses
   1  known client and measurement traffic
   1  website with known dependencies
   1  scanning reports from three commercial services
   1  2,111 threat actor names and 4,071 malware family names from curated sources
   1  stix vocabularies and keywords from attack pattern, malware instance, target information, and threat actor objects
   1  curated threat-actor and malware keyword lists
   1  redpajama-data membership and post-release source dates
   1  two-author independent coding
   1  human similarity ratings on perturbed speech
   1  3gpp recommendations and requirements
   1  five high-internet-freedom vantage points
   1  ciphertexts with known decoding success or failure
   1  research-team review of participant attempts
   1  action labels recorded by alipay-b callback functions
   1  historical data, assumed predominantly benign
   1  fingerprint pattern labels for collected fingers
   1  takedown request emails
   1  github badbits information
   1  prior-work alias lists
   1  three random probes per /96 prefix
   1  self-curated motion-analysis labels
   1  self-curated displacement labels
   1  hmm labels filtered by clustering consistency checks
   1  40,000 cnn/dailymail sentences for transition probabilities
   1  20 prior properties and 7 new properties
   1  vpnapi.io ip classification
   1  mitre att&ck technique descriptions and detection strategies
   1  mitre threat groups, software, and campaigns, augmented with malpedia
   1  candidate matches from mitre att&ck, malpedia, and keybert
   1  participant interpretation with researcher safety screening
   1  databases of known spam/phishing emails
   1  participant interview accounts categorized using thomas et al. and zheng et al. taxonomies
   1  preliminary coding of responses from about 10% of participants
   1  ground-truth membership labels for training and public samples
   1  controlled harvester configurations
   1  controlled energy availability conditions
   1  rpki validation states, valley-free criterion, as relationships, and domain experts
   1  rpki validation results
   1  112,623 non-login and 2,242 login ui images
   1  acceptance of certificates with four known validation defects
   1  stack traces from hooked certificate-validation functions
   1  packages appearing in more than five apps
   1  domains contacted by more than five apps
   1  calls to default certificate validation override functions
   1  15,114 responses from six candidate llms
   1  matched versus unmatched head-movement traces and video fingerprints
   1  known train/validation membership split
   1  real-scene images versus monitors displaying photographs
   1  electron security recommendations and prior exploits
   1  resolver process status
   1  fpdns fingerprint database
   1  official brand logos
   1  322,327 english posts
   1  libradar library detection and domain-name matching
   1  fresh installations and vendor showcase/known deployment lists
   1  top-10 most severe cves affecting studied application versions
   1  two-researcher survey and validation
   1  public security advisories, conference programs, research blogs, and vendor security bulletins
   1  known i1, q1, u1, and u2 operations
   1  twitter and facebook application traces
   1  100 policies identified by fingerprint keyword searching
   1  bhr-blocked alerts reviewed for benign triggers
   1  soc forensic incident reports
   1  soc analysts' contextual inquiries and organizational records
   1  android app permissions and declared dss practices
   1  valid component sequences from ufc101 and bdd100k
   1  sigma rules labeled malicious; enterprise benign events labeled benign
   1  feodo botnet-family labels and benign/malicious source lists
   1  similarity graph of address features and transaction histories
   1  blur season 1 airdrop records and wallet transaction histories
   1  20 ordinal inscription collections from ordinalhub
   1  pink-slime, national-news, and local-news outlet categories
   1  experts in operations and maintenance
   1  national vulnerability database
   1  genderize.io labels with probability above 90%
   1  skilldetective data-type list
   1  historical pickup-request records
   1  industry-partner labels derived from threat intelligence and a detection model
   1  online industry-engine comparison
   1  participant wishes and needs
   1  five control nodes and repeated vp detections
   1  control-node ases, titles, prompts, tls certificates, and connections
   1  regional internet registry data
   1  api responses and user-page status
   1  platform-provided suspension labels
   1  coingecko-verified genuine tokens and exchange-defined tokens
   1  ethereum erc-20 transfer records
   1  2,746 phishing accounts labeled by users and security companies; randomly collected benign eoas
   1  ground-truth labels provided by [29], further validated by domain experts
   1  time-aligned timit phonetic transcripts
   1  study condition requirements
   1  five hired workers labeling contexts and topics
   1  developer feedback and vulnerability fixes
   1  three independent turkers agreeing on the sequence
   1  filter responses to membership queries
   1  malware samples resolving domains
   1  reverse-engineered dga families
   1  isc ptr scan
   1  known reference implementations for tea, aes, rc4, md5, and rsa
   1  known authorship of forum documents
   1  known same-author and different-author blog pairs; cross-forum email matches
   1  reprodroid researcher-labelled source/sink flows
   1  historical vulnerability and framework-commit analysis
   1  known vulnerabilities at the time of collection
   1  explicitly labeled legitimate-user and impersonator samples
   1  inclusion criterion that papers discuss a vulnerability
   1  cisa kev catalog
   1  android apk xml declarations and view initialization routines
   1  safedns
   1  82 scanner-identified malicious models plus 252 benign models
   1  default weights-only unpickler policy
   1  binary labels identifying adversarial and vehicle-surface points
   1  device acknowledgments, normal control-command responses, and recovery behavior
   1  successful ssh dos or ftp poisoning
   1  known transmitted modulation scheme
   1  known backdoored and non-backdoored model architectures constructed by the authors
   1  consensus coding by three study members
   1  research-team coding of collected screenshots
   1  research-team clustering of 22 platforms
   1  class label text embeddings
   1  risc-v opcodes repository
   1  official scanner ip lists, reverse dns, and cyberspace-search-engine correlation
   1  scanner-organization labels from service-level predictions
   1  automatic payload verification and runtime monitoring
   1  literature-derived code patterns
   1  440 sampled static data flows
   1  youtube and bilibili selected videos plus other-class videos
   1  15 selected android applications
   1  4,000 evenly split signal traces
   1  samples of twitter and telegram posts
   1  measured physical-address and cache-slice tuples
   1  tchecker reports
   1  experimenter-assigned physical locations and contexts
   1  normal conferencing audio versus audio containing in-channel sensing signals
   1  google's lists of unsafe web resources
   1  curated by security-expert partners
   1  matching github repository ids
   1  web-interface inspection and lfs traffic analysis
   1  creator and interpreter responses
   1  post-task interview transcripts and study notes
   1  curated open database maintained by censored planet
   1  two trained lawyers analyzing all 24 policies
   1  oauth/oidc parameter inspection
   1  iterative coding of 440 s&p-related posts
   1  known binary-format sections, addresses, and permissions
   1  prior reverse engineering of plc binaries
   1  exploit code and tracing-tool system calls
   1  operator-provided ground truth
   1  cookie creator domains, script stack traces, and outbound requests
   1  cookie-value substrings and derived base64, md5, and sha1 forms
   1  webpage for each vpn service
   1  ipinfo long-term geolocation trends
   1  tls-key-assisted labeling of rpc api packet sequences
   1  public blockchain ledger transaction histories
   1  major web pki root stores and ccadb
   1  ad-specific keywords encountered during the empirical study
   1  observed promotion graph relations
   1  tls certificate san inspection
   1  authors' outlier definitions
   1  admin, crawler, and lower-privilege user responses
   1  researchers' task-specific gdpr criteria
   1  k-shell values on the actual forwarding graph gm
   1  k-shell decomposition of the actual forwarding graph
   1  campus it investigation and validation
   1  ground-truth sample posts
   1  sampled posts
   1  privacypolicy and iot sensitive data taxonomies, with randomly sampled non-hyponyms
   1  movielens-100k and steam-200k interaction records
   1  dark visitors bot-category list
   1  certora human-written properties
   1  participant survey demographic responses
   1  one researcher coded follow-up responses; another applied the codes
   1  published rov measurements in [41]
   1  reviewers' inspection of uafx warnings and bug-triggering pocs
   1  category examples co-developed with a legal scholar
   1  ieee-assigned ouis
   1  apkid packer and obfuscator labels
   1  in-protocol signal labels identifying circuit types and sessions
   1  pre-recorded voice commands and words
   1  related research articles and gdpr examples
   1  cookie jar and sent cookies comparison
   1  known iot-device clock frequencies
   1  cpu-utilization model, battery voltage, and hamming-distance power model
   1  rpki data from all five rirs
   1  gpt-3.5 turbo labels and active-learning labels
   1  unaltered-image baseline detection results
   1  website and tracker organization ownership
   1  historical provider defensive registrations
   1  known hamming-weight relationships used for poi correlation and cluster labeling
   1  persona-specific target labels
   1  3gpp ts 24.501 and ts 23.502
   1  required nas responses specified by 3gpp
   1  actual access-control enforcement implemented by aosp apis
   1  1hosts github tracker list
   1  cookie database
   1  model labels from hugging face model collections
   1  analysis of transaction data, logs, tokens transferred, and call traces
   1  etherscan labels
   1  big-15, mb-24, and malwaredrift malware labels
   1  original big-15 labels when available
   1  two co-authors' independent coding of 55 documents
   1  controlled hammering of eight vulnerable locations
   1  six experimenter-defined camera activities
   1  key press timestamps from /dev/input/eventx
   1  40 common english words
   1  seized wallet data and known private-key constraints
   1  uk financial conduct authority handbook
   1  hlr lookup results
   1  uk companies house register
   1  public php issue reports, conversations, and patches
   1  flowfusion-discovered sanitizer reports and bug analysis
   1  killnet-associated ip addresses and networks
   1  9 disclosed free-input vulnerabilities and fixes from 7 public audit reports
   1  buggy functions in ddisclosed
   1  published ivi vulnerability records
   1  active dnsbl verification
   1  same-model positive pairs and different-model negative pairs
   1  duckduckgo entity list
   1  majority responses across repeated prompts
   1  63,742,912 aggregated password vaults
   1  known locally deployed software versions
   1  responses from 164 locally deployed phpmyadmin versions
   1  public issue reports and maintainer decisions
   1  consented user-provided attributes verified against alipay server data
   1  open-ended responses from study participants
   1  person pos tags
   1  reference list of 1,230 unique names aggregated from three sources
   1  trace labels from fbsad
   1  msad attack labels
   1  nas-layer attack intervals
   1  etherscan/bscscan reputation lists and coinranking
   1  arkham intel labels
   1  conference track, title, and abstract
   1  confirmed mining and benign addresses from the confirmation module
   1  authors published and modified test packages on domains
   1  enron name-email pairs
   1  100 freelaw training members and 100 freelaw test non-members
   1  two-author qualitative coding of conversational data
   1  automated equivalence checking after test-case filtering
   1  opp-115, ppgdpr, c3pa, and author-labeled samples
   1  majority vote across eight libraries
   1  commit messages from 2020 to december 2024
   1  source patches and compiled kernel modules
   1  acceptable ads standard and acceptable ads committee validation
   1  dynamic verification of candidate mirror services
   1  authors selected three known malware and three benign applications per session
   1  power-trace measurements from proximity-sensor events
   1  gnb trace files and ue capture files
   1  experimental traces from controlled tor clients and hidden services
   1  verified trusted node data from tor metrics
   1  file-format standards and testing
   1  three cyber-threat detection platforms
   1  rpki roas and validated rpki objects
   1  comparison of invalid announcements with rpki-valid coverage and prefix lengths
   1  rovista rov protection scores
   1  tweets made by u.s. senators in 2021
   1  apwg ecx phishing urls
   1  word-rank features from gpt2-xl
   1  calibration measurements: five hit and five miss samples
   1  merchant ids extracted by scammagnifier and linked to known fraudulent websites
   1  authors’ assigned benign or misleading-recapture labels
   1  messages containing “climate change” or “global warming”
   1  first message posted in one of six selected activist subreddits
   1  fr200k user ratings
   1  validation set containing approximately 2,000 samples
   1  list of 23 state-affiliated russian websites
   1  keywords commonly associated with qanon
   1  human-reviewer comparison on 270 posts
   1  60 survey participants
   1  device-model database
   1  opensea-provided openrarity ranks
   1  2,000 widgets labeled by exact ui text into eight categories
   1  curated political elites with known ideological leanings
   1  phase 1 ideology estimates for 94,000 users
   1  203 attack transactions and 40,000 normal transactions
   1  labeled desktop windows with hco scans and no hco motion
   1  desktop and vr attack/no-attack evaluation windows
   1  vulnerable, patched/mitigated, and non-vulnerable software versions
   1  ens official normalization standard and complete domain list
   1  ens official normalization standard
   1  daily surveys and recruitment survey
   1  ccpa requirements and response records
   1  donation solicitation posts plus flagged communication channels
   1  url regular expressions from bakshy et al.
   1  behavioral interactions among reddit communities
   1  labeled phishing and non-phishing ethereum users
   1  mediabias/factcheck
   1  ads.txt and sellers.json relationships
   1  sellers.json entries
   1  keyword patterns and known malware references
   1  owasp top 10 targeted risks
   1  fixed smurf implementation
   1  verified (site, src, path) triples from source-to-sink analysis
   1  etherscan and bscscan account labels
   1  few-shot validation of taint-analysis alerts
   1  two-author abstract screening
   1  snyk database vulnerability records
   1  participant judgments
   1  training members and held-out test non-members
   1  reviewer consensus over paper text
   1  iab taxonomy
   1  14 government employees specializing in discrimination, racism, and bias
   1  industry-trusted address labels
   1  wikimia and mimir member/non-member labels
   1  exact-match searches of public webpages
   1  internet archive and january 2023 common crawl snapshot
   1  ds direct/background and relayed connections
   1  successful dns response to a scanning query
   1  attack verification by examining client responses
   1  builder addresses from etherscan and eigenphi
   1  locally recorded routerinfo publication logs from 10 host routers
   1  115 incident labels and attack transaction hashes
   1  cryptographic calls, snowballing, and global-variable analysis
   1  two independent researchers, with discrepancies resolved by a third researcher
   1  normal and attack labels in cic-ids2018, cic-ddos2019, and cicapt-iiot2024
   1  known normal and malicious traffic samples
   1  500 kakaotalk and 500 non-kakaotalk login logo images
   1  spike golden-reference model
   1  tests triggering known vulnerabilities or improving processor coverage
   1  known virtual-keyboard layout
   1  application identity labels from selected meta quest applications
   1  selected webxr application identities and categories
   1  experimentally rendered furniture, avatars, and participant-count scenarios
   1  controlled set of test devices
   1  urlscan scam-website homepage text
   1  crimson-detected scam websites
   1  standardized phrases in commit titles
   1  patchdb non-security labels
   1  su et al.'s list, refined using mdn and nguyen et al.
   1  curated list of abusable imports
   1  1,663 scam websites and 9,800 category-matched benign websites
   1  model trained on 7,000 queries and category tokens
   1  fine-tuning inclusion versus exclusion
   1  participant perceptions of risk in 89 interview scenarios covering 114 automations
   1  public individual voting records from snapshot
   1  existing literature and domain knowledge
   1  labels of each trace's intra-site pattern
   1  568 pgd pages and 410481 non-pgd pages
   1  place-name prompts with latitude and longitude
   1  nine security engineers in three groups
   1  survey findings and platform analysis
   1  baseline data without covert traffic
   1  publicly documented providers
   1  torres et al.'s public analysis code
   1  97 verified apks and legitimate wallet references
   1  bitton et al. isa taxonomy
   1  security patches and exploit paths
   1  categories identified through trigrams and sample inspection
   1  two privacy and software-development experts, with third-expert adjudication
   1  200-dimensional rater representations and co-rating histories
   1  media bias/fact check categories
   1  mediabias/factcheck labels
   1  science feedback credibility scores
   1  matching publisher ads.txt entries with ad-network sellers.json identifiers
   1  original prompts and reconstructed prompts
   1  scripted victim accesses and known cache state
   1  offline hit-miss histograms and scripted website accesses
   1  prediction validity under csmia and lomia
   1  ground-truth answers and original answers
   1  whether follow-up udp payloads were received
   1  successful direct http/3 requests
   1  experimentally assigned expectancy conditions
   1  randomized sensitive attributes and membership bits in privacy games
   1  four independent board-certified pathologists; senior pathologist review
   1  password-pair similarity, popular-password membership, and semantic reuse definitions
   1  human-expert validation of 20,000 requests
   1  400 traffic flows from each of three classes
   1  gorilla identifiers in binaries and contacted c2 server
   1  concept and disease labels in derm7pt, skincon, pbc, and thyroid
   1  44,434 video pairs labeled by research assistants
   1  research assistants' policy-based labels
   1  prior wash-trading methodologies
   1  goplus token security
   1  criteria used in [58]
   1  data winnowing and corrected silhouette-free frames
   1  subtitle files for the suspect video library
   1  comparison with coordinator api transaction lists
   1  active coordinator api monitoring
   1  crisismmd humanitarian class labels
   1  crisismmd class labels and text rationales
   1  longenough movie identities
   1  audiomnist training and fsdd fine-tuning
   1  operator confirmations and external validation
   1  exactly retrained models
   1  ground-truth layer label maps
   1  ground-truth hyperparameter values
   1  ofac-sanctioned address set
   1  embedded versus unwatermarked or incorrect candidate keys
   1  true embedded key and non-matching candidate keys
   1  shanghai university ranking
   1  known vulnerable functions identified for 27 cves
   1  transactions observed in the public mempool and matched against on-chain transactions
   1  itemized price sheets, service menus, and pinned posts
   1  observed communication patterns and functions
   1  static vehicle ids in 135,697 brussels mmv entries; xu et al. approach for status validation
   1  static-id brussels entries and successor-id matching
   1  lamp-qa personalized rubrics
   1  three medical professionals and a senior meta-reviewer
   1  anonymized gdsii layouts, known cell functions, and known substitutions
   1  claimed design labels and known exchanged-cell substitutions
   1  observed onetrust category names
   1  participant yes/no permission responses
   1  participant permission preferences
   1  trufflehog outputs
   1  x ads api taxonomy
   1  ground-truth actions and ui elements
   1  xcal device-side diagnostic logs
   1  ca-published revocation reason codes, signtool results, and certificate information
   1  company registration records
   1  ground-truth code samples
   1  expected outputs
   1  gartner product categories
   1  unfavorable financial term taxonomy
   1  official app store search results
   1  app descriptions and supplemental online sources
   1  labeled pips across ten categories
   1  two independent labelers
   1  public incident databases and documented real-world cases
   1  developer confirmation of potential vulnerabilities
   1  etherscan label cloud
   1  url ground truth used for correctness decisions
   1  on-chain profits exceeding 100 usd and eigenphi records
   1  rpki roa data and bgp routing tables
   1  hard-coded strings, extracted device values, and regular expressions
   1  three co-authors' independent categorization and majority decision
   1  first author's coding of free-response answers
   1  population samples and target samples
   1  participant privacy ratings on a 1–7 scale
   1  stimulus privacy attributes from vispr and participant ratings
   1  root-cause analysis of confirmed bugs in firmware binaries
   1  3gpp normative sentences and asn.1 definitions
   1  known blog authors and google-profile-linked blogs
   1  google profiles listing multiple blogs
   1  foe-absent training trials and controlled foe-present workload trials
   1  exact-match verification and developer confirmation
   1  build-time inclusion of inserted assert statements
   1  open directory project taxonomy
   1  five-user swipe experiments
   1  known app identities in training and testing traces
   1  facebook's legitimate ssl certificates and rsa public keys
   1  kernel configuration files and elf symbols
   1  aosp base policies and compiled vendor policies
   1  android .rc files
   1  installed anti-spyware applications' scan outputs
   1  deductive codes from interview guide and inductive transcript coding
   1  magika test-set file-type labels
   1  publicdns.xyz list
   1  ppone enforcement-action labels
   1  researcher-coded sample of 100 rationales
   1  android cdd and prior-work mitigation requirements
   1  android cdd-required hardening techniques
   1  extracted .cil files from firmware images
   1  androidmanifest.xml package names and intent filters
   1  known eea inputs from ecdsa signature-verification traces
   1  publicly available satellite and bus-system information
   1  tls server name indication
   1  gps and configured hardware-in-the-loop locations
   1  human identification of attack root causes and terminal impacts, followed by provenance tracing
   1  dnsviz error and validation codes
   1  dnsviz validation diagnostics
   1  dnsviz error codes and zone context
   1  published llc slice functions and /proc/<pid>/pagemap
   1  remote service validation of candidate credentials
   1  25 randomly selected posts from each of 86 clusters
   1  game titles labeled in ground-truth pcap sessions and server logs
   1  ground-truth activity-stage transitions
   1  server responses, tls handshakes, http content, and censor packet behavior
   1  domains consistently appearing in top lists for 30 days, plus monthly lists
   1  sinkdb and misp sinkhole lists
   1  command-sequence similarity
   1  historic ipv4 announcement data and as tags
   1  ip reputation and malware infrastructure feeds
   1  operator-confirmed emnify configuration
   1  etherscan labels and transaction relationships
   1  acquired toolkits and files from reported phishing websites
   1  custom keyword list
   1  dynamic verification with controlled redirect payloads
   1  controlled local server receiving unique test identifiers
   1  collaborative coding by legal and security researchers
   1  prohibited-library announcements and malicious examples
   1  malicious content-rendering examples
   1  wechat platform operational regulations
   1  publicized malware examples and interactions with wechat security teams
   1  default ca bundle on ubuntu 22.04
   1  routersploit exploits and linked security advisories
   1  device trees, kernel configurations, and elf headers
   1  linux kernel banner strings
   1  extracted file mime signatures and linux filesystem paths
   1  android's official permission categories
   1  bösch privacy taxonomy and gray dark-pattern categories
   1  cookie stability, length, expiry, and entropy characteristics
   1  dns cname chains aligned with known tracking entities
   1  gpt-4 judgments comparing outputs with gpt-3.5-turbo-0613
   1  50 reddit imss websites and 51 verified nextdns websites
   1  cm security bulletins
   1  cm bulletins and named discoverers
   1  tinycheck stalkerware indicators of compromise
   1  pre-compiled tracker databases
   1  human development report, world economic outlook, global wealth report, democracy index, and western-society criteria
   1  datadome decisions on each request
   1  botd decisions on each request
   1  datadome and botd detection decisions
   1  karma device signatures and vulnerability database
   1  codeql alarms and source-sink traces
   1  lira, attack r, and rmia attack results
   1  dns zone files and 214 domains with known registration and expiration periods
   1  network traffic, decompiled apk code, package names, and app structure
   1  urls automatically loaded or present in html anchor tags
   1  custom ait attack labeling method
   1  known rodinia and nvidia cuda sample application identities
   1  known blender studio character identities
   1  browser popup triggered by selenium
   1  tchecker source-sink paths
   1  known streamed video representations
   1  ca/browser forum s/mime baseline requirements
   1  microsoft outlook, google chrome, apple macos, and mozilla root stores
   1  real training trajectories
   1  jailbreakbench goals
   1  successful retrieval of bidding records via the pbjs api
   1  baidu indexed search results
   1  known labeled attack samples
   1  iana ipv4 special-purpose address registry
   1  asdb
   1  as2org
   1  asrank
   1  accepted and rejected fp crs with potential benign indicators
   1  detailed ground-truth documents accompanying the attacks
   1  historical collection of 60 confirmed reentrancy attacks
   1  two independent experts, with additional experts resolving disagreements
   1  reference spectrograms for walking and standing up
   1  public vulnerability databases and vendor advisories
   1  mukerjee et al. [13]
   1  mosleh and rand [12]; robertson et al. [16]
   1  correlation of binary instruction patterns, group-name strings, and runtime behavior
   1  fact-checked events from politifact and the china internet joint rumor-refuting platform
   1  documented base models and fine-tuning provenance
   1  not-stated
   1  participant demographic questionnaire
   1  australian census and australian federal election study
   1  known pool members and excluded non-members
   1  literature-derived questions used as semantic retrieval seeds
   1  authoritative sources and expert domain knowledge
   1  pixel-wise ground truth from green-screen compositing
   1  package references across apks and developer identities
   1  filesystem hierarchy standard directory presence
   1  official foundation project lists and hosting-platform organizational information
   1  platform policies and unified cross-platform behavioral definitions
   1  official advertising policy documents
   1  dynamic interaction, runtime logs, stack traces, and reverse engineering
   1  original threat reports
   1  measured as paths in pl-bgp-rand
   1  20 unrelated libraries and 20 libraries with different versions
   1  android apk decompilation, cfgs, and backtracking
   1  declared android app permissions
   1  clinically diagnosed brain tumor cases
   1  vulnerability patches and reconstructed vulnerable functions
   1  commercial threat-intelligence labels
   1  greynoise benign-scanner labels
   1  original system prompt and user question
   1  four previously identified builder-searcher integrations
   1  websites and online sources associated with each as
   1  institutional scanner list
   1  user-reported activity
   1  file hashes and retrieved malware
   1  26 signatures across 18 c2 families
   1  participant-typed keys and statuses
   1  participant-typed key labels
   1  participant-typed word labels
   1  dynamic taint flows from privacy-sensitive apis
   1  dynamic native-method invocation traces
   1  discussion among the research team
   1  two testers' assessment of app functionality and lan necessity
   1  known in/out/unlearned shadow-model roles
   1  aggregated shadow-model populations
   1  315 reported cases and normal-user statistics
   1  users with at least 10 transactions under sampled headers
   1  200 vlp-32c images and 532 infrared attack-trace images
   1  benign detection traces
   1  observed bit flips and successful addressing-function recovery
   1  complete routeviews bgp table snapshot
   1  http protocol parsing and custom json/configuration grammars
   1  app categories retrieved from apple and google app-store apis
   1  3,000 confirmed phishpedia phishing websites
   1  open-source community-managed domain list
   1  curated malicious-package labels and benign-package verification with osv and snyk
   1  network traffic inspection using dummy pii
   1  webauthn security and privacy requirements
   1  known pin digits entered during controlled recordings
   1  passwords conforming to validation-set templates
   1  extracted video and subtitle keypoints
   1  alias mechanisms experimentally learned from 28 email providers
   1  common platform enumeration identifiers and public vulnerability databases
   1  voluntarily revealed or cooperating users' records and netflix ratings
   1  known user records
   1  prompted and submitted password pairs
   1  survey-collected extension lists
   1  496 github sbom-related discussions
   1  nine controlled subdomains with dnssec misconfigurations and one well-configured subdomain
   1  ten consecutive servfail responses or timeout after controlled cache injection
   1  one researcher coded all dialogues; a second independently verified 60 dialogues
   1  binary user feedback on presented traces
   1  developer feedback on api-usage traces
   1  labeled spectra produced by emlora and legitimate system processes
   1  35 popular mobile app development frameworks
   1  known correlated tor/exit flow pairs from controlled collection
   1  stable mac addresses for approximately 15 minutes
   1  wayback machine, google references, and request patterns
   1  archived android applications and observed dns queries
   1  event and command logs from one week
   1  user-agent string
   1  authors' knowledge of attack workflows
   1  experimentally configured foreground, background, and not-present states
   1  authors' hand inspection of linux kernel source
   1  database of known images of child exploitation
   1  known stalkerware list
   1  facebook and owasp account/session guidelines
   1  facebook account-merging guidelines
   1  bert-predicted comment sentiments
   1  website source-code keyword matching
   1  asvspoof2019 challenge labels
   1  original sample transcriptions
   1  five verified family names from multiple sources
   1  urlhaus reports
   1  alienvault otx reports
   1  token formats, use contexts, and equality across test rounds
   1  adc clock frequencies of the 13 tested voice recorders
   1  known websites selected for each browsing session
   1  known typed keys from participant sessions
   1  known typed keys from adversary sessions
   1  participant activity labels and typed sentences
   1  adversary's reconstructed keyboard
   1  selenium scripts and recorded ui elements
   1  valid answers from the authors' nameserver
   1  version.bind query and fpdns fingerprints
   1  black-box measurements of rate limits, timeouts, packet sizes, and response timing
   1  recorded calls to the reporting function
   1  pre-stress memory dumps and known programmed data
   1  51 power-cycle measurements per device
   1  observed state transitions and runtime behavior
   1  version.bind query and fpdns fingerprinting
   1  mawilab benign traffic excerpts
   1  vendor confirmation of reported vulnerabilities
   1  surrounding comments, commit messages, and post-version diffs
   1  computed from shared users and password reuse rates in cit0day
   1  iterative open coding of video transcripts
   1  retire.js vulnerability database
   1  owner survey, supplemented by service banners and webui content
   1  authors' defined vulnerable, limited, and secure ui-layer criteria
   1  collaborative open coding of five exercises
   1  open coding of organizer interview responses
   1  publicly disclosed vulnerabilities
   1  known nginx/openssl binary and reference access patterns
   1  1,090 helpful reviews and 1,090 non-informative ar-miner reviews
   1  audiomnist and speech commands labels
   1  legitimate-user, other-user, replay, and fake-hand experimental cases
   1  shared wallet addresses or github accounts
   1  researcher-developed open coding
   1  youtube-provided video categories
   1  known rsa multiplication/squaring patterns
   1  40 labeled phoronix applications
   1  linux kernel cves project
   1  paired outputs from protected and unprotected shadow models
   1  corresponding clean outputs of shadow models
   1  official mm api set
   1  blag output
   1  violated ppl formulas, runtime states, configuration values, and flight-stage tests
   1  rnti extracted using network signal guru
   1  decompiler error messages and comparisons of original and decompiled code
   1  expert examination of algorithm candidates
   1  corresponding vendors' confirmation and fixes
   1  waf bypass plus webapp parameter recognition
   1  labeled collision-graph neighbors from d1
   1  experiment-defined app and activity classes
   1  experiment-defined keyboard and activity traces
   1  rustsec advisory database review
   1  known target and surrounding devices
   1  three known attacks and four newly discovered attacks
   1  formal dy properties and extracted claims
   1  custom fingerprinting test pages and browserleaks demo pages
   1  over 40,000 ad disclosure texts

scripts/annotation_audit.mjs. Four maps, each entry carrying the deciding phrase. Running the file re-derives every probe and exits non-zero on any paper a map does not cover.

annotation_audit.mjs

// Hand audits for statistics:annotation. Two of them, both recorded here rather
// than in prose, so the page's two most load-bearing counts can be re-derived.
//
// (1) PROBE_AUDIT — the 30 papers matched by the `llm_annotator` title+summary
//     probe in scripts/gap_probe_roadmap.mjs, each read and given a verdict.
//     The probe is what put this page on the roadmap; the audit is what stops
//     its 30 becoming the page's population.
//
//       annotator  — the model assigns labels/extractions to items in the
//                    paper's own study population, and those labels are a
//                    result or feed one
//       instrument — the model generates artefacts (fuzzing seeds, interface
//                    specs, decompiled code) rather than labels
//       object     — the model, or an attack on it, is what is being studied;
//                    it annotates nothing
//
// (2) TEMPERATURE_AUDIT — the 55 papers, of the 175 that use an LLM to classify
//     something, whose full text contains the word "temperature". Read in
//     context, because the word has four other senses in this corpus (colour
//     temperature of a smart bulb, a weather API's payload, the softmax /
//     Gumbel-Softmax τ, simulated annealing) and one paper only cites a title.
//
//       value      — states a numeric decoding temperature for its own runs
//       default    — says "default settings", no number
//       discussed  — the word is about LLM decoding but no setting is given
//                    for the authors' own labelling runs
//       other      — a different sense of the word, or a citation title only
//
// Each entry carries the deciding phrase. Run this file directly to check both
// maps against the live probe/regex and fail loudly on any paper neither
// covers:  node scripts/annotation_audit.mjs
 
export const PROBE_AUDIT = {
  'CCS/2023/large-language-models-for-code-security-hardening-and-adversarial-testing': ['object', 'the LLM is the artefact being hardened and attacked'],
  'CCS/2023/protecting-intellectual-property-of-large-language-model-based-code-generation-a': ['object', 'watermarking a code-generation API; nothing is annotated'],
  'USENIX/2023/lost-at-c-a-user-study-on-the-security-implications-of-large-language-model-code': ['object', 'Codex is the treatment in a user study; the authors code the output by hand'],
  'CCS/2024/optimization-based-prompt-injection-attack-to-llm-as-a-judge': ['object', 'an attack ON LLM-as-a-judge, not a use of one'],
  'IMC/2024/analyzing-corporate-privacy-policies-using-ai-chatbots': ['annotator', 'GPT-4 Turbo extracts structured annotations from crawled policies'],
  'WWW/2024/human-vs-chatgpt-effect-of-data-annotation-in-interpretable-crisis-related-micro': ['annotator', 'GPT-3.5 Turbo rationale annotations compared against human ones'],
  'IEEE-SP/2025/make-a-feint-to-the-east-while-attacking-in-the-west-blinding-llm-based-code-aud': ['object', 'Flashboom attacks on LLM code auditors'],
  'IEEE-SP/2025/gptracker-a-large-scale-measurement-of-misused-gpts': ['annotator', 'gpt-4o-mini classifies crawled GPTs for misuse, manually validated'],
  'IEEE-SP/2025/understanding-users-security-and-privacy-concerns-and-attitudes-towards-conversa': ['annotator', 'GPT-4o classifies 2.5M Reddit posts alongside manual coding'],
  'IMC/2025/an-in-depth-investigation-of-data-collection-in-llm-app-ecosystems': ['annotator', 'GPT-4o and o1 label Action data collection and policy disclosures'],
  'IMC/2025/learning-as-to-organization-mappings-with-borges': ['annotator', 'GPT-4o-mini extracts and classifies organisation names, manually validated'],
  'USENIX/2025/abusability-of-automation-apps-in-intimate-partner-violence': ['annotator', 'Qwen2.5-Coder-32B-Instruct classifies 12,962 iOS Shortcuts recipes'],
  'USENIX/2025/make-agent-defeat-agent-automatic-detection-of-taint-style-vulnerabilities-in-ll': ['object', 'LLM-based agents are the target of the fuzzing'],
  'WWW/2025/suigpt-mad-move-ai-decompiler-to-improve-transparency-and-auditability-on-non-op': ['object', 'the LLM decompiler is the system being evaluated'],
  'WWW/2025/harmful-terms-and-where-to-find-them-measuring-and-modeling-unfavorable-financia': ['annotator', 'GPT-4o classifies 1.9M terms-and-conditions clauses'],
  'IMC/2025/from-voice-to-ads-auditing-commercial-smart-speakers-for-targeted-advertising-ba': ['annotator', 'GPT-4o extracts ad attributes from 420 hours of recordings'],
  'USENIX/2025/no-way-to-sign-out-unpacking-non-compliance-with-google-plays-app-account-deleti': ['annotator', 'GPT-4 classifies account-deletion disclosures, manually verified'],
  'IEEE-SP/2026/parasites-in-the-toolchain-a-large-scale-analysis-of-attacks-on-the-mcp-ecosyste': ['annotator', 'GPT-4o labels 1,360 MCP servers and 12,230 tools, manually validated'],
  'NDSS/2026/what-do-they-fix-llm-aided-categorization-of-security-patches-for-critical-memory-bugs': ['annotator', 'GPT-4-turbo categorises kernel security patches'],
  'USENIX/2026/pangolin-fuzzing-multilingual-iot-firmware-with-llm-driven-code-analysis': ['instrument', 'DeepSeek-V3 generates interface and parameter specifications for a fuzzer'],
  'USENIX/2026/sok-philter-uncovering-security-and-functional-gaps-in-ai-based-phishing-website': ['annotator', 'o4-mini and gemini-2.5-pro extract evidence from 55 papers against a codebook'],
  'PETS/2026/the-role-of-online-forums-in-developer-understanding-of-privacy-law-a-reddit-cas': ['annotator', 'GPT-5 and GPT-5-mini classify 2,248 subreddit posts, held-out test set'],
  'WWW/2026/tgnn-enhancing-pixel-tracking-detection-via-llm-driven-annotation-and-gat-powere': ['annotator', 'Qwen3-Plus labels web requests for pixel tracking, held-out test set'],
  'WWW/2026/bowling-with-chatgpt-on-the-evolving-user-interactions-with-conversational-ai-sy': ['annotator', 'GPT-4o annotates 825,672 donated ChatGPT turns'],
  'USENIX/2026/autonomy-comes-with-costs-detecting-denial-of-service-vulnerabilities-caused-by': ['instrument', 'GPT-4.1 generates prompts to drive a DoS fuzzer; validation none-reported'],
  'PETS/2026/word-level-annotation-of-gdpr-transparency-compliance-in-privacy-policies-using': ['annotator', 'GPT-4.1 annotates policy passages against a 200-policy human gold set'],
  'WWW/2026/longitudinal-trends-in-global-climate-change-discourse-on-facebook': ['annotator', 'Qwen2-7B-Instruct assigns topic labels to 299,329 Facebook posts'],
  'PETS/2026/operationalizing-the-motivated-intruder-a-codebook-guided-inference-framework-fo': ['annotator', 'Gemini 2.5 Pro classifies 227 prompts against a codebook, with human annotation'],
  'PETS/2026/personal-data-flows-and-privacy-policy-traceability-in-third-party-llm-apps-in-t': ['annotator', 'GPT-4o-mini labels privacy policies in the GPT ecosystem'],
  'USENIX/2026/cutting-the-gordian-knot-detecting-malicious-pypi-packages-via-a-knowledge-minin': ['annotator', 'GPT-4.1 labels package code context; a held-out test set on the RAG variant'],
};
 
export const TEMPERATURE_AUDIT = {
  'WWW/2023/cam-a-large-language-model-based-creative-analogy-mining-framework': ['value', 'GPT-3 text-davinci-001 "with temperature=0"'],
  'IMC/2024/diffaudit-auditing-privacy-practices-of-online-services-for-children-and-adolesc': ['value', 'sweeps GPT-4 temperature 0.7/0.8 and majority-votes across them'],
  'CCS/2024/accurate-and-efficient-recurring-vulnerability-detection-for-iot-firmware': ['value', '"temperature parameter set to zero"'],
  'CCS/2024/prophetfuzz-fully-automated-prediction-and-fuzzing-of-high-risk-option-combinati': ['value', '"0.7 for tasks needing diverse outcomes"'],
  'IEEE-SP/2024/smartinv-multimodal-learning-for-smart-contract-invariant-inference': ['value', '"we set temperature, top-k, top-p, and repeated penalty to 0"'],
  'PETS/2024/connecting-the-dots-tracing-data-endpoints-in-iot-devices': ['other', 'a word cloud containing "local temperature station"'],
  'USENIX/2024/knowphish-large-language-models-meet-multimodal-knowledge-graphs-for-enhancing-r': ['other', 'only in a cited paper’s title, "KL-divergence guided temperature sampling"'],
  'USENIX/2024/less-defined-knowledge-and-more-true-alarms-reference-based-phishing-detection-w': ['value', '"with the temperature set to 0"'],
  'USENIX/2024/llm-fuzzer-scaling-assessment-of-large-language-model-jailbreaks': ['value', '"we set the temperature parameter to 1.0"'],
  'USENIX/2024/malla-demystifying-real-world-large-language-model-integrated-malicious-services': ['discussed', '"21 different temperature settings" is a property of the services measured, not of the authors’ own labelling'],
  'USENIX/2024/when-threads-meet-interrupts-effective-static-detection-of-interrupt-based-deadl': ['value', '"we set the temperature as 0.1 and set Top-p as 0.2"'],
  'WWW/2024/human-vs-chatgpt-effect-of-data-annotation-in-interpretable-crisis-related-micro': ['value', '"We set the temperature parameter to 0.2"'],
  'CCS/2025/zvdetector-state-guided-vulnerability-detection-system-for-zigbee-devices': ['other', 'the Tuya smart bulb’s "Color Temperature" attribute'],
  'IEEE-SP/2025/gptracker-a-large-scale-measurement-of-misused-gpts': ['other', 'a weather API payload `{"temperature": 15℃}` in a figure'],
  'IMC/2025/learning-as-to-organization-mappings-with-borges': ['value', 'GPT-4o-mini "with a temperature set to 0 and a Top P"'],
  'NDSS/2025/generating-api-parameter-security-rules-with-llm-for-api-misuse-detection': ['value', '"the temperature of LLM is set to 0, 1, 0, and 1 respectively"'],
  'USENIX/2025/great-now-write-an-article-about-that-the-crescendo-multi-turn-llm-jailbreak-att': ['value', '"we set the temperature for all models to 0.5"'],
  'USENIX/2025/torchlight-shedding-light-on-real-world-attacks-on-cloudless-iot-devices-conceal': ['value', '"a temperature setting of 0.95" and "0.6"'],
  'USENIX/2025/mbfuzzer-a-multi-party-protocol-fuzzer-for-mqtt-brokers': ['default', '"with its default temperature parameter"; the other hit is a softmax T'],
  'WWW/2025/thematic-lm-a-llm-based-multi-agent-system-for-large-scale-thematic-analysis': ['value', '"temperature and top_p are set at the default value of one"'],
  'NDSS/2026/firmagent-leveraging-fuzzing-to-assist-llm-agents-with-iot-firmware-vulnerability-discovery': ['value', '"temperature of 0.7"'],
  'NDSS/2026/bsfuzzer-context-aware-semantic-fuzzing-for-ble-logic-flaw-detection': ['value', '"we set the LLM temperature to 0 when parsing structured information"'],
  'NDSS/2026/a-causal-perspective-for-enhancing-jailbreak-attack-and-defense': ['value', '"a generation temperature of 0.7"'],
  'PETS/2026/alexa-do-not-say-that-in-front-of-my-boss-a-cross-cultural-comparison-of-user-an': ['value', '"applied a temperature of 0.3 for all models, and ran both tasks three times"'],
  'NDSS/2026/iotbec-an-accurate-and-efficient-recurring-vulnerability-detection-framework-for-black-box-iot-devices': ['value', '"the temperature parameter set to 0.1"'],
  'PETS/2026/audagent-automated-auditing-of-privacy-policy-compliance-in-ai-agents': ['value', '"Gemini with the highest temperature (2.0)"'],
  'WWW/2025/supernotes-driving-consensus-in-crowd-sourced-fact-checking': ['value', '"set the temperature = 0.95, top_p = 0.8"'],
  'NDSS/2026/benchmarking-and-understanding-safety-risks-in-ai-character-platforms': ['default', '"we use the default settings", no number'],
  'USENIX/2026/pangolin-fuzzing-multilingual-iot-firmware-with-llm-driven-code-analysis': ['value', '"configure the temperature of DeepSeek-V3 with 0.7"'],
  'USENIX/2026/sok-philter-uncovering-security-and-functional-gaps-in-ai-based-phishing-website': ['value', '"default API settings (temperature = 1 and top-p = 1 for o4-mini...)"'],
  'WWW/2026/adaptive-location-hierarchy-learning-for-long-tailed-mobility-prediction': ['other', 'the Gumbel-Softmax τ'],
  'WWW/2026/inference-cost-attacks-for-retrieval-augmented-large-language-models': ['value', '"(temperature set to 0)"'],
  'USENIX/2026/autonomy-comes-with-costs-detecting-denial-of-service-vulnerabilities-caused-by': ['value', '"the temperature fixed at 0 to ensure deterministic outputs"'],
  'WWW/2025/llmcloudhunter-harnessing-llms-for-automated-extraction-of-detection-rules-from': ['value', '"we use a low temperature of zero"'],
  'PETS/2026/websp-eval-evaluating-web-agents-on-website-security-and-privacy-tasks': ['value', '"a temperature of 1.0 (recommended/default setting for reasoning models)"'],
  'WWW/2026/semfuzz-a-semantics-aware-fuzzing-framework-for-network-protocol-implementations': ['value', '"top_p and temperature were set to 0.1 and 0.5"'],
  'WWW/2026/arulecon-agentic-security-rule-conversion': ['value', '"configured with a temperature of 0.3"'],
  'WWW/2026/longitudinal-trends-in-global-climate-change-discourse-on-facebook': ['other', '"temperature deviations" and "temperature anomalies" — the climate variable'],
  'USENIX/2026/semantics-over-syntax-uncovering-pre-authentication-5g-baseband-vulnerabilities': ['value', '"configured with temperature=0.0"'],
  'CCS/2025/asymmetry-vulnerability-and-physical-attacks-on-online-map-construction-for-auto': ['other', 'the "color temperature" of a light source'],
  'CCS/2025/automatically-detecting-online-deceptive-patterns': ['other', 'only in a cited title, "The effect of sampling temperature on problem solving"'],
  'IMC/2025/do-spammers-dream-of-electric-sheep-characterizing-the-prevalence-of-llm-generat': ['value', '"the default temperature of 1" and "a generation temperature of 0 for rewriting"'],
  'PETS/2025/sheeps-clothing-wolfish-intent-automated-detection-and-evaluation-of-problematic': ['value', '"we set temperature to 0"'],
  'USENIX/2025/selfdefend-llms-can-defend-themselves-against-jailbreaking-in-a-practical-manner': ['value', '"max_tokens, temperature, n, and top_p to 128, 0, ..."'],
  'NDSS/2026/indicator-of-benignity-an-industry-view-of-false-positive-in-malicious-domain-detection-and-its-mitigation': ['value', '"temperature = 0.1, top p = 0.95, frequency penalty = 0"'],
  'USENIX/2026/sirens-whisper-inaudible-near-ultrasonic-jailbreaks-of-speech-driven-llms': ['other', 'the simulated-annealing temperature T'],
  'WWW/2026/moral-outrage-shapes-commitments-beyond-attention-multimodal-moral-emotions-on-y': ['other', 'a topic label, "Fluctuating Weather & Temperatures"'],
  'PETS/2026/disclosure-divergence-measuring-privacy-policy-and-data-safety-misalignment-at-s': ['value', '"deterministic decoding using temperature = 0.0, top-p = 1.0" (split across columns)'],
  'WWW/2026/when-ads-become-profiles-uncovering-the-invisible-risk-of-web-advertising-at-sca': ['value', '"we set temperature to 0.0" and "a temperature of 0.3"'],
  'PETS/2026/chatbot-confessions-large-scale-analysis-of-private-data-disclosure-in-shared-ai': ['value', '"We configured temperature=0.1"'],
  'USENIX/2026/chameleon-channels-measuring-youtube-accounts-repurposed-for-deception-and-profi': ['value', '"with temperature zero and top p of one"'],
  'CCS/2025/you-cant-steal-nothing-mitigating-prompt-leakages-in-llms-via-system-vectors': ['value', '"We set the “temperature” parameter in the OpenAI API to 0.6"'],
  'WWW/2026/cfvbench-a-comprehensive-video-benchmark-for-fine-grained-multimodal-retrieval-a': ['value', '"we set the temperature to 0.1 and top-p to 1"'],
  'PETS/2026/ai-in-the-loop-privacy-preserving-real-time-scam-detection-and-conversational-sc': ['value', '"use temperature = 0.95 and top-p = 0.9"'],
  'WWW/2026/visual-content-moderation-in-messaging-systems': ['other', 'a temperature-scaled softmax'],
};
 
 
// (3) PROMPT_AUDIT — the 31 of the 175 whose full text contains a sentence
//     pointing at a reproduced prompt. Read in context, because "the exact
//     prompt mentioned in the policy" and "we combine the above 5 parts to
//     form a complete prompt" both match the regex and neither publishes
//     anything. One paper matches because it says it is DELIBERATELY not
//     publishing its prompts, which is the opposite of what the probe is for
//     and is worth its own verdict.
//
//       published — a sentence says the prompt (or prompt template) is
//                   reproduced in an appendix, figure, listing or artifact
//       declined  — the paper says it is withholding the prompts
//       other     — the sentence is about something else
//
//     This count is a LOWER BOUND in both directions from the probe: a paper
//     that prints its prompt in an appendix figure without a sentence pointing
//     at it is invisible here.
export const PROMPT_AUDIT = {
  'IMC/2024/diffaudit-auditing-privacy-practices-of-online-services-for-children-and-adolesc': ['published', '"See Appendix B for details about the model prompt"'],
  'CCS/2025/zvdetector-state-guided-vulnerability-detection-system-for-zigbee-devices': ['published', '"using the LLM prompt shown in Figure"'],
  'NDSS/2025/generating-api-parameter-security-rules-with-llm-for-api-misuse-detection': ['published', '"All the complete prompts used by GPTAid"'],
  'NDSS/2026/bsfuzzer-context-aware-semantic-fuzzing-for-ble-logic-flaw-detection': ['published', '"see the field validation prompt in Appendix A.7"'],
  'PETS/2026/audagent-automated-auditing-of-privacy-policy-compliance-in-ai-agents': ['published', '"using the prompts in Appendix C.1"'],
  'NDSS/2026/benchmarking-and-understanding-safety-risks-in-ai-character-platforms': ['published', '"the complete prompt templates ... provided in the Appendix"'],
  'WWW/2025/llmcloudhunter-harnessing-llms-for-automated-extraction-of-detection-rules-from': ['published', '"transcription prompts are provided in Appendix F"'],
  'WWW/2025/harmful-terms-and-where-to-find-them-measuring-and-modeling-unfavorable-financia': ['published', '"(see prompts in Appendix C)"'],
  'PETS/2026/word-level-annotation-of-gdpr-transparency-compliance-in-privacy-policies-using': ['published', '"The full prompts used for classification, annotation, and self-correction are provided in Appendix"'],
  'WWW/2026/arulecon-agentic-security-rule-conversion': ['published', '"detailed prompts in Appendix B"'],
  'USENIX/2026/semantics-over-syntax-uncovering-pre-authentication-5g-baseband-vulnerabilities': ['published', '"The full prompt template"'],
  'CCS/2025/asymmetry-vulnerability-and-physical-attacks-on-online-map-construction-for-auto': ['published', '"Full prompt details are available in Appendix Fig. 13"'],
  'PETS/2026/chatbot-confessions-large-scale-analysis-of-private-data-disclosure-in-shared-ai': ['published', '"The complete prompt construction details, including the six-layer architecture"'],
  'PETS/2026/ai-in-the-loop-privacy-preserving-real-time-scam-detection-and-conversational-sc': ['published', '"prompt templates are provided in the Appendices"'],
  'NDSS/2025/hidden-and-lost-control-on-security-design-risks-in-iot-user-facing-matter-controller': ['published', '"Detailed prompt templates are provided in Appendix A"'],
  'USENIX/2025/unsafe-llm-based-search-quantitative-analysis-and-mitigation-of-safety-risks-in': ['published', '"The refinement prompt is shown in Appendix C"'],
  'WWW/2025/fact-based-counter-narrative-generation-to-combat-hate-speech': ['published', '"The detailed prompt is provided in Appendix A.3"'],
  'WWW/2025/50-shades-of-deceptive-patterns-a-unified-taxonomy-multimodal-detection-and-secu': ['published', '"We provide the initial prompt and the final best prompt in Appendix C and Appendix D"'],
  'NDSS/2026/cache-me-catch-you-cache-related-security-threats-in-llm-serving-frameworks': ['published', '"see Appendix H for example prompts"'],
  'NDSS/2026/protocolguard-detecting-protocol-non-compliance-bugs-via-llm-guided-static-analysis-and-dynamic-verification': ['published', '"the complete prompt template is provided in the Appendix A"'],
  'NDSS/2026/an-llm-driven-fuzzing-framework-for-detecting-logic-instruction-bugs-in-plcs': ['published', '"prompt shown in Figure 5"'],
  'USENIX/2026/a-large-scale-study-of-personalized-phishing-using-large-language-models': ['published', '"The full prompt for this step is shown"'],
  'USENIX/2026/stayin-alive-how-global-stolen-data-markets-thrive-on-telegram': ['published', '"The full prompt is available in Appendix A.2"'],
  'IMC/2025/fishing-for-smishing-understanding-sms-phishing-infrastructure-and-strategies-by': ['published', '"Our prompt is available in Appendix D.2"'],
  'IMC/2025/analyzing-compliance-and-complications-of-integrating-internationalized-x-509-ce': ['published', '"The prompt templates are detailed in Appendix C"'],
  'PETS/2026/on-the-suitability-of-llm-driven-agents-for-dark-pattern-audits': ['published', '"The full prompting configurations are illustrated in \u00a7A.3, Figures 2 to 4"'],
  'CCS/2024/moderator-moderating-text-to-image-diffusion-models-through-fine-grained-context': ['other', '"moderating the exact prompt mentioned in the policy" — about the systems studied'],
  'USENIX/2025/when-llms-go-online-the-emerging-threat-of-web-enabled-llms': ['declined', '"disclosing the exact prompts used for these attacks could enable malicious actors to replicate them"'],
  'NDSS/2026/beyond-jailbreak-unveiling-risks-in-llm-applications-arising-from-blurred-capability-boundaries': ['other', '"Prompt templates are central to eliciting" — describes the platforms measured'],
  'WWW/2026/towards-multi-label-text-interpretation-with-chain-of-thought-prompting-and-cont': ['other', '"contextual information from the complete prompt" — an ablation row'],
  'IEEE-SP/2025/my-model-is-malware-to-you-transforming-ai-models-into-malware-by-abusing-tensor': ['other', '"We combine the above 5 parts to form a complete prompt" — describes the attack'],
};
 
 
// (4) GROUPED_SPLIT_AUDIT — of the 300 web-platform papers whose classification
//     carries `held-out-test-set` or `cross-validation`, the 10 whose full text
//     contains any language about splitting or evaluating across a GROUP
//     (website, domain, page, publisher) rather than across rows. Read in
//     context, because "cross-domain" means a topic category in one paper and a
//     different online platform in another, and one paper matches only because
//     it says it did NOT do this.
//
//       grouped — the train/test boundary really is drawn at a group, or the
//                 evaluation is explicitly on groups not seen in training
//       other   — a different sense of the phrase, or a declared limitation
//
//     A LOWER BOUND: a paper that split by site and did not write a sentence
//     about it is invisible, and so is one that used a phrasing outside the
//     regex. The regex is printed in the report and in the provenance page.
export const GROUPED_SPLIT_AUDIT = {
  'WWW/2012/echoes-of-power-language-effects-and-power-differences-in-social-interaction': ['grouped', '"a means to assess cross-domain generalization" — trained on one discussion corpus, tested on another'],
  'PETS/2016/blogs-twitter-feeds-and-reddit-comments-cross-domain-authorship-attribution': ['grouped', 'the whole paper is the cross-platform setting: train on one platform, test on another'],
  'PETS/2019/var-cnn-a-data-efficient-website-fingerprinting-attack-based-on-deep-learning': ['grouped', '"there is no overlap between unmonitored training and testing sites"'],
  'CCS/2023/cookiegraph-understanding-and-detecting-first-party-tracking-cookies': ['grouped', '"We ensure that there is no overlap in the websites used for training and testing in each fold"'],
  'USENIX/2023/automated-cookie-notice-analysis-and-enforcement': ['grouped', '"on previously unseen domains"'],
  'NDSS/2026/lightening-the-load-a-cluster-based-framework-for-a-lower-overhead-provable-website-fingerprinting-defense': ['grouped', '"out-of-training settings with unseen websites"'],
  'IEEE-SP/2022/device-fingerprinting-with-peripheral-timestamps': ['grouped', '"an 80/20 grouped split between train and test sets, i.e., train/test set devices are mutually exclusive" — the group is a device, not a site'],
  'IMC/2023/evolving-bots-the-new-generation-of-comment-bots-and-their-underlying-scam-campa': ['other', '"the partitions of the graph in Figure 7 by domains" is a figure, not a split'],
  'NDSS/2026/loki-proactively-discovering-online-scams-by-mining-toxic-search-queries': ['other', '"Cross-Domain Validation Across Major Scam Categories" — domain means topic here'],
  'IEEE-SP/2012/on-the-feasibility-of-internet-scale-author-identification': ['other', '"we have not tested it in a cross-domain setting" — a declared limitation'],
};
 
// The regex behind (4). Exported so the report and the provenance page print
// the same one, and so widening it cannot silently change a published number.
export const GROUPED_SPLIT_RE =
  /(group(ed)?[- ]?(k[- ]?fold|shuffle ?split|split|cross[- ]validation)|GroupKFold|GroupShuffleSplit|(split|partition|divid\w+|separat\w+)[^.]{0,60}\b(by|per|at the|across) (site|domain|website|host|publisher|organi[sz]ation|page)s?\b|(site|domain|website|publisher|page)[- ](level|wise|disjoint)[- ]?(split|partition|evaluation|cross)|disjoint (sets? of )?(sites|domains|websites|pages)|(no|without) overlap[^.]{0,70}(sites|domains|websites|pages)|unseen (sites|domains|websites)|held[- ]out (sites|domains|websites)|cross[- ](site|domain|website)[- ]?(evaluation|validation|generalis|generaliz|setting)|leave[- ]one[- ](site|domain|website)[- ]out)/i;
 
 
// (5) RESIDUE_SAMPLE — a systematic 30-string sample of the ground-truth fold's
//     unmapped residue, hand-classified into the family each string WOULD have
//     joined if the fold reached it. Added 2026-09-11 after a review caught the
//     page asserting, with three invented examples, that the residue was
//     dominated by by-construction setups. It is not: all three of those
//     strings in fact fold correctly, and two of them are `gt_fold.mjs`'s own
//     self-test fixtures. This map is what replaced that claim.
//
//     Drawn as every 73rd line of `node scripts/gt_fold.mjs --residue`, which
//     is deterministic for a given extractions.jsonl. The strings are stored
//     literally rather than re-sampled, so the sample survives a corpus change;
//     the self-check below warns for any that have left the residue.
//
//     `unclear` means a reasonable person could not place the string from the
//     string alone — not that it belongs nowhere.
export const RESIDUE_SAMPLE = {
  'server-side recurring-device cookies': 'unclear',
  "authors' downloaded fraudulent websites": 'by-construction',
  'noaa city snowfall observations': 'existing-dataset',
  'authentication-token testing by submitting requests with cookie subsets': 'execution-or-oracle',
  'known dodo-telstra, bell-dery, and libyan disruption events': 'existing-dataset',
  'opencorporates company directory': 'third-party-service',
  'device settings': 'unclear',
  '774 known dark jargons and corresponding hypernyms': 'existing-dataset',
  'five years of nanog operator mailing-list threads': 'existing-dataset',
  'media bias/fact check ratings for 1,540 media-source domains': 'third-party-service',
  'paired acceleration and speech spectrograms': 'by-construction',
  'exact addon-id search results containing xbmc or kodi': 'rule-or-tool-output',
  'webxray same-entity list': 'curated-list',
  '928 registry cookies plus 10,000 randomly sampled cookies': 'unclear',
  "http status 418, requests containing 'wp-' or 'access.log'": 'rule-or-tool-output',
  'penn discourse treebank, wordnet, and causal constructions': 'existing-dataset',
  'validation against 41 public resolvers': 'execution-or-oracle',
  'python-captured key press and release events': 'by-construction',
  'service names and identified parent owners': 'unclear',
  'labeled privacy-notice sentences and privacy-policy notices': 'existing-dataset',
  'curated threat-actor and malware keyword lists': 'curated-list',
  'online industry-engine comparison': 'third-party-service',
  'ipinfo long-term geolocation trends': 'third-party-service',
  'open-ended responses from study participants': 'human-annotation',
  'participant judgments': 'human-annotation',
  "research assistants' policy-based labels": 'human-annotation',
  "installed anti-spyware applications' scan outputs": 'rule-or-tool-output',
  'real training trajectories': 'by-construction',
  'extracted video and subtitle keypoints': 'by-construction',
  'corresponding clean outputs of shadow models': 'by-construction',
};
 
if (import.meta.url === `file://${process.argv[1]}`) {
  const { loadExtractions, dataRoot, pct } = await import('./lib.mjs');
  const fs = await import('node:fs');
  const path = await import('node:path');
  const rows = loadExtractions();
  const key = (p) => `${p.venue}/${p.year}/${p.slug}`;
  let bad = 0;
 
  const PROBE_RE =
    /(LLM|GPT|large language model)[^•]{0,60}(annotat|label|classif|code|judge)|LLM-as-a-judge|prompt(ing)? (strategy|engineering)[^•]{0,40}(annotat|label)/i;
  const hits = rows.filter((p) => PROBE_RE.test(`${p.title ?? ''} • ${p.summary ?? ''}`));
  for (const p of hits)
    if (!(key(p) in PROBE_AUDIT)) { console.log(`UNAUDITED probe hit: ${key(p)}`); bad += 1; }
  for (const k of Object.keys(PROBE_AUDIT))
    if (!hits.some((p) => key(p) === k)) { console.log(`STALE probe audit entry: ${k}`); bad += 1; }
  const verdicts = {};
  for (const [, [v]] of Object.entries(PROBE_AUDIT)) verdicts[v] = (verdicts[v] ?? 0) + 1;
  console.log(`probe candidates: ${hits.length}`);
  for (const [v, n] of Object.entries(verdicts).sort((a, b) => b[1] - a[1]))
    console.log(`  ${v.padEnd(12)} ${String(n).padStart(3)}  ${pct(n, hits.length)}`);
 
  const USED = new Set(['used', 'produced']);
  const LLM = rows.filter((p) => (p.classification ?? []).some((c) => c.method === 'llm' && USED.has(c.usedOrMentioned)));
  const norm = (s) => s.replace(/­/g, '').replace(/-\s*\n\s*/g, '').replace(/\s+/g, ' ');
  const tempHits = LLM.filter((p) =>
    /temperature/i.test(norm(fs.readFileSync(path.join(dataRoot(), 'fulltext', String(p.year), p.venue, p.slug, 'paper.cols.txt'), 'utf8')))
  );
  for (const p of tempHits)
    if (!(key(p) in TEMPERATURE_AUDIT)) { console.log(`UNAUDITED temperature hit: ${key(p)}`); bad += 1; }
  for (const k of Object.keys(TEMPERATURE_AUDIT))
    if (!tempHits.some((p) => key(p) === k)) { console.log(`STALE temperature audit entry: ${k}`); bad += 1; }
  const tv = {};
  for (const [, [v]] of Object.entries(TEMPERATURE_AUDIT)) tv[v] = (tv[v] ?? 0) + 1;
  console.log(`\nLLM papers (used/produced): ${LLM.length}; full text containing "temperature": ${tempHits.length}`);
  for (const [v, n] of Object.entries(tv).sort((a, b) => b[1] - a[1]))
    console.log(`  ${v.padEnd(12)} ${String(n).padStart(3)}  ${pct(n, LLM.length)} of the ${LLM.length}`);
 
  const PROMPT_RE =
    /(full|exact|complete|verbatim) prompts?|prompts? (are|is|used are)? ?(listed|shown|given|provided|included|available|reproduced) in (the )?(appendix|artifact|repository|supplementary|table|figure|listing)|prompts? in (the )?appendix|see appendix[^.]{0,40}prompt|prompt templates? (in|are) /i;
  const promptHits = LLM.filter((p) =>
    PROMPT_RE.test(norm(fs.readFileSync(path.join(dataRoot(), 'fulltext', String(p.year), p.venue, p.slug, 'paper.cols.txt'), 'utf8')))
  );
  for (const p of promptHits)
    if (!(key(p) in PROMPT_AUDIT)) { console.log(`UNAUDITED prompt hit: ${key(p)}`); bad += 1; }
  for (const k of Object.keys(PROMPT_AUDIT))
    if (!promptHits.some((p) => key(p) === k)) { console.log(`STALE prompt audit entry: ${k}`); bad += 1; }
  const pv = {};
  for (const [, [v]] of Object.entries(PROMPT_AUDIT)) pv[v] = (pv[v] ?? 0) + 1;
  console.log(`\nfull text says a prompt is reproduced: ${promptHits.length} candidates`);
  for (const [v, n] of Object.entries(pv).sort((a, b) => b[1] - a[1]))
    console.log(`  ${v.padEnd(12)} ${String(n).padStart(3)}  ${pct(n, LLM.length)} of the ${LLM.length}`);
 
  const HOW = rows.filter((p) => p.platforms.includes('web') &&
    (p.classification ?? []).some((c) => c.validation === 'held-out-test-set' || c.validation === 'cross-validation'));
  const gHits = HOW.filter((p) =>
    GROUPED_SPLIT_RE.test(norm(fs.readFileSync(path.join(dataRoot(), 'fulltext', String(p.year), p.venue, p.slug, 'paper.cols.txt'), 'utf8')))
  );
  for (const p of gHits)
    if (!(key(p) in GROUPED_SPLIT_AUDIT)) { console.log(`UNAUDITED grouped-split hit: ${key(p)}`); bad += 1; }
  for (const k of Object.keys(GROUPED_SPLIT_AUDIT))
    if (!gHits.some((p) => key(p) === k)) { console.log(`STALE grouped-split audit entry: ${k}`); bad += 1; }
  const gv = {};
  for (const [, [v]] of Object.entries(GROUPED_SPLIT_AUDIT)) gv[v] = (gv[v] ?? 0) + 1;
  console.log(`\nweb papers with a held-out test set or cross-validation: ${HOW.length}; grouped-split candidates: ${gHits.length}`);
  for (const [v, n] of Object.entries(gv).sort((a, b) => b[1] - a[1]))
    console.log(`  ${v.padEnd(12)} ${String(n).padStart(3)}  ${pct(n, HOW.length)} of the ${HOW.length}`);
 
  const { foldGroundTruth } = await import('./gt_fold.mjs');
  const residueStrings = new Set();
  for (const p of rows)
    for (const c of p.classification ?? [])
      if (c.groundTruthSource && foldGroundTruth(c.groundTruthSource) === null)
        residueStrings.add(c.groundTruthSource.toLowerCase());
  const rv = {};
  let gone = 0;
  for (const [str, v] of Object.entries(RESIDUE_SAMPLE)) {
    rv[v] = (rv[v] ?? 0) + 1;
    if (!residueStrings.has(str)) { console.log(`SAMPLED STRING NO LONGER IN RESIDUE: ${str}`); gone += 1; }
  }
  console.log(`\nresidue sample: ${Object.keys(RESIDUE_SAMPLE).length} strings hand-classified into the family they would have joined`);
  for (const [v, n] of Object.entries(rv).sort((a, b) => b[1] - a[1])) console.log(`  ${v.padEnd(22)} ${String(n).padStart(3)}`);
  if (gone) { console.log(`${gone} sampled string(s) have left the residue — the fold changed; re-sample.`); bad += 1; }
 
  if (bad) { console.log(`\n${bad} PROBLEM(S) — the audit maps and the probes disagree.`); process.exit(1); }
  console.log('\nboth audits cover their probe exactly.');
}