Measure The Web

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== The Publication Corpus Behind These Pages ======

Many pages on this site carry a section headed //Use in Publications// whose figures come from one source: a structured extraction over the full text of **5,859 papers** from seven security, privacy and measurement venues, 2010–2026. This page is that source's methodology. It exists so that every figure on this site can be discounted correctly — you cannot judge "1,228 of the 3,908 papers that took a measurement from a vantage point — 31.4% — say where they measured from" without knowing which papers were eligible to be counted, which were never eligible, and how the 31.4% was produced.

It is not a description of a public dataset. **The corpus is not published.** See [[#How to check a figure yourself]] for what that means for you.

<WRAP important>
**One rule governs every number here and on every page that draws on this corpus: name the denominator before the numerator.** The corpus is seven //broad// venues, not a web-measurement corpus. Only 1,120 of the 5,859 papers ran a crawl; only 1,762 ran statistical inference; only 402 assessed compliance with a law. A percentage "of 5,859 papers" almost always answers a question nobody asked.
</WRAP>

===== The provenance namespace =====

Every page whose figures come from this corpus has a second, published page holding the working notes behind it: each query with its population and denominator, the report script and its unedited output, the folds and their unmapped residue, the quotes that were spot-checked, and what could not be established.

**Those pages live under ''provenance:'' and mirror the content page's id exactly.** A content page's figures are auditable at ''provenance:'' followed by its own id — no lookup table, no suffix, no guessing:

^ Content page ^ Its working notes ^
| [[design:crawling_location]] | [[provenance:design:crawling_location]] |
| [[design:ip_classification]] | [[provenance:design:ip_classification]] |
| [[design:website_classification]] | [[provenance:design:website_classification]] |
| [[privacy:fingerprinting]] | [[provenance:privacy:fingerprinting]] |
| [[privacy:javascript]] | [[provenance:privacy:javascript]] |
| [[programming:crawler]] | [[provenance:programming:crawler]] |

This page is the root of that namespace. Corpus-level caveats — everything below — are stated here once, and each ''provenance:'' page carries only what is specific to its own page.

===== 1. Scope: seven venues, 2010–2026 =====

^ Venue ^ Venue-years ^ First ^ Last ^ Records in the venue index ^
| ACM CCS | 16 | 2010 | 2025 | 3,381 |
| IEEE S&P | 17 | 2010 | 2026 | 1,837 |
| ACM IMC | 16 | 2010 | 2025 | 862 |
| NDSS | 17 | 2010 | 2026 | 1,577 |
| PETS / PoPETs | 17 | 2010 | 2026 | 1,253 |
| USENIX Security | 17 | 2010 | 2026 | 3,012 |
| TheWebConf (WWW) | 17 | 2010 | 2026 | 4,942 |

**117 venue-years** of a possible 119. CCS 2026 and IMC 2026 are absent because those conferences are held in October and November and had not taken place when the index was built.

==== What is absent, and what that means ====

**EuroS&P, ACSAC, RAID, AsiaCCS, CHI and SOUPS are not in the corpus at all.** Neither are PAM, TMA, ANRW, SIGCOMM or ACM CCR, nor the machine-learning venues where a growing share of privacy work now appears. The consequences are not evenly distributed:

  * **Usable privacy and HCI are missing by construction.** SOUPS and CHI are where most user-study methodology for privacy is published. This site's pages on user studies, consent interfaces and qualitative coding are therefore built on whatever those seven venues happened to publish, which is a biased and much smaller sample of that literature. Treat any claim here about user-study practice as a claim about security venues' user studies.
  * **Internet measurement outside IMC is missing.** For network-layer topics — IP geolocation, routing, anycast — PAM and TMA are core venues and their absence is a serious gap. [[design:ip_classification]] says so in its own limitations section.
  * **Every claim on this site is a claim about seven venues.** Not about "the literature", not about "the field". Where a page words it otherwise, that is an error worth reporting.

===== 2. The funnel =====

Each row is counted from the artefact that stage produces, not from a note about it.

^ Stage ^ Papers ^ Lost here ^ Is the loss random or systematic? ^
| Venue-year metadata records (117 venue-years, across 7 venues) | 16,864 | — | — |
| …carrying an abstract | 15,800 | −1,064 | **systematic**, in three unrelated blocks — below |
| Screened on title + abstract | 15,800 | 0 | none: screening covers every abstract |
| Selected as in scope | 6,103 | −9,697 | by design |
| PDF retrieved | 5,873 | −230 | **systematic**: paywall, or not yet published |
| Text extracted and prepared | 5,873 | 0 | none |
| **Structured extraction record** | **5,859** | −14 | **systematic**: an IEEE S&P rate-limit tail |

96.0% of selected papers made it to an extraction record; 34.7% of all indexed records did.

==== Where each loss comes from ====

**The 1,064 records with no abstract are three unrelated problems sharing a row, and only two of them go away with time.**

^ Era ^ Records ^ Why ^ Will waiting fix it? ^
| 2010–2019 | 443 | **For 426 of them there is no abstract to fetch.** NDSS archive pages for 2010–2018 link straight to the PDF and publish no abstract, and NDSS has no DBLP DOIs, so neither retrieval path applies (260 papers). Pre-PoPETs PETS workshop schedule pages likewise publish none (126). CCS 2011's DOI-less poster entries sit under a DOI-less parent proceedings (40). The remaining 17 are scattered singles across PETS, USENIX and NDSS, mostly front matter. | **No** for the 426; the 17 are recoverable |
| 2020–2024 | 58 | Scattered — mostly one pass where the OpenAlex daily quota ran out mid-run. | Yes, with a re-run |
| 2025–2026 | 563 | OpenAlex has the DOI before it has the abstract: WWW 2026 (324) and IEEE S&P 2026 (194) dominate. | Yes, in a few months |

This matters more than it looks, because **screening runs on abstracts. A paper with no abstract is never screened, so it can never be selected, so it can never appear in any figure on this site.** It is excluded for a reason that has nothing to do with its relevance. All 1,064 are present in the index with their titles, so they are countable rather than silently absent — which is how the table above can exist at all.

**The 230 selected papers with no PDF** are concentrated, not spread:

^ Venue-year ^ Selected but never retrieved ^
| USENIX 2026 | 104 |
| IMC 2012 | 22 |
| IMC 2013 | 12 |
| WWW 2024 | 8 |
| IMC 2020 | 6 |
| CCS 2018 | 6 |
| everything else | 72, spread over 31 more venue-years |

USENIX Security 2026 alone is 45.2% of the loss, for the simple reason that the conference has not happened yet and its papers are not online. Per venue, the share of selected papers with no PDF is 0% for IEEE S&P, NDSS and PETS, 2.5% for WWW, 4.4% for CCS, 7.2% for USENIX and 7.7% for IMC.

**The 14 papers with prepared text but no extraction record.** One is a USENIX 2017 slide deck excluded deliberately; the other 13 are IEEE S&P papers — eleven from 2024, one from 2023, one from 2021 — whose extraction requests ended on ''520 status code (no body)'' and were never retried. Across the whole run 559 papers failed at least once and 546 were recovered by a retry; these 13 are the tail nobody went back for. The effect is a mild under-representation of IEEE S&P 2024 in every figure on this site, worth roughly 1.7% of that venue.

==== What "5,859 papers" actually contains ====

Two structural properties of the funnel that nothing above shows, both of which change how a silence figure should be read.

  * **138 of the 5,859 records are posters**, 96 from CCS and 42 from IMC, identifiable by their own slug and title. The venue index does not filter front matter and posters out, and screening did not reject them. 89 of them sit inside the 3,908-paper ''measuredFrom'' population and 15 inside the 1,120-paper ''crawled'' one. A poster is one or two pages with its methodology compressed out, so it is systematically more likely to be scored as "did not state", and it biases exactly the reporting-gap figures this site is built on — upward. 251 records are four pages or fewer. If a silence rate matters to you, exclude short records and see whether it moves.
  * **Three papers are in the corpus twice.** The consistency check on duplicates is keyed on ''(venue, year, slug)'' and returns zero, but that key cannot see the duplication the venue index warns about: //Detecting and Characterizing Social Spam Campaigns// is indexed under both CCS 2010 and IMC 2010, //UIScope// under both NDSS 2020 and NDSS 2021, and //DRAWN APART// twice within NDSS 2022, once under an ''auto-draft-242'' slug. Keyed on the normalised title instead, the corpus has 3 extra records. Three in 5,859 changes nothing arithmetically; it matters because "zero duplicates" was being reported by a check that could not have found these.

==== The selection rule, verbatim ====

Screening is itself an LLM pass — ''gpt-5.6-luna'', the same model as the extraction, last run on 2026-08-07. It produces one label record per abstract with seven fields, and a paper is selected if **any** of them is set:

<code>
--any securityMeasurement,privacyMeasurement,manualAnalysis,domains,darkPatterns,laws,crawlers
</code>

The instruction to the screening model was explicitly over-inclusive — verbatim from ''scripts/build_selection_batch.mjs'': //"Include a paper if it plausibly matches the criteria — be deliberately over-inclusive, since a later pass reads the full text and a wrongly excluded paper is never seen again."// The rule was re-applied for this page and checked both ways: **every one of the 5,873 retrieved papers falls inside it, and no directory exists outside it.**

What has **never** been measured is whether the screening decisions are //correct//. There is no human-screened control set to score them against, and the labels are model output. A relevant paper wrongly dropped at screening leaves no trace anywhere in this pipeline.

===== 3. IEEE S&P: the worked example of a systematic loss =====

Until 10 August 2026, IEEE S&P looked like a partly-paywalled venue: **333 of its 780 selected papers were on disk**, and the corpus documentation carried a standing caveat that "IEEE S&P is 43% retrieved" and that cross-venue percentages were biased against it. Several pages on this site repeated that caveat.

It was not a paywall. **The Xplore PDF is served from ''stampPDF/getPDF.jsp''; ''stamp.jsp'' is a viewer shell that only embeds the file.** The fetcher scraped the viewer page, found no PDF link, and logged "No PDF found" — a message indistinguishable in the logs from "this paper is behind a paywall". With institutional access and the correct URL, all 447 remaining papers were retrievable.

^ ^ Before 2026-08-10 ^ Now ^
| IEEE S&P selected | 780 | 780 |
| …PDF retrieved | 333 (42.7%) | **780 (100%)** |
| …structured extraction record | — | 767 (98.3%) |

Two things are worth taking from this beyond the fixed number.

First, **the old caveat is wrong and must be deleted wherever it survives.** Any page still saying IEEE S&P is 43% retrieved is stale.

Second, and more usefully for your own crawl: a retrieval failure and an access failure produced the same log line, and it took months to notice. That is the general shape of a systematic loss — it does not announce itself, it looks like a property of the world.

===== 4. Provisional years, and nine venue-years that are holes =====

<WRAP important>
**Never present a per-year trend that ends in 2026 as if 2026 were a complete year.** Either cut the series at 2024, or label 2025–2026 provisional in the table itself.
</WRAP>

^ Year ^ Venue-years ^ Index records ^ With abstract ^ Selected ^ Extracted ^
| 2019 | 7 | 1,002 | 99.7% | 407 | 402 |
| 2020 | 7 | 953 | 99.9% | 414 | 404 |
| 2021 | 7 | 1,167 | 99.9% | 386 | 379 |
| 2022 | 7 | 1,348 | 99.9% | 556 | 546 |
| 2023 | 7 | 1,598 | 99.8% | 721 | 719 |
| 2024 | 7 | 1,901 | 97.3% | 712 | 690 |
| **2025** //(provisional)// | 7 | 2,032 | 98.0% | 774 | 770 |
| **2026** //(provisional)// | **5** | 1,989 | 73.7% | 521 | 415 |

2026 is thin for three compounding reasons, none of them relevance: CCS and IMC 2026 have not been held; USENIX Security 2026 has been selected but not published; and IEEE S&P 2026 (58 abstracts of 252 records) and WWW 2026 (630 of 954) are largely un-indexed. **Abstract-based selection cannot select a paper whose abstract is not yet indexed.** 2025 is nearly complete on this measure but is still the newest year and the least likely to have been checked by anyone.

==== Nine venue-years contribute nothing at all ====

The same mechanism produces silence, not thinness, in nine older venue-years. None of these 309 papers has an abstract in the index, so none was screened, so none is in the corpus:

^ Venue-year ^ Index records ^ With abstract ^ In the corpus ^
| NDSS 2010 | 24 | 0 | 0 |
| NDSS 2011 | 28 | 0 | 0 |
| NDSS 2016 | 60 | 0 | 0 |
| NDSS 2018 | 71 | 0 | 0 |
| PETS 2010 | 22 | 0 | 0 |
| PETS 2011 | 27 | 0 | 0 |
| PETS 2012 | 25 | 0 | 0 |
| PETS 2013 | 22 | 0 | 0 |
| PETS 2014 | 30 | 0 | 0 |

**PETS 2010–2014 is absent in its entirety**, and PETS is the most privacy-focused venue in the set. If you are looking here for early privacy-measurement methodology and finding little, that is why. Nothing here can be fixed by re-running the metadata build: the pre-PoPETs PETS workshop schedule pages and the NDSS archive pages for those years publish no abstracts at all, so there is nothing to fetch. Closing it means a different retrieval path — reading the abstract off the paper's own first page — which has not been implemented.

===== 5. How the structured extraction was made =====

^ ^ ^
| What | One structured record per paper, describing what the authors did |
| Input | The paper's whole text, as ''paper.cols.txt'' (5,852 records) or, where ''pdftotext'' produced a garbled encoding, OCR'd ''paper.mistral.md'' (7). ''paper.norm.txt'' only gates eligibility. |
| Method | **An LLM extraction against a fixed schema — a single pass, one request per paper** |
| Model | ''gpt-5.6-luna'', structured outputs, all 5,859 records |
| Key | ''(venue, year, slug)'' |
| Relation families | 13: ''tools'', ''population'', ''crawlConfig'', ''classification'', ''detection'', ''vantage'', ''temporal'', ''statistics'', ''humanAnnotation'', ''participants'', ''ethics'', ''artifacts'', ''legal'' |
| Evidence | **135,025 tuples carry a verbatim quote and the section it came from** |

<WRAP important>
**This is not human coding and must not be read as such.** There is no second coder, no adjudication step, and no inter-rater statistic — the concepts do not apply to a single generative pass. What it has instead is a checkable quote on every tuple, which is a weaker guarantee of a different kind: it lets you verify that a figure traces to a real sentence, not that a careful reader would have coded the sentence the same way.
</WRAP>

Three schema decisions shape every query you can run against it:

  * **Silence is a first-class value.** The schema's sentinels are ''not-stated'', ''none-mentioned'', ''not-applicable'', ''unclear'' and ''unknown'', and the model is asked to record them rather than omit the field. This is what lets a page report "of the 1,120 papers that ran a crawl, 53 say anything about robots.txt" instead of quietly dropping the other 1,067.
  * **''usedOrMentioned'' on anything citable** — ''used'' (84.3% of tool tuples), ''produced'' (9.4%), ''compared'' (5.0%), ''mentioned'' (1.2%), ''unclear'' (0.1%). A baseline the authors compared against reads as adoption if you skip this filter, and the tuples that are not ''used'' concentrate in exactly the names a page is about.
  * **The measured results live in ''detection[].prevalence''** — 26,316 of the 27,241 detection tuples (96.6%), across 5,572 papers, carry the paper's own figure. But ''prevalence'' is the model's summary of a result, not a quote from it. Grep the full text for any prevalence figure before publishing it, not just the attached quote.

Evidence quotes per family:

^ Family ^ Tuples with a quote ^ ^ Family ^ Tuples with a quote ^
| ''tools'' | 37,359 | | ''artifacts'' | 5,524 |
| ''detection'' | 27,241 | | ''ethics'' | 4,800 |
| ''population'' | 17,016 | | ''vantage'' | 4,685 |
| ''classification'' | 11,330 | | ''participants'' | 2,018 |
| ''statistics'' | 8,787 | | ''crawlConfig'' | 1,080 |
| ''temporal'' | 8,775 | | ''legal'' | 632 |
| ''humanAnnotation'' | 5,778 | | | |

===== 6. Reliability, measured and stated as a limit =====

==== Quote groundedness, re-measured on this corpus ====

Every quote is supposed to be verbatim, so every quote can be looked for in the text the extractor read. All **135,025** were, on 2026-08-12, by ''scripts/report_corpus.mjs --quotes'':

^ Verdict ^ Quotes ^ Share ^
| Present exactly, after collapsing whitespace and line-break hyphenation | 78,450 | 58.1% |
| An 8-word run of it survives in the source | 49,425 | 36.6% |
| Fragmented — every content word inside one window, order broken | 5,442 | 4.0% |
| **Not found** | **1,708** | **1.3%** |

Only the last row counts against the extraction, and even it is mostly not what it looks like. Five "not found" quotes were read against their papers by hand on 2026-08-12. **Every one is present in the paper**:

^ Paper ^ Extracted quote (fragment) ^ What actually happened ^
| ''CCS/2011/ispy-automatic-reconstruction-of-typed-input…'' | "the Scale Invariant Feature Transform, or SIFT [22]" | Column splice: //"Scale In-"// ends one column, //"variant Feature Transform"// starts the next |
| ''CCS/2011/how-to-tell-if-your-cloud-files-are-vulnerable…'' | "We experimented with drives from Fujitsu, Hitachi, HP, and Seagate." | Column splice mid-word: //"Fujitsu, Hi-"// / //"tachi, HP7, and Seagate"// |
| ''CCS/2012/aligot-cryptographic-function-identification…'' | "Among all tracing tools available in such environment, we chose Pin" | Column splice: //"such environ-"// / //"ment, we chose Pin"// |
| ''CCS/2010/detecting-and-characterizing-social-spam-campaigns'' | "We carry out the study on Facebook … We choose 8 regional networks …" | The extraction's own ellipsis: both halves are in the paper, the join is not |
| ''USENIX/2024/you-can-obfuscate-but-you-cannot-hide…'' | "For each destination, bots can gather their available bandwidth through tools like Pathneck [17]." | **A bug in the checker, not the extraction.** This paper's ''pdftotext'' output is a garbled font encoding; its text was OCR-repaired and the extractor read the repaired file. The first version of the script checked the garbled one |

Those five were the first entries in the script's not-found list, which is filled in sorted-key order and therefore starts in CCS 2010–2012 — the era where column-splice damage is most likely, so it is a favourable sample and not a random one. **Five more were drawn from 2025 and 2026**, where the not-found counts are largest, and read the same way:

^ Paper ^ What actually happened ^
| ''USENIX/2025/a-framework-for-abusability-analysis…'' | Present, spliced through the bibliography: //"rg/10. [other column] 5281/zenodo.14745290."// |
| ''CCS/2025/a-sea-of-cyber-threats…'' | Present, with a participant table interleaved mid-sentence: //"open coding methodology [95], refin-P10 35 Chief Mate … ing the codebook"// |
| ''USENIX/2025/addressing-the-address-books…'' | Present twice; the extraction dropped the "(90.5%)" parenthesis's surrounding clause |
| ''NDSS/2026/a-causal-perspective-for-enhancing-jailbreak…'' | Present; the extraction reconstructed a URL broken across a line, and ''causal-learn'' and ''py-why'' are both in the text |
| ''NDSS/2026/aliens-among-us…'' | **Not a quote at all.** The extraction wrote //"The paper does not report ethics review, affected-party notification, regulator contact…"// — a sentence about the paper's silence rather than a sentence from the paper |

That last one is a different failure mode from layout damage, and it is the only one worth worrying about. It is also rare: **12 of the 135,025 quotes** describe a paper's silence instead of quoting it, concentrated in ''ethics'' and ''classification''. A further 15.9% of the not-found quotes contain an ellipsis, meaning the extraction spliced two parts of a source sentence and marked the join — not verbatim, but not invented either.

A practical consequence for anyone checking a quote by hand: **normalise whitespace before concluding a quote is absent.** ''grep -F'' on a multi-word quote fails about half the time against this text because it still wraps mid-sentence; ''tr -s '[:space:]' ' ' | grep -iF'' finds them.

The failures are also spread thin rather than concentrated: 1,143 of 5,859 papers (19.5%) have at least one, and the ten worst papers hold only 82 of the 1,708. The rate is flat across all seventeen years, between 0.8% and 1.5% in every single one, which is what you would expect of a PDF-layout artefact and not of a model that got worse or better over time.

**Two caveats about this measurement.** It reproduces the dataset's own audit of the earlier corpus almost exactly — that audit reported 57.9% / 37.0% / 4.2% / 0.9% on the same four buckets — but the two implementations are independent and the "fragmented" test in particular is an approximation of the original, so the 4.0%/1.3% split between the last two rows should be read as one combined 5.3% that is mostly layout damage. And the first measurement of this ran the check against the wrong file for seven papers whose text had been OCR-repaired, which put one paper at the top of the not-found ranking with 29 phantom failures; the script now reads whichever rendering the extractor read.

==== Field stability: measured once, on the earlier corpus, never since ====

Two independent extraction runs over the same 100 papers were compared to see which fields survive being turned into a percentage. **That comparison was run against the 4,322-paper corpus and has not been repeated** — repeating it means paying for a second full extraction pass. Treat the figures below as the right order of magnitude, not as current measurements.

^ Run-to-run agreement ^ Fields ^ How to use them ^
| **90–100%** | has-authors'-own-link (100), ''crawlConfig'' fired (99), ''.statefulness'' (98), ''.interactionDepth'' (97), ''participants'' fired (96), ''isEmpirical'' (95), ''legal.law'' (95), ''humanAnnotation'' fired (94), ''.consentAction'' (93), ''artifacts.availability'' (90) | Publish the percentage. |
| **65–80%** | ''platforms'' (77), ''ethics.robotsTxt'' (76), ''temporal.mode'' (69), ''vantage.infrastructure'' (69), ''ethics.reviewOutcome'' (68), ''statistics.kind'' (68), ''ethics.notifiedAffectedParties'' (67) | Publish, with the caveat that a repeat run moves it a few points. |
| **under 60%** | ''classification.method'' (58), **''studyTypes'' (57) — the least stable field in the schema** | Report as a ranking or a rough share. Never a precise figure. |
| **about 20%** | the free-text names: ''detection.phenomenon'', ''classification.resourceName'', ''statistics.method'' | Fold before counting. Never aggregate by exact string. |

The bottom two rows are not the model being unreliable about what a paper says. ''studyTypes'' and ''classification.method'' are the two fields where a paper genuinely admits several defensible readings — a crawl that also reanalyses a public dataset is honestly both — and multi-valued fields are scored here by exact //set// equality, so one extra tuple flips them. The 20% row is wording variance: the same finding described as "third-party cookie setting" and "setting of third-party cookies" is one disagreement about English, not two readings of the paper.

===== 7. The rules every corpus-backed page here follows =====

Four rules, each with the failure it prevents. If a page on this site breaks one, the figure is wrong.

**1. Name the denominator before the numerator.** Each query defines its own population. The table below is the one legitimate use of 5,859 as a denominator — it exists to show how rarely 5,859 is the right one:

^ Population ^ Papers ^ Share of 5,859 ^
| ''sampled'' — drew a study population | 5,712 | 97.5% |
| ''empirical'' — measured the world as it is | 5,118 | 87.4% |
| ''classified'' — classified or labelled something | 4,439 | 75.8% |
| ''measuredFrom'' — measured from a vantage point | 3,908 | 66.7% |
| ''annotated'' — coded data by hand | 3,318 | 56.6% |
| ''inferential'' — ran statistical inference | 1,762 | 30.1% |
| ''web'' — measured the web platform | 1,622 | 27.7% |
| ''humanSubjects'' — recruited participants | 1,357 | 23.2% |
| ''crawled'' — ran an automated web crawl | 1,120 | 19.1% |
| ''legal'' — assessed compliance with a law | 402 | 6.9% |

**2. A sentinel is never an answer.** The five sentinels are ''not-stated'', ''none-mentioned'', ''not-applicable'', ''unclear'' and ''unknown''. Worked example — ''ethics.robotsTxt'' over the 1,120 crawling papers: 53 papers (4.7%) state a real value; 992 (88.6%) hold //some// value, the rest of which is ''not-stated''. Counting the sentinel turns "one crawling paper in twenty says anything about robots.txt" — which is the finding — into "seven in eight do".

**3. Count papers, never tuples.** A paper naming EasyList three times is one paper: 145 ''EasyList'' tuples across 94 papers, 1.54 per paper.

**4. Fold free-text names before aggregating, and print what the fold could not map.** This is the rule that is easiest to skip and most expensive to skip. The worked case is ''vantage[].locations'', which is free text — "Frankfurt, Germany", "US-East", "a US-based university network":

^ How the United States is counted ^ Papers ^
| Papers whose ''vantage.locations'' contains the exact string ''United States'' | 238 |
| Folded by the dataset's own ''name_fold.mjs'' — case, punctuation, and a short alias list | 367 |
| **Folded on meaning by ''scripts/geo.mjs'' — states, cities, regions and abbreviations** | **639** |

**A 42.6% undercount, on the exact quantity such a page is about**, and the middle row is what the dataset's own convenience query reports. 308 distinct raw strings fold to the United States: ''US'', ''USA'', ''U.S.'', ''California'', ''New York'', ''Oregon'', ''Los Angeles'', ''US East Coast'', ''Silicon Valley'', ''US-East'' and 298 more. Case-and-punctuation folding does not touch any of them.

Every fold written for a page on this site therefore returns its **unmapped residue**, and the residue is printed in full on that page's ''provenance:'' page. A residue that exists only inside a local script's output is a residue nobody will ever look at. Folds also age. When the corpus was extended to 2025–2026, [[privacy:fingerprinting]]'s subject fold went from 7 unmapped tuples to 59 **with no code change at all** (''scripts/report_fingerprinting.mjs'', not this page's script) — the newer papers fingerprint new things (DPI boxes, LLMs, AR/VR apps). It was extended and is back to 7. A fold that does not print its residue would have absorbed the difference silently.

===== 8. How to check a figure yourself =====

**What you can check.** Every figure on a corpus-backed page traces to a sentence in a published paper, and the ''provenance:'' page for that content page gives you the query, the population, the denominator, the fold, and the report script's unedited output. Two things follow that you can act on without any access to the corpus:

  - **Check the arithmetic and the population.** If a percentage's denominator is not named on the page or on its ''provenance:'' page, that is a defect in the page and worth reporting. So is a share of a population the page never defines.
  - **Read the papers a page names.** Where a page names its papers — the worked examples, the tool tables, the "read these first" lists — the papers are public and are the evidence. The dataset is not.

**What you cannot check.** The corpus itself — the venue index, the screening labels, the retrieved PDFs and ''extractions.jsonl'' — is a local dataset on the maintainer's machine. It is not published, not downloadable, and there is no API. The largest consequence is the plainest one, and it is not softened anywhere else on this page:

<WRAP important>
**An aggregate count on this site cannot be independently verified.** When a page says 182 papers crawled from the United States, the list of those 182 exists only in the unpublished extraction. You can check that the denominator is named, that the fold is described, that the arithmetic holds and that the handful of quotes the ''provenance:'' page spot-checked are real. You cannot recount the 182. Weigh these figures accordingly: they are a documented and internally audited measurement, not a reproducible one.
</WRAP>

Two further consequences follow:

  * **The extraction cannot be reproduced to the same values.** It is generative model output; two runs over identical text disagree, which is the whole point of the stability table in [[#Field stability: measured once, on the earlier corpus, never since]]. Re-running it would produce a similar but not identical dataset.
  * **The extraction currently exists on one disk with no archive.** That is a known, unresolved risk recorded in the dataset's own runbook, and it is stated here rather than left out because a reader deciding how much to lean on these figures should know it.

===== 9. The report script and its output =====

Every figure on this page comes from one script, which prints all of them with their denominators, plus the checks that the funnel is internally consistent. Its unedited output, as printed on 2026-08-12, is in the [[#Appendix: the report script's full output|appendix]] at the foot of this page and is downloadable from there.

<code bash>
node scripts/report_corpus.mjs                     # the funnel, scope, folds and rules
node scripts/report_corpus.mjs --quotes            # + all 135,025 quotes re-checked (~20 s)
node scripts/report_corpus.mjs --quotes --wiki     # the same, as DokuWiki tables

# the staleness guard: assert every number on this page appears in the report
node scripts/check_page_numbers.mjs pages/literature_corpus.txt out/new/report_corpus.txt --code
</code>

Two figures on this page are not produced by the script, and both name their source: the **2,870** papers whose token ledger was lost, which comes from the extraction's own ''README.md'', and the **29** phantom quote failures caused by a bug in an early version of this page's own script, described in [[#11. Run log]]. Every other figure on the page, including all four quote-groundedness percentages and the whole funnel, is produced by the script.

What the guard actually reported on 2026-08-12 is worth stating exactly, because it says as much about the guard as about the page. It flagged six figures: the **2,870**, and five digit fragments of quoted paper text (a DOI, a percentage and a participant-table row inside the quotes in [[#Quote groundedness, re-measured on this corpus|section 6]]). It did **not** flag the **29** — because 29 also happens to appear in the report as the count of crawling papers that discuss robots.txt.

That is the shape of this kind of guard. It asks whether a number //appears// in the report, so it gives a false pass whenever a stale figure collides with an unrelated live one, and it is noisy wherever a page quotes someone else's numbers. It is a floor on staleness checking, not a proof of freshness, and its output has to be read rather than expected to come back clean.

The script also prints seven consistency checks on the funnel — retrieved papers outside the selection rule, fulltext directories outside it, extraction records with no prepared text on disk, duplicate records, labelled papers with no abstract, abstracts never screened, and truncated extractor inputs. All seven return zero. A nonzero value in any of them means the funnel above is wrong, not that the corpus is.


===== 10. What this page could not establish =====

Stated plainly, because a provenance page that overstates its own rigour is worse than none.

  * **Whether the screening is accurate.** No human-screened control set exists. A relevant paper wrongly dropped at abstract screening leaves no trace in any artefact, so the false-negative rate of selection is unknown and unknowable from what is on disk. This is the largest unquantified risk in the funnel and it sits at the widest step of it.
  * **Whether field stability has moved.** The agreement figures in §6 are from the 4,322-paper corpus. Re-measuring means a second full extraction pass; it has not been done, and every page quoting those figures should say so.
  * **Whether the 14 unextracted papers matter.** Thirteen are IEEE S&P, eleven of them from 2024. Nobody has read them to see whether they would have changed anything.
  * **Whether the 309 papers in the nine empty venue-years matter.** The //cause// is established and documented — those venue pages publish no abstracts — but nobody has read the 309 titles to see how much relevant methodology is sitting outside the corpus. That is a cheap check nobody has done.
  * **The per-run token and cost ledger.** The extraction's own accounting was lost for roughly the first 2,870 papers of the original run because the ledger was written only on clean exit and every restart killed the process first. The total is a measured tail plus an extrapolation, and it is not re-derivable.
  * **Anything about a venue that is not one of the seven.** This is not a limitation that better tooling fixes; it is the scope.

===== 11. Run log =====

^ ^ ^
| Page written | 2026-08-12 |
| Corpus at the time | 5,859 papers, 7 venues, 117 venue-years, 2010–2026 |
| Script | ''scripts/report_corpus.mjs'' (new for this page) |
| Model | Claude Opus 5 |
| Figures carried over from earlier notes | **None.** Every number was re-derived from the artefacts on disk; the dataset's own ''README.md'' still quotes the 4,322-paper figures and was not used as a source. |
| Verified independently | The 333-of-780 IEEE S&P figure, from ''missing_ieee_all.jsonl'' holding exactly 447 records (780 − 447 = 333). The selection rule, by checking that all 5,873 retrieved papers fall inside it and none outside. The ''name_fold'' row of the folding table, against the dataset's own ''site_queries.mjs --page vantage'', which reports the same 367. |
| Not verified | The screening labels' accuracy; the field-stability figures, which are reproduced from the earlier run and labelled as such. |
| Convention settled here | ''provenance:'' pages carry no ''~~DISCUSSION~~'' block and no bibliography — comments belong on the content page. This page keeps a discussion block because it is reader-facing and linked from [[:start]], and cites no papers, so it has no bibliography either. |
| Why there is no ''provenance:literature:corpus'' | This page //is// a provenance page: sections 9 to 11 are its own working log. A provenance page for the provenance page would recurse without adding anything. |
| Wired into | [[:start]]; the six ''provenance:'' pages, which already link here; and the //Methodology and limitations of these figures// section of each corpus-backed content page, where the generic "these venues are absent, 2025–2026 are provisional" text was replaced by a pointer here plus the page-specific consequence. |
| Mistakes caught before review | The quote check initially read ''paper.cols.txt'' for all papers, including the 7 whose extractor input was OCR text; that scored a broken font encoding as 29 fabricated quotes in a single paper and put it top of the not-found ranking. Fixed by keying on each record's own ''textSource''. |

==== Review, 2026-08-12 ====

Three reviewers (Claude Sonnet 5), each given the page, the script and its output, and each told explicitly that the summary they were given might not be exhaustive. What they found, and what was done about it — the rejections matter as much as the fixes.

^ Reviewer ^ Finding ^ Verdict ^
| Figures vs script | **''report_corpus.mjs'' §9c printed hardcoded literals** — ''13 / 780'', ''104 / 230'' and ''4.0% + 1.3% = 5.3%'' — in the very block whose job is to make derived figures auditable. All three happened to be correct against the current data, and all three would have gone stale silently on the next corpus growth //while the staleness guard kept passing//, because the guard only asks whether a number appears in the report. | **Accepted.** All three now computed. |
| Figures vs script | "twelve of them from 2024" — the real split of the 13 unextracted IEEE S&P papers is 11 from 2024, 1 from 2023, 1 from 2021. Contradicted by the page's own embedded output. Invisible to the guard because the number was spelled out as a word. | **Accepted.** |
| Figures vs script | "flat across sixteen years" — 2010–2026 is seventeen. | **Accepted.** |
| Claims vs sources | The stability table read ''has-an-artifact-link'', where the source says ''has-authors'-own-link''. Those are different claims: any artifact link at all, versus the authors releasing their own. | **Accepted.** |
| Claims vs sources | The three named causes of the 443 pre-2020 missing abstracts sum to 426, not 443. The remaining 17 are a fourth cause of a different kind — scattered singles, mostly front matter, recoverable in principle — so a flat "waiting will not fix it" was wrong for those. | **Accepted.** The report now prints the residue and the table names it. |
| Claims vs sources | The screening instruction is verbatim from ''build_selection_batch.mjs'', not ''label_papers.mjs'', which carries a similar but distinct rubric. Not an error, but easy for a later editor to mis-attribute. | **Accepted** as a clarification; the script is now named on the page. |
| Reader fit | The page's opening example — "31.4% of papers that measured something" — did not name its denominator, which is the exact failure the page exists to prevent. | **Accepted.** It now names the 3,908. |
| Reader fit | The sentinel rule named two sentinels where the schema has five. A reader trusting this page as the authoritative statement would not know ''unclear'' and ''unknown'' also count as silence. | **Accepted.** |
| Reader fit | The raw output block, which is most of the page by line count, sat between "how to check a figure yourself" and the two sections a sceptical reader most needs. | **Accepted.** It is now an appendix after the run log. |
| Reader fit | The population table's header, "Share of 5,859", reads as the page breaking its own rule two sections after stating it. | **Accepted**, reworded to say why that denominator is the point of that particular table. |
| Reader fit | ''[[#How to check a figure yourself]]'' was reported as a broken anchor, because the heading is numbered "8." and the link is not. | **Rejected.** Checked against the published HTML: DokuWiki's ''cleanID'' strips the leading ''8. '' when it builds the section id, so the heading's id //is// ''how_to_check_a_figure_yourself'' and the link resolves. Both the numbered and unnumbered forms work. The rendered page has no broken anchors and no red links. |

==== Second review, 2026-08-12 ====

A fourth reviewer (Claude Fable 5) was given no checklist and asked for whatever the first three were not looking for. It found more than they did, and two of its findings are properties of the corpus that nobody had written down.

^ Finding ^ Verdict ^
| **138 of the 5,859 records are posters**, unfiltered by the venue index and not rejected by screening, 89 of them inside the ''measuredFrom'' population. Posters compress their methodology out, so they push every //silence// figure on this site upward — which is the class of figure the site is mostly about. | **Accepted.** Measured, and now a named property of the funnel in [[#What "5,859 papers" actually contains]]. |
| **The duplicate check was blind to the duplication mode the venue index warns about.** Keyed on ''(venue, year, slug)'' it returns zero; keyed on the title, three papers are in the corpus twice. "Zero duplicates" was being reported by a check that could not have found them. | **Accepted.** The script now checks both keys and names the three. |
| The five hand-checked quotes were all from CCS 2010–2012, because the script's not-found list is filled in sorted-key order — a favourable sample presented as if it generalised, while the years with the most failures had none checked. | **Accepted.** Five more were drawn from 2025–2026 and read by hand; one of them turned out to be a different failure mode (a sentence about the paper's silence rather than a quote), which was then measured corpus-wide at 12 of 135,025. |
| §8's "read the paper" was technically true and practically misleading: corpus-backed tables do not name their papers, so **an aggregate count cannot be independently recounted at all** — the biggest item, and it was missing from "what you cannot check". | **Accepted.** It is now the boxed statement in [[#8. How to check a figure yourself]]. |
| The script printed a literal ''input paper.norm.txt'' in §7 of its own output, contradicting the ''textSource'' counts three screens above it. Same defect class as the hardcoded arithmetic the first reviewer caught, but hiding in a string. | **Accepted.** Computed now. |
| "twelve of them from 2024" survived in §10 after being fixed in §2 — the first review log said "Accepted", which read as "fixed everywhere". | **Accepted.** Fixed, and worth recording that a review log can flatter a partial fix. |
| The funnel's first row read "(7 venues × 117 venue-years)", which parses as a product. The screening model was never named. "Two honest caveats" and "worth being blunt about" label the page's own candour. | **All accepted.** |
| The dataset's own ''extract/README.md'' and runbook still carry the dead "IEEE S&P is 43% retrieved" caveat and a stale permanent-gaps table. | **Accepted as a note**: if you follow a citation from this page into those files, their IEEE S&P figures predate the 2026-08-10 repair described in [[#3. IEEE S&P: the worked example of a systematic loss]]. |
| Sibling page [[design:crawling_location]] carries the same "sixteen years" slip that was fixed here. | **Accepted**, fixed on that page. |

Working notes for individual pages are under ''provenance:''; see [[#The provenance namespace]].

===== Appendix: the report script's full output =====

Unedited, as printed on 2026-08-12. It is at the end of the page rather than beside the commands because a reader checking a number wants it and a reader learning the caveats does not.

<file text report_corpus-output.txt>
report_corpus.mjs — every figure on literature:corpus
data root: /workspace/publications_dataset/data
generated: 2026-08-12

== 1. Scope ==

venues                 7  (CCS, IEEE-SP, IMC, NDSS, PETS, USENIX, WWW)
years                  2010–2026
venue-year files       117
  of a possible        119 (7 venues x 17 years)

absent venue-years     2: CCS 2026, IMC 2026

  Venue    Venue-years  First  Last  Metadata records
  -------  -----------  -----  ----  ----------------
  CCS      16           2010   2025  3,381           
  IEEE-SP  17           2010   2026  1,837           
  IMC      16           2010   2025  862             
  NDSS     17           2010   2026  1,577           
  PETS     17           2010   2026  1,253           
  USENIX   17           2010   2026  3,012           
  WWW      17           2010   2026  4,942           


== 2. The funnel ==

  Stage                                      N       Lost here  Loss is                                       
  -----------------------------------------  ------  ---------  ----------------------------------------------
  Venue-year metadata records                16,864                                                           
  …with an abstract                          15,800  −1,064     systematic: see the two eras below            
  Screened (one label record each)           15,800  0          none: screening is complete over the abstracts
  Selected by the relevance rule             6,103   −9,697     by design: out of scope                       
  PDF retrieved (>=1 KB)                     5,873   −230       systematic: paywall / not yet published       
  pdftotext produced paper.txt               5,873   0          none                                          
  Text prepared, so eligible for extraction  5,873   0          none                                          
  Structured extraction record               5,859   −14        systematic: IEEE-SP rate-limit tail           

Text preparation, in more detail than the funnel row shows:
  paper.txt        5873   (pdftotext -layout, raw, columns side by side)
  paper.cols.txt   5869   (decolumn.mjs; 4 PDFs have no usable text layer)
  paper.norm.txt   5873   (normalize_text.mjs; also carries the OCR text for those 4)
The extractor picks per paper. Of the 5859 records: {"cols":5852,"mistral":7}

Selected -> extracted overall: 5859 of 6103 = 96.0%
Metadata -> extracted overall: 5859 of 16864 = 34.7%

The 1,064 records with no abstract split into two eras:
  2010-2019 (venue pages and OpenAlex never had them): 443
  2020-2024                                          : 58
  2025-2026 (OpenAlex has the DOI, not yet the abstract): 563

Missing abstracts by venue-year, every block of 10 or more:

  Venue-year    Records with no abstract
  ------------  ------------------------
  WWW 2026      324                     
  IEEE-SP 2026  194                     
  NDSS 2018     71                      
  NDSS 2016     60                      
  CCS 2011      40                      
  NDSS 2014     32                      
  PETS 2014     30                      
  NDSS 2011     28                      
  CCS 2025      27                      
  PETS 2011     27                      
  PETS 2012     25                      
  NDSS 2010     24                      
  WWW 2024      23                      
  CCS 2024      22                      
  PETS 2010     22                      
  PETS 2013     22                      
  NDSS 2013     20                      
  NDSS 2012     15                      

NDSS 2010-2018 block: 260   (venue pages publish no abstract, and NDSS has no DBLP DOIs)
PETS 2010-2014 block: 126   (pre-PoPETs workshop schedule pages publish no abstracts)
CCS 2011:             40    (DOI-less poster entries under a DOI-less parent proceedings)
those three account for 426 of the 443 pre-2020 records;
the remaining 17 are scattered singles across PETS, USENIX and NDSS, mostly front matter,
and are recoverable in principle, unlike the three structural blocks.
Causes per block are documented in data/corpus2/README.md §5; the counts here are re-derived.

CHECK  retrieved papers outside the selection rule: 0
CHECK  fulltext directories outside the selection rule: 0
CHECK  extraction records with no paper.norm.txt on disk: 0
CHECK  duplicate extraction records, keyed on (venue, year, slug): 0
CHECK  duplicate extraction records, keyed on normalised TITLE: 3 extra records across 3 titles
         CCS/2010/detecting-and-characterizing-social-spam-campaigns  |  IMC/2010/detecting-and-characterizing-social-spam-campaigns
         NDSS/2020/uiscope-accurate-instrumentation-free-and-visible-attack-investigation-for-gui-applications  |  NDSS/2021/uiscope-accurate-instrumentation-free-and-visible-attack-investigation-for-gui-applications
         NDSS/2022/auto-draft-242  |  NDSS/2022/drawn-apart-a-device-identification-technique-based-on-remote-gpu-fingerprinting
CHECK  extraction records that are posters: 138 {"CCS":96,"IMC":42}
         of those, in measuredFrom: 89; in crawled: 15
         records of 4 pages or fewer: 251
CHECK  labelled papers with no abstract in metadata: 0
CHECK  abstracts never screened: 0
CHECK  extraction records flagged textTruncated: 0
CHECK  textSource of extractor input: {"cols":5852,"mistral":7}
CHECK  extraction model: {"gpt-5.6-luna":5859}
CHECK  screening model: gpt-5.6-luna (service tier flex), last segment finished 2026-08-07T15:00:50.647Z


== 3. Where the extraction lost papers ==

paper.norm.txt on disk but no extraction record: 14

  Paper                                                                                          Reason                   
  ---------------------------------------------------------------------------------------------  -------------------------
  USENIX/2017/oss-fuzz-googles-continuous-fuzzing-service-for-open-source-software               no failure logged        
  IEEE-SP/2021/happer-unpacking-android-apps-via-a-hardware-assisted-approach                    520 status code (no body)
  IEEE-SP/2023/three-birds-with-one-stone-efficient-partitioning-attacks-on-interdependent-cryp  520 status code (no body)
  IEEE-SP/2024/a-systematic-study-of-physical-sensor-attack-hardness                             520 status code (no body)
  IEEE-SP/2024/conning-the-crypto-conman-end-to-end-analysis-of-cryptocurrency-based-technical   520 status code (no body)
  IEEE-SP/2024/jasmine-scale-up-javascript-static-security-analysis-with-computation-based-sema  520 status code (no body)
  IEEE-SP/2024/nyx-detecting-exploitable-front-running-vulnerabilities-in-smart-contracts        520 status code (no body)
  IEEE-SP/2024/on-sms-phishing-tactics-and-infrastructure                                        520 status code (no body)
  IEEE-SP/2024/pandora-principled-symbolic-validation-of-intel-sgx-enclave-runtimes              520 status code (no body)
  IEEE-SP/2024/pomabuster-detecting-price-oracle-manipulation-attacks-in-decentralized-finance   520 status code (no body)
  IEEE-SP/2024/pulling-off-the-mask-forensic-analysis-of-the-deceptive-creator-wallets-behind-s  520 status code (no body)
  IEEE-SP/2024/revisiting-automotive-attack-surfaces-a-practitioners-perspective                 520 status code (no body)
  IEEE-SP/2024/routing-attacks-on-cryptocurrency-mining-pools                                    520 status code (no body)
  IEEE-SP/2024/sok-a-comprehensive-analysis-and-evaluation-of-docker-container-attack-and-defen  520 status code (no body)

extraction failures logged (retries included): 693 over 559 distinct papers
  Failure reason             Log lines
  -------------------------  ---------
  520 status code (no body)  603      
  Connection error.          64       
  Request timed out.         25       
  terminated                 1        

of those 559 papers, 546 were recovered by a retry


== 4. Funnel by venue ==

  Venue    Metadata  Abstract  Screened  Selected  PDF    Extracted  Extracted / selected
  -------  --------  --------  --------  --------  -----  ---------  --------------------
  CCS      3,381     3,290     3,290     1,036     990    990        95.6%               
  IEEE-SP  1,837     1,642     1,642     780       780    767        98.3%               
  IMC      862       854       854       691       638    638        92.3%               
  NDSS     1,577     1,317     1,317     701       701    701        100.0%              
  PETS     1,253     1,113     1,113     510       510    510        100.0%              
  USENIX   3,012     2,997     2,997     1,520     1,411  1,410      92.8%               
  WWW      4,942     4,587     4,587     865       843    843        97.5%               


== 5. Funnel by year — where 2025 and 2026 are thin ==

  Year                Venue-years  Metadata  Abstract  Abstract share  Selected  Extracted
  ------------------  -----------  --------  --------  --------------  --------  ---------
  2010                7            496       450       90.7%           123       119      
  2011                7            352       257       73.0%           119       116      
  2012                7            426       384       90.1%           178       151      
  2013                7            484       442       91.3%           141       125      
  2014                7            526       460       87.5%           169       166      
  2015                7            573       566       98.8%           193       190      
  2016                7            620       557       89.8%           192       182      
  2017                7            686       680       99.1%           237       231      
  2018                7            711       638       89.7%           260       254      
  2019                7            1,002     999       99.7%           407       402      
  2020                7            953       952       99.9%           414       404      
  2021                7            1,167     1,166     99.9%           386       379      
  2022                7            1,348     1,347     99.9%           556       546      
  2023                7            1,598     1,595     99.8%           721       719      
  2024                7            1,901     1,849     97.3%           712       690      
  2025 (provisional)  7            2,032     1,992     98.0%           774       770      
  2026 (provisional)  5            1,989     1,466     73.7%           521       415      

Venue-years with abstract coverage below 90% (a paper with no abstract is never screened,
so it can never be selected — those venue-years are under-selected by construction):

  Venue-year    Metadata  With abstract  Share
  ------------  --------  -------------  -----
  NDSS 2010     24        0              0.0% 
  NDSS 2011     28        0              0.0% 
  NDSS 2016     60        0              0.0% 
  NDSS 2018     71        0              0.0% 
  PETS 2010     22        0              0.0% 
  PETS 2011     27        0              0.0% 
  PETS 2012     25        0              0.0% 
  PETS 2013     22        0              0.0% 
  PETS 2014     30        0              0.0% 
  IEEE-SP 2026  252       58             23.0%
  NDSS 2014     78        46             59.0%
  CCS 2011      102       62             60.8%
  WWW 2026      954       630            66.0%
  NDSS 2013     59        39             66.1%
  NDSS 2012     54        39             72.2%

Venue-years contributing ZERO papers to the extracted corpus: 9
These are holes, not thin spots. The wiki is silent about them by construction.

  Venue-year  Metadata records  With abstract  Selected
  ----------  ----------------  -------------  --------
  NDSS 2010   24                0              0       
  NDSS 2011   28                0              0       
  NDSS 2016   60                0              0       
  NDSS 2018   71                0              0       
  PETS 2010   22                0              0       
  PETS 2011   27                0              0       
  PETS 2012   25                0              0       
  PETS 2013   22                0              0       
  PETS 2014   30                0              0       


== 6. IEEE S&P — a systematic loss, found and repaired ==

IEEE-SP selected      780
IEEE-SP PDF retrieved 780  (100.0% of selected)
IEEE-SP extracted     767  (98.3% of selected)
IEEE-SP not retrieved 0

  Year  IEEE-SP selected but no PDF
  ----  ---------------------------

The retrieval gap that was repaired on 2026-08-10: the Xplore PDF is at
stampPDF/getPDF.jsp, not the stamp.jsp viewer shell, so scraping the viewer
found no PDF link and logged "No PDF found" — indistinguishable from a paywall.
(data/RUNBOOK_new_conference_year.md §3.)

Every other venue, for comparison — selected papers with no PDF on disk:

  Venue    Selected  No PDF  Share missing
  -------  --------  ------  -------------
  CCS      1036      46      4.4%         
  IEEE-SP  780       0       0.0%         
  IMC      691       53      7.7%         
  NDSS     701       0       0.0%         
  PETS     510       0       0.0%         
  USENIX   1520      109     7.2%         
  WWW      865       22      2.5%         

The retrieval loss is spread over 37 venue-years. The 10 largest:

  Venue-year   Selected but no PDF
  -----------  -------------------
  USENIX 2026  104                
  IMC 2012     22                 
  IMC 2013     12                 
  WWW 2024     8                  
  IMC 2020     6                  
  CCS 2018     6                  
  WWW 2022     5                  
  CCS 2016     4                  
  CCS 2012     4                  
  CCS 2021     4                  

top 6 venue-years: 158 papers
remaining 31 venue-years: 72 papers


== 7. What the extraction is ==

records                5,859, one per paper, keyed on (venue, year, slug)
model                  gpt-5.6-luna
passes                 1 (single pass; no adjudication, no second coder)
input                  paper.cols.txt (5,852), paper.mistral.md (OCR) (7); paper.norm.txt only gates eligibility
relation families      13: tools, population, crawlConfig, classification, detection, vantage, temporal, statistics, humanAnnotation, participants, ethics, artifacts, legal

evidence quotes        135,025 across 5,859 papers
  Family           Tuples carrying a quote
  ---------------  -----------------------
  tools            37,359                 
  detection        27,241                 
  population       17,016                 
  classification   11,330                 
  statistics       8,787                  
  temporal         8,775                  
  humanAnnotation  5,778                  
  artifacts        5,524                  
  ethics           4,800                  
  vantage          4,685                  
  participants     2,018                  
  crawlConfig      1,080                  
  legal            632                    


== 7b. Populations — the denominators a page may use ==

  Population     Definition                               Papers  Share of 5859
  -------------  ---------------------------------------  ------  -------------
  sampled        drew a study population                  5,712   97.5%        
  empirical      measured the world as it is              5,118   87.4%        
  classified     classified or labelled something         4,439   75.8%        
  measuredFrom   took a measurement from a vantage point  3,908   66.7%        
  annotated      coded data by hand                       3,318   56.6%        
  inferential    ran statistical inference                1,762   30.1%        
  web            measured the web platform                1,622   27.7%        
  humanSubjects  recruited human participants             1,357   23.2%        
  crawled        ran an automated web crawl               1,120   19.1%        
  legal          assessed compliance with a law           402     6.9%         


== 8. Folding is not optional — the vantage/United States case ==

papers whose vantage.locations contains the exact string "United States": 238
…folded by the dataset's name_fold.mjs (what site_queries.mjs reports):   367
…folded on meaning (scripts/geo.mjs):                                     639
undercount from name_fold alone: 272 papers = 42.6% of the folded total
distinct raw strings that fold to the United States: 308

The 25 most common of them:

  Raw string               Tuples
  -----------------------  ------
  United States            262   
  US                       62    
  USA                      51    
  California               25    
  U.S.                     22    
  New York                 16    
  Oregon                   10    
  Los Angeles              9     
  New York City            9     
  Virginia                 7     
  San Francisco            6     
  Ohio                     6     
  Seattle                  5     
  Chicago                  5     
  Atlanta                  5     
  New Jersey               4     
  America                  4     
  US East Coast            4     
  Boston, Massachusetts    4     
  Silicon Valley           4     
  California, USA          4     
  Illinois                 3     
  Boston, MA, USA          3     
  Northeast United States  3     
  US-East                  3     


== 8b. The other two rules, as numbers ==

Sentinels are never answers — ethics.robotsTxt over the papers that ran a crawl:
  population (crawled)                       1120
  ethics object fired at all                 992
  robotsTxt holds a real value               53 = 4.7%
  robotsTxt holds any value incl. sentinels  992 = 88.6%
  full distribution: {"not-stated":939,"(no ethics object at all)":128,"discussed":29,"respected":15,"ignored":9}
  Counting the sentinel as an answer turns "one crawling paper in twenty says anything
  about robots.txt" into "seven in eight do".

Papers, never tuples — three names, counted both ways:

  Exact name  Tuples  Papers  Tuples per paper
  ----------  ------  ------  ----------------
  EasyList    145     94      1.54            
  OpenWPM     61      61      1.00            
  Selenium    238     237     1.00            

Filter usedOrMentioned — tool tuples by relationship:

  usedOrMentioned  Tuples  Share
  ---------------  ------  -----
  used             31,496  84.3%
  produced         3,528   9.4% 
  compared         1,876   5.0% 
  mentioned        438     1.2% 
  unclear          21      0.1% 

Where the measured results are — detection[].prevalence:
  detection tuples                 27,241 across 5,655 papers
  …carrying a prevalence figure    26,316 across 5,572 papers = 96.6% of tuples


== 9. Quote groundedness ==

papers checked         5,859 (whole corpus)
quotes checked         135,025
quotes skipped (the extractor's source file is not on disk): 0

  Verdict                                                        Quotes  Share of checked
  -------------------------------------------------------------  ------  ----------------
  exact, after collapsing whitespace and line-break hyphenation  78,450  58.1%           
  an 8-word run survives                                         49,425  36.6%           
  fragmented — every content word in one window, order broken    5,442   4.0%            
  not found                                                      1,708   1.3%            

quotes that describe the paper's silence rather than quoting it: 12 (0.0%)

of the 1,708 not-found quotes, 271 (15.9%) contain an ellipsis,
i.e. the extraction spliced two parts of the source and marked the join.

papers with at least one not-found quote: 1143 of 5859 = 19.5%
the 10 worst papers hold 82 of the 1708 not-found quotes:

  Paper                                                                                                Not-found quotes
  ---------------------------------------------------------------------------------------------------  ----------------
  CCS/2017/how-unique-is-your-onion-an-analysis-of-the-fingerprintability-of-tor-onion-serv            12              
  NDSS/2019/countering-malicious-processes-with-process-dns-association                                10              
  WWW/2024/identifying-risky-vendors-in-cryptocurrency-p2p-marketplaces                                9               
  IEEE-SP/2024/smartinv-multimodal-learning-for-smart-contract-invariant-inference                     8               
  USENIX/2024/attention-please-an-investigation-of-the-app-tracking-transparency-permission            8               
  CCS/2019/charting-the-attack-surface-of-trigger-action-iot-platforms                                 7               
  IMC/2019/measuring-ewhoring                                                                          7               
  NDSS/2026/rethinking-fake-speech-detection-a-generalized-framework-leveraging-spectrogram-magnitude  7               
  PETS/2018/panoptispy-characterizing-audio-and-video-exfiltration-from-android-applications           7               
  WWW/2026/anomaly-detection-of-interaction-behaviors-in-streaming-graphs                              7               

Not-found rate by year — the text pipeline is the same throughout, so a trend here
is about the PDFs, not about the extraction:

  Year  Quotes checked  Not found  Rate
  ----  --------------  ---------  ----
  2010  2,212           21         0.9%
  2011  2,416           21         0.9%
  2012  2,938           34         1.2%
  2013  2,485           20         0.8%
  2014  3,521           40         1.1%
  2015  4,189           49         1.2%
  2016  4,121           53         1.3%
  2017  5,319           80         1.5%
  2018  5,829           79         1.4%
  2019  9,225           103        1.1%
  2020  9,619           92         1.0%
  2021  9,157           96         1.0%
  2022  12,724          166        1.3%
  2023  16,310          245        1.5%
  2024  16,007          208        1.3%
  2025  18,686          243        1.3%
  2026  10,267          158        1.5%

Up to 25 not-found quotes, for hand-checking:

  Paper                                                                                      Family           Quote (truncated)                                                                                             
  -----------------------------------------------------------------------------------------  ---------------  --------------------------------------------------------------------------------------------------------------
  CCS/2010/a-methodology-for-empirical-analysis-of-permission-based-security-models-and-its  detection        applications can be assigned to the nearest neuron, effectively clustering the applications requesting similar
  CCS/2010/detecting-and-characterizing-social-spam-campaigns                                vantage          We carry out the study on Facebook ... We choose 8 regional networks ... as targets for data collection.      
  CCS/2010/fingerprinting-websites-using-remote-traffic-analysis                             population       The results are shown in Table 1. Classification accuracy ... 12 ...                                          
  CCS/2010/fingerprinting-websites-using-remote-traffic-analysis                             population       The results are shown in Table 1. Classification accuracy ... 24 ...                                          
  CCS/2011/how-to-tell-if-your-cloud-files-are-vulnerable-to-drive-crashes                   population       We experimented with drives from Fujitsu, Hitachi, HP, and Seagate.                                           
  CCS/2011/ispy-automatic-reconstruction-of-typed-input-from-compromising-reflections        tools            For this we use the Scale Invariant Feature Transform, or SIFT [22].                                          
  CCS/2011/waptec-whitebox-analysis-of-web-applications-for-parameter-tampering-exploit-con  artifacts        A detailed report of exploits found by our approach can be found at http://sisl.rites.uic.edu/waptec.         
  CCS/2012/aligot-cryptographic-function-identification-in-obfuscated-binary-programs        tools            Among all tracing tools available in such environment, we chose Pin, the dynamic binary instrumentation framew
  CCS/2012/authenticated-secret-key-extraction-using-channel-characteristics-for-body-area   tools            This becomes a generalization of single-source single-sink maximum-flow problem. Therefore, we let each sensor
  CCS/2012/cross-vm-side-channels-and-their-use-to-extract-private-keys                      temporal         The data collection lasted about six hours, during which roughly 1000 key-related fragments were recovered fro
  CCS/2012/peerpress-utilizing-enemies-p2p-strength-against-them                             population       We built one virtual environment with six virtual machines. All virtual machines installed Windows XP SP1 with
  CCS/2012/peerpress-utilizing-enemies-p2p-strength-against-them                             vantage          We built one virtual environment with six virtual machines.                                                   
  CCS/2012/populated-ip-addresses-classification-and-applications                            tools            We use the LIBSVM [8] and LIBLINEAR [11] toolkits in our implementation.                                      
  CCS/2012/protecting-location-privacy-optimal-strategy-against-localization-attacks         temporal         The length of the considered traces is one month.                                                             
  CCS/2012/sabot-specification-based-payload-generation-for-programmable-logic-controllers   ethics           We evaluate S ABOT using four metrics: Accuracy ... Adaptability ... Performance ... Scalability              
  CCS/2013/beheading-hydras-performing-effective-botnet-takedowns                            classification   Our heuristics successfully classified 97% of the samples' contingency plans correctly.                       
  CCS/2013/chucky-exposing-missing-checks-in-source-code-for-vulnerability-discovery         tools            The parser is based on an island grammar [see 20] for the parser generator ANTLR [25].                        
  CCS/2013/delta-automatic-identification-of-unknown-web-based-infection-campaigns           vantage          via a 10-node cluster of custom crawlers running an adaptive fetch schedule                                   
  CCS/2013/protocol-misidentification-made-easy-with-format-transforming-encryption          detection        The resulting regexes achieved perfect misclassification for all classifier/target protocol combinations, as s
  CCS/2013/the-robustness-of-hollow-captchas                                                 tools            We use Lee's algorithm [10] to automatically detect and then repair broken contours.                          
  CCS/2013/users-get-routed-traffic-correlation-on-tor-by-realistic-adversaries              temporal         BGP paths gathered during March 2013 ... CAIDA ... from December 2012 ... CAIDA AS Relationships Dataset for J
  CCS/2013/viceroi-catching-click-spam-in-search-ad-networks                                 humanAnnotation  Working with the ad network we manually investigated around hundred websites associated with the publishers we
  CCS/2014/harvesting-high-value-foreign-currency-transactions-from-emv-contactless-credit   detection        Average transaction duration (card discovery to transaction approval) 478ms Standard deviation 36ms Fastest tr
  CCS/2014/nothing-else-mator-s-monitoring-the-anonymity-of-tors-path-selection              artifacts        The monitor programs are available on our website [1].                                                        
  CCS/2014/poster-positioning-attack-on-proximity-based-people-discovery                     tools            Android emulator(s) installed with mock location provider and customized plug-ins for various PBPD apps       


== 9b. Field stability — REPRODUCED, NOT RE-MEASURED ==

The block below is copied verbatim from data/extract/README.md. It was measured by
comparing two independent extraction runs over the same 100 papers, ON THE EARLIER
4,322-PAPER CORPUS. Re-measuring it needs a second full extraction pass and that has
not been done. It is printed here so that every figure the page carries has a source
in this file — not because this script computed it.

  Agreement  Fields                                                                                                                                                                                                                                  
  ---------  ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  90-100%    has-authors'-own-link (100), crawlConfig fired (99), .statefulness (98), .interactionDepth (97), participants fired (96), isEmpirical (95), legal.law (95), humanAnnotation fired (94), .consentAction (93), artifacts.availability (90)
  65-80%     platforms (77), ethics.robotsTxt (76), temporal.mode (69), vantage.infrastructure (69), ethics.reviewOutcome (68), statistics.kind (68), ethics.notifiedAffectedParties (67)                                                            
  under 60%  classification.method (58), studyTypes (57)                                                                                                                                                                                             
  about 20%  free-text names: detection.phenomenon, classification.resourceName, statistics.method                                                                                                                                                   

The same file reports the earlier corpus-wide quote audit as:
  exact 57.9%  |  8-word run 37.0%  |  fragmented 4.2%  |  not found 0.9%
Section 9 above re-measures that on the current corpus with an independent
implementation. Compare the two before quoting either.


== 9c. Figures the page derives by arithmetic ==

Each of these appears on the page and is a combination of two figures above.
They are printed so the staleness guard can account for them.

crawling papers that say NOTHING about robots.txt   1120 - 53 = 1067
IEEE-SP tail as a share of the venue                13 / 780 = 1.7%
  by year: 2021: 1, 2023: 1, 2024: 11
USENIX 2026 share of the retrieval loss           104 / 230 = 45.2%
US-folding raw strings beyond the 10 listed         308 - 10 = 298
papers in the nine zero-contribution venue-years    309
IEEE-SP retrieval before the 2026-08-10 repair      780 - 447 = 333 on disk = 42.7%
  (447 = line count of data/fulltext/missing_ieee_all.jsonl, the work list for that repair)
quote verdicts that are layout damage, not exact     4.0% + 1.3% = 5.3%
vantage location stated, the page's opening example 1228 of 3908 = 31.4%


== 10. What this script does not measure ==

- Run-to-run agreement of the extraction. That needs a second extraction pass over
  the same text; data/extract/stability100/ holds one, but it was run against the
  4,322-paper corpus and has not been repeated. Every agreement figure quoted on the
  page is from that older run and is labelled as such.
- Whether a selection decision was correct. The screening labels are model output and
  data/extract/README.md says plainly they are not ground truth; there is no
  human-screened control set to score them against.
- Whether a paper that produced no tuple for a family is silent or was missed.
</file>

/* This enables discussion under this article. */
~~DISCUSSION~~