Many pages on this site carry a section headed Use in Publications whose figures come from one source: a structured extraction over the full text of 5,859 papers from seven security, privacy and measurement venues, 2010–2026. This page is that source's methodology. It exists so that every figure on this site can be discounted correctly — you cannot judge “31.4% of papers that measured something say where they measured from” without knowing which papers were eligible to be counted, which were never eligible, and how the 31.4% was produced.

It is not a description of a public dataset. The corpus is not published. See How to check a figure yourself for what that means for you.

Every page whose figures come from this corpus has a second, published page holding the working notes behind it: each query with its population and denominator, the report script and its unedited output, the folds and their unmapped residue, the quotes that were spot-checked, and what could not be established.

Those pages live under provenance: and mirror the content page's id exactly. A content page's figures are auditable at provenance: followed by its own id — no lookup table, no suffix, no guessing:

This page is the root of that namespace. Corpus-level caveats — everything below — are stated here once, and each provenance: page carries only what is specific to its own page.

117 venue-years of a possible 119. CCS 2026 and IMC 2026 are absent because those conferences are held in October and November and had not taken place when the index was built.

EuroS&P, ACSAC, RAID, AsiaCCS, CHI and SOUPS are not in the corpus at all. Neither are PAM, TMA, ANRW, SIGCOMM or ACM CCR, nor the machine-learning venues where a growing share of privacy work now appears. The consequences are not evenly distributed:

• Usable privacy and HCI are missing by construction. SOUPS and CHI are where most user-study methodology for privacy is published. This site's pages on user studies, consent interfaces and qualitative coding are therefore built on whatever those seven venues happened to publish, which is a biased and much smaller sample of that literature. Treat any claim here about user-study practice as a claim about security venues' user studies.
• Internet measurement outside IMC is missing. For network-layer topics — IP geolocation, routing, anycast — PAM and TMA are core venues and their absence is a serious gap. ip_classification says so in its own limitations section.
• Every claim on this site is a claim about seven venues. Not about “the literature”, not about “the field”. Where a page words it otherwise, that is an error worth reporting.

Each row is counted from the artefact that stage produces, not from a note about it.

96.0% of selected papers made it to an extraction record; 34.7% of all indexed records did.

The 1,064 records with no abstract are three unrelated problems sharing a row, and only two of them go away with time.

This matters more than it looks, because screening runs on abstracts. A paper with no abstract is never screened, so it can never be selected, so it can never appear in any figure on this site. It is excluded for a reason that has nothing to do with its relevance. All 1,064 are present in the index with their titles, so they are countable rather than silently absent — which is how the table above can exist at all.

The 230 selected papers with no PDF are concentrated, not spread:

USENIX Security 2026 alone is 45.2% of the loss, for the simple reason that the conference has not happened yet and its papers are not online. Per venue, the share of selected papers with no PDF is 0% for IEEE S&P, NDSS and PETS, 2.5% for WWW, 4.4% for CCS, 7.2% for USENIX and 7.7% for IMC.

The 14 papers with prepared text but no extraction record. One is a USENIX 2017 slide deck excluded deliberately; the other 13 are IEEE S&P papers, twelve of them from 2024, whose extraction requests ended on 520 status code (no body) and were never retried. Across the whole run 559 papers failed at least once and 546 were recovered by a retry; these 13 are the tail nobody went back for. The effect is a mild under-representation of IEEE S&P 2024 in every figure on this site, worth roughly 1.7% of that venue.

Screening produces one label record per abstract with seven fields. A paper is selected if any of them is set:

--any securityMeasurement,privacyMeasurement,manualAnalysis,domains,darkPatterns,laws,crawlers

The instruction to the screening model was explicitly over-inclusive: “Include a paper if it plausibly matches the criteria — be deliberately over-inclusive, since a later pass reads the full text and a wrongly excluded paper is never seen again.” The rule was re-applied for this page and checked both ways: every one of the 5,873 retrieved papers falls inside it, and no directory exists outside it.

What has never been measured is whether the screening decisions are correct. There is no human-screened control set to score them against, and the labels are model output. A relevant paper wrongly dropped at screening leaves no trace anywhere in this pipeline.

Until 10 August 2026, IEEE S&P looked like a partly-paywalled venue: 333 of its 780 selected papers were on disk, and the corpus documentation carried a standing caveat that “IEEE S&P is 43% retrieved” and that cross-venue percentages were biased against it. Several pages on this site repeated that caveat.

It was not a paywall. The Xplore PDF is served from stampPDF/getPDF.jsp; stamp.jsp is a viewer shell that only embeds the file. The fetcher scraped the viewer page, found no PDF link, and logged “No PDF found” — a message indistinguishable in the logs from “this paper is behind a paywall”. With institutional access and the correct URL, all 447 remaining papers were retrievable.

Two things are worth taking from this beyond the fixed number.

First, the old caveat is wrong and must be deleted wherever it survives. Any page still saying IEEE S&P is 43% retrieved is stale.

Second, and more usefully for your own crawl: a retrieval failure and an access failure produced the same log line, and it took months to notice. That is the general shape of a systematic loss — it does not announce itself, it looks like a property of the world.

2026 is thin for three compounding reasons, none of them relevance: CCS and IMC 2026 have not been held; USENIX Security 2026 has been selected but not published; and IEEE S&P 2026 (58 abstracts of 252 records) and WWW 2026 (630 of 954) are largely un-indexed. Abstract-based selection cannot select a paper whose abstract is not yet indexed. 2025 is nearly complete on this measure but is still the newest year and the least likely to have been checked by anyone.

The same mechanism produces silence, not thinness, in nine older venue-years. None of these 309 papers has an abstract in the index, so none was screened, so none is in the corpus:

PETS 2010–2014 is absent in its entirety, and PETS is the most privacy-focused venue in the set. If you are looking here for early privacy-measurement methodology and finding little, that is why. Nothing here can be fixed by re-running the metadata build: the pre-PoPETs PETS workshop schedule pages and the NDSS archive pages for those years publish no abstracts at all, so there is nothing to fetch. Closing it means a different retrieval path — reading the abstract off the paper's own first page — which has not been implemented.

Three schema decisions shape every query you can run against it:

• not-stated and none-mentioned are first-class values. The schema asks the model to record silence rather than omit it. This is what lets a page report “of the 1,120 papers that ran a crawl, 53 say anything about robots.txt” instead of quietly dropping the other 1,067.
• usedOrMentioned on anything citable — used (84.3% of tool tuples), produced (9.4%), compared (5.0%), mentioned (1.2%), unclear (0.1%). A baseline the authors compared against reads as adoption if you skip this filter, and the tuples that are not used concentrate in exactly the names a page is about.
• The measured results live in detection[].prevalence — 26,316 of the 27,241 detection tuples (96.6%), across 5,572 papers, carry the paper's own figure. But prevalence is the model's summary of a result, not a quote from it. Grep the full text for any prevalence figure before publishing it, not just the attached quote.

Evidence quotes per family:

Every quote is supposed to be verbatim, so every quote can be looked for in the text the extractor read. All 135,025 were, on 2026-08-12, by scripts/report_corpus.mjs –quotes:

Only the last row counts against the extraction, and even it is mostly not what it looks like. Five “not found” quotes were read against their papers by hand on 2026-08-12. Every one is present in the paper:

A further 15.9% of the not-found quotes contain an ellipsis, meaning the extraction spliced two parts of a source sentence and marked the join — not verbatim, but not invented either.

A practical consequence for anyone checking a quote by hand: normalise whitespace before concluding a quote is absent. grep -F on a multi-word quote fails about half the time against this text because it still wraps mid-sentence; tr -s '[:space:]' ' ' | grep -iF finds them.

The failures are also spread thin rather than concentrated: 1,143 of 5,859 papers (19.5%) have at least one, and the ten worst papers hold only 82 of the 1,708. The rate is flat across sixteen years, between 0.8% and 1.5% in every single year, which is what you would expect of a PDF-layout artefact and not of a model that got worse or better over time.

Two honest caveats about this measurement. It reproduces the dataset's own audit of the earlier corpus almost exactly — that audit reported 57.9% / 37.0% / 4.2% / 0.9% on the same four buckets — but the two implementations are independent and the “fragmented” test in particular is an approximation of the original, so the 4.0%/1.3% split between the last two rows should be read as one combined 5.3% that is mostly layout damage. And the first measurement of this ran the check against the wrong file for seven papers whose text had been OCR-repaired, which put one paper at the top of the not-found ranking with 29 phantom failures; the script now reads whichever rendering the extractor read.

Two independent extraction runs over the same 100 papers were compared to see which fields survive being turned into a percentage. That comparison was run against the 4,322-paper corpus and has not been repeated — repeating it means paying for a second full extraction pass. Treat the figures below as the right order of magnitude, not as current measurements.

The bottom two rows are not the model being unreliable about what a paper says. studyTypes and classification.method are the two fields where a paper genuinely admits several defensible readings — a crawl that also reanalyses a public dataset is honestly both — and multi-valued fields are scored here by exact set equality, so one extra tuple flips them. The 20% row is wording variance: the same finding described as “third-party cookie setting” and “setting of third-party cookies” is one disagreement about English, not two readings of the paper.

Four rules, each with the failure it prevents. If a page on this site breaks one, the figure is wrong.

1. Name the denominator before the numerator. Each query defines its own population:

2. A sentinel is never an answer. ethics.robotsTxt over the 1,120 crawling papers: 53 papers (4.7%) state a real value; 992 (88.6%) hold some value, the rest of which is not-stated. Counting the sentinel turns “one crawling paper in twenty says anything about robots.txt” — which is the finding — into “seven in eight do”.

3. Count papers, never tuples. A paper naming EasyList three times is one paper: 145 EasyList tuples across 94 papers, 1.54 per paper.

4. Fold free-text names before aggregating, and print what the fold could not map. This is the rule that is easiest to skip and most expensive to skip. The worked case is vantage[].locations, which is free text — “Frankfurt, Germany”, “US-East”, “a US-based university network”:

A 42.6% undercount, on the exact quantity such a page is about, and the middle row is what the dataset's own convenience query reports. 308 distinct raw strings fold to the United States: US, USA, U.S., California, New York, Oregon, Los Angeles, US East Coast, Silicon Valley, US-East and 298 more. Case-and-punctuation folding does not touch any of them.

Every fold written for a page on this site therefore returns its unmapped residue, and the residue is printed in full on that page's provenance: page. A residue that exists only inside a local script's output is a residue nobody will ever look at. Folds also age. When the corpus was extended to 2025–2026, fingerprinting's subject fold went from 7 unmapped tuples to 59 with no code change at all (scripts/report_fingerprinting.mjs, not this page's script) — the newer papers fingerprint new things (DPI boxes, LLMs, AR/VR apps). It was extended and is back to 7. A fold that does not print its residue would have absorbed the difference silently.

What you can check. Every figure on a corpus-backed page traces to a sentence in a published paper, and the provenance: page for that content page gives you the query, the population, the denominator, the fold, and the report script's unedited output. Two things follow that you can act on without any access to the corpus:

1. Read the paper. The corpus is public literature. If a page says n papers do something and names them, the papers are the evidence, not the dataset.
2. Check the arithmetic and the population. If a percentage's denominator is not named on the page or on its provenance: page, that is a defect in the page and worth reporting.

What you cannot check. The corpus itself — the venue index, the screening labels, the retrieved PDFs and extractions.jsonl — is a local dataset on the maintainer's machine. It is not published, not downloadable, and there is no API. Two further consequences are worth being blunt about:

• The extraction cannot be reproduced to the same values. It is generative model output; two runs over identical text disagree, which is the whole point of the stability table in Field stability: measured once, on the earlier corpus, never since. Re-running it would produce a similar but not identical dataset.
• The extraction currently exists on one disk with no archive. That is a known, unresolved risk recorded in the dataset's own runbook, and it is stated here rather than left out because a reader deciding how much to lean on these figures should know it.

Every figure on this page comes from one script, which prints all of them with their denominators, plus the checks that the funnel is internally consistent. The output below is what it printed on 2026-08-12, unedited.

node scripts/report_corpus.mjs                     # the funnel, scope, folds and rules
node scripts/report_corpus.mjs --quotes            # + all 135,025 quotes re-checked (~20 s)
node scripts/report_corpus.mjs --quotes --wiki     # the same, as DokuWiki tables
 
# the staleness guard: assert every number on this page appears in the report
node scripts/check_page_numbers.mjs pages/literature_corpus.txt out/new/report_corpus.txt --code

On 2026-08-12 the guard reported exactly two figures on this page that the report cannot produce, both from named sources outside the corpus: the 2,870 papers whose token ledger was lost, which comes from the extraction's own README.md, and the 29 phantom quote failures caused by a bug in an early version of this page's own script, described in 11. Run log. Every other figure on the page, including all four quote-groundedness percentages and the whole funnel, is produced by the script.

The script also prints seven consistency checks on the funnel — retrieved papers outside the selection rule, fulltext directories outside it, extraction records with no prepared text on disk, duplicate records, labelled papers with no abstract, abstracts never screened, and truncated extractor inputs. All seven return zero. A nonzero value in any of them means the funnel above is wrong, not that the corpus is.

report_corpus-output.txt

report_corpus.mjs — every figure on literature:corpus
data root: /workspace/publications_dataset/data
generated: 2026-08-12
 
== 1. Scope ==
 
venues                 7  (CCS, IEEE-SP, IMC, NDSS, PETS, USENIX, WWW)
years                  2010–2026
venue-year files       117
  of a possible        119 (7 venues x 17 years)
 
absent venue-years     2: CCS 2026, IMC 2026
 
  Venue    Venue-years  First  Last  Metadata records
  -------  -----------  -----  ----  ----------------
  CCS      16           2010   2025  3,381           
  IEEE-SP  17           2010   2026  1,837           
  IMC      16           2010   2025  862             
  NDSS     17           2010   2026  1,577           
  PETS     17           2010   2026  1,253           
  USENIX   17           2010   2026  3,012           
  WWW      17           2010   2026  4,942           
 
 
== 2. The funnel ==
 
  Stage                                      N       Lost here  Loss is                                       
  -----------------------------------------  ------  ---------  ----------------------------------------------
  Venue-year metadata records                16,864                                                           
  …with an abstract                          15,800  −1,064     systematic: see the two eras below            
  Screened (one label record each)           15,800  0          none: screening is complete over the abstracts
  Selected by the relevance rule             6,103   −9,697     by design: out of scope                       
  PDF retrieved (>=1 KB)                     5,873   −230       systematic: paywall / not yet published       
  pdftotext produced paper.txt               5,873   0          none                                          
  Text prepared, so eligible for extraction  5,873   0          none                                          
  Structured extraction record               5,859   −14        systematic: IEEE-SP rate-limit tail           
 
Text preparation, in more detail than the funnel row shows:
  paper.txt        5873   (pdftotext -layout, raw, columns side by side)
  paper.cols.txt   5869   (decolumn.mjs; 4 PDFs have no usable text layer)
  paper.norm.txt   5873   (normalize_text.mjs; also carries the OCR text for those 4)
The extractor picks per paper. Of the 5859 records: {"cols":5852,"mistral":7}
 
Selected -> extracted overall: 5859 of 6103 = 96.0%
Metadata -> extracted overall: 5859 of 16864 = 34.7%
 
The 1,064 records with no abstract split into two eras:
  2010-2019 (venue pages and OpenAlex never had them): 443
  2020-2024                                          : 58
  2025-2026 (OpenAlex has the DOI, not yet the abstract): 563
 
Missing abstracts by venue-year, every block of 10 or more:
 
  Venue-year    Records with no abstract
  ------------  ------------------------
  WWW 2026      324                     
  IEEE-SP 2026  194                     
  NDSS 2018     71                      
  NDSS 2016     60                      
  CCS 2011      40                      
  NDSS 2014     32                      
  PETS 2014     30                      
  NDSS 2011     28                      
  CCS 2025      27                      
  PETS 2011     27                      
  PETS 2012     25                      
  NDSS 2010     24                      
  WWW 2024      23                      
  CCS 2024      22                      
  PETS 2010     22                      
  PETS 2013     22                      
  NDSS 2013     20                      
  NDSS 2012     15                      
 
NDSS 2010-2018 block: 260   (venue pages publish no abstract, and NDSS has no DBLP DOIs)
PETS 2010-2014 block: 126   (pre-PoPETs workshop schedule pages publish no abstracts)
Causes per block are documented in data/corpus2/README.md §5; the counts here are re-derived.
 
CHECK  retrieved papers outside the selection rule: 0
CHECK  fulltext directories outside the selection rule: 0
CHECK  extraction records with no paper.norm.txt on disk: 0
CHECK  duplicate extraction records: 0
CHECK  labelled papers with no abstract in metadata: 0
CHECK  abstracts never screened: 0
CHECK  extraction records flagged textTruncated: 0
CHECK  textSource of extractor input: {"cols":5852,"mistral":7}
CHECK  extraction model: {"gpt-5.6-luna":5859}
 
 
== 3. Where the extraction lost papers ==
 
paper.norm.txt on disk but no extraction record: 14
 
  Paper                                                                                          Reason                   
  ---------------------------------------------------------------------------------------------  -------------------------
  USENIX/2017/oss-fuzz-googles-continuous-fuzzing-service-for-open-source-software               no failure logged        
  IEEE-SP/2021/happer-unpacking-android-apps-via-a-hardware-assisted-approach                    520 status code (no body)
  IEEE-SP/2023/three-birds-with-one-stone-efficient-partitioning-attacks-on-interdependent-cryp  520 status code (no body)
  IEEE-SP/2024/a-systematic-study-of-physical-sensor-attack-hardness                             520 status code (no body)
  IEEE-SP/2024/conning-the-crypto-conman-end-to-end-analysis-of-cryptocurrency-based-technical   520 status code (no body)
  IEEE-SP/2024/jasmine-scale-up-javascript-static-security-analysis-with-computation-based-sema  520 status code (no body)
  IEEE-SP/2024/nyx-detecting-exploitable-front-running-vulnerabilities-in-smart-contracts        520 status code (no body)
  IEEE-SP/2024/on-sms-phishing-tactics-and-infrastructure                                        520 status code (no body)
  IEEE-SP/2024/pandora-principled-symbolic-validation-of-intel-sgx-enclave-runtimes              520 status code (no body)
  IEEE-SP/2024/pomabuster-detecting-price-oracle-manipulation-attacks-in-decentralized-finance   520 status code (no body)
  IEEE-SP/2024/pulling-off-the-mask-forensic-analysis-of-the-deceptive-creator-wallets-behind-s  520 status code (no body)
  IEEE-SP/2024/revisiting-automotive-attack-surfaces-a-practitioners-perspective                 520 status code (no body)
  IEEE-SP/2024/routing-attacks-on-cryptocurrency-mining-pools                                    520 status code (no body)
  IEEE-SP/2024/sok-a-comprehensive-analysis-and-evaluation-of-docker-container-attack-and-defen  520 status code (no body)
 
extraction failures logged (retries included): 693 over 559 distinct papers
  Failure reason             Log lines
  -------------------------  ---------
  520 status code (no body)  603      
  Connection error.          64       
  Request timed out.         25       
  terminated                 1        
 
of those 559 papers, 546 were recovered by a retry
 
 
== 4. Funnel by venue ==
 
  Venue    Metadata  Abstract  Screened  Selected  PDF    Extracted  Extracted / selected
  -------  --------  --------  --------  --------  -----  ---------  --------------------
  CCS      3,381     3,290     3,290     1,036     990    990        95.6%               
  IEEE-SP  1,837     1,642     1,642     780       780    767        98.3%               
  IMC      862       854       854       691       638    638        92.3%               
  NDSS     1,577     1,317     1,317     701       701    701        100.0%              
  PETS     1,253     1,113     1,113     510       510    510        100.0%              
  USENIX   3,012     2,997     2,997     1,520     1,411  1,410      92.8%               
  WWW      4,942     4,587     4,587     865       843    843        97.5%               
 
 
== 5. Funnel by year — where 2025 and 2026 are thin ==
 
  Year                Venue-years  Metadata  Abstract  Abstract share  Selected  Extracted
  ------------------  -----------  --------  --------  --------------  --------  ---------
  2010                7            496       450       90.7%           123       119      
  2011                7            352       257       73.0%           119       116      
  2012                7            426       384       90.1%           178       151      
  2013                7            484       442       91.3%           141       125      
  2014                7            526       460       87.5%           169       166      
  2015                7            573       566       98.8%           193       190      
  2016                7            620       557       89.8%           192       182      
  2017                7            686       680       99.1%           237       231      
  2018                7            711       638       89.7%           260       254      
  2019                7            1,002     999       99.7%           407       402      
  2020                7            953       952       99.9%           414       404      
  2021                7            1,167     1,166     99.9%           386       379      
  2022                7            1,348     1,347     99.9%           556       546      
  2023                7            1,598     1,595     99.8%           721       719      
  2024                7            1,901     1,849     97.3%           712       690      
  2025 (provisional)  7            2,032     1,992     98.0%           774       770      
  2026 (provisional)  5            1,989     1,466     73.7%           521       415      
 
Venue-years with abstract coverage below 90% (a paper with no abstract is never screened,
so it can never be selected — those venue-years are under-selected by construction):
 
  Venue-year    Metadata  With abstract  Share
  ------------  --------  -------------  -----
  NDSS 2010     24        0              0.0% 
  NDSS 2011     28        0              0.0% 
  NDSS 2016     60        0              0.0% 
  NDSS 2018     71        0              0.0% 
  PETS 2010     22        0              0.0% 
  PETS 2011     27        0              0.0% 
  PETS 2012     25        0              0.0% 
  PETS 2013     22        0              0.0% 
  PETS 2014     30        0              0.0% 
  IEEE-SP 2026  252       58             23.0%
  NDSS 2014     78        46             59.0%
  CCS 2011      102       62             60.8%
  WWW 2026      954       630            66.0%
  NDSS 2013     59        39             66.1%
  NDSS 2012     54        39             72.2%
 
Venue-years contributing ZERO papers to the extracted corpus: 9
These are holes, not thin spots. The wiki is silent about them by construction.
 
  Venue-year  Metadata records  With abstract  Selected
  ----------  ----------------  -------------  --------
  NDSS 2010   24                0              0       
  NDSS 2011   28                0              0       
  NDSS 2016   60                0              0       
  NDSS 2018   71                0              0       
  PETS 2010   22                0              0       
  PETS 2011   27                0              0       
  PETS 2012   25                0              0       
  PETS 2013   22                0              0       
  PETS 2014   30                0              0       
 
 
== 6. IEEE S&P — a systematic loss, found and repaired ==
 
IEEE-SP selected      780
IEEE-SP PDF retrieved 780  (100.0% of selected)
IEEE-SP extracted     767  (98.3% of selected)
IEEE-SP not retrieved 0
 
  Year  IEEE-SP selected but no PDF
  ----  ---------------------------
 
The retrieval gap that was repaired on 2026-08-10: the Xplore PDF is at
stampPDF/getPDF.jsp, not the stamp.jsp viewer shell, so scraping the viewer
found no PDF link and logged "No PDF found" — indistinguishable from a paywall.
(data/RUNBOOK_new_conference_year.md §3.)
 
Every other venue, for comparison — selected papers with no PDF on disk:
 
  Venue    Selected  No PDF  Share missing
  -------  --------  ------  -------------
  CCS      1036      46      4.4%         
  IEEE-SP  780       0       0.0%         
  IMC      691       53      7.7%         
  NDSS     701       0       0.0%         
  PETS     510       0       0.0%         
  USENIX   1520      109     7.2%         
  WWW      865       22      2.5%         
 
The retrieval loss is spread over 37 venue-years. The 10 largest:
 
  Venue-year   Selected but no PDF
  -----------  -------------------
  USENIX 2026  104                
  IMC 2012     22                 
  IMC 2013     12                 
  WWW 2024     8                  
  IMC 2020     6                  
  CCS 2018     6                  
  WWW 2022     5                  
  CCS 2016     4                  
  CCS 2012     4                  
  CCS 2021     4                  
 
top 6 venue-years: 158 papers
remaining 31 venue-years: 72 papers
 
 
== 7. What the extraction is ==
 
records                5,859, one per paper, keyed on (venue, year, slug)
model                  gpt-5.6-luna
passes                 1 (single pass; no adjudication, no second coder)
input                  paper.norm.txt, whole paper text
relation families      13: tools, population, crawlConfig, classification, detection, vantage, temporal, statistics, humanAnnotation, participants, ethics, artifacts, legal
 
evidence quotes        135,025 across 5,859 papers
  Family           Tuples carrying a quote
  ---------------  -----------------------
  tools            37,359                 
  detection        27,241                 
  population       17,016                 
  classification   11,330                 
  statistics       8,787                  
  temporal         8,775                  
  humanAnnotation  5,778                  
  artifacts        5,524                  
  ethics           4,800                  
  vantage          4,685                  
  participants     2,018                  
  crawlConfig      1,080                  
  legal            632                    
 
 
== 7b. Populations — the denominators a page may use ==
 
  Population     Definition                               Papers  Share of 5859
  -------------  ---------------------------------------  ------  -------------
  sampled        drew a study population                  5,712   97.5%        
  empirical      measured the world as it is              5,118   87.4%        
  classified     classified or labelled something         4,439   75.8%        
  measuredFrom   took a measurement from a vantage point  3,908   66.7%        
  annotated      coded data by hand                       3,318   56.6%        
  inferential    ran statistical inference                1,762   30.1%        
  web            measured the web platform                1,622   27.7%        
  humanSubjects  recruited human participants             1,357   23.2%        
  crawled        ran an automated web crawl               1,120   19.1%        
  legal          assessed compliance with a law           402     6.9%         
 
 
== 8. Folding is not optional — the vantage/United States case ==
 
papers whose vantage.locations contains the exact string "United States": 238
…folded by the dataset's name_fold.mjs (what site_queries.mjs reports):   367
…folded on meaning (scripts/geo.mjs):                                     639
undercount from name_fold alone: 272 papers = 42.6% of the folded total
distinct raw strings that fold to the United States: 308
 
The 25 most common of them:
 
  Raw string               Tuples
  -----------------------  ------
  United States            262   
  US                       62    
  USA                      51    
  California               25    
  U.S.                     22    
  New York                 16    
  Oregon                   10    
  Los Angeles              9     
  New York City            9     
  Virginia                 7     
  San Francisco            6     
  Ohio                     6     
  Seattle                  5     
  Chicago                  5     
  Atlanta                  5     
  New Jersey               4     
  America                  4     
  US East Coast            4     
  Boston, Massachusetts    4     
  Silicon Valley           4     
  California, USA          4     
  Illinois                 3     
  Boston, MA, USA          3     
  Northeast United States  3     
  US-East                  3     
 
 
== 8b. The other two rules, as numbers ==
 
Sentinels are never answers — ethics.robotsTxt over the papers that ran a crawl:
  population (crawled)                       1120
  ethics object fired at all                 992
  robotsTxt holds a real value               53 = 4.7%
  robotsTxt holds any value incl. sentinels  992 = 88.6%
  Counting the sentinel as an answer turns "one crawling paper in twenty says anything
  about robots.txt" into "seven in eight do".
 
Papers, never tuples — three names, counted both ways:
 
  Exact name  Tuples  Papers  Tuples per paper
  ----------  ------  ------  ----------------
  EasyList    145     94      1.54            
  OpenWPM     61      61      1.00            
  Selenium    238     237     1.00            
 
Filter usedOrMentioned — tool tuples by relationship:
 
  usedOrMentioned  Tuples  Share
  ---------------  ------  -----
  used             31,496  84.3%
  produced         3,528   9.4% 
  compared         1,876   5.0% 
  mentioned        438     1.2% 
  unclear          21      0.1% 
 
Where the measured results are — detection[].prevalence:
  detection tuples                 27,241 across 5,655 papers
  …carrying a prevalence figure    26,316 across 5,572 papers = 96.6% of tuples
 
 
== 9. Quote groundedness ==
 
papers checked         5,859 (whole corpus)
quotes checked         135,025
quotes skipped (the extractor's source file is not on disk): 0
 
  Verdict                                                        Quotes  Share of checked
  -------------------------------------------------------------  ------  ----------------
  exact, after collapsing whitespace and line-break hyphenation  78,450  58.1%           
  an 8-word run survives                                         49,425  36.6%           
  fragmented — every content word in one window, order broken    5,442   4.0%            
  not found                                                      1,708   1.3%            
 
of the 1,708 not-found quotes, 271 (15.9%) contain an ellipsis,
i.e. the extraction spliced two parts of the source and marked the join.
 
papers with at least one not-found quote: 1143 of 5859 = 19.5%
the 10 worst papers hold 82 of the 1708 not-found quotes:
 
  Paper                                                                                                Not-found quotes
  ---------------------------------------------------------------------------------------------------  ----------------
  CCS/2017/how-unique-is-your-onion-an-analysis-of-the-fingerprintability-of-tor-onion-serv            12              
  NDSS/2019/countering-malicious-processes-with-process-dns-association                                10              
  WWW/2024/identifying-risky-vendors-in-cryptocurrency-p2p-marketplaces                                9               
  IEEE-SP/2024/smartinv-multimodal-learning-for-smart-contract-invariant-inference                     8               
  USENIX/2024/attention-please-an-investigation-of-the-app-tracking-transparency-permission            8               
  CCS/2019/charting-the-attack-surface-of-trigger-action-iot-platforms                                 7               
  IMC/2019/measuring-ewhoring                                                                          7               
  NDSS/2026/rethinking-fake-speech-detection-a-generalized-framework-leveraging-spectrogram-magnitude  7               
  PETS/2018/panoptispy-characterizing-audio-and-video-exfiltration-from-android-applications           7               
  WWW/2026/anomaly-detection-of-interaction-behaviors-in-streaming-graphs                              7               
 
Not-found rate by year — the text pipeline is the same throughout, so a trend here
is about the PDFs, not about the extraction:
 
  Year  Quotes checked  Not found  Rate
  ----  --------------  ---------  ----
  2010  2,212           21         0.9%
  2011  2,416           21         0.9%
  2012  2,938           34         1.2%
  2013  2,485           20         0.8%
  2014  3,521           40         1.1%
  2015  4,189           49         1.2%
  2016  4,121           53         1.3%
  2017  5,319           80         1.5%
  2018  5,829           79         1.4%
  2019  9,225           103        1.1%
  2020  9,619           92         1.0%
  2021  9,157           96         1.0%
  2022  12,724          166        1.3%
  2023  16,310          245        1.5%
  2024  16,007          208        1.3%
  2025  18,686          243        1.3%
  2026  10,267          158        1.5%
 
Up to 25 not-found quotes, for hand-checking:
 
  Paper                                                                                      Family           Quote (truncated)                                                                                             
  -----------------------------------------------------------------------------------------  ---------------  --------------------------------------------------------------------------------------------------------------
  CCS/2010/a-methodology-for-empirical-analysis-of-permission-based-security-models-and-its  detection        applications can be assigned to the nearest neuron, effectively clustering the applications requesting similar
  CCS/2010/detecting-and-characterizing-social-spam-campaigns                                vantage          We carry out the study on Facebook ... We choose 8 regional networks ... as targets for data collection.      
  CCS/2010/fingerprinting-websites-using-remote-traffic-analysis                             population       The results are shown in Table 1. Classification accuracy ... 12 ...                                          
  CCS/2010/fingerprinting-websites-using-remote-traffic-analysis                             population       The results are shown in Table 1. Classification accuracy ... 24 ...                                          
  CCS/2011/how-to-tell-if-your-cloud-files-are-vulnerable-to-drive-crashes                   population       We experimented with drives from Fujitsu, Hitachi, HP, and Seagate.                                           
  CCS/2011/ispy-automatic-reconstruction-of-typed-input-from-compromising-reflections        tools            For this we use the Scale Invariant Feature Transform, or SIFT [22].                                          
  CCS/2011/waptec-whitebox-analysis-of-web-applications-for-parameter-tampering-exploit-con  artifacts        A detailed report of exploits found by our approach can be found at http://sisl.rites.uic.edu/waptec.         
  CCS/2012/aligot-cryptographic-function-identification-in-obfuscated-binary-programs        tools            Among all tracing tools available in such environment, we chose Pin, the dynamic binary instrumentation framew
  CCS/2012/authenticated-secret-key-extraction-using-channel-characteristics-for-body-area   tools            This becomes a generalization of single-source single-sink maximum-flow problem. Therefore, we let each sensor
  CCS/2012/cross-vm-side-channels-and-their-use-to-extract-private-keys                      temporal         The data collection lasted about six hours, during which roughly 1000 key-related fragments were recovered fro
  CCS/2012/peerpress-utilizing-enemies-p2p-strength-against-them                             population       We built one virtual environment with six virtual machines. All virtual machines installed Windows XP SP1 with
  CCS/2012/peerpress-utilizing-enemies-p2p-strength-against-them                             vantage          We built one virtual environment with six virtual machines.                                                   
  CCS/2012/populated-ip-addresses-classification-and-applications                            tools            We use the LIBSVM [8] and LIBLINEAR [11] toolkits in our implementation.                                      
  CCS/2012/protecting-location-privacy-optimal-strategy-against-localization-attacks         temporal         The length of the considered traces is one month.                                                             
  CCS/2012/sabot-specification-based-payload-generation-for-programmable-logic-controllers   ethics           We evaluate S ABOT using four metrics: Accuracy ... Adaptability ... Performance ... Scalability              
  CCS/2013/beheading-hydras-performing-effective-botnet-takedowns                            classification   Our heuristics successfully classified 97% of the samples' contingency plans correctly.                       
  CCS/2013/chucky-exposing-missing-checks-in-source-code-for-vulnerability-discovery         tools            The parser is based on an island grammar [see 20] for the parser generator ANTLR [25].                        
  CCS/2013/delta-automatic-identification-of-unknown-web-based-infection-campaigns           vantage          via a 10-node cluster of custom crawlers running an adaptive fetch schedule                                   
  CCS/2013/protocol-misidentification-made-easy-with-format-transforming-encryption          detection        The resulting regexes achieved perfect misclassification for all classifier/target protocol combinations, as s
  CCS/2013/the-robustness-of-hollow-captchas                                                 tools            We use Lee's algorithm [10] to automatically detect and then repair broken contours.                          
  CCS/2013/users-get-routed-traffic-correlation-on-tor-by-realistic-adversaries              temporal         BGP paths gathered during March 2013 ... CAIDA ... from December 2012 ... CAIDA AS Relationships Dataset for J
  CCS/2013/viceroi-catching-click-spam-in-search-ad-networks                                 humanAnnotation  Working with the ad network we manually investigated around hundred websites associated with the publishers we
  CCS/2014/harvesting-high-value-foreign-currency-transactions-from-emv-contactless-credit   detection        Average transaction duration (card discovery to transaction approval) 478ms Standard deviation 36ms Fastest tr
  CCS/2014/nothing-else-mator-s-monitoring-the-anonymity-of-tors-path-selection              artifacts        The monitor programs are available on our website [1].                                                        
  CCS/2014/poster-positioning-attack-on-proximity-based-people-discovery                     tools            Android emulator(s) installed with mock location provider and customized plug-ins for various PBPD apps       
 
 
== 9b. Field stability — REPRODUCED, NOT RE-MEASURED ==
 
The block below is copied verbatim from data/extract/README.md. It was measured by
comparing two independent extraction runs over the same 100 papers, ON THE EARLIER
4,322-PAPER CORPUS. Re-measuring it needs a second full extraction pass and that has
not been done. It is printed here so that every figure the page carries has a source
in this file — not because this script computed it.
 
  Agreement  Fields                                                                                                                                                                                                                                  
  ---------  ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  90-100%    has-authors'-own-link (100), crawlConfig fired (99), .statefulness (98), .interactionDepth (97), participants fired (96), isEmpirical (95), legal.law (95), humanAnnotation fired (94), .consentAction (93), artifacts.availability (90)
  65-80%     platforms (77), ethics.robotsTxt (76), temporal.mode (69), vantage.infrastructure (69), ethics.reviewOutcome (68), statistics.kind (68), ethics.notifiedAffectedParties (67)                                                            
  under 60%  classification.method (58), studyTypes (57)                                                                                                                                                                                             
  about 20%  free-text names: detection.phenomenon, classification.resourceName, statistics.method                                                                                                                                                   
 
The same file reports the earlier corpus-wide quote audit as:
  exact 57.9%  |  8-word run 37.0%  |  fragmented 4.2%  |  not found 0.9%
Section 9 above re-measures that on the current corpus with an independent
implementation. Compare the two before quoting either.
 
 
== 9c. Figures the page derives by arithmetic ==
 
Each of these appears on the page and is a combination of two figures above.
They are printed so the staleness guard can account for them.
 
crawling papers that say NOTHING about robots.txt   1120 - 53 = 1067
IEEE-SP 2024 tail as a share of the venue           13 / 780 = 1.7%
USENIX 2026 share of the retrieval loss             104 / 230 = 45.2%
US-folding raw strings beyond the 10 listed         308 - 10 = 298
papers in the nine zero-contribution venue-years    309
IEEE-SP retrieval before the 2026-08-10 repair      780 - 447 = 333 on disk = 42.7%
  (447 = line count of data/fulltext/missing_ieee_all.jsonl, the work list for that repair)
quote verdicts that are layout damage rather than exact  4.0% + 1.3% = 5.3%
vantage location stated, the page's opening example 1228 of 3908 = 31.4%
 
 
== 10. What this script does not measure ==
 
- Run-to-run agreement of the extraction. That needs a second extraction pass over
  the same text; data/extract/stability100/ holds one, but it was run against the
  4,322-paper corpus and has not been repeated. Every agreement figure quoted on the
  page is from that older run and is labelled as such.
- Whether a selection decision was correct. The screening labels are model output and
  data/extract/README.md says plainly they are not ground truth; there is no
  human-screened control set to score them against.
- Whether a paper that produced no tuple for a family is silent or was missed.

Stated plainly, because a provenance page that overstates its own rigour is worse than none.

• Whether the screening is accurate. No human-screened control set exists. A relevant paper wrongly dropped at abstract screening leaves no trace in any artefact, so the false-negative rate of selection is unknown and unknowable from what is on disk. This is the largest unquantified risk in the funnel and it sits at the widest step of it.
• Whether field stability has moved. The agreement figures in §6 are from the 4,322-paper corpus. Re-measuring means a second full extraction pass; it has not been done, and every page quoting those figures should say so.
• Whether the 14 unextracted papers matter. Twelve are IEEE S&P 2024. Nobody has read them to see whether they would have changed anything.
• Whether the 309 papers in the nine empty venue-years matter. The cause is established and documented — those venue pages publish no abstracts — but nobody has read the 309 titles to see how much relevant methodology is sitting outside the corpus. That is a cheap check nobody has done.
• The per-run token and cost ledger. The extraction's own accounting was lost for roughly the first 2,870 papers of the original run because the ledger was written only on clean exit and every restart killed the process first. The total is a measured tail plus an extrapolation, and it is not re-derivable.
• Anything about a venue that is not one of the seven. This is not a limitation that better tooling fixes; it is the scope.

Working notes for individual pages are under provenance:; see The provenance namespace.