| Both sides previous revisionPrevious revision | |
| programming:crawler [2026/08/12 10:05] – Review pass (Fable): fix stale figures that sat OUTSIDE the corpus section and so were missed by the windowed staleness guard. 11.6->12.0%, 21.8->21.6%, 6.3->6.8%, 4.1->3.8%, 12.9->12.5%, 34->75 unnamed bespoke crawlers; Playwright rewritten (10 -> 34 pap karel.kubicek.claude | programming:crawler [2026/08/12 15:45] (current) – Methodology section: replaced the generic corpus caveats (absent venues, provisional 2025-2026) with a pointer to the new literature:corpus page, keeping the page-specific consequence. Authored by Claude. karel.kubicek.claude |
|---|
| * **''used'' and ''produced'' both count** as driving a crawl — a paper that built its own crawler crawled with it. Tools only //compared// or //cited// do not, which is why the last table separates the two. | * **''used'' and ''produced'' both count** as driving a crawl — a paper that built its own crawler crawled with it. Tools only //compared// or //cited// do not, which is why the last table separates the two. |
| * **Quotes were spot-checked**, by ''scripts/quote_check.mjs'' rather than by hand. Of the **106** evidence quotes attached to an OpenWPM or Playwright tool mention anywhere in the corpus, 58 match the source text exactly once whitespace and line-break hyphenation are normalised, and 32 more match on at least 60% of their five-word windows. The remaining 16 were then read by hand against the full text: **all sixteen are present in the paper and none was unsupported.** The mismatch is always either a bracketed citation marker the extraction dropped (''the Open-WPM platform [52] on the Firefox browser'') or the two-column reading order still interleaving mid-sentence (''we created a Playwright**constrains the valid child elements, and everything else is moved based** [18] crawler''). | * **Quotes were spot-checked**, by ''scripts/quote_check.mjs'' rather than by hand. Of the **106** evidence quotes attached to an OpenWPM or Playwright tool mention anywhere in the corpus, 58 match the source text exactly once whitespace and line-break hyphenation are normalised, and 32 more match on at least 60% of their five-word windows. The remaining 16 were then read by hand against the full text: **all sixteen are present in the paper and none was unsupported.** The mismatch is always either a bracketed citation marker the extraction dropped (''the Open-WPM platform [52] on the Firefox browser'') or the two-column reading order still interleaving mid-sentence (''we created a Playwright**constrains the valid child elements, and everything else is moved based** [18] crawler''). |
| * **Venue coverage.** EuroS&P, ACSAC, RAID, AsiaCCS, CHI and SOUPS are absent entirely; any claim here is a claim about seven venues. Notably, several of the tool papers this page recommends were published //outside// them — Krumnow et al. at CoNEXT, Klein et al. at EuroS&P, Libert in a communication journal — so a tool's paper count in this table is a lower bound on its standing. | * **Venue coverage.** Seven venues only; the scope, the selection funnel and what each stage of it costs are on [[literature:corpus]]. Notably, several of the tool papers this page recommends were published //outside// them — Krumnow et al. at CoNEXT, Klein et al. at EuroS&P, Libert in a communication journal — so a tool's paper count in this table is a lower bound on its standing. |
| * **''crawled'' is defined** as a paper whose crawl configuration was recorded or whose study types include an automated web crawl (1,120 papers, 19.1% of the corpus). This corpus is seven broad security venues, not a web-measurement corpus, so shares of all 5,859 papers would be meaningless here. | * **''crawled'' is defined** as a paper whose crawl configuration was recorded or whose study types include an automated web crawl (1,120 papers, 19.1% of the corpus). This corpus is seven broad security venues, not a web-measurement corpus, so shares of all 5,859 papers would be meaningless here. |
| |
