| Both sides previous revisionPrevious revision | |
| provenance:programming:crawler_detection [2026/09/17 10:45] – Record consentAction schema clarification; Authored by Claude karel.kubicek.claude | provenance:programming:crawler_detection [2026/09/21 13:42] (current) – Quote-check refresh 2026-09-21: regenerated cd_quotecheck and report_crawler_detection output blocks; 13 below threshold -> 11 rescued + 2 below in both. Authored by Claude karel.kubicek.claude |
|---|
| ============================================================================== | ============================================================================== |
| scripts/cd_quotecheck.mjs, last line of its output: | scripts/cd_quotecheck.mjs, last line of its output: |
| 161 quotes: 111 exact, 37 partial, 13 below threshold | 161 quotes: 111 exact, 37 partial, 11 rescued from the PDF, 2 below threshold in BOTH renderings |
| Below-threshold is NOT "unsupported": every one was read by hand and all are | Below-threshold is NOT "unsupported": every one was read by hand and all are |
| present in the paper, spliced by the two-column reading order. | present in the paper, spliced by the two-column reading order. |
| ### verdicts_all.json | ### verdicts_all.json |
| partial CCS/2010/spam-the-underground-on-140-characters-or-less 93% of 15 windows | partial CCS/2010/spam-the-underground-on-140-characters-or-less 93% of 15 windows |
| MISSING CCS/2011/text-based-captcha-strengths-and-weaknesses 45% of 22 windows | RESCUED CCS/2011/text-based-captcha-strengths-and-weaknesses stored=45% pdf=82% |
| "Many websites use CAPTCHAs, or Completely Automated Public Turing tests to tell Computers and Humans Apart, in an attempt to block automated" | |
| partial CCS/2016/protecting-insecure-communications-with-topology-aware-network-tunnels 76% of 17 windows | partial CCS/2016/protecting-insecure-communications-with-topology-aware-network-tunnels 76% of 17 windows |
| MISSING CCS/2018/yet-another-text-captcha-solver-a-generative-adversarial-network-based-approach 56% of 9 windows | RESCUED CCS/2018/yet-another-text-captcha-solver-a-generative-adversarial-network-based-approach stored=56% pdf=100% |
| "The real captchas are automatically collected using a web crawler written in Python." | |
| partial CCS/2022/im-spartacus-no-im-spartacus-proactively-protecting-users-from-phishing-by-inten 61% of 28 windows | partial CCS/2022/im-spartacus-no-im-spartacus-proactively-protecting-users-from-phishing-by-inten 61% of 28 windows |
| partial IEEE-SP/2014/hunting-the-red-fox-online-understanding-and-detection-of-mass-redirect-script-i 67% of 9 windows | partial IEEE-SP/2014/hunting-the-red-fox-online-understanding-and-detection-of-mass-redirect-script-i 67% of 9 windows |
| MISSING IEEE-SP/2019/measuring-and-analyzing-search-engine-poisoning-of-linguistic-collisions 55% of 11 windows | RESCUED IEEE-SP/2019/measuring-and-analyzing-search-engine-poisoning-of-linguistic-collisions stored=55% pdf=100% |
| "To ensure that the search engine servers would not be overloaded, we rate-limited our crawlers." | |
| partial IMC/2015/who-is-com-learning-to-parse-whois-records 80% of 10 windows | partial IMC/2015/who-is-com-learning-to-parse-whois-records 80% of 10 windows |
| partial IMC/2018/how-tracking-companies-circumvented-ad-blockers-using-websockets 92% of 49 windows | partial IMC/2018/how-tracking-companies-circumvented-ad-blockers-using-websockets 92% of 49 windows |
| partial IMC/2023/ecn-with-quic-challenges-in-the-wild 63% of 32 windows | partial IMC/2023/ecn-with-quic-challenges-in-the-wild 63% of 32 windows |
| partial IMC/2025/canvassing-the-fingerprinters-characterizing-canvas-fingerprinting-use-across-th 86% of 29 windows | partial IMC/2025/canvassing-the-fingerprinters-characterizing-canvas-fingerprinting-use-across-th 86% of 29 windows |
| MISSING NDSS/2019/how-bad-can-it-git-characterizing-secret-leakage-in-public-github-repositories 46% of 13 windows | RESCUED NDSS/2019/how-bad-can-it-git-characterizing-secret-leakage-in-public-github-repositories stored=46% pdf=100% |
| "we can run all queries every thirty minutes within the rate limits using a single API key." | RESCUED NDSS/2020/deceptive-previews-a-study-of-the-link-preview-trustworthiness-in-social-platforms stored=50% pdf=100% |
| MISSING NDSS/2020/deceptive-previews-a-study-of-the-link-preview-trustworthiness-in-social-platforms 50% of 2 windows | RESCUED NDSS/2025/scammagnifier-piercing-the-veil-of-fraudulent-shopping-website-campaigns stored=35% pdf=81% |
| "posting prevented due to bot detection" | |
| MISSING NDSS/2025/scammagnifier-piercing-the-veil-of-fraudulent-shopping-website-campaigns 35% of 26 windows | |
| "During the development of our framework, SCAMMAGNIFIER, we faced several challenges including dealing with diverse fraudulent shopping websi" | |
| MISSING PETS/2017/topics-of-controversy-an-empirical-analysis-of-web-censorship-lists 56% of 9 windows | MISSING PETS/2017/topics-of-controversy-an-empirical-analysis-of-web-censorship-lists 56% of 9 windows |
| "For instance, LinkedIn blocked us from accessing user profile pages and job listings." | "For instance, LinkedIn blocked us from accessing user profile pages and job listings." |
| partial PETS/2024/automatic-generation-of-web-censorship-probe-lists 74% of 23 windows | partial PETS/2024/automatic-generation-of-web-censorship-probe-lists 74% of 23 windows |
| partial USENIX/2015/measuring-the-longitudinal-evolution-of-the-online-anonymous-marketplace-ecosyst 87% of 30 windows | partial USENIX/2015/measuring-the-longitudinal-evolution-of-the-online-anonymous-marketplace-ecosyst 87% of 30 windows |
| MISSING USENIX/2017/characterizing-the-nature-and-dynamics-of-tor-exit-blocking 56% of 18 windows | RESCUED USENIX/2017/characterizing-the-nature-and-dynamics-of-tor-exit-blocking stored=56% pdf=100% |
| "Websites in these categories are also observed to be the most aggressive-with 50% of them blocking over 60% of the chosen relays." | RESCUED USENIX/2022/measurement-by-proxy-on-the-accuracy-of-online-marketplace-measurements stored=46% pdf=100% |
| MISSING USENIX/2022/measurement-by-proxy-on-the-accuracy-of-online-marketplace-measurements 46% of 13 windows | |
| "On session start, we provided the scraper with a session cookie manually obtained after solving a CAPTCHA." | |
| partial USENIX/2022/the-security-lottery-measuring-client-side-web-security-inconsistencies 72% of 18 windows | partial USENIX/2022/the-security-lottery-measuring-client-side-web-security-inconsistencies 72% of 18 windows |
| partial USENIX/2023/a-study-of-multi-factor-and-risk-based-authentication-availability 70% of 27 windows | partial USENIX/2023/a-study-of-multi-factor-and-risk-based-authentication-availability 70% of 27 windows |
| partial USENIX/2025/are-captchas-still-bot-hard-generalized-visual-captcha-solving-with-agentic-visi 93% of 15 windows | partial USENIX/2025/are-captchas-still-bot-hard-generalized-visual-captcha-solving-with-agentic-visi 93% of 15 windows |
| partial USENIX/2025/beyond-exploit-scanning-a-functional-change-driven-approach-to-remote-software-v 86% of 14 windows | partial USENIX/2025/beyond-exploit-scanning-a-functional-change-driven-approach-to-remote-software-v 86% of 14 windows |
| MISSING USENIX/2025/s-mine-collecting-and-analyzing-s-mime-certificates-at-scale 56% of 18 windows | RESCUED USENIX/2025/s-mine-collecting-and-analyzing-s-mime-certificates-at-scale stored=56% pdf=100% |
| "except in 2024, when we found a new server that provided over a million SM2 certificates until they blocked our IP address." | |
| partial USENIX/2025/the-ransomware-decade-the-creation-of-a-fine-grained-dataset-and-a-longitudinal 73% of 15 windows | partial USENIX/2025/the-ransomware-decade-the-creation-of-a-fine-grained-dataset-and-a-longitudinal 73% of 15 windows |
| partial WWW/2018/crimebb-enabling-cybercrime-research-on-underground-forums-at-scale 93% of 15 windows | partial WWW/2018/crimebb-enabling-cybercrime-research-on-underground-forums-at-scale 93% of 15 windows |
| no-quote WWW/2022/reproducibility-and-replicability-of-web-measurement-studies (offtopic) | no-quote WWW/2022/reproducibility-and-replicability-of-web-measurement-studies (offtopic) |
| partial WWW/2023/online-advertising-in-ukraine-and-russia-during-the-2022-russian-invasion 79% of 19 windows | partial WWW/2023/online-advertising-in-ukraine-and-russia-during-the-2022-russian-invasion 79% of 19 windows |
| MISSING WWW/2024/navigating-the-post-api-dilemma 58% of 24 windows | RESCUED WWW/2024/navigating-the-post-api-dilemma stored=58% pdf=100% |
| "Moreover, given the severity and the impact of the topic surrounding COVID19, search engines may have placed stricter moderation policies on" | |
| partial WWW/2025/welcome-to-the-dark-side-analyzing-the-revenue-flows-of-fraud-in-the-online-ad-e 63% of 16 windows | partial WWW/2025/welcome-to-the-dark-side-analyzing-the-revenue-flows-of-fraud-in-the-online-ad-e 63% of 16 windows |
| MISSING WWW/2025/whats-in-phishers-a-longitudinal-study-of-security-configurations-in-phishing-we 52% of 25 windows | RESCUED WWW/2025/whats-in-phishers-a-longitudinal-study-of-security-configurations-in-phishing-we stored=52% pdf=100% |
| "This approach provides a comprehensive view of the phishing webpages by fully rendering all client-side resources and helps evade anti-bot t" | |
| |
| ### recall_verdicts.json | ### recall_verdicts.json |
| partial USENIX/2026/network-level-prompt-and-trait-leakage-in-local-research-agents 72% of 18 windows | partial USENIX/2026/network-level-prompt-and-trait-leakage-in-local-research-agents 72% of 18 windows |
| |
| 161 quotes: 111 exact, 37 partial, 13 below threshold | 161 quotes: 111 exact, 37 partial, 11 rescued from the PDF, 2 below threshold in BOTH renderings |
| </file> | </file> |
| |
| All 13 below-threshold quotes were read by hand against ''paper.cols.txt''. **All 13 are | **Re-run 2026-09-21 with the PDF fallback: 11 of the 13 are rescued, 2 are below threshold in both renderings.** See //Quote-check refresh, 2026-09-21// at the foot of this page. The old figure was **13 below threshold**; 11 of those 13 are located in an independent ''pypdf'' rendering of the same ''paper.pdf'' and are a defect in the stored text rather than in the extraction. ''exact'' (111) and ''partial'' (37) did not move. |
| present in the paper.** The mismatch is always the two-column reading order splicing | |
| | All 13 of the then-below-threshold quotes were read by hand against ''paper.cols.txt''. **All 13 are |
| | present in the paper**, a hand result the fallback independently confirms for 11 of them. The mismatch is always the two-column reading order splicing |
| another column through the sentence. Three were rewritten after that reading, and the | another column through the sentence. Three were rewritten after that reading, and the |
| rewrite is recorded in the verdict file's ''note'' field: | rewrite is recorded in the verdict file's ''note'' field: |
| | ''node scripts/check_page_numbers.mjs'' (windowed **and** whole-page, with ''%%--code%%'') | OK | | | ''node scripts/check_page_numbers.mjs'' (windowed **and** whole-page, with ''%%--code%%'') | OK | |
| | ''node scripts/check_attributions.mjs'' | OK — 19 attributions | | | ''node scripts/check_attributions.mjs'' | OK — 19 attributions | |
| | ''node scripts/cd_quotecheck.mjs'' | 161 quotes: 111 exact, 37 partial, 13 read by hand | | | ''node scripts/cd_quotecheck.mjs'' | 161 quotes: 111 exact, 37 partial, 11 rescued from the PDF, 2 below threshold in both renderings. All 13 of the pre-fallback below-threshold quotes were read by hand (2026-08-29); re-run 2026-09-21 | |
| | ''node scripts/verify_crawler_detection_figures.mjs'' | 22 per-paper figures checked against the papers, 0 not found | | | ''node scripts/verify_crawler_detection_figures.mjs'' | 22 per-paper figures checked against the papers, 0 not found | |
| | ''bash scripts/external_checks_crawler_detection.sh'' | 0 FAILED | | | ''bash scripts/external_checks_crawler_detection.sh'' | 0 FAILED | |
| |
| **Rejected finding.** No schema value was overwritten with 55; the audited number is separately defined and published by [[privacy:consent]]. | **Rejected finding.** No schema value was overwritten with 55; the audited number is separately defined and published by [[privacy:consent]]. |
| | |
| | ===== Quote-check refresh, 2026-09-21 ===== |
| | |
| | The 2026-09-04 ''cols''-vs-PDF audit on [[:provenance:literature:corpus]] showed that 73.1% of evidence quotes that cannot be located in ''paper.cols.txt'' **are** present in an independent ''pypdf'' rendering of the same ''paper.pdf''. ''cd_quotecheck.mjs'' already carried the fallback; what was stale was this page's copy of its output, which came from the 2026-08-29 run. Re-run, last line, unedited: |
| | |
| | <code> |
| | $ node scripts/cd_quotecheck.mjs |
| | 161 quotes: 111 exact, 37 partial, 11 rescued from the PDF, 2 below threshold in BOTH renderings |
| | </code> |
| | |
| | ^ Figure ^ Was ^ Is ^ Why ^ |
| | | quotes checked | 161 | 161 | population unchanged | |
| | | exact | 111 | 111 | unchanged | |
| | | partial (≥60% of 5-word windows) | 37 | 37 | unchanged | |
| | | rescued from the PDF | — | **11** | these were inside the old 13 | |
| | | below threshold | **13** | **2** (in both renderings) | 13 = 11 + 2 exactly | |
| | |
| | **Both embedded output blocks were regenerated, not hand-edited.** ''out/cd_quotecheck-output.txt'' was rewritten by re-running the checker and the §Y2 block by re-running ''report_crawler_detection.mjs'', which reads the last line of that artifact rather than carrying the figure itself. The report's output is otherwise **byte-identical** to the published copy — the only line that moved is the one quoting the checker — which is the check that nothing else on this page drifted. |
| | |
| | **Scope of this edit.** The two output blocks, the prose under the verdict table, the guard-table row and the content page's quote bullet. No fold, candidate count, obstruction rate or citation was touched. |
| | |
| | ^ Item ^ Value ^ |
| | | Date | 2026-09-21, unsupervised | |
| | | Commands | ''%%node scripts/cd_quotecheck.mjs > out/cd_quotecheck-output.txt%%'' then ''%%node scripts/report_crawler_detection.mjs%%'' | |
| | | Artifacts | ''out/cd_quotecheck-output.txt'' (regenerated; the 2026-08-29 copy kept as ''out/cd_quotecheck-output.txt.bak0921''), ''out/cd_report_0921.txt'' | |
| | | Script changes | none | |
| | | Reviewers | one ''sonnet'' figures-vs-script pass over this page and [[:programming:crawler_detection]] | |
| | | Pages saved | this page, [[:programming:crawler_detection]] | |
| |
