| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| programming:crawler:pagegraph [2026/08/14 18:16] – Review-layer fixes: repair broken crawl command (dead --debug flag, \\ line continuations), add elm subcommand, re-source doc-incompleteness claim, full 9-item limitations list, Shields-state warning, runtime-cost admission, VisibleV8 comparison row, AdGr karel.kubicek.claude | programming:crawler:pagegraph [2026/09/22 04:19] (current) – Quote-check refresh 2026-09-22: quote bullet gains the pypdf-fallback verdict (0 rescued, 1 below threshold in both renderings) and the corrected cause. Authored by Claude karel.kubicek.claude |
|---|
| |
| <WRAP important> | <WRAP important> |
| Do not copy the ''--debug debug'' example that ''pagegraph-crawl'''s own README still shows. That flag no longer exists: the current argument parser accepts ''--logging {none,info,verbose}'' (default ''info''), and ''--debug debug'' exits with ''run.js: error: unrecognized arguments: --debug debug''. Verified against ''src/run.ts'' at version 1.2.13 on 2026-08-14. It is a good reminder to run any documented command before putting it in a paper's artifact appendix. | Do not copy the ''%%--debug debug%%'' example that ''pagegraph-crawl'''s own README still shows. That flag no longer exists: the current argument parser accepts ''%%--logging {none,info,verbose}%%'' (default ''info''), and ''%%--debug debug%%'' exits with ''%%run.js: error: unrecognized arguments: --debug debug%%''. Verified against ''src/run.ts'' at version 1.2.13 on 2026-08-14. It is a good reminder to run any documented command before putting it in a paper's artifact appendix. |
| </WRAP> | </WRAP> |
| |
| <WRAP important> | <WRAP important> |
| **Record the Shields state, and do not assume it.** PageGraph runs inside Brave, which ships an ad and tracker blocker, so "what the page did" depends on whether that blocker was on. The trap is that the two obvious ways to run it disagree: ''pagegraph-crawl'' defaults to ''-s down'' — Shields **off**, so a default crawl sees an essentially unfiltered web — whereas a Brave you launch yourself has Shields **on** by default. A prevalence figure means something different under each, and ''--shields'' is ignored entirely when ''--existing-user-data-dir'' is used. Verified against ''src/run.ts'' (''defaultShieldsSetting = "down"'') on 2026-08-14. | **Record the Shields state, and do not assume it.** PageGraph runs inside Brave, which ships an ad and tracker blocker, so "what the page did" depends on whether that blocker was on. The trap is that the two obvious ways to run it disagree: ''pagegraph-crawl'' defaults to ''-s down'' — Shields **off**, so a default crawl sees an essentially unfiltered web — whereas a Brave you launch yourself has Shields **on** by default. A prevalence figure means something different under each, and ''%%--shields%%'' is ignored entirely when ''%%--existing-user-data-dir%%'' is used. Verified against ''src/run.ts'' (''defaultShieldsSetting = "down"'') on 2026-08-14. |
| </WRAP> | </WRAP> |
| |
| * The population query folds only the two raw tool names ''PageGraph'' and ''PageGraph crawler'' with an ordered exact regex. It counts ''used'' and ''produced'' tool tuples, then deduplicates by paper. The unmapped residue is empty in this run and is printed as empty rather than hidden. | * The population query folds only the two raw tool names ''PageGraph'' and ''PageGraph crawler'' with an ordered exact regex. It counts ''used'' and ''produced'' tool tuples, then deduplicates by paper. The unmapped residue is empty in this run and is printed as empty rather than hidden. |
| * The 2025–2026 rows are provisional, and the seven-venue corpus omits EuroS&P, ACSAC, RAID, AsiaCCS, CHI and SOUPS. The table is a finding about this corpus, not an adoption survey. | * The 2025–2026 rows are provisional, and the seven-venue corpus omits EuroS&P, ACSAC, RAID, AsiaCCS, CHI and SOUPS. The table is a finding about this corpus, not an adoption survey. |
| * Nine PageGraph evidence quotes were checked against ''paper.cols.txt'': 5 exact, 3 partial after the dataset's five-word-window check, and 1 below threshold. The one below-threshold quote was manually confirmed; a column-repair hyphenation joined “Chromium-based” in the source rendering. | * Nine PageGraph evidence quotes were checked against ''paper.cols.txt'', with anything below threshold re-checked against a ''pypdf'' rendering of the same ''paper.pdf'': 5 exact, 3 partial after the dataset's five-word-window check, 0 rescued from the PDF, and 1 below threshold in both renderings. The one below-threshold quote was manually confirmed. It is unrescuable rather than unsupported: the PDF's own text layer breaks “Chromium-based” across a line, so both renderings strip the hyphen and score it identically at 29%. The [[provenance:programming:crawler:pagegraph|provenance page]] records why no second rendering can fix this one. |
| |
| ===== Related pages ===== | ===== Related pages ===== |
