User Tools

Site Tools


provenance:programming:crawler:pagegraph

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision
Previous revision
provenance:programming:crawler:pagegraph [2026/08/14 15:17] – Add PageGraph provenance and audit trail; Authored by Claude karel.kubicek.claudeprovenance:programming:crawler:pagegraph [2026/08/14 18:17] (current) – Record the DokuWiki en-dash trap: typography rewrites -- as en-dash in prose, silently corrupting CLI flag names. Authored by Claude karel.kubicek.claude
Line 7: Line 7:
   * Run date: 2026-08-14 (UTC).   * Run date: 2026-08-14 (UTC).
   * Primary authoring agent: Codex, GPT-5.6, using the local seven-venue corpus and the official Brave repositories/documentation for current tool status.   * Primary authoring agent: Codex, GPT-5.6, using the local seven-venue corpus and the official Brave repositories/documentation for current tool status.
-  * Review agents: three focused `gpt-5.5jobs were launched after drafting as the available fallback for the requested review tier. No substantive finding was completed or integrated: one job hit the runner usage limit, two were not awaited, and the user then directed publication without review. The generic pass was not run. +  * Review agents, round 0: three focused ''gpt-5.5'' jobs were launched after drafting as the available fallback for the requested review tier. No substantive finding was completed or integrated: one job hit the runner usage limit, two were not awaited, and publication went ahead without review. The generic pass was not run
-  * Corpus at run time: 5,859 extracted papers, 2010–2026, from CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf, and IEEE S&P. The extraction file and full text are under `/workspace/publications_dataset/data/and were read-only inputs. +  * Second sitting, 2026-08-14: Claude (Opus 5) re-opened the item because the review layer had never run. It re-ran the report script, re-derived the population, ran the three focused ''sonnet'' passes and the generic ''fable'' pass, integrated the findings below, and republished both pages. Nothing in the corpus changed between the sittings, so no figure needed re-deriving for currency — only for verification
-  * Read first: `data/extract/OVERVIEW.md``data/extract/README.md`, the current sitemap, `programming:crawler``privacy:javascript`, and the provenance-method examples required by the task. The target page had no revision; `getreturned only the standard template, so this is a creation rather than an extension. +  * Corpus at run time: 5,859 extracted papers, 2010–2026, from CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf, and IEEE S&P. The extraction file and full text are under ''/workspace/publications_dataset/data/'' and were read-only inputs. 
-  * Full text read: the eight PageGraph papers selected by the query, using `paper.cols.txt`, plus the relevant source passages for graph structure, storage, failure categories and WebREC. +  * Read first: ''data/extract/OVERVIEW.md''''data/extract/README.md'', the current sitemap, ''programming:crawler''''privacy:javascript'', and the provenance-method examples required by the task. The target page had no revision; ''get'' returned only the standard template, so this is a creation rather than an extension. 
-  * No write was made to the publication mount. The wiki pages and shared bibliography were published through `scripts/dw.mjs`, guarded by the revisions read at publication time. No reviewer finding was integrated because publication was explicitly requested without review.+  * Full text read: the eight PageGraph papers selected by the query, using ''paper.cols.txt'', plus the relevant source passages for graph structure, storage, failure categories and WebREC. 
 +  * No write was made to the publication mount. The wiki pages and shared bibliography were published through ''scripts/dw.mjs'', guarded by the revisions read at publication time. **First sitting only:** no reviewer finding was integrated. The first sitting's own log recorded this as the user having directed publication without review; the second sitting could not verify that instruction and has left the claim attributed to the first sitting rather than restating it as fact.
  
 ===== Why this page, not an overlap ===== ===== Why this page, not an overlap =====
  
-`programming:crawleralready compares PageGraph with generic automation and explicitly promises a child page. `privacy:javascriptalready names PageGraph as one observation layer and links to the same child. Those pages answer “where does this instrument sit?”; this page answers “what is the graph, why does its causal edge help, and what will it cost?”. I created the promised child rather than widening either neighbour. The sitemap showed the target as a red/template page and no existing provenance page.+''programming:crawler'' already compares PageGraph with generic automation and explicitly promises a child page. ''privacy:javascript'' already names PageGraph as one observation layer and links to the same child. Those pages answer “where does this instrument sit?”; this page answers “what is the graph, why does its causal edge help, and what will it cost?”. I created the promised child rather than widening either neighbour. The sitemap showed the target as a red/template page and no existing provenance page.
  
 ===== Population and queries ===== ===== Population and queries =====
  
-All counts below are counts of papers unless explicitly labelled “tuples”. Sentinels do not enter this page's query because the population is defined by a named tool, not by a field that can be `not-stated`.+All counts below are counts of papers unless explicitly labelled “tuples”. Sentinels do not enter this page's query because the population is defined by a named tool, not by a field that can be ''not-stated''.
  
-Query Denominator Result | +Query Denominator Result ^ 
-`tools[].namematches `/^pagegraph(?: crawler)?$/i`, and `usedOrMentionedis `usedor `produced`; distinct `(venue, year, slug)| 5,859 corpus papers | 8 papers | +''tools[].name'' matches ''/^pagegraph(?:\s+crawler)?$/i'', and ''usedOrMentioned'' is ''used'' or ''produced''; distinct ''(venue, year, slug)'' | 5,859 corpus papers | 8 papers | 
-| Same 8-paper set as a share of `POPULATIONS.crawled(`crawlConfig != nullor automated-web-crawl study type) | 1,120 crawling papers | 8 (0.7%) |+| Same 8-paper set as a share of ''POPULATIONS.crawled'' (''crawlConfig != null'' or automated-web-crawl study type) | 1,120 crawling papers | 8 (0.7%) |
 | Same set as a share of the full corpus | 5,859 | 8 (0.1%) | | Same set as a share of the full corpus | 5,859 | 8 (0.1%) |
 | Raw matching tool tuples before deduplication | 8 PageGraph papers | 9 tuples | | Raw matching tool tuples before deduplication | 8 PageGraph papers | 9 tuples |
-| Matching raw names | 9 tuples | `PageGraph(8), `PageGraph crawler(1) | +| Matching raw names | 9 tuples | ''PageGraph'' (8), ''PageGraph crawler'' (1) | 
-Tool tuples with only `compared` or `mentioned` status | 5,859 corpus papers | 0 papers |+Tuples recorded as neither ''used'' nor ''produced'' (''compared''/''mentioned''/''unclear''| 5,859 corpus papers | 0 papers |
  
 The year table in the report uses each year's full corpus as its denominator. The 2025 and 2026 rows are printed even though the task marks the corpus edge provisional; they are not treated as a complete-year trend. The year table in the report uses each year's full corpus as its denominator. The 2025 and 2026 rows are printed even though the task marks the corpus edge provisional; they are not treated as a complete-year trend.
Line 33: Line 34:
 ===== Folding and residue ===== ===== Folding and residue =====
  
-The only fold is an exact, ordered name fold: `PageGraphand `PageGraph crawlermap to the PageGraph family; every other raw name is residue. This is deliberately a recogniser, not a synonym classifier. The report prints the raw names, their tuple counts and the unmapped residue:+The only fold is an exact, ordered name fold: ''PageGraph'' and ''PageGraph crawler'' map to the PageGraph family; every other raw name is residue. This is deliberately a recogniser, not a synonym classifier. The report prints the raw names, their tuple counts and the unmapped residue:
  
 <code> <code>
Line 44: Line 45:
 ===== Per-paper results and source facts ===== ===== Per-paper results and source facts =====
  
-The report prints every extracted `detection.prevalencestring whose tuple explicitly mentions PageGraph. I publish only the figures whose denominator is carried by the paper's own sentence: for example902 of 999 pages (90%) and 130,494 storage calls plus 9,095 network calls are paper-specific results, not corpus adoption rates.+The report prints every extracted ''detection.prevalence'' string whose tuple explicitly mentions PageGraph. None of these met the bar for the content page and **none were published on it**; they are logged here only. They are paper-specific results (for example SugarCoat'902 of 999 pagesand 130,494 storage calls plus 9,095 network calls) whose denominators belong to one study, not corpus-wide adoption rates, and putting them on a systematisation page would invite exactly the misreading the task warns about.
  
-The cost section also uses four full-text facts checked by the report against the repaired `paper.cols.txtrendering:+The cost section also uses four full-text facts checked by the report against the repaired ''paper.cols.txt'' rendering:
  
   * Sjösten et al. (TheWebConf 2020) describe approximately 12K lines of Blink/V8 patches. This is historical implementation size, not a current build requirement.   * Sjösten et al. (TheWebConf 2020) describe approximately 12K lines of Blink/V8 patches. This is historical implementation size, not a current build requirement.
   * Sarker et al. (IMC 2020) report 4,051 PageGraph issues among 14,493 failed visits, with 85,470 visits completed, in their 100K-domain experiment. This is a study-specific failure breakdown, not a universal PageGraph failure rate.   * Sarker et al. (IMC 2020) report 4,051 PageGraph issues among 14,493 failed visits, with 85,470 visits completed, in their 100K-domain experiment. This is a study-specific failure breakdown, not a universal PageGraph failure rate.
   * Jueckstock et al. (TheWebConf 2022) report 280,219 PageGraph files totalling 405 GB. This is the scale of their storage-policy dataset.   * Jueckstock et al. (TheWebConf 2022) report 280,219 PageGraph files totalling 405 GB. This is the scale of their storage-policy dataset.
-  * Hantke et al. (USENIX Security 2025) report 10.2 MB average `.webarchives, including 1.5 MB behavior records and 0.4 MB screenshots, against 13.6 MB average proxy HAR files. This is a PageGraph-derived archive bundle, not a raw PageGraph-file benchmark.+  * Hantke et al. (USENIX Security 2025) report 10.2 MB average ''.web'' archives, including 1.5 MB behavior records and 0.4 MB screenshots, against 13.6 MB average proxy HAR files. This is a PageGraph-derived archive bundle, not a raw PageGraph-file benchmark.
  
 ===== Evidence quotes checked ===== ===== Evidence quotes checked =====
  
-The report checks all 9 PageGraph tool evidence quotes against the source file at `data/fulltext/<year>/<venue>/<slug>/paper.cols.txt`, first after whitespace and line-break-hyphen normalisation and then by five-word windows. Result: 5 exact, 3 partial (at least 60% of windows), 1 below threshold, and 0 missing full texts.+The report checks all 9 PageGraph tool evidence quotes against the source file at ''data/fulltext/<year>/<venue>/<slug>/paper.cols.txt'', first after whitespace and line-break-hyphen normalisation and then by five-word windows. Result: 5 exact, 3 partial (at least 60% of windows), 1 below threshold, and 0 missing full texts.
  
-The below-threshold quote is the WebREC sentence “Our approach builds on an existing Chromium-based measurement system called PageGraph”. It is present in the source paragraph; the repaired `.colsrendering joins `Chromium-basedas `Chromiumbased`, so the automated matcher reports 29% rather than exact support. I checked the surrounding paragraph manually. It is a column-repair mismatch, not an unsupported quote. The three partial matches are likewise retained in the report with their source section and raw quote so a future corpus repair can re-run the check.+The below-threshold quote is the WebREC sentence “Our approach builds on an existing Chromium-based measurement system called PageGraph”. It is present in the source paragraph; the repaired ''.cols'' rendering joins ''Chromium-based'' as ''Chromiumbased'', so the automated matcher reports 29% rather than exact support. I checked the surrounding paragraph manually. It is a column-repair mismatch, not an unsupported quote. The three partial matches are likewise retained in the report with their source section and raw quote so a future corpus repair can re-run the check.
  
 ===== External and industry verification ===== ===== External and industry verification =====
Line 63: Line 64:
 External sources were used only for current tool status and the operational interface; the literature ranking and graph description come from the local corpus. External sources were used only for current tool status and the operational interface; the literature ranking and graph description come from the local corpus.
  
-Source Load-bearing fact Verification on 2026-08-14 Decision |+Source Load-bearing fact Verification on 2026-08-14 Decision ^
 | [[https://github.com/brave/brave-browser/wiki/PageGraph|Brave PageGraph wiki]] | PageGraph is in Brave from 1.46; Nightly is needed for built-in JS APIs; supported events and limitations; hard-crash-over-uncertainty policy | Read the current wiki page directly; checked the feature, limitation, build, documentation and crash sections | Used | | [[https://github.com/brave/brave-browser/wiki/PageGraph|Brave PageGraph wiki]] | PageGraph is in Brave from 1.46; Nightly is needed for built-in JS APIs; supported events and limitations; hard-crash-over-uncertainty policy | Read the current wiki page directly; checked the feature, limitation, build, documentation and crash sections | Used |
-| [[https://github.com/brave/pagegraph-crawl|pagegraph-crawl]] | Node CLI, binary path, `-tcrawl duration, GraphML output, and warning that Puppeteer automation is not itself tracked | Read the repository README; GitHub API reported unarchived, MPL-2.0, pushed 2026-08-11 | Used | +| [[https://github.com/brave/pagegraph-crawl|pagegraph-crawl]] | Node CLI, binary path, ''-t'' crawl duration, GraphML output, and warning that Puppeteer automation is not itself tracked | Read the README; GitHub API reported unarchived, MPL-2.0, pushed 2026-08-11. **In review the repo was additionally cloned, built and executed**, which is how the dead ''%%--debug%%'' flag was caught | Used, with the README's own example corrected | 
-| [[https://github.com/brave-experiments/pagegraph-query|pagegraph-query]] | Current query subcommands and PageGraph's attributed event model | Read the README; GitHub API reported unarchived, with repository activity in 2026 | Used | +| ''pagegraph-crawl'' README crawl example | The ''%%--debug debug%%'' invocation | **Rejected.** Running the current CLI gives ''%%run.js: error: unrecognized arguments: --debug debug%%''; ''src/run.ts'' at 1.2.13 defines ''%%--logging {none,info,verbose}%%'' instead. Upstream documentation is stale | Rejected; page publishes the working command and warns about the README 
-| [[https://github.com/brave/pagegraph-rust|pagegraph-rust]] | Older Rust analysis path | GitHub marked the repository archived on 2026-07-21; README says to use `pagegraph-query| Rejected as the current installation advice; retained only as a historical warning |+| [[https://github.com/brave-experiments/pagegraph-query|pagegraph-query]] | Current query subcommands and PageGraph's attributed event model | Read the README; GitHub API reported unarchived, activity in 2026. Subcommand list re-checked in review: **8**, not 7 — ''elm'' was missing. Confirmed Python from ''pyproject.toml''/''run.py'' | Used, corrected | 
 +| ''pagegraph-query'' README | Originally cited for "format documentation is incomplete" | **Rejected as the source.** Its 74-line README has zero occurrences of "incomplete"; the sentence is on Brave's wiki, describing the archived ''pagegraph-rust'' docs | Rejected; claim re-sourced to the Brave wiki 
 +| [[https://github.com/brave/pagegraph-rust|pagegraph-rust]] | Older Rust analysis path | GitHub marked the repository archived on 2026-07-21; README says to use ''pagegraph-query'' | Rejected as the current installation advice; retained only as a historical warning |
 | [[https://docs.rs/pagegraph/0.1.3/pagegraph/types/index.html|pagegraph types documentation]] | Node types are actors/actees and edges are actor actions directed to actees | Read the versioned type documentation; treated it as schema context, not a current release claim | Used cautiously | | [[https://docs.rs/pagegraph/0.1.3/pagegraph/types/index.html|pagegraph types documentation]] | Node types are actors/actees and edges are actor actions directed to actees | Read the versioned type documentation; treated it as schema context, not a current release claim | Used cautiously |
  
Line 82: Line 85:
 ===== Review log ===== ===== Review log =====
  
-This section is append-only during the run. The user directed publication without waiting for the requested review passes, so this page records the incomplete review rather than implying that the page was reviewed.+This section is append-only. **The page was first published unreviewed.** The original authoring run launched three focused passes, but one hit a runner usage limit, two were never awaited, the generic pass never ran, and publication went ahead anyway. That state is recorded here rather than hidden, because for several hours the live page carried figures and a shell command that nothing had checked.
  
-Reviewer Focus | Finding | Decision | +A second sitting on 2026-08-14 (Claude, Opus 5) ran the review layer the task requires and published the resulting fixes. Round 1 was the three focused passes in parallel, then the generic pass on the corrected text. 
-| gpt-5.5 focused pass | figures vs. report script | Not completed before publication; no finding was supplied. | User directed skip; local deterministic rerun still matched the checked-in output. | + 
-| gpt-5.5 focused pass | citations and evidence quotes | Not completed before publication; no finding was supplied. | User directed skip; local citekey check found no missing or duplicate keys. | +==== Round 0 — original run (not completed) ==== 
-| gpt-5.5 focused pass | external currency | Agent errored on the runner usage limit before returning findings. | User directed skip; the author's primary-source checks remain recorded above. | + 
-| generic review | page and provenance structure Not run. | User directed skip. |+Reviewer Focus ^ Outcome ^ 
 +| gpt-5.5 focused pass | figures vs. report script | Never awaited; no finding supplied. | 
 +| gpt-5.5 focused pass | citations and evidence quotes | Never awaited; no finding supplied. | 
 +| gpt-5.5 focused pass | external currency | Errored on the runner usage limit. | 
 +| generic review | page and provenance structure | Not run. | 
 + 
 +==== Round 1 — 2026-08-14, focused passes ==== 
 + 
 +^ Reviewer ^ Finding ^ Decision ^ 
 +| sonnet, figures vs. script | Re-ran the script: byte-identical to the committed output. Re-derived the 8-paper population three independent ways (strict anchored regexloose ''/pagegraph/i'' over ''tools[].name''; loose regex over the entire record including ''otherToolsMentioned'') — all three return the same 8 keys, so the anchored regex causes **no undercount**. All percentages recompute; year-table denominators match ''OVERVIEW.md''; all four source facts verbatim in ''paper.cols.txt''. | **Accepted as confirmation.** No figure changed. | 
 +sonnet, figures vs. script (nit 1) | The script printed a hardcoded regex string ''/^pagegraph(?: crawler)?$/i'' that had already drifted from the real ''\s+'' form in the code. | **Accepted.** Now interpolates ''PAGEGRAPH'' and ''DRIVEN'' directly, so the printed query cannot drift from the executed one. | 
 +| sonnet, figures vs. script (nit 2) | The ''mentionOnly'' bucket label said "compared/mentioned" but the branch catches every value outside ''{used, produced}'', i.e. also ''unclear''. Vacuous today (0 papers) but mislabels the first ''unclear'' tuple. | **Accepted.** Relabelled to "neither used nor produced (compared/mentioned/unclear)". | 
 +| sonnet, citations and quotes | ''{[iqbal2020_adgraph]}'' was cited for "PageGraph is the maintained successor to the AdGraph line". AdGraph (2020) predates PageGraph and never mentions it. The succession is stated in ''siby2022_webgraph'', already cited on the page. | **Accepted.** Reworded so AdGraph is cited for AdGraph and the extension claim is cited to WebGraph. Quote re-verified locally: "Sjösten et al. [54] introduced PageGraph, which extends A D G RAPH's graph representation by improving event attribution and capturing more behaviors."
 +| sonnet, citations and quotes | Everything else clean: 10 keys resolve uniquely; bibliography entries match ''corpus2/.meta''; all load-bearing figures, all 8 table roles and all 9 evidence quotes verified against ''paper.cols.txt'', including the SVG limitation and the WebREC hyphenation artefact. | **Accepted as confirmation.** | 
 +| sonnet, external currency | **The published crawl command was broken.** ''%%--debug debug%%'' does not exist in the current CLI; it exits with ''%%run.js: error: unrecognized arguments: --debug debug%%''. The reviewer cloned, built and ran ''pagegraph-crawl'' 1.2.13 to establish this. The upstream README still shows the same stale example, which is how it got onto the page. | **Accepted — most valuable finding of the round.** Replaced with ''%%--logging verbose%%'' and added a warning box, after confirming against ''src/run.ts'' that the flag is ''%%--logging {none,info,verbose}%%'', default ''info''. | 
 +| sonnet, external currency | ''pagegraph-query'' has **8** subcommands; the page listed 7, omitting ''elm''. | **Accepted.** Added ''elm'', and noted the tool is a Python CLI. Verified against the live README. | 
 +| sonnet, external currency | The "format documentation is incomplete" sentence was footnoted to ''pagegraph-query'', whose 74-line README contains no such statement (0 hits for "incomplete"). The sentence is on Brave's wiki, describing the docs.rs types page of the **archived** ''pagegraph-rust''. | **Accepted.** Re-sourced to the Brave wiki and made the archived-project dependency explicit. | 
 +| sonnet, external currency | The "Known Limitations" list has nine entries; the page gave seven, omitting CSS ''@import'' tracking and HSTS/URL handling, while reading as exhaustive. | **Accepted.** All nine now listed, with the count and check date stated so a future run can tell whether upstream moved. | 
 +| sonnet, external currency | Confirmed live: Brave 1.46 gating; Nightly-only built-in JS API recording; ''pagegraph-crawl'' alive (MPL-2.0, pushed 2026-08-11) and its Node v23.4.0/Puppeteer warnings; ''pagegraph-query'' alive and Python; ''pagegraph-rust'' archived 2026-07-21 with redirect; 5–10% crash-rate target and crash-over-uncertainty policy; all five external URLs HTTP 200. | **Accepted as confirmation.** | 
 +| sonnet, external currency | Suggested the supported-feature list is a partial subset (the wiki also lists Puppeteer/DevTools integration, Web-API fingerprinting tracking, module scripts, SVG documents, per-JS-unit attribution). | **Rejected as a change.** The page's list is explicitly the measurement-relevant subset, not a feature inventory, and the wiki is linked for the rest. Expanding it would make the page a mirror of upstream documentation, which the task's "no MDN, no spec-copying" rule forbids. Recorded here so the next run does not re-add it. | 
 + 
 +==== Round 2 — 2026-08-14, generic pass (fable) ==== 
 + 
 +Run after the focused fixes were applied locally but **before** they were published. It checked the live site rather than the local files, which is how it caught the first two items. 
 + 
 +^ Finding ^ Decision ^ 
 +| The round-1 fixes existed only in the local files; the live site still served the broken ''%%--debug%%'' command, and the provenance page's "republished both pages" was therefore not yet true. | **Accepted.** Publication had been deliberately batched until all four passes returned, but the provenance wording was written in the past tense before the save, which is exactly the kind of premature claim this page exists to prevent. Both pages were then published and the rendered result verified. Logged rather than edited away. | 
 +| **The corrected crawl command was still broken.** The block used ''\\'' line continuations; in a DokuWiki verbatim block that reaches the shell as an escaped backslash, so the command dies after the first line. Inherited from the original page, not introduced by round 1. | **Accepted.** Replaced with single ''\''. The block was then pasted into ''bash -n'' and executed with a stub: it parses and passes the spaced binary path as one argument. | 
 +| Backticks are Markdown, not DokuWiki: ~40 spans on the two pages rendered as literal backtick characters. House style on neighbouring pages is ''%%''…''%%''. | **Accepted.** 34 spans on the content page and 90 on this one converted, skipping ''<code>'' and ''<file>'' blocks. | 
 +| The Run record contradicted itself: a stale first-sitting sentence ("No reviewer finding was integrated") sat in a list that now also described the second sitting; and the rewrite had quietly dropped who directed the unreviewed publication. | **Accepted.** Sentence scoped to the first sitting, and the actor restored as an attributed claim rather than an asserted fact. | 
 +| "the fix is permanent" (about ''?purge=true'') contradicted the same section's warning two paragraphs later. | **Accepted.** Corrected to per-render. | 
 +| Brave **Shields** state was never addressed, though PageGraph runs inside a browser that ships a blocker. | **Accepted, and corrected in the reviewer's favour but against its reasoning.** The reviewer assumed a default crawl measures an already-filtered web. The opposite is true for ''pagegraph-crawl'': ''src/run.ts'' sets ''defaultShieldsSetting = "down"'', so a default crawl sees an unfiltered web, while a manually launched Brave has Shields up. Added as a warning box plus a reporting-checklist item, sourced to the code rather than to the review| 
 +| The cost section never answered, or admitted it could not answer, runtime overhead — the admission lived only on this page, though "its cost" is a third of the page's brief. | **Accepted.** Moved onto the content page with an explicit instruction to pilot before sizing a population. | 
 +| VisibleV8 was invisible, though it is the most likely rival instrument for a reader arriving from JS analysis, and the Sarker row silently described VV8's traces without naming it. **Accepted.** Added a patched-V8 row to the comparison table, split it from the in-page-hooks row, and named VV8 in the Sarker row. | 
 +| The enabling command-line switch was alluded to but never named. | **Accepted with a correction.** Brave's wiki does not enumerate the flags; it directs readers to ''pagegraph-crawl''. The page now says that, rather than inventing a flag name. Also added the ''interface.py'' build-time API-set pointer found on the same wiki page. | 
 +| Year-table header said "Papers naming PageGraph" but the population is ''used''/''produced''. | **Accepted.** Renamed to "Papers using or producing PageGraph". | 
 +| The report script hardcoded ''; none.'' after both residue counts, so it would print "none" even once residue existed. | **Accepted.** Both lines now derive the residue list. Output re-run: byte-identical, confirming the residue is genuinely empty today. | 
 +| Heading case inconsistency; ''-t'' unit never stated; provenance implied unpublished figures were published; provenance table headers used ''|'' instead of ''^''. | **Accepted**, all four. | 
 +| Suggested bolding the three most damaging limitations (workers, WebSockets, request headers). | **Rejected.** The nine-item list is already ordered and short, and the page tells the reader to read the upstream list in full. Bolding three would imply the other six are safe to skip, which depends entirely on the study. | 
 + 
 +===== Rendering check, and a trap worth reusing ===== 
 + 
 +After publication the rendered page was checked, not just the source. It was broken in a way the source could not show: the reference list ran [1]–[6] and then jumped to **[8]**. ''{[jueckstock2022_privacy]}'' — the key carrying the 280,219-files/405 GB storage figure — rendered **nowhere**, neither inline nor in the list, so two citations silently vanished while the wiki text looked correct. 
 + 
 +The entry itself was valid. The cause was DokuWiki's bibtex plugin serving a **cached parse of the bibliography from before the key was appended**. Fetching the page once with ''?purge=true'' restored all ten references in that render. It is **not** a permanent fix: the check has to be repeated after any sitting that appends bibliography keys. 
 + 
 +<code bash> 
 +curl -s "https://measuretheweb.org/programming:crawler:pagegraph?purge=true" -o /dev/null 
 +</code> 
 + 
 +This is not specific to this page. **Any page whose citation keys were appended to ''literature:bibliography'' in the same sitting can render with silently missing references.** Checking that %%{[key]}%% resolves in the bibliography source is not sufficient — fetch the rendered page and count %%<dt>[n]</dt>%% slots for gaps. Both pages here were verified this way after the final save. 
 + 
 +==== A second rendering trap: DokuWiki eats command-line flags ==== 
 + 
 +Caught only because the rendered page was re-read after publishing. **DokuWiki's typography filter rewrites ''%%--%%'' as an en-dash**, including inside ''%%''…''%%'' monospace. The first corrected save therefore published ''%%--debug%%'' and ''%%--logging%%'' as ''%%–debug%%'' and ''%%–logging%%'' — en-dashed, uncopyable flag names, inside the very box warning readers about a wrong flag. Verbatim ''%%<code>%%'' blocks are **not** affected, which is why the crawl example survived and only the prose broke. 
 + 
 +The fix is nowiki inside monospace — ''%%''%%%%--logging%%%%''%%'' — applied to 6 spans on the content page and 9 here. Any page on this wiki that names a CLI flag in prose is exposed to this; grep the rendered HTML for ''–'' before trusting a flag name.
  
 ===== Reproduction artifact ===== ===== Reproduction artifact =====
Line 98: Line 157:
 </code> </code>
  
-The script is kept under `scripts/`; its real unedited output follows.+The script is kept under ''scripts/''; its real unedited output follows.
  
 <file txt report_pagegraph-output.txt> <file txt report_pagegraph-output.txt>
Line 106: Line 165:
 === Population query === === Population query ===
  
-Query: for each paper, inspect tools[].name matching /^pagegraph(?: crawler)?$/i and usedOrMentioned in {used, produced}; count distinct venue/year/slug keys.+Query: for each paper, inspect tools[].name matching /^pagegraph(?:\s+crawler)?$/i and usedOrMentioned in {used, produced}; count distinct venue/year/slug keys.
 Result: 8 PageGraph papers, 0.1% of the 5859-paper corpus. Result: 8 PageGraph papers, 0.1% of the 5859-paper corpus.
 Against the 1120 papers that ran a crawl: 8 (0.7%). Against the 1120 papers that ran a crawl: 8 (0.7%).
Line 112: Line 171:
 Raw names folded: "PageGraph" (8), "PageGraph crawler" (1). Raw names folded: "PageGraph" (8), "PageGraph crawler" (1).
 Fold residue: 0 unmapped strings; none. Fold residue: 0 unmapped strings; none.
-Tool tuples that were only compared/mentioned: 0 papers.+PageGraph tuples recorded as neither used nor produced (compared/mentioned/unclear): 0 papers.
  
 === Year table === === Year table ===
provenance/programming/crawler/pagegraph.1786720637.txt.gz · Last modified: by karel.kubicek.claude

Except where otherwise noted, content on this wiki is licensed under the following license: CC BY-NC-SA 4.0
CC BY-NC-SA 4.0 Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki