| Next revision | Previous revision |
| provenance:design:existing_datasets [2026/08/28 19:04] – Provenance for design:existing_datasets: populations, the three-field disagreement, the fold and its full 3,192-string residue, the residue hand-audit, quote checks, external sources verified and rejected, judgement calls, the four-pass review log, and th karel.kubicek.claude | provenance:design:existing_datasets [2026/09/17 07:44] (current) – Markup rendering sweep correction: preserve table heading and escape residual CLI flag. Authored by Claude. karel.kubicek.claude |
|---|
| ==== Pass A — figures vs script (Sonnet) ==== | ==== Pass A — figures vs script (Sonnet) ==== |
| |
| Re-ran the script, diffed it against the committed output, checked every number and denominator on the page, audited the script internals, swept ''--members'' for every published family, and reproduced four headline figures independently in Python from ''extractions.jsonl''. | Re-ran the script, diffed it against the committed output, checked every number and denominator on the page, audited the script internals, swept ''%%--members%%'' for every published family, and reproduced four headline figures independently in Python from ''extractions.jsonl''. |
| |
| ^ Finding ^ Verdict ^ What was done ^ | ^ Finding ^ Verdict ^ What was done ^ |
| * "It is the modal sentence" (about undated dataset references) was an unsupported superlative. Replaced with the figure that //is// supported: 43.3% of the 2,534 existing-dataset papers give neither a first nor a last date. | * "It is the modal sentence" (about undated dataset references) was an unsupported superlative. Replaced with the figure that //is// supported: 43.3% of the 2,534 existing-dataset papers give neither a first nor a last date. |
| * A section on **reusing another paper's released artifact** was missing, even though the item this page answers names "last year's crawl" explicitly. Added, built on two figures the page already has (9 of 60 residue strings name a prior paper's dataset by citation; 57.7% artifact release) plus {[demir2022_reproducibility]}. The Demir citation was first written as "measured how much a repeated crawl of the same list differs from itself", which is **not** what that paper did; corrected to its actual design (4.5M pages, 24 setups) and its own sentence. | * A section on **reusing another paper's released artifact** was missing, even though the item this page answers names "last year's crawl" explicitly. Added, built on two figures the page already has (9 of 60 residue strings name a prior paper's dataset by citation; 57.7% artifact release) plus {[demir2022_reproducibility]}. The Demir citation was first written as "measured how much a repeated crawl of the same list differs from itself", which is **not** what that paper did; corrected to its actual design (4.5M pages, 24 setups) and its own sentence. |
| * The dataset-family table originally showed 24 of the fold's 35 Internet/web/security families, cutting the tail silently — including a 34-paper family. Replaced with the complete table generated from ''--wiki'', plus a sentence naming the counts in the four scopes that are **not** tabulated, so nothing is dropped without a number. | * The dataset-family table originally showed 24 of the fold's 35 Internet/web/security families, cutting the tail silently — including a 34-paper family. Replaced with the complete table generated from ''%%--wiki%%'', plus a sentence naming the counts in the four scopes that are **not** tabulated, so nothing is dropped without a number. |
| | |
| | ==== Mechanical guards, after every fix ==== |
| | |
| | Run against the final page text, not against a draft: |
| | |
| | <code> |
| | node scripts/check_wrap.mjs pages/draft/design_existing_datasets.txt pages/draft/provenance_design_existing_datasets.txt |
| | node scripts/check_tables.mjs pages/draft/design_existing_datasets.txt pages/draft/provenance_design_existing_datasets.txt |
| | node scripts/check_page_numbers.mjs pages/draft/design_existing_datasets.txt scripts/report_existing_datasets-output.txt |
| | </code> |
| | |
| | ''check_wrap'' and ''check_tables'' pass on both pages. ''check_page_numbers'' reports **29 unaccounted figures**, and all 29 were read by hand: every one is from a cited paper (Huang et al.'s 95 / 52.07 / 85.71 / 75.00 / 80.00, Wu et al.'s 1,407 / 49 / 72 / 24.69 / 19.17, Izhikevich et al.'s 443, Demir et al.'s 4.5, the IMC 2016 and IMC 2018 exemplars' 09, 03, 4.1 and 489,580,002), an external primary source (27,616,457 / 308 / 13.6 / 200 / 405 / 86,372 / 6962 / 2019), a hypothetical in the page's own prose (a 10,000-host validation sample), or plain arithmetic on a figure the report **does** print (61.9% = 100 − 38.1%). **No corpus figure on the page is missing from the report.** That is the only thing this guard can prove, and it is worth saying explicitly: it traces digits, so it cannot tell a right number from a wrong one, and it never sees a quoted phrase. |
| |
| ==== Pass D — generic (Fable) ==== | ==== Pass D — generic (Fable) ==== |
| | Main agent | Claude (Opus 5) — corpus queries, fold, report script, page text, provenance page | | | Main agent | Claude (Opus 5) — corpus queries, fold, report script, page text, provenance page | |
| | Sub-agents | one currency sweep (Sonnet) before writing; four reviewers after — figures-vs-script (Sonnet), citations-and-quotes (Sonnet), external-currency (Sonnet), generic (Fable) | | | Sub-agents | one currency sweep (Sonnet) before writing; four reviewers after — figures-vs-script (Sonnet), citations-and-quotes (Sonnet), external-currency (Sonnet), generic (Fable) | |
| | Write path | ''node scripts/dw.mjs'' (JSON-RPC), every save with ''--if-rev'' | | | Write path | ''node scripts/dw.mjs'' (JSON-RPC), every save with ''<nowiki>--if-rev</nowiki>'' | |
| | Bibliography | appended to a fresh export of the live page, not to a local copy; duplicate check by key, DOI and title before appending | | | Bibliography | appended to a fresh export of the live page, not to a local copy; duplicate check by key, DOI and title before appending | |
| | Accidental exposure or mistakes caught in review | the ''artifacts.links'' definition bug in §11; four fold regex bugs in §6; two working hypotheses about external services refuted in §9. All caught before publication. | | | Accidental exposure or mistakes caught in review | the ''artifacts.links'' definition bug in §11; four fold regex bugs in §6; two working hypotheses about external services refuted in §9. All caught before publication. | |
| * [[literature:corpus]] — the dataset, its funnel, its venue scope and its provisional years. | * [[literature:corpus]] — the dataset, its funnel, its venue scope and its provisional years. |
| * [[provenance:design:archives]] — the neighbouring provenance page, for the archive half of the same problem. | * [[provenance:design:archives]] — the neighbouring provenance page, for the archive half of the same problem. |
| | |
| | ===== Markup sweep, 2026-09-17 ===== |
| | |
| | Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving. |
| |
