Differences

This shows you the differences between two versions of the page.

--- provenance:design:archives [2026/08/13 22:45] – New provenance page for design:archives: populations and denominators, the archive-name fold with full residue, quotes spot-checked, external sources verified and rejected, four review passes with findings and dispositions, report script output. Authored karel.kubicek.claude
+++ provenance:design:archives [2026/09/21 13:37] (current) – Quote-check refresh 2026-09-21: re-ran quote_check.mjs --temporal with the pypdf fallback; 21 below threshold -> 13 rescued + 8 below in both; all three hand-read cases are still in the 8. Authored by Claude karel.kubicek.claude
@@ Line 11: / Line 11: @@
 | Report script | ''scripts/report_archives.mjs'' |
 | Fold it depends on | ''scripts/arch_fold.mjs'' (archive-name families) |
-| Quote checker extension | ''scripts/quote_check.mjs'' gained a ''--temporal'' mode; no other flag reaches ''temporal'' tuples |
+| Quote checker extension | ''scripts/quote_check.mjs'' gained a ''%%--temporal%%'' mode; no other flag reaches ''temporal'' tuples |
 | Data | ''data/extract/run1/extractions.jsonl'', 5,859 papers, 7 venues, 2010–2026 |
 | Bibliography | 3 new entries (''lerner2017_rewriting'', ''singh2026_empire'', ''roth2022_security''), 1 entry corrected in place (''hantke2025web'': arXiv preprint → published USENIX Security 2025) |
@@ Line 133: / Line 133: @@
 ===== 8. Quotes and figures spot-checked =====
-''node scripts/quote_check.mjs --temporal '...' '' over the archive-related temporal tuples: **118 quotes checked, 62 exact, 35 partial (≥60% of five-word windows), 21 below threshold, 0 with no full text on disk.** Three of the below-threshold cases were then read by hand against ''paper.cols.txt'':
+''node scripts/quote_check.mjs %%--temporal%% '...' '' over the archive-related temporal tuples. **Re-run 2026-09-21** with the PDF fallback — see //Quote-check refresh, 2026-09-21// at the foot of this page: **118 quotes checked, 62 exact, 35 partial (≥60% of five-word windows), 13 rescued from the PDF, 8 below threshold in both renderings, 0 with no full text on disk.** The old figure was **21** below threshold; 13 of those 21 are located in an independent ''pypdf'' rendering of the same ''paper.pdf'' and are a defect in the stored text rather than in the extraction. ''exact'' (62) and ''partial'' (35) did not move.
+Three of the then-21 were read by hand against ''paper.cols.txt''. **All three are still below threshold in both renderings**, so they are in the 8 and the hand read below remains the only evidence for them — the fallback rescues 13 other quotes and none of these:
 ^ Paper ^ Quote ^ Verdict ^
@@ Line 139: / Line 141: @@
 | NDSS 2025, //Duumviri// | "using Wayback Machine [28]." | **Real**, line 696. Fails only because a five-word-window test on a five-word quote is degenerate. |
 | CCS 2016, //Lurking Malice in the Cloud// | "Searching the Common Crawl dataset, collected in February 2015" | **Real.** The source reads "Common Crawl [?] dataset, collected in February 2015" — the extractor dropped an unresolved citation marker. |
+**Why none of the three moved is the point, not a disappointment.** Two of them (//Duumviri// at 0%, //Through the Lens// at 0%) are short quotes, where a five-word-window test is degenerate regardless of how the PDF renders; the third is a dropped citation marker, which lives in the extraction's own string and no rendering can repair. The fallback fixes de-columning, and de-columning was not what broke these three. The remaining five of the 8 have **not** been read: ''WWW/2020/an-empirical-study-of-the-use-of-integrity-verification-mechanisms…'' (58%), ''PETS/2021/privacy-preference-signals…'' (56%), ''PETS/2021/the-cname-of-the-game…'' (43% and 29%, two tuples in the same paper) and ''CCS/2023/black-ostrich…'' (33%).
 Figures quoted on the content page from individual papers were read in their own full text, not taken from the extraction:
@@ Line 155: / Line 159: @@
 | "only around 5% of the archived pages available on the Wayback Machine were crawled with a browser" | {[zhu2025_toward]} §1 | same — see §10, the appendix that derives it is not in the extracted text |
 | 3× throughput gap, browser vs browserless | {[zhu2025_toward]} §1 | same |
-| 16.1% of requests attempted to escape; 17.2% of URLs; 11.5% of domains. Robots exclusion 2.0% of requests, not-archived 1.4% of requests — **re-read after finding B1, which caught the first draft using the wrong Table 2 row** | {[lerner2016internet]} Table 2 and §5 | ''2016/USENIX/internet-jones…/paper.cols.txt'' |
+| 16.1% of requests attempted to escape; 17.2% of URLs; 11.5% of domains. Robots exclusion 2.0% of requests, not-archived 1.4% of requests — **re-read after finding B1, which caught the first draft using the wrong Table 2 row** | {[lerner2016_internet]} Table 2 and §5 | ''2016/USENIX/internet-jones…/paper.cols.txt'' |
 | 73% of Top 500 / 80% of Top Million vulnerable | {[lerner2017_rewriting]} §6 | ''2017/CCS/rewriting-history…/paper.cols.txt'' |
 | 423,265 of 758,192 URLs (55.8%); median 3 snapshots | {[weinberg2017_topics]} §4.2 | ''2017/PETS/topics-of-controversy…/paper.cols.txt'' |
@@ Line 560: / Line 564: @@
 | Accepted | 29 |
 | Rejected | 1 — repairing four pre-existing duplicate key pairs in the shared bibliography. Out of scope for this item and riskier than leaving them: both keys in each pair resolve, and deleting one could break a page we are not editing. Recorded in §10 instead. |
-| Code written | ''scripts/arch_fold.mjs'' (new), ''scripts/report_archives.mjs'' (new), ''scripts/quote_check.mjs'' (''--temporal'' mode added) |
+| Code written | ''scripts/arch_fold.mjs'' (new), ''scripts/report_archives.mjs'' (new), ''scripts/quote_check.mjs'' (''%%--temporal%%'' mode added) |
 | Pages saved | [[design:archives]] (extended), [[provenance:design:archives]] (new), [[literature:bibliography]] (3 entries added, 1 corrected in place) |
+| Discussion block | None, following the convention already set by ''provenance:design:sampling'' and ''provenance:design:crawling_location'' — comments belong on the content page. The citekeys here therefore render as markers with no reference list, which is the convention rather than an omission. |
+| Publication order | ''literature:bibliography'' (3 added, 1 corrected) → [[design:archives]] → this page, all 2026-08-13, after all four review passes had returned and been applied. |
 **Mistakes made and caught in review**, recorded because the clean version of this page would be worth less:
@@ Line 574: / Line 581: @@
 Every one of these was found by a reviewer, none by the author. The pattern worth carrying forward: the two most serious defects (D1, C1) were **not** the kind a checklist finds — one was a category error inside code everyone had been handed, the other was a claim that something does not exist. Both needed a reader who was not told what to look for.
+===== Amendment, 2026-09-04: citekey consolidation =====
+  * ''lerner2016internet'' was one of two keys for the same paper in [[:literature:bibliography]]. The wiki-wide consolidation of 2026-09-04 (drain item ''dedup-regional-filter-lists-bibkey'') kept ''lerner2016_internet'' and deleted the other entry.
+  * 3 citation markers on [[:design:archives]] and 1 citation marker on this page were repointed to the kept key. No prose on either page changed, and no figure moved. Statements above that name the deleted key describe the state when they were written. Full query log and the invariants checked before saving: [[:provenance:literature:bibliography]].
+[[design:archives|← back to the content page]] · [[literature:corpus|corpus-level provenance]]
+===== Markup sweep, 2026-09-17 =====
+Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving.
+===== Quote-check refresh, 2026-09-21 =====
+The 2026-09-04 ''cols''-vs-PDF audit on [[:provenance:literature:corpus]] showed that 73.1% of evidence quotes that cannot be located in ''paper.cols.txt'' **are** present in an independent ''pypdf'' rendering of the same ''paper.pdf''. ''scripts/quote_check.mjs'' was patched the same day to re-check everything below threshold against that second rendering and report a fourth verdict, **RESCUED**. §8's figure predates the patch. Re-run with the same ''%%--temporal%%'' pattern §3 records, unedited first line:
+<code>
+$ node scripts/quote_check.mjs --temporal 'wayback|internet archive|archive\.org|web-archive|common ?crawl|http ?archive' --show 30
+quotes checked: 62 exact, 35 partial (>=60% of 5-word windows), 13 rescued from the PDF, 8 below threshold in both renderings, 0 with no full text on disk.
+</code>
+The command is the code block above — the pattern is a regex alternation, and a DokuWiki table cell cannot hold a ''%%|%%''.
+^ Figure ^ Was ^ Is ^ Why ^
+| quotes checked | 118 | 118 | population unchanged — the corpus has not moved |
+| exact | 62 | 62 | unchanged |
+| partial (≥60% of 5-word windows) | 35 | 35 | unchanged |
+| rescued from the PDF | — | **13** | new verdict; these were inside the old 21 |
+| below threshold | **21** | **8** (in both renderings) | 21 = 13 + 8 exactly; nothing else moved |
+| below-threshold rate | 17.8% | **6.8%** | 8 of 118 |
+**This is the page where the fallback and the hand pass do not overlap at all.** All three quotes §8 read by hand are still below threshold in both renderings, because two are too short for the window test and the third is a dropped citation marker. §8 now says so and names the five of the 8 that nobody has read.
+**Scope of this edit.** §8's bulk figure and the paragraph under its verdict table. ''arch_fold.mjs'' and ''report_archives.mjs'' were **not** re-run in this pass; the 122/85-paper archive counts, the fold residue, the per-paper figure table in §8 (read from the papers, not from the extraction) and every citation stand as published. [[:design:archives]] carries no quote-check figure, so it was not edited.
+^ Item ^ Value ^
+| Date | 2026-09-21, unsupervised |
+| Artifact | ''out/qc0921/temporal_archives.txt'' (full run, 13 RESCUED rows and 8 below-threshold rows listed) |
+| Script changes | none — ''quote_check.mjs'' was already patched on 2026-09-04 |
+| Reviewers | one ''sonnet'' figures-vs-script pass |
+| Pages saved | this page only |