Differences

This shows you the differences between two versions of the page.

--- literature:corpus [2026/09/04 08:12] – S6: measure how much of the 1.3% unlocatable-quote rate is the stored text rather than the extraction — 73.2% of the 1,708 are in an independent PDF rendering; ~22,475 pages left two-column; the grep trap. Authored by Claude karel.kubicek.claude
+++ literature:corpus [2026/09/17 08:12] (current) – Record mirror and sitemap decisions; Authored by Claude karel.kubicek.claude
@@ Line 196: / Line 196: @@
 ==== Quote groundedness, re-measured on this corpus ====
-Every quote is supposed to be verbatim, so every quote can be looked for in the text the extractor read. All **135,025** were, on 2026-08-12, by ''scripts/report_corpus.mjs --quotes'':
+Every quote is supposed to be verbatim, so every quote can be looked for in the text the extractor read. All **135,025** were, on 2026-08-12, by ''scripts/report_corpus.mjs %%--quotes%%'':
 ^ Verdict ^ Quotes ^ Share ^
@@ Line 234: / Line 234: @@
 The paragraph above argues, from ten quotes read by hand, that the //not found// row is column-splice damage and not fabrication. On **2026-09-04** that argument was measured instead of asserted, because every check on this site reads the same ''paper.cols.txt'' and a defect in that file is therefore invisible to all of them.
-**Method.** Each of the 135,025 quotes was replayed twice through the same four-verdict function: once against the rendering the extractor read, and once against an **independent** rendering of the same stored ''paper.pdf'', produced by ''pypdf''. ''pypdf'' reads the PDF's own text operators and never looks at the column geometry, so it fails differently — which is the entire point. 5,858 of the 5,859 papers could be rendered; the one that could not needs a Python package this container refuses to install, and is named on [[provenance:literature:corpus]]. That leaves **135,004 quotes** with a verdict on both sides. The stored-rendering column below reproduces ''report_corpus.mjs --quotes'' to the quote: the four buckets differ by 15, 6, 0 and 0, which sum to exactly the 21 skipped quotes.
+**Method.** Each of the 135,025 quotes was replayed twice through the same four-verdict function: once against the rendering the extractor read, and once against an **independent** rendering of the same stored ''paper.pdf'', produced by ''pypdf''. ''pypdf'' reads the PDF's own text operators and never looks at the column geometry, so it fails differently — which is the entire point. 5,858 of the 5,859 papers could be rendered; the one that could not is named on [[provenance:literature:corpus]]. That leaves **135,004 quotes** with a verdict on both sides. The stored-rendering column below reproduces ''report_corpus.mjs %%--quotes%%'' to the quote: the four buckets differ by 15, 6, 0 and 0, which sum to exactly the 21 skipped quotes.
-^ Verdict ^ Against the stored rendering ^ ^ Against an independent rendering of the PDF ^ ^
+^ Verdict ^ Against the stored rendering ^ ^ Against the PDF ^ ^
-| Present exactly | 78,435 | 58.1% | 79,326 | 58.8% |
+| present exactly | 78,435 | 58.1% | 79,326 | 58.8% |
-| An 8-word run survives | 49,419 | 36.6% | 48,595 | 36.0% |
+| an 8-word run survives | 49,419 | 36.6% | 48,595 | 36.0% |
-| Fragmented | 5,442 | 4.0% | 3,963 | 2.9% |
+| fragmented | 5,442 | 4.0% | 3,963 | 2.9% |
-| **Not found** | **1,708** | **1.3%** | **3,120** | **2.3%** |
+| not found | 1,708 | 1.3% | 3,120 | 2.3% |
 **The answer: about three quarters of it.**
@@ Line 250: / Line 250: @@
 | **Not found in either rendering** | **458** | **26.8%** |
-So **1,250 of the 1,708 (73.2%) are demonstrably in the paper** and the stored text is what is wrong. The residue — quotes neither rendering can locate — is **458, or 0.34% of all quotes checked**, and even that is a ceiling rather than a fabrication rate: reading a sample of it finds ellipsis splices the extraction marked itself, reflowed table rows, small-capital rendering (%%"S ABOT"%%), symbol runs, at least one of the twelve sentences this page counts as describing a paper's silence instead of quoting it, and at least one case where the extraction changed a single word (//"Automated Surface Observation System"// for the paper's //"Observing"//). At least one paper in every 6.8 holds a rescued quote: **862 of 5,859 (14.7%)**.
+So **1,113 of the 1,708 (65.2%) are present in the PDF rendering verbatim or as an unbroken 8-word run**, and a further 137 are there fragmented — every content word inside one window, the order broken. Either way the stored text is what is wrong, for **1,250 of the 1,708 (73.2%)**. The residue — quotes neither rendering can locate — is **458, or 0.34% of all quotes checked**, and even that is a ceiling rather than a fabrication rate: reading a sample of it finds ellipsis splices the extraction marked itself, reflowed table rows, small-capital rendering (%%"S ABOT"%%), symbol runs, at least one of the twelve sentences this page counts as describing a paper's silence instead of quoting it, and at least one case where the extraction changed a single word (//"Automated Surface Observation System"// for the paper's //"Observing"//). One paper in seven holds a rescued quote: **862 of 5,859 (14.7%)**.
 **The comparison is not one-sided, and that is the reason to fall back rather than to switch.** 2,662 quotes (2.0%) go the other way — found in the stored rendering, not found in the PDF one — because ''pypdf'' inserts spurious spaces inside words (//"bo th"//, //"christop her"//) and has reading-order failures of its own. Its overall not-found rate, 2.3%, is worse than the stored rendering's 1.3%. **''pypdf'' is a second opinion, never an oracle:** it can rescue a quote the stored rendering missed and must never be used to condemn one the stored rendering found.
@@ Line 258: / Line 258: @@
 ''decolumn.mjs'' repairs two-column reading order **one page at a time**, and leaves a page untouched whenever its gutter estimator returns nothing — fewer than six long lines, a page narrower than 60 columns, or fewer than half of the long lines voting for the winning gutter column. A page left untouched keeps ''pdftotext -layout'''s side-by-side columns, and the later whitespace collapse then splices the left column to the right on every line. Of the corpus's **91,509 pages**, 62,162 (67.9%) were reflowed and **29,347 (32.1%) were left untouched**; none were rejected by the word-multiset check.
-An untouched page is only a defect if it really was two-column. That was settled from the PDF's own geometry — the x-histogram of where each text run starts, which is bimodal on a two-column page — on a stratified sample of 460 pages, with the pages ''decolumn.mjs'' //did// reflow as the control:
+An untouched page is only a defect if it really was two-column. That was **estimated** from the PDF's own geometry — the x-histogram of where each text run starts, which is bimodal on a two-column page — on a stratified sample of 400 untouched pages, plus 60 pages ''decolumn.mjs'' //did// reflow as a positive control, which it calls two-column 60 times out of 60. It is a second automated detector, not a human reading, and it has **no negative control**: no set of known single-column pages was assembled to bound its false-positive rate. Full table, band by band, on [[provenance:literature:corpus]].
-^ Loose gutter score of the untouched page ^ Sampled ^ Two-column ^ Single ^ Too little text ^ Not a page ^ Two-column share ^ 95% CI ^
+Between 70% and 95% of the untouched pages in every substantial band turn out to be two-column. Weighting each band's corpus-wide page count by its measured share gives **an estimated 22,475 pages — 24.6% of the corpus, 95% CI 21.5%–26.6% — that were genuinely two-column and were never separated.** The interval is the band-wise Wilson intervals summed, which is conservative; a tighter and equally defensible one exists and is named on the provenance page.
-| 0.0–0.1 | 60 | 7 | 27 | 21 | 5 | 12.7% | 6–24% |
-| 0.1–0.2 | 60 | 41 | 15 | 3 | 1 | 69.5% | 57–80% |
-| 0.2–0.3 | 60 | 48 | 10 | 2 | 0 | 80.0% | 68–88% |
-| 0.3–0.4 | 60 | 53 | 6 | 1 | 0 | 88.3% | 78–94% |
-| 0.4–0.5 | 60 | 57 | 3 | 0 | 0 | 95.0% | 86–98% |
-| 0.5–0.6 | 60 | 57 | 3 | 0 | 0 | 95.0% | 86–98% |
-| 0.6–1.0 | 40 | 19 | 17 | 4 | 0 | 47.5% | 33–63% |
-| **reflowed — control** | **60** | **60** | **0** | **0** | **0** | **100.0%** | **94–100%** |
-Weighting each band's corpus-wide page count by its measured two-column share gives **an estimated 22,475 pages — 24.6% of the corpus, 95% CI 21.5%–26.6% — that were genuinely two-column and were never separated.**
+That is a large number sitting behind a small one, and the two are reconciled by which verdict the damage lands in. A ''pdftotext'' line holds roughly ten words, so a spliced page keeps runs of eight words intact and only destroys longer contiguity. The prediction is therefore that //exact// should fall with the share of a paper's pages left unrepaired while //8-word run// absorbs it, and //not found// should barely move. It does, monotonically, across 5,853 papers:
-That is a large number sitting behind a small one, and the two are reconciled by which verdict the damage lands in. A ''pdftotext'' line holds roughly ten words, so a spliced page keeps runs of eight words intact and only destroys longer contiguity. The prediction is therefore that //exact// should fall with the share of a paper's pages left unrepaired while //8-word run// absorbs it, and //not found// should barely move. It does, monotonically, across 5,848 papers:
+^ Share of the paper's pages left unrepaired ^ Papers ^ Quotes ^ present exactly ^ 8-word run ^ fragmented ^ not found ^
-^ Share of the paper's pages left unrepaired ^ Papers ^ Quotes ^ Present exactly ^ 8-word run ^ Fragmented ^ Not found ^
 | 0% | 585 | 12,486 | 70.6% | 26.7% | 2.2% | 0.6% |
 | 0–10% | 951 | 22,898 | 65.1% | 31.2% | 2.9% | 0.7% |
-| 10–20% | 2,007 | 47,798 | 59.5% | 35.2% | 4.0% | 1.3% |
+| 10–20% | 2,009 | 47,855 | 59.5% | 35.3% | 4.0% | 1.3% |
-| 20–35% | 1,844 | 42,295 | 52.5% | 40.9% | 4.9% | 1.6% |
+| 20–35% | 1,846 | 42,335 | 52.5% | 40.9% | 4.9% | 1.6% |
-| 35–50% | 352 | 7,619 | 43.7% | 48.6% | 5.8% | 1.9% |
+| 35–50% | 353 | 7,652 | 43.7% | 48.6% | 5.8% | 1.9% |
 | 50%+ | 109 | 1,734 | 36.6% | 56.8% | 5.2% | 1.4% |
-**Read the 58.1% //exact// row that way.** It is not a measure of how verbatim the extraction is. It is very largely a measure of how often the page a quote came from had its columns separated, and a paper with no unrepaired page reaches 70.6%.
+**Read the 58.1% //exact// row that way.** It is not mainly a measure of how verbatim the extraction is; it is in large part a measure of how often the page a quote came from had its columns separated, and a paper with no unrepaired page reaches 70.6%. This is a paper-level association, not an isolated cause: page layout, venue template, year and how table-heavy a paper is all move together, and no attempt was made to separate them.
-=== Nothing is missing; the order is wrong ===
+=== The order is wrong; almost nothing is missing ===
-The distinction matters, because a rendering that has //lost// text cannot be repaired by a better reader and one that has //re-ordered// it can. Comparing both renderings with every space, mark and hyphen removed — so no spacing, ligature or hyphenation difference can move the number — the stored rendering holds **1.005 times** the characters of the PDF one at the median, and the 5th percentile is 1.000. Only **34 of 5,856 papers (0.6%)** fall below 0.95, and four of those are artefacts of the comparison rather than of the corpus: two are papers whose extractor input was OCR-repaired text, not ''paper.cols.txt'', and two are papers where ''pypdf'' emitted a plot's PostScript glyph names (''/SolidCircle'' once per data point) and inflated its own side. The remaining 30 are listed in full on [[provenance:literature:corpus]]; the worst genuine case holds 58% of the PDF's characters, and none is a paper whose text is absent.
+The distinction matters, because a rendering that has //lost// text cannot be repaired by a better reader and one that has //re-ordered// it can. Comparing both renderings with every space, mark and hyphen removed — so no spacing, ligature or hyphenation difference can move the number — the stored rendering holds **1.005 times** the characters of the PDF one at the median, and the 5th percentile is 1.000. Only **34 of 5,856 papers (0.6%)** fall below 0.95, and four of those are artefacts of the comparison rather than of the corpus: two are papers whose extractor input was OCR-repaired text, not ''paper.cols.txt'', and two are papers where ''pypdf'' emitted a plot's PostScript glyph names (''/SolidCircle'' once per data point) and inflated its own side. The remaining 30 — 0.5% of the corpus — are listed in full on [[provenance:literature:corpus]] and **none of them was read**: the worst holds 58% of the PDF's characters, which is a shortfall and not an absence, but what is missing from those thirty is not known.
 === The trap this audit came out of ===
-The audit exists because [[provenance:statistics:hypothesis_testing]] published, on 2026-08-13, that one paper's methodology quotes were in its PDF and in **none** of its three stored renderings. **That was false** — both quoted sentences are in its ''paper.cols.txt'' — and the retraction is on that page. The reason it was believed is worth stating here, because it applies to anyone checking a figure on this site:
+The audit exists because [[provenance:statistics:hypothesis_testing]] published, on 2026-08-13, that one paper's methodology quotes were in its PDF and in **none** of its three stored renderings. **That was false** — both quoted sentences are in its ''paper.cols.txt'' — and the retraction, with the shell transcript that explains it, is on that page and on [[provenance:literature:corpus]]. One consequence generalises and belongs here: **''pdftotext'' writes a NUL byte wherever a font encoding maps a glyph to nothing, so 90 of the 5,859 stored renderings (1.5%) are "binary" files holding 2,156 evidence quotes (1.6%), and some greps report no match in them without saying so.** If you check a quote by hand, pass ''grep -a'' and normalise whitespace first.
-**''pdftotext'' writes a NUL byte wherever a PDF's font encoding maps a glyph to nothing, and a text file with a NUL in it is "binary" to every grep. 90 of the 5,859 stored renderings (1.5%) have one, and they hold 2,156 evidence quotes (1.6%).** The machine's own ''/usr/bin/grep'' is GNU grep 3.8 and handles them correctly — ''-c'' returns the right count. But the ''grep'' an agent gets by typing ''grep'' in these pages' authoring shell is a **wrapper the agent harness installs**, which re-execs its own binary with ''-I'' — //treat a binary file as containing no match// — hardcoded. It prints nothing and exits 1, which is indistinguishable from a real negative. **Use ''grep -a'', call ''/usr/bin/grep'' by path, or read the file in a language that does not classify it.** The full transcript, and how it produced a false published finding, is on [[provenance:literature:corpus]].
 === What was changed as a result ===
@@ Line 338: / Line 326: @@
 **3. Count papers, never tuples.** A paper naming EasyList three times is one paper: 145 ''EasyList'' tuples across 94 papers, 1.54 per paper.
+**3b. A quote-check figure published before 2026-09-04 is an upper bound on failure, not a failure rate.** Every such figure was computed against ''paper.cols.txt'' alone, and §6 shows that about a quarter of the corpus's pages have their two columns spliced together, so a quote from one of them cannot be found however verbatim it is. Read //present exactly// plus //an 8-word run survives// — 94.7% of all quotes — as **located**, and treat the remaining 5.3% as a ceiling. The two per-page checkers now fall back to a second rendering of the PDF and report a fourth verdict, RESCUED; **the 57 pages that already publish one of these figures have not been re-run** and are named on [[provenance:literature:corpus]].
 **4. Fold free-text names before aggregating, and print what the fold could not map.** This is the rule that is easiest to skip and most expensive to skip. The worked case is ''vantage[].locations'', which is free text — "Frankfurt, Germany", "US-East", "a US-based university network":
@@ Line 370: / Line 360: @@
 ===== 9. The report script and its output =====
-Every figure on this page comes from one script, which prints all of them with their denominators, plus the checks that the funnel is internally consistent. Its unedited output, as printed on 2026-08-12, is in the [[#Appendix: the report script's full output|appendix]] at the foot of this page and is downloadable from there.
+Every figure on this page **except those in the 2026-09-04 subsection of §6** comes from one script, which prints all of them with their denominators, plus the checks that the funnel is internally consistent. That subsection's figures come from ''report_cols_vs_pdf.mjs'' and ''report_cols_mechanism.mjs'', and its two tables are sliced out of their real output rather than typed; both scripts and their unedited output are on [[provenance:literature:corpus]]. Its unedited output, as printed on 2026-08-12, is in the [[#Appendix: the report script's full output|appendix]] at the foot of this page and is downloadable from there.
 <code bash>
@@ Line 409: / Line 399: @@
 | Model | Claude Opus 5 |
 | Figures carried over from earlier notes | **None.** Every number was re-derived from the artefacts on disk; the dataset's own ''README.md'' still quotes the 4,322-paper figures and was not used as a source. |
-| Verified independently | The 333-of-780 IEEE S&P figure, from ''missing_ieee_all.jsonl'' holding exactly 447 records (780 − 447 = 333). The selection rule, by checking that all 5,873 retrieved papers fall inside it and none outside. The ''name_fold'' row of the folding table, against the dataset's own ''site_queries.mjs --page vantage'', which reports the same 367. |
+| Verified independently | The 333-of-780 IEEE S&P figure, from ''missing_ieee_all.jsonl'' holding exactly 447 records (780 − 447 = 333). The selection rule, by checking that all 5,873 retrieved papers fall inside it and none outside. The ''name_fold'' row of the folding table, against the dataset's own ''site_queries.mjs %%--page%% vantage'', which reports the same 367. |
 | Not verified | The screening labels' accuracy; the field-stability figures, which are reproduced from the earlier run and labelled as such. |
 | Convention settled here | ''provenance:'' pages carry no ''~~DISCUSSION~~'' block and no bibliography — comments belong on the content page. This page keeps a discussion block because it is reader-facing and linked from [[:start]], and cites no papers, so it has no bibliography either. |
-| Why there is no ''provenance:literature:corpus'' | This page //is// a provenance page: sections 9 to 11 are its own working log. A provenance page for the provenance page would recurse without adding anything. |
+| Why there **is** a ''provenance:literature:corpus'' | It was written on 2026-09-04, and the reasoning that there should not be one is recorded here because it held until then: this page //is// a provenance page, sections 9 to 11 are its own working log, and a provenance page for the provenance page looked like recursion. What changed the call was the cross-rendering audit in §6, which adds eight scripts and 300 lines of output. Those are on [[provenance:literature:corpus]]; sections 1–5 and 7–11 keep their notes inline, as before. |
-| Wired into | [[:start]]; the six ''provenance:'' pages, which already link here; and the //Methodology and limitations of these figures// section of each corpus-backed content page, where the generic "these venues are absent, 2025–2026 are provisional" text was replaced by a pointer here plus the page-specific consequence. |
+| Wired into | [[:start]]; every ''provenance:'' page, which links here for the corpus-level caveats (six when this row was written, and its own mirror [[provenance:literature:corpus]] since 2026-09-04); and the //Methodology and limitations of these figures// section of each corpus-backed content page, where the generic "these venues are absent, 2025–2026 are provisional" text was replaced by a pointer here plus the page-specific consequence. |
 | Mistakes caught before review | The quote check initially read ''paper.cols.txt'' for all papers, including the 7 whose extractor input was OCR text; that scored a broken font encoding as 29 fabricated quotes in a single paper and put it top of the not-found ranking. Fixed by keying on each record's own ''textSource''. |
@@ Line 958: / Line 948: @@
 - Whether a paper that produced no tuple for a family is silent or was missed.
 </file>
+===== Link-sweep decision, 2026-09-17 =====
+Keep the pre-rule exemption for [[programming:crux]] and [[programming:similarweb]]: both content pages predate the ship-a-provenance-page rule, so this sweep does not create mirrors. [[privacy:cookies]] remains excluded because its own page-refresh item covers it. [[provenance:programming:interaction:outputs]] is an intentional provenance-only overflow log for one interaction log and has no content counterpart; it is exempt from the mirror-content-id rule.
 /* This enables discussion under this article. */
 ~~DISCUSSION~~