| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| literature:corpus [2026/08/12 16:06] – Second review round: name the 138 poster records and the 3 title-level duplicates as structural properties of the funnel; add a boxed statement that an aggregate count cannot be independently recounted; widen the hand-checked quote sample to 2025-2026 and karel.kubicek.claude | literature:corpus [2026/09/17 08:12] (current) – Record mirror and sitemap decisions; Authored by Claude karel.kubicek.claude |
|---|
| |
| This page is the root of that namespace. Corpus-level caveats — everything below — are stated here once, and each ''provenance:'' page carries only what is specific to its own page. | This page is the root of that namespace. Corpus-level caveats — everything below — are stated here once, and each ''provenance:'' page carries only what is specific to its own page. |
| | |
| | The rule applies to this page too: its own working notes are at **[[provenance:literature:corpus]]**. Sections 1–5 and 7–11 below were written on 2026-08-12 and keep their notes inline, in §9 and §11; the 2026-09-04 cross-rendering audit in §6 has its query log, its scripts and its unedited output there. |
| |
| ===== 1. Scope: seven venues, 2010–2026 ===== | ===== 1. Scope: seven venues, 2010–2026 ===== |
| ==== Quote groundedness, re-measured on this corpus ==== | ==== Quote groundedness, re-measured on this corpus ==== |
| |
| Every quote is supposed to be verbatim, so every quote can be looked for in the text the extractor read. All **135,025** were, on 2026-08-12, by ''scripts/report_corpus.mjs --quotes'': | Every quote is supposed to be verbatim, so every quote can be looked for in the text the extractor read. All **135,025** were, on 2026-08-12, by ''scripts/report_corpus.mjs %%--quotes%%'': |
| |
| ^ Verdict ^ Quotes ^ Share ^ | ^ Verdict ^ Quotes ^ Share ^ |
| The failures are also spread thin rather than concentrated: 1,143 of 5,859 papers (19.5%) have at least one, and the ten worst papers hold only 82 of the 1,708. The rate is flat across all seventeen years, between 0.8% and 1.5% in every single one, which is what you would expect of a PDF-layout artefact and not of a model that got worse or better over time. | The failures are also spread thin rather than concentrated: 1,143 of 5,859 papers (19.5%) have at least one, and the ten worst papers hold only 82 of the 1,708. The rate is flat across all seventeen years, between 0.8% and 1.5% in every single one, which is what you would expect of a PDF-layout artefact and not of a model that got worse or better over time. |
| |
| **Two caveats about this measurement.** It reproduces the dataset's own audit of the earlier corpus almost exactly — that audit reported 57.9% / 37.0% / 4.2% / 0.9% on the same four buckets — but the two implementations are independent and the "fragmented" test in particular is an approximation of the original, so the 4.0%/1.3% split between the last two rows should be read as one combined 5.3% that is mostly layout damage. And the first measurement of this ran the check against the wrong file for seven papers whose text had been OCR-repaired, which put one paper at the top of the not-found ranking with 29 phantom failures; the script now reads whichever rendering the extractor read. | **Two caveats about this measurement.** It reproduces the dataset's own audit of the earlier corpus almost exactly — that audit reported 57.9% / 37.0% / 4.2% / 0.9% on the same four buckets — but the two implementations are independent and the "fragmented" test in particular is an approximation of the original, so the 4.0%/1.3% split between the last two rows should be read as one combined 5.3% that is mostly layout damage. **That last clause was an argument from ten examples until 2026-09-04, when it was measured; the next subsection is the measurement.** And the first measurement of this ran the check against the wrong file for seven papers whose text had been OCR-repaired, which put one paper at the top of the not-found ranking with 29 phantom failures; the script now reads whichever rendering the extractor read. |
| | |
| | ==== How much of that 1.3% is the stored text rather than the extraction? ==== |
| | |
| | The paragraph above argues, from ten quotes read by hand, that the //not found// row is column-splice damage and not fabrication. On **2026-09-04** that argument was measured instead of asserted, because every check on this site reads the same ''paper.cols.txt'' and a defect in that file is therefore invisible to all of them. |
| | |
| | **Method.** Each of the 135,025 quotes was replayed twice through the same four-verdict function: once against the rendering the extractor read, and once against an **independent** rendering of the same stored ''paper.pdf'', produced by ''pypdf''. ''pypdf'' reads the PDF's own text operators and never looks at the column geometry, so it fails differently — which is the entire point. 5,858 of the 5,859 papers could be rendered; the one that could not is named on [[provenance:literature:corpus]]. That leaves **135,004 quotes** with a verdict on both sides. The stored-rendering column below reproduces ''report_corpus.mjs %%--quotes%%'' to the quote: the four buckets differ by 15, 6, 0 and 0, which sum to exactly the 21 skipped quotes. |
| | |
| | ^ Verdict ^ Against the stored rendering ^ ^ Against the PDF ^ ^ |
| | | present exactly | 78,435 | 58.1% | 79,326 | 58.8% | |
| | | an 8-word run survives | 49,419 | 36.6% | 48,595 | 36.0% | |
| | | fragmented | 5,442 | 4.0% | 3,963 | 2.9% | |
| | | not found | 1,708 | 1.3% | 3,120 | 2.3% | |
| | |
| | **The answer: about three quarters of it.** |
| | |
| | ^ Of the 1,708 quotes the stored rendering cannot locate ^ Quotes ^ Share ^ |
| | | Present **verbatim** in the PDF rendering | 511 | 29.9% | |
| | | Present as an 8-word run | 602 | 35.2% | |
| | | Present but fragmented | 137 | 8.0% | |
| | | **Not found in either rendering** | **458** | **26.8%** | |
| | |
| | So **1,113 of the 1,708 (65.2%) are present in the PDF rendering verbatim or as an unbroken 8-word run**, and a further 137 are there fragmented — every content word inside one window, the order broken. Either way the stored text is what is wrong, for **1,250 of the 1,708 (73.2%)**. The residue — quotes neither rendering can locate — is **458, or 0.34% of all quotes checked**, and even that is a ceiling rather than a fabrication rate: reading a sample of it finds ellipsis splices the extraction marked itself, reflowed table rows, small-capital rendering (%%"S ABOT"%%), symbol runs, at least one of the twelve sentences this page counts as describing a paper's silence instead of quoting it, and at least one case where the extraction changed a single word (//"Automated Surface Observation System"// for the paper's //"Observing"//). One paper in seven holds a rescued quote: **862 of 5,859 (14.7%)**. |
| | |
| | **The comparison is not one-sided, and that is the reason to fall back rather than to switch.** 2,662 quotes (2.0%) go the other way — found in the stored rendering, not found in the PDF one — because ''pypdf'' inserts spurious spaces inside words (//"bo th"//, //"christop her"//) and has reading-order failures of its own. Its overall not-found rate, 2.3%, is worse than the stored rendering's 1.3%. **''pypdf'' is a second opinion, never an oracle:** it can rescue a quote the stored rendering missed and must never be used to condemn one the stored rendering found. |
| | |
| | === The mechanism, and its size === |
| | |
| | ''decolumn.mjs'' repairs two-column reading order **one page at a time**, and leaves a page untouched whenever its gutter estimator returns nothing — fewer than six long lines, a page narrower than 60 columns, or fewer than half of the long lines voting for the winning gutter column. A page left untouched keeps ''pdftotext -layout'''s side-by-side columns, and the later whitespace collapse then splices the left column to the right on every line. Of the corpus's **91,509 pages**, 62,162 (67.9%) were reflowed and **29,347 (32.1%) were left untouched**; none were rejected by the word-multiset check. |
| | |
| | An untouched page is only a defect if it really was two-column. That was **estimated** from the PDF's own geometry — the x-histogram of where each text run starts, which is bimodal on a two-column page — on a stratified sample of 400 untouched pages, plus 60 pages ''decolumn.mjs'' //did// reflow as a positive control, which it calls two-column 60 times out of 60. It is a second automated detector, not a human reading, and it has **no negative control**: no set of known single-column pages was assembled to bound its false-positive rate. Full table, band by band, on [[provenance:literature:corpus]]. |
| | |
| | Between 70% and 95% of the untouched pages in every substantial band turn out to be two-column. Weighting each band's corpus-wide page count by its measured share gives **an estimated 22,475 pages — 24.6% of the corpus, 95% CI 21.5%–26.6% — that were genuinely two-column and were never separated.** The interval is the band-wise Wilson intervals summed, which is conservative; a tighter and equally defensible one exists and is named on the provenance page. |
| | |
| | That is a large number sitting behind a small one, and the two are reconciled by which verdict the damage lands in. A ''pdftotext'' line holds roughly ten words, so a spliced page keeps runs of eight words intact and only destroys longer contiguity. The prediction is therefore that //exact// should fall with the share of a paper's pages left unrepaired while //8-word run// absorbs it, and //not found// should barely move. It does, monotonically, across 5,853 papers: |
| | |
| | ^ Share of the paper's pages left unrepaired ^ Papers ^ Quotes ^ present exactly ^ 8-word run ^ fragmented ^ not found ^ |
| | | 0% | 585 | 12,486 | 70.6% | 26.7% | 2.2% | 0.6% | |
| | | 0–10% | 951 | 22,898 | 65.1% | 31.2% | 2.9% | 0.7% | |
| | | 10–20% | 2,009 | 47,855 | 59.5% | 35.3% | 4.0% | 1.3% | |
| | | 20–35% | 1,846 | 42,335 | 52.5% | 40.9% | 4.9% | 1.6% | |
| | | 35–50% | 353 | 7,652 | 43.7% | 48.6% | 5.8% | 1.9% | |
| | | 50%+ | 109 | 1,734 | 36.6% | 56.8% | 5.2% | 1.4% | |
| | |
| | **Read the 58.1% //exact// row that way.** It is not mainly a measure of how verbatim the extraction is; it is in large part a measure of how often the page a quote came from had its columns separated, and a paper with no unrepaired page reaches 70.6%. This is a paper-level association, not an isolated cause: page layout, venue template, year and how table-heavy a paper is all move together, and no attempt was made to separate them. |
| | |
| | === The order is wrong; almost nothing is missing === |
| | |
| | The distinction matters, because a rendering that has //lost// text cannot be repaired by a better reader and one that has //re-ordered// it can. Comparing both renderings with every space, mark and hyphen removed — so no spacing, ligature or hyphenation difference can move the number — the stored rendering holds **1.005 times** the characters of the PDF one at the median, and the 5th percentile is 1.000. Only **34 of 5,856 papers (0.6%)** fall below 0.95, and four of those are artefacts of the comparison rather than of the corpus: two are papers whose extractor input was OCR-repaired text, not ''paper.cols.txt'', and two are papers where ''pypdf'' emitted a plot's PostScript glyph names (''/SolidCircle'' once per data point) and inflated its own side. The remaining 30 — 0.5% of the corpus — are listed in full on [[provenance:literature:corpus]] and **none of them was read**: the worst holds 58% of the PDF's characters, which is a shortfall and not an absence, but what is missing from those thirty is not known. |
| | |
| | === The trap this audit came out of === |
| | |
| | The audit exists because [[provenance:statistics:hypothesis_testing]] published, on 2026-08-13, that one paper's methodology quotes were in its PDF and in **none** of its three stored renderings. **That was false** — both quoted sentences are in its ''paper.cols.txt'' — and the retraction, with the shell transcript that explains it, is on that page and on [[provenance:literature:corpus]]. One consequence generalises and belongs here: **''pdftotext'' writes a NUL byte wherever a font encoding maps a glyph to nothing, so 90 of the 5,859 stored renderings (1.5%) are "binary" files holding 2,156 evidence quotes (1.6%), and some greps report no match in them without saying so.** If you check a quote by hand, pass ''grep -a'' and normalise whitespace first. |
| | |
| | === What was changed as a result === |
| | |
| | * ''quote_check.mjs'' and ''ht_quotecheck.mjs'' now re-check everything below threshold against the PDF rendering and report a fourth verdict, **RESCUED**, instead of counting it against the paper. Rendering is on demand and cached, so a clean run costs nothing. |
| | * Both scripts now key on each record's own ''textSource'' rather than always reading ''paper.cols.txt''. Seven papers were OCR-repaired and checking those against ''paper.cols.txt'' scores a broken font encoding as a fabricated quote — the bug ''report_corpus.mjs'' had already had to fix, still live in the two per-page checkers. ((That fix was described here as done when it had only been made in ''quote_check.mjs''; a reviewer found ''ht_quotecheck.mjs'' still taking whichever of ''cols''/''norm'' existed first. Fixing it moved one hypothesis-test quote from below-threshold to //partial//, so [[statistics:hypothesis_testing]] now reads 541 partial and 216 below threshold rather than 540 and 217.)) |
| | * ''pypdf'''s 75 MB decompression limits are raised in both audit scripts, because five of the corpus's PDFs are legitimately larger and were leaving 151 quotes with no second rendering. |
| | * **The four records with no ''paper.cols.txt'' are not a defect.** All four are papers with no usable text layer; ''decolumn.mjs'' deliberately skips a ''paper.txt'' under 5,000 characters and removes any stale output, and the extractor read ''paper.mistral.md'' for them. |
| | * **Not fixed: the renderings themselves.** The right repair is a better gutter estimator in ''decolumn.mjs'' and a re-render, which would move the ''exact'' rate on this page substantially. The dataset is mounted read-only here, so that is the dataset repository's job and is logged as a follow-up. |
| | * **Not re-run: the per-page quote-check figures already published across the site.** Roughly sixty pages carry one, and each would fall by a similar proportion. Only [[statistics:hypothesis_testing]], whose own figures prompted the audit, has been refreshed. |
| | |
| | Every query, every script, its unedited output, the folds, the hand-read quotes and what could not be established are on **[[provenance:literature:corpus]]**. |
| |
| ==== Field stability: measured once, on the earlier corpus, never since ==== | ==== Field stability: measured once, on the earlier corpus, never since ==== |
| |
| **3. Count papers, never tuples.** A paper naming EasyList three times is one paper: 145 ''EasyList'' tuples across 94 papers, 1.54 per paper. | **3. Count papers, never tuples.** A paper naming EasyList three times is one paper: 145 ''EasyList'' tuples across 94 papers, 1.54 per paper. |
| | |
| | **3b. A quote-check figure published before 2026-09-04 is an upper bound on failure, not a failure rate.** Every such figure was computed against ''paper.cols.txt'' alone, and §6 shows that about a quarter of the corpus's pages have their two columns spliced together, so a quote from one of them cannot be found however verbatim it is. Read //present exactly// plus //an 8-word run survives// — 94.7% of all quotes — as **located**, and treat the remaining 5.3% as a ceiling. The two per-page checkers now fall back to a second rendering of the PDF and report a fourth verdict, RESCUED; **the 57 pages that already publish one of these figures have not been re-run** and are named on [[provenance:literature:corpus]]. |
| |
| **4. Fold free-text names before aggregating, and print what the fold could not map.** This is the rule that is easiest to skip and most expensive to skip. The worked case is ''vantage[].locations'', which is free text — "Frankfurt, Germany", "US-East", "a US-based university network": | **4. Fold free-text names before aggregating, and print what the fold could not map.** This is the rule that is easiest to skip and most expensive to skip. The worked case is ''vantage[].locations'', which is free text — "Frankfurt, Germany", "US-East", "a US-based university network": |
| ===== 9. The report script and its output ===== | ===== 9. The report script and its output ===== |
| |
| Every figure on this page comes from one script, which prints all of them with their denominators, plus the checks that the funnel is internally consistent. Its unedited output, as printed on 2026-08-12, is in the [[#Appendix: the report script's full output|appendix]] at the foot of this page and is downloadable from there. | Every figure on this page **except those in the 2026-09-04 subsection of §6** comes from one script, which prints all of them with their denominators, plus the checks that the funnel is internally consistent. That subsection's figures come from ''report_cols_vs_pdf.mjs'' and ''report_cols_mechanism.mjs'', and its two tables are sliced out of their real output rather than typed; both scripts and their unedited output are on [[provenance:literature:corpus]]. Its unedited output, as printed on 2026-08-12, is in the [[#Appendix: the report script's full output|appendix]] at the foot of this page and is downloadable from there. |
| |
| <code bash> | <code bash> |
| | Model | Claude Opus 5 | | | Model | Claude Opus 5 | |
| | Figures carried over from earlier notes | **None.** Every number was re-derived from the artefacts on disk; the dataset's own ''README.md'' still quotes the 4,322-paper figures and was not used as a source. | | | Figures carried over from earlier notes | **None.** Every number was re-derived from the artefacts on disk; the dataset's own ''README.md'' still quotes the 4,322-paper figures and was not used as a source. | |
| | Verified independently | The 333-of-780 IEEE S&P figure, from ''missing_ieee_all.jsonl'' holding exactly 447 records (780 − 447 = 333). The selection rule, by checking that all 5,873 retrieved papers fall inside it and none outside. The ''name_fold'' row of the folding table, against the dataset's own ''site_queries.mjs --page vantage'', which reports the same 367. | | | Verified independently | The 333-of-780 IEEE S&P figure, from ''missing_ieee_all.jsonl'' holding exactly 447 records (780 − 447 = 333). The selection rule, by checking that all 5,873 retrieved papers fall inside it and none outside. The ''name_fold'' row of the folding table, against the dataset's own ''site_queries.mjs %%--page%% vantage'', which reports the same 367. | |
| | Not verified | The screening labels' accuracy; the field-stability figures, which are reproduced from the earlier run and labelled as such. | | | Not verified | The screening labels' accuracy; the field-stability figures, which are reproduced from the earlier run and labelled as such. | |
| | Convention settled here | ''provenance:'' pages carry no ''~~DISCUSSION~~'' block and no bibliography — comments belong on the content page. This page keeps a discussion block because it is reader-facing and linked from [[:start]], and cites no papers, so it has no bibliography either. | | | Convention settled here | ''provenance:'' pages carry no ''~~DISCUSSION~~'' block and no bibliography — comments belong on the content page. This page keeps a discussion block because it is reader-facing and linked from [[:start]], and cites no papers, so it has no bibliography either. | |
| | Why there is no ''provenance:literature:corpus'' | This page //is// a provenance page: sections 9 to 11 are its own working log. A provenance page for the provenance page would recurse without adding anything. | | | Why there **is** a ''provenance:literature:corpus'' | It was written on 2026-09-04, and the reasoning that there should not be one is recorded here because it held until then: this page //is// a provenance page, sections 9 to 11 are its own working log, and a provenance page for the provenance page looked like recursion. What changed the call was the cross-rendering audit in §6, which adds eight scripts and 300 lines of output. Those are on [[provenance:literature:corpus]]; sections 1–5 and 7–11 keep their notes inline, as before. | |
| | Wired into | [[:start]]; the six ''provenance:'' pages, which already link here; and the //Methodology and limitations of these figures// section of each corpus-backed content page, where the generic "these venues are absent, 2025–2026 are provisional" text was replaced by a pointer here plus the page-specific consequence. | | | Wired into | [[:start]]; every ''provenance:'' page, which links here for the corpus-level caveats (six when this row was written, and its own mirror [[provenance:literature:corpus]] since 2026-09-04); and the //Methodology and limitations of these figures// section of each corpus-backed content page, where the generic "these venues are absent, 2025–2026 are provisional" text was replaced by a pointer here plus the page-specific consequence. | |
| | Mistakes caught before review | The quote check initially read ''paper.cols.txt'' for all papers, including the 7 whose extractor input was OCR text; that scored a broken font encoding as 29 fabricated quotes in a single paper and put it top of the not-found ranking. Fixed by keying on each record's own ''textSource''. | | | Mistakes caught before review | The quote check initially read ''paper.cols.txt'' for all papers, including the 7 whose extractor input was OCR text; that scored a broken font encoding as 29 fabricated quotes in a single paper and put it top of the not-found ranking. Fixed by keying on each record's own ''textSource''. | |
| |
| - Whether a paper that produced no tuple for a family is silent or was missed. | - Whether a paper that produced no tuple for a family is silent or was missed. |
| </file> | </file> |
| | |
| | ===== Link-sweep decision, 2026-09-17 ===== |
| | |
| | Keep the pre-rule exemption for [[programming:crux]] and [[programming:similarweb]]: both content pages predate the ship-a-provenance-page rule, so this sweep does not create mirrors. [[privacy:cookies]] remains excluded because its own page-refresh item covers it. [[provenance:programming:interaction:outputs]] is an intentional provenance-only overflow log for one interaction log and has no content counterpart; it is exempt from the mirror-content-id rule. |
| |
| /* This enables discussion under this article. */ | /* This enables discussion under this article. */ |
| ~~DISCUSSION~~ | ~~DISCUSSION~~ |
| |
