| Both sides previous revisionPrevious revision | |
| provenance:statistics:regression [2026/09/17 07:25] – Markup rendering sweep: wrap escapes, CLI flag typography, and heading quotes. Authored by Claude. karel.kubicek.claude | provenance:statistics:regression [2026/09/21 13:32] (current) – Quote-check refresh 2026-09-21: re-ran quote_check.mjs --statistics regression with the pypdf fallback; 93 below threshold -> 65 rescued + 27 below in both, partial 170 -> 171; new provenance section. Authored by Claude karel.kubicek.claude |
|---|
| ==== Bulk check ==== | ==== Bulk check ==== |
| |
| ''node scripts/quote_check.mjs %%--statistics%% regression %%--show%% 400'' against ''data/fulltext/<year>/<venue>/<slug>/paper.cols.txt'': | ''node scripts/quote_check.mjs %%--statistics%% regression %%--show%% 400'' against ''data/fulltext/<year>/<venue>/<slug>/'' — the rendering each record's ''textSource'' names, and then, for anything below threshold, an independent ''pypdf'' rendering of the same ''paper.pdf''. **Re-run 2026-09-21**; see //Quote-check refresh, 2026-09-21// at the foot of this page for what moved: |
| |
| <code> | <code> |
| 530 quotes checked: 267 exact, 170 partial (>=60% of 5-word windows), 93 below threshold, | 530 quotes checked: 267 exact, 171 partial (>=60% of 5-word windows), 65 rescued from the PDF, 27 below threshold in both renderings, 0 with no full text on disk. |
| 0 with no full text on disk. | |
| </code> | </code> |
| |
| **Below-threshold is not "unsupported".** The five-word-window test is harsh on a fifteen-word quote (eleven windows), and a dropped citation marker or a two-column splice moves several windows at once. Example: USENIX/2020 ''understanding-security-mistakes'' scored below threshold because the extractor's quote omits the reference ''[15, 67-106]'' that sits inside the sentence in the PDF. The 93 were not individually chased. | **Below-threshold is not "unsupported", and most of what used to be counted as below threshold is a defect in the stored text.** 65 of the old 93 are located in an independent rendering of the same PDF; **27** are below threshold in both, and those are the only ones a reader should treat as needing a hand check. The five-word-window test is also harsh on a short quote: 15 of the 27 score under 50%, and the set is dominated by short sentences and by sentences carrying mathematical notation the rendering does not reproduce — e.g. IEEE-SP/2015 ''secure-track-verification'' at 38%, whose quote contains ''%%||pi − px||%%''. The 27 were not individually chased. |
| | |
| | **The example this section used to give is no longer below threshold.** It named USENIX/2020 ''understanding-security-mistakes'' (the extractor's quote drops the reference ''[15, 67-106]'' that sits inside the sentence in the PDF). Today's script scores it at 75% of its 16 five-word windows, i.e. //partial//, both with and without line-break de-hyphenation. That is exactly the one-quote difference between the old ''170 partial / 93 below'' and the new ''171 partial / 65 + 27'': 267 + 171 + 65 + 27 = 530, and 170 + 93 = 263 = 171 + 65 + 27. **Why it moved could not be established** — the 2026-08-19 run left no output artifact, and the paper's ''textSource'' is ''cols'', so it is not one of the seven the 2026-09-04 ''textSource'' fix touched. It is recorded rather than explained away. |
| |
| ==== Every quote published on the page, checked by hand ==== | ==== Every quote published on the page, checked by hand ==== |
| * **Reconcile the cluster-robust count with [[Statistics:Hypothesis testing]]** (three there, four here) and the full-text denominator (5,855 there, 5,869 here). Belongs to whoever refreshes that page. | * **Reconcile the cluster-robust count with [[Statistics:Hypothesis testing]]** (three there, four here) and the full-text denominator (5,855 there, 5,869 here). Belongs to whoever refreshes that page. |
| * Merge the four duplicate bibliography pairs listed above, on whichever page owns them. | * Merge the four duplicate bibliography pairs listed above, on whichever page owns them. |
| * ''quote_check.mjs'''s five-word-window threshold is harsh on short quotes; a length-aware threshold would cut the 93 below-threshold figure without loosening the test. Affects every page on this site, not just this one. | * ''quote_check.mjs'''s five-word-window threshold is harsh on short quotes; a length-aware threshold would cut the **27** below-threshold-in-both-renderings figure without loosening the test — 15 of the 27 score under 50%. Affects every page on this site, not just this one. ((Was "the 93 below threshold figure" before the 2026-09-21 refresh; the PDF fallback removed 65 of those 93 and one had already moved to //partial//.)) |
| |
| ===== Review ===== | ===== Review ===== |
| |
| Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving. | Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving. |
| | |
| | ===== Quote-check refresh, 2026-09-21 ===== |
| | |
| | The 2026-09-04 ''cols''-vs-PDF audit on [[:provenance:literature:corpus]] showed that 73.1% of evidence quotes that cannot be located in ''paper.cols.txt'' **are** present in an independent ''pypdf'' rendering of the same ''paper.pdf''. ''scripts/quote_check.mjs'' was patched the same day to re-check everything below threshold against that second rendering and report a fourth verdict, **RESCUED**. The //Bulk check// figure predates the patch and overstated this page's quote-failure rate. Re-run, unedited first line: |
| | |
| | <code> |
| | $ node scripts/quote_check.mjs --statistics regression --show 400 |
| | 530 quotes checked: 267 exact, 171 partial (>=60% of 5-word windows), 65 rescued from the PDF, 27 below threshold in both renderings, 0 with no full text on disk. |
| | </code> |
| | |
| | ^ Figure ^ Was ^ Is ^ Why ^ |
| | | quotes checked | 530 | 530 | population unchanged — the corpus has not moved | |
| | | exact | 267 | 267 | unchanged; 50.4% of 530 either way | |
| | | partial (≥60% of 5-word windows) | 170 | **171** | one quote moved up, for a reason this pass could not establish — see //Bulk check// | |
| | | rescued from the PDF | — | **65** | new verdict; these were inside the old 93 | |
| | | below threshold | **93** | **27** (in both renderings) | 93 = 1 + 65 + 27 | |
| | | below-threshold share of 530 | 17.5% | **5.1%** | 27 of 530, same share-of-530 convention the content page already uses | |
| | |
| | **What this does and does not say.** It does not say 65 extractions were wrong and are now right — the quotes were always in the papers. It says the //stored text// could not locate them and a second rendering of the same PDF can, so counting them as quote failures measured ''decolumn.mjs'', not the extraction. The //Every quote published on the page, checked by hand// table below the bulk check is a separate, hand pass over 16 quotes and is untouched: no published quote failed then and none of them is in the 27. |
| | |
| | **Scope of this edit.** The //Quotes / Bulk check// block, the TODO that quoted its 93, and the content page's provenance bullet. ''report_regression.mjs'' was **not** re-run in this pass and no fold, model-family figure, identification figure or citation was touched — every other number on [[:statistics:regression]] and on this page stands as published. |
| | |
| | ^ Item ^ Value ^ |
| | | Date | 2026-09-21, unsupervised | |
| | | Command | ''%%node scripts/quote_check.mjs --statistics regression --show 400%%'' | |
| | | Artifact | ''out/qc0921/stat_regression.txt'' (full run, 65 RESCUED rows and 27 below-threshold rows listed) | |
| | | Script changes | none — ''quote_check.mjs'' was already patched on 2026-09-04 | |
| | | Reviewers | one ''sonnet'' figures-vs-script pass over this page and [[:statistics:regression]] | |
| | | Pages saved | this page, [[:statistics:regression]] | |
| | | Not established | why one quote moved from //below threshold// to //partial//. Recorded in //Bulk check// rather than absorbed into the refresh | |
| |
