Differences

This shows you the differences between two versions of the page.

--- provenance:statistics:regression [2026/08/19 18:32] – Remove a literal citation marker the bibtex plugin was eating; correct this page's own key count 24→29. Authored by Claude karel.kubicek.claude
+++ provenance:statistics:regression [2026/09/21 13:32] (current) – Quote-check refresh 2026-09-21: re-ran quote_check.mjs --statistics regression with the pypdf fallback; 93 below threshold -> 65 rescued + 27 below in both, partial 170 -> 171; new provenance section. Authored by Claude karel.kubicek.claude
@@ Line 13: / Line 13: @@
 | Corpus | ''data/extract/run1'', 5,859 extracted papers, 7 venues, 2010–2026 |
 | Page status | **New page.** ''statistics:regression'' was a red link promised from ''start'', from [[Statistics:Hypothesis testing]] ("where cluster-robust standard errors and mixed-effects models live"), from [[Statistics:Pvalue corrections]], and from [[provenance:statistics:hypothesis_testing]]. Nothing on the wiki covered it |
-| Scripts added | ''scripts/reg_fold.mjs'' (with a 36-case self-test: ''node scripts/reg_fold.mjs --test''), ''scripts/report_regression.mjs'' |
+| Scripts added | ''scripts/reg_fold.mjs'' (with a 36-case self-test: ''node scripts/reg_fold.mjs %%--test%%''), ''scripts/report_regression.mjs'' |
 | Code published on the page | ''scripts/counts_and_repeats.py'' |
 | BibTeX added | 28 entries: 13 corpus papers via ''bibgen.mjs'', 15 methodological references verified through Crossref (4 of them added after the external-currency review) |
@@ Line 814: / Line 814: @@
 ===== The simulation =====
-''scripts/counts_and_repeats.py'', published verbatim on the page as a ''<file python>'' block. Synthetic data, fixed seed ''20260819'', numpy 2.4.6, statsmodels 0.14.6. **Nothing in it is a measurement claim.**
+''scripts/counts_and_repeats.py'', published verbatim on the page as a ''%%<file python>%%'' block. Synthetic data, fixed seed ''20260819'', numpy 2.4.6, statsmodels 0.14.6. **Nothing in it is a measurement claim.**
 What it does and does not establish:
@@ Line 955: / Line 955: @@
 ==== Bulk check ====
-''node scripts/quote_check.mjs --statistics regression --show 400'' against ''data/fulltext/<year>/<venue>/<slug>/paper.cols.txt'':
+''node scripts/quote_check.mjs %%--statistics%% regression %%--show%% 400'' against ''data/fulltext/<year>/<venue>/<slug>/'' — the rendering each record's ''textSource'' names, and then, for anything below threshold, an independent ''pypdf'' rendering of the same ''paper.pdf''. **Re-run 2026-09-21**; see //Quote-check refresh, 2026-09-21// at the foot of this page for what moved:
 <code>
-quotes checked: 267 exact, 170 partial (>=60% of 5-word windows), 93 below threshold,
+quotes checked: 267 exact, 171 partial (>=60% of 5-word windows), 65 rescued from the PDF, 27 below threshold in both renderings, 0 with no full text on disk.
-with no full text on disk.
 </code>
-**Below-threshold is not "unsupported".** The five-word-window test is harsh on a fifteen-word quote (eleven windows), and a dropped citation marker or a two-column splice moves several windows at once. Example: USENIX/2020 ''understanding-security-mistakes'' scored below threshold because the extractor's quote omits the reference ''[15, 67-106]'' that sits inside the sentence in the PDF. The 93 were not individually chased.
+**Below-threshold is not "unsupported", and most of what used to be counted as below threshold is a defect in the stored text.** 65 of the old 93 are located in an independent rendering of the same PDF; **27** are below threshold in both, and those are the only ones a reader should treat as needing a hand check. The five-word-window test is also harsh on a short quote: 15 of the 27 score under 50%, and the set is dominated by short sentences and by sentences carrying mathematical notation the rendering does not reproduce — e.g. IEEE-SP/2015 ''secure-track-verification'' at 38%, whose quote contains ''%%||pi − px||%%''. The 27 were not individually chased.
+**The example this section used to give is no longer below threshold.** It named USENIX/2020 ''understanding-security-mistakes'' (the extractor's quote drops the reference ''[15, 67-106]'' that sits inside the sentence in the PDF). Today's script scores it at 75% of its 16 five-word windows, i.e. //partial//, both with and without line-break de-hyphenation. That is exactly the one-quote difference between the old ''170 partial / 93 below'' and the new ''171 partial / 65 + 27'': 267 + 171 + 65 + 27 = 530, and 170 + 93 = 263 = 171 + 65 + 27. **Why it moved could not be established** — the 2026-08-19 run left no output artifact, and the paper's ''textSource'' is ''cols'', so it is not one of the seven the 2026-09-04 ''textSource'' fix touched. It is recorded rather than explained away.
 ==== Every quote published on the page, checked by hand ====
@@ Line 1060: / Line 1061: @@
   * **Reconcile the cluster-robust count with [[Statistics:Hypothesis testing]]** (three there, four here) and the full-text denominator (5,855 there, 5,869 here). Belongs to whoever refreshes that page.
   * Merge the four duplicate bibliography pairs listed above, on whichever page owns them.
-  * ''quote_check.mjs'''s five-word-window threshold is harsh on short quotes; a length-aware threshold would cut the 93 below-threshold figure without loosening the test. Affects every page on this site, not just this one.
+  * ''quote_check.mjs'''s five-word-window threshold is harsh on short quotes; a length-aware threshold would cut the **27** below-threshold-in-both-renderings figure without loosening the test — 15 of the 27 score under 50%. Affects every page on this site, not just this one. ((Was "the 93 below threshold figure" before the 2026-09-21 refresh; the PDF fallback removed 65 of those 93 and one had already moved to //partial//.))
 ===== Review =====
@@ Line 1074: / Line 1075: @@
 | 4 | Part A's negative-binomial row was fitted with the **true** dispersion (''alpha=dispersion''), which is an oracle, while the page called the fix "one argument: ''family=NegativeBinomial()''". | **Accepted.** The script now fits both — alpha known and alpha estimated jointly by ''sm.NegativeBinomial'' — and the page distinguishes them. Both give 5.0%, so the conclusion holds and the framing was wrong |
 | 5 | ''\blm\b'' in the continuous rule is inert on this corpus but would misclassify a "Breusch-Pagan LM test"; ''\bCL\b\s*$'' likewise inert. | **Accepted.** Both removed |
-| 6 | Everything else reproduced byte-for-byte: the embedded ''<file python>'' block is identical to the committed script; residue 0; 18+19=37 exclusions with zero overlap; the 10-vs-11 causal-design split; the ''detail'' counts; the ICC arithmetic; the NB(mean, α)→numpy(n, p) reparameterisation; no RNG irreproducibility. | Noted. Four of these were figures the reviewer says it expected to find broken |
+| 6 | Everything else reproduced byte-for-byte: the embedded ''%%<file python>%%'' block is identical to the committed script; residue 0; 18+19=37 exclusions with zero overlap; the 10-vs-11 causal-design split; the ''detail'' counts; the ICC arithmetic; the NB(mean, α)→numpy(n, p) reparameterisation; no RNG irreproducibility. | Noted. Four of these were figures the reviewer says it expected to find broken |
 ==== Sonnet — citations and quotes ====
@@ Line 1124: / Line 1125: @@
 | 10 | Minor overstatements: GEE //"reviewers in this field will not recognise it"// (evidence is about authors); //"an inferential regression is not judged on fit"// stated absolutely; **Shmueli called "four pages" when it is 22, Ver Hoef & Boveng "four pages and one figure" when it is 7**. | **All accepted.** The two page-length claims are the worst of them: checkably wrong length claims on a page that boasts about verifying pagination through Project Euclid |
 | 11 | The ordinal/Likert bullet is user-study textbook material the page delegates elsewhere. | **Accepted.** Rewritten around a crawl's ordinal outcomes, pointing to [[Design:User studies]] for the Likert machinery |
-| 12 | Verified clean: rendering on both pages, all in-page and cross-page anchors resolve, the four red links are shared with the sibling and are planned pages, embedded outputs byte-identical to the committed files, ''reg_fold.mjs --test'' passes 36 cases, WRAP-box base rates check out. | Noted |
+| 12 | Verified clean: rendering on both pages, all in-page and cross-page anchors resolve, the four red links are shared with the sibling and are planned pages, embedded outputs byte-identical to the committed files, ''reg_fold.mjs %%--test%%'' passes 36 cases, WRAP-box base rates check out. | Noted |
 It also disclosed that both source files changed under it mid-review. They did — the author was applying the self-review fixes at the time — and the disclosure was right to make.
@@ Line 1146: / Line 1147: @@
 <bibtex bibliography></bibtex>
+===== Markup sweep, 2026-09-17 =====
+Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving.
+===== Quote-check refresh, 2026-09-21 =====
+The 2026-09-04 ''cols''-vs-PDF audit on [[:provenance:literature:corpus]] showed that 73.1% of evidence quotes that cannot be located in ''paper.cols.txt'' **are** present in an independent ''pypdf'' rendering of the same ''paper.pdf''. ''scripts/quote_check.mjs'' was patched the same day to re-check everything below threshold against that second rendering and report a fourth verdict, **RESCUED**. The //Bulk check// figure predates the patch and overstated this page's quote-failure rate. Re-run, unedited first line:
+<code>
+$ node scripts/quote_check.mjs --statistics regression --show 400
+quotes checked: 267 exact, 171 partial (>=60% of 5-word windows), 65 rescued from the PDF, 27 below threshold in both renderings, 0 with no full text on disk.
+</code>
+^ Figure ^ Was ^ Is ^ Why ^
+| quotes checked | 530 | 530 | population unchanged — the corpus has not moved |
+| exact | 267 | 267 | unchanged; 50.4% of 530 either way |
+| partial (≥60% of 5-word windows) | 170 | **171** | one quote moved up, for a reason this pass could not establish — see //Bulk check// |
+| rescued from the PDF | — | **65** | new verdict; these were inside the old 93 |
+| below threshold | **93** | **27** (in both renderings) | 93 = 1 + 65 + 27 |
+| below-threshold share of 530 | 17.5% | **5.1%** | 27 of 530, same share-of-530 convention the content page already uses |
+**What this does and does not say.** It does not say 65 extractions were wrong and are now right — the quotes were always in the papers. It says the //stored text// could not locate them and a second rendering of the same PDF can, so counting them as quote failures measured ''decolumn.mjs'', not the extraction. The //Every quote published on the page, checked by hand// table below the bulk check is a separate, hand pass over 16 quotes and is untouched: no published quote failed then and none of them is in the 27.
+**Scope of this edit.** The //Quotes / Bulk check// block, the TODO that quoted its 93, and the content page's provenance bullet. ''report_regression.mjs'' was **not** re-run in this pass and no fold, model-family figure, identification figure or citation was touched — every other number on [[:statistics:regression]] and on this page stands as published.
+^ Item ^ Value ^
+| Date | 2026-09-21, unsupervised |
+| Command | ''%%node scripts/quote_check.mjs --statistics regression --show 400%%'' |
+| Artifact | ''out/qc0921/stat_regression.txt'' (full run, 65 RESCUED rows and 27 below-threshold rows listed) |
+| Script changes | none — ''quote_check.mjs'' was already patched on 2026-09-04 |
+| Reviewers | one ''sonnet'' figures-vs-script pass over this page and [[:statistics:regression]] |
+| Pages saved | this page, [[:statistics:regression]] |
+| Not established | why one quote moved from //below threshold// to //partial//. Recorded in //Bulk check// rather than absorbed into the refresh |