Differences

This shows you the differences between two versions of the page.

--- provenance:statistics:interrater_agreement [2026/08/20 21:22] – New page: provenance for statistics:interrater_agreement — every query with its denominator, the report script and probes with unedited output, the fold and its full residue, quote spot-checks, external source verification, and the five-reviewer log inclu karel.kubicek.claude
+++ provenance:statistics:interrater_agreement [2026/09/17 08:14] (current) – Qualify root start link; Authored by Claude karel.kubicek.claude
@@ Line 13: / Line 13: @@
 | Page status | **New page.** Nothing on the wiki covered inter-rater agreement before this. |
 | Main agent | Claude Opus 5 |
-| Sub-agents | 1 research (external reference verification, Sonnet), 4 review (3 × Sonnet, 1 × Fable) — see [[#Review log]] |
+| Sub-agents | 1 research (external reference verification, Sonnet); 5 review passes — 3 focused (Sonnet) in parallel, 1 generic (Fable), then a re-run of the figures pass after its findings were acted on (Sonnet). The citations pass spawned two of its own for DOI resolution and quote checking. See [[#Review log]] |
 | Scripts committed | ''scripts/report_interrater.mjs'', ''scripts/ira_fold.mjs'', ''scripts/ira_fulltext_probes.sh'', ''scripts/kappa_on_skewed_labels.py'' |
@@ Line 23: / Line 23: @@
   * **[[Statistics:Hypothesis testing]]** is about inference on measurements. Reliability of the measurements is upstream of it, and that page is already 63 kB.
-The ''statistics:'' namespace was chosen over ''design:'' because the deliverable is a coefficient and a reporting standard, matching its five siblings. It is linked from [[start]] in the statistics line and from [[Practices:Ethics]]; those edits were made in the same sitting as this page, from the same run.
+The ''statistics:'' namespace was chosen over ''design:'' because the deliverable is a coefficient and a reporting standard, matching its five siblings. It is linked from [[:start]] in the statistics line and from [[Practices:Ethics]]; those edits were made in the same sitting as this page, from the same run.
 ===== Denominators used on the page =====
@@ Line 445: / Line 445: @@
 ===== Review log =====
-Four reviewers, all told explicitly that the author's context may not be exhaustive, all handed the page text, the report script, its unedited output and these notes.
+Five review passes, all told explicitly that the author's context may not be exhaustive, all handed the page text, the report script, its unedited output and these notes.
-=== The four reviewers ===
+=== The five review passes ===
 ^ Pass ^ Model ^ Verdict ^
@@ Line 453: / Line 453: @@
 | Citations and quotes | Sonnet | Resolved all 23 new DOIs; verified quotes against ''paper.cols.txt''. **3 findings, one serious.** |
 | External currency | Sonnet | Fetched PyPI, CRAN, Crossref, arXiv and the SAGE catalogue. **2 findings + 1 suggestion.** |
-| Generic | Fable | See below. |
+| Generic | Fable | No checklist. **17 findings**, including the three most serious defects on this page. |
+| Figures vs script, re-run | Sonnet | Run again after the probe script was rewritten. All three scripts reproduce byte-for-byte; **2 findings**, both minor. |
 === Findings accepted and fixed ===
@@ Line 459: / Line 460: @@
 ^ # ^ Reviewer ^ Finding ^ Action ^
 | 1 | citations | **A quote was attributed to the wrong paper.** The page had Sun, Vekaria & Nithyanand {[sun2026_suitability]} saying their annotators worked //"without seeing LLM responses"//. A full-file grep of that paper finds the phrase nowhere. The sentence is in fact from Zhang et al. {[zhang2025_abusability]}, USENIX Security 2025, whose extraction record sits a few lines away in the same query output. **This is the most serious defect the review caught and it was an authoring error, not an extraction error** — the quote was real, the attribution was not. | Re-attributed to {[zhang2025_abusability]}, verified in that paper's full text, entry added to the bibliography, spot-check table row added above. |
-| 2 | figures | **Undisclosed denominator.** "the median double-coded sample here is 385 items and 22% double-code fewer than 50" appeared on the content page with no population named. The figures are over the **457** papers that name a statistic //and// state a coded sample size (89.3% of the 512). | Denominator stated in the ''<WRAP>'' block, again beside the simulation, and again in the methodology list. |
+| 2 | figures | **Undisclosed denominator.** "the median double-coded sample here is 385 items and 22% double-code fewer than 50" appeared on the content page with no population named. The figures are over the **457** papers that name a statistic //and// state a coded sample size (89.3% of the 512). | Denominator stated in the ''%%<WRAP>%%'' block, again beside the simulation, and again in the methodology list. |
 | 3 | citations | **A quote reordered its source's clauses.** The page printed //"average 0.81; range 0.64–0.93"// as a direct quote from {[skolka2019_anything]}; the paper says //"The pairwise agreement ranges between 0.64 and 0.93, with an average of 0.81"//. Content identical, word order not. | Replaced with the paper's own sentence, verbatim. |
 | 4 | citations | **Bibliographic error.** {[khatun2026_disclosure]} was entered with ''number = {3}''; the DOI record says issue 4. | Corrected before the bibliography was saved. |
@@ Line 495: / Line 496: @@
 <bibtex bibliography></bibtex>
+===== Markup sweep, 2026-09-17 =====
+Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving.
+===== Link-hygiene sweep, 2026-09-17 =====
+The fresh rendered-DOM sweep found this page's bare ''start'' link resolved to ''provenance:statistics:start''. It is now root-anchored; the post-save DOM was re-checked for red links. No figures or citations changed.