Differences

This shows you the differences between two versions of the page.

--- provenance:programming:crawler:webxray:random_sample [2026/09/11 16:12] – Third-sitting fixes from the figures reviewer: absent-set invariant asserted in the kappa estimator, deterministic pe, guard extended to figures that live only in script text output (6 new mutations, 18 total). Authored by Claude karel.kubicek.claude
+++ provenance:programming:crawler:webxray:random_sample [2026/09/11 19:20] (current) – Point to the new residue_pass appendix; refresh the owner_verify_sources.py block after its whois:// / charset / rate-limit fixes. Authored by Claude karel.kubicek.claude
@@ Line 7: / Line 7: @@
 **inter-rater re-adjudication of 2026-09-11** (sections AA–AK), which measured
 how far a second model reading the same brief reaches the same verdicts.
+The **second pass over the residue**, run later the same day — which
+re-adjudicated the 40 domains this sample could not settle, moved 28 of them
+into the rates, and changed the content page's headline table and its Fisher
+table — has its own appendix, for exactly the reason given below:
+[[provenance:programming:crawler:webxray:residue_pass]]. Its blocks are another
+kB, most of it one probe log, and adding them here would have made this page
+kB. That page computes both figures when it is generated, from the same file
+lists both manifests use.
 Read [[provenance:programming:crawler:webxray|the provenance page]] first: the
@@ Line 17: / Line 26: @@
 AF–AK their outputs.
-**Why this is a separate page.** These blocks are 282 kB, and the parent
+**Why this is a separate page.** These blocks are 285 kB, and the parent
 provenance page is already large without them. Raw size alone would not decide
 it — ''literature:bibliography'' is 388 kB and renders fine — but ~60 kB of this
@@ Line 48: / Line 57: @@
 | C | ''out/adj/INSTRUCTIONS.md'' | 4,965 | ''1985026abac3d888'' | no |
 | D | ''scripts/owner_merge_rows.py'' | 4,572 | ''84ee130f2e8c2310'' | no |
-| E | ''scripts/owner_verify_sources.py'' | 8,860 | ''9d610c42aba5f08e'' | no |
+| E | ''scripts/owner_verify_sources.py'' | 12,417 | ''6c3778961aa5672c'' | no |
 | F | ''scripts/owner_verify_rendered.mjs'' | 2,923 | ''0bf6b4615724e84d'' | no |
 | G | ''scripts/owner_corrections.py'' | 6,847 | ''f145c0908d4bf9a6'' | no |
@@ Line 603: / Line 612: @@
     s = re.sub(r"[^a-z0-9\u00c0-\uffff]+", " ", s)
     return re.sub(r"\s+", " ", s).strip()
+def decode(body):
+    """Decode fetched bytes using the charset the document declares.
+    Decoding as UTF-8 with ``errors="replace"`` turns every byte of a
+    windows-1251 or shift_jis page into U+FFFD, so a Cyrillic or Japanese quote
+    can never match and the row is reported NOTFOUND -- a fabrication verdict on
+    a correct citation. The normaliser was already fixed once for this class of
+    bug (it used to delete non-Latin letters); this is the same bug one layer
+    earlier, found on 2026-09-11 when ``help.i.ua/agreement/`` was cited: that
+    page declares ``charset=windows-1251`` and every Ukrainian word in it
+    decoded to replacement characters.
+    Sniffed from the bytes rather than from the HTTP header, because the header
+    is not cached -- the cache holds the body alone. If nothing is declared, or
+    the declared codec is unknown, the old behaviour is the fallback.
+    """
+    m = re.search(rb"""charset=["']?([A-Za-z0-9_.:-]{2,24})""", body[:4096], re.I)
+    if m:
+        enc = m.group(1).decode("ascii", "ignore")
+        if enc.lower().replace("-", "") not in ("utf8", "utf"):
+            try:
+                return body.decode(enc)
+            except (LookupError, UnicodeDecodeError):
+                pass
+    return body.decode("utf8", "replace")
@@ Line 623: / Line 659: @@
         except Exception as e:
             return None, f"tls: {e}"
+        os.makedirs(cache, exist_ok=True)
+        open(path, "wb").write(out)
+        return out, None
+    if url.startswith("whois://"):
+        # The 2026-09-11 second pass over the residue cites registry and
+        # registrar WHOIS records, which are not reachable over HTTP: the
+        # protocol is a port-43 socket. Without this branch every such row
+        # FETCHFAILs, which reads as an unverifiable citation when in fact the
+        # checker had no client. The URL is `whois://<server>/<domain>` and the
+        # query is re-issued against the *named server*, not re-resolved
+        # through IANA, so the row is checked against the record it cites.
+        rest = url[len("whois://"):]
+        server, _, dom = rest.partition("/")
+        if not dom:
+            return None, "whois: URL must be whois://<server>/<domain>"
+        sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+        import time
+        import whois43
+        out = b""
+        # Two attempts, 20 s apart. One retry is enough: the observed limit is a
+        # burst limit, and both stubs in the first run answered correctly 25 s
+        # later. Retrying forever would turn a registrar's block into a hang.
+        for attempt in range(2):
+            if attempt:
+                time.sleep(20)
+            try:
+                out = whois43.ask(server, dom).encode("utf8")
+            except Exception as e:
+                return None, f"whois: {e}"
+            if len(out) >= 400 and dom.encode().lower() in out.lower():
+                break
+        # Registrar WHOIS servers rate-limit. MarkMonitor answers the fourth
+        # rapid query with a stub that has no registrant block, and caching that
+        # stub turns a correct citation into a permanent NOTFOUND: it happened
+        # to ``sa-as.com`` and ``blogblog.com`` in this script's first run on
+        # 2026-09-11, and both verified on a retry 25 seconds later. A short or
+        # domain-less answer is a transport failure, not evidence, so it is
+        # neither returned nor cached.
+        if len(out) < 400 or dom.encode().lower() not in out.lower():
+            return None, (f"whois: {len(out)}-byte answer from {server} does not name "
+                          f"{dom} after two attempts -- rate-limited, not cached")
         os.makedirs(cache, exist_ok=True)
         open(path, "wb").write(out)
@@ Line 674: / Line 751: @@
                             "url": url, "error": err})
             continue
-        hay = norm(body.decode("utf8", "replace"))
+        hay = norm(decode(body))
         nq = norm(quote)
         # A quote the adjudicator elided ("A ... B") is checked fragment by
@@ Line 686: / Line 763: @@
         # strip for every row would let a quote match an HTML comment or an
         # attribute it was never on.
-        raw_hay = norm(body.decode("utf8", "replace"), strip_tags=False)
+        raw_hay = norm(decode(body), strip_tags=False)
         if not nq:
             status = "NOQUOTE"
@@ Line 5683: / Line 5760: @@
   * [[provenance:programming:crawler:webxray]] — the provenance page these blocks belong to: queries, denominators, corrections, limits, reviewers.
   * [[programming:crawler:webxray]] — the content page the figures are on.
+  * [[provenance:programming:crawler:webxray:residue_pass]] — the same treatment for the 2026-09-11 second pass over the 40 rows this sample could not settle.
   * [[literature:corpus]] — corpus scope, the selection funnel, extraction stability.