Differences

This shows you the differences between two versions of the page.

--- programming:crawler:foxhound [2026/08/17 17:50] – Fix a bug in the published reducer found by an adversarial stress harness: script_of required a scripthash and discarded the filename, so the documented --unit script fallback never fired and every unhashed flow collapsed to one empty key. Adds regression karel.kubicek.claude
+++ programming:crawler:foxhound [2026/08/17 17:56] (current) – Fix a false negative in the published reducer, found by the figures re-review and reproduced before fixing: begin/end are UTF-16 code-unit offsets, and slicing with them in Python (which indexes by code point) reads the wrong substring whenever an astral karel.kubicek.claude
@@ Line 23: / Line 23: @@
 The **taint metadata** — ''report.detail.str.taint'', an array with one entry per tainted range:
-  * ''begin'' / ''end'' — **character offsets into ''str''**. This is the part people miss: Foxhound tells you //which five characters// of a 22-character string came from the source, so "a tainted value reached ''innerHTML''" can be qualified by how much of the sink argument the attacker controls.
+  * ''begin'' / ''end'' — **UTF-16 code-unit offsets into ''str''**, because a SpiderMonkey string is UTF-16. Slice with them in a language that indexes by code point — Python, Go, Rust — and a single emoji earlier in the string silently shifts every later index, so you read the wrong substring and can conclude a dangerous flow was harmless.  This is the part people miss: Foxhound tells you //which five characters// of a 22-character string came from the source, so "a tainted value reached ''innerHTML''" can be qualified by how much of the sink argument the attacker controls.
   * ''flow'' — an array of operation nodes, ordered **sink first, source last**. Each node carries ''operation'' (''concat'', ''substr'', ''unescape'', ''innerHTML'', ''location.hash'', or ''function'' for an application call), ''builtin'', ''source'', ''arguments'', and a ''location'' with ''filename'', ''line'', ''pos'', ''scriptline'' and a ''scripthash''.
@@ Line 236: / Line 236: @@
 # Sinks where HTML or JavaScript syntax in the tainted substring is what makes a
-# flow dangerous. For a network sink, syntax characters are irrelevant.
+# flow dangerous. `iframe.srcdoc` is in here because its content is parsed as a
+# whole HTML document, exactly like `document.write`. Navigation sinks
+# (`location.href`, `a.href`, `window.open`) are deliberately NOT here: they are
+# dangerous through the URL scheme (`javascript:`), which this syntax screen does
+# not model, so silence about them is honest rather than reassuring.
 HTML_JS_SINKS = {
     "innerHTML", "outerHTML", "insertAdjacentHTML", "document.write",
     "document.writeln", "eval", "Function.ctor", "script.text",
-    "script.innerHTML", "eventHandler", "setTimeout", "setInterval",
+    "script.innerHTML", "script.textContent", "eventHandler", "setTimeout",
-    "Range.createContextualFragment(fragment)",
+    "setInterval", "Range.createContextualFragment(fragment)", "iframe.srcdoc",
 }
 ENCODING_OPS = {"encodeURI", "encodeURIComponent", "escape"}
@@ Line 319: / Line 323: @@
             return False
     return False
+def utf16_slice(value: str, begin: int, end: int) -> str:
+    """Slice `value` by UTF-16 code units, which is how the engine counts.
+    Foxhound's `begin`/`end` are offsets into a SpiderMonkey string, and JS
+    strings are UTF-16. Python slices by CODE POINT, so a single character
+    outside the Basic Multilingual Plane anywhere earlier in the string (an
+    emoji, some CJK extensions) shifts every later Python index by one and the
+    slice silently returns the wrong substring. On "\U0001F600\U0001F600<>PADDING"
+    the engine's offsets 4..6 bound "<>", while `value[4:6]` returns "PA" — which
+    would report a dangerous flow as having held no syntax character.
+    """
+    units = value.encode("utf-16-le")
+    return units[2 * begin:2 * end].decode("utf-16-le", errors="replace")
+def utf16_len(value: str) -> int:
+    """Length of `value` in UTF-16 code units, i.e. in the engine's own unit."""
+    return len(value.encode("utf-16-le")) // 2
@@ Line 329: / Line 353: @@
     if sink not in HTML_JS_SINKS:
         return False
-    substring = value[taint_range["begin"]:taint_range["end"]]
+    substring = utf16_slice(value, taint_range["begin"], taint_range["end"])
     return not DANGEROUS.search(substring)
@@ Line 351: / Line 375: @@
                 "scripthash": scripthash,
                 "script": filename,
+                # In UTF-16 code units, the unit the offsets are expressed in.
                 "chars": taint_range["end"] - taint_range["begin"],
+                "chars_of": utf16_len(value),
                 "operations": [n["operation"] for n in flow],
                 "jit_blind": is_jit_blind(flow, harness_re),
@@ Line 549: / Line 575: @@
     check("no scripthash -> filename fallback", nh["script"], "https://domgo.at/inline")
     check("script unit falls back to filename", UNITS["script"](nh), "https://domgo.at/inline")
+    # UTF-16 offsets: an astral character before the range must not shift it.
+    astral = copy.deepcopy(WIKI_EXAMPLE)
+    astral["detail"]["str"] = "\U0001F600\U0001F600<>PADDING"
+    astral["detail"]["str_taint"][0]["begin"] = 4
+    astral["detail"]["str_taint"][0]["end"] = 6
+    check("utf16_slice finds the real substring", utf16_slice(astral["detail"]["str"], 4, 6), "<>")
+    check("naive python slice would have been wrong", astral["detail"]["str"][4:6], "PA")
+    check("astral shift does not hide a dangerous substring",
+          flows([astral])[0]["no_syntax_chars"], False)
+    check("string length is counted in UTF-16 units", utf16_len(astral["detail"]["str"]), 13)
+    # iframe.srcdoc is parsed as HTML, so it is in the syntax-screen sink set.
+    srcdoc = copy.deepcopy(WIKI_EXAMPLE)
+    srcdoc["detail"]["sink"] = "iframe.srcdoc"
+    check("srcdoc is screened", flows([srcdoc])[0]["no_syntax_chars"], True)
+    nav = copy.deepcopy(WIKI_EXAMPLE)
+    nav["detail"]["sink"] = "location.href"
+    check("navigation sinks are not screened", flows([nav])[0]["no_syntax_chars"], False)
     # An empty flow must not crash and must not be attributed to anything.
@@ Line 620: / Line 665: @@
 </file>
-Its self-test runs the flow from the project's own documentation plus ten mutations of it — including the three that matter most, a harness-only ''function'' node, an encode followed by a decode, and a location with no ''scripthash'' — and its real output is:
+Its self-test runs the flow from the project's own documentation plus thirteen mutations of it — including the four that matter most: a harness-only ''function'' node, an encode followed by a decode, a location with no ''scripthash'', and a tainted range sitting behind an astral character — and its real output is:
 <code>
 $ python3 foxhound_flows.py --selftest
-selftest: 25 checks passed
+selftest: 31 checks passed
 flows: 1   sites: 1   pages: 1   scripts: 1