Differences

This shows you the differences between two versions of the page.

--- programming:crawler:foxhound [2026/08/17 17:48] – Re-review fixes: attribute the DOM-XSS-decline reading to Sabino et al.'s multi-factor discussion instead of a 'rather than the web got safer' dichotomy they do not assert; restore the paper's 'likely' hedge on why PanoptiChrome's authors missed its unres karel.kubicek.claude
+++ programming:crawler:foxhound [2026/09/21 14:32] (current) – Reconciling footnote updated: the 8 on programming:crawler and privacy:javascript is now 9; what still differs is the cites-only column. Authored by Claude karel.kubicek.claude
@@ Line 23: / Line 23: @@
 The **taint metadata** — ''report.detail.str.taint'', an array with one entry per tainted range:
-  * ''begin'' / ''end'' — **character offsets into ''str''**. This is the part people miss: Foxhound tells you //which five characters// of a 22-character string came from the source, so "a tainted value reached ''innerHTML''" can be qualified by how much of the sink argument the attacker controls.
+  * ''begin'' / ''end'' — **UTF-16 code-unit offsets into ''str''**, because a SpiderMonkey string is UTF-16. Slice with them in a language that indexes by code point — Python, Go, Rust — and a single emoji earlier in the string silently shifts every later index, so you read the wrong substring and can conclude a dangerous flow was harmless.  This is the part people miss: Foxhound tells you //which five characters// of a 22-character string came from the source, so "a tainted value reached ''innerHTML''" can be qualified by how much of the sink argument the attacker controls.
   * ''flow'' — an array of operation nodes, ordered **sink first, source last**. Each node carries ''operation'' (''concat'', ''substr'', ''unescape'', ''innerHTML'', ''location.hash'', or ''function'' for an application call), ''builtin'', ''source'', ''arguments'', and a ''location'' with ''filename'', ''line'', ''pos'', ''scriptline'' and a ''scripthash''.
@@ Line 236: / Line 236: @@
 # Sinks where HTML or JavaScript syntax in the tainted substring is what makes a
-# flow dangerous. For a network sink, syntax characters are irrelevant.
+# flow dangerous. `iframe.srcdoc` is in here because its content is parsed as a
+# whole HTML document, exactly like `document.write`. Navigation sinks
+# (`location.href`, `a.href`, `window.open`) are deliberately NOT here: they are
+# dangerous through the URL scheme (`javascript:`), which this syntax screen does
+# not model, so silence about them is honest rather than reassuring.
 HTML_JS_SINKS = {
     "innerHTML", "outerHTML", "insertAdjacentHTML", "document.write",
     "document.writeln", "eval", "Function.ctor", "script.text",
-    "script.innerHTML", "eventHandler", "setTimeout", "setInterval",
+    "script.innerHTML", "script.textContent", "eventHandler", "setTimeout",
-    "Range.createContextualFragment(fragment)",
+    "setInterval", "Range.createContextualFragment(fragment)", "iframe.srcdoc",
 }
 ENCODING_OPS = {"encodeURI", "encodeURIComponent", "escape"}
@@ Line 253: / Line 257: @@
 # call that reports the flow as an ordinary `function` node, so without this the
 # JIT-blindness metric and the scripthash key both describe your own code.
-# Override with --harness if your harness script is named differently.
+# This is a NAME heuristic and it cuts both ways: a page function called
+# something like `reportTaintSinkStats` would be misclassified as harness code.
+# Set --harness to a pattern that matches your harness and nothing else.
 HARNESS_RE = re.compile(r"taint_reporting|ReportTaintSink|__playwright", re.I)
@@ Line 277: / Line 283: @@
 def script_of(flow: list[dict], harness_re: re.Pattern) -> tuple[str, str]:
-    """(scripthash, filename) of the node nearest the sink that is page code."""
+    """(scripthash, filename) of the node nearest the sink that is page code.
+    A node can carry a filename with no scripthash (inline handlers, and nodes
+    the engine could not attribute to a compiled script). Keep the first such
+    filename so `--unit script` has something to fall back on rather than
+    silently collapsing every unhashed flow into one empty key.
+    """
+    fallback = ""
     for node in flow:
         if is_harness(node, harness_re):
@@ Line 284: / Line 297: @@
         if loc.get("scripthash"):
             return loc["scripthash"], loc.get("filename", "")
-    return "", ""
+        if not fallback and loc.get("filename"):
+            fallback = loc["filename"]
+    return "", fallback
@@ Line 308: / Line 323: @@
             return False
     return False
+def utf16_slice(value: str, begin: int, end: int) -> str:
+    """Slice `value` by UTF-16 code units, which is how the engine counts.
+    Foxhound's `begin`/`end` are offsets into a SpiderMonkey string, and JS
+    strings are UTF-16. Python slices by CODE POINT, so a single character
+    outside the Basic Multilingual Plane anywhere earlier in the string (an
+    emoji, some CJK extensions) shifts every later Python index by one and the
+    slice silently returns the wrong substring. On "\U0001F600\U0001F600<>PADDING"
+    the engine's offsets 4..6 bound "<>", while `value[4:6]` returns "PA" — which
+    would report a dangerous flow as having held no syntax character.
+    """
+    units = value.encode("utf-16-le")
+    return units[2 * begin:2 * end].decode("utf-16-le", errors="replace")
+def utf16_len(value: str) -> int:
+    """Length of `value` in UTF-16 code units, i.e. in the engine's own unit."""
+    return len(value.encode("utf-16-le")) // 2
@@ Line 318: / Line 353: @@
     if sink not in HTML_JS_SINKS:
         return False
-    substring = value[taint_range["begin"]:taint_range["end"]]
+    substring = utf16_slice(value, taint_range["begin"], taint_range["end"])
     return not DANGEROUS.search(substring)
@@ Line 340: / Line 375: @@
                 "scripthash": scripthash,
                 "script": filename,
+                # In UTF-16 code units, the unit the offsets are expressed in.
                 "chars": taint_range["end"] - taint_range["begin"],
+                "chars_of": utf16_len(value),
                 "operations": [n["operation"] for n in flow],
                 "jit_blind": is_jit_blind(flow, harness_re),
@@ Line 526: / Line 563: @@
           "d7069063759edbf2dcf45741802bc405")
     check("harness script still does not clear jit_blind", flows([hh])[0]["jit_blind"], True)
+    # A filename with no scripthash is still a usable key for --unit script.
+    nohash = copy.deepcopy(WIKI_EXAMPLE)
+    for n in nohash["detail"]["str_taint"][0]["flow"]:
+        loc = n.get("location") or {}
+        loc.pop("scripthash", None)
+        loc["filename"] = "https://domgo.at/inline"
+        n["location"] = loc
+    nh = flows([nohash])[0]
+    check("no scripthash -> empty hash", nh["scripthash"], "")
+    check("no scripthash -> filename fallback", nh["script"], "https://domgo.at/inline")
+    check("script unit falls back to filename", UNITS["script"](nh), "https://domgo.at/inline")
+    # UTF-16 offsets: an astral character before the range must not shift it.
+    astral = copy.deepcopy(WIKI_EXAMPLE)
+    astral["detail"]["str"] = "\U0001F600\U0001F600<>PADDING"
+    astral["detail"]["str_taint"][0]["begin"] = 4
+    astral["detail"]["str_taint"][0]["end"] = 6
+    check("utf16_slice finds the real substring", utf16_slice(astral["detail"]["str"], 4, 6), "<>")
+    check("naive python slice would have been wrong", astral["detail"]["str"][4:6], "PA")
+    check("astral shift does not hide a dangerous substring",
+          flows([astral])[0]["no_syntax_chars"], False)
+    check("string length is counted in UTF-16 units", utf16_len(astral["detail"]["str"]), 13)
+    # iframe.srcdoc is parsed as HTML, so it is in the syntax-screen sink set.
+    srcdoc = copy.deepcopy(WIKI_EXAMPLE)
+    srcdoc["detail"]["sink"] = "iframe.srcdoc"
+    check("srcdoc is screened", flows([srcdoc])[0]["no_syntax_chars"], True)
+    nav = copy.deepcopy(WIKI_EXAMPLE)
+    nav["detail"]["sink"] = "location.href"
+    check("navigation sinks are not screened", flows([nav])[0]["no_syntax_chars"], False)
+    # An empty flow must not crash and must not be attributed to anything.
+    empty = copy.deepcopy(WIKI_EXAMPLE)
+    empty["detail"]["str_taint"][0]["flow"] = []
+    er = flows([empty])[0]
+    check("empty flow source", er["source"], "unattributed")
+    check("empty flow is jit_blind", er["jit_blind"], True)
+    check("empty flow not encoded", er["encoded_at_sink"], False)
     # No source-flagged node must not be relabelled.
@@ Line 589: / Line 665: @@
 </file>
-Its self-test runs the flow from the project's own documentation plus seven mutations of it — including the two that matter most, a harness-only ''function'' node and an encode followed by a decode — and its real output is:
+Its self-test runs the flow from the project's own documentation plus thirteen mutations of it — including the four that matter most: a harness-only ''function'' node, an encode followed by a decode, a location with no ''scripthash'', and a tainted range sitting behind an astral character — and its real output is:
 <code>
 $ python3 foxhound_flows.py --selftest
-selftest: 19 checks passed
+selftest: 31 checks passed
 flows: 1   sites: 1   pages: 1   scripts: 1
@@ Line 609: / Line 685: @@
 ===== Use in publications =====
-The full-text sweep ''/fox ?hound/i'' over the corpus returns **13 papers**. Two are homographs — an ImageNet class label ''americanfoxhound'' and "English Foxhound" as an example crowdsourcing label — and are excluded from every figure here. Two cite the project without running it, one of them in order to reject it {[liu2025_domino]}. That leaves **9 papers that actually ran the browser**, out of the **1,120 papers in the corpus that ran a crawl** (0.8%).((This is one more than the 8 in the specialised-crawler table on [[Programming:Crawler]] and in the tool table on [[Privacy:Javascript]], and the difference is a real correction rather than a different denominator. Those tables count extracted ''tools[]'' tuples by ''usedOrMentioned'', which files Khodayari et al.'s NDSS 2025 paper under ''compared'' and therefore in their "cites only" column; reading the sentence shows it ran the browser on 42,288 pages as one of six baseline detectors. Those two pages are queued for an errata edit; this page is the deeper audit.)) The corpus is the seven venues on [[literature:corpus]]; EuroS&P, where the tool is described, is not among them, and the project's own list of publications naming Foxhound has 14 entries across more venues.((wiki [[https://github.com/SAP/project-foxhound/wiki/Publications|Publications]] page, checked 2026-08-17.))
+The full-text sweep ''/fox ?hound/i'' over the corpus returns **13 papers**. Two are homographs — an ImageNet class label ''americanfoxhound'' and "English Foxhound" as an example crowdsourcing label — and are excluded from every figure here. Two cite the project without running it, one of them in order to reject it {[liu2025_domino]}. That leaves **9 papers that actually ran the browser**, out of the **1,120 papers in the corpus that ran a crawl** (0.8%).((**The two tool tables that used to say 8 now say 9, corrected on 2026-09-21.** The specialised-crawler table on [[Programming:Crawler]] and the tool table on [[Privacy:Javascript]] count extracted ''tools[]'' tuples by ''usedOrMentioned'', which filed Khodayari et al.'s NDSS 2025 paper under ''compared'' and therefore in their "cites only" column; reading the sentence shows it ran the browser on 42,288 pages as one of six baseline detectors. Both report scripts now read the role verdicts below for this tool, so the three pages agree at 9. They still differ in what else they can see: neither table counts {[liu2025_domino]} or the PanoptiChrome paper, because a paper that only cites Foxhound in related work has no ''tools[]'' tuple for it, so their "cites only" columns read 0 where this page reports 2.)) The corpus is the seven venues on [[literature:corpus]]; EuroS&P, where the tool is described, is not among them, and the project's own list of publications naming Foxhound has 14 entries across more venues.((wiki [[https://github.com/SAP/project-foxhound/wiki/Publications|Publications]] page, checked 2026-08-17.))
 ^ Year ^ Corpus papers ^ Ran Foxhound ^ Share of that year ^
@@ Line 655: / Line 731: @@
   * **The confirmation step**, if you claim vulnerabilities rather than flows: payload generation, canary, or manual review, with the confirmed / refuted / unconfirmed counts kept separate. Do not report flows as vulnerabilities.
   * **What you did with flows whose chain recorded no application function call**, if any claim depends on the operations in the chain — the JIT blind spot is systematic.
-  * Counts of attempted, loaded, crashed and timed-out pages, **counted by you**: the shipped build has ''--disable-crashreporter'', so nothing else will count them. Its user agent is also roughly a year behind stable, which belongs in the limitations.
+  * Counts of attempted, loaded, crashed and timed-out pages, **counted by you**: the shipped build has ''%%--disable-crashreporter%%'', so nothing else will count them. Its user agent is also roughly a year behind stable, which belongs in the limitations.
   * Your harness. The init script, the binding, and the reducer that turns reports into rows are where the interesting decisions live, and none of them are visible from "we used Foxhound with Playwright".