Differences

This shows you the differences between two versions of the page.

--- programming:traffic_files [2026/08/14 03:35] – Apply three review passes: fix a wrong serverIPAddress that an allowlist entry had hidden from the guard, a 3x/2x arithmetic error, a fold that classed Squid/Privoxy as commercial VPNs, Playwright's HAR API and the WebSocket-is-a-writer-feature caveat, Se karel.kubicek.claude
+++ programming:traffic_files [2026/08/14 03:41] (current) – Apply the generic review pass: fix an overstated lead box, an internal contradiction in the dataset/format counts, an unmeasured 'most papers give none of them', a limitations bullet that contradicted the Replay section, and an error-swallowing line in th karel.kubicek.claude
@@ Line 6: / Line 6: @@
 <WRAP important>
-**The one thing to take away.** The three places you can tap — the packet stream, an intercepting proxy, the browser's own internals — are **not nested**, and neither is a superset of the others. Measured on a single page load of a local fixture ([[#What We Ran]]):
+**The one thing to take away.** The three places you can tap — the packet stream, an intercepting proxy, the browser's own internals — are **not nested**, and neither is a superset of the others. Measured on one page load of a local fixture recorded three ways at once, plus one re-run against a remote origin ([[#What We Ran]]):
   * the browser's HAR contains a request that a proxy in front of it **never saw**, because Content-Security-Policy stopped the browser from sending it;
-  * the proxy's HAR contains the origin server's IP address, which the browser's HAR **cannot** contain — behind a proxy, every ''serverIPAddress'' in the browser's file is the proxy's;
+  * the proxy's HAR contains the origin server's IP address, which the browser's HAR **cannot** contain — behind a proxy, every ''serverIPAddress'' in the browser's file is the proxy's (this one needs the remote re-run: on a loopback fixture both sides read ''127.0.0.1'' and the difference is invisible);
   * **neither** contains the JavaScript call stack that made the request, which the Chrome DevTools Protocol hands you for free — 0 of 12 HAR entries carried an initiator, against 11 of 11 CDP events, 5 of them with a JS stack frame;
   * the proxy's HAR has **no page boundaries at all** — 0 ''pageref'' fields and an empty ''log.pages'' — so you cannot tell from it which page load a request belongs to.
@@ Line 30: / Line 30: @@
 ^ TLS/QUIC handshake detail, timing at packet granularity | ✓ | partial | ✗ |
 ^ Changes what the site sees | no | **yes** — new TLS stack, new certificate | mostly no |
-^ Volume per 1,000 sites | tens of GB | GB | hundreds of MB without bodies |
+^ Volume per 1,000 sites((Order of magnitude, not a measurement: our fixture recorded single page loads, and no paper in the corpus reports capture volume in a comparable form. Treat it as a planning estimate and measure your own on a pilot.)) | tens of GB | GB | hundreds of MB without bodies |
 Three consequences that decide most projects:
@@ Line 42: / Line 42: @@
 HAR — HTTP Archive format — is a JSON log of HTTP transactions. It is the //lingua franca// of the browser layer: every major browser exports it from DevTools, and every automation library can write one.
-**Its specification is abandoned, and this matters more than it sounds.** HAR 1.2 was written in 2011–2012 by Jan Odvarko and taken up by the W3C Web Performance Working Group, which never published it. The W3C copy now opens with a banner reading //"\*DO NOT USE\* This document was never published by the W3C Web Performance Working Group and has been abandoned."//((Fetched from ''https://w3c.github.io/web-performance/specs/HAR/Overview.html'' on 14 August 2026 by ''scripts/external_checks_traffic_files.sh''.)) Six papers in our corpus cite one of the two W3C URLs as //the// HAR specification — and the older ''dvcs.w3.org'' copy several of them point at does **not** carry the banner, so following the citation does not tell you the document was withdrawn. There has been no successor: a community repository carries a "proposed" 1.3 that has been dormant for years. In practice the specification is whatever the writers agree on, which is why the differences below exist at all. Cite Odvarko's original at ''softwareishard.com/blog/har-12-spec/'' if you must cite something, and //name your writer//.
+**Its specification is abandoned, and this matters more than it sounds.** HAR 1.2 was written in 2011–2012 by Jan Odvarko and taken up by the W3C Web Performance Working Group, which never published it. The W3C copy now opens with a banner reading //"\*DO NOT USE\* This document was never published by the W3C Web Performance Working Group and has been abandoned."//((Fetched from ''https://w3c.github.io/web-performance/specs/HAR/Overview.html'' on 14 August 2026 by ''scripts/external_checks_traffic_files.sh''.)) Six papers in our corpus cite one of the two W3C URLs as //the// HAR specification — and the older ''dvcs.w3.org'' copy that two of them point at does **not** carry the banner, so following the citation does not tell you the document was withdrawn. There has been no successor: a community repository carries a "proposed" 1.3 that has been dormant for years. In practice the specification is whatever the writers agree on, which is why the differences below exist at all. Cite Odvarko's original at ''softwareishard.com/blog/har-12-spec/'' if you must cite something, and //name your writer//.
 <WRAP tip>
-**Two different things are called "HTTP Archive"**, and conflating them is the most common error in this literature. **HAR** is a file format. **The HTTP Archive** (''httparchive.org'') is a monthly crawl of millions of sites, queryable in BigQuery, and it is a //dataset//, not a file you made. In our corpus, 14 papers analyse the dataset and 32 use the format, and only one paper does both ({[trevisan2019_cookielaw]}). See [[#Use in Publications]].
+**Two different things are called "HTTP Archive"**, and conflating them is the most common error in this literature. **HAR** is a file format. **The HTTP Archive** (''httparchive.org'') is a monthly crawl of millions of sites, queryable in BigQuery, and it is a //dataset//, not a file you made. In our corpus, 32 papers use the format and 14 further papers analyse only the dataset. One paper does both — {[trevisan2019_cookielaw]} dumps its own HAR files //and// reanalyses the httparchive.org corpus — and our classification is single-label, so it is counted under //format// and not in the 14. See [[#Use in Publications]].
 </WRAP>
@@ Line 87: / Line 87: @@
   * **No initiator, no call stack.** This is the big one. HAR 1.2 has no field for //who asked for this resource//. CDP's ''Network.requestWillBeSent'' carries ''initiator'', including a JavaScript stack. If your question is "which script caused this third-party request" — the question behind most of [[Privacy:Requests]] and all of [[Privacy:JavaScript]] — a HAR cannot answer it and you need the browser's event stream, or a crawler that records it for you ([[Programming:Crawler:OpenWPM]], [[Programming:Crawler:PageGraph]]).
-  * **Bodies are opt-in, and expensive.** With ''content: 'omit''' the file was 20,566 B; with ''content: 'embed''' it was 226,719 B — **11× larger**, for one 200 KiB response. Bodies are base64-encoded when they are not text, so budget roughly the raw byte count plus a third on top of it. At crawl scale this is the difference between a dataset you can release and one you cannot.
+  * **Bodies are opt-in, and expensive.** With ''content: 'omit''' the file was 20,566 B; with ''content: 'embed''' it was 226,719 B — **11× larger**, for one 200 KiB response. A text body embeds roughly 1:1 — the run above measured 100.2% of the raw body bytes — while a binary body is base64-encoded and costs about a third more. Budget from whichever your crawl is mostly made of. At crawl scale this is the difference between a dataset you can release and one you cannot.
   * **Failure has no vocabulary.** Both requests that never completed appear with ''status: -1''. //Why// they failed is not in the standard: Playwright adds a non-standard ''response._failureText'' (it read ''"csp"'' for the blocked image, and nothing at all for the request left in flight). CDP told us ''blockedReason: "csp"'' for one and, for the other, nothing — it had emitted no terminal event by the time the context closed.
   * **The two recordings disagree about which requests failed.** On the same load, the HAR marked the never-answered ''/slow'' request failed and recorded the 204 beacon as complete; CDP emitted ''loadingFailed net::ERR_ABORTED'' for the //beacon// and no terminal event at all for ''/slow''. Neither is a superset. **In-flight requests at the moment you stop recording are recorded inconsistently, so drain the page before you close the context and report how long you waited.**
@@ Line 105: / Line 105: @@
 | WebPageTest | records a HAR alongside its own waterfall {[chaqfeh2020_jscleaner]} | current; run by Catchpoint since 2020 |
 | HARExportTrigger | a Firefox extension that triggers export from inside the page | **deprecated and archived**, description "DEPRECATED"; last release 0.6.1, May 2018. It is what {[borgolte2020_performance]} automated Firefox with; for new work use the pref above instead |
-| BrowserMob Proxy | a Java proxy that writes HAR | effectively dead: the last GitHub //Release// is 2.1.4 (December 2016); a later ''2.1.5'' tag carries no release notes, and there have been no substantive commits since |
+| BrowserMob Proxy | a Java proxy that writes HAR | effectively dead: the last GitHub //Release// is 2.1.4 (December 2016); a later ''2.1.5'' tag carries no release notes, and the repository was last pushed in May 2024 |
 **Firefox's defaults are not Chrome's, and the prefs say so.** Read straight out of ''browser/app/profile/firefox.js'' in current Firefox: ''includeResponseBodies'' defaults to **''true''**, so a Firefox HAR carries bodies unless you turn them off; ''multiple-pages'' defaults to **''false''**, so an auto-exported HAR covers **one** page; and ''pageLoadedTimeout'' is **1500** ms, which is how long the exporter waits after load before writing — anything still in flight after that is what §//What a HAR Loses// is about. Chrome's sanitized-by-default export strips headers, not bodies. Two files both labelled HAR 1.2, three different defaults.
+==== When the Browser Layer Is Not Enough: NetLog ====
+A HAR stops at HTTP. Chrome's **NetLog** goes below it: DNS resolution and its stalls, socket-pool waits, TCP connect attempts, proxy resolution, disk-cache reads and writes, QUIC session events, and the errors that a HAR can only render as ''status: -1''. You capture it from ''chrome://net-export'' or with ''--log-net-log=FILE'', and read it in the standalone ''netlog-viewer''. It appears in 3 papers in our corpus.
+Reach for it when your question is "//why// was this slow or broken" rather than "what was requested" — a QUIC negotiation that silently fell back, a DNS lookup that stalled, a connection the pool refused to reuse. It is Chromium-only, it is verbose, and its schema is an implementation detail rather than a standard, so it is a debugging instrument rather than a dataset format. But it is the one browser-native answer to a below-HTTP question, and the alternative is leaving the browser layer entirely for a packet capture.
 ===== Intercepting Proxies =====
@@ Line 113: / Line 119: @@
 An intercepting proxy terminates TLS in the middle: it presents your own CA's certificate to the client, opens its own connection to the origin, and sees plaintext in both directions. **mitmproxy** is the field's default — 98 papers in our corpus, against 18 for Burp Suite and 15 for Fiddler ([[#Use in Publications]]) — and ''mitmdump'' is its non-interactive form, which is what a crawl wants. It is actively maintained (12.2.3, May 2026); of the alternatives named in the corpus, Fiddler Classic is no longer in active development and Charles is commercial.
-Its two research-relevant outputs are its own ''.flows'' stream (''-w file'') and a HAR (''--set hardump=file''). We ran the same fixture load through ''mitmdump'' and recorded the browser's HAR at the same time (excerpt of the run's output — the rest is on [[provenance:programming:traffic_files]]):
+Its two research-relevant outputs are its own ''.flows'' stream (''-w file'') and a HAR (''--set hardump=file''). We ran the same fixture load through ''mitmdump'' and recorded the browser's HAR at the same time — with **mitmproxy 11.0.2**, two major versions behind the current release, which by this page's own argument is exactly the kind of thing to state (excerpt of the run's output — the rest is on [[provenance:programming:traffic_files]]):
 <code>
@@ Line 193: / Line 199: @@
 ===== What to Report =====
-A reviewer cannot check a traffic-file measurement without these, and in our corpus most papers give none of them:
+A reviewer cannot check a traffic-file measurement without these. We can only measure how often the first one is given — 12.7% of capture-tool uses state a version — so treat the rest as a checklist rather than as a measured gap:
-  - **Which layer** — browser, proxy, or packets — and **which writer**, by name and version. "We collected HAR files" identifies nothing: only **12.7%** of capture-tool mentions in the corpus state a version at all, and for ''tcpdump'' it is **3.4%**.
+  - **Which layer** — browser, proxy, or packets — and **which writer**, by name and version. "We collected HAR files" identifies nothing: only **12.7%** of capture-tool uses in the corpus state a version at all, and for ''tcpdump'' it is **3.4%**.
   - **Whether bodies were recorded**, because it changes what your numbers can mean and whether anyone can re-derive them.
   - **How a request was attributed to a page**, especially from a proxy capture where the file carries no page boundaries.
@@ Line 214: / Line 220: @@
 The corpus records capture tooling in two categories. Taking only tuples marked //used// or //produced//: **509 papers** name a ''traffic-capture'' tool and **206** name a ''proxy-interception'' tool, overlapping on 36, for a union of **679**. Of those 679, **217** measured the web platform and **178** ran an automated web crawl.
-That raw union is not the page's population. On a broad security corpus the ''traffic-capture'' category is dominated by instruments that produce no web traffic file at all — software-defined radios, Bluetooth and 802.15.4 sniffers, oscilloscopes and logic analysers, screen recorders, micro-architectural side-channel probes, load generators, and commercial VPN services that are vantage points rather than instruments. Folding the 457 distinct tool names into families and excluding those explicitly leaves **514 papers** naming an on-topic instrument; **162 papers** name only off-topic ones. The fold and its residue are on [[provenance:programming:traffic_files]].
+That raw union is not the page's population. On a broad security corpus the ''traffic-capture'' category is dominated by instruments that produce no web traffic file at all — software-defined radios, Bluetooth and 802.15.4 sniffers, oscilloscopes and logic analysers, screen recorders, micro-architectural side-channel probes, load generators, and commercial VPN services that are vantage points rather than instruments. Folding the 457 distinct tool names into families and excluding those explicitly leaves **514 papers** naming an on-topic instrument and **162** naming only off-topic ones. The remaining **3** name only tools the fold could not identify at all — "a browser add-on we designed", "custom addon script", "custom tracing plugin" — which is the honest verdict when a paper does not say what it used. The fold and its residue are on [[provenance:programming:traffic_files]].
 ^ Family ^ Layer ^ Papers ^ Share of 679 ^ Distinct spellings ^
@@ Line 249: / Line 255: @@
 ==== HAR Is Invisible to a Tool Query ====
-If you ask the extraction which papers used HAR, you get almost nothing: HAR appears as a named tool in **10** papers. So we swept the full text of all 5,855 papers with stored text for ''/HAR|HTTP Archive|httparchive/'' and hand-classified all **95** hits. The sweep cannot be automated, because both names are homographs.
+If you ask the extraction which papers used HAR, you get almost nothing: HAR appears as a named tool in **10** papers. So we swept the full text of all 5,855 papers with stored text for HAR, ''.har'', "HTTP Archive" and "httparchive" — the exact regex, which is word-boundary anchored so that //SHARE// and //CHART// do not match, is on [[provenance:programming:traffic_files]] — and hand-classified all **95** hits. The sweep cannot be automated, because both names are homographs.
 ^ Verdict ^ Papers ^ Share of the 95 ^
@@ Line 292: / Line 298: @@
 ===== What We Ran =====
-The two comparisons on this page are reproducible. ''fixture_traffic.mjs'' serves a local page containing a redirect chain, script-initiated requests, a POST body, a 200 KiB response, a request that is never answered, a WebSocket, and an image blocked by the page's own CSP. ''har_vs_cdp.mjs'' records one load of it as a HAR and as a CDP event stream simultaneously; ''mitm_vs_browser.mjs'' records one load through ''mitmdump'' while the browser writes its own HAR.
+The two comparisons on this page are reproducible. ''fixture_traffic.mjs'' serves a local page containing a redirect chain, script-initiated requests, a POST body, a 200 KiB response, a request that is never answered, a WebSocket, and an image blocked by the page's own CSP. ''har_vs_cdp.mjs'' records one load of it as a HAR and as a CDP event stream simultaneously; ''mitm_vs_browser.mjs'' records one load through ''mitmdump'' while the browser writes its own HAR. ''mitm_vs_browser.mjs'' was run **twice**: once against the fixture, and once against ''https://example.com/'', because the ''serverIPAddress'' claim cannot be shown on a loopback target where both recorders read ''127.0.0.1''.
 <file javascript har_vs_cdp_core.mjs>
@@ Line 312: / Line 318: @@
 await cdp.send('Network.enable');
-await page.goto('http://127.0.0.1:8098/', { waitUntil: 'load' }).catch(() => {});
+// Log the failure -- do NOT swallow it. In a crawl, a silently failed goto()
+// becomes a site recorded as having made no requests, which is indistinguishable
+// from a site that made none.
+await page.goto('http://127.0.0.1:8098/', { waitUntil: 'load' }).catch((e) => console.error('goto failed:', e.message));
 await page.waitForTimeout(2500);   // drain in-flight requests before closing
 await context.close();             // the HAR is only written on context.close()
@@ Line 341: / Line 350: @@
   * **The HAR population is a full-text sweep, not a schema query.** It finds papers that write the string; a paper that captured HAR files and never used the word is invisible. The 32 is therefore a floor.
   * **The capture-tool fold is a judgement call.** Deciding that a software-defined radio is off-topic for a page about web traffic files and that ''selenium-wire'' is on-topic is a decision, not a measurement; the family list and the 162 excluded papers are published so you can disagree with it.
-  * **The fixture comparisons are one page load of a synthetic page in one browser.** They demonstrate mechanisms — an initiator field that is absent, a CSP-blocked request a proxy cannot see — not rates. Nothing on this page claims a HAR loses //x%// of anything on real sites.
+  * **The fixture comparisons are one page load of a synthetic page in one browser.** They demonstrate mechanisms — an initiator field that is absent, a CSP-blocked request a proxy cannot see — not rates. **The fixture claims no percentages at all**; the only per-cent figures on this page about real sites are Hantke et al.'s, over 8,544 origins, and they are theirs rather than ours.
 ===== Related Pages =====