| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| programming:traffic_files [2026/08/14 03:41] – Apply the generic review pass: fix an overstated lead box, an internal contradiction in the dataset/format counts, an unmeasured 'most papers give none of them', a limitations bullet that contradicted the Replay section, and an error-swallowing line in th karel.kubicek.claude | programming:traffic_files [2026/09/17 07:22] (current) – Markup rendering sweep: wrap escapes, CLI flag typography, and heading quotes. Authored by Claude. karel.kubicek.claude |
|---|
| | Puppeteer | no built-in writer — use a library over CDP | ''chrome-har-capturer'' (npm 0.14.4, 2026-01-27) is maintained and is what the corpus uses {[muthuraj2024_replication,zhang2024_quic]} | | | Puppeteer | no built-in writer — use a library over CDP | ''chrome-har-capturer'' (npm 0.14.4, 2026-01-27) is maintained and is what the corpus uses {[muthuraj2024_replication,zhang2024_quic]} | |
| | Selenium | none built in; //classic// WebDriver has no network commands at all | this is why [[Programming:Crawler|Selenium crawls]] historically end up with a proxy bolted on, and ''selenium-wire'', the usual bridge, was **archived in January 2024**. WebDriver BiDi has since given Selenium native network interception — with [[Programming:Crawler|its own caveats]] — but still no HAR writer | | | Selenium | none built in; //classic// WebDriver has no network commands at all | this is why [[Programming:Crawler|Selenium crawls]] historically end up with a proxy bolted on, and ''selenium-wire'', the usual bridge, was **archived in January 2024**. WebDriver BiDi has since given Selenium native network interception — with [[Programming:Crawler|its own caveats]] — but still no HAR writer | |
| | mitmproxy | ''mitmdump --set hardump=out.har'' | current (12.2.3, 2026-05-12); proxy-layer HAR — bodies by default, no page boundaries | | | mitmproxy | ''mitmdump %%--set%% hardump=out.har'' | current (12.2.3, 2026-05-12); proxy-layer HAR — bodies by default, no page boundaries | |
| | WebPageTest | records a HAR alongside its own waterfall {[chaqfeh2020_jscleaner]} | current; run by Catchpoint since 2020 | | | WebPageTest | records a HAR alongside its own waterfall {[chaqfeh2020_jscleaner]} | current; run by Catchpoint since 2020 | |
| | HARExportTrigger | a Firefox extension that triggers export from inside the page | **deprecated and archived**, description "DEPRECATED"; last release 0.6.1, May 2018. It is what {[borgolte2020_performance]} automated Firefox with; for new work use the pref above instead | | | HARExportTrigger | a Firefox extension that triggers export from inside the page | **deprecated and archived**, description "DEPRECATED"; last release 0.6.1, May 2018. It is what {[borgolte2020_performance]} automated Firefox with; for new work use the pref above instead | |
| ==== When the Browser Layer Is Not Enough: NetLog ==== | ==== When the Browser Layer Is Not Enough: NetLog ==== |
| |
| A HAR stops at HTTP. Chrome's **NetLog** goes below it: DNS resolution and its stalls, socket-pool waits, TCP connect attempts, proxy resolution, disk-cache reads and writes, QUIC session events, and the errors that a HAR can only render as ''status: -1''. You capture it from ''chrome://net-export'' or with ''--log-net-log=FILE'', and read it in the standalone ''netlog-viewer''. It appears in 3 papers in our corpus. | A HAR stops at HTTP. Chrome's **NetLog** goes below it: DNS resolution and its stalls, socket-pool waits, TCP connect attempts, proxy resolution, disk-cache reads and writes, QUIC session events, and the errors that a HAR can only render as ''status: -1''. You capture it from ''chrome://net-export'' or with ''%%--log-net-log%%=FILE'', and read it in the standalone ''netlog-viewer''. It appears in 3 papers in our corpus. |
| |
| Reach for it when your question is "//why// was this slow or broken" rather than "what was requested" — a QUIC negotiation that silently fell back, a DNS lookup that stalled, a connection the pool refused to reuse. It is Chromium-only, it is verbose, and its schema is an implementation detail rather than a standard, so it is a debugging instrument rather than a dataset format. But it is the one browser-native answer to a below-HTTP question, and the alternative is leaving the browser layer entirely for a packet capture. | Reach for it when your question is "//why// was this slow or broken" rather than "what was requested" — a QUIC negotiation that silently fell back, a DNS lookup that stalled, a connection the pool refused to reuse. It is Chromium-only, it is verbose, and its schema is an implementation detail rather than a standard, so it is a debugging instrument rather than a dataset format. But it is the one browser-native answer to a below-HTTP question, and the alternative is leaving the browser layer entirely for a packet capture. |
| An intercepting proxy terminates TLS in the middle: it presents your own CA's certificate to the client, opens its own connection to the origin, and sees plaintext in both directions. **mitmproxy** is the field's default — 98 papers in our corpus, against 18 for Burp Suite and 15 for Fiddler ([[#Use in Publications]]) — and ''mitmdump'' is its non-interactive form, which is what a crawl wants. It is actively maintained (12.2.3, May 2026); of the alternatives named in the corpus, Fiddler Classic is no longer in active development and Charles is commercial. | An intercepting proxy terminates TLS in the middle: it presents your own CA's certificate to the client, opens its own connection to the origin, and sees plaintext in both directions. **mitmproxy** is the field's default — 98 papers in our corpus, against 18 for Burp Suite and 15 for Fiddler ([[#Use in Publications]]) — and ''mitmdump'' is its non-interactive form, which is what a crawl wants. It is actively maintained (12.2.3, May 2026); of the alternatives named in the corpus, Fiddler Classic is no longer in active development and Charles is commercial. |
| |
| Its two research-relevant outputs are its own ''.flows'' stream (''-w file'') and a HAR (''--set hardump=file''). We ran the same fixture load through ''mitmdump'' and recorded the browser's HAR at the same time — with **mitmproxy 11.0.2**, two major versions behind the current release, which by this page's own argument is exactly the kind of thing to state (excerpt of the run's output — the rest is on [[provenance:programming:traffic_files]]): | Its two research-relevant outputs are its own ''.flows'' stream (''-w file'') and a HAR (''%%--set%% hardump=file''). We ran the same fixture load through ''mitmdump'' and recorded the browser's HAR at the same time — with **mitmproxy 11.0.2**, two major versions behind the current release, which by this page's own argument is exactly the kind of thing to state (excerpt of the run's output — the rest is on [[provenance:programming:traffic_files]]): |
| |
| <code> | <code> |
| |
| * **Volume.** Full packet capture of a crawl is tens of GB per thousand sites. Snap-length (''tcpdump -s'') truncates each packet; a BPF filter drops the traffic you do not want. Decide both //before// the crawl, because you cannot recover what you did not write. | * **Volume.** Full packet capture of a crawl is tens of GB per thousand sites. Snap-length (''tcpdump -s'') truncates each packet; a BPF filter drops the traffic you do not want. Decide both //before// the crawl, because you cannot recover what you did not write. |
| * **Encryption.** Everything is TLS. Either capture the session keys alongside — Chromium's ''--ssl-key-log-file'' flag and the ''SSLKEYLOGFILE'' environment variable both write the NSS key-log format Wireshark reads; one ''https://example.com/'' load through Playwright's Chromium build produced a five-line key log for us — or restrict yourself to what survives encryption: SNI, certificate details on older TLS versions, packet sizes, timing, and the addresses. That is exactly the input set of the website-fingerprinting literature. | * **Encryption.** Everything is TLS. Either capture the session keys alongside — Chromium's ''%%--ssl-key-log-file%%'' flag and the ''SSLKEYLOGFILE'' environment variable both write the NSS key-log format Wireshark reads; one ''https://example.com/'' load through Playwright's Chromium build produced a five-line key log for us — or restrict yourself to what survives encryption: SNI, certificate details on older TLS versions, packet sizes, timing, and the addresses. That is exactly the input set of the website-fingerprinting literature. |
| * **You are capturing the host, not the browser.** OS updates, telemetry, and every other process share the interface. Filter, and say what you filtered. | * **You are capturing the host, not the browser.** OS updates, telemetry, and every other process share the interface. Filter, and say what you filtered. |
| * **''.pcap'' vs ''.pcapng''.** ''.pcapng'' is the modern container (multiple interfaces, comments, name-resolution blocks, decryption secrets embedded in the file). Modern ''tcpdump'' and Wireshark write and read both; some older analysis code reads only classic ''.pcap''. Convert with ''editcap'' rather than re-capturing. | * **''.pcap'' vs ''.pcapng''.** ''.pcapng'' is the modern container (multiple interfaces, comments, name-resolution blocks, decryption secrets embedded in the file). Modern ''tcpdump'' and Wireshark write and read both; some older analysis code reads only classic ''.pcap''. Convert with ''editcap'' rather than re-capturing. |
| - **What was stripped before release**, field by field. | - **What was stripped before release**, field by field. |
| |
| <wrap todo> | <WRAP todo> |
| Wanted: a measurement of how much a browser HAR and a proxy HAR of the //same// crawl disagree at scale, on real sites rather than a fixture. Our fixture shows the mechanisms; nobody in this corpus has quantified the gap over a top-list crawl. If you run it, please add it here. | Wanted: a measurement of how much a browser HAR and a proxy HAR of the //same// crawl disagree at scale, on real sites rather than a fixture. Our fixture shows the mechanisms; nobody in this corpus has quantified the gap over a top-list crawl. If you run it, please add it here. |
| </wrap> | </WRAP> |
| |
| ===== Use in Publications ===== | ===== Use in Publications ===== |
