A crawl is an event. A traffic file is the artefact it leaves behind: a re-readable record of what went over the network, written once and analysed many times. Writing one is what lets you answer a question you had not thought of on the day of the crawl, hand the evidence to a reviewer, and let somebody else recompute your number instead of trusting it.

This page is about choosing where to tap, and what the resulting file can and cannot answer. It is not about how HTTP works — for the wire format go to the RFCs, and for the browser API go to MDN. It pairs with the crawler comparison (what drives the browser), Archives (reading somebody else's recording), Requests (what you do with the requests once you have them) and IP classification (one field in the file that most people miss).

Three consequences that decide most projects:

• A proxy changes the measurement. Your crawler now presents mitmproxy's TLS fingerprint and a certificate signed by a CA you generated. Sites that pin certificates break, and sites that fingerprint the TLS handshake see a different client. If your research question is “what do sites do to a normal browser”, an intercepting proxy is the one instrument that can invalidate the answer. Fingerprinting covers what is fingerprintable.
• A packet capture cannot see inside TLS unless you also capture the session keys. Chrome and Firefox will write them to the path in the SSLKEYLOGFILE environment variable, and Wireshark reads that file — but a keylog file is a decryption key for every session in the capture, so it inherits every rule in Before You Publish The File.
• The browser layer is the only one that sees the browser's own decisions: a request blocked by CSP, by an extension, by tracking protection, or served from cache leaves no packets and reaches no proxy. If you are measuring blocking, the browser layer is not a convenience, it is the requirement.

HAR — HTTP Archive format — is a JSON log of HTTP transactions. It is the lingua franca of the browser layer: every major browser exports it from DevTools, and every automation library can write one.

Its specification is abandoned, and this matters more than it sounds. HAR 1.2 was written in 2011–2012 by Jan Odvarko and taken up by the W3C Web Performance Working Group, which never published it. The W3C copy now opens with a banner reading “\*DO NOT USE\* This document was never published by the W3C Web Performance Working Group and has been abandoned.”¹⁾ Six papers in our corpus cite one of the two W3C URLs as the HAR specification — and the older dvcs.w3.org copy several of them point at does not carry the banner, so following the citation does not tell you the document was withdrawn. There has been no successor: a community repository carries a “proposed” 1.3 that has been dormant for years. In practice the specification is whatever the writers agree on, which is why the differences below exist at all. Cite Odvarko's original at softwareishard.com/blog/har-12-spec/ if you must cite something, and name your writer.

Only a handful of the schema's fields carry a measurement. In rough order of how often you will touch them:

We recorded one load of a local fixture three ways at once — Playwright's HAR with bodies omitted, the same with bodies embedded, and a raw CDP session on the same page — and diffed them field by field. The fixture deliberately contains a redirect chain, a script-initiated fetch, a POST with a body, a 200 KiB response, a request that is never answered, a WebSocket, and an image the page's own CSP forbids. This is the comparison table from the run's output; the full output, including the per-URL outcome table and the extension-field list, is on traffic_files.

                                                         HAR (omit)  HAR (embed)  CDP
entries / requests                                       12          12           11
  …of which redirect hops                                2           2            2
  …of which never completed (status -1 / loadingFailed)  2           2            2
entries carrying response body text                      0           6            on demand (Network.getResponseBody)
entries with an initiator / call stack                   0           0            11
  …with a JS stack frame                                 0           0            5
entries with a resource type                             12          12           11
entries with a frame id                                  12          12           11
entries with serverIPAddress                             8           8            7
WebSocket connections                                    1           1            1
WebSocket frames                                         0           4            4

Reading that table, in order of how much it costs you:

• No initiator, no call stack. This is the big one. HAR 1.2 has no field for who asked for this resource. CDP's Network.requestWillBeSent carries initiator, including a JavaScript stack. If your question is “which script caused this third-party request” — the question behind most of Requests and all of JavaScript — a HAR cannot answer it and you need the browser's event stream, or a crawler that records it for you (OpenWPM, PageGraph).
• Bodies are opt-in, and expensive. With content: 'omit' the file was 20,566 B; with content: 'embed' it was 226,719 B — 11× larger, for one 200 KiB response. Bodies are base64-encoded when they are not text, so budget roughly the raw byte count plus a third on top of it. At crawl scale this is the difference between a dataset you can release and one you cannot.
• Failure has no vocabulary. Both requests that never completed appear with status: -1. Why they failed is not in the standard: Playwright adds a non-standard response._failureText (it read “csp” for the blocked image, and nothing at all for the request left in flight). CDP told us blockedReason: “csp” for one and, for the other, nothing — it had emitted no terminal event by the time the context closed.
• The two recordings disagree about which requests failed. On the same load, the HAR marked the never-answered /slow request failed and recorded the 204 beacon as complete; CDP emitted loadingFailed net::ERR_ABORTED for the beacon and no terminal event at all for /slow. Neither is a superset. In-flight requests at the moment you stop recording are recorded inconsistently, so drain the page before you close the context and report how long you waited.
• A HAR is a transaction log, not a cookie jar. The main document's response.cookies correctly contained the HttpOnly cookie — that part of the folklore is wrong, the flag hides a cookie from document.cookie, not from the recorder. But at the end of the load the browser jar held 4 cookies and one of them, set by document.cookie in a script, appears in no HAR entry, because it never crossed the network. Counting cookies from a HAR counts cookies observed in HTTP headers, which is a different quantity from cookies in the jar. Say which one you mean. Cookies is the page for the rest of that problem.
• The WebSocket rows are about the writer, not the format. HAR 1.2 has no representation for a WebSocket at all. Playwright records the connection as an entry and the frames under a non-standard _webSocketMessages, and it only started doing so in version 1.61 (June 2026) — so a HAR written by an older Playwright, or by a different tool, may show nothing. Check rather than assume.
• Everything interesting is a vendor extension. The spec reserves _-prefixed names for custom fields, and the useful ones all live there. Playwright 1.62.1 emitted exactly _resourceType, _securityDetails, _serverPort, _webSocketMessages, response._failureText and response._transferSize. Chrome DevTools emits a different set. So an analysis script written against one writer's HAR quietly produces empty columns against another's.

Firefox's defaults are not Chrome's, and the prefs say so. Read straight out of browser/app/profile/firefox.js in current Firefox: includeResponseBodies defaults to true, so a Firefox HAR carries bodies unless you turn them off; multiple-pages defaults to false, so an auto-exported HAR covers one page; and pageLoadedTimeout is 1500 ms, which is how long the exporter waits after load before writing — anything still in flight after that is what §What a HAR Loses is about. Chrome's sanitized-by-default export strips headers, not bodies. Two files both labelled HAR 1.2, three different defaults.

An intercepting proxy terminates TLS in the middle: it presents your own CA's certificate to the client, opens its own connection to the origin, and sees plaintext in both directions. mitmproxy is the field's default — 98 papers in our corpus, against 18 for Burp Suite and 15 for Fiddler (Use in Publications) — and mitmdump is its non-interactive form, which is what a crawl wants. It is actively maintained (12.2.3, May 2026); of the alternatives named in the corpus, Fiddler Classic is no longer in active development and Charles is commercial.

Its two research-relevant outputs are its own .flows stream (-w file) and a HAR (–set hardump=file). We ran the same fixture load through mitmdump and recorded the browser's HAR at the same time (excerpt of the run's output — the rest is on traffic_files):

                                  browser HAR   mitmproxy HAR
entries                           12            11
entries with response body text   0             10
entries with _resourceType        12            1
entries with a pageref            12            0
log.pages                         1             0
WebSocket entries                 1             0

in the browser HAR only:   http://127.0.0.99:1/blocked-by-csp.gif, ws://127.0.0.1:8098/ws
in the mitmproxy HAR only: /ws

file sizes
  browser HAR (content omitted)     20453 B
  mitmproxy HAR (bodies included)   261692 B
  mitmproxy .flows (its own format) 232420 B
  ratio, mitmproxy HAR / browser HAR 12.8x

• The proxy never saw the CSP-blocked request. That is not a bug; it is the definition of the layer. Any measurement of blocking — by CSP, by an extension, by the browser's own tracking protection — is invisible from in front of the browser.
• No page boundaries. The mitmproxy HAR has zero pageref values and an empty log.pages. In a crawl of n sites through one proxy you must reconstruct “which site was this” yourself, normally by having the crawler stamp a header or by writing one flow file per site. Doing it by timestamp is a bug waiting to happen: page loads overlap.
• The WebSocket is in both files under different URLs — ws://…/ws in the browser's, http://…/ws (the upgrade) in the proxy's. Joining the two files on URL silently loses it.
• Bodies by default. 10 of 11 entries carried response text, and the file is 12.8× the browser's body-less HAR. This is what you want for content analysis and what you do not want for a 100k-site crawl.
• Every serverIPAddress the browser records becomes the proxy's. Re-running the same script against https://example.com/ on 14 August 2026, the browser HAR recorded 127.0.0.1 — the proxy — and the mitmproxy HAR recorded 104.20.23.154, the origin. (That second address is not a constant: example.com sits on Cloudflare's anycast network and the edge address rotates. The point is which side each recorder sees, not the octets.) If you need origin addresses and you are proxying, take them from the proxy side. IP classification is where that matters.

What the proxy buys you that the browser layer does not: traffic from a mobile app or an IoT device that has no DevTools at all — which is why much of the mobile-privacy literature is built on it [7Kollnig, Konrad; Shuba, Anastasia; Binns, Reuben; Van Kleek, Max; Shadbolt, Nigel (2022): "Are iPhones Really Better for Privacy? A Comparative Study of iOS and Android Apps", Proceedings on Privacy Enhancing Technologies 2022(2):6-24. (DOI)] — and full request and response bodies without the browser's cooperation. It is not the only way onto a phone: [8Figueira, Olivia; Trimananda, Rahmadi; Markopoulou, Athina; Jordan, Scott (2024): "DiffAudit: Auditing Privacy Practices of Online Services for Children and Adolescents", in: Proceedings of the ACM Internet Measurement Conference. (DOI)] goes to the packet layer there instead, with an on-device capture app, and keeps the browser layer for the web half of the same study.

What it costs. Certificate pinning breaks. TLS fingerprinting sees mitmproxy, not Chrome. QUIC and HTTP/3 interception is newer and less complete than HTTP/1.1 and HTTP/2 — mitmproxy enables it by default but documents the support as limited, and browsers do not accept a custom CA over QUIC the way they do over TLS — so a proxied crawl can silently downgrade the protocol under measurement. That is fatal when the protocol is the measurement, which is why the QUIC website-fingerprinting work captures at the browser and packet layers instead [9Siby, Sandra; Barman, Ludovic; Wood, Christopher; Fayed, Marwan; Sullivan, Nick; Troncoso, Carmela (2023): "Evaluating Practical QUIC Website Fingerprinting Defenses for the Masses", Proceedings on Privacy Enhancing Technologies 2023(4):79-95. (DOI)]. And you have added a component that can fail: a flow the proxy drops looks exactly like a request the site did not make.

tcpdump (4.99.6) and Wireshark (4.6.8) are, by a wide margin, the most-named capture instruments in the corpus — 119 and 128 papers — and for web measurement they are usually the wrong layer. They are there because the corpus is seven security and networking venues, and most of those papers are not measuring the web at all (Use in Publications separates the two).

Reach for a packet capture when the question is genuinely below HTTP:

• traffic analysis and website fingerprinting, where packet sizes and timing are the signal;
• DNS, QUIC transport behaviour, connection reuse, TLS handshake contents;
• anything where you must not perturb the connection, because a capture is passive and a proxy is not;
• a device you cannot instrument, where the network is the only place you can stand.

Practicalities that bite:

• Volume. Full packet capture of a crawl is tens of GB per thousand sites. Snap-length (tcpdump -s) truncates each packet; a BPF filter drops the traffic you do not want. Decide both before the crawl, because you cannot recover what you did not write.
• Encryption. Everything is TLS. Either capture the session keys alongside — Chromium's –ssl-key-log-file flag and the SSLKEYLOGFILE environment variable both write the NSS key-log format Wireshark reads; one https://example.com/ load through Playwright's Chromium build produced a five-line key log for us — or restrict yourself to what survives encryption: SNI, certificate details on older TLS versions, packet sizes, timing, and the addresses. That is exactly the input set of the website-fingerprinting literature.
• You are capturing the host, not the browser. OS updates, telemetry, and every other process share the interface. Filter, and say what you filtered.
• .pcap vs .pcapng. .pcapng is the modern container (multiple interfaces, comments, name-resolution blocks, decryption secrets embedded in the file). Modern tcpdump and Wireshark write and read both; some older analysis code reads only classic .pcap. Convert with editcap rather than re-capturing.
• Neither format is standardised, which surprises people. As of 14 August 2026 draft-ietf-opsawg-pcap (rev 08) and draft-ietf-opsawg-pcapng (rev 05) are still Internet-Drafts at the IETF; the first is even headed for Historic status rather than Standards Track. In practice the reference implementation is libpcap, and “what Wireshark reads” is the specification. Cite the drafts and the version of libpcap you used.

A traffic file is a recording, and a recording can be played back: serve the stored responses to a real browser so the page executes without touching the network again. That is the reproducibility argument for storing traffic in the first place, and it is where HAR and WARC meet — WARC being the archival format that Archives covers, and the one the Wayback Machine and pywb speak.

Replay is not free of the recording's blind spots, and Hantke et al. measured the cost [10Hantke, Florian; Snyder, Peter; Haddadi, Hamed; Stock, Ben (2025): "Web Execution Bundles: Reproducible, Accurate, and Archivable Web Measurements", in: Proceedings of the USENIX Security Symposium. (Link)]. Replaying archived responses to a browser and comparing the JavaScript API calls against a live baseline, they found an average per-origin difference of 0.7% for their own execution-recording format, against 13.9% for HAR and 13.3% for WARC — “the drawback of replaying content rather than recording and inspecting the executions directly”. Replaying itself fails sometimes: over 8,523 origins they hit 311 HAR and 44 WARC issues replaying responses.

The same paper is also the best available check on whether a HAR captures the requests: across 8,544 origins they counted 778,500 requests in the HAR after filtering out non-page requests, against 776,229 recorded inside the browser — less than 0.3% difference. So a HAR is a faithful record of what left the browser; it is the why and the what happened next that it drops. Their framing is worth copying: “HAR is developed to record all requests that leave the browser, while WebREC captures every activity happening on the page.”

There is currently no maintained general-purpose HAR sanitiser, and this is a claim we tried to disprove rather than assert. Every candidate we could find is archived or dormant as of 14 August 2026: Google's har-sanitizer and Cloudflare's har-sanitizer are both archived read-only, as is the sanitizhar Chrome extension; Beyond Identity's har-sanitize has not been touched since November 2023; harmor was last committed in April 2025; and Edgio's har-tools is orphaned, its publisher having shut down in January 2025. If one of these has revived since, it is a better answer than the paragraph below — check before you write your own. What survives is vendor-specific: Okta's own sanitiser strips all cookies and the signature of ID/access/SAML tokens but explicitly “There is no attempt to remove user information as part of the sanitization process” and leaves JWT claims and SAML attributes intact, and recommends capturing with a test account instead. If you need sanitisation in a crawl pipeline, you are writing it.

Before a traffic file goes into an artefact, a support ticket, or a shared drive:

1. Decide bodies before the crawl, not after. If you do not need response bodies, do not record them: it is the single largest reduction in both size and risk.
2. Strip credential-bearing fields, at minimum request.headers Authorization/Cookie/Proxy-Authorization, response.headers Set-Cookie, request.cookies/response.cookies, and any bearer/session token in request.queryString or request.postData. Chrome DevTools has sanitised its HAR export by default since Chrome 130 (October 2024) — but read what that means: “the network log exported in HAR format will no longer contain Cookie, Set-Cookie, and Authorization headers by default”, and nothing else. Query-string tokens, POST bodies, JWT claims and non-standard auth headers all survive it.
3. Remember what you cannot strip. Response bodies contain whatever the site returned about the account you were logged into. If the crawl was authenticated (Registration), the safe assumption is that the file cannot be released at all.
4. Your own vantage point is in there too. Client IP addresses, internal hostnames, proxy addresses and the crawl machine's clock skew are all personal or identifying data in some jurisdictions — see Ethics.
5. Release the derived table, not the raw capture, unless the raw capture is the contribution. A CSV of (site, request URL, resource type, classification) is reproducible, reviewable and safe; a directory of HARs is none of those things.

A reviewer cannot check a traffic-file measurement without these, and in our corpus most papers give none of them:

1. Which layer — browser, proxy, or packets — and which writer, by name and version. “We collected HAR files” identifies nothing: only 12.7% of capture-tool mentions in the corpus state a version at all, and for tcpdump it is 3.4%.
2. Whether bodies were recorded, because it changes what your numbers can mean and whether anyone can re-derive them.
3. How a request was attributed to a page, especially from a proxy capture where the file carries no page boundaries.
4. What you did with incomplete requests — the status: -1 entries — and how long you waited before closing the context.
5. What was filtered or truncated at capture time: BPF filter, snap length, URL exclusions, robots.txt handling.
6. What was stripped before release, field by field.

Wanted: a measurement of how much a browser HAR and a proxy HAR of the same crawl disagree at scale, on real sites rather than a fixture. Our fixture shows the mechanisms; nobody in this corpus has quantified the gap over a top-list crawl. If you run it, please add it here.

Every figure below comes from the publication corpus of 5,859 papers across CCS, IMC, NDSS, PoPETs, USENIX Security, TheWebConf and IEEE S&P, 2010–2026. Each names its own denominator; the full query log is on traffic_files.

The corpus records capture tooling in two categories. Taking only tuples marked used or produced: 509 papers name a traffic-capture tool and 206 name a proxy-interception tool, overlapping on 36, for a union of 679. Of those 679, 217 measured the web platform and 178 ran an automated web crawl.

That raw union is not the page's population. On a broad security corpus the traffic-capture category is dominated by instruments that produce no web traffic file at all — software-defined radios, Bluetooth and 802.15.4 sniffers, oscilloscopes and logic analysers, screen recorders, micro-architectural side-channel probes, load generators, and commercial VPN services that are vantage points rather than instruments. Folding the 457 distinct tool names into families and excluding those explicitly leaves 514 papers naming an on-topic instrument; 162 papers name only off-topic ones. The fold and its residue are on traffic_files.

The named instruments, folded across spellings and counted by paper:

The layer flips when you restrict to the web. Of the 514 on-topic papers, 61.9% name a packet-capture tool and 37.7% a proxy. Of the 188 whose platform includes web, it is 50.5% proxy against 45.7% packet — the packet layer's dominance is an artefact of the venues, not advice about web measurement.

If you ask the extraction which papers used HAR, you get almost nothing: HAR appears as a named tool in 10 papers. So we swept the full text of all 5,855 papers with stored text for /HAR|HTTP Archive|httparchive/ and hand-classified all 95 hits. The sweep cannot be automated, because both names are homographs.

The 11 homographs are worth naming, because a regex over full text will otherwise inflate any HAR count by a tenth: Human Activity Recognition is the standard expansion in the sensing and ML literature (UCI-HAR, KU-HAR and UTAH-STM-HAR are benchmark datasets, and five papers use them), one XR-biometrics paper defines HAR as Harmonic Amplitude Ratio, and four hits are surnames and identifiers that the corpus's two-column repair split across a line break — S HAR, B HAR-GAVA, D HAR-MDASANI, IS_WORD_C HAR.

Of the 32 papers that use HAR as an instrument, only 9 (28.1%) name it in any tool field, and only 16 (50.0%) have any traffic-capture or proxy tuple at all. A structured tool query would have found at most half of them. That is a fact about how the field reports its instruments, not about the extraction: writing HAR files is treated as too ordinary to name.

Where they are, by year bucket: 2010–2013 1, 2014–2017 3, 2018–2021 12, 2022–2024 9, 2025–2026 7 (2025–2026 is provisional — CCS 2026 and IMC 2026 have not been held). By venue: IMC 9, PoPETs 8, TheWebConf 7, IEEE S&P 4, USENIX Security 3, NDSS 1. No CCS paper in the corpus uses HAR as an instrument, and that is not a small-n artefact: CCS contributes 163 crawling papers, more than IMC's 132, and IMC contributes 9 of the 32.

And it stays a minority practice inside crawling. Of the 1,120 papers that ran an automated web crawl, 156 (13.9%) name an on-topic capture instrument and 26 (2.3%) use HAR; the union is 172 (15.4%). Six papers in seven that crawl the web either never wrote a traffic file or never said so.

Of the 669 tool tuples in on-topic capture families, 85 (12.7%) state a version.

mitmproxy is reported twice as often as Wireshark and seven times as often as tcpdump, which is the right way round — mitmproxy's behaviour changes between major versions in ways that matter, and researchers seem to know it.

For HAR the equivalent question is which writer produced the file, and it can only be answered as an upper bound: a keyword probe over the full text of the 32 instrument papers finds a recognisable writer in 25 (78.1%), but it counts any mention of Selenium or DevTools anywhere in the paper, so the true figure is lower. Seven of the 32 name no recognisable writer anywhere at all.

• [10Hantke, Florian; Snyder, Peter; Haddadi, Hamed; Stock, Ben (2025): "Web Execution Bundles: Reproducible, Accurate, and Archivable Web Measurements", in: Proceedings of the USENIX Security Symposium. (Link)] (USENIX Security 2025) — the reference point for this whole page. Measures HAR and WARC replay against direct execution recording, and is the only paper in the corpus that treats the choice of traffic file as a research question rather than an implementation detail. Read it first.
• [11Englehardt, Steven; Narayanan, Arvind (2016): "Online Tracking: A 1-million-site Measurement and Analysis", in: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 1388–1401. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)] (CCS 2016) — the 1-million-site measurement, and the paper that made “browser plus proxy” a standard architecture: “After examining several Python HTTP proxies, we chose to use Mitmproxy to record all HTTP Request and Response headers.”
• [12Butkiewicz, Michael; Madhyastha, Harsha V.; Sekar, Vyas (2011): "Understanding website complexity: measurements, metrics, and implications", in: Proceedings of the ACM Internet Measurement Conference. (DOI)] (IMC 2011) — the earliest HAR use in the corpus, and still the clearest exposition: it prints an annotated HAR snippet as a figure.
• [13Hounsel, Austin; Borgolte, Kevin; Schmitt, Paul; Holland, Jordan; Feamster, Nick (2020): "Comparing the Effects of DNS, DoT, and DoH on Web Performance", in: Proceedings of the ACM Web Conference. (DOI)] (TheWebConf 2020) — the best worked example of a HAR artefact: they found that the first DNS query in a HAR can show a lookup time of 0 ms even when that is impossible, because a redirect means the first query the browser actually performed is not in the file.
• [14Singanamalla, Sudheesh; Paracha, Muhammad Talha; Ahmad, Suleman; Hoyland, Jonathan; Valenta, Luke; Safronov, Yevgen; Wu, Peter; Galloni, Andrew; Heimerl, Kurtis; Sullivan, Nick; Wood, Christopher A.; Fayed, Marwan (2022): "Respect the ORIGIN!: a best-case evaluation of connection coalescing in the wild", in: Proceedings of the ACM Internet Measurement Conference. (DOI)] (IMC 2022) — HAR timelines used as the primary measurement, reconstructing what connection coalescing would have saved.
• [8Figueira, Olivia; Trimananda, Rahmadi; Markopoulou, Athina; Jordan, Scott (2024): "DiffAudit: Auditing Privacy Practices of Online Services for Children and Adolescents", in: Proceedings of the ACM Internet Measurement Conference. (DOI)] (IMC 2024) — HAR for the web side and pcap for the mobile side of the same study, converted to a common JSON representation; the model for a two-layer design.
• [15Karnam, Sai Keerthana; Dash, Abhisek; Das, Antariksh; Mousavi, Sepehr; Bechtold, Stefan; Gummadi, Krishna P.; Mukherjee, Animesh; Weber, Ingmar; Zannettou, Savvas (2026): "Setting the Course, but Forgetting to Steer: Analyzing Compliance with GDPR's Right of Access to Data by Instagram, TikTok, and Youtube", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)] (IEEE S&P 2026) — HAR logs used as ground truth to score what a platform's GDPR data-download package left out. A different use of the file: not the measurement, the yardstick.
• [16Innocenti, Tommaso; Jannett, Louis; Mainka, Christian; Mladenov, Vladislav; Kirda, Engin (2025): ""Only as Strong as the Weakest Link": On the Security of Brokered Single Sign-On on the Web", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)] (IEEE S&P 2025) — reuses somebody else's published HAR dataset rather than crawling, which is the reproducibility payoff of storing traffic in the first place.

The two comparisons on this page are reproducible. fixture_traffic.mjs serves a local page containing a redirect chain, script-initiated requests, a POST body, a 200 KiB response, a request that is never answered, a WebSocket, and an image blocked by the page's own CSP. har_vs_cdp.mjs records one load of it as a HAR and as a CDP event stream simultaneously; mitm_vs_browser.mjs records one load through mitmdump while the browser writes its own HAR.

har_vs_cdp_core.mjs

// The core of sandbox/har_vs_cdp.mjs: one page load, two recordings, one diff.
// Full script and fixture: see provenance:programming:traffic_files.
import fs from 'node:fs';
import { chromium } from 'playwright';
 
const harPath = '/tmp/one-load.har';
const browser = await chromium.launch({ executablePath: process.env.PW_EXEC, headless: true });
// content: 'omit' | 'embed' | 'attach' -- this one flag is the 11x size difference.
const context = await browser.newContext({ recordHar: { path: harPath, content: 'omit' } });
const page = await context.newPage();
 
// A CDP session on the SAME page, so both recordings see exactly one load.
const cdp = await context.newCDPSession(page);
const sent = [];
cdp.on('Network.requestWillBeSent', (e) => sent.push(e));
await cdp.send('Network.enable');
 
await page.goto('http://127.0.0.1:8098/', { waitUntil: 'load' }).catch(() => {});
await page.waitForTimeout(2500);   // drain in-flight requests before closing
await context.close();             // the HAR is only written on context.close()
await browser.close();
 
const log = JSON.parse(fs.readFileSync(harPath, 'utf8')).log;
console.log('HAR entries with an initiator:', log.entries.filter((e) => e._initiator).length);
console.log('CDP requests with an initiator:', sent.filter((e) => e.initiator).length);
console.log('CDP requests with a JS stack:  ', sent.filter((e) => e.initiator?.stack).length);

Run against the fixture, that prints:

HAR entries with an initiator: 0
CDP requests with an initiator: 11
CDP requests with a JS stack:   5

Environment: Playwright 1.62.1 driving its own chromium_headless_shell build, mitmproxy 11.0.2, on Linux aarch64, 14 August 2026. Both comparisons were run three times; the per-URL outcome table was identical each time apart from entry ordering.

Every corpus figure on this page comes from scripts/report_traffic_files.mjs; every query, its denominator, the tool-name fold with its unmapped residue, the 95 hand-classified HAR verdicts with the deciding sentence for each, and the spot-checked quotes are on traffic_files. The corpus-wide caveats — what the seven venues do and do not cover, why 2025–2026 is provisional, and how stable each field is — are on Corpus.

Three limits specific to this page:

• The HAR population is a full-text sweep, not a schema query. It finds papers that write the string; a paper that captured HAR files and never used the word is invisible. The 32 is therefore a floor.
• The capture-tool fold is a judgement call. Deciding that a software-defined radio is off-topic for a page about web traffic files and that selenium-wire is on-topic is a decision, not a measurement; the family list and the 162 excluded papers are published so you can disagree with it.
• The fixture comparisons are one page load of a synthetic page in one browser. They demonstrate mechanisms — an initiator field that is absent, a CSP-blocked request a proxy cannot see — not rates. Nothing on this page claims a HAR loses x% of anything on real sites.

• Crawler — what drives the browser, and which control channels can record the network at all.
• Archives — reading somebody else's recording, and the WARC side of replay.
• Requests — classifying the requests once you have them.
• IP classification — serverIPAddress, and why it is the proxy's when you proxy.
• Cookies — why a HAR's cookie arrays are not a cookie jar.
• Ethics — traffic files as personal data.
• Artifacts — what to release, and in what shape.

[1]

Trevisan, Martino; Traverso, Stefano; Bassi, Eleonora; Mellia, Marco (2019): "4 Years of EU Cookie Law: Results and Lessons Learned", Proceedings on Privacy Enhancing Technologies 2019(2):126-145. (DOI)

[2]

Wang, Kailong; Zhang, Junzhe; Bai, Guangdong; Ko, Ryan K. L.; Dong, Jin Song (2021): "It's Not Just the Site, It's the Contents: Intra-domain Fingerprinting Social Media Websites Through CDN Bursts", in: Proceedings of the ACM Web Conference. (DOI)

[3]

Muthuraj, Naveenraj; Eghbal, Nooshin; Lu, Paul (2024): "Replication: "Taking a long look at QUIC"", in: Proceedings of the ACM Internet Measurement Conference. (DOI)

[4]

Zhang, Xumiao; Jin, Shuowei; He, Yi; Hassan, Ahmad; Mao, Z. Morley; Qian, Feng; Zhang, Zhi-Li (2024): "QUIC is not Quick Enough over Fast Internet", in: Proceedings of the ACM Web Conference. (DOI)

[5]

Chaqfeh, Moumena; Zaki, Yasir; Hu, Jacinta; Subramanian, Lakshmi (2020): "JSCleaner: De-Cluttering Mobile Webpages Through JavaScript Cleanup", in: Proceedings of the ACM Web Conference. (DOI)

[6]

Borgolte, Kevin; Feamster, Nick (2020): "Understanding the Performance Costs and Benefits of Privacy-focused Browser Extensions", in: Proceedings of the ACM Web Conference. (DOI)

[7]

Kollnig, Konrad; Shuba, Anastasia; Binns, Reuben; Van Kleek, Max; Shadbolt, Nigel (2022): "Are iPhones Really Better for Privacy? A Comparative Study of iOS and Android Apps", Proceedings on Privacy Enhancing Technologies 2022(2):6-24. (DOI)

[8]

Figueira, Olivia; Trimananda, Rahmadi; Markopoulou, Athina; Jordan, Scott (2024): "DiffAudit: Auditing Privacy Practices of Online Services for Children and Adolescents", in: Proceedings of the ACM Internet Measurement Conference. (DOI)

[9]

Siby, Sandra; Barman, Ludovic; Wood, Christopher; Fayed, Marwan; Sullivan, Nick; Troncoso, Carmela (2023): "Evaluating Practical QUIC Website Fingerprinting Defenses for the Masses", Proceedings on Privacy Enhancing Technologies 2023(4):79-95. (DOI)

[10]

Hantke, Florian; Snyder, Peter; Haddadi, Hamed; Stock, Ben (2025): "Web Execution Bundles: Reproducible, Accurate, and Archivable Web Measurements", in: Proceedings of the USENIX Security Symposium. (Link)

[11]

Englehardt, Steven; Narayanan, Arvind (2016): "Online Tracking: A 1-million-site Measurement and Analysis", in: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pp. 1388–1401. Association for Computing Machinery, New York, NY, USA. (DOI) (Link)

[12]

Butkiewicz, Michael; Madhyastha, Harsha V.; Sekar, Vyas (2011): "Understanding website complexity: measurements, metrics, and implications", in: Proceedings of the ACM Internet Measurement Conference. (DOI)

[13]

Hounsel, Austin; Borgolte, Kevin; Schmitt, Paul; Holland, Jordan; Feamster, Nick (2020): "Comparing the Effects of DNS, DoT, and DoH on Web Performance", in: Proceedings of the ACM Web Conference. (DOI)

[14]

Singanamalla, Sudheesh; Paracha, Muhammad Talha; Ahmad, Suleman; Hoyland, Jonathan; Valenta, Luke; Safronov, Yevgen; Wu, Peter; Galloni, Andrew; Heimerl, Kurtis; Sullivan, Nick; Wood, Christopher A.; Fayed, Marwan (2022): "Respect the ORIGIN!: a best-case evaluation of connection coalescing in the wild", in: Proceedings of the ACM Internet Measurement Conference. (DOI)

[15]

Karnam, Sai Keerthana; Dash, Abhisek; Das, Antariksh; Mousavi, Sepehr; Bechtold, Stefan; Gummadi, Krishna P.; Mukherjee, Animesh; Weber, Ingmar; Zannettou, Savvas (2026): "Setting the Course, but Forgetting to Steer: Analyzing Compliance with GDPR's Right of Access to Data by Instagram, TikTok, and Youtube", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)

[16]

Innocenti, Tommaso; Jannett, Louis; Mainka, Christian; Mladenov, Vladislav; Kirda, Engin (2025): ""Only as Strong as the Weakest Link": On the Security of Brokered Single Sign-On on the Web", in: Proceedings of the IEEE Symposium on Security and Privacy. (DOI)