Working notes behind traffic_files — every query with its population and denominator, the scripts and their unedited output, the folds and their residue, the hand classification and its deciding sentences, the figures checked against the source papers, the external sources and how each was verified, and what could not be established. Corpus-level caveats that apply to every page on this site are on corpus and are not restated here.

This page is new, not an extension. start has promised Programming:Traffic files (“using traffic files, e.g., HAR”) since before this corpus existed, and ip_classification links to it twice — once for serverIPAddress and once for “extracting server addresses from a HAR or a packet capture”, both marked not yet written. Nothing on the wiki overlapped: crawler covers what drives the browser and stops at the control channel; archives covers reading somebody else's recording (WARC, the Wayback Machine) and explicitly not making your own; requests starts after you already have the requests. The gap this page fills is the artefact in between.

Judgement call on scope. The item as specified asked for “the HAR schema, what it loses versus a full CDP trace, mitmproxy/mitmdump, and browser-native export”. We widened it to include packet capture, because the corpus makes that unavoidable: tcpdump and Wireshark are the two most-named capture instruments in it (119 and 128 papers) and a page that omitted them would misrepresent what the literature does. We did not widen it to replay and archiving beyond one section, because archives already covers that well and duplicating it would have been worse than cross-linking.

Three separate populations, because no single field in the extraction means “wrote a traffic file”.

usedOrMentioned is filtered to used/produced everywhere. Including compared and mentioned would count a paper that named Wireshark as related work as a paper that captured packets.

traffic-capture on a seven-venue security corpus is not a web-measurement category. Of 679 papers, 162 name only instruments that produce no web traffic file at all. Those are folded into seven explicitly named offtopic: families and excluded with a printed count:

A paper can match several families, so the rows sum to more than the union (210 papers match at least one off-topic family; 162 match only off-topic families).

The VPN row is the one a reasonable person could disagree with. A commercial VPN in a proxy-interception tuple is a vantage point, which is crawling_location's subject, not an instrument that writes a file. We excluded it. The residential-proxy question is genuinely two questions and this page answers only one of them.

This row was wrong in the first published version and was fixed on review. It also swallowed Squid, Squid ICAP modules, Privoxy, Tinyproxy, 3proxy and wwwoffle — self-hosted forward and caching proxies, which sit exactly where an intercepting proxy sits and can log the same traffic (Squid with SSL-bump, Privoxy's filters). They are neither commercial VPNs nor vantage points, and excluding them as such was indefensible. They are now their own on-topic family, forward / caching proxy (self-hosted), 7 papers. ProxySQL (a database proxy), HA-Proxy, dnscrypt-proxy, doh-proxy and netcat were in the same row for the same bad reason and are now in a separate off-topic family named for what they actually are. The reviewer's point stands generally: an off-topic family's name is a claim about everything in it, and it has to be true of every member, not of the ones you had in mind when you wrote the regex.

cd /workspace/artifacts/wiki
 
# every figure on the page, with its denominator
node scripts/report_traffic_files.mjs                > out/traffic_files-output.txt
node scripts/report_traffic_files.mjs --list         > out/traffic_files-list.txt
node scripts/report_traffic_files.mjs --wiki                        # DokuWiki tables
node scripts/report_traffic_files.mjs --quotes 'mitm'               # evidence behind a family
 
# every literal per-paper figure and quote, checked against paper.cols.txt
node scripts/verify_traffic_figures.mjs              > out/traffic_files_figures.txt
 
# every external fact, re-checked against its primary source (needs network)
sh scripts/external_checks_traffic_files.sh          > out/traffic_files_external.txt 2>&1
 
# the two experiments (needs Playwright's headless_shell and mitmproxy)
export PLAYWRIGHT_BROWSERS_PATH=/workspace/.playwright
export PW_EXEC=$(ls -d /workspace/.playwright/chromium_headless_shell-*/chrome-linux/headless_shell | head -1)
cd sandbox && node fixture_traffic.mjs &            # http://127.0.0.1:8098/
node sandbox/har_vs_cdp.mjs                          > out/har_vs_cdp-output.txt
node sandbox/har_vs_cdp_core.mjs                     > out/har_vs_cdp_core-output.txt
MITMDUMP=/tmp/mitmlib/bin/mitmdump MITMLIB=/tmp/mitmlib \
  node sandbox/mitm_vs_browser.mjs                   > out/mitm_vs_browser-output.txt
MITMDUMP=/tmp/mitmlib/bin/mitmdump MITMLIB=/tmp/mitmlib \
  node sandbox/mitm_vs_browser.mjs https://example.com/ > out/mitm_vs_browser-remote-output.txt
 
# stale-number guard, whole page, including <file> blocks
cat out/traffic_files-output.txt out/traffic_files-list.txt out/traffic_files_figures.txt \
    out/har_vs_cdp-output.txt out/mitm_vs_browser-output.txt \
    out/mitm_vs_browser-remote-output.txt out/traffic_files_external.txt \
    > out/traffic_files_all_evidence.txt
node scripts/check_page_numbers.mjs pages/programming_traffic_files.txt \
    out/traffic_files_all_evidence.txt --code

The guard reports OK — every figure in the page traces to out/traffic_files_all_evidence.txt as of 2026-08-14. It took three passes to get there, and each one found something — see §9.

CAPTURE_FAMILIES in scripts/traffic_fold.mjs is an ordered list of regex families; first match wins, so offtopic families are listed first and specific names come before the generic patterns that would also swallow them (“tcpdump with PF_RING” must be packet capture, not high-rate hardware; “Tor SOCKS proxy server” must be a vantage point, not a TLS interceptor).

Unmapped residue: 9 distinct names across 9 papers, printed in full by the report and reproduced here so it does not live only in a local file:

Google Translate (1); browser add-on we designed (1); custom program (1); lb (1);
Facebook tracking pixel (1); SSL measurements (1); custom addon script (1);
custom tracing plugin (1); iPhone 16 (1)

All nine were read by hand. Six are extraction noise in a capture category (“Google Translate”, “Facebook tracking pixel”, “iPhone 16”, “lb”, “custom program”, “SSL measurements”). Three are genuinely unidentifiable instruments — “browser add-on we designed”, “custom addon script”, “custom tracing plugin” — where the paper names no tool at all. Leaving them unmapped is the honest outcome: they belong to no family because the paper does not say which.

Spelling variance, which is why the fold exists at all. tcpdump appears as tcpdump, Tcpdump, TCPdump, TCPDump, TCPDUMP, TCP-Dump, tcpdump/libpcap and tcpdump with PF_RING; Wireshark as Wireshark, WireShark, Wireshark GUI, WireShark/tshark, custom Wireshark dissectors and Wireshark plugin; mitmproxy as mitmproxy, Mitmproxy, MitmProxy, MITMProxy, mitm-proxy, mitmproxy.io, Customized Mitm Proxy and mitmdump.

The tool schema sees HAR in 10 papers. That is not a measurement of the field; it is a measurement of what people bother to call a tool. So the HAR population comes from a full-text sweep of every data/fulltext/<year>/<venue>/<slug>/paper.cols.txt (5,855 of the 5,859 records have one), matching

/\bHAR\b|HTTP Archive|HTTP archive|\.har\b|HttpArchive|httparchive/

95 papers match. HAR_VERDICT in scripts/traffic_fold.mjs carries one verdict and one deciding sentence per paper; the report fails loudly if the sweep returns a paper the map does not cover, or if the map covers a paper the sweep no longer returns.

This is the part that cannot be automated, and the reason the verdicts are hand-made.

Four of the eleven are artefacts of the corpus's own two-column repair, not of the papers. That is worth knowing generally: a full-text regex over this corpus will match text that does not exist as a token in the PDF.

Both were candidates for instrument and were rejected on reading:

• IMC 2011 towards-understanding-modern-web-traffic — “While their use of HTTP archive record (HAR) format [25] allows a precise detection of page load events, their data set consists of only the top-level pages of randomly chosen 2,000 Web sites”. That is a comparison to related work; this paper captures packets.
• PETS 2022 fp-radar-… — “HTTP Archive [38] is a related project that archives millions of URLs each month. However, it does not store …”. Considered and rejected in favour of the Wayback Machine.

The deciding sentence for every one is in HAR_VERDICT and printed by report_traffic_files.mjs §E.

“Does the paper say which writer produced the HAR” cannot be answered from the schema, so it is a keyword probe over the full text of the 32 instrument papers. It finds a recognisable writer in 25 (78.1%). It over-counts: a paper that cites Selenium anywhere scores a hit for “Selenium / WebDriver” (13 of the 25) whether or not Selenium wrote the HAR — and Selenium cannot write one. The probe is published because the negative side is sound: the 7 papers it finds nothing in genuinely name no writer.

The full output of node scripts/report_traffic_files.mjs is below, unedited. –list output (the 514-paper population with its families) is not reproduced here for length; regenerate it with the command in §3.

report_traffic_files.txt

corpus: 5859 papers, 7 venues, 2010–2026
data root: /workspace/publications_dataset/data
 
 
==== A. POPULATION ====
 
Any tool tuple in category traffic-capture, used or produced   509
Any tool tuple in category proxy-interception, used or produced 206
Union (the raw capture-tooling population)                      679
Overlap                                                          36
Of the union, platforms includes "web"                           217
Of the union, ran an automated web crawl                         178
 
Distinct tool names across both categories (used/produced): 457
 
── Capture-instrument families (of 679 papers naming any capture or proxy tool) ──
Family                                                          Layer     Papers  Share  Spellings
--------------------------------------------------------------  --------  ------  -----  ---------
offtopic: radio / SDR / RF sniffer                              offtopic  90      13.3%  105
offtopic: hardware probe / lab instrument                       offtopic  30      4.4%   34
offtopic: host / side-channel instrumentation                   offtopic  30      4.4%   29
offtopic: traffic generation / replay load                      offtopic  17      2.5%   12
offtopic: commercial VPN / residential proxy (a vantage point)  offtopic  27      4.0%   28
offtopic: proxies something other than web traffic              offtopic  5       0.7%   5
offtopic: passive dataset / telescope / vantage platform        offtopic  17      2.5%   16
browser-native HTTP log (HAR / NetLog)                          browser   8       1.2%   8
record-and-replay / web archive                                 browser   9       1.3%   9
forward / caching proxy (self-hosted)                           proxy     7       1.0%   6
TLS-intercepting proxy                                          proxy     189     27.8%  79
packet capture (libpcap family)                                 packet    287     42.3%  72
flow-level export (NetFlow / IPFIX)                             flow      13      1.9%   11
IDS / stream analyser                                           packet    28      4.1%   10
high-rate capture (kernel / NIC offload)                        packet    15      2.2%   9
on-device mobile / IoT capture                                  mobile    21      3.1%   15
 
On-topic families, union of papers      514
Off-topic families, union of papers     210
Off-topic ONLY (excluded from the page) 162
  (sum of family counts != union: a paper may match several families)
Unmapped residue: 9 distinct names, 9 papers
  Google Translate (1); browser add-on we designed (1); custom program (1); lb (1); Facebook tracking pixel (1); SSL measurements (1); custom addon script (1); custom tracing plugin (1); iPhone 16 (1)
 
── Capture layer (of 514 papers naming an on-topic capture instrument) ──
Layer    Papers  Share
-------  ------  -----
packet   318     61.9%
proxy    194     37.7%
mobile   21      4.1%
browser  17      3.3%
flow     13      2.5%
 
── Capture layer, web subset only (of 188 on-topic papers whose platform includes web) ──
Layer    Papers  Share of web subset
-------  ------  -------------------
proxy    95      50.5%
packet   86      45.7%
browser  13      6.9%
mobile   5       2.7%
flow     4       2.1%
 
── Named instruments in the on-topic families (papers; of 679) ──
Instrument                                                   Papers  Share
-----------------------------------------------------------  ------  -----
Wireshark / tshark / dumpcap                                 128     18.9%
tcpdump                                                      119     17.5%
mitmproxy / mitmdump                                         98      14.4%
Bro / Zeek                                                   24      3.5%
Scapy                                                        18      2.7%
Burp Suite                                                   18      2.7%
Fiddler                                                      15      2.2%
PF_RING                                                      12      1.8%
HAR (browser HTTP log)                                       10      1.5%
record-and-replay (WPR / pywb / Mahimahi)                    8       1.2%
Charles Proxy                                                5       0.7%
proxy libraries in the crawler (selenium-wire / BrowserMob)  4       0.6%
NetLog (Chrome)                                              3       0.4%
OWASP ZAP                                                    2       0.3%
 
 
==== B. IS THE INSTRUMENT IDENTIFIED? ====
 
── Version reporting ──
Tuples in on-topic capture families: 669; state a version: 85 (12.7%)
Tool              Papers  Tuples  Tuples with a version  Share
----------------  ------  ------  ---------------------  -----
Wireshark/tshark  121     130     16                     12.3%
tcpdump           119     119     4                      3.4%
mitmproxy         98      101     25                     24.8%
 
 
==== C. DRIFT OVER TIME ====
 
── Capture layer over time, share of the papers in each bucket that name an on-topic instrument ──
Bucket      Papers  On-topic  Share  packet  proxy  browser  flow  mobile
----------  ------  --------  -----  ------  -----  -------  ----  ------
2010–2013   511     54        10.6%  37      16     3        2     1
2014–2017   769     96        12.5%  59      35     2        2     3
2018–2021   1439    110       7.6%   67      40     4        4     6
2022–2024   1955    157       8.0%   98      60     6        3     9
2025–2026*  1185    97        8.2%   57      43     2        2     2
* 2025–2026 is provisional: CCS 2026 and IMC 2026 have not been held, and IEEE S&P/WWW 2026 abstracts are not fully in OpenAlex.
 
 
==== D. HAR: THE FULL-TEXT SWEEP AND ITS HOMOGRAPHS ====
 
── Full-text sweep ──
Papers with a paper.cols.txt: 5855
Matching /HAR|HTTP Archive|httparchive/: 95
Hand-classified: 95; UNCLASSIFIED (fix the hand map): 0
 
Verdict     Papers  Share of the sweep
----------  ------  ------------------
instrument  32      33.7%
dataset     14      14.7%
citation    36      37.9%
mention     2       2.1%
homograph   11      11.6%
 
── The 32 papers that use HAR as their own instrument ──
Venue    Year  Slug
-------  ----  --------------------------------------------------------------------------------
IMC      2011  understanding-website-complexity-measurements-metrics-and-implications
NDSS     2015  parking-sensors-analyzing-and-detecting-parked-domains
IMC      2017  the-ad-wars-retrospective-measurement-and-analysis-of-anti-adblock-filter-lists
USENIX   2017  characterizing-the-nature-and-dynamics-of-tor-exit-blocking
IEEE-SP  2018  surveylance-automatically-detecting-online-survey-scams
WWW      2018  proxytorrent-untangling-the-free-http-s-proxy-ecosystem
IMC      2019  an-empirical-study-of-the-cost-of-dns-over-https
PETS     2019  4-years-of-eu-cookie-law-results-and-lessons-learned
PETS     2019  oblivious-dns-practical-privacy-for-dns-queries
WWW      2020  comparing-the-effects-of-dns-dot-and-doh-on-web-performance
WWW      2020  jscleaner-de-cluttering-mobile-webpages-through-javascript-cleanup
WWW      2020  understanding-the-performance-costs-and-benefits-of-privacy-focused-browser-exte
IEEE-SP  2021  crawlphish-large-scale-analysis-of-client-side-cloaking-techniques-in-phishing
PETS     2021  oblivious-dns-over-https-odoh-a-practical-privacy-enhancement-to-dns
WWW      2021  its-not-just-the-site-its-the-contents-intra-domain-fingerprinting-social-media
WWW      2021  where-are-you-taking-me-understanding-abusive-traffic-distribution-systems
IMC      2022  respect-the-origin-a-best-case-evaluation-of-connection-coalescing-in-the-wild
PETS     2022  are-iphones-really-better-for-privacy-a-comparative-study-of-ios-and-android-app
IMC      2023  the-prevalence-of-single-sign-on-on-the-web-towards-the-next-generation-of-web-c
PETS     2023  evaluating-practical-quic-website-fingerprinting-defenses-for-the-masses
USENIX   2023  reversing-breaking-and-fixing-the-french-legislative-election-e-voting-protocol
IMC      2024  diffaudit-auditing-privacy-practices-of-online-services-for-children-and-adolesc
IMC      2024  of-choices-and-control-a-comparative-analysis-of-government-hosting
IMC      2024  replication-taking-a-long-look-at-quic
WWW      2024  quic-is-not-quick-enough-over-fast-internet
IEEE-SP  2025  only-as-strong-as-the-weakest-link-on-the-security-of-brokered-single-sign-on-on
IMC      2025  where-in-the-world-are-my-trackers-mapping-web-tracking-flow-across-diverse-geog
PETS     2025  tracker-installations-are-not-created-equal-understanding-tracker-configuration
PETS     2025  gig-work-at-what-cost-exploring-privacy-risks-of-gig-work-platform-participation
USENIX   2025  web-execution-bundles-reproducible-accurate-and-archivable-web-measurements
IEEE-SP  2026  setting-the-course-but-forgetting-to-steer-analyzing-compliance-with-gdprs-right
PETS     2026  more-space-less-privacy-measuring-the-effectiveness-of-ip-based-website-fingerpr
 
By year bucket: 2010–2013 1 | 2014–2017 3 | 2018–2021 12 | 2022–2024 9 | 2025–2026* 7
By venue: IMC 9 | PETS 8 | WWW 7 | IEEE-SP 4 | USENIX 3 | NDSS 1
 
── How many HAR-instrument papers does a tool-name query find? ──
Of the 32 HAR-instrument papers:
  name HAR / "HTTP Archive" in any tools[] or otherToolsMentioned[] tuple: 9 (28.1%)
  have ANY traffic-capture / proxy-interception tuple at all:              16 (50.0%)
  named in tools[]: PETS/2019/4-years-of-eu-cookie-law-results-and-lessons-learned, WWW/2020/comparing-the-effects-of-dns-dot-and-doh-on-web-performance, WWW/2020/understanding-the-performance-costs-and-benefits-of-privacy-focused-browser-exte, IMC/2024/diffaudit-auditing-privacy-practices-of-online-services-for-children-and-adolesc, IMC/2024/of-choices-and-control-a-comparative-analysis-of-government-hosting, IMC/2024/replication-taking-a-long-look-at-quic, WWW/2024/quic-is-not-quick-enough-over-fast-internet, IEEE-SP/2026/setting-the-course-but-forgetting-to-steer-analyzing-compliance-with-gdprs-right, IEEE-SP/2018/surveylance-automatically-detecting-online-survey-scams
 
── Crawling papers vs HAR-instrument papers, per venue ──
Venue    Papers  Ran a crawl  HAR-instrument papers
-------  ------  -----------  ---------------------
CCS      990     163          0
IMC      638     132          9
NDSS     701     129          1
PETS     510     123          8
USENIX   1410    221          3
WWW      843     242          7
IEEE-SP  767     110          4
 
── Homographs (why the sweep cannot be automated) ──
  [homograph] USENIX/2018/schrodinger-s-rat-profiling-the-stakeholders-in-the-remote-access-trojan-ecosyst
      Surname split across a column break: "D HAR - MDASANI"
  [homograph] CCS/2019/privacy-risks-of-securing-machine-learning-models-against-adversarial-examples
      "the UCI Human Activity Recognition (HAR) dataset"
  [homograph] USENIX/2019/stack-overflow-considered-helpful-deep-learning-security-nudges-towards-stronger
      Surname split across a column break: "B HAR - GAVA"
  [homograph] CCS/2023/large-language-models-for-code-security-hardening-and-adversarial-testing
      Identifier split across a column break: "IS_WORD_C HAR"
  [homograph] USENIX/2023/sqirl-grey-box-detection-of-sql-injection-vulnerabilities-using-reinforcement-le
      Surname split across a column break: "S HAR , L. K."
  [homograph] NDSS/2024/campro-camera-based-anti-facial-recognition
      "Vision-based human activity recognition (HAR)"
  [homograph] CCS/2025/harnessing-vital-sign-vibration-harmonics-for-effortless-and-inbuilt-xr-user-aut
      "we derive the Harmonic Amplitude Ratio (HAR)"
  [homograph] NDSS/2025/radsee-see-your-handwriting-through-walls-using-fmcw-radar
      "human activity recognition (HAR) using different sensing"
  [homograph] NDSS/2026/svdefense-effective-defense-against-gradient-inversion-attacks-via-singular-value-decomposition
      "Human Activity Recognition (HAR). HAR identifies daily activities like walking or sitting"
  [homograph] NDSS/2026/beyond-conventional-triggers-auto-contextualized-covert-triggers-for-android-logic-bombs
      "the Human Activity Recognition Dataset (KU-HAR)"
  [homograph] PETS/2026/sensor-privacy-as-a-spectrum-quantifying-privacy-in-edge-and-multimodal-systems
      "three widely-used benchmarks: UCI-HAR … UTAH-STM-HAR"
  [mention] IMC/2011/towards-understanding-modern-web-traffic
      "While their use of HTTP archive record (HAR) format [25] allows a precise detection of page load events, their data set consists of only …" — describing related work; the paper itself captures packets
  [mention] PETS/2022/fp-radar-longitudinal-measurement-and-early-detection-of-browser-fingerprinting
      "HTTP Archive [38] is a related project that archives millions of URLs each month. However, it does not store …" — considered and rejected in favour of the Wayback Machine
 
── Does a HAR-instrument paper name the writer that produced the file? ──
Of 32 instrument papers, 25 (78.1%) mention at least one recognisable writer somewhere in the full text.
Writer named anywhere in the paper      Papers
--------------------------------------  ------
Selenium / WebDriver                    13
Chrome DevTools / Network panel         8
mitmproxy / a proxy                     6
WebPageTest                             4
chrome-har-capturer                     3
PhantomJS                               3
Puppeteer / Playwright                  2
HARExportTrigger                        1
Firefox pref devtools.netmonitor.har.*  1
sitespeed.io                            1
No recognisable writer named:
  IMC/2011/understanding-website-complexity-measurements-metrics-and-implications
  PETS/2021/oblivious-dns-over-https-odoh-a-practical-privacy-enhancement-to-dns
  PETS/2023/evaluating-practical-quic-website-fingerprinting-defenses-for-the-masses
  USENIX/2023/reversing-breaking-and-fixing-the-french-legislative-election-e-voting-protocol
  IEEE-SP/2025/only-as-strong-as-the-weakest-link-on-the-security-of-brokered-single-sign-on-on
  IEEE-SP/2026/setting-the-course-but-forgetting-to-steer-analyzing-compliance-with-gdprs-right
  PETS/2026/more-space-less-privacy-measuring-the-effectiveness-of-ip-based-website-fingerpr
NOTE: this is a full-text keyword probe, so it OVER-counts — a paper that merely
cites Selenium elsewhere scores a hit. Read it as an upper bound on how often the
HAR writer is identifiable, not as a measurement of correct reporting.
 
── Traffic files inside a crawl ──
Papers that ran an automated web crawl:                       1120
  …that name an on-topic capture instrument in tools[]:      156 (13.9%)
  …that use HAR as an instrument (full-text sweep):          26 (2.3%)
  …union of the two:                                         172 (15.4%)
 
 
==== E. WHAT THE HAR PAPERS SAY THEY DID WITH IT ====
 
── Deciding sentence per instrument paper ──
  IEEE-SP/2018/surveylance-automatically-detecting-online-survey-scams
      "SURVEYLANCE implements HTTP Archive 1.2 specification [44], and stores the network traffic traces as HAR objects"
  IEEE-SP/2021/crawlphish-large-scale-analysis-of-client-side-cloaking-techniques-in-phishing
      "CrawlPhish stores source code using HAR files [57], which capture all HTTP requests/responses"
  IEEE-SP/2025/only-as-strong-as-the-weakest-link-on-the-security-of-brokered-single-sign-on-on
      "Input: HAR Files. We use a public dataset from prior work [34] … Each SSO flow is captured in the JSON-based HAR data format"
  IEEE-SP/2026/setting-the-course-but-forgetting-to-steer-analyzing-compliance-with-gdprs-right
      "we captured interactions with the browser by downloading the HAR (HTTP Archive) files"; HAR logs are the ground truth against which data-download packages are scored
  IMC/2011/understanding-website-complexity-measurements-metrics-and-implications
      Figure 1 is an example HAR snippet; "This extension generates a report in the HTTP archive record (HAR) format [4]"
  IMC/2017/the-ad-wars-retrospective-measurement-and-analysis-of-anti-adblock-filter-lists
      "we store all HTTP requests/responses in a HAR file [25]"; discards partial HARs below 10% of average size
  IMC/2019/an-empirical-study-of-the-cost-of-dns-over-https
      "we use the sitespeed.io project to instruct Firefox for the measurements and collect HAR files with the performance statistics"
  IMC/2022/respect-the-origin-a-best-case-evaluation-of-connection-coalescing-in-the-wild
      "Chrome developer tools were used to retrieve and write the page load data as an HTTP Archive format (HAR) file"
  IMC/2023/the-prevalence-of-single-sign-on-on-the-web-towards-the-next-generation-of-web-c
      "the HTTP transaction log (HAR format)"
  IMC/2024/diffaudit-auditing-privacy-practices-of-online-services-for-children-and-adolesc
      "we export the trace as a HAR (HTTP Archive) file [49], as provided by the Network Panel" for web, PCAP for mobile
  IMC/2024/of-choices-and-control-a-comparative-analysis-of-government-hosting
      "which are then consolidated into an HTTP Archive (HAR) file"
  IMC/2024/replication-taking-a-long-look-at-quic
      "PLT values are extracted from ‘onLoad’ field of HAR files [33] produced by the chrome-har-capturer tool"
  IMC/2025/where-in-the-world-are-my-trackers-mapping-web-tracking-flow-across-diverse-geog
      "recording HAR files and all network requests during page loads"
  NDSS/2015/parking-sensors-analyzing-and-detecting-parked-domains
      "recorded a trace of all HTTP requests initiated by the web page (HAR)"; five of its features are HAR features
  PETS/2019/4-years-of-eu-cookie-law-results-and-lessons-learned
      "they dump to file the HTTP Archive (HAR) [55], a JSON-formatted … We look at all HTTP responses with Set-Cookie header in the HAR file." Also analyses the httparchive.org corpus — counted in both.
  PETS/2019/oblivious-dns-practical-privacy-for-dns-queries
      "Chrome webdriver and record HAR files for each browsing session"
  PETS/2021/oblivious-dns-over-https-odoh-a-practical-privacy-enhancement-to-dns
      "capture the page load session into an HTTP Archive (HAR) file additionally recording the results of the window.performance API"
  PETS/2022/are-iphones-really-better-for-privacy-a-comparative-study-of-ios-and-android-app
      "using mitmproxy to capture network traffic in the HAR format"
  PETS/2023/evaluating-practical-quic-website-fingerprinting-defenses-for-the-masses
      "We perform HAR captures and we identify the protocols used by those websites"
  PETS/2025/gig-work-at-what-cost-exploring-privacy-risks-of-gig-work-platform-participation
      "collect all HTTP traffic generated during our interactions with these platforms in the form of HAR files"
  PETS/2025/tracker-installations-are-not-created-equal-understanding-tracker-configuration
      "both through the network HAR file and using Chrome’s manifest v2 webNavigation feature"
  PETS/2026/more-space-less-privacy-measuring-the-effectiveness-of-ip-based-website-fingerpr
      "we capture HTTP Archive (.har) files to record all domain connections initiated during the browsing session"
  USENIX/2017/characterizing-the-nature-and-dynamics-of-tor-exit-blocking
      "we first rule out cases of client and network errors through HAR file analysis"
  USENIX/2023/reversing-breaking-and-fixing-the-french-legislative-election-e-voting-protocol
      "These are gathered into HTTP Archive format (HAR) files that can be easily generated by major browsers"
  USENIX/2025/web-execution-bundles-reproducible-accurate-and-archivable-web-measurements
      Measures HAR and WARC replay against its own .web format: "we successfully visited 8,523 origins with 311 HAR and 44 WARC (overlapping) issues replaying responses"
  WWW/2018/proxytorrent-untangling-the-free-http-s-proxy-ecosystem
      "Data is collected as an HTTP Archive (HAR); for this, we have extended PhantomJS’s HAR capturer"
  WWW/2020/comparing-the-effects-of-dns-dot-and-doh-on-web-performance
      "For each HAR file that we collected with our automated browser, we extract the onLoad timing"; reports the first-query 0 ms DNS-timing artefact
  WWW/2020/jscleaner-de-cluttering-mobile-webpages-through-javascript-cleanup
      "Webpagetest also records the full waterfall HTTP Archive (HAR) representation of the session"
  WWW/2020/understanding-the-performance-costs-and-benefits-of-privacy-focused-browser-exte
      "we automatically extract the HAR through a modified version of the open source HARExportTrigger extension"
  WWW/2021/its-not-just-the-site-its-the-contents-intra-domain-fingerprinting-social-media
      "We capture the HAR (HTTP Archive) files … by setting the preference devtools.netmonitor.har.enableAutoExportToFile in Firefox"; also runs tcpdump
  WWW/2021/where-are-you-taking-me-understanding-abusive-traffic-distribution-systems
      "Manually investigating HTTP archive files, we verified if any of the 1,339 pages labeled earlier … are truly impersonating"
  WWW/2024/quic-is-not-quick-enough-over-fast-internet
      "Using chrome-har-capturer [12], we build scripts to collect HTTP Archive (HAR) files"
 
 
==== Z. FIGURES ON THE PAGE THAT ARE NOT FROM THIS CORPUS ====
 
Every number below is external to the publications dataset. Its primary source
is recorded on provenance:programming:traffic_files; scripts/external_checks_traffic_files.sh
re-fetches each one.
 
  This block is a POINTER, not a source of truth. The authoritative list of
  external figures, each with the primary source it was fetched from, is in
  scripts/verify_traffic_figures.mjs (EXTERNAL FIGURES) and is re-fetched by
  scripts/external_checks_traffic_files.sh. Two sibling scripts drifted apart
  here once already — this block carried mitmproxy 12.1.2 / Wireshark 4.6.2 /
  Chrome 127 as "current" while the page correctly said 12.2.3 / 4.6.8 / 130,
  and the stale copy was reproduced verbatim in the provenance page's
  "unedited report output". Caught by review on 2026-08-14.
 
    node scripts/verify_traffic_figures.mjs
    sh   scripts/external_checks_traffic_files.sh

Both run against sandbox/fixture_traffic.mjs, a local page built specifically to hit the places where the recordings disagree: a three-hop 302 chain, four script-initiated fetch calls, a POST with a JSON body, a 200 KiB response, a route that is never answered, a WebSocket with two frames each way, and an <img> whose origin the page's own CSP forbids.

Environment: Playwright 1.62.1 driving chromium_headless_shell from /workspace/.playwright; mitmproxy 11.0.2 installed into /tmp/mitmlib (the current release is 12.2.3 — the page says so); Linux aarch64; 14 August 2026.

har_vs_cdp.txt

fixture:        http://127.0.0.1:8098/
playwright:     1.62.1
har creator:    Playwright 1.62.1, HAR version 1.2
har files:      /tmp/harcdp-POW6Tb/har-omit.har (20566 B), /tmp/harcdp-POW6Tb/har-embed.har (226719 B)
 
                                                         HAR (omit)  HAR (embed)  CDP
entries / requests                                       12          12           11
  …of which redirect hops                                2           2            2
  …of which never completed (status -1 / loadingFailed)  2           2            2
entries carrying response body text                      0           6            on demand (Network.getResponseBody)
entries with an initiator / call stack                   0           0            11
  …with a JS stack frame                                 0           0            5
entries with a resource type                             12          12           11
entries with a frame id                                  12          12           11
entries with serverIPAddress                             8           8            7
WebSocket connections                                    1           1            1
WebSocket frames                                         0           4            4
 
--- what each recording says about the same four things ---
 
/beacon    HAR keys on the entry: pageref, startedDateTime, time, request, response, cache, timings, _resourceType
           HAR timings:           {"dns":-1,"connect":-1,"ssl":-1,"send":0,"wait":0.868,"receive":-1}
           HAR startedDateTime:   2026-08-14T02:56:47.617Z
/post      HAR request.postData:  {"mimeType":"application/json","text":"{\"id\":\"u-42\",\"ts\":1}","params":[]}
/big       content.size / text len (embed): 204800 / 204800
/big       content.size / text len (omit):  204800 / absent
 
--- cookies ---
response.cookies on the main document (HAR): [{"name":"http_cookie","value":"xyz789","path":"/","httpOnly":true}]
Set-Cookie response headers on the main document (HAR): [{"name":"set-cookie","value":"http_cookie=xyz789; Path=/; HttpOnly"}]
cookies in the browser jar at the end of the load (context.cookies()): http_cookie(httpOnly=true), pixel_id(httpOnly=false), js_cookie(httpOnly=false), beacon_id(httpOnly=false)
  → the jar has 4; the HAR's per-entry cookie arrays are per-transaction, not a jar dump.
 
--- URLs each recording saw ---
HAR only: ws://127.0.0.1:8098/ws
CDP only: (none)
both:     11 URLs
CSP-blocked image requested by the document: in HAR? true; in CDP? true
 
CDP loadingFailed:
  http://127.0.0.99:1/blocked-by-csp.gif errorText="" blockedReason="csp"
  http://127.0.0.1:8098/beacon?u=http%3A%2F%2F127.0.0.1%3A8098%2F errorText="net::ERR_ABORTED" blockedReason=undefined
 
--- the two requests the browser never completed ---
blocked-by-csp
    HAR: status=-1 statusText="" _failureText="csp" bodySize=-1
    CDP: errorText="" blockedReason="csp" canceled=false
/slow
    HAR: status=-1 statusText="" _failureText=undefined bodySize=-1
    CDP: errorText=undefined blockedReason=undefined canceled=undefined
 
--- non-standard ("_"-prefixed) fields this writer emits ---
  log.entries._resourceType
  log.entries._securityDetails
  log.entries._serverPort
  log.entries._webSocketMessages
  log.entries.response._failureText
  log.entries.response._transferSize
 
--- file size ---
  content:'omit'   20566 B
  content:'embed'  226719 B  (11.0x, for one 200 KiB response body)
  sum of response.content.size across all entries: 205679 B
  base64 inflation on the embedded bodies: 100.2% of the raw body bytes

Stability. Run three times consecutively. The per-URL outcome table was identical each time; only the ordering of entries within the HAR varied, and the byte counts moved by tens of bytes (timestamps). The two disagreements the page reports — HAR marks /slow failed and the beacon complete, CDP the reverse — reproduced on all three runs.

mitm_vs_browser.txt

mitmproxy creator: mitmproxy 11.0.2, HAR 1.2
browser creator:   Playwright 1.62.1, HAR 1.2
 
                                  browser HAR   mitmproxy HAR
entries                           12            11
distinct URLs                     12            11
entries with response body text   0             10
entries with _resourceType        12            1
entries with a pageref            12            0
log.pages                         1             0
entries with serverIPAddress      8             11
WebSocket entries                 1             0
 
in the browser HAR only:   http://127.0.0.99:1/blocked-by-csp.gif, ws://127.0.0.1:8098/ws
in the mitmproxy HAR only: /ws
 
file sizes
  browser HAR (content omitted)     20453 B
  mitmproxy HAR (bodies included)   261692 B
  mitmproxy .flows (its own format) 232420 B
  ratio, mitmproxy HAR / browser HAR 12.8x
 
serverIPAddress recorded by each side
  browser HAR:   127.0.0.1   (the address the browser connected to = the proxy)
  mitmproxy HAR: 127.0.0.1   (the address the proxy connected to = the origin)
  NOTE: on this target both are loopback, so the two values coincide and this run
  does not by itself show the difference — re-run against a remote origin.
 
mitmproxy HAR entry keys: _resourceType, _webSocketMessages, cache, request, response, serverIPAddress, startedDateTime, time, timings
browser  HAR entry keys: _resourceType, _securityDetails, _serverPort, cache, pageref, request, response, serverIPAddress, startedDateTime, time, timings

Re-run against a remote origin, to test the serverIPAddress claim that the loopback fixture cannot show:

mitm_vs_browser_remote.txt

mitmproxy creator: mitmproxy 11.0.2, HAR 1.2
browser creator:   Playwright 1.62.1, HAR 1.2
 
                                  browser HAR   mitmproxy HAR
entries                           1             1
distinct URLs                     1             1
entries with response body text   0             1
entries with _resourceType        1             0
entries with a pageref            1             0
log.pages                         1             0
entries with serverIPAddress      1             1
WebSocket entries                 0             0
 
in the browser HAR only:   (none)
in the mitmproxy HAR only: (none)
 
file sizes
  browser HAR (content omitted)     2491 B
  mitmproxy HAR (bodies included)   5831 B
  mitmproxy .flows (its own format) 8253 B
  ratio, mitmproxy HAR / browser HAR 2.3x
 
serverIPAddress recorded by each side
  browser HAR:   127.0.0.1   (the address the browser connected to = the proxy)
  mitmproxy HAR: 104.20.23.154   (the address the proxy connected to = the origin)
  Behind a proxy, every serverIPAddress in the browser's HAR is the proxy's.
 
mitmproxy HAR entry keys: cache, request, response, serverIPAddress, startedDateTime, time, timings
browser  HAR entry keys: _resourceType, _securityDetails, _serverPort, cache, pageref, request, response, serverIPAddress, startedDateTime, time, timings

sandbox/har_vs_cdp_core.mjs is the code in the page's <file javascript> block, extracted from the page source and run, so that the three-line output quoted beside it is that script's real output rather than a retyped one:

HAR entries with an initiator: 0
CDP requests with an initiator: 11
CDP requests with a JS stack:   5

Every literal figure the page attributes to a paper was checked against that paper's own paper.cols.txt with whitespace normalised — not against evidence.quote, because detection[].prevalence is a model summary of a result rather than a quotation. 17 literals, 0 missing.

traffic_files_figures.txt

=== per-paper figures, checked against paper.cols.txt (whitespace normalised)
 
USENIX/2025/web-execution-bundles-reproducible-accurate-and-archivable-web-measurements
  ok      0.7%
  ok      13.9%
  ok      13.3%
  ok      8,523
  ok      311
  ok      44
  ok      8,544
  ok      778,500
  ok      776,229
  ok      0.3%
  ok      6,043
 
=== verbatim quotes
 
USENIX/2025/web-execution-bundles-reproducible-accurate-and-archivable-web-measurements
  ok      "HAR is developed to record all requests that leave the browser"
  ok      "the drawback of replaying content rather than recording and inspecting the executions directly"
 
CCS/2016/online-tracking-a-1-million-site-measurement-and-analysis
  ok      "we chose to use Mitmproxy"
  ok      "to record all HTTP Re"
 
WWW/2020/comparing-the-effects-of-dns-dot-and-doh-on-web-performance
  ok      "the first query that a HAR contains can show DNS timings of 0 ms"
 
IMC/2022/respect-the-origin-a-best-case-evaluation-of-connection-coalescing-in-the-wild
  ok      "as an HTTP Archive format (HAR) file"
 
17 literals checked, 0 missing.
 
NOTE: presence is not pairing. "X of Y" was additionally read in context — see
provenance:programming:traffic_files §quotes spot-checked.
 
 
=== EXTERNAL FIGURES — every number on the page that is NOT from the corpus
Each was fetched by scripts/external_checks_traffic_files.sh on 2026-08-14.
 
  HAR 1.2                spec version; written 2011, W3C draft dated 2012, abandoned
  Chrome 130             October 2024; sanitized HAR export becomes the default
  chrome-har-capturer 0.14.4   npm, published 2026-01-27
  mitmproxy 12.2.3       released 2026-05-12 (the runs on this page used 11.0.2)
  Wireshark 4.6.8        current stable
  tcpdump 4.99.6         latest tag
  libpcap 1.10.6         latest tag
  Playwright 1.62.1      the version the fixture runs used
  Playwright 1.60        tracing.startHar()/stopHar() added (May 2026)
  Playwright 1.61        "HAR and trace recordings now include WebSocket requests" (June 2026)
  HARExportTrigger 0.6.1 last release, May 2018; repo archived, description "DEPRECATED"
  selenium-wire          repo archived January 2024
  BrowserMob Proxy       last tagged release 2016; last commit 2024
  draft-ietf-opsawg-pcap rev 08, intended status Historic
  draft-ietf-opsawg-pcapng rev 05, Informational
  WARC 1.1 / ISO 28500:2017  the WARC standard design:archives covers
  Okta: 28 Sept – 17 Oct 2023, 5 customers' sessions hijacked, 134 customers initially
        reported, scope revised 29 November 2023
  1500 ms                Firefox devtools.netmonitor.har.pageLoadedTimeout default
  five-line key log      one https://example.com/ load with --ssl-key-log-file, 2026-08-14
 
=== TOKENS — the exact digit tokens the above produce, isolated so that
    check_page_numbers.mjs (which splits on the second dot in a version string)
    can match them. Each is accounted for immediately above.
    0.14 4 12.2 3 4.6 8 4.99 6 1.10 6 1.62 1 1.60 1.61 0.6 1 08 05 130 1500 1.2 5 134 2.1 4
    1,000 802.15 12.8 127.0 0.1 127.0.0.99 104.20 23.154 8098 8081 2500
 
=== NOT FIGURES — digit fragments the page contains that are not measurements
  127.0.0.1, 127.0.0.99                   loopback addresses in the fixture output
  104.20.23.154                           example.com's address as the proxy saw it on
                                          2026-08-14. NOT a constant: example.com is on
                                          Cloudflare anycast and this rotates. It was
                                          listed here as an unchanging "not a figure"
                                          once, which let a WRONG value (172.66.147.243,
                                          from an unrelated earlier curl) survive the
                                          stale-number guard. Caught by review 2026-08-14.
  8098, 8081                              fixture and proxy ports
  802.15.4                                a radio standard named in the fold
  2500                                    a waitForTimeout argument in published code
  200 KiB, 204, 302, 101, -1              HTTP sizes and status codes

Presence is not pairing, so the Hantke et al. figures were additionally read in context. The sentence behind “less than 0.3%” is: “In total, we count 778,500 requests in HAR after filtering out non-page-related requests like service workers or initial redirects, while we see 776,229 requests captured via WebREC for 8,544 successfully crawled origins. This makes a negligible difference of less than 0.3%.” Numerator, denominator and percentage are one sentence. Likewise “we observed an average difference across all origins of 0.7% between .web and BL, we measured 13.9% for HAR and 13.3% for WARC replayed responses” and “we successfully visited 8,523 origins with 311 HAR and 44 WARC (overlapping) issues replaying responses”.

node scripts/quote_check.mjs –tools 'mitmproxy|tcpdump|wireshark|tshark' over the whole corpus:

357 quotes checked: 229 exact, 87 partial (>=60% of 5-word windows), 41 below threshold, 0 with no full text on disk.

Below-threshold is not “unsupported”. The lowest-scoring one is the single most-quoted sentence on the content page, Englehardt & Narayanan's, at 20% — and it is verbatim in the paper. The .cols rendering reads “After examining several Python HTTP proxies, we chose to use Mitmproxy6 to record all HTTP Re-quest and Response headers”: a footnote marker and a hyphenation split by the two-column repair. Checked by hand; the extraction is faithful. Two others were spot-checked the same way (respect-the-origin, comparing-the-effects-of-dns-dot-and-doh) and both are present verbatim.

scripts/external_checks_traffic_files.sh re-fetches every non-corpus fact. Run 2026-08-14; every check returned. Findings that changed what the page says:

• har.fyi as a HAR-format reference. It is the HTTP Archive dataset documentation. A sub-agent's first pass proposed it as the modern spec home; rejected after reading its own tagline.
• Secondary reporting on the Okta incident. Widely-repeated sentences attributed to Cloudflare and BeyondTrust about what was inside their HAR files could not be traced to a primary source: BeyondTrust's blog returns HTTP 403 to every non-browser client, and 1Password's own post says only that it “detected suspicious activity on our Okta instance” with “no 1Password user data accessed”. A draft of the page asserted that all three “reported follow-on intrusion attempts using session tokens lifted from those files”. That sentence was cut and replaced with Okta's own words, which are primary and sufficient.
• Fiddler licensing news. A widely-cited August 2026 post about Progress/Telerik restricting Fiddler Classic to non-commercial use is by Fiddler's original author on a personal blog, not by the vendor. The page says only that Fiddler Classic is no longer in active development, which the vendor's own product pages support.
• “Other HAR-leak incidents”. Searched for; none found with a primary source. The page therefore names exactly one incident and does not imply a pattern.
• A licence for the HTTP Archive BigQuery dataset. Could not be established from httparchive.org or har.fyi. The page does not state one.

• How much a browser HAR and a proxy HAR of the same crawl disagree at scale. The fixture demonstrates the mechanisms on one synthetic page; nothing in this corpus quantifies the gap over a top-list crawl. This is on the page as a <wrap todo>.
• Whether Chrome's HAR export includes response bodies by default. Firefox's answer is in the pref file; Chrome's current documentation does not restate a default for body inclusion, only for the header sanitisation. The page states the Firefox default and does not claim a Chrome one.
• Whether venue artifact-evaluation processes say anything about releasing traffic captures. The public CFP and artifact pages of IEEE S&P 2026, IMC 2025/2026 and PoPETs were fetched and contain no clause specific to HAR files, pcaps or network captures — guidance stops at the generic Menlo-Report level. That is a verified negative, and the page does not claim venue guidance exists.
• The 32 HAR-instrument papers are a floor, not a count. A paper that captured HAR files and never wrote the word is invisible to a full-text sweep.

Kept as a log, because the useful part is which check fired.

Review passes are logged in §10 once run.

Three focused passes and the author's own read, all against the frozen snapshot rev 1786677596 (pages/programming_traffic_files.txt md5 d165a98f56316b0d2c618128bc9d801d). Each reviewer was told its context might not be exhaustive and was handed the page, the provenance, the scripts and their committed output. Nothing was edited while they ran.

The same pass re-ran the report (plain and –wiki, byte-identical to the committed output), re-ran the figure verifier, re-ran both experiments three times, and confirmed papers-not-tuples, sentinel handling, usedOrMentioned filtering, both-ended year buckets and every denominator in “Use in Publications”. No other issue found.

All 16 {[key]} citations resolve; 279 keys in the bibliography, 279 unique; all 14 new entries match Crossref on authors, title, year, DOI, and the three hand-written PoPETs entries match on volume, issue and pages. Every quotation verified verbatim after whitespace normalisation. Every claim-to-paper attribution checked, including all eleven Hantke et al. figures for numerator/denominator/percentage coming from the same sentence. The “six papers cite a W3C HAR URL” figure was independently reproduced (two dvcs.w3.org, four w3c.github.io). No missing citations, no footnote/citation mismatches.

The same pass independently re-verified the HAR abandonment banner, the two dvcs.w3.org/w3c.github.io copies, har.fyi’s subject, both IETF drafts, WARC/ISO 28500, seven tool versions, every maintenance-status call, the Chrome 130 wording, all ten Firefox prefs read from firefox.js, –set hardump, the QUIC/custom-CA limitation, SSLKEYLOGFILE for both browsers, and ~20 URLs for liveness. One 403 (iso.org), already disclosed in §8.3. Nothing else superseded.

Three fixes found by re-reading rather than by any check: the WebPageTest row put its citation where it looked like it supported a vendor-ownership claim; “papers that used HARExportTrigger cannot be reproduced with it today” asserted something never tested (the extension is archived, which is the actual point); and “the absence of CCS is real” invited a causal reading. The last was sharpened rather than softened after measuring it — CCS contributes 163 crawling papers to IMC's 132 and produces 0 HAR-instrument papers to IMC's 9, which is a stronger statement than the original and is now backed by a per-venue table in the report.

Ten accepted findings, zero rejected. The three passes disagreed usefully: only the figures pass could have found F3 and F4 (they need the scripts re-run), only the citation pass could have found C2 (it needs the cited paper read), and only the currency pass could have found E1 (it needs today's release notes). A fourth, generic pass (fable, no checklist) was launched after the other three had been applied, and had not returned when this log was first published. It is reading the page and this provenance page against the “no MDN, no textbook” test, for overstated claims, unmeasured framing sentences, internal contradictions and scope overlap with archives and crawler. Its findings will be appended here as §10.6 rather than folded silently into the sections above, so that what the first three passes missed stays visible.

This provenance page carries no ~~DISCUSSION~~ block, following the convention set by the existing provenance: pages: comments belong on the content page, where a reader is. It adds no bibliography entries of its own — it cites the same {[key]} keys as traffic_files against the shared bibliography.

• traffic_files — the content page this backs.
• corpus — the corpus-wide provenance: venue scope, selection funnel, field stability, and why 2025–2026 is provisional.