Measure The Web

User Tools

Site Tools

Trace:
design:archives

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
design:archives [2026/08/13 22:44] – Extend: what an archive preserves and what it does not (static capture, escapes, coverage, vantage), corpus-backed Use in Publications, best-practice checklist, anachronism trap, self-recording alternative; correct five stale/wrong facts (Wayback size, Sa karel.kubicek.claudedesign:archives [2026/09/04 17:29] (current) – Citekey consolidation 2026-09-04: repoint duplicate bibliography keys (fouad2022my/boettger2025_regional/ahmad2026_ipfp/bouhoula2024automated/lerner2016internet) to the kept key; no prose or figure changed. Authored by Claude karel.kubicek.claude
Line 3: Line 3:
 A web archive lets you measure a page you cannot visit — because the date has passed, because the site is gone, or because you want everyone else to be able to measure exactly the same bytes. That is a genuinely different instrument from a crawler, and it fails in genuinely different ways. This page is about **what an archive can and cannot answer**, not about how archiving works. A web archive lets you measure a page you cannot visit — because the date has passed, because the site is gone, or because you want everyone else to be able to measure exactly the same bytes. That is a genuinely different instrument from a crawler, and it fails in genuinely different ways. This page is about **what an archive can and cannot answer**, not about how archiving works.
  
-Researchers have used archives for longitudinal analyses of privacy {[lerner2016internet,jha2024privacy,amos2021privacy,dimova2021cname,singh2026_empire]} and of security features {[pletinckx2021out,roth2020complex,stock2017web]}. The attraction is reproducibility: a snapshot is static, so a later reader can fetch the same one, and nobody has to write another crawler. The cost is that a snapshot is a **partial recording made by someone else's crawler, from their vantage point, at a moment they chose**, and almost every pitfall below follows from that one sentence.+Researchers have used archives for longitudinal analyses of privacy {[lerner2016_internet,jha2024privacy,amos2021privacy,dimova2021cname,singh2026_empire]} and of security features {[pletinckx2021out,roth2020complex,stock2017web]}. The attraction is reproducibility: a snapshot is static, so a later reader can fetch the same one, and nobody has to write another crawler. The cost is that a snapshot is a **partial recording made by someone else's crawler, from their vantage point, at a moment they chose**, and almost every pitfall below follows from that one sentence.
  
 Two numbers frame everything that follows. In our corpus of seven security and privacy venues (see [[#Use in Publications]]), **115 of 5,118 empirical papers (2.2%) name a web archive at all**, against 1,236 (24.2%) that ran a live crawl. Of the papers that do, **only about a quarter collected nothing of their own** — 58.2% ran a live crawl as well. The archive is normally a supplement, not the instrument. Two numbers frame everything that follows. In our corpus of seven security and privacy venues (see [[#Use in Publications]]), **115 of 5,118 empirical papers (2.2%) name a web archive at all**, against 1,236 (24.2%) that ran a live crawl. Of the papers that do, **only about a quarter collected nothing of their own** — 58.2% ran a live crawl as well. The archive is normally a supplement, not the instrument.
Line 16: Line 16:
  
   - {[hantke2023you]} — the systematic study. Compares seven archives, quantifies what goes wrong in the Internet Archive, and ends with a best-practice list. If you read one, read this.   - {[hantke2023you]} — the systematic study. Compares seven archives, quantifies what goes wrong in the Internet Archive, and ends with a best-practice list. If you read one, read this.
-  - {[lerner2016internet]} — the founding archive-based measurement in this literature (tracking, 1996–2016), and still the only paper that measured how often replay escapes to the live web.+  - {[lerner2016_internet]} — the founding archive-based measurement in this literature (tracking, 1996–2016), and still the only paper that measured how often replay escapes to the live web.
   - {[zhu2025_toward]} — why a static crawl misses half the page, measured in 2025, from the archive operators' side of the problem.   - {[zhu2025_toward]} — why a static crawl misses half the page, measured in 2025, from the archive operators' side of the problem.
   - {[singh2026_empire]} — the model recent application: a 2026 longitudinal study that states its snapshot rule, its rate limiting and its handling of classifier anachronism, all of which you can copy.   - {[singh2026_empire]} — the model recent application: a 2026 longitudinal study that states its snapshot rule, its rate limiting and its handling of classifier anachronism, all of which you can copy.
Line 58: Line 58:
 ==== Escapes: when an archived page measures today ==== ==== Escapes: when an archived page measures today ====
  
-URL rewriting inside a replayed page is not perfect, chiefly because it cannot rewrite URLs that JavaScript builds at runtime. When it fails, the replayed page issues a request to the **live** web. Lerner et al. measured this on archived snapshots of the Alexa top 500 and found that **16.1% of all requests attempted to escape the archive** — 17.2% of unique URLs and 11.5% of domains {[lerner2016internet]}. That measurement is from 2016, on 2015-era replay, and **nobody has repeated it since**; treat it as evidence that escapes matter, not as today's rate.+URL rewriting inside a replayed page is not perfect, chiefly because it cannot rewrite URLs that JavaScript builds at runtime. When it fails, the replayed page issues a request to the **live** web. Lerner et al. measured this on archived snapshots of the Alexa top 500 and found that **16.1% of all requests attempted to escape the archive** — 17.2% of unique URLs and 11.5% of domains {[lerner2016_internet]}. That measurement is from 2016, on 2015-era replay, and **nobody has repeated it since**; treat it as evidence that escapes matter, not as today's rate.
  
 To put that in proportion: in the same table, robots.txt exclusion accounted for 2.0% of requests and the resource simply not being archived for 1.4%. **Escapes were by far the largest single failure mode — eight times robots exclusion.** Their crawler blocked those requests, which is the right default: an escaped request measures the present under a past timestamp. If your tooling does not block them, your "2014 measurement" silently contains 2026 responses. Blocking is one rule in whatever drives your browser — abort any request whose URL is not under ''web.archive.org'' — and **count what you blocked**, because that count is a fidelity statistic worth reporting. The same mechanism is a security problem, not only a measurement one — Lerner et al. later showed that an attacker who controls an escape destination can rewrite what a client sees in the archive, and that **73% of Top 500 sites and 80% of Top Million domains** had snapshots vulnerable to at least one such attack {[lerner2017_rewriting]}. To put that in proportion: in the same table, robots.txt exclusion accounted for 2.0% of requests and the resource simply not being archived for 1.4%. **Escapes were by far the largest single failure mode — eight times robots exclusion.** Their crawler blocked those requests, which is the right default: an escaped request measures the present under a past timestamp. If your tooling does not block them, your "2014 measurement" silently contains 2026 responses. Blocking is one rule in whatever drives your browser — abort any request whose URL is not under ''web.archive.org'' — and **count what you blocked**, because that count is a fidelity statistic worth reporting. The same mechanism is a security problem, not only a measurement one — Lerner et al. later showed that an attacker who controls an escape destination can rewrite what a client sees in the archive, and that **73% of Top 500 sites and 80% of Top Million domains** had snapshots vulnerable to at least one such attack {[lerner2017_rewriting]}.
Line 290: Line 290:
 ===== Open Questions ===== ===== Open Questions =====
  
-  * <wrap todo>**The archive comparison needs redoing, and the instrument for redoing it shrank.** Memento Time Travel was decommissioned in 2025; MemGator survives but reaches 13 archives rather than 30-plus, so Hantke et al.'s enumeration {[hantke2023you]} cannot be repeated as written and its 2022 coverage numbers are now four years old. Nobody has published a current census of which public web archives still crawl.</wrap> +<WRAP todo> 
-  * <wrap todo>**Nobody has re-measured static-vs-dynamic archive bias at scale.** The 73.9%/95.3% tracker gap {[hantke2023you]} rests on 2,026 sites, limited by rate-limiting. Zhu et al.'s {[zhu2025_toward]} 45.8% fidelity gap is measured on their own crawls, not on the Internet Archive's holdings. A large-scale measurement of what the Internet Archive itself is missing, per resource type, does not exist.</wrap> +  * **The archive comparison needs redoing, and the instrument for redoing it shrank.** Memento Time Travel was decommissioned in 2025; MemGator survives but reaches 13 archives rather than 30-plus, so Hantke et al.'s enumeration {[hantke2023you]} cannot be repeated as written and its 2022 coverage numbers are now four years old. Nobody has published a current census of which public web archives still crawl. 
-  * <wrap todo>**Monoculture risk is unquantified.** Roughly two thirds of archive-using papers depend on one archive, one crawler and one vantage point. What that shared bias does to the field's longitudinal results has not been studied — and Common Crawl, the only comparison Hantke et al. could make, was rejected on coverage rather than validated on agreement.</wrap> +  * **Nobody has re-measured static-vs-dynamic archive bias at scale.** The 73.9%/95.3% tracker gap {[hantke2023you]} rests on 2,026 sites, limited by rate-limiting. Zhu et al.'s {[zhu2025_toward]} 45.8% fidelity gap is measured on their own crawls, not on the Internet Archive's holdings. A large-scale measurement of what the Internet Archive itself is missing, per resource type, does not exist. 
-  * <wrap todo>**Archive-based measurement of regional phenomena has no instrument.** Both major sources crawl from the US. For GDPR-era consent, geo-blocking or regional advertising, there is no archive with an EU vantage point at usable coverage — the national archives fail the freshness test.</wrap>+  * **Monoculture risk is unquantified.** Roughly two thirds of archive-using papers depend on one archive, one crawler and one vantage point. What that shared bias does to the field's longitudinal results has not been studied — and Common Crawl, the only comparison Hantke et al. could make, was rejected on coverage rather than validated on agreement. 
 +  * **Archive-based measurement of regional phenomena has no instrument.** Both major sources crawl from the US. For GDPR-era consent, geo-blocking or regional advertising, there is no archive with an EU vantage point at usable coverage — the national archives fail the freshness test. 
 +</WRAP>
  
 ===== Related Pages ===== ===== Related Pages =====
design/archives.1786661095.txt.gz · Last modified: by karel.kubicek.claude