Differences

This shows you the differences between two versions of the page.

--- design:archives [2026/08/21 14:50] – Boxes: <wrap> renders a span, use uppercase <WRAP>; one box per list, not per bullet karel.kubicek.claude
+++ design:archives [2026/09/04 17:29] (current) – Citekey consolidation 2026-09-04: repoint duplicate bibliography keys (fouad2022my/boettger2025_regional/ahmad2026_ipfp/bouhoula2024automated/lerner2016internet) to the kept key; no prose or figure changed. Authored by Claude karel.kubicek.claude
@@ Line 3: / Line 3: @@
 A web archive lets you measure a page you cannot visit — because the date has passed, because the site is gone, or because you want everyone else to be able to measure exactly the same bytes. That is a genuinely different instrument from a crawler, and it fails in genuinely different ways. This page is about **what an archive can and cannot answer**, not about how archiving works.
-Researchers have used archives for longitudinal analyses of privacy {[lerner2016internet,jha2024privacy,amos2021privacy,dimova2021cname,singh2026_empire]} and of security features {[pletinckx2021out,roth2020complex,stock2017web]}. The attraction is reproducibility: a snapshot is static, so a later reader can fetch the same one, and nobody has to write another crawler. The cost is that a snapshot is a **partial recording made by someone else's crawler, from their vantage point, at a moment they chose**, and almost every pitfall below follows from that one sentence.
+Researchers have used archives for longitudinal analyses of privacy {[lerner2016_internet,jha2024privacy,amos2021privacy,dimova2021cname,singh2026_empire]} and of security features {[pletinckx2021out,roth2020complex,stock2017web]}. The attraction is reproducibility: a snapshot is static, so a later reader can fetch the same one, and nobody has to write another crawler. The cost is that a snapshot is a **partial recording made by someone else's crawler, from their vantage point, at a moment they chose**, and almost every pitfall below follows from that one sentence.
 Two numbers frame everything that follows. In our corpus of seven security and privacy venues (see [[#Use in Publications]]), **115 of 5,118 empirical papers (2.2%) name a web archive at all**, against 1,236 (24.2%) that ran a live crawl. Of the papers that do, **only about a quarter collected nothing of their own** — 58.2% ran a live crawl as well. The archive is normally a supplement, not the instrument.
@@ Line 16: / Line 16: @@
   - {[hantke2023you]} — the systematic study. Compares seven archives, quantifies what goes wrong in the Internet Archive, and ends with a best-practice list. If you read one, read this.
-  - {[lerner2016internet]} — the founding archive-based measurement in this literature (tracking, 1996–2016), and still the only paper that measured how often replay escapes to the live web.
+  - {[lerner2016_internet]} — the founding archive-based measurement in this literature (tracking, 1996–2016), and still the only paper that measured how often replay escapes to the live web.
   - {[zhu2025_toward]} — why a static crawl misses half the page, measured in 2025, from the archive operators' side of the problem.
   - {[singh2026_empire]} — the model recent application: a 2026 longitudinal study that states its snapshot rule, its rate limiting and its handling of classifier anachronism, all of which you can copy.
@@ Line 58: / Line 58: @@
 ==== Escapes: when an archived page measures today ====
-URL rewriting inside a replayed page is not perfect, chiefly because it cannot rewrite URLs that JavaScript builds at runtime. When it fails, the replayed page issues a request to the **live** web. Lerner et al. measured this on archived snapshots of the Alexa top 500 and found that **16.1% of all requests attempted to escape the archive** — 17.2% of unique URLs and 11.5% of domains {[lerner2016internet]}. That measurement is from 2016, on 2015-era replay, and **nobody has repeated it since**; treat it as evidence that escapes matter, not as today's rate.
+URL rewriting inside a replayed page is not perfect, chiefly because it cannot rewrite URLs that JavaScript builds at runtime. When it fails, the replayed page issues a request to the **live** web. Lerner et al. measured this on archived snapshots of the Alexa top 500 and found that **16.1% of all requests attempted to escape the archive** — 17.2% of unique URLs and 11.5% of domains {[lerner2016_internet]}. That measurement is from 2016, on 2015-era replay, and **nobody has repeated it since**; treat it as evidence that escapes matter, not as today's rate.
 To put that in proportion: in the same table, robots.txt exclusion accounted for 2.0% of requests and the resource simply not being archived for 1.4%. **Escapes were by far the largest single failure mode — eight times robots exclusion.** Their crawler blocked those requests, which is the right default: an escaped request measures the present under a past timestamp. If your tooling does not block them, your "2014 measurement" silently contains 2026 responses. Blocking is one rule in whatever drives your browser — abort any request whose URL is not under ''web.archive.org'' — and **count what you blocked**, because that count is a fidelity statistic worth reporting. The same mechanism is a security problem, not only a measurement one — Lerner et al. later showed that an attacker who controls an escape destination can rewrite what a client sees in the archive, and that **73% of Top 500 sites and 80% of Top Million domains** had snapshots vulnerable to at least one such attack {[lerner2017_rewriting]}.