Working log behind Biases. Every query with its denominator, the report script and its unedited output, the folds and their full residue, the probes that were run and the ones that were rejected, the quotes checked, the external sources verified and rejected, and what could not be established.

Corpus-level caveats — the venue scope, the selection funnel, the provisional years, the duplicate records — are on Corpus and are not restated here. Citation keys are shared with the content page and the single Bibliography; this page adds no bibliography entries of its own.

Voice here is a working log, not prose. It is for someone checking a number.

Three neighbours overlap this page and each was considered as the host instead.

Deliberate division of labour. These belong to the neighbours and are linked, not restated: the list catalogue and their provenance (Website selection); the per-list spelling undercount and the sampler that records its own provenance (Sampling); choosing and verifying a vantage point, and the EEA/GDPR trend (Crawling location); non-independence and clustering (Hypothesis testing); fixing the exclusion rule in advance (Study preregistration).

Two figures are re-derived that a neighbour also reports, on purpose:

• The Alexa→Tranco transition. Sampling computes it on its own 1,153-paper web-unit population; this page computes it on the 793 papers using any ranking list, by an independent regex fold. The two land within 3% on Alexa (463 there, 475 here) and that is the only external check either page has on its fold — see the fold section for why “within 3%” is the honest phrasing and “agree” was not.
• Vantage-point location shares. Crawling location reports 182 US papers of 296 stating on the crawled population; this page reports 639 of 1,135 on measuredFrom. Different populations, same shape. Stated as such on both.

One thing the page deliberately does not attempt. It does not rank the four biases by size. The effect sizes come from different papers with different designs and denominators, and a ranking would imply a comparison that was never run.

Every query names its own population. There is no page-wide denominator, which is the point of the page.

population    definition                                                 papers
------------  ---------------------------------------------------------  ------
all           every extracted paper in the corpus                        5859
sampled       has >=1 population tuple (drew a study population)         5712
crawled       crawlConfig != null OR studyTypes has automated-web-crawl  1120
webCrawled    crawled AND platforms includes web                         857
measuredFrom  has >=1 vantage tuple                                      3908
repeated      crawled AND some temporal.snapshots > 1                    250

crawled, measuredFrom and webCrawled come from scripts/lib.mjs unchanged and match the counts published in data/extract/OVERVIEW.md. Two are defined in biases_report.mjs itself:

const sampled  = rows.filter((p) => p.population.length > 0);
const repeated = crawled.filter((p) =>
  p.temporal.some((t) => typeof t.snapshots === 'number' && t.snapshots > 1));

repeated is a floor, not a count of longitudinal work. Only 409 of the 1,120 crawling papers (36.5%) state a snapshot count at all, so a paper that crawled monthly for a year and never wrote the number down is not in it. This is said on the page.

repeated is also too narrow in a way that was caught by accident. It requires crawled, and the extraction classifies [1Efstratiou, Alexandros (2025): "On YouTube Search API Use in Research", in: Proceedings of the ACM Internet Measurement Conference. (DOI)] — a paper with a section headed 4.3 Attrition, sixteen snapshots, and the best treatment of query-sampler attrition in the corpus — as network-scan-or-probe with crawlConfig: null. So the paper most relevant to this page's survivorship section is not in the population the survivorship probe reads. The report therefore prints the probe a second time over every paper with more than one snapshot regardless of study type (817 readable), and the rate is the same, 0.7%. Both numbers are on the page.

population.sourceList is free text. It is folded to an alphanumeric skeleton by skeleton() in lib.mjs and then matched against nine unanchored regular expressions:

const RANKING_LISTS = [
  ['Tranco', /tranco/],
  ['Alexa', /alexa/],
  ['Cisco Umbrella', /umbrella|opendns/],
  ['Majestic', /majestic/],
  ['Chrome UX Report (CrUX)', /crux|chromeuxreport|chromeuserexperience/],
  ['Quantcast', /quantcast/],
  ['SimilarWeb', /similarweb/],
  ['Common Crawl', /commoncrawl/],
  ['Statvoo / Domcop / other rank vendor', /statvoo|domcop|ahrefs|semrush|builtwith/],
];

Two bugs in the first version of this fold, both found in review, both of which moved published figures. They are recorded in detail because either one alone would have produced a table that looks authoritative and is wrong.

A third bug, produced by the fix for the second, and found on re-review. Unanchoring /alexa/ re-exposed the substring problem the anchors had been suppressing, and the first exclusion list was built only from the strings the author had looked at. The re-review found two it missed:

Fixed with an assistant-ecosystem keyword list plus a negative lookahead for the surname, because alexander is a prefix collision rather than a topic. Final figures: 793 rank-list papers, 479 of webCrawled, Alexa 475. The lesson is that the exclusion list is only as good as the strings someone looked at, so all 16 are printed by the report, and anyone unanchoring another pattern should expect the same class of surprise.

Why anchoring was there in the first place, and what replaced it. “Alexa” is also Amazon's voice assistant, and the corpus contains skill-store populations. Unanchoring /alexa/ therefore needs an explicit exclusion instead:

const ALEXA_IS_THE_LIST = /alexa(?!nder)/;               // surname, not a list
const ALEXA_NOT_A_LIST =
  /skill|marketplace|voiceassistant|echodot|amazonecho|\balexastore|smartspeaker
   |googleassistant|smartthings|homekit|openhab|homeassistant|ifttt|siri|googlehome/;

All 16 excluded strings, printed by the report so the exclusion is auditable:

  2x  US Alexa skill store
  1x  third-party Alexa skill-name aggregation database
  1x  Amazon Alexa skill market
  1x  Alexa skill market
  1x  Amazon Alexa Skill Store
  1x  Amazon Alexa skills store
  1x  Alexa skill stores: US, UK, AU, CA, DE, JP, FR
  1x  Alexa, Google Assistant, IFTTT, SmartThings, Apple HomeKit, OpenHAB, and HomeAssistant integration lists
  1x  Amazon Alexa marketplace
  1x  Amazon Alexa marketplace, US market
  1x  Alexa skills with complete traceability
  1x  Alexander Mylnikov
  1x  Alexa skill marketplace
  1x  Alexa skill marketplaces
  1x  privacy-policy pairs across Alexa marketplaces
  1x  custom extracted email list from Amazon Alexa skill descriptions

Every one is the voice assistant. None is a ranking-list population. The remaining unanchored patterns were checked the same way: /tranco/, /majestic/, /quantcast/ and /crux/ produced no non-list matches in the corpus at all.

This is still a matcher, not a synonym fold. A list named in a way none of the nine catch is counted as not a ranking list, so 793 remains a lower bound and every share against it is conservative in the same direction.

Residue: 10,011 distinct unmatched sourceList skeletons. They are not ranking lists — the head is custom seed list (516 papers), Google Play (143), Prolific (122), Google Play Store (83), GitHub (73), Amazon Mechanical Turk (68), AndroZoo (54), CIFAR-10 (52), RIPE Atlas (48), VirusTotal (39). The report prints the top 25 and the total. Printing all 10,011 was considered and rejected: they are the corpus's entire population vocabulary, most of it about apps, datasets and participant pools, and none of it about web popularity rankings.

Cross-check against the independent fold on Sampling. That page's sample_fold.mjs reaches 463 Alexa papers and 262 Tranco on its own 1,153-paper web-unit population; this page reaches 475 and 267 on a different population by a different method.

And this is where the cross-check has to be described honestly, because it is the weakest thing on the page. It “held” through all three bug fixes — the buggy version reported 464 / 253, which is closer to Sampling's 463 / 262 than the corrected 475 / 267 is. A cross-check that is satisfied equally by the right answer and by three bugs is not a check on correctness; it only bounds how far apart two methods can be. All three bugs were found by reading the code and by a reviewer reading the corpus, not by the cross-check. The page therefore says the two folds “land within 3% of each other” and does not say they “agree”.

vantage.locations is folded through scripts/geo.mjs (normalizeLocation), reused unchanged from Crawling location. It maps a string to country, region, multi or unmapped.

tuple-level fold kinds: [["country",2405],["unmapped",403],["region",331],["multi",203]]
403 of 3342 location strings unmapped (12.1%), 354 distinct

Why the exact-string figure is on the page. Folded, 639 papers name the United States. Counting the exact string United States gives 238 — a 62.8% undercount. The task brief records this measured at 280 → 498 on 2026-08-05 via site_queries.mjs –page vantage; the numbers here differ because the population is measuredFrom (3,908 papers) rather than the crawling papers, and because geo.mjs folds more aggressively than name_fold.mjs. The ratio is the claim, and it survived the change of population.

The full 354-string residue is in the report output, reproduced verbatim below, and it is worth reading rather than skimming: it is not noise. It contains real countries the fold lacks (Gambia, Libya, Niger, Zambia, Sudan, Lesotho, Dominican Republic, Jamaica, Trinidad and Tobago), US state abbreviations (MA, NH, RI, CT, AZ, MD, NC, NV), cities (Hangzhou, Guangzhou, Belo Horizonte, Tallinn, Moscow), cloud region codes (ap-southeast-2, westus, FRA, SYD), and — the largest group — indoor micro-locations from wireless and IoT papers (anechoic chamber, balcony, underground parking lot, football field, coffee shop). That last group is the honest reason the residue is 12% rather than 2%: measuredFrom includes papers whose “vantage point” is a room, and a country fold cannot map a room. The page's 71.0%-silence figure is not affected — those papers are in the “no mappable location” row either way — but a reader building a country table from this field should exclude non-web platforms first.

Fixing the residue was considered and rejected for this run: adding the missing countries would move the United States figure by 0 and the tail by single papers, and adding the micro-locations would require a second fold dimension (indoor/outdoor) that no page needs. It is left visible instead, which is the rule.

Both probes read data/fulltext/<year>/<venue>/<slug>/paper.cols.txt, collapse all whitespace to single spaces, and lower-case — because a PDF line break inside a phrase silently undercounts an uncollapsed match.

Seventeen phrases over the 250 repeated papers. All 250 had a readable paper.cols.txt (0 missing).

61 papers (24.4%) contain at least one attrition phrase
unreachable 19 | churn 19 | no longer available 12 | failed to load 9 | still alive 4
could not be reached 3 | went offline 3 | survivorship 2 | attrition 0 | survivor bias 0
dropped out of 0 | no longer resolve 0 | no longer resolved 0 | no longer reachable 0
ceased to exist 0 | sites that disappeared 0 | domains that disappeared 0

The named-bias subset (attrition, survivorship, survivor bias, survivorship bias) across four populations:

Why the four rows. One row would be a curiosity; four rows that agree at 0.7% are a finding about the field's vocabulary rather than about one population definition. attrition returning exactly 0 in 250 papers was treated as suspicious and checked independently: grep -rli attrition over the whole data/fulltext tree returns 22 paths, none of which is in the repeated set. (That grep is only a sanity check, not a figure — the workdir notes record that grep -rl silently skips papers whose paper.cols.txt is binary-detected, which is why every published figure comes from the streaming Node probe instead.)

What the probe cannot see. It finds phrases, not concepts. A paper that writes “we successfully loaded 91,204 of 100,000 sites” and never uses the vocabulary is a miss. So 24.4% is an upper bound on papers that say anything, and 0.8% is close to the true rate for papers that name the bias — the page says both and does not present either as a measure of good practice. Turning the bound into an estimate would need hand-coding; it is listed as an open question on the page.

A heading, not the word — matched against a whole line, so a paper that mentions limitations in prose does not count:

const HEAD = /^\s*(?:[0-9]+(?:\.[0-9]+)*\.?\s+|[IVX]+\.\s+|[A-Z]\.\s+)?
              (?:limitations?|threats? to validity)(?:\s+and\s+[a-z ]{1,30})?\s*$/i;

Run on the raw (not whitespace-collapsed) line array, because a heading is a line. Every figure is a lower bound: a paper that discusses limitations without a heading, or whose heading is Discussion and Caveats, counts as a miss. That is stated on the page next to the table.

Rejected variants, each because it inflated the count with false positives on inspection: matching limitation anywhere in a line (hits every “a limitation of this approach” sentence and every related-work summary); matching discussion (a discussion section is not a limitations section); dropping the end-of-line anchor (hits table-of-contents lines and running headers).

scripts/biases_quotecheck.mjs checks the 32 quotes and per-paper figures the page publishes, against three invented-sentence negative controls.

The exact-match problem, and how it was solved. decolumn.mjs can emit a figure caption or a running header in the middle of a sentence that is contiguous in the PDF, and the insertion often lands mid-clause. An exact substring match therefore fails on quotes that are genuinely verbatim. Three approaches were tried:

The adopted rule: cut the quote into overlapping 3-word shingles, look each up, and take the maximal runs of consecutive missing shingles. One inserted span costs k−1 = 2 consecutive shingles, or k = 3 if it also splits a hyphenated word (post-2020 → post … 2020 loses three, not two). A quote passes iff every missing run is ≤ 3 long and there are ≤ 5 runs. A paraphrase fails, because its missing shingles come in long or scattered runs — the three invented-sentence controls score runs of [1,1,12], [12,3] and [1,8,4].

The bound of 5 runs is set by the worst real case, and the bound was raised after seeing it, which is worth saying out loud because it is how a check gets fitted to its data. Hils et al.'s methods sentence is spliced four times by the decolumner, runs [2,2,3,2], and failed at ≤ 3 runs. The sentence had already been read in the PDF and is verbatim, so the choice was between dropping a correct quote and loosening the bound. The bound was loosened, and the three controls were re-run at the looser bound and still fail by a wide margin (their worst runs are 12, 12 and 8, against a cap of 3). A reader who thinks that is too generous should tighten it to ≤ 3 runs and expect exactly one failure, hils2021_privacy.

Every quote was also read in context by hand, which is how the one real problem was found and resolved:

Unedited output:

PASS  scheitle2018_long   26 shingles  [paper.cols.txt:100% runs=[-] paper.txt:77% runs=[3,2,1]]  "we find the amount of NX-DOMAIN responses in both the Umbrella (11…"
PASS  scheitle2018_long    7 shingles  [paper.cols.txt:100% runs=[-] paper.txt:100% runs=[-]]  "all three only on 99k out of 1M domains."
PASS  scheitle2018_long   29 shingles  [paper.cols.txt:100% runs=[-] paper.txt:100% runs=[-]]  "all Top 1M lists exceeding the general population by at least a fa…"
PASS  scheitle2018_long   17 shingles  [paper.cols.txt:100% runs=[-] paper.txt:94% runs=[1]]  "At 11-13%, we find IPv6 enablement across top lists to significant…"
PASS  scheitle2018_long    7 shingles  [paper.cols.txt:100% runs=[-] paper.txt:100% runs=[-]]  "revealing daily churn of up to 50% of domains."
PASS  scheitle2018_long   10 shingles  [paper.cols.txt:80% runs=[2] paper.txt:70% runs=[2,1]]  "10k probes at 1 query per day ... achieve a rank of 38k"
PASS  ruth2022_toppling   12 shingles  [paper.cols.txt:83% runs=[2] paper.txt:83% runs=[2]]  "Top lists exhibit noticeable and irregular geographic biases ... a…"
PASS  ruth2022_toppling    7 shingles  [paper.cols.txt:86% runs=[1] paper.txt:86% runs=[1]]  "is the most accurate top list compared to Cloud"
PASS  ruth2022_toppling    1 shingles  [paper.cols.txt:100% runs=[-] paper.txt:100% runs=[-]]  "across all metrics"
PASS  ruth2022_toppling   16 shingles  [paper.cols.txt:100% runs=[-] paper.txt:88% runs=[2]]  "of the 1,790 domains we measure in the Alexa top 10K, 70% of them …"
PASS  ruth2022_toppling   15 shingles  [paper.cols.txt:100% runs=[-] paper.txt:93% runs=[1]]  "our analysis reveals certain categories of websites (e.g., adult a…"
PASS  ruth2022_toppling   10 shingles  [paper.cols.txt:100% runs=[-] paper.txt:70% runs=[3]]  "Top lists better approximate client behavior on desktop platforms …"
PASS  hils2021_privacy   30 shingles  [paper.cols.txt:70% runs=[2,2,3,2] paper.txt:70% runs=[2,2,3,2]]  "in order to avoid survivorship bias in our observations. Picking a…"
PASS  singh2017_torexit   35 shingles  [paper.cols.txt:100% runs=[-] paper.txt:74% runs=[2,3,3,1]]  "several of the selected exit relays intermittently went offline, w…"
PASS  jueckstock2021_realistic   13 shingles  [paper.cols.txt:77% runs=[2,1] paper.txt:77% runs=[2,1]]  "Around 5% of content-providing domains show significant measuremen…"
PASS  jueckstock2021_realistic   14 shingles  [paper.cols.txt:79% runs=[1,2] paper.txt:79% runs=[1,2]]  "sites that always failed to load from a particular vantage point (…"
PASS  jueckstock2021_realistic    2 shingles  [paper.cols.txt:100% runs=[-] paper.txt:100% runs=[-]]  "nearly 20% of domains"
PASS  jueckstock2021_realistic    4 shingles  [paper.cols.txt:50% runs=[2] paper.txt:50% runs=[2]]  "strongly correlated to choice of BC."
PASS  jueckstock2021_realistic   10 shingles  [paper.cols.txt:100% runs=[-] paper.txt:100% runs=[-]]  "over 7% of domains and over 5% of total HTTP traffic volume"
PASS  zeber2020representativeness   32 shingles  [paper.cols.txt:100% runs=[-] paper.txt:100% runs=[-]]  "we find that the average similarity in third parties is low, with …"
PASS  zeber2020representativeness   21 shingles  [paper.cols.txt:100% runs=[-] paper.txt:81% runs=[2,2]]  "crawler site visits issued requests to a median of 11.6 third-part…"
PASS  annamalai2025_beyond   17 shingles  [paper.cols.txt:100% runs=[-] paper.txt:82% runs=[2,1]]  "the real user browsing sessions detected 471 such fingerprinting w…"
PASS  urban2020beyond   30 shingles  [paper.cols.txt:100% runs=[-] paper.txt:83% runs=[3,2]]  "subsites set considerably more (36 %) cookies than the respective …"
PASS  urban2020beyond   16 shingles  [paper.cols.txt:100% runs=[-] paper.txt:88% runs=[2]]  "of the measured websites do not embed any trackers on the landing …"
PASS  demir2022_reproducibility   29 shingles  [paper.cols.txt:93% runs=[2] paper.txt:93% runs=[2]]  "profiles from the USA are tracked most in terms of distinct tracke…"
PASS  demir2023_similarity   13 shingles  [paper.cols.txt:85% runs=[2] paper.txt:85% runs=[2]]  "The results show that when comparing two different profiles, 48% o…"
PASS  demir2023_similarity   13 shingles  [paper.cols.txt:100% runs=[-] paper.txt:92% runs=[1]]  "only 32% of the cookies appear in all profiles and 42% only in one…"
PASS  efstratiou2025_youtube    9 shingles  [paper.cols.txt:89% runs=[1] paper.txt:89% runs=[1]]  "The results suggest that drop-ins and drop-outs are the normative …"
PASS  efstratiou2025_youtube    5 shingles  [paper.cols.txt:100% runs=[-] paper.txt:100% runs=[-]]  "tends to return shorter, more popular videos"
PASS  efstratiou2025_youtube   29 shingles  [paper.cols.txt:93% runs=[2] paper.txt:93% runs=[2]]  "API samples videos from empirical distributions, returning results…"
PASS  singh2025_where   12 shingles  [paper.cols.txt:100% runs=[-] paper.txt:83% runs=[2]]  "websites in 91% of the examined countries (21/23) embed trackers h…"
PASS  lerner2016_internet    4 shingles  [paper.cols.txt:100% runs=[-] paper.txt:100% runs=[-]]  "16.1% of all requests attempted to"
PASS  CONTROL invented (ruth)   16 shingles  [paper.cols.txt:13% runs=[1,1,12] BAD paper.txt:6% runs=[3,12] BAD]  "we conclude that the Tranco list is the single most accurate ranki…"
PASS  CONTROL invented (jueckstock)   17 shingles  [paper.cols.txt:12% runs=[12,3] BAD paper.txt:12% runs=[12,3] BAD]  "we found that the choice of vantage point had no measurable effect…"
PASS  CONTROL invented (efstratiou)   15 shingles  [paper.cols.txt:13% runs=[1,8,4] BAD paper.txt:13% runs=[1,8,4] BAD]  "the YouTube Search API returned a stable and reproducible set of v…"

35/35 checks passed (32 quotes must locate, 3 controls must not); 0 failed.
Rule: a quote locates iff every run of missing 3-shingles is <=3 long and there are <=5 runs.

Note how often paper.txt fails where paper.cols.txt passes. That is the two-column reading-order repair working, and it is the reason the brief says to read .cols.

node scripts/biases_report.mjs    > scripts/biases_report-output.txt
node scripts/biases_quotecheck.mjs > scripts/biases_quotecheck-output.txt
cat scripts/biases_report-output.txt scripts/biases_quotecheck-output.txt > out/biases_all.txt
node scripts/check_page_numbers.mjs pages/statistics_biases.txt out/biases_all.txt
# -> OK — every figure in the page traces to out/biases_all.txt

Run whole-page, with no section markers, because the page repeats corpus figures in its opening box, its bias sections, its currency table and its worked example — all outside any “Use in Publications” window. The workdir notes record 29 stale figures surviving a refresh because the check was windowed.

What the guard does and does not prove. It proves every number on the page appears somewhere in the report output. It does not prove a number is the right one for the sentence it sits in — and that gap bit this page. After the fold fix moved 15 figures, three stale ones survived in the “Methodology and limitations” bullets (773 is a floor, 463 against 464 here, 25 evidence quotes) and the guard passed, because all three numbers appear in the report for unrelated reasons: 773 is the tail of the page's own “2,773”, 463 is the cross-check figure from Sampling, and 25 is a row count. They were caught by re-reading the section by hand. Anyone refreshing this page after a fold change must re-read the prose, not just re-run the guard.

The guard's first run reported 23 unaccounted figures, every one of them a legitimate non-corpus number. Rather than adding 23 entries to the shared ALLOW map in check_page_numbers.mjs, biases_report.mjs gained a Section Z that prints every such number with its source: per-paper figures with the paper's key, figures taken from a neighbouring wiki page, corpus facts documented in data/extract/README.md, the external dates above, and the three illustrative numbers in worked examples that are not measurements at all. Section Z is the audit trail for the digits the report does not compute, and it is why the guard now passes with no shared-state edit.

scripts/biases_report.mjs, 763 lines of output. Reproduce with:

node scripts/biases_report.mjs        # ~20 s, reads 6,105 full texts across two probes
node scripts/biases_quotecheck.mjs    # ~5 s, 35 checks

The output is committed at scripts/biases_report-output.txt and scripts/biases_quotecheck-output.txt. It contains, in order: the population table; the ranking-list fold with its 25-row residue head; the Alexa/Tranco period table; samplingMethod over sampled with the sentinel counted separately; top-n against stratified over webCrawled; list-version reporting over three populations; the repeated population and snapshot distribution; probe 1 in full including the four-population named-bias table; probe 2 with its per-period table; the location fold with all 354 unmapped strings printed; single-vantage exposure; vantage.infrastructure; population.unit over webCrawled; sampling-method exposure for keyword-derived populations; the detection-prevalence denominator-risk block; the per-venue and per-year corpus tables; and Section Z.

• Which sites drop out of a crawl, and how. No paper in this corpus characterises the composition of the set that fails to load. Everything the page says about the direction of survivorship bias is reasoning from mechanism, not a measurement, and is worded as such. Sampling reached the same conclusion independently.
• How many repeated-crawl papers report attrition numerically without the vocabulary. The probe cannot see it. This is the single figure that would turn the page's 0.8% from a bound into an estimate.
• Whether LLM-based classification carries its own selection bias in this literature. LLM classification appears in the 2025–2026 slice; a bias characterisation of it does not. Stated as an open question rather than speculated about.
• A defensible ranking of the four biases by magnitude. The effect sizes come from incomparable designs. Not attempted.
• Denominator risk, quantified. The report prints that 4,390 of 5,655 papers with a detection tuple (77.6%) state at least one prevalence containing a %, and that 45.9% of the 26,316 stated prevalences do. This was not put on the page as a figure: metric and prevalence being separate fields makes it possible for them to disagree, but establishing that they do disagree in a given paper needs the paper's own denominator read from paper.cols.txt, which was not done at scale. The block stays in the report as a pointer for a future run.

Both were run after saving, and both found something.

1. In-page anchors — one was broken. The page's own section links rely on DokuWiki's section-ID cleaning, which strips punctuation, colons and a leading 1.. Extracting every id= from the rendered DOM and every href=“#…” and differencing them found one broken link out of 62: [[#Worked Example]] does not resolve, because the heading is Worked Example: The Biases of This Site's Own Corpus and DokuWiki's id for it is worked_examplethe_biases_of_this_site_s_own_corpus. A partial heading text is not a valid anchor, even though a truncated one reads fine in the source. The digit-prefixed headings ([[#1. Selection Bias: Your Sample Is a Ranking List]]) do resolve, because the link side and the heading side clean identically. Fixed by linking the cleaned id with a display label.

# the check, worth re-running after any heading edit
curl -s "https://measuretheweb.org/doku.php?id=statistics:biases" |
python3 -c 'import re,sys; h=sys.stdin.read();
ids=set(re.findall(r"<h[1-6][^>]*\bid=\"([^\"]+)\"",h));
links=set(re.findall(r"href=\"#([^\"]+)\"",h));
print([l for l in links if l not in ids and not l.startswith(("fn","ref__","dokuwiki__","discussion__"))])'
# -> [] after the fix

2. Citations needed a cache purge. On first load the content page rendered 16 of 18 references: the markers [9] and [14] were missing, and those are exactly the two entries added to Bibliography in this run. The bibtex4dw plugin caches the parsed bibliography, so a page saved in the same sitting as a new entry renders as if the entry did not exist — with no error and no unresolved marker to notice. Fetching the bibliography and then the page with &purge=true fixed it; both pages then rendered every reference.

curl -s ".../doku.php?id=literature:bibliography&purge=true" > /dev/null
curl -s ".../doku.php?id=statistics:biases&purge=true"      > /dev/null
# then count rendered <dt>[n]</dt> against the page's distinct {[key]} count

Do not check this by looking for a broken citation. There isn't one. The only symptom is a reference list shorter than the number of distinct keys, which is why the check is a count and not a read.

3. Rendered structure, both pages. statistics:biases: 18 references, 11 h2, 25 h3, 17 tables, 4 WRAP boxes, 3 footnotes. provenance:statistics:biases: 1 reference, 17 h2, 14 h3, 17 tables, 11 pre blocks, 0 footnotes. No unclosed block, no literal wiki markup in the output, no truncated page.

• Two duplicate citekey pairs exist in Bibliography and were noticed while checking this page's keys: lerner2016_internet / lerner2016internet and boettger2025_regional / bottger2025_regional (also bouhoula2024_automated / bouhoula2024automated). This page uses lerner2016_internet. Not fixed here — deduplicating keys touches every page that cites them and is its own item.
• repeated should probably not require crawled. See the efstratiou2025_youtube case above. Any future page about longitudinal measurement should define it on temporal.snapshots alone and say so.
• population.unit puts 41.4% of web-crawling papers in other. Any unit-based figure from this field, on any page, should print that share next to it.

Three focused reviewers in parallel on a frozen snapshot, then fixes, then a generic pass. All were told explicitly that the author's context may not be exhaustive, and all were handed the page text, both scripts, their unedited output and these notes.

Run first, because the sibling logs record that reviewers do not check rhetorical sentences. It ran twice — once before the reviewers and once after their fixes were in — and found sixteen things in prose and in the methodology bullets. Twelve were outright wrong, including two self-contradictions and two claims stated backwards. None of the five focused reviewer passes flagged any of them.

Re-review after the fixes (same reviewer, its context intact, new snapshot): all citekeys still resolve, no new duplicates, C2/C4/C5 confirmed fixed, and it independently verified two things the author had changed on his own initiative — the VP/BC re-attribution (it reconstructed the spliced sentence and confirmed 5% is the VP axis while “over 7% of domains and over 5% of total HTTP traffic volume” and “nearly 20% of domains' traffic … strongly correlated to choice of BC” are both the BC axis) and the bozzolan2026_llmweb abstract quote and both claims drawn from it. One remaining finding: a third efstratiou2025_youtube quote (“tends to return shorter, more popular videos”) was outside the check's list — the same class of gap as C4 and C5. It had in fact been added between the reviewer's snapshot and its report, so it was already closed; the list stands at 31 quotes and the reviewer's point about relying on “verified by tooling” rather than “verified by hand” is the right one to record.

C4 and C5 together are the most useful thing the review layer produced. Both quotes were outside the quotecheck's list, and both were wrong in exactly the way the quotecheck exists to prevent. The lesson is procedural: the check is only as good as the completeness of its input list, and the list was assembled by hand from the extraction's evidence.quote values — so the two quotes the author typed from a paper rather than lifting from the extraction were the two that were unchecked and wrong. Nine quotes have since been added; the list now stands at 32.

• Biases — the page these notes are for.
• Corpus — corpus-wide provenance: venues, funnel, provisional years, duplicates.
• sampling and hypothesis_testing — the sibling logs whose populations this page's figures were cross-checked against.

[1]

Efstratiou, Alexandros (2025): "On YouTube Search API Use in Research", in: Proceedings of the ACM Internet Measurement Conference. (DOI)