| Next revision | Previous revision |
| provenance:design:blocking_and_geodifference [2026/09/10 23:47] – Working log for design:blocking_and_geodifference: the inclusion rule and its amendment, nine probes, 275 hand verdicts, the censor and location folds with residue, 94 quote checks, external and rendered-dashboard checks, five review passes with dispositi karel.kubicek.claude | provenance:design:blocking_and_geodifference [2026/09/10 23:50] (current) – The sentence describing the namespace-relative-link bug contained a live example of it and rendered a red link; escape it with %% and say so. Authored by Claude. karel.kubicek.claude |
|---|
| ====== Provenance: Blocking and Geodifference ====== | ====== Provenance: Blocking and Geodifference ====== |
| |
| Working log for [[design:blocking_and_geodifference]]. Every figure on that page, the query that produced it, the population it is a share of, the folds and their residue, the quotes that were checked against the source PDFs, the external sources that were fetched and the ones that were rejected, and the judgement calls — including the ones a reasonable person would have made differently. | Working log for [[:design:blocking_and_geodifference]]. Every figure on that page, the query that produced it, the population it is a share of, the folds and their residue, the quotes that were checked against the source PDFs, the external sources that were fetched and the ones that were rejected, and the judgement calls — including the ones a reasonable person would have made differently. |
| |
| Corpus-level caveats — venue scope, the selection funnel, the provisional 2025–2026 years, extraction stability — are on [[Literature:Corpus]] and are not restated here. | Corpus-level caveats — venue scope, the selection funnel, the provisional 2025–2026 years, extraction stability — are on [[:Literature:Corpus]] and are not restated here. |
| |
| **Run:** 2026-09-10. Corpus at the time: ''data/extract/run1/extractions.jsonl'', **5,859** extracted papers, 7 venues (CCS, IMC, NDSS, PoPETs, USENIX Security, TheWebConf, IEEE S&P), 2010–2026. Authored by Claude (Opus 5) with five review passes (see §12). No part of this page was carried over from an earlier run: this page did not exist before, and the queued estimate that scoped it (74 candidate papers) was re-derived rather than reused — see §3. | **Run:** 2026-09-10. Corpus at the time: ''data/extract/run1/extractions.jsonl'', **5,859** extracted papers, 7 venues (CCS, IMC, NDSS, PoPETs, USENIX Security, TheWebConf, IEEE S&P), 2010–2026. Authored by Claude (Opus 5) with five review passes (see §12). No part of this page was carried over from an earlier run: this page did not exist before, and the queued estimate that scoped it (74 candidate papers) was re-derived rather than reused — see §3. |
| * **Probe 8 came from reading the shared bibliography's newest entry.** ''holzbauer2025_tracking'', //Tracking Internet Disruptions in Ukraine// (IMC 2025), was in ''literature:bibliography'' already, is squarely about whether clients could reach the Internet, and **no probe caught it** — while a draft of the page asserted that shutdowns as an event class were absent from the corpus. They are not: there are **12** outage-detection papers, and what they lack is not measurement but **attribution**. The distinction became an adjacent family (§4) and the page's claim was rewritten around it. This is the finding to remember: a candidate set built from one vocabulary cannot see a literature that uses another, and the counter-example was already on the wiki. | * **Probe 8 came from reading the shared bibliography's newest entry.** ''holzbauer2025_tracking'', //Tracking Internet Disruptions in Ukraine// (IMC 2025), was in ''literature:bibliography'' already, is squarely about whether clients could reach the Internet, and **no probe caught it** — while a draft of the page asserted that shutdowns as an event class were absent from the corpus. They are not: there are **12** outage-detection papers, and what they lack is not measurement but **attribution**. The distinction became an adjacent family (§4) and the page's claim was rewritten around it. This is the finding to remember: a candidate set built from one vocabulary cannot see a literature that uses another, and the counter-example was already on the wiki. |
| |
| **The queued estimate was wrong in both directions and is not comparable.** [[roadmap]] queued this page on a title-and-summary probe that matched 74 papers, 38 of them on the ''web'' platform, 30 in 2020–2023 and 22 in 2024–2026. **That probe is not in ''scripts/gap_probe_roadmap.mjs''** — it came from the 2026-09-02 brainstorm pass and only its alternation is recorded in prose — so it cannot be re-run as such. Running the recorded alternation (''censorship|censored|OONI|Censored Planet|GFW|geoblock|blockpage|internet shutdown'') over the current corpus gives **68 / 38 / 27 / 21**. The ''web'' column matches exactly and the other three do not, so either the corpus moved under it or the recorded alternation is not quite what was run; this run cannot distinguish those, and does not claim to. And 68 is not a floor for this page's population either. Of those 68 hits, **38 are in the population and 30 are not** — circumvention systems, homonyms and incidental mentions. In the other direction, **16 of the population's 63 papers are caught by no title probe at all**, and probe 1 (a wider title probe than the queued one) excludes **36 of its own 83 hits, 43.4%**. A candidate count justifies looking, and nothing more. | **The queued estimate was wrong in both directions and is not comparable.** [[:roadmap]] queued this page on a title-and-summary probe that matched 74 papers, 38 of them on the ''web'' platform, 30 in 2020–2023 and 22 in 2024–2026. **That probe is not in ''scripts/gap_probe_roadmap.mjs''** — it came from the 2026-09-02 brainstorm pass and only its alternation is recorded in prose — so it cannot be re-run as such. Running the recorded alternation (''censorship|censored|OONI|Censored Planet|GFW|geoblock|blockpage|internet shutdown'') over the current corpus gives **68 / 38 / 27 / 21**. The ''web'' column matches exactly and the other three do not, so either the corpus moved under it or the recorded alternation is not quite what was run; this run cannot distinguish those, and does not claim to. And 68 is not a floor for this page's population either. Of those 68 hits, **38 are in the population and 30 are not** — circumvention systems, homonyms and incidental mentions. In the other direction, **16 of the population's 63 papers are caught by no title probe at all**, and probe 1 (a wider title probe than the queued one) excludes **36 of its own 83 hits, 43.4%**. A candidate count justifies looking, and nothing more. |
| |
| ===== 4. Verdicts: all 275, and the off-topic families in full ===== | ===== 4. Verdicts: all 275, and the off-topic families in full ===== |
| ^ Off-topic family ^ Papers ^ Why ^ | ^ Off-topic family ^ Papers ^ Why ^ |
| | ''off:incidental'' | 115 | one incidental mention, or a related-work citation only | | | ''off:incidental'' | 115 | one incidental mention, or a related-work citation only | |
| | ''off:vpn-proxy-ecosystem'' | 8 | the VPN or proxy ecosystem itself → [[Design:Crawling location]] | | | ''off:vpn-proxy-ecosystem'' | 8 | the VPN or proxy ecosystem itself → [[:Design:Crawling location]] | |
| | ''off:homonym-tor-metrics'' | 7 | names **Tor Metrics**, a Tor network dataset, not a blocking observatory | | | ''off:homonym-tor-metrics'' | 7 | names **Tor Metrics**, a Tor network dataset, not a blocking observatory | |
| | ''off:takedown-effects'' | 5 | what a takedown did to the target, not whether a client could reach content | | | ''off:takedown-effects'' | 5 | what a takedown did to the target, not whether a client could reach content | |
| | ''off:blockchain-sanctions'' | 3 | sanctions and blocklists on a blockchain | | | ''off:blockchain-sanctions'' | 3 | sanctions and blocklists on a blockchain | |
| | ''off:homonym-statistical-censoring'' | 2 | "censored"/"censoring" in the statistics or ML sense (interval-censored Hawkes processes; adversarial censoring of demographic features) | | | ''off:homonym-statistical-censoring'' | 2 | "censored"/"censoring" in the statistics or ML sense (interval-censored Hawkes processes; adversarial censoring of demographic features) | |
| | ''off:adblocking'' | 2 | blocking by the user's own ad blocker → [[Programming:Filter lists]] | | | ''off:adblocking'' | 2 | blocking by the user's own ad blocker → [[:Programming:Filter lists]] | |
| | ''off:security-blocking'' | 2 | blocking malware or phishing | | | ''off:security-blocking'' | 2 | blocking malware or phishing | |
| | ''off:llm-guardrails'' | 2 | model refusals of what the user asked | | | ''off:llm-guardrails'' | 2 | model refusals of what the user asked | |
| | ''off:homonym-ml-concept-censorship'' | 1 | concept censorship in a diffusion model | | | ''off:homonym-ml-concept-censorship'' | 1 | concept censorship in a diffusion model | |
| | ''off:age-gate'' | 1 | age assurance → queued as ''privacy:age_assurance'' | | | ''off:age-gate'' | 1 | age assurance → queued as ''privacy:age_assurance'' | |
| | ''off:bot-blocking'' | 1 | blocked for looking like a bot → [[Programming:Crawler detection]] | | | ''off:bot-blocking'' | 1 | blocked for looking like a bot → [[:Programming:Crawler detection]] | |
| | ''off:censor-as-attack-surface'' | 1 | censorship middleboxes abused as an amplifier | | | ''off:censor-as-attack-surface'' | 1 | censorship middleboxes abused as an amplifier | |
| | ''off:moderation-system'' | 1 | builds a moderation classifier rather than measuring blocking | | | ''off:moderation-system'' | 1 | builds a moderation classifier rather than measuring blocking | |
| ==== Vantage locations ==== | ==== Vantage locations ==== |
| |
| ''vantage[].locations'' is free text and is folded through ''scripts/geo.mjs'' (shared with [[Design:Crawling location]] and [[Design:IP classification]], so the pages agree). **Unmapped strings are still counted as distinct locations** — an unmapped string is a place the paper named, and dropping it would make "two or more vantage points" an artefact of the alias list — and every one is printed: | ''vantage[].locations'' is free text and is folded through ''scripts/geo.mjs'' (shared with [[:Design:Crawling location]] and [[:Design:IP classification]], so the pages agree). **Unmapped strings are still counted as distinct locations** — an unmapped string is a place the paper named, and dropping it would make "two or more vantage points" an artefact of the alias list — and every one is printed: |
| |
| <code> | <code> |
| | OONI Probe CLI v3.30.0, 2026-07-27 | GitHub releases API, and ''/releases?per_page=3'' printed as well as ''/releases/latest'', because a prerelease can lead the list | | | OONI Probe CLI v3.30.0, 2026-07-27 | GitHub releases API, and ''/releases?per_page=3'' printed as well as ''/releases/latest'', because a prerelease can lead the list | |
| | Citizen Lab test lists are maintained | last commit 2026-09-09; 150 per-country CSVs plus ''global.csv'' | | | Citizen Lab test lists are maintained | last commit 2026-09-09; 150 per-country CSVs plus ''global.csv'' | |
| | Censored Planet's URLs have moved | ''censoredplanet.org/data/raw'' → **HTTP 404**; ''dashboard.'', ''data.'' and ''docs.censoredplanet.org'' → 200. The site is a hash-router SPA, so ''curl'' sees an empty ''<div id="root">'' and the //Raw Data// link is only visible after rendering | | | Censored Planet's URLs have moved | ''censoredplanet.org/data/raw'' → **HTTP 404**; ''dashboard.'', ''data.'' and ''docs.censoredplanet.org'' → 200. The site is a hash-router SPA, so ''curl'' sees an empty ''%%<div id="root">%%'' and the //Raw Data// link is only visible after rendering | |
| | the dashboard's counters | rendered: 116.924.640.072 total, 744.324.571 in the last 30 days, 236 countries | | | the dashboard's counters | rendered: 116.924.640.072 total, 744.324.571 in the last 30 days, 236 countries | |
| | Censored Planet's tools, with dates | GitHub repo API per repo: ''geoinspector'' last push 2023-05-19, ''CenTrace'' 2023-03-30, ''CenFuzz'' 2024-02-04, ''dns_blockpage_fingerprint'' 2025-05-01, ''geodiff-app'' 2025-05-01, ''censoredplanet-analysis'' 2025-06-29, ''chinese-llm-blocking'' 2025-07-16; none archived | | | Censored Planet's tools, with dates | GitHub repo API per repo: ''geoinspector'' last push 2023-05-19, ''CenTrace'' 2023-03-30, ''CenFuzz'' 2024-02-04, ''dns_blockpage_fingerprint'' 2025-05-01, ''geodiff-app'' 2025-05-01, ''censoredplanet-analysis'' 2025-06-29, ''chinese-llm-blocking'' 2025-07-16; none archived | |
| | RFC 9110 defines the HTTP semantics referenced | ''rfc9110.json'': INTERNET STANDARD, June 2022, not obsoleted or updated | | | RFC 9110 defines the HTTP semantics referenced | ''rfc9110.json'': INTERNET STANDARD, June 2022, not obsoleted or updated | |
| | **Regulation (EU) 2018/302 is about the internal market, not about US publishers walling the EU** | CELEX ''32018R0302'' fetched from the Publications Office **CELLAR** service (''publications.europa.eu/resource/celex/…''), because EUR-Lex answers automated clients with an AWS-WAF challenge. Title confirmed verbatim | | | **Regulation (EU) 2018/302 is about the internal market, not about US publishers walling the EU** | CELEX ''32018R0302'' fetched from the Publications Office **CELLAR** service (''publications.europa.eu/resource/celex/…''), because EUR-Lex answers automated clients with an AWS-WAF challenge. Title confirmed verbatim | |
| | Cloudflare Radar's outage centre exists but is bot-walled | HTTP **403** to ''curl'' with a browser User-Agent; the page therefore points at [[Programming:Cloudflare Radar]] and makes no claim about its contents | | | Cloudflare Radar's outage centre exists but is bot-walled | HTTP **403** to ''curl'' with a browser User-Agent; the page therefore points at [[:Programming:Cloudflare Radar]] and makes no claim about its contents | |
| |
| **Sources considered and rejected:** | **Sources considered and rejected:** |
| | 6 | The bot-management open question contradicted the page's own method section — telling the two apart per request is routine; what is missing is a **quantification** | **Accepted and rewritten** to the quantified version | | | 6 | The bot-management open question contradicted the page's own method section — telling the two apart per request is routine; what is missing is a **quantification** | **Accepted and rewritten** to the quantified version | |
| | 7 | "IMC … by a factor of three" is 2.1× | **Accepted**; already fixed in the author's re-read | | | 7 | "IMC … by a factor of three" is 2.1× | **Accepted**; already fixed in the author's re-read | |
| | 8 | **The ''design:'' namespace listing omits [[Design:Existing datasets]]**, a live page this one links three times, while asserting its table is the namespace's outline | **Accepted and fixed** on that page (15 rows), with the omission recorded there rather than silently corrected. The suggestion that ''sitemap.mjs'' assert every ''design:*'' id appears in that table is a good one and is **not** done here — it is a change to a shared script and belongs to whoever owns it | | | 8 | **The ''design:'' namespace listing omits [[:Design:Existing datasets]]**, a live page this one links three times, while asserting its table is the namespace's outline | **Accepted and fixed** on that page (15 rows), with the omission recorded there rather than silently corrected. The suggestion that ''sitemap.mjs'' assert every ''design:*'' id appears in that table is a good one and is **not** done here — it is a change to a shared script and belongs to whoever owns it | |
| | 9 | Provenance housekeeping: **''===== 12. Review ====='' appeared twice**, cross-references pointed at §10 and §11 for sections that are §12 and §16, and "every figure is produced by the report script" is overstated because the currency table comes from two other scripts | **Accepted and fixed**, all four | | | 9 | Provenance housekeeping: **''===== 12. Review ====='' appeared twice**, cross-references pointed at §10 and §11 for sections that are §12 and §16, and "every figure is produced by the report script" is overstated because the currency table comes from two other scripts | **Accepted and fixed**, all four | |
| | 10 | Minor: the probe enumeration summed to six, "whether a manipulated DNS answer is manipulated", "the one-sentence version" is three sentences, FilterMap described as combining Censys data when Censys is used to check its signatures, an unscoped "exactly one paper in the corpus", a heading promising a denominator column, and **Khattak et al. (NDSS 2016) cited as if it were corpus evidence** | **All accepted and fixed.** The last one turned out to be the most useful: NDSS 2016 is one of **eleven venue-years with no extracted papers at all** (NDSS 2010, 2011, 2016, 2018; PoPETs 2010–2014; CCS and IMC 2026), which the report now prints and the page now states in //What the corpus does not cover// | | | 10 | Minor: the probe enumeration summed to six, "whether a manipulated DNS answer is manipulated", "the one-sentence version" is three sentences, FilterMap described as combining Censys data when Censys is used to check its signatures, an unscoped "exactly one paper in the corpus", a heading promising a denominator column, and **Khattak et al. (NDSS 2016) cited as if it were corpus evidence** | **All accepted and fixed.** The last one turned out to be the most useful: NDSS 2016 is one of **eleven venue-years with no extracted papers at all** (NDSS 2010, 2011, 2016, 2018; PoPETs 2010–2014; CCS and IMC 2026), which the report now prints and the page now states in //What the corpus does not cover// | |
| |
| ^ Call ^ What was decided ^ What a reasonable person might have decided instead ^ | ^ Call ^ What was decided ^ What a reasonable person might have decided instead ^ |
| | **Split or extend?** | A new page, **with two bullets rewritten on the neighbour**. [[Design:Crawling location]] is 30 KB and is about //choosing// a vantage point; this page is about the phenomenon and the claim. Its "Content: geoblocking and geo-differentiation" subsection had three bullets; the geoblocking and geo-differentiation ones were **merged into one bullet that hands the phenomenon to this page** (the citations {[mcdonald2018_forbidden]} and {[kumar2022_investigation]} stay there, because they are also the reason to care about vantage choice), and the personalisation bullet stayed and gained the distinction it was missing — personalisation changes the //ordering// of what you are served, not whether you are served. A link to this page was added to that page's //Related pages// list. Nothing numeric was moved, so no figure on that page went stale, and its citekey counts are unchanged (checked by diffing them before and after) | Move the whole subsection out and leave a bare pointer. Rejected: the subsection is one of the three reasons that page gives for why the vantage point matters, and deleting it would leave that argument with a hole. Or leave the subsection untouched and duplicate it here — rejected as exactly the duplication the split was supposed to avoid | | | **Split or extend?** | A new page, **with two bullets rewritten on the neighbour**. [[:Design:Crawling location]] is 30 KB and is about //choosing// a vantage point; this page is about the phenomenon and the claim. Its "Content: geoblocking and geo-differentiation" subsection had three bullets; the geoblocking and geo-differentiation ones were **merged into one bullet that hands the phenomenon to this page** (the citations {[mcdonald2018_forbidden]} and {[kumar2022_investigation]} stay there, because they are also the reason to care about vantage choice), and the personalisation bullet stayed and gained the distinction it was missing — personalisation changes the //ordering// of what you are served, not whether you are served. A link to this page was added to that page's //Related pages// list. Nothing numeric was moved, so no figure on that page went stale, and its citekey counts are unchanged (checked by diffing them before and after) | Move the whole subsection out and leave a bare pointer. Rejected: the subsection is one of the three reasons that page gives for why the vantage point matters, and deleting it would leave that argument with a hole. Or leave the subsection untouched and duplicate it here — rejected as exactly the duplication the split was supposed to avoid | |
| | **The page id** | ''design:blocking_and_geodifference'', as fixed on [[roadmap]] on 2026-09-07 and enforced by ''scripts/sitemap.mjs''. Writing it under any other id would have left the roadmap promising a page that does not exist | ''design:blocking'' is shorter, and the //geodifference// half is arguably [[Design:Crawling location]]' or [[Design:Platforms]]'. Keeping the fixed id was the cheaper choice and the roadmap row was removed in the same sitting | | | **The page id** | ''design:blocking_and_geodifference'', as fixed on [[:roadmap]] on 2026-09-07 and enforced by ''scripts/sitemap.mjs''. Writing it under any other id would have left the roadmap promising a page that does not exist | ''design:blocking'' is shorter, and the //geodifference// half is arguably [[:Design:Crawling location]]' or [[:Design:Platforms]]'. Keeping the fixed id was the cheaper choice and the roadmap row was removed in the same sitting | |
| | **Circumvention excluded** | The 28 circumvention papers are counted and named but not in the population: a paper that builds an evasion system measures blocking //instrumentally//, and including them would put Geneva and domain fronting in a table about how a blocking claim is made | Include them — they contain some of the best blocking measurements in the corpus (Geneva's and GET /out's censor models especially). The compromise is that the count is published, so a reader who disagrees can add 28 and see what moves | | | **Circumvention excluded** | The 28 circumvention papers are counted and named but not in the population: a paper that builds an evasion system measures blocking //instrumentally//, and including them would put Geneva and domain fronting in a table about how a blocking claim is made | Include them — they contain some of the best blocking measurements in the corpus (Geneva's and GET /out's censor models especially). The compromise is that the count is published, so a reader who disagrees can add 28 and see what moves | |
| | **Protective DNS included** | In, as the whole ''filter'' family (1 paper): its motive is security, but its claim structure is this page's | Out, as security blocking. Subtracting one paper is the whole difference | | | **Protective DNS included** | In, as the whole ''filter'' family (1 paper): its motive is security, but its claim structure is this page's | Out, as security blocking. Subtracting one paper is the whole difference | |
| - **Two "quotes" came from the extraction's model-written ''summary'' field** and were replaced with the papers' own words (§9). Nothing was published in that state, but the checker had to be extended to catch the class rather than the instance. | - **Two "quotes" came from the extraction's model-written ''summary'' field** and were replaced with the papers' own words (§9). Nothing was published in that state, but the checker had to be extended to catch the class rather than the instance. |
| - **''scripts/sitemap.mjs'' reported a failure that was not one.** It caches page source under ''out/pages/'' and never expires it, so a roadmap row removed twenty minutes earlier still showed as "WRITTEN but still on the roadmap". The cache entry was refreshed by hand; the script's cache policy is a known trap rather than a bug found here. | - **''scripts/sitemap.mjs'' reported a failure that was not one.** It caches page source under ''out/pages/'' and never expires it, so a roadmap row removed twenty minutes earlier still showed as "WRITTEN but still on the roadmap". The cache entry was refreshed by hand; the script's cache policy is a known trap rather than a bug found here. |
| | - **A literal ''%%<div id="root">%%'' inside monospace truncated this page from section 11 onward**, and it was live in that state for about two minutes. DokuWiki's inline code does **not** escape a tag: everything after it was dropped, silently, with no error and no failing check — ''check_wrap.mjs'' and ''check_tables.mjs'' both passed on the source. It was caught by fetching the **rendered DOM** and comparing its heading, table and ''<file>'' counts against the source's. Nothing but reading the render catches this class. |
| | - **Every internal link on this page was namespace-relative.** From ''provenance:design:'', ''%%[[roadmap]]%%'' resolves to ''provenance:design:roadmap''. Every link here is now root-anchored with a leading colon, and the rendered DOM was checked for ''wikilink2'' (red-link) classes. **Writing that sentence produced a third instance of the same bug**: the example link, in monospace, was still a live link and rendered red, because inline code does not escape a wiki link any more than it escapes a tag. It needs ''%%…%%'' — which is what the sentence you are reading now uses. |
| - **''report_blocking_geodifference.mjs'' exhausted the V8 heap on its first run**, because it cached every paper's full text (0.5 GB on disk, over 4 GB as JS strings). It now does a single streaming pass for the whole-corpus regexes and caches only the 63 population texts. | - **''report_blocking_geodifference.mjs'' exhausted the V8 heap on its first run**, because it cached every paper's full text (0.5 GB on disk, over 4 GB as JS strings). It now does a single streaming pass for the whole-corpus regexes and caches only the 63 population texts. |
| |
| ===== Related ===== | ===== Related ===== |
| |
| * **[[design:blocking_and_geodifference|The page these notes are for]]** | * **[[:design:blocking_and_geodifference|The page these notes are for]]** |
| * [[literature:corpus]] — venue scope, the selection funnel, and the provisional 2025–2026 years | * [[:literature:corpus]] — venue scope, the selection funnel, and the provisional 2025–2026 years |
| * ''provenance:design:crawling_location'' does not exist (deliberately not created here — that page's figures were not touched by this run); the namespace's own log is [[provenance:design]] | * ''provenance:design:crawling_location'' does not exist (deliberately not created here — that page's figures were not touched by this run); the namespace's own log is [[:provenance:design]] |
| * [[roadmap]] / [[provenance:roadmap]] — where this page's id and queued estimate came from | * [[:roadmap]] / [[:provenance:roadmap]] — where this page's id and queued estimate came from |
| |
| <bibtex bibliography></bibtex> | <bibtex bibliography></bibtex> |
| |
