Differences

This shows you the differences between two versions of the page.

--- design:crawling_location [2026/09/10 23:47] – Hand the geoblocking and geo-differentiation phenomenon to the new design:blocking_and_geodifference: merge the two bullets that duplicated it into one pointer, keep the personalisation bullet (and say what makes it different), add the page to Related. No karel.kubicek.claude
+++ design:crawling_location [2026/09/17 10:45] (current) – ConsentAction audit propagation and EU/EEA figures; Authored by Claude karel.kubicek.claude
@@ Line 9: / Line 9: @@
   - **How your IP is treated.** Datacenter, university, Tor and residential addresses receive measurably different treatment from bot management and from trackers.
-This page covers all three, how researchers have actually chosen vantage points, the practical options, and how to verify that the vantage point you think you have is the one you got. It pairs with [[Design:IP classification]] (classifying //other people's// addresses), [[Design:Website selection]] (which sites), and [[Programming:Crawler]] (which tool).
+This page covers all three, how researchers have actually chosen vantage points, the practical options, and how to verify that the vantage point you think you have is the one you got. It pairs with [[Design:IP classification]] (classifying //other people's// addresses), [[Design:Website selection]] (which sites), and [[Programming:Crawler]] (which tool). [[Privacy:Age assurance]] is the sharpest current case of the first bullet: since 2025 several jurisdictions //require// a site to behave differently for visitors they believe are theirs, so an age-gate prevalence figure without a stated vantage point is not a quantity.
 ===== Why the Vantage Point Changes Your Results =====
@@ Line 140: / Line 140: @@
 <WRAP important>
-The gap is worse where it matters most. Of the **349 papers that state what their crawler did with the consent banner** — accept, reject, a CMP-specific choice, or explicitly no interaction — only **68 (19.5%) state an EU/EEA vantage, and 210 give no location at all.** Consent behaviour is the single most geo-dependent thing on the web, and the majority of papers interacting with it do not report the jurisdiction they observed it from.
+The gap is worse where it matters most. Of the **55 papers whose audited full-text record states what their crawler did with the consent banner** — accept, reject, a CMP-specific choice, or explicitly no interaction — **37 (67.3%) state an EU/EEA vantage, and 11 state no location at all.** Consent behaviour is the single most geo-dependent thing on the web, and the jurisdiction is still missing for a substantial minority of those audited paper claims.
-Both figures come from the same per-paper record, so the 349 are a subset of the 1,120 crawling papers rather than a separately sampled group: the consent action and the vantage point were extracted in one pass from the same full text, and a paper counts here only if it stated its consent action explicitly.
+Both figures use the same per-paper record, but the consent denominator is the audited one, not the raw schema count: the 2026-09-05 audit read all 349 non-sentinel ''crawlConfig.consentAction'' labels and rejected 279 of the 313 ''no-interaction'' defaults. The resulting 55 are a subset of the 1,120 crawling papers, and a paper counts here only if the audit found a consent action claim in its full text.
 </WRAP>
@@ Line 202: / Line 202: @@
   * **Geolocation databases disagree with each other.** They agree well at country level and poorly below it, which is the subject of [[Design:IP classification]].
-The following script checks both. It queries several free services for your egress IP, reports whether they agree, flags datacenter/VPN/proxy addresses, and exits non-zero if the country is not the one you expected — so it can gate a crawl rather than merely inform you. Free tiers are rate-limited, so call it once when a vantage point comes up and again when it goes down, not per request. Adding a ''--json'' branch that dumps the same report as a dict is a two-line change if you want to log it alongside the crawl.
+The following script checks both. It queries several free services for your egress IP, reports whether they agree, flags datacenter/VPN/proxy addresses, and exits non-zero if the country is not the one you expected — so it can gate a crawl rather than merely inform you. Free tiers are rate-limited, so call it once when a vantage point comes up and again when it goes down, not per request. Adding a ''%%--json%%'' branch that dumps the same report as a dict is a two-line change if you want to log it alongside the crawl.
 <file python verify_vantage.py>