| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| design:crawling_location [2026/09/15 17:16] – Cross-reference privacy:age_assurance: since 2025 several jurisdictions require a site to behave differently by visitor location. Authored by Claude karel.kubicek.claude | design:crawling_location [2026/09/17 10:45] (current) – ConsentAction audit propagation and EU/EEA figures; Authored by Claude karel.kubicek.claude |
|---|
| |
| <WRAP important> | <WRAP important> |
| The gap is worse where it matters most. Of the **349 papers that state what their crawler did with the consent banner** — accept, reject, a CMP-specific choice, or explicitly no interaction — only **68 (19.5%) state an EU/EEA vantage, and 210 give no location at all.** Consent behaviour is the single most geo-dependent thing on the web, and the majority of papers interacting with it do not report the jurisdiction they observed it from. | The gap is worse where it matters most. Of the **55 papers whose audited full-text record states what their crawler did with the consent banner** — accept, reject, a CMP-specific choice, or explicitly no interaction — **37 (67.3%) state an EU/EEA vantage, and 11 state no location at all.** Consent behaviour is the single most geo-dependent thing on the web, and the jurisdiction is still missing for a substantial minority of those audited paper claims. |
| |
| Both figures come from the same per-paper record, so the 349 are a subset of the 1,120 crawling papers rather than a separately sampled group: the consent action and the vantage point were extracted in one pass from the same full text, and a paper counts here only if it stated its consent action explicitly. | Both figures use the same per-paper record, but the consent denominator is the audited one, not the raw schema count: the 2026-09-05 audit read all 349 non-sentinel ''crawlConfig.consentAction'' labels and rejected 279 of the 313 ''no-interaction'' defaults. The resulting 55 are a subset of the 1,120 crawling papers, and a paper counts here only if the audit found a consent action claim in its full text. |
| </WRAP> | </WRAP> |
| |
| * **Geolocation databases disagree with each other.** They agree well at country level and poorly below it, which is the subject of [[Design:IP classification]]. | * **Geolocation databases disagree with each other.** They agree well at country level and poorly below it, which is the subject of [[Design:IP classification]]. |
| |
| The following script checks both. It queries several free services for your egress IP, reports whether they agree, flags datacenter/VPN/proxy addresses, and exits non-zero if the country is not the one you expected — so it can gate a crawl rather than merely inform you. Free tiers are rate-limited, so call it once when a vantage point comes up and again when it goes down, not per request. Adding a ''--json'' branch that dumps the same report as a dict is a two-line change if you want to log it alongside the crawl. | The following script checks both. It queries several free services for your egress IP, reports whether they agree, flags datacenter/VPN/proxy addresses, and exits non-zero if the country is not the one you expected — so it can gate a crawl rather than merely inform you. Free tiers are rate-limited, so call it once when a vantage point comes up and again when it goes down, not per request. Adding a ''%%--json%%'' branch that dumps the same report as a dict is a two-line change if you want to log it alongside the crawl. |
| |
| <file python verify_vantage.py> | <file python verify_vantage.py> |
