design:dns
Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
| design:dns [2026/08/27 23:15] – Figures re-check applied: drop the dnsdb.info A record (it rotated between two runs an hour apart) and correct the Rapid7 guard's tuple count. Authored by Claude. karel.kubicek.claude | design:dns [2026/09/25 03:03] (current) – Is-this-your-page row: squatted, parked, expired or dangling names and WHOIS/RDAP registration data -> Security:Domain abuse. Authored by Claude. karel.kubicek.claude | ||
|---|---|---|---|
| Line 4: | Line 4: | ||
| <WRAP important> | <WRAP important> | ||
| - | **A DNS answer is a property of the query, not only of the name.** It depends on which resolver you asked, where you asked from, what your source port happened to be, and what was already in a cache. In a run on 2026-08-27 from one machine, Google, Cloudflare and Quad9 **did not all return the same addresses for 29 of the 75** Tranco top-100 names that had an address at all (38.7%), and for **8** of those no two of the three shared a single address. Another **25 of the 100** had no A record at the apex at all. Details and the script are in [[#Measure your own resolver dependence]]. | + | **A DNS answer is a property of the query, not only of the name.** It depends on which resolver you asked, where you asked from, what your source port happened to be, and what was already in a cache. In a run on 2026-08-27 from one machine, Google, Cloudflare and Quad9 **did not all return the same addresses for 29 of the 75** Tranco top-100 names that had an IPv4 address at all (38.7%), and for **10** of those no two of the three shared a single address. Another **25 of the 100** have no A record at the apex at all. Details and the script are in [[#Measure your own resolver dependence]]. |
| </ | </ | ||
| - | Everything below about "the literature" | + | Everything below about "the literature" |
| ===== Is this your page? ===== | ===== Is this your page? ===== | ||
| Line 18: | Line 18: | ||
| | whether to crawl, scan, or unpack an app at all | [[Design: | | whether to crawl, scan, or unpack an app at all | [[Design: | ||
| | who owns a name (WHOIS, RDAP, the Public Suffix List) | not here — see [[Design: | | who owns a name (WHOIS, RDAP, the Public Suffix List) | not here — see [[Design: | ||
| + | | whether a name is a squat, parked, expired and re-registered, | ||
| | the ethics of probing other people' | | the ethics of probing other people' | ||
| Line 84: | Line 85: | ||
| | **Rapid7 Open Data** (Sonar FDNS / RDNS) | 8 | **Still live**, contrary to what a 2022-era memory will tell you: '' | | **Rapid7 Open Data** (Sonar FDNS / RDNS) | 8 | **Still live**, contrary to what a 2022-era memory will tell you: '' | ||
| | **ICANN CZDS** / zone files | 3 | Current. Per-TLD application. | The authoritative list of registered names in participating zones. It is the denominator a "we resolved the top 1M" study does not have. | | | **ICANN CZDS** / zone files | 3 | Current. Per-TLD application. | The authoritative list of registered names in participating zones. It is the denominator a "we resolved the top 1M" study does not have. | | ||
| - | | **Active DNS Project**; **360 / 114DNS**; **Spamhaus**; **DNS Observatory** | 2 / 4 / — / 1 | Various. | Regional and vendor feeds; the Chinese feeds are the only route to some of that namespace. | | + | | **Active DNS Project**; **360 / 114DNS**; **DNS Observatory** | 2 / 4 / 1 | Various. | Regional and vendor feeds; the Chinese feeds are the only route to some of that namespace. Spamhaus' |
| ==== Where you ask from ==== | ==== Where you ask from ==== | ||
| Line 90: | Line 91: | ||
| **RIPE Atlas** appears in **75** papers corpus-wide and **24** of the 244 here — and **21** of those 24 are IMC papers, which is another way of saying the same thing as the venue table. It is not in the membership rule because most of the 75 use it for traceroute, not DNS. But it is the standard way to ask the same DNS question from thousands of vantage points, and DNS is one of its native measurement types: {[randall2021_home]} localises interception with it, {[moura2019_cache]} probes recursive resolvers worldwide with it, {[aldalky2019_look]} uses it for globally distributed probes, and {[randall2020_trufflehunter]} uses it to emulate geographically distributed users filling resolver caches. {[bhaskar2022_many]}, | **RIPE Atlas** appears in **75** papers corpus-wide and **24** of the 244 here — and **21** of those 24 are IMC papers, which is another way of saying the same thing as the venue table. It is not in the membership rule because most of the 75 use it for traceroute, not DNS. But it is the standard way to ask the same DNS question from thousands of vantage points, and DNS is one of its native measurement types: {[randall2021_home]} localises interception with it, {[moura2019_cache]} probes recursive resolvers worldwide with it, {[aldalky2019_look]} uses it for globally distributed probes, and {[randall2020_trufflehunter]} uses it to emulate geographically distributed users filling resolver caches. {[bhaskar2022_many]}, | ||
| - | **Generic scanners** — ZMap and XMap (112 papers), ZGrab (31), Scapy (33) — appear in 38 of the 244. ZMap is how you find open resolvers; it is not a DNS toolkit. [[Design: | + | **Generic scanners** — ZMap and XMap (110 papers), ZGrab (32), Scapy (33) — appear in 37 of the 244. ZMap is how you find open resolvers; it is not a DNS toolkit. Those first two counts are the **same paper sets** [[Programming: |
| ===== Five ways a DNS measurement goes wrong ===== | ===== Five ways a DNS measurement goes wrong ===== | ||
| Line 96: | Line 97: | ||
| ==== 1. The answer depends on which resolver you asked ==== | ==== 1. The answer depends on which resolver you asked ==== | ||
| - | This is the oldest finding on the page and it has never stopped being true. Ager et al. {[ager2010_comparing]} sent the same queries to the local resolver, Google and OpenDNS from 60-plus vantage points in 2010 and found that //"the answers to the DNS resolvers differ in terms of subnets for approximately 2,000 out of our 10,000 host names. In half of these cases, the returned IP addresses even belong to different ASs and countries."// | + | This is the oldest finding on the page and it has never stopped being true. Ager et al. {[ager2010_comparing]} sent the same queries to the local resolver, Google and OpenDNS from 60-plus vantage points in 2010 and found that //"the answers to the DNS resolvers differ in terms of subnets for approximately 2,000 out of our 10,000 host names. In half of these cases, the returned IP addresses even belong to different ASs and countries."// |
| Two mechanisms do most of it. CDNs and GeoDNS answer according to where they think you are, and **EDNS Client Subnet** (RFC 7871 — note it is **Informational**, | Two mechanisms do most of it. CDNs and GeoDNS answer according to where they think you are, and **EDNS Client Subnet** (RFC 7871 — note it is **Informational**, | ||
| Line 134: | Line 135: | ||
| | **DoH** — DNS over HTTPS | RFC 8484, October 2018, Proposed Standard | 0 / 0 / 23 / 16 / 9 | The one the browsers deployed. The most-measured of the encrypted transports and still the default assumption. | | | **DoH** — DNS over HTTPS | RFC 8484, October 2018, Proposed Standard | 0 / 0 / 23 / 16 / 9 | The one the browsers deployed. The most-measured of the encrypted transports and still the default assumption. | | ||
| | **DoQ** — DNS over QUIC | RFC 9250, May 2022, Proposed Standard | 0 / 0 / 4 / 1 / 1 | Standardised, | | **DoQ** — DNS over QUIC | RFC 9250, May 2022, Proposed Standard | 0 / 0 / 4 / 1 / 1 | Standardised, | ||
| - | | **DoC** — DNS over CoAP | RFC 9953, **March 2026**, Proposed Standard | 0 / 0 / 0 / 0 / 0 | Five months old at the time of writing | + | | **DoC** — DNS over CoAP | RFC 9953, **March 2026**, Proposed Standard | 0 / 0 / 0 / 0 / 0 | Five months old at the time of writing. The zero is a **run probe, |
| | **ODoH** — Oblivious DoH | RFC 9230, June 2022, **Experimental** | 0 / 0 / 5 / 4 / 2 | Experimental in the RFC sense as well as the deployment sense. Do not describe it as a deployed default. | | | **ODoH** — Oblivious DoH | RFC 9230, June 2022, **Experimental** | 0 / 0 / 5 / 4 / 2 | Experimental in the RFC sense as well as the deployment sense. Do not describe it as a deployed default. | | ||
| | **DDR** — Discovery of Designated Resolvers | RFC 9462, November 2023, Proposed Standard (with DNR, RFC 9463) | 0 / 0 / 0 / 0 / 1 | The frontier, and the answer to "how does a client find an encrypted resolver at all". **Exactly one paper in this corpus measures it.** | | | **DDR** — Discovery of Designated Resolvers | RFC 9462, November 2023, Proposed Standard (with DNR, RFC 9463) | 0 / 0 / 0 / 0 / 1 | The frontier, and the answer to "how does a client find an encrypted resolver at all". **Exactly one paper in this corpus measures it.** | | ||
| Line 168: | Line 169: | ||
| ===== Active DNS measurement is scanning ===== | ===== Active DNS measurement is scanning ===== | ||
| - | Sending queries to resolvers you do not own is a scan, and it has a failure mode the rest of the scan branch does not: **an open resolver will amplify whatever you send it at somebody else.** Kührer et al. {[khrer2015_going]} //" | + | Sending queries to resolvers you do not own is a scan, and it has a failure mode the rest of the scan branch does not: **an open resolver will amplify whatever you send it at somebody else.** Kührer et al. {[kuhrer2015_going]} //" |
| - | The DNS population states more about ethics than the corpus average on three of four fields, and **less** on the one that names an ethics board — which is the pattern you would expect from work that mitigates harm operationally and does not go to an IRB because it has no human subjects: | + | The DNS population states more about ethics than the corpus average on three of four fields, and **less** on the one that names an ethics board. A plausible reading |
| ^ Field ^ DNS population (241 empirical of the 244) ^ Whole corpus (5,118 empirical) ^ | ^ Field ^ DNS population (241 empirical of the 244) ^ Whole corpus (5,118 empirical) ^ | ||
| Line 196: | Line 197: | ||
| </ | </ | ||
| - | A DNS methods section is complete when a replicator can answer all of these. | + | A DNS methods section is complete when a replicator can answer all of these. |
| * **Which resolver**, by address and by name, including whether it was your own recursive server, the system resolver, or a public one. If more than one, which for which measurement. | * **Which resolver**, by address and by name, including whether it was your own recursive server, the system resolver, or a public one. If more than one, which for which measurement. | ||
| * **Where from** — the source addresses and networks, and whether they were datacenter or residential ([[Design: | * **Where from** — the source addresses and networks, and whether they were datacenter or residential ([[Design: | ||
| - | * **The transport** — Do53 over UDP or TCP, DoT, DoH, DoQ — and whether you fell back. | + | * **The transport** — Do53 over UDP or TCP, DoT, DoH, DoQ — and whether you fell back, including what you did when a resolver set the **TC bit** on a UDP answer. Only **9.0%** of the 244 mention truncation or TCP fallback at all. |
| + | * **Which record types, and which address family.** A study that queries A only and a study that queries A and AAAA produce different "did not resolve" | ||
| * **EDNS0 and ECS**: whether you sent an OPT record, and whether you sent a client-subnet option and with what prefix. This changes the answer, so an unstated setting makes the result unreproducible. | * **EDNS0 and ECS**: whether you sent an OPT record, and whether you sent a client-subnet option and with what prefix. This changes the answer, so an unstated setting makes the result unreproducible. | ||
| * **Query rate and retry policy**, and what you did on SERVFAIL and timeout. A timeout silently recorded as "no record" | * **Query rate and retry policy**, and what you did on SERVFAIL and timeout. A timeout silently recorded as "no record" | ||
| Line 223: | Line 225: | ||
| Python 3.8+, standard library only: no dnspython, no dig, nothing to install. | Python 3.8+, standard library only: no dnspython, no dig, nothing to install. | ||
| + | |||
| + | **This script queries A records only.** That is a deliberate limit and a real | ||
| + | one: a name with only AAAA records, or one reachable through an HTTPS/SVCB | ||
| + | record, is reported here as having no address. If your study is dual-stack, | ||
| + | query both families and report both -- "the name did not resolve" | ||
| + | has no IPv4 address" | ||
| + | the second. | ||
| What it reports, per name: | What it reports, per name: | ||
| - | * the A-record set each resolver returned | + | * the A-record set each resolver returned, and whether the answer was |
| + | truncated (TC set) rather than complete | ||
| * whether all resolvers agree, agree on the /24, or disagree outright | * whether all resolvers agree, agree on the /24, or disagree outright | ||
| * the TTL each resolver returned, and the TTL on an immediate second query | * the TTL each resolver returned, and the TTL on an immediate second query | ||
| Line 300: | Line 310: | ||
| def parse_answers(buf, | def parse_answers(buf, | ||
| - | """ | + | """ |
| + | |||
| + | The TC (truncation) bit matters here and is easy to skip. This script sends | ||
| + | no EDNS0 OPT record on purpose (see build_query), | ||
| + | capped at 512 bytes; a resolver with more addresses than fit sets TC and | ||
| + | returns a partial answer. Recording that as the resolver' | ||
| + | would manufacture " | ||
| + | is set the result is flagged rather than silently trusted -- a real | ||
| + | measurement would retry over TCP, which this script deliberately does not | ||
| + | do, because the point is to show you the condition, not to paper over it. | ||
| + | | ||
| if len(buf) < 12: | if len(buf) < 12: | ||
| raise ValueError(" | raise ValueError(" | ||
| Line 307: | Line 327: | ||
| raise ValueError(" | raise ValueError(" | ||
| rcode = flags & 0x0F | rcode = flags & 0x0F | ||
| + | truncated = bool(flags & 0x0200) | ||
| off = 12 | off = 12 | ||
| for _ in range(qd): | for _ in range(qd): | ||
| Line 318: | Line 339: | ||
| out.append((ttl, | out.append((ttl, | ||
| off += rdlen | off += rdlen | ||
| - | return rcode, out | + | return rcode, truncated, out |
| def query(server, | def query(server, | ||
| """ | """ | ||
| - | txid, pkt = build_query(name, | ||
| sock = socket.socket(socket.AF_INET, | sock = socket.socket(socket.AF_INET, | ||
| started = time.time() | started = time.time() | ||
| deadline = started + timeout | deadline = started + timeout | ||
| try: | try: | ||
| + | # Inside the try: an unencodable name must return a dict like every | ||
| + | # other failure, not raise. The docstring promises that. | ||
| + | txid, pkt = build_query(name, | ||
| sock.sendto(pkt, | sock.sendto(pkt, | ||
| # Accept only a datagram from the address we asked. This does not make | # Accept only a datagram from the address we asked. This does not make | ||
| Line 347: | Line 370: | ||
| if peer[0] == server: | if peer[0] == server: | ||
| break | break | ||
| - | rcode, answers = parse_answers(data, | + | rcode, truncated, answers = parse_answers(data, |
| return {" | return {" | ||
| + | " | ||
| " | " | ||
| " | " | ||
| Line 354: | Line 378: | ||
| except Exception as exc: # timeout, refused, malformed | except Exception as exc: # timeout, refused, malformed | ||
| return {" | return {" | ||
| + | " | ||
| " | " | ||
| " | " | ||
| Line 521: | Line 546: | ||
| elif ttl2 == ttl1: | elif ttl2 == ttl1: | ||
| cached = f" | cached = f" | ||
| + | trunc = " | ||
| print(f" | print(f" | ||
| - | f" | + | f" |
| record.append({" | record.append({" | ||
| print() | print() | ||
| Line 570: | Line 596: | ||
| < | < | ||
| # resolvers: Google (8.8.8.8), Cloudflare (1.1.1.1), Quad9 (9.9.9.9) | # resolvers: Google (8.8.8.8), Cloudflare (1.1.1.1), Quad9 (9.9.9.9) | ||
| - | # names: 100 run at 2026-08-27T22:56:27Z | + | # names: 100 run at 2026-08-27T23:17:26Z |
| - | gstatic.com -> disjoint | + | googleapis.com -> disjoint |
| - | Google | + | Google |
| - | Cloudflare | + | Cloudflare |
| - | Quad9 NOERROR | + | Quad9 NOERROR |
| summary A: all 100 names | summary A: all 100 names | ||
| identical | identical | ||
| - | partial | + | partial |
| - | disjoint | + | disjoint |
| no-A-record | no-A-record | ||
| summary B: the 75 names where at least two resolvers returned an address | summary B: the 75 names where at least two resolvers returned an address | ||
| identical | identical | ||
| - | partial | + | partial |
| - | disjoint | + | disjoint |
| NOT identical | NOT identical | ||
| </ | </ | ||
| - | Four things to take from it, none of which is a claim about the Internet — it is one machine, one instant, three resolvers, one hundred names: | + | Five things to take from it, none of which is a claim about the Internet — it is one machine, one instant, three resolvers, one hundred names: |
| - | - **The number to quote is "not identical", | + | - **The number to quote is "not identical", |
| + | - **And "not identical" | ||
| - **25 of the 100 names have no A record at the apex.** They are CDN and infrastructure zones. A pipeline that resolves and drops failures loses them silently. | - **25 of the 100 names have no A record at the apex.** They are CDN and infrastructure zones. A pipeline that resolves and drops failures loses them silently. | ||
| - | - **Summary B is the denominator.** Summary | + | - **This is IPv4 only.** The script asks for A records, so "no address" |
| - **Read the verdict logic before trusting any of this.** The first version of this script intersected all three resolvers' | - **Read the verdict logic before trusting any of this.** The first version of this script intersected all three resolvers' | ||
| Line 611: | Line 638: | ||
| | Ververis et al., PoPETs 2025, //Path to Encrypted DNS with DDR// {[ververis2025_path]} | Where the encrypted-DNS question is now, and how thin the evidence still is. | | | Ververis et al., PoPETs 2025, //Path to Encrypted DNS with DDR// {[ververis2025_path]} | Where the encrypted-DNS question is now, and how thin the evidence still is. | | ||
| - | Also worth your time, by topic: {[lu2019_over]} and {[chhabra2021_over]} on the cost and performance of encrypted DNS; {[li2024_worldwide]} on its reachability; | + | ==== Four current RFCs to measure ==== |
| + | |||
| + | The four documents below are current Proposed Standards in the [[https:// | ||
| + | |||
| + | **[[https:// | ||
| + | |||
| + | **[[https:// | ||
| + | |||
| + | **[[https:// | ||
| + | |||
| + | **[[https:// | ||
| + | |||
| + | Also worth your time, by topic: {[lu2019_over]} and {[chhabra2021_over]} on the cost and performance of encrypted DNS; {[li2024_worldwide]} on its reachability; | ||
| ===== Use in publications ===== | ===== Use in publications ===== | ||
| - | Figures from '' | + | Figures from '' |
| ==== What the instrument families look like ==== | ==== What the instrument families look like ==== | ||
| Line 627: | Line 666: | ||
| | dnssec-tool | 3 | 3 | yes | | | dnssec-tool | 3 | 3 | yes | | ||
| | open-resolver-census | 1 | 1 | yes | | | open-resolver-census | 1 | 1 | yes | | ||
| - | | scanner-generic (ZMap, ZGrab, Scapy) | 156 | 38 | no | | + | | scanner-generic (ZMap, ZGrab, Scapy) | 154 | 37 | no | |
| | vantage-platform (RIPE Atlas 75, MobileAtlas 2) | 77 | 24 | no | | | vantage-platform (RIPE Atlas 75, MobileAtlas 2) | 77 | 24 | no | | ||
design/dns.1787872554.txt.gz · Last modified: by karel.kubicek.claude
