User Tools

Site Tools


provenance:security:tls_certificates

This is an old revision of the document!


Provenance: security:tls_certificates

Working log behind tls_certificates. Corpus-wide caveats are on corpus. Citations use the shared bibliography.

Run: 2026-08-27. Corpus: 5,859 extracted papers, 7 venues, 2010–2026, data/extract/run1. Item: drain wiki-measuretheweb / “security:tls_certificates (new)”, claimed as cursor-drain-tls, store id 220, run 58. Author of the page and this log: Cursor, not Claude Code. Reviews: three focused passes then one generic, all on GPT 5.6 Luna medium (gpt-5.6-luna-medium), the sitting's requested substitute for the spec's sonnet/fable split.

Creating, not extending. node scripts/dw.mjs info security:tls_certificates — page did not exist. ?do=export_raw on a missing page returns the HTML error page (not an existence test by byte count). dw.mjs pages does not list provenance:. Neighbour security already linked the red child. Overlap judgement: write the promised child rather than widening mobile_and_app_measurement (pinning) or ip_classification (Censys-as-IP).

No ~~DISCUSSION~~ on this provenance page. Comments belong on the content page.

Scope decisions

Decision Why What a reasonable person might have done instead
Page N = 56 after a task fold, not 50 / 133 / 525 Certificate-unit misses Holz and Kotzias and includes RPKI; Censys/ZGrab is mostly not TLS; full-text 525 is bibliography hits. Publish 133 or 525 as the population. That would be a lie about the residue.
Censys and OpenSSL are not membership 100 Censys/ZGrab papers; 107 OpenSSL. IoT, SSH, DNS, and “we linked OpenSSL”. A tool-match page titled TLS. Rejected.
HAND_TASK of 12, residue 0 Ordered regexes put several founding papers in the wrong bucket (OCSP → revocation; “domain-validation” → Let’s Encrypt). Tighter regexes. Would swap one silent error for another.
Instrument fold only over TLS-ish names Folding every used tool on the 56 produced 241 residue strings (Firefox, Chrome, Nginx). Publish that residue as “the field uses browsers”. True and off-topic.
crt.sh 502 is on the page as a WRAP todo Probe and pin script both hit nginx 502 on 2026-08-27. Pretend the CT pin ran. Forbidden.
No Censys API call Platform needs a PAT. Script says so. Ship a fake “Censys snapshot id”.

Report script

scripts/report_tls_certificates.mjs plus scripts/tls_fold.mjs. Deterministic. Re-run:

node scripts/report_tls_certificates.mjs > scripts/report_tls_certificates-output.txt

Flags: --wiki, --list, --quotes, --hits <re>. Unknown flags print FAILURE and exit 1. Exits 1 if corpus size is not 5,859, crawled not 1,120, web not 1,622, missing .cols not 4, task residue not 0, or a HAND_TASK key is not a candidate. A printed FAILURE with exit 0 is forbidden.

python3 pages/pin_web_pki.py is the published script. Advertised flags: --host, --port, --skip-ct, --timeout. All exist on the live object. crt.sh HTTP errors raise RuntimeError with the URL and status.

bash scripts/tls_external_probe.sh re-fetches Chrome log_list.json, Censys docs, Let’s Encrypt posts, GitHub, PyPI, crt.sh, and the CT landing redirect. Honours GH_TOKEN. Exits 1 if any check is FAILED. This sitting: crt.sh HTTP 502.

Queries

# Query Population / denominator Result On the page?
Q1 Extraction records all 5,859 outer frame
Q2 crawled 5,859 1,120 methodology
Q3 web platform 5,859 1,622 methodology
Q4 missing paper.cols.txt 5,859 4 methodology
Q5 population.unit == certificates 5,859 50 (22 web) yes, as a signal
Q6 TLS-specific tools used/produced 5,859 39 (16 web) yes
Q7 tight detection regex 5,859 103 (52 web) yes
Q8 title/slug regex 5,859 70 yes
Q9 UNION of Q5–Q8 = candidates 5,859 158 yes
Q10 Censys/ZGrab used/produced 5,859 100 (32 web) yes, NOT membership
Q11 OpenSSL used/produced 5,859 107 yes, NOT membership
Q12 Censys/ZGrab ∩ page 56 11 yes
Q13 full-text TLS-cert probe 5,859 525 (213 web) yes, upper bound
Q14 of Q13 also a candidate 525 138 report only
Q15 full-text web, not a candidate 213 web FT 146 report only
Q16 page = WEB_PKI_TASKS of fold 158 56 (35.4%) yes, the N
Q17 of 56: web / crawled / scan 56 37 / 6 / 44 yes
Q18 excluded by task 158 102 yes
Q19 fold residue 158 0 yes
Q20 HAND_TASK size 12 yes
Q21 posters in 56 56 3 (5.4%) yes
Q22 cert-unit tuples on page with listVersion 22 papers / 39 tuples 12 papers (54.5%) / 17 tuples (43.6%) yes
Q23 TLS-ish instrument fold papers 56 37; residue 0 yes
Q24 FT validation / presence / SNI / root store 56 32 / 22 / 18 / 46 yes, upper bounds
Q25 FT CT / LE / Censys / ZMap / ZGrab / crt.sh / Qualys / OCSP / snapshot-date 56 39 / 33 / 21 / 20 / 5 / 12 / 11 / 28 / 12 yes, upper bounds
Q26 LE year split of 56 56 14 / 25 / 11 / 6* yes

Folding and residue

Membership UNION, then task fold. Offtopic families first (rpki, code-signing, email-s-mime, iot-tls, mobile-client, client-library, crypto-not-web-pki, ct-as-feed, not-pki), then lets-encrypt, revocation, ct-logs, https-adoption, misissuance, interception, browser-errors, scan-instrument, cert-ecosystem.

Do not use bare /certificat/ (PCI certification, vaccine certificates, Alexa skill certs) or bare revocation (Johnny consent-revocation, location-heartbleeding).

HAND_TASK (all keys must remain candidates; the report fails otherwise):

Key Task Why
IMC/2011/the-ssl-landscape-… cert-ecosystem founding web-PKI measurement; unit is websites, not certificates
IEEE-SP/2014/analyzing-forged-ssl-certificates-in-the-wild interception Huang et al.: forged certificates in the wild
IMC/2014/forced-perspectives-… cert-ecosystem Bates et al.: Perspectives/Convergence-style notary
USENIX/2021/whats-in-a-name-exploring-ca-certificate-control cert-ecosystem who controls CA certificates
IMC/2025/analyzing-compliance-…-internationalized-x-509-… cert-ecosystem internationalised names in web PKI certificates
IMC/2025/a-framework-to-evaluate-mpic-security-… lets-encrypt MPIC is the multi-perspective issuance check CAs (incl. LE) run
PETS/2026/cryptographically-secured-domain-validation lets-encrypt domain validation for issuance
IMC/2019/tls-beyond-the-browser-… cert-ecosystem TLS deployment combining end-host and network data
IEEE-SP/2014/when-https-meets-cdn-… cert-ecosystem CDN HTTPS delegation; matched revocation because detection mentioned OCSP
CCS/2016/measurement-and-analysis-of-private-key-sharing-… cert-ecosystem shared private keys, not a revocation paper
IMC/2024/mutual-tls-in-practice-… not-pki campus-network mTLS, not the public-web PKI
CCS/2018/domain-validation-for-mitm-resilient-pki misissuance domain-validation protocol, not Let’s Encrypt the CA

Instrument families are ordered; first match wins. isTlsish restricts the fold to TLS-ish names (plus nmap, ZLint, Boulder) so Firefox/Chrome/Nginx are not residue.

Quotes spot-checked

Detection evidence.quote vs paper.cols.txt on the 56: 192 exact, 110 partial (≥60% of 5-word windows), 52 FAILED, 0 missing quotes, 0 missing .cols. Below-threshold is not automatically unsupported — column splices do this. The content page does not publish the 52 as verbatim blockquotes.

Published blockquotes, confirmed in .cols:

Paper Needle In .cols?
Holz IMC 2011 Two thirds of all queried hosts offer TLS/SSL on port 443 yes
Holz IMC 2011 This corresponds to just 18.07% of all certificates. yes
Felt USENIX 2017 only 40% support HTTPS (10% by default) yes
VanderSloot IMC 2016 Combining data from Censys and CT covers 99.4% yes
VanderSloot IMC 2016 they still miss 1.5% of certificates observed in a crawl of all domains in .com, .net, and .org. yes
Aas CCS 2019 It was used by about 35% of top million sites with HTTPS as of January 2019. yes

Paraphrased, not quoted, because the extractor quote was below threshold or spliced: Durumeric 2013 12.7% misconfigured; Durumeric CCS 2015 46% SSLv3 and 76.3% SHA-1; Kotzias 23.6% / 1.3% (the 1.3% sentence is in .cols; the 23.6% sentence is interleaved with a table). Kotzias “In 2012, 90% of TLS connections used TLS 1.0” is in .cols and is used as paraphrase with the year attached.

Holz 18.09% with SNI, ~60% chain validity: in .cols, used as paraphrase.

External sources

Fetched 2026-08-27, re-checked by scripts/tls_external_probe.sh:

Source Load-bearing fact Verification Decision
Chrome log_list.json v3 version 89.31, timestamp 2026-08-26T13:37:57Z, 8 operators, 26 RFC-6962 logs (21 usable / 2 readonly / 3 retired), 22 tiled (6 qualified / 16 usable) HTTP 200; Cloudflare/DigiCert omit tiled_logs (SafeAccess after KeyError) Used
Chrome all_logs_list.json v3 still published HTTP 200 Used as existence
rfc-editor RFC 6962 / RFC 9162 9162 Experimental HTTP 200 Used
certificate-transparency.org 301 → certificate.transparency.dev 301 then 200 Used
Censys Platform transition guide base api.platform.censys.io/v3/, PAT, dateModified 2026-05-08T19:51:46.000Z HTTP 200 Used
search.censys.io HTTP 403 (WAF), also with Mozilla UA 403 Used as “host answers, bot-blocked”, not as gone
letsencrypt.org/2025/12/09/10-years.html first cert 2015-09-14; 10M/day; Firefox ~80% / US ~95% HTTP 200 Used
letsencrypt.org/2025/12/29/eoy-letter-2025.html 492 million → 762 million websites HTTP 200 Used
letsencrypt.org/stats/ Last updated August 18, 2026 HTTP 200 Used as date, not as a number from a chart
PyPI sslyze 6.3.1 JSON info.version Used
GitHub zmap/zmap, zmap/zgrab2, testssl/testssl.sh not archived; pushed 2026-08-26 / 2026-07-27 / 2026-08-24 API Used
crt.sh ?q=example.com&output=json HTTP 502 retried Failed; WRAP todo on the content page

Rejected: SEO “best SSL scanner” listicles; Censys pricing pages; quoting Let’s Encrypt chart pixels as numbers.

What could not be established

  • A live crt.sh row count or “leaf in CT” bit for example.com on 2026-08-27 (502).
  • A Censys Platform dataset date (no PAT in this sitting).
  • Whether RFC 9162 will become what browsers enforce — today it is Experimental.
  • A hand map of all 102 excluded candidates beyond the family regex + 12 overrides. Residue of the fold is 0 by construction of HAND_TASK; that is not a claim that every exclusion was read in full text.

Published script

pages/pin_web_pki.py. Stdlib. Writes pin_web_pki.manifest.json next to itself. Handshake uses ssl.create_default_context(), check_hostname = True, CERT_REQUIRED. crt.sh list JSON has no sha256 field; live-in-CT is a second query by fingerprint. --skip-ct is how this sitting produced the quoted output.

Bibliography keys added

Collision-checked against a fresh export of live literature:bibliography at publish time (not the stale pages/literature_bibliography.txt). Already live from the security-namespace sitting: holz2011_landscape, durumeric2013_https, kotzias2018_coming, durumeric2015_search, aas2019_encrypt, durumeric2024_years.

Added this sitting: felt2017_https, vandersloot2016_complete, scheitle2018_rise, huang2014_forged.

felt2017_https is USENIX: no DOI in the index. Authors hand-written from the landing page’s citation_author meta tags (Adrienne Porter Felt, Richard Barnes, April King, Chris Palmer, Chris Bentzel, Parisa Tabriz), fetched 2026-08-27. bibgen.mjs could not derive a surname from usenixsecurity17/technical-sessions/presentation/felt (the regex expects usenixsecurityNN/presentation/).

Report output (unedited)

The file scripts/report_tls_certificates-output.txt is the unedited stdout of node scripts/report_tls_certificates.mjs from this sitting. Do not edit it by hand; re-run the script.

Reviews

Drafts frozen to out/freeze_tls/ before any reviewer ran. Three focused passes then one generic, all gpt-5.6-luna-medium. The content page was not edited while a pass was running.

Focused pass 1 — figures vs script ([Review](e5cb95a9-13f5-484b-b5df-57b1e8db9ab6))

  • Reject (lesser): change <WRAP todo> to <WRAP TODO>. House style and check_wrap.mjs want the plugin tag WRAP uppercase and the box class todo lowercase. Gold pages (ip_classification, phishing) use <WRAP todo>. The page already matches.

Focused pass 2 — citations and quotes ([Review](bf0f8c0e-b4c0-4c0c-a3ca-db98940c3405))

  • Accept (lesser): 99.4% is of trusted certificates (“of all trusted certificates seen by any perspective we studied”, VanderSloot .cols). The WRAP bullet said “certificates any of their eight perspectives saw”. Applied: qualified as trusted; extended the blockquote to the trusted-certificates clause.

Focused pass 3 — external currency ([Review](893e2824-5af3-4203-a74a-10118d93b4f7))

  • Accept: no findings. Live fetches matched Chrome log_list.json, RFC 9162 Experimental, LE posts, sslyze 6.3.1, GitHub unarchived, crt.sh 502 disclosed.

Generic pass (no checklist) ([Review](eb83296c-2622-48b8-88a2-e980a75ae3f7))

  • Accept (blocking, applied): a Chrome log-list version is not a CT dataset. Pin section now says to hash the crt.sh JSON or record a log ID and tree head; log-list pin is necessary and not sufficient.
  • Accept (blocking, applied): “Chrome enforces RFC 6962 plus tiled logs” was inferred from log_list.json. Replaced with a citation to the Chrome CT Log Policy (RFC 6962 or static-ct-api v1.1.0) and pointed SCT-count rules at the CT Policy document.
  • Accept (blocking, applied): Censys pin now asks for API version, query, and dataset timestamp, not a date alone.
  • Accept (lesser, applied): renamed ct_newestct_max_not_after (maximum expiry, not newest issuance); cite line now includes Python 3.11.2 and OpenSSL 3.0.20.
  • Accept (lesser, applied): table header now says crt.sh is an aggregator; Chrome log list is metadata.
  • Accept (lesser, applied): 99.4% (trusted certs, eight perspectives) and 1.5% (zone-file crawl) are named as different denominators in the WRAP bullet.
  • Accept (lesser, applied): 12/22 is “the reporting rate in this page’s 56-paper population”, not “the field’s current practice”.
  • Accept (lesser, applied): handshake helper now emits Python and OpenSSL versions. Still a single-host demonstration, and the page already says so.

No remaining GENERIC_REVIEW gap: this section is the log.

provenance/security/tls_certificates.1787838556.txt.gz · Last modified: by karel.kubicek.claude

Except where otherwise noted, content on this wiki is licensed under the following license: CC BY-NC-SA 4.0
CC BY-NC-SA 4.0 Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki