| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| provenance:artifacts [2026/08/27 13:24] – Freeze snapshot revs before review. Authored by Claude. karel.kubicek.claude | provenance:artifacts [2026/08/27 13:37] (current) – Generic-review accept/reject log; regenerated host figures. Authored by Claude. karel.kubicek.claude |
|---|
| |
| * Run date: 2026-08-27 (UTC). | * Run date: 2026-08-27 (UTC). |
| * Authoring agent: Cursor Grok 4.6, executing the drain item ''artifacts (new; namespace page with real content)'' as ''cursor-drain-artifacts'' (item 180, run 56). Not via ''claude -p'' / ''drain-sandbox.sh''. Review requested: three focused passes on GPT 5.6 Luna medium, generic pass on GPT 5.6 Luna max (''gpt-5.6-luna-max''). Available Task slugs this session: inherit, claude-opus-5-thinking-high, composer-2.5-fast, cursor-grok-4.5-high, cursor-grok-4.6-high, gpt-5.6-sol-medium. ''gpt-5.6-luna-max'' and luna@medium are not in that list. All four passes therefore run on ''gpt-5.6-sol-medium'' — same recorded substitution as programming:tranco and programming:docker this day, not a silent swap. | * Authoring agent: Cursor Grok 4.6, executing the drain item ''artifacts (new; namespace page with real content)'' as ''cursor-drain-artifacts'' (item 180, run 56). Not via ''claude -p'' / ''drain-sandbox.sh''. Review requested: three focused passes on GPT 5.6 Luna medium, generic pass on GPT 5.6 Luna max (''gpt-5.6-luna-max''). Available Task slugs this session: inherit, claude-opus-5-thinking-high, composer-2.5-fast, cursor-grok-4.5-high, cursor-grok-4.6-high, gpt-5.6-sol-medium. ''gpt-5.6-luna-max'' and luna@medium are not in that list. All four passes therefore run on ''gpt-5.6-sol-medium'' — same recorded substitution as programming:tranco and programming:docker this day, not a silent swap. The generic pass ran after the focused findings were applied. |
| * Corpus at run time: 5,859 extracted papers, 2010–2026, CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf, IEEE S&P. Read-only inputs under ''/workspace/publications_dataset/data/''. | * Corpus at run time: 5,859 extracted papers, 2010–2026, CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf, IEEE S&P. Read-only inputs under ''/workspace/publications_dataset/data/''. |
| * Read first: ''data/extract/OVERVIEW.md'', the extraction schema's ''Artifacts'' object, ''statistics:study_preregistration'' (already quotes the 3,321 figure), ''practices:ethics'', ''design:website_selection'', ''start'', USENIX/S&P/CCS/NDSS/WWW/IMC/PETS live AE pages. | * Read first: ''data/extract/OVERVIEW.md'', the extraction schema's ''Artifacts'' object, ''statistics:study_preregistration'' (already quotes the 3,321 figure), ''practices:ethics'', ''design:website_selection'', ''start'', USENIX/S&P/CCS/NDSS/WWW/IMC/PETS live AE pages. |
| Year buckets from ''lib.mjs'' ''YEAR_BUCKETS'' (both ends bounded; last starred): 23.7% / 38.4% / 50.6% / 65.0% / 76.5%*. | Year buckets from ''lib.mjs'' ''YEAR_BUCKETS'' (both ends bounded; last starred): 23.7% / 38.4% / 50.6% / 65.0% / 76.5%*. |
| |
| USENIX 2025–2026: 293 papers, 276 own-link (94.2%), Zenodo 219 (74.7% of the 293). | USENIX 2025–2026: 293 papers, 276 own-link (94.2%), Zenodo 217 (74.1% of the 293). |
| |
| ===== Folding and residue ===== | ===== Folding and residue ===== |
| |
| ''scripts/host_fold.mjs''. Ordered families, first match wins, DOI prefixes before a catch-all ''doi.org'' / ''dx.doi.org'' row. Self-tests run on import. A paper matching GitHub and Zenodo is in both; union of archival families is 622 / 3,321 = 18.7%. | ''scripts/host_fold.mjs''. Ordered families, first match wins, DOI prefixes before a catch-all ''doi.org'' / ''dx.doi.org'' row. Self-tests run on import. A paper matching GitHub and Zenodo is in both; union of archival families is 617 / 3,321 = 18.6%. DOI-prefix matching applies only to doi.org / dx.doi.org / ''doi:'' URLs. |
| |
| Exact-string undercount: a regex on ''github.com'' without gist / raw / ''%%*.github.io%%'' reported 1,785 GitHub papers; the fold reports **1,896**. The year/venue tables use the fold, not the regex. | Exact-string undercount: a regex on ''github.com'' without gist / raw / ''%%*.github.io%%'' reported 1,785 GitHub papers; the fold reports **1,896**. The year/venue tables use the fold, not the regex. DOI-prefix matching (Zenodo ''10.5281'', OSF ''10.17605'', …) applies only to doi.org / dx.doi.org / ''doi:'' URLs, not to a GitHub path that happens to contain those digits. |
| |
| Unparsed authors'-own URLs after teaching the fold ''doi:10.5281/zenodo.N'': **0**. The report throws if that is not zero. | Unparsed authors'-own URLs after teaching the fold ''doi:10.5281/zenodo.N'': **0**. The report throws if that is not zero. |
| |
| Residue: **667 distinct unmapped hosts**, 870 paper-host pairs, 47 hosts with ≥3 papers (printed in the report). The long tail is lab pages. Full list: ''node scripts/report_artifacts.mjs --residue''. Not dropped. | Residue: **669 distinct unmapped hosts**, 875 paper-host pairs, 48 hosts with ≥3 papers (printed in the report). High-count hosts are mostly lab pages; the tail includes pastebins, form hosts, and registries. Full list: ''node scripts/report_artifacts.mjs --residue''. Not dropped. |
| |
| Rejected probes: | Rejected probes: |
| * Whether promised-not-yet-available (289) later appeared. Would need a second pass over camera-ready PDFs vs submission PDFs. | * Whether promised-not-yet-available (289) later appeared. Would need a second pass over camera-ready PDFs vs submission PDFs. |
| * CHI / SOUPS artifact culture. Out of corpus. | * CHI / SOUPS artifact culture. Out of corpus. |
| | * Demir et al. 2022 has no ''paper.cols.txt'' in this mount (only ''fetch.json''). The 117 / 18 / 4.5M / C9 67% / results-open 24% figures were checked against a fetched copy of the paper, not against ''.cols''. |
| |
| ===== Report script, unedited ===== | ===== Report script, unedited ===== |
| --------------------------------------------- -------------- ------ ------------- | --------------------------------------------- -------------- ------ ------------- |
| GitHub mutable-repo 1896 57.1% | GitHub mutable-repo 1896 57.1% |
| Zenodo archival 436 13.1% | Zenodo archival 431 13.0% |
| Google Sites (project page) project-page 157 4.7% | Google Sites (project page) project-page 157 4.7% |
| OSF archival 112 3.4% | OSF archival 112 3.4% |
| Software Heritage archival 2 0.1% | Software Heritage archival 2 0.1% |
| |
| Sum of family counts = 3164, union of mapped papers = 2724 / 3321. A paper with GitHub AND Zenodo is in both rows. | Sum of family counts = 3159, union of mapped papers = 2722 / 3321. A paper with GitHub AND Zenodo is in both rows. |
| Union of archival families (Zenodo, OSF, Figshare, Dryad, Dataverse, 4TU, Software Heritage): 622 / 3321 = 18.7%. | Union of archival families (Zenodo, OSF, Figshare, Dryad, Dataverse, 4TU, Software Heritage): 617 / 3321 = 18.6%. |
| |
| Unmapped hostnames (residue), hosts with ≥3 own-link papers: | Unmapped hostnames (residue), hosts with ≥3 own-link papers: |
| pastebin.com 5 | pastebin.com 5 |
| personalization.ccs.neu.edu 5 | personalization.ccs.neu.edu 5 |
| | 10.0.20.161 4 |
| addons.mozilla.org 4 | addons.mozilla.org 4 |
| caida.org 4 | caida.org 4 |
| traces.simpleweb.org 3 | traces.simpleweb.org 3 |
| |
| Residue: 667 distinct unmapped hosts, 870 paper-host pairs. 47 hosts have ≥3 papers (printed). Run with --residue for the full host list. The long tail is lab / university project pages. | Residue: 669 distinct unmapped hosts, 875 paper-host pairs. 48 hosts have ≥3 papers (printed). Run with --residue for the full host list. The long tail is lab / university project pages. |
| |
| ## E. GitHub vs Zenodo over time (of own-link papers in the year) | ## E. GitHub vs Zenodo over time (of own-link papers in the year) |
| 2022 546 338 61.9% 234 69.2% 5 1.5% | 2022 546 338 61.9% 234 69.2% 5 1.5% |
| 2023 719 475 66.1% 321 67.6% 21 4.4% | 2023 719 475 66.1% 321 67.6% 21 4.4% |
| 2024 690 457 66.2% 307 67.2% 42 9.2% | 2024 690 457 66.2% 307 67.2% 41 9.0% |
| 2025* 770 592 76.9% 318 53.7% 228 38.5% | 2025* 770 592 76.9% 318 53.7% 224 37.8% |
| 2026* 415 315 75.9% 202 64.1% 132 41.9% | 2026* 415 315 75.9% 202 64.1% 132 41.9% |
| |
| ------- ------ -------- ----- ------------------ ------------------ | ------- ------ -------- ----- ------------------ ------------------ |
| CCS 990 467 47.2% 252 25 | CCS 990 467 47.2% 252 25 |
| IEEE-SP 767 423 55.1% 270 12 | IEEE-SP 767 423 55.1% 270 11 |
| IMC 638 313 49.1% 163 12 | IMC 638 313 49.1% 163 12 |
| NDSS 701 443 63.2% 310 102 | NDSS 701 443 63.2% 310 100 |
| PETS 510 258 50.6% 158 5 | PETS 510 258 50.6% 158 5 |
| USENIX 1410 980 69.5% 460 233 | USENIX 1410 980 69.5% 460 231 |
| WWW 843 437 51.8% 283 47 | WWW 843 437 51.8% 283 47 |
| |
| USENIX 2025–2026 (open-science policy years): 293 papers, 276 own-link (94.2%), Zenodo 219 (79.3% of own-link; 74.7% of USENIX 2025–2026), GitHub 82. | USENIX 2025–2026 (open-science policy years): 293 papers, 276 own-link (94.2%), Zenodo 217 (78.6% of own-link; 74.1% of USENIX 2025–2026), GitHub 82. |
| |
| ## G. Schema badge field — almost empty | ## G. Schema badge field — almost empty |
| USENIX/2010/scantegrity-ii-municipal-election-at-takoma-park-the-first-e2e-binding-governmen | USENIX/2010/scantegrity-ii-municipal-election-at-takoma-park-the-first-e2e-binding-governmen |
| All source code was released under the GPLv2 software license. | All source code was released under the GPLv2 software license. |
| CCS/2012/operating-system-framed-in-case-of-mistaken-identity-measuring-the-success-of-we | |
| Go to: http://saucers.cups.cs.cmu.edu/yacot/mnt/wtk/survey.php?i=workerID | |
| IMC/2012/confused-timid-and-unstable-picking-a-video-streaming-rate-is-hard | |
| RTMPDump. http: //rtmpdump.mplayerhq.hu/. | |
| USENIX/2012/enemy-of-the-state-a-state-aware-black-box-web-vulnerability-scanner | |
| GARGOYLESOFTWARE INC. HtmlUnit. http://htmlunit.sourceforge.net/. | |
| WWW/2012/leveraging-user-comments-for-aesthetic-aware-image-search-reranking | |
| DPChallenge1 ... http://www.dpchallenge.com | |
| USENIX/2014/exit-from-hell-reducing-the-impact-of-amplification-ddos-attacks | |
| NETWORK MAPPER. http://nmap.org/, 2014. | |
| USENIX/2018/debloating-software-through-piece-wise-compilation-and-loading | |
| using ROPgadget [33]. | |
| NDSS/2019/the-crux-of-voice-insecurity-a-brain-study-of-speaker-legitimacy-detection | |
| Festvox, http://festvox.org/, 2014, accessed:05-11-2018. | |
| WWW/2019/mobile-app-risk-ranking-via-exclusive-sparse-coding | |
| CMU privacygrad4 . https://www.hcii.cmu.edu/research/privacygrade | |
| NDSS/2021/more-than-a-fair-share-network-data-remanence-attacks-against-secret-sharing-based-schemes | |
| Available: https://www.opennetworking.org/wp-content/uploads/2014/10/openflow-spec-v1.3.0.pdf | |
| IEEE-SP/2022/transfer-attacks-revisited-a-large-scale-empirical-study-in-real-computer-vision | |
| Code & Results: https://github.com/AlgebraLoveme/Transfer-Attacks-Revisited-A-Large-Scale-Empirical-Study-in-Real-Computer-Vision-Settings | |
| USENIX/2023/lalaine-measuring-and-characterizing-non-compliance-of-apple-privacy-labels | |
| We will release Lalaine. | |
| USENIX/2024/i-chose-to-fight-be-brave-and-to-deal-with-it-threat-experiences-and-security-pr | |
| https://www.usenix.org/conference/usenixsecurity24/presentation/gröber-content-creators | |
| USENIX/2024/knowphish-large-language-models-meet-multimodal-knowledge-graphs-for-enhancing-r | |
| Knowphish github repository. https://github.com/imethanlee/KnowPhish. | |
| USENIX/2024/whisperfuzz-white-box-fuzzing-for-detecting-and-locating-timing-vulnerabilities | |
| Synopsys VCS. https://www.synopsys.com/verification/simulation/vcs.html, 2022. | |
| IMC/2025/from-webgl-to-webgpu-a-reality-check-of-browser-based-gpu-acceleration | |
| 2024. Emscripten Documentation. https://emscripten.org/docs/getting_started/index.html. | |
| IMC/2025/unlocking-crowdsourced-propagation-measurements-accuracy-guarantees-for-mobile-p | |
| G-NetTrack Lite. https://play.google.com/store/apps/details?id=com.gyokovsolutions.gnettracklite. (2023). | |
| USENIX/2025/gnss-wasp-gnss-wide-area-spoofing | |
| Artifacts available at https://zenodo.org/records/14734238. | |
| PETS/2026/i-don-t-think-it-needs-to-be-political-privacy-experiences-and-concerns-of-femhe | |
| see Appendix §A.1 | |
| NDSS/2026/rtrace-towards-better-visibility-of-shared-library-execution | |
| We open-source RTrace. | |
| IEEE-SP/2024/please-tell-me-more-privacy-impact-of-explainability-through-the-lens-of-members | |
| "Cifar dataset," https://www.cs.toronto.edu/∼kriz/cifar.html, 2012. | |
| |
| ## Y. Arithmetic-derived figures the page may print | ## Y. Arithmetic-derived figures the page may print |
| Demir et al. TheWebConf 2022 (the paper this page tells you to read first): 117 web-measurement | Demir et al. TheWebConf 2022 (the paper this page tells you to read first): 117 web-measurement |
| papers, 18 reproducibility criteria, 4.5-million-page experiment. | papers, 18 reproducibility criteria, 4.5-million-page experiment. |
| | C9 crawler-publicly-available omitted by 67%; results openly available 24%. |
| |
| TheWebConf 2026 artifact badging (www2026.thewebconf.org/calls/artifact-badging.html, 2026-08-27): | TheWebConf 2026 artifact badging (www2026.thewebconf.org/calls/artifact-badging.html, 2026-08-27): |
| * rendered ''https://measuretheweb.org/artifacts'' after purge: 0 "could not be found", ''bibtex_citekey'' present (34 markers), all 9 keys resolve, no leftover ''{[...]}'' | * rendered ''https://measuretheweb.org/artifacts'' after purge: 0 "could not be found", ''bibtex_citekey'' present (34 markers), all 9 keys resolve, no leftover ''{[...]}'' |
| |
| Findings and accept/reject land in the table below after the three focused passes and the generic pass. There is no checklist for the generic pass. Requested models luna@medium / ''gpt-5.6-luna-max'' were unavailable; passes run on ''gpt-5.6-sol-medium'' as recorded in the run record. | Findings and accept/reject. Reviewers: three focused passes then one generic pass, all on ''gpt-5.6-sol-medium'' (requested luna@medium / ''gpt-5.6-luna-max'' unavailable). |
| | |
| | ^ Pass ^ Finding ^ Decision ^ |
| | | figures | 622 "archival host" includes 34 unverified doi.org resolver URLs | **accepted.** Intro now splits named archives from other-DOI. | |
| | | figures | ''--require-doi'' accepted a generic ACM publisher DOI | **accepted.** Generic doi.org is now ''unknown'', not archival. | |
| | | figures | GitHub path containing ''10.5281'' classified as Zenodo | **accepted.** DOI-prefix matching only on doi.org / ''doi:'' URLs. | |
| | | figures | ''zenodo.org/not-a-record'' exited 0 under ''--require-doi'' | **accepted.** No record id → unknown, exit 1. | |
| | | figures | ''rebrand.ly'' / ''shorturl.at'' in host_fold but not pin_artifact | **accepted.** Shortener set synced. | |
| | | figures | "219 of 293 USENIX papers that year" is 2025–2026 combined | **accepted.** Wording fixed. | |
| | | figures | "almost all of the Zenodo mass (233 of 436)" is 53.4% | **accepted.** "more than half". | |
| | | citations | Bertram quote spans a column splice in ''.cols'' | **accepted.** Shortened to the contiguous //"We cannot directly reveal a sample mapping"//. | |
| | | citations | Thomas quote spans a column splice; "aggregates were the artifact" unsupported | **accepted.** Quote shortened; sentence now "Only aggregates could be retained." | |
| | | citations | Lyons quote capitalisation; reason was "Device identifiers" | **accepted.** Verbatim footnote on re-identification. | |
| | | citations | Demir: failures are not "configuration, not the repository" (C9 67% omitted; results open 24%) | **accepted.** Sentence rewritten from the paper. | |
| | | citations | Demir ''paper.cols.txt'' missing from the corpus mount | **rejected as a page fix.** Mount is read-only; 117 / 18 / 4.5M verified from the fetched PDF. Logged under what could not be established. | |
| | | external | CCS row claimed a mandatory open-science appendix | **accepted.** CCS AE is optional; appendix is encouraged after evaluation. | |
| | | external | USENIX "DOI-backed archive required" overstates "long-term stable reference or DOI" | **accepted.** Table aligned with the badges page. | |
| | | external | PETS row headed "2026 rule" while live page is PoPETs 2027 | **accepted.** Column retitled; cell names 2027. | |
| | | generic | host_fold DOI-prefix still matched GitHub paths containing ''10.5281'' (pin_artifact was already fixed) | **accepted.** Prefix matching restricted to doi.org / ''doi:'' in host_fold too; self-test added; figures regenerated. | |
| | | generic | ''--require-doi'' accepted ''osf.io/'' and ''figshare.com/account'' | **accepted.** Empty / account / login paths on named archives are now unknown. | |
| | | generic | parse() stripped spaces from URLs | **accepted.** Whitespace now raises. | |
| | | generic | "minimum to regenerate the headline" omitted analysis scripts / env pin; column 2 was only links | **accepted.** Added the analysis-pipeline row; column 2 is now a reason. | |
| | | generic | "What you can always release" contradicts dual-use | **accepted.** "often still"; Kim cited. | |
| | | generic | Causal "the jump is USENIX policy" / "optional AE does not produce deposits" | **accepted.** "coincides with"; last sentence weakened. | |
| | | generic | "GitHub's default branch has neither" a licence; "wrong to cite" | **accepted.** Distinguished licence file vs DOI vs Available deposit. | |
| | | generic | Run record said all four passes ran while the log said generic had not | **accepted.** Run record and this table now agree. | |
| | | generic | FAILED-quote dump uncapped when the 5-word window list was empty | **accepted.** Cap is 15 in both branches. | |
| | | generic | "almost all" 667 residue hosts are lab pages, unmeasured | **accepted.** High-count hosts vs tail. | |
| | | generic | Reclassify the 34 other-DOI resolver URLs as non-archival | **rejected.** They are institutional deposit DOIs via doi.org; the named-archive split is already disclosed. Changing permanence would pretend we resolved each DOI. | |
| |
