User Tools

Site Tools


provenance:artifacts

This is an old revision of the document!


Provenance: Artifacts

Back to Artifacts. Corpus-wide selection and extraction notes are on corpus. This is the page-specific query log.

Run record

  • Run date: 2026-08-27 (UTC).
  • Authoring agent: Cursor Grok 4.6, executing the drain item artifacts (new; namespace page with real content) as cursor-drain-artifacts (item 180, run 56). Not via claude -p / drain-sandbox.sh. Review requested: three focused passes on GPT 5.6 Luna medium, generic pass on GPT 5.6 Luna max (gpt-5.6-luna-max). Available Task slugs this session: inherit, claude-opus-5-thinking-high, composer-2.5-fast, cursor-grok-4.5-high, cursor-grok-4.6-high, gpt-5.6-sol-medium. gpt-5.6-luna-max and luna@medium are not in that list. All four passes therefore run on gpt-5.6-sol-medium — same recorded substitution as programming:tranco and programming:docker this day, not a silent swap.
  • Corpus at run time: 5,859 extracted papers, 2010–2026, CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf, IEEE S&P. Read-only inputs under /workspace/publications_dataset/data/.
  • Read first: data/extract/OVERVIEW.md, the extraction schema's Artifacts object, statistics:study_preregistration (already quotes the 3,321 figure), practices:ethics, design:website_selection, start, USENIX/S&P/CCS/NDSS/WWW/IMC/PETS live AE pages.
  • Target page had no revision. ?do=export_raw on artifacts returned the HTML error page. This is a creation. start and Website selection already red-link it; that is the overlap judgement: create the promised namespace page rather than broaden ethics or preregistration.
  • No write to the publication mount. Wiki saves through scripts/dw.mjs. New pages have no –if-rev; start and literature:bibliography do.

Why this page, not an overlap

Neighbour What it already answers What it does not
study_preregistration preregistration is not a badge; 3,321 own-link vs 15 preregistered hosts, kinds, venue AE, what to put in a crawl artefact
ethics IRB, harm, dual-use, crawler identification the release decision, Zenodo vs GitHub, Available-badge rules
website_selection why pin a list how to publish the pinned list
crawler how to crawl how to archive the crawler

The item asked for a namespace page with real content: badge policies, Zenodo DOIs, personal-data release. No child pages are planned. A stub namespace outline would have been the contributing default; the item overrode it.

Population and queries

All counts are papers unless labelled tuples. artifacts === null (320 papers) is a schema-level “said nothing”, distinct from availability === none-mentioned (2,183). Direct field access after that branch; no .get().

Query Denominator Result
artifacts.links[] with belongsToAuthors === true, distinct papers 5,859 3,321 (56.7%)
same, empirical 5,118 2,872 (56.1%)
same, crawled 1,120 684 (61.1%)
availability stated (not null, not none-mentioned), empirical 5,118 2,890 (56.5%) — OVERVIEW.md's row
availability == public but no own-link 5,859 54
own-link but availability not public 3,321 530
artifacts.badge non-null 5,859 15 (0.3%)
kind == preregistration, own-link papers 3,321 12 — schema signal; the prereg page's 15 is the full-text count
posters ∪ ≤4-page records dropped 5,859 → 5,608 own-link 56.7% → 58.4%
data/extract/artifact_links.jsonl rows 4,322. STALE. Not used. The script refuses if that file ever matches the current corpus size, so a refresh cannot silently start using it.

Year buckets from lib.mjs YEAR_BUCKETS (both ends bounded; last starred): 23.7% / 38.4% / 50.6% / 65.0% / 76.5%*.

USENIX 2025–2026: 293 papers, 276 own-link (94.2%), Zenodo 219 (74.7% of the 293).

Folding and residue

scripts/host_fold.mjs. Ordered families, first match wins, DOI prefixes before a catch-all doi.org / dx.doi.org row. Self-tests run on import. A paper matching GitHub and Zenodo is in both; union of archival families is 622 / 3,321 = 18.7%.

Exact-string undercount: a regex on github.com without gist / raw / *.github.io reported 1,785 GitHub papers; the fold reports 1,896. The year/venue tables use the fold, not the regex.

Unparsed authors'-own URLs after teaching the fold doi:10.5281/zenodo.N: 0. The report throws if that is not zero.

Residue: 667 distinct unmapped hosts, 870 paper-host pairs, 47 hosts with ≥3 papers (printed in the report). The long tail is lab pages. Full list: node scripts/report_artifacts.mjs –residue. Not dropped.

Rejected probes:

  • Using artifact_links.jsonl as the URL source. OVERVIEW.md recommends it. It is the 4,322-paper corpus. Using it would have silently dropped every 2025–2026 paper, which is exactly where Zenodo moved.
  • Publishing 15/5,859 as a badge-award rate. The field is empty because papers do not write the badge name, not because venues do not award badges. NDSS 2026's own results page (114 / 112 / 97 / 70) is the source for that venue.
  • Unioning schema kind == preregistration (12) with the prereg page's 15. They disagree on purpose; this page points at that one.

Evidence quotes checked

5,539 papers with an artifacts record. artifacts.evidence.quote against paper.cols.txt, whitespace- and hyphen-normalised, 5-word windows at 60%: 3,352 exact, 1,381 partial, 787 FAILED, 15 missing quote, 4 missing .cols.

The first 15 FAILED rows were read. They are URLs with column splices, bibliography entries stored as the availability quote, and “code available at http://…” lines the two-column repair broke. Same class quote_check.mjs warns about. None looked fabricated. No FAILED quote is used on the content page. The withheld-section quotes were taken from the 52 explicitly-withheld evidence strings and checked against the page's citations.

External and industry verification

Source Load-bearing fact Verification 2026-08-27 Decision
secartifacts.github.io/usenixsec2026/badges GitHub/GitLab/personal pages not acceptable for Available; Zenodo recommended; Available mandatory fetched; external_checks_artifacts.sh Used
USENIX Security 2026 CFP open-science appendix at submission fetched Used
sp2026.ieee-security.org/cfartifacts.html Optional AE; Available needs DOI (Zenodo/FigShare/Dryad) fetched Used
www2026.thewebconf.org/calls/artifact-badging.html Available only; DOI required; GitHub alongside DOI fetched Used
conferences.sigcomm.org/imc/2026/cfp/ availability declaration; shepherding; community award fetched Used
petsymposium.org/artifacts.php optional Available/Functional/Reproduced fetched Used
secartifacts.github.io/ndss2026/results 114 / 112 / 97 / 70 fetched Used
www.sigsac.org/ccs/CCS2026/call-for/call-for-artifacts.html ACM v1.1 vocabulary; GitHub not adequate for Available; Zenodo recommended fetched Used
ACM Artifact Review and Badging v1.1 (acm.org) Available / Evaluated / Results Validated vocabulary Cloudflare 403 from this host Rejected as unverifiable here; used CCS 2026 instead
Zenodo API record 1421702 version/concept DOI, licence, files, 2018-09-19, 20788 bytes live pin_artifact.py Used as API demo, not as a measurement paper to imitate
github.com/mozilla/OpenWPM classified mutable-repo; –require-doi exits 1 live Used as the counterexample

Rejected: SEO “best places to share research data” listicles; GitHub's own “citing this repository” page as if it satisfied USENIX Available (it mints a Software Heritage link if you follow it — that is extra work, not automatic).

Judgement calls

  • Headline is own-link, not availability==public. They disagree by 54 + 530 papers. OVERVIEW.md's year column is own-link. Matching it keeps the site internally consistent.
  • Namespace page with real content, not an outline. The item said so. There are no children.
  • Did not re-litigate preregistration. 12 vs 15 is already documented next door.
  • Did not publish a badge-award rate from this corpus.
  • Zenodo 1421702 as the API demo is an arbitrary live record, chosen because the first dump hit it. The page says so. Using a “nice” measurement Zenodo would have implied endorsement of that deposit's contents.
  • start page one-liner updated in the same sitting so the new page is reachable with an accurate promise, not just a red link that turned blue.

What could not be established

  • How often each venue awards badges. Not in the extraction. NDSS 2026 publishes it; the others were not scraped into a table.
  • Whether GitHub URLs from 2016 still resolve. A live HEAD of 1,896 repos is a different item.
  • Whether promised-not-yet-available (289) later appeared. Would need a second pass over camera-ready PDFs vs submission PDFs.
  • CHI / SOUPS artifact culture. Out of corpus.
  • Demir et al. 2022 has no paper.cols.txt in this mount (only fetch.json). The 117 / 18 / 4.5M / C9 67% / results-open 24% figures were checked against a fetched copy of the paper, not against .cols.

Report script, unedited

scripts/report_artifacts.mjs output from the run that the content page was written against. Do not edit this block by hand; re-run the script.

corpus: 5859 papers, 7 venues, 2010-2026
population ALL:        5859
population EMPIRICAL:  5118
population CRAWLED:    1120
population OWN-LINK:   3321  (authors'-own artifact URL)
artifact_links.jsonl:  4322 rows — STALE (old corpus); not used
generated by scripts/report_artifacts.mjs

## A. Released an authors-own artifact link

Population  N     Authors-own link  Share
----------  ----  ----------------  -----
all papers  5859  3321              56.7%
empirical   5118  2872              56.1%
crawled     1120  684               61.1%

This is OVERVIEW.md's "Releases an artifact link" column and the 3,321 figure on statistics:study_preregistration. belongsToAuthors === true, paper-counted.

## B. artifacts.availability

Against all papers:
availability                Papers  Share of 5859
--------------------------  ------  -------------
public                      2845    48.6%
on-request                  91      1.6%
restricted                  79      1.3%
promised-not-yet-available  289     4.9%
explicitly-withheld         52      0.9%
none-mentioned              2183    37.3%
no artifacts record (null)  320     5.5%

Against empirical (OVERVIEW.md's 5,118 denominator):
availability                Papers  Share of 5118
--------------------------  ------  -------------
public                      2439    47.7%
on-request                  86      1.7%
restricted                  73      1.4%
promised-not-yet-available  240     4.7%
explicitly-withheld         52      1.0%
none-mentioned              1964    38.4%
no artifacts record (null)  264     5.2%

Stated (not null, not none-mentioned) among empirical: 2890 / 5118 = 56.5%. This is OVERVIEW.md's artifacts.availability row (2,890 / 56.5%).

## C. Kind of authors-own artifact (paper-counted, of own-link papers)

kind                          Papers  Share of 3321
----------------------------  ------  -------------
code-and-data                 1307    39.4%
source-code                   1175    35.4%
project-page                  508     15.3%
dataset                       448     13.5%
web-demo-or-service           296     8.9%
extended-version-or-appendix  236     7.1%
other                         94      2.8%
survey-instrument             88      2.6%
browser-extension             23      0.7%
mobile-app                    12      0.4%
preregistration               12      0.4%

Sum of kind counts = 4199, union = 3321. A paper with source-code AND dataset is in both rows. preregistration kind = 12 — the schema signal, not the full-text count on statistics:study_preregistration (15 papers preregistered; 12 of them have a preregistration-kind artifact link).

## D. Host family of authors-own links (paper-counted, of own-link papers)

Family                                         Permanence      Papers  Share of 3321
---------------------------------------------  --------------  ------  -------------
GitHub                                         mutable-repo    1896    57.1%
Zenodo                                         archival        436     13.1%
Google Sites (project page)                    project-page    157     4.7%
OSF                                            archival        112     3.4%
arXiv                                          publisher-page  97      2.9%
USENIX paper page                              publisher-page  91      2.7%
YouTube / Vimeo (demo)                         other           65      2.0%
URL shortener                                  shortener       59      1.8%
Google Drive / Docs                            mutable-repo    41      1.2%
anonymous.4open.science (double-blind GitHub)  anonymized      36      1.1%
other DOI repository                           archival        34      1.0%
Figshare                                       archival        28      0.8%
GitLab                                         mutable-repo    25      0.8%
IACR ePrint                                    publisher-page  24      0.7%
Chrome Web Store / Play Store                  other           15      0.5%
Bitbucket                                      mutable-repo    13      0.4%
Dropbox                                        mutable-repo    13      0.4%
4TU.ResearchData                               archival        8       0.2%
Hugging Face                                   mutable-repo    6       0.2%
Harvard Dataverse                              archival        4       0.1%
Dryad                                          archival        2       0.1%
Software Heritage                              archival        2       0.1%

Sum of family counts = 3164, union of mapped papers = 2724 / 3321. A paper with GitHub AND Zenodo is in both rows.
Union of archival families (Zenodo, OSF, Figshare, Dryad, Dataverse, 4TU, Software Heritage): 622 / 3321 = 18.7%.

Unmapped hostnames (residue), hosts with ≥3 own-link papers:
Host                                Papers
----------------------------------  ------
athinagroup.eng.uci.edu             9
publications.teamusec.de            9
ant.isi.edu                         8
scans.io                            7
vusec.net                           7
mediatum.ub.tum.de                  6
research.microsoft.com              6
securepki.org                       6
comsec.ethz.ch                      5
gfw.report                          5
pastebin.com                        5
personalization.ccs.neu.edu         5
addons.mozilla.org                  4
caida.org                           4
censoredplanet.org                  4
crysp.uwaterloo.ca                  4
cse.chalmers.se                     4
hub.docker.com                      4
moa-lab.net                         4
mobitec.ie.cuhk.edu.hk              4
owlink.org                          4
pdf-insecurity.org                  4
sand-project.nl                     4
adanalyst.mpi-sws.org               3
apps.facebook.com                   3
ar-sec.cs.washington.edu            3
arima.cylab.cmu.edu                 3
aspredicted.org                     3
blaseur.com                         3
cambridgecybercrime.uk              3
cmand.org                           3
code.google.com                     3
crypto.stanford.edu                 3
cs.ucsb.edu                         3
cs.uic.edu                          3
cs.umd.edu                          3
forms.gle                           3
geneva.cs.umd.edu                   3
gitee.com                           3
go.wisc.edu                         3
isi.edu                             3
moniotrlab.khoury.northeastern.edu  3
nymity.ch                           3
sandlab.cs.uchicago.edu             3
ter.ps                              3
thuir.cn                            3
traces.simpleweb.org                3

Residue: 667 distinct unmapped hosts, 870 paper-host pairs. 47 hosts have ≥3 papers (printed). Run with --residue for the full host list. The long tail is lab / university project pages.

## E. GitHub vs Zenodo over time (of own-link papers in the year)

Year   Papers  Own-link  Own-link share  GitHub (of own-link)  GitHub share  Zenodo (of own-link)  Zenodo share
-----  ------  --------  --------------  --------------------  ------------  --------------------  ------------
2010   119     29        24.4%           0                     0.0%          0                     0.0%
2011   116     28        24.1%           1                     3.6%          0                     0.0%
2012   151     37        24.5%           2                     5.4%          0                     0.0%
2013   125     27        21.6%           6                     22.2%         0                     0.0%
2014   166     56        33.7%           9                     16.1%         0                     0.0%
2015   190     71        37.4%           17                    23.9%         0                     0.0%
2016   182     72        39.6%           13                    18.1%         0                     0.0%
2017   231     96        41.6%           35                    36.5%         0                     0.0%
2018   254     118       46.5%           57                    48.3%         1                     0.8%
2019   402     176       43.8%           99                    56.3%         2                     1.1%
2020   404     206       51.0%           129                   62.6%         1                     0.5%
2021   379     228       60.2%           146                   64.0%         4                     1.8%
2022   546     338       61.9%           234                   69.2%         5                     1.5%
2023   719     475       66.1%           321                   67.6%         21                    4.4%
2024   690     457       66.2%           307                   67.2%         42                    9.2%
2025*  770     592       76.9%           318                   53.7%         228                   38.5%
2026*  415     315       75.9%           202                   64.1%         132                   41.9%

* 2025–2026 are provisional: CCS/IMC 2026 unheld; IEEE S&P/WWW 2026 incompletely selected.

Year buckets (YEAR_BUCKETS from lib.mjs):
Window      Papers  Own-link  Share
----------  ------  --------  -----
2010–2013   511     121       23.7%
2014–2017   769     295       38.4%
2018–2021   1439    728       50.6%
2022–2024   1955    1270      65.0%
2025–2026*  1185    907       76.5%

## F. Venue (all papers, then own-link papers' GitHub/Zenodo split)

Venue    Papers  Own-link  Share  GitHub of own-link  Zenodo of own-link
-------  ------  --------  -----  ------------------  ------------------
CCS      990     467       47.2%  252                 25
IEEE-SP  767     423       55.1%  270                 12
IMC      638     313       49.1%  163                 12
NDSS     701     443       63.2%  310                 102
PETS     510     258       50.6%  158                 5
USENIX   1410    980       69.5%  460                 233
WWW      843     437       51.8%  283                 47

USENIX 2025–2026 (open-science policy years): 293 papers, 276 own-link (94.2%), Zenodo 219 (79.3% of own-link; 74.7% of USENIX 2025–2026), GitHub 82.

## G. Schema badge field — almost empty

artifacts.badge non-null: 15 / 5859 = 0.3%.
badge string                                                              Papers  Keys
------------------------------------------------------------------------  ------  -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Available, Functional, Reproduced                                         2       IEEE-SP/2026/one-tap-to-hijack-them-all-a-security-analysis-of-the-google-fast-pair-protocol, IEEE-SP/2026/rain-transiently-leaking-data-from-public-clouds-using-old-vulnerabilities
Artifact evaluated: passed                                                1       USENIX/2020/big-numbers-big-troubles-systematically-analyzing-nonce-leakage-in-ec-dsa-implem
ARTIFACT EVALUATED: PASSED                                                1       USENIX/2020/halucinator-firmware-re-hosting-through-abstraction-layer-emulation
PASSED                                                                    1       USENIX/2020/montage-a-neural-network-language-model-guided-javascript-engine-fuzzer
Partially evaluated by the USENIX Security artifact evaluation committee  1       USENIX/2021/evaluating-in-workflow-messages-for-improving-mental-models-of-end-to-end-encryp
AEC-approved artifact                                                     1       NDSS/2024/facilitating-non-intrusive-in-vivo-firmware-testing-with-stateless-instrumentation
Available and Functional                                                  1       NDSS/2025/attributing-open-source-contributions-is-critical-but-difficult-a-systematic-analysis-of-github-practices-and-their-impact-on-software-supply-chain-security
AVAILABLE FUNCTIONAL REPRODUCED                                           1       IEEE-SP/2026/deepsurf-detecting-memory-safety-vulnerabilities-in-rust-through-fuzzing-llm-aug
artifact evaluation badges                                                1       USENIX/2026/privacyshield-relaying-ble-beacons-to-counter-unsolicited-tracking
Available badge                                                           1       NDSS/2026/understanding-the-status-and-strategies-of-the-code-signing-abuse-ecosystem
Available and Functional badges requested                                 1       NDSS/2026/mutato-enhancing-fuzz-drivers-with-adaptive-api-option-mutation
AVAILABLE FUNCTIONAL                                                      1       IEEE-SP/2026/banshee-target-switch-attacks-on-gimbal-stabilized-visual-tracking-systems-via-a
artifact-evaluated as reproducible                                        1       NDSS/2025/icsquartz-scan-cycle-aware-and-vendor-agnostic-fuzzing-for-industrial-control-systems
artifact available and artifact functional badge                          1       NDSS/2026/one-email-many-faces-a-deep-dive-into-identity-confusion-in-email-aliases

This is NOT a badge-award rate. Papers almost never write the badge name in a sentence the extractor can attach to artifacts.badge. Do not publish 15/5,859 as "the field does not use badges".

## H. Full-text sweep for artifact-evaluation wording

Probe                          Papers  Missing .cols  Share of files read
-----------------------------  ------  -------------  -------------------
artifact evaluation            59      4              1.0%
artifacts available (phrase)   21      4              0.4%
artifacts functional           3       4              0.1%
results reproduced             2       4              0.0%
open science appendix/section  89      4              1.5%

These are upper bounds: "artifact evaluation" is also the name of a committee, a badge, and a sentence about evaluating an attack artifact. They are NOT a badge-award rate. Use --hits to read contexts. The schema badge field (section G) is the lower bound.

## I. availability vs own-link — they are not the same question

Cut                              Papers
-------------------------------  ------
availability == public           2845
…and no authors-own link         54
authors-own link                 3321
…and availability is not public  530

Own-link papers by availability:
availability                Own-link papers  Share of 3321
--------------------------  ---------------  -------------
public                      2791             84.0%
none-mentioned              278              8.4%
promised-not-yet-available  135              4.1%
restricted                  62               1.9%
on-request                  47               1.4%
explicitly-withheld         8                0.2%

54 "public" papers have no authors-own URL in artifacts.links — typically "we release as part of scamper" / a WINE dataset id / a citation, not a URL the extractor kept as belongsToAuthors. 530 papers have an authors-own URL but are labelled promised / none-mentioned / restricted / on-request / withheld. The page's headline uses the URL, not the enum.

## J. Withheld, restricted, on-request, promised

availability                Papers  Share of 5859  Of which crawled
--------------------------  ------  -------------  ----------------
explicitly-withheld         52      0.9%           10
restricted                  79      1.3%           24
on-request                  91      1.6%           32
promised-not-yet-available  289     4.9%           72

explicitly-withheld evidence quotes (all of them — 52 is small enough):
  IMC/2011/an-analysis-of-underground-forums
    Unfortunately, the data is not ours to share. As mentioned in the paper, however, others have leaked the data.
  IMC/2013/profiling-high-school-students-with-facebook-how-online-privacy-laws-can-actuall
    Because of the sensitive nature of the information we gathered and inferred, we will not be making our data sets public and we will not explicitly identify the high schools involved.
  IEEE-SP/2015/ad-injection-at-scale-assessing-deceptive-advertisement-modifications
    never sharing raw data outside of Google; and setting a short lifetime on the data collected after which only aggregates could be stored.
  PETS/2018/touch-and-you-re-trapp-ck-ed-quantifying-the-uniqueness-of-touch-gestures-for-tr
    The data collected was not released publicly.
  CCS/2019/five-years-of-the-right-to-be-forgotten
    We cannot directly reveal a sample mapping between URLs and our annotations, nor can we reveal the exact URLs requested and the justification for delisting verdicts.
  IEEE-SP/2019/characterizing-pixel-tracking-through-the-lens-of-disposable-email-services
    The dataset is stored on a local server with strict access control. We keep the dataset strictly to ourselves.
  IEEE-SP/2019/touching-the-untouchables-dynamic-security-analysis-of-the-lte-control-plane
    We plan to privately release LTEFuzz to these carriers and vendors in the near future. A public release is not planned as LTEFuzz can be used for malicious purposes.
  IMC/2019/measuring-security-practices-and-how-they-impact-security
    The feature data set is only accessible to members of our group, subject to IRB and our agreements with campus, and will not (and cannot) be shared further.
  IMC/2020/a-characterization-of-the-covid-19-pandemic-impact-on-a-mobile-network-operator
    All datasets used in this work are covered by NDAs prohibiting any re-sharing with 3rd parties even for research purposes.
  IMC/2020/towards-a-user-level-understanding-of-ipv6-behavior
    We will not be sharing the analyzed data.
  IMC/2020/where-things-roam-uncovering-cellular-iot-m2m-connectivity
    Both datasets used in this work are collected from operators and covered by NDAs prohibiting any re-sharing with 3rd parties even for research purposes.
  USENIX/2020/empirical-measurement-of-systemic-2fa-usability
    we are unable to make this data publicly available.
  USENIX/2020/shim-shimmeny-evaluating-the-security-and-privacy-contributions-of-link-shimming
    The analyzed data cannot be publicly shared due to privacy constraints.
  WWW/2020/the-chameleon-attack-manipulating-content-display-in-online-social-media
    Chameleon pages, posts, and tweets are publicly available. Links can be found in the GitHub repository. Source code is not provided to reduce misuse.
  CCS/2022/watch-out-for-race-condition-attacks-when-using-android-external-storage
    Considering that the dataset may contain the volunteers' sensitive information, we will not make this dataset public unless with all the volunteers' agreement.
  IEEE-SP/2020/a-tale-of-sea-and-sky-on-the-security-of-maritime-vsat-communications
    we have elected not to release GSExtract until the issues identified in this study are addressed
  NDSS/2022/auto-draft-229
    the non-anonymized user datasets cannot be made publicly available due to obvious ethical concerns and privacy regulations.
  PETS/2022/a-novel-reconstruction-attack-on-foreign-trade-official-statistics-with-a-brazil
    For ethical reasons we have also chosen not to disclose in this paper other points of access to the data used in the description of our attack.
  USENIX/2022/behind-the-tube-exploitative-monetization-of-content-on-youtube
    We do not make our full account marketplace/forum data public, and present only a portion of the data.
  USENIX/2022/online-website-fingerprinting-evaluating-website-fingerprinting-attacks-on-tor-i
    We destroy all classification models upon completion of our evaluation.
  NDSS/2023/hope-of-delivery-extracting-user-locations-from-mobile-instant-messengers
    The raw data contain private location data we prefer not to share publicly.
  PETS/2023/how-website-owners-face-privacy-issues-thematic-analysis-of-responses-from-a-cov
    the dataset may only be accessed by specific researchers but not released publicly.
  USENIX/2023/diving-into-robocall-content-with-snorcall
    We are under a nondisclosure agreement to not release raw data or other content that could potentially identify a caller
  IMC/2024/bounce-in-the-wild-a-deep-dive-into-email-delivery-failures-from-a-large-email-s
    Due to email sensitivities and privacy concerns, we do not publish any datasets.
  WWW/2023/the-chameleon-on-the-web-an-empirical-study-of-the-insidious-proactive-web-defac
    The URLs of landing pages, control scripts, and content providers are anonymized in the paper; they are kept private and will not be shared with the public.
  USENIX/2023/log-it-s-big-it-s-heavy-it-s-filled-with-personal-data-measuring-the-logging-of
    We are not releasing our data.
  IMC/2024/mutual-tls-in-practice-a-deep-dive-into-certificate-configurations-and-privacy-i
    We unfortunately are unable to provide the original campus network traffic data due to Infosec and IRB rules given their sensitive nature.
  USENIX/2023/problematic-advertising-and-its-disparate-exposure-on-facebook
    we do not plan on making this data generally available to protect the privacy of our users.
  USENIX/2023/v-cloak-intelligibility-naturalness-timbre-preserving-real-time-voice-anonymizat
    withholding the release of the code of V-C LOAK for 90 days after notification.
  NDSS/2024/eavesdropping-on-controller-acoustic-emanation-for-keystroke-inference-attack-in-virtual-reality
    Heimdall has never been used in other ways or released to other parties.
  NDSS/2024/masterkey-automated-jailbreaking-of-large-language-model-chatbots
    We have decided not to release our complete jailbreak dataset before issues are properly addressed.
  IEEE-SP/2024/surveilling-the-masses-with-wi-fi-based-positioning-systems
    The data collected during this study is stored on the authors' machines and will not be publicly released.
  IMC/2024/through-the-telco-lens-a-countrywide-empirical-study-of-cellular-handovers
    Our datasets and actual, unnormalized, numbers in the figures cannot be published openly due to privacy guidelines of the MNO
  USENIX/2024/donapi-malicious-npm-packages-detector-using-behavior-sequence-knowledge-mapping
    a large multinational enterprise has already deployed our detector internally for security interception, so we cannot release the code due to contract limitations and copyright issues.
  WWW/2024/triage-of-messages-and-conversations-in-a-large-scale-child-victimization-corpus
    We have not permitted to share the data or reveal any information that could identify the platform or the victims.
  IEEE-SP/2016/sending-out-an-sms-characterizing-the-security-of-the-sms-ecosystem-with-public
    Because we do not make our data available to others, this study does not change - in severity or duration - the harm done by the existence and use of the gateway.
  IEEE-SP/2025/resolution-without-dissent-in-path-per-query-sanitization-to-defeat-surreptitiou
    Although source code for TunTight is not available
  IEEE-SP/2025/understanding-users-security-and-privacy-concerns-and-attitudes-towards-conversa
    The data is used exclusively for internal analysis and has not been redistributed. Additionally, we do not release any derivative products, such as our classifier
  IMC/2025/time-to-scan-digging-into-ntp-based-ipv6-scanning
    we refrain from publishing any of the collected data to also avoid others from doing so.
  PETS/2025/can-social-media-privacy-and-safety-features-protect-targets-of-interpersonal-at
    To prevent the attacks we demonstrated from being misused in practice, we do not release the specific attack steps anywhere.
  USENIX/2025/deanonymizing-ethereum-validators-the-p2p-network-has-a-privacy-issue
    Therefore, we refrain from making them publicly available.
  USENIX/2025/ghost-clusters-evaluating-attribution-of-illicit-services-through-cryptocurrency
    It is not legally possible to make the data in this paper public.
  USENIX/2025/on-the-virtues-of-information-security-in-the-uk-climate-movement
    we have chosen not to share the full interview transcripts, interview scripts or field notes as part of our dataset.
  USENIX/2025/i-can-tell-your-secrets-inferring-privacy-attributes-from-mini-app-interaction-h
    Due to AliPay's IRB and business regulations, and strict user privacy concerns, all code and data are processed on supervised servers.
  USENIX/2025/the-doom-of-device-drivers-your-android-device-most-likely-has-n-day-kernel-vuln
    Consequently, we do not recommend open-sourcing these resources. However, if the USENIX committee holds a different opinion, we will share all our findings and tools accordingly.
  WWW/2025/beyond-the-crawl-unmasking-browser-fingerprinting-in-real-user-interactions
    The source code for the automated crawler can be found at https://github.com/spalabucr/beyond-the-crawl.
  IEEE-SP/2017/leakage-abuse-attacks-against-order-revealing-encryption
    For privacy reasons we will not include links to these datasets, but they are available from the authors by request.
  IEEE-SP/2025/characterizing-robocalls-with-multiple-vantage-points
    we have confidentiality agreements in place that prevent us from sharing raw data.
  USENIX/2025/glados-location-aware-denial-of-service-of-cellular-networks
    these constraints prevent us from making the code publicly available.
  USENIX/2025/preventing-artificially-inflated-sms-attacks-through-large-scale-traffic-inspect
    the ML model, feature engineering, and model training source codes cannot be shared
  USENIX/2025/phishing-attacks-against-password-manager-browser-extensions
    Publishing individual records without consent would not be compatible with our institution's ethical standards, as it could have affected participants' opt-out decision.
  NDSS/2026/on-borrowed-time-measurement-informed-understanding-of-the-ntp-pools-robustness-to-monopoly-attacks
    Because our findings have potential security implications for the pool, we do not make our complete dataset publicly available

## K. Poster / short-record sensitivity

Cut                                       Papers  Own-link  Share
----------------------------------------  ------  --------  -----
all                                       5859    3321      56.7%
posters (slug poster-* or title Poster:)  138     17        12.3%
≤4 pages                                  251     44        17.5%
trimmed (drop posters ∪ ≤4 pages)         5608    3277      58.4%

Union dropped = 251 (every poster is also ≤4 pages, so the union is the short-record count). The own-link rate moves 56.7% → 58.4%.

## L. Quote check of artifacts.evidence.quote against paper.cols.txt

Verdict                                                         Papers
--------------------------------------------------------------  ------
exact (whitespace/hyphen normalised)                            3352
partial (≥60% of 5-word windows)                                1381
FAILED (below threshold) — listed, not treated as fabrications  787
missing quote                                                   15
missing paper.cols.txt                                          4
papers with an artifacts record                                 5539

First FAILED quotes (max 15). Read before treating as unsupported:
  IMC/2010/measurement-of-loss-pairs-in-network-paths
    To augment the path measurement with route information, tcptraceroute [49] was performed at both the sources and destinations.
  IMC/2010/measurement-and-analysis-of-real-world-802-11-mesh-networks
    The data set analyzed in this thesis includes measurements collected from 110 different production Meraki wireless mesh networks located around the world
  CCS/2010/dismantling-securememory-cryptomemory-and-cryptorf
    Sample code available at http://www.libnfc.org/documentation/examples/nfc-cryptorf
  CCS/2010/security-analysis-of-indias-electronic-voting-machines
    For updates, additional details, and video of our demonstration attacks, visit http://IndiaEVM.org.
  CCS/2010/the-security-of-modern-password-expiration-an-algorithmic-framework-and-empirica
    We employed various approaches to crack passwords: dictionary-based password cracking using "John the Ripper" (http://www.openwall.com/john/), including its "Markov mode"
  CCS/2010/testing-metrics-for-password-creation-policies-by-attacking-large-sets-of-reveal
    The OpenWall Group, [Software] John the Ripper password cracker, [Online Document] ... Available HTTP http://www.openwall.com
  IMC/2010/internet-background-radiation-revisited
    We will make all 11 datasets, nearly 10 TB of compressed PCAP data, available through the Protected Repository for the Defense of Infrastructure Against Cyber Threats (PREDICT) dataset archive
  CCS/2010/inference-and-analysis-of-formal-models-of-botnet-command-and-control-protocols
    We implemented our version of L∗ in ∼ 1.7 KLOC of C++ and the bot emulator and experimental infrastructure in ∼ 2.3 KLOC of Python and Bash scripts.
  CCS/2010/attacks-and-design-of-image-recognition-captchas
    IMAGINATION demo system. http://goldbach.cse.psu.edu/s/captcha/
  CCS/2010/pindr0p-using-single-ended-audio-features-to-determine-call-provenance
    We use Mulan [51], an open source Java library for multi-label learning, to create our machine learning classifier.
  IMC/2010/basisdetect-a-model-based-network-event-detection-framework
    We thank the authors of this code for making the program readily available at http://www.eecs.umich.edu/∼cscott/code/mnscann.zip
  IMC/2010/primitives-for-active-internet-topology-mapping-toward-high-frequency-characteri
    Acknowledgments ... CAIDA for measurement infrastructure support
  USENIX/2010/dude-wheres-that-ip-circumventing-measurement-based-ip-geolocation
    [21] Planetlab, 2010. http://www.planet-lab.org/.
  USENIX/2010/making-linux-protection-mechanisms-egalitarian-with-userfs
    DokuWiki. http://www.dokuwiki.org/dokuwiki.
  USENIX/2010/scantegrity-ii-municipal-election-at-takoma-park-the-first-e2e-binding-governmen
    All source code was released under the GPLv2 software license.
  CCS/2012/operating-system-framed-in-case-of-mistaken-identity-measuring-the-success-of-we
    Go to: http://saucers.cups.cs.cmu.edu/yacot/mnt/wtk/survey.php?i=workerID
  IMC/2012/confused-timid-and-unstable-picking-a-video-streaming-rate-is-hard
    RTMPDump. http: //rtmpdump.mplayerhq.hu/.
  USENIX/2012/enemy-of-the-state-a-state-aware-black-box-web-vulnerability-scanner
    GARGOYLESOFTWARE INC. HtmlUnit. http://htmlunit.sourceforge.net/.
  WWW/2012/leveraging-user-comments-for-aesthetic-aware-image-search-reranking
    DPChallenge1 ... http://www.dpchallenge.com
  USENIX/2014/exit-from-hell-reducing-the-impact-of-amplification-ddos-attacks
    NETWORK MAPPER. http://nmap.org/, 2014.
  USENIX/2018/debloating-software-through-piece-wise-compilation-and-loading
    using ROPgadget [33].
  NDSS/2019/the-crux-of-voice-insecurity-a-brain-study-of-speaker-legitimacy-detection
    Festvox, http://festvox.org/, 2014, accessed:05-11-2018.
  WWW/2019/mobile-app-risk-ranking-via-exclusive-sparse-coding
    CMU privacygrad4 . https://www.hcii.cmu.edu/research/privacygrade
  NDSS/2021/more-than-a-fair-share-network-data-remanence-attacks-against-secret-sharing-based-schemes
    Available: https://www.opennetworking.org/wp-content/uploads/2014/10/openflow-spec-v1.3.0.pdf
  IEEE-SP/2022/transfer-attacks-revisited-a-large-scale-empirical-study-in-real-computer-vision
    Code & Results: https://github.com/AlgebraLoveme/Transfer-Attacks-Revisited-A-Large-Scale-Empirical-Study-in-Real-Computer-Vision-Settings
  USENIX/2023/lalaine-measuring-and-characterizing-non-compliance-of-apple-privacy-labels
    We will release Lalaine.
  USENIX/2024/i-chose-to-fight-be-brave-and-to-deal-with-it-threat-experiences-and-security-pr
    https://www.usenix.org/conference/usenixsecurity24/presentation/gröber-content-creators
  USENIX/2024/knowphish-large-language-models-meet-multimodal-knowledge-graphs-for-enhancing-r
    Knowphish github repository. https://github.com/imethanlee/KnowPhish.
  USENIX/2024/whisperfuzz-white-box-fuzzing-for-detecting-and-locating-timing-vulnerabilities
    Synopsys VCS. https://www.synopsys.com/verification/simulation/vcs.html, 2022.
  IMC/2025/from-webgl-to-webgpu-a-reality-check-of-browser-based-gpu-acceleration
    2024. Emscripten Documentation. https://emscripten.org/docs/getting_started/index.html.
  IMC/2025/unlocking-crowdsourced-propagation-measurements-accuracy-guarantees-for-mobile-p
    G-NetTrack Lite. https://play.google.com/store/apps/details?id=com.gyokovsolutions.gnettracklite. (2023).
  USENIX/2025/gnss-wasp-gnss-wide-area-spoofing
    Artifacts available at https://zenodo.org/records/14734238.
  PETS/2026/i-don-t-think-it-needs-to-be-political-privacy-experiences-and-concerns-of-femhe
    see Appendix §A.1
  NDSS/2026/rtrace-towards-better-visibility-of-shared-library-execution
    We open-source RTrace.
  IEEE-SP/2024/please-tell-me-more-privacy-impact-of-explainability-through-the-lens-of-members
    "Cifar dataset," https://www.cs.toronto.edu/∼kriz/cifar.html, 2012.

## Y. Arithmetic-derived figures the page may print

own-link share all: 3321/5859 = 56.7%
own-link share empirical: 2872/5118 = 56.1%
own-link share crawled: 684/1120 = 61.1%
2010-2013 own-link: see section E buckets (23.7%)
2022-2024 own-link: see section E buckets (65.0%)
2024 own-link: 457/690 = 66.2%
2025 own-link: 592/770 = 76.9% (provisional)
null artifacts records: 320

## Z. External figures the page may print (NOT from this corpus)

Every number below is external. Listed so check_page_numbers.mjs can pass a whole-page audit.

  USENIX Security 2026 Available badge (secartifacts.github.io/usenixsec2026/badges, 2026-08-27):
    GitHub / GitLab / personal pages are NOT acceptable for Artifacts Available.
    Zenodo recommended; FigShare, Dryad, Software Heritage named as alternatives.
    Three badges: Available (mandatory for accepted papers), Functional (optional), Reproduced (optional).

  NDSS 2026 AE results (secartifacts.github.io/ndss2026/results, 2026-08-27):
    114 artifacts evaluated; 112 Available; 97 Functional; 70 Reproduced; 3 Distinguished Artifact Awards.

  ACM Artifact Review and Badging Version 1.1, cited by CCS 2026 call-for-artifacts (2026-08-27):
    Available / Evaluated (Functional or Reusable) / Results Validated. CCS 2026 uses this vocabulary.
    acm.org/publications/policies/artifact-review-and-badging-current is Cloudflare-blocked from this host;
    Version 1.1 is verified on www.sigsac.org/ccs/CCS2026/call-for/call-for-artifacts.html.
    Demir et al. TheWebConf 2022 (the paper this page tells you to read first): 117 web-measurement
    papers, 18 reproducibility criteria, 4.5-million-page experiment.
    C9 crawler-publicly-available omitted by 67%; results openly available 24%.

  TheWebConf 2026 artifact badging (www2026.thewebconf.org/calls/artifact-badging.html, 2026-08-27):
    Artifacts Available only; light review; DOI required; GitHub+Zenodo both in the resource-availability statement.
    Camera-ready deadlines: main 18 February 2026, companion 11 March 2026.

  IEEE S&P 2026 (sp2026.ieee-security.org/cfartifacts.html, 2026-08-27):
    Optional AE; Available requires DOI-backed deposit (Zenodo/FigShare/Dryad); GitHub allowed as extra, not instead.

  IMC 2026 CFP (conferences.sigcomm.org/imc/2026/cfp/, 2026-08-27):
    Artifact-availability declaration required (full / partial / none). Shepherding of accepted papers.
    Community Contribution Award requires public data or code by camera-ready. Replicability Track EoI 29 Jan 2026.

  PoPETs 2027 AE (petsymposium.org/artifacts.php, 2026-08-27):
    Optional; Available / Functional / Reproduced. PETS 2026 Artifact Award already announced.

  Zenodo (zenodo.org, 2026-08-27): CERN-hosted; version DOI + concept DOI; GitHub integration mints a DOI per release.

  pin_artifact.py is published on the page; its sample output is from a live run against Zenodo record 1421702
    and github.com/mozilla/OpenWPM. Re-run it; do not freeze byte counts of files that change.
    Live run 2026-08-27: concept DOI 10.5281/zenodo.1421701 (record 1421701); created 2018-09-19; licence other-open;
    1 file Eichhoernchen/monero-digging-paper-v1.zip (20788 bytes). --require-doi on OpenWPM exits 1.

Review log

Freeze snapshot (before any reviewer ran):

  • content artifacts rev 1787836784 (30,955 bytes), file pages/artifacts.txt copied to out/artifacts/frozen/artifacts.txt
  • provenance this page first published as rev 1787836785 (46,280 bytes); this freeze-note save is the next rev
  • bibliography rev 1787836782 after appending 7 keys
  • start rev 1787836787
  • report out/artifacts-output.txt / scripts/report_artifacts.mjs copied into out/artifacts/frozen/
  • rendered https://measuretheweb.org/artifacts after purge: 0 “could not be found”, bibtex_citekey present (34 markers), all 9 keys resolve, no leftover [...]

Findings and accept/reject. Reviewers: three focused passes on gpt-5.6-sol-medium (requested luna@medium / gpt-5.6-luna-max unavailable). Generic pass not yet run.

Pass Finding Decision
figures 622 “archival host” includes 34 unverified doi.org resolver URLs accepted. Intro now splits named archives from other-DOI.
figures –require-doi accepted a generic ACM publisher DOI accepted. Generic doi.org is now unknown, not archival.
figures GitHub path containing 10.5281 classified as Zenodo accepted. DOI-prefix matching only on doi.org / doi: URLs.
figures zenodo.org/not-a-record exited 0 under –require-doi accepted. No record id → unknown, exit 1.
figures rebrand.ly / shorturl.at in host_fold but not pin_artifact accepted. Shortener set synced.
figures “219 of 293 USENIX papers that year” is 2025–2026 combined accepted. Wording fixed.
figures “almost all of the Zenodo mass (233 of 436)” is 53.4% accepted. “more than half”.
citations Bertram quote spans a column splice in .cols accepted. Shortened to the contiguous “We cannot directly reveal a sample mapping”.
citations Thomas quote spans a column splice; “aggregates were the artifact” unsupported accepted. Quote shortened; sentence now “Only aggregates could be retained.”
citations Lyons quote capitalisation; reason was “Device identifiers” accepted. Verbatim footnote on re-identification.
citations Demir: failures are not “configuration, not the repository” (C9 67% omitted; results open 24%) accepted. Sentence rewritten from the paper.
citations Demir paper.cols.txt missing from the corpus mount rejected as a page fix. Mount is read-only; 117 / 18 / 4.5M verified from the fetched PDF. Logged under what could not be established.
external CCS row claimed a mandatory open-science appendix accepted. CCS AE is optional; appendix is encouraged after evaluation.
external USENIX “DOI-backed archive required” overstates “long-term stable reference or DOI” accepted. Table aligned with the badges page.
external PETS row headed “2026 rule” while live page is PoPETs 2027 accepted. Column retitled; cell names 2027.

Generic pass: not yet run. This table is the focused-pass log, not a GENERIC_REVIEW placeholder.

provenance/artifacts.1787837459.txt.gz · Last modified: by karel.kubicek.claude

Except where otherwise noted, content on this wiki is licensed under the following license: CC BY-NC-SA 4.0
CC BY-NC-SA 4.0 Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki