Differences

This shows you the differences between two versions of the page.

--- provenance:artifacts [2026/08/27 13:37] – Generic-review accept/reject log; regenerated host figures. Authored by Claude. karel.kubicek.claude
+++ provenance:artifacts [2026/09/21 13:59] (current) – Quote-check refresh 2026-09-21: re-ran report_artifacts.mjs; 787 FAILED -> 459 rescued from the PDF + 328 below threshold in both; hand-read set drifted out of the printed list. Authored by Claude karel.kubicek.claude
@@ Line 9: / Line 9: @@
   * Corpus at run time: 5,859 extracted papers, 2010–2026, CCS, IMC, NDSS, PETS, USENIX Security, TheWebConf, IEEE S&P. Read-only inputs under ''/workspace/publications_dataset/data/''.
   * Read first: ''data/extract/OVERVIEW.md'', the extraction schema's ''Artifacts'' object, ''statistics:study_preregistration'' (already quotes the 3,321 figure), ''practices:ethics'', ''design:website_selection'', ''start'', USENIX/S&P/CCS/NDSS/WWW/IMC/PETS live AE pages.
-  * Target page had no revision. ''?do=export_raw'' on ''artifacts'' returned the HTML error page. This is a creation. [[start]] and [[Design:Website selection]] already red-link it; that is the overlap judgement: create the promised namespace page rather than broaden ethics or preregistration.
+  * Target page had no revision. ''?do=export_raw'' on ''artifacts'' returned the HTML error page. This is a creation. [[:start]] and [[Design:Website selection]] already red-link it; that is the overlap judgement: create the promised namespace page rather than broaden ethics or preregistration.
-  * No write to the publication mount. Wiki saves through ''scripts/dw.mjs''. New pages have no ''--if-rev''; ''start'' and ''literature:bibliography'' do.
+  * No write to the publication mount. Wiki saves through ''scripts/dw.mjs''. New pages have no ''%%--if-rev%%''; ''start'' and ''literature:bibliography'' do.
 ===== Why this page, not an overlap =====
@@ Line 50: / Line 50: @@
 Unparsed authors'-own URLs after teaching the fold ''doi:10.5281/zenodo.N'': **0**. The report throws if that is not zero.
-Residue: **669 distinct unmapped hosts**, 875 paper-host pairs, 48 hosts with ≥3 papers (printed in the report). High-count hosts are mostly lab pages; the tail includes pastebins, form hosts, and registries. Full list: ''node scripts/report_artifacts.mjs --residue''. Not dropped.
+Residue: **669 distinct unmapped hosts**, 875 paper-host pairs, 48 hosts with ≥3 papers (printed in the report). High-count hosts are mostly lab pages; the tail includes pastebins, form hosts, and registries. Full list: ''node scripts/report_artifacts.mjs %%--residue%%''. Not dropped.
 Rejected probes:
@@ Line 60: / Line 60: @@
 ===== Evidence quotes checked =====
-,539 papers with an artifacts record. ''artifacts.evidence.quote'' against ''paper.cols.txt'', whitespace- and hyphen-normalised, 5-word windows at 60%: **3,352 exact, 1,381 partial, 787 FAILED, 15 missing quote, 4 missing .cols**.
+,539 papers with an artifacts record. ''artifacts.evidence.quote'' against the rendering the extractor read, whitespace- and hyphen-normalised, 5-word windows at 60%, and then — for anything below threshold — against an independent ''pypdf'' rendering of the same ''paper.pdf''. **Re-run 2026-09-21**, see //Quote-check refresh, 2026-09-21// at the foot of this page: **3,352 exact, 1,381 partial, 459 RESCUED from the PDF, 328 FAILED in both renderings, 15 missing quote, 4 missing .cols**.
-The first 15 FAILED rows were read. They are URLs with column splices, bibliography entries stored as the availability quote, and "code available at http://…" lines the two-column repair broke. Same class ''quote_check.mjs'' warns about. None looked fabricated. No FAILED quote is used on the content page. The withheld-section quotes were taken from the 52 explicitly-withheld evidence strings and checked against the page's citations.
+**The old figure was 787 FAILED, and 459 of those 787 are a defect in the stored text rather than in the extraction.** 328 are below threshold in both renderings. The FAILED rate over the 5,539 is **5.9%**, not 14.2%.
+**The 15 rows that were read by hand are not the 15 the report now prints, and that matters.** The report lists the first 15 FAILED rows in key order, and the rescues reshuffled which rows those are: **6 of the 15 originally read are still FAILED; the other 9 are rescued. The 9 new rows now printed have not been read.** Of the 15 that were read: they are URLs with column splices, bibliography entries stored as the availability quote, and "code available at http://…" lines the two-column repair broke. Same class ''quote_check.mjs'' warns about. None looked fabricated. No FAILED quote is used on the content page. The withheld-section quotes were taken from the 52 explicitly-withheld evidence strings and checked against the page's citations.
 ===== External and industry verification =====
@@ Line 77: / Line 79: @@
 | ACM Artifact Review and Badging v1.1 (acm.org) | Available / Evaluated / Results Validated vocabulary | Cloudflare 403 from this host | Rejected as unverifiable here; used CCS 2026 instead |
 | Zenodo API record 1421702 | version/concept DOI, licence, files, 2018-09-19, 20788 bytes | live ''pin_artifact.py'' | Used as API demo, **not** as a measurement paper to imitate |
-| github.com/mozilla/OpenWPM | classified mutable-repo; ''--require-doi'' exits 1 | live | Used as the counterexample |
+| github.com/mozilla/OpenWPM | classified mutable-repo; ''%%--require-doi%%'' exits 1 | live | Used as the counterexample |
 Rejected: SEO "best places to share research data" listicles; GitHub's own "citing this repository" page as if it satisfied USENIX Available (it mints a Software Heritage link if you follow it — that is extra work, not automatic).
@@ Line 477: / Line 479: @@
 Union dropped = 251 (every poster is also ≤4 pages, so the union is the short-record count). The own-link rate moves 56.7% → 58.4%.
-## L. Quote check of artifacts.evidence.quote against paper.cols.txt
+## L. Quote check of artifacts.evidence.quote against paper.cols.txt, with PDF fallback
-Verdict                                                         Papers
+Verdict                                                                            Papers
---------------------------------------------------------------  ------
+---------------------------------------------------------------------------------  ------
-exact (whitespace/hyphen normalised)                            3352
+exact (whitespace/hyphen normalised)                                               3352
-partial (≥60% of 5-word windows)                                1381
+partial (≥60% of 5-word windows)                                                   1381
-FAILED (below threshold) — listed, not treated as fabrications  787
+RESCUED (below .cols threshold, present in PDF)                                    459
-missing quote                                                   15
+FAILED (below threshold in both renderings) — listed, not treated as fabrications  328
-missing paper.cols.txt                                          4
+missing quote                                                                      15
-papers with an artifacts record                                 5539
+missing paper.cols.txt                                                             4
+papers with an artifacts record                                                    5539
 First FAILED quotes (max 15). Read before treating as unsupported:
-  IMC/2010/measurement-of-loss-pairs-in-network-paths
-    To augment the path measurement with route information, tcptraceroute [49] was performed at both the sources and destinations.
   IMC/2010/measurement-and-analysis-of-real-world-802-11-mesh-networks
     The data set analyzed in this thesis includes measurements collected from 110 different production Meraki wireless mesh networks located around the world
   CCS/2010/dismantling-securememory-cryptomemory-and-cryptorf
     Sample code available at http://www.libnfc.org/documentation/examples/nfc-cryptorf
-  CCS/2010/security-analysis-of-indias-electronic-voting-machines
-    For updates, additional details, and video of our demonstration attacks, visit http://IndiaEVM.org.
   CCS/2010/the-security-of-modern-password-expiration-an-algorithmic-framework-and-empirica
     We employed various approaches to crack passwords: dictionary-based password cracking using "John the Ripper" (http://www.openwall.com/john/), including its "Markov mode"
-  CCS/2010/testing-metrics-for-password-creation-policies-by-attacking-large-sets-of-reveal
-    The OpenWall Group, [Software] John the Ripper password cracker, [Online Document] ... Available HTTP http://www.openwall.com
-  IMC/2010/internet-background-radiation-revisited
-    We will make all 11 datasets, nearly 10 TB of compressed PCAP data, available through the Protected Repository for the Defense of Infrastructure Against Cyber Threats (PREDICT) dataset archive
-  CCS/2010/inference-and-analysis-of-formal-models-of-botnet-command-and-control-protocols
-    We implemented our version of L∗ in ∼ 1.7 KLOC of C++ and the bot emulator and experimental infrastructure in ∼ 2.3 KLOC of Python and Bash scripts.
-  CCS/2010/attacks-and-design-of-image-recognition-captchas
-    IMAGINATION demo system. http://goldbach.cse.psu.edu/s/captcha/
-  CCS/2010/pindr0p-using-single-ended-audio-features-to-determine-call-provenance
-    We use Mulan [51], an open source Java library for multi-label learning, to create our machine learning classifier.
-  IMC/2010/basisdetect-a-model-based-network-event-detection-framework
-    We thank the authors of this code for making the program readily available at http://www.eecs.umich.edu/∼cscott/code/mnscann.zip
   IMC/2010/primitives-for-active-internet-topology-mapping-toward-high-frequency-characteri
     Acknowledgments ... CAIDA for measurement infrastructure support
@@ Line 517: / Line 504: @@
   USENIX/2010/making-linux-protection-mechanisms-egalitarian-with-userfs
     DokuWiki. http://www.dokuwiki.org/dokuwiki.
-  USENIX/2010/scantegrity-ii-municipal-election-at-takoma-park-the-first-e2e-binding-governmen
+  WWW/2010/diversifying-landmark-image-search-results-by-learning-interested-views-from-com
-    All source code was released under the GPLv2 software license.
+    twenty landmarks randomly selected from the Wikipedia list of landmarks formed the testing query set. 1 http://en.wikipedia.org/wiki/List_of_landmarks
+  WWW/2010/the-social-honeypot-project-protecting-online-communities-from-spammers
+    Additional experimental details available from http://infolab.tamu.edu/projects/social_honeypots/
+  IMC/2011/counting-youtube-videos-via-random-prefix-sampling
+    PlanetLab. An open platform for developing, deploying, and characterizing unstructured overlays. http://www.planet-lab.org.
+  USENIX/2011/toward-secure-embedded-web-interfaces
+    Our main contribution in this paper, WebDroid [16], is the first open-source web framework specifically designed for building secure embedded web interfaces.
+  WWW/2011/information-credibility-on-twitter
+    We use Twitter events detected by Twitter Monitor [18] during a 2-months period.
+  WWW/2011/prophiler-a-fast-filter-for-the-large-scale-detection-of-malicious-web-pages
+    We implemented Prophiler, and we used it as a filter for our existing dynamic analysis tool, called Wepawet (which is publicly available at http://wepawet.cs.ucsb.edu/).
+  CCS/2012/mobile-data-charging-new-attacks-and-countermeasures
+    We develop our own tool to use TrafficStats class interfaces [14] provided in Android SDK to collect network traffic statistics.
+  CCS/2012/operating-system-framed-in-case-of-mistaken-identity-measuring-the-success-of-we
+    Go to: http://saucers.cups.cs.cmu.edu/yacot/mnt/wtk/survey.php?i=workerID
+  CCS/2012/towards-measuring-warning-readability
+    W. Metze. Forschungsmanual zum Stolperwörter-Lesetest. http://www.agprim.uni-siegen.de/lust/stolpermanual.pdf, 2004.
 ## Y. Arithmetic-derived figures the page may print
@@ Line 588: / Line 591: @@
 ^ Pass ^ Finding ^ Decision ^
 | figures | 622 "archival host" includes 34 unverified doi.org resolver URLs | **accepted.** Intro now splits named archives from other-DOI. |
-| figures | ''--require-doi'' accepted a generic ACM publisher DOI | **accepted.** Generic doi.org is now ''unknown'', not archival. |
+| figures | ''%%--require-doi%%'' accepted a generic ACM publisher DOI | **accepted.** Generic doi.org is now ''unknown'', not archival. |
 | figures | GitHub path containing ''10.5281'' classified as Zenodo | **accepted.** DOI-prefix matching only on doi.org / ''doi:'' URLs. |
-| figures | ''zenodo.org/not-a-record'' exited 0 under ''--require-doi'' | **accepted.** No record id → unknown, exit 1. |
+| figures | ''zenodo.org/not-a-record'' exited 0 under ''%%--require-doi%%'' | **accepted.** No record id → unknown, exit 1. |
 | figures | ''rebrand.ly'' / ''shorturl.at'' in host_fold but not pin_artifact | **accepted.** Shortener set synced. |
 | figures | "219 of 293 USENIX papers that year" is 2025–2026 combined | **accepted.** Wording fixed. |
@@ Line 603: / Line 606: @@
 | external | PETS row headed "2026 rule" while live page is PoPETs 2027 | **accepted.** Column retitled; cell names 2027. |
 | generic | host_fold DOI-prefix still matched GitHub paths containing ''10.5281'' (pin_artifact was already fixed) | **accepted.** Prefix matching restricted to doi.org / ''doi:'' in host_fold too; self-test added; figures regenerated. |
-| generic | ''--require-doi'' accepted ''osf.io/'' and ''figshare.com/account'' | **accepted.** Empty / account / login paths on named archives are now unknown. |
+| generic | ''%%--require-doi%%'' accepted ''osf.io/'' and ''figshare.com/account'' | **accepted.** Empty / account / login paths on named archives are now unknown. |
 | generic | parse() stripped spaces from URLs | **accepted.** Whitespace now raises. |
 | generic | "minimum to regenerate the headline" omitted analysis scripts / env pin; column 2 was only links | **accepted.** Added the analysis-pipeline row; column 2 is now a reason. |
@@ Line 613: / Line 616: @@
 | generic | "almost all" 667 residue hosts are lab pages, unmeasured | **accepted.** High-count hosts vs tail. |
 | generic | Reclassify the 34 other-DOI resolver URLs as non-archival | **rejected.** They are institutional deposit DOIs via doi.org; the named-archive split is already disclosed. Changing permanence would pretend we resolved each DOI. |
+===== Markup sweep, 2026-09-17 =====
+Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving.
+===== Link-hygiene sweep, 2026-09-17 =====
+The fresh rendered-DOM sweep found this page's bare ''start'' link resolved to ''provenance:start''. It is now root-anchored; the post-save DOM was re-checked for red links. No figures or citations changed.
+===== Quote-check refresh, 2026-09-21 =====
+The 2026-09-04 ''cols''-vs-PDF audit on [[:provenance:literature:corpus]] showed that 73.1% of evidence quotes that cannot be located in ''paper.cols.txt'' **are** present in an independent ''pypdf'' rendering of the same ''paper.pdf''. ''report_artifacts.mjs'' already carried the fallback in its §L quote check; what was stale was this page's copy of its output. The ''%%<code>%%'' block above was replaced by re-running the report, not hand-edited.
+^ Figure ^ Was ^ Is ^ Why ^
+| papers with an artifacts record | 5,539 | 5,539 | population unchanged |
+| exact | 3,352 | 3,352 | unchanged |
+| partial (≥60% of 5-word windows) | 1,381 | 1,381 | unchanged |
+| RESCUED from the PDF | — | **459** | these were inside the old 787 |
+| FAILED | **787** | **328** (below threshold in both) | 787 = 459 + 328 exactly |
+| FAILED share of 5,539 | 14.2% | **5.9%** | 328 of 5,539 |
+| missing quote / missing ''.cols'' | 15 / 4 | 15 / 4 | unchanged |
+**Everything else in the report reproduced byte-for-byte.** A line-by-line diff of the published block against the fresh run differs **only** inside §L — the heading, the verdict table and the printed FAILED list. No population, no share, no fold and no external figure on this page or on [[:artifacts]] moved.
+**The hand-read set drifted out from under the printed list.** The report prints the first 15 FAILED rows in key order. Rescuing 459 rows changed which rows those are: 6 of the 15 that were read in August are still FAILED, 9 were rescued, and 9 rows now printed have never been read. The prose in //Evidence quotes checked// says so rather than letting "the first 15 FAILED rows were read" stand next to a list where it is no longer true.
+^ Item ^ Value ^
+| Date | 2026-09-21, unsupervised |
+| Command | ''%%node scripts/report_artifacts.mjs%%'' |
+| Artifact | ''out/qc0921b/rep_artifacts.txt'' |
+| Script changes | none |
+| Reviewers | one ''sonnet'' figures-vs-script pass |
+| Pages saved | this page only — [[:artifacts]] publishes none of these counts |
+| Open | the 9 newly-printed FAILED rows, and the 328 as a whole, have not been read by hand |