| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| provenance:design:platforms [2026/08/27 20:37] – Add the full review log for the three focused passes (accepted and rejected findings), the misattributed-quote and omitted-TikTok corrections, and observations for the shared bibliography (5 duplicate key pairs, an unresolved DOI, a null byte that makes g karel.kubicek.claude | provenance:design:platforms [2026/09/22 01:33] (current) – Embed the real scripts/auth145_audit.mjs in the provenance log, per the page's convention. Authored by Claude karel.kubicek.claude |
|---|
| [CCS 2012] malware infection | [CCS 2012] malware infection |
| [IMC 2012] Facebook gifting application activity | [IMC 2012] Facebook gifting application activity |
| | |
| | ## Cross-page reconciliation: how many papers name "Alexa" in a stated population source? |
| | papers with >=1 stated population sourceList 5492 |
| | ... of which at least one tuple has a web unit 1143 |
| | name "Alexa" in any stated sourceList (no role filter) 486 |
| | ... and have a web-unit population tuple 464 <- comparable to design:website_selection's 463 |
| | assigned role=ranking by plat_fold.mjs (what this page publishes) 412 |
| | The gap is the role rule: Alexa-as-skill-store strings are diverted to |
| | role=subject, and a string naming several lists is tagged per platform. |
| | Neither number is wrong; they answer different questions. |
| </file> | </file> |
| |
| const noMode = anySubject.filter((r) => (r.temporal || []).every((t) => !t.mode || isSentinel(t.mode))).length; | const noMode = anySubject.filter((r) => (r.temporal || []).every((t) => !t.mode || isSentinel(t.mode))).length; |
| console.log(` (no stated mode) ${String(noMode).padStart(4)} ${pct(noMode, anySubject.length)}`); | console.log(` (no stated mode) ${String(noMode).padStart(4)} ${pct(noMode, anySubject.length)}`); |
| | // temporal.mode is MULTI-VALUED, so "383 existing-dataset" is not "383 papers |
| | // that collected nothing": 175 of them also name a primary-collection mode. |
| | // Publishing the multi-valued count as an exclusive one overstates dataset |
| | // reuse and understates primary collection. Exclusive split: |
| | const PRIMARY = new Set(['live-crawl', 'active-probing', 'passive-collection']); |
| | let bothModes = 0; |
| | let onlyExisting = 0; |
| | let onlyPrimary = 0; |
| | for (const r of anySubject) { |
| | const m = new Set((r.temporal || []).map((t) => t.mode).filter((x) => x && !isSentinel(x))); |
| | const ex = m.has('existing-dataset'); |
| | const pr = [...m].some((x) => PRIMARY.has(x)); |
| | if (ex && pr) bothModes += 1; |
| | else if (ex) onlyExisting += 1; |
| | else if (pr) onlyPrimary += 1; |
| | } |
| | console.log(` EXCLUSIVE split of the same ${anySubject.length} papers:`); |
| | console.log(` existing-dataset ONLY (no primary collection) ${String(onlyExisting).padStart(4)} ${pct(onlyExisting, anySubject.length)}`); |
| | console.log(` primary collection ONLY (no existing dataset) ${String(onlyPrimary).padStart(4)} ${pct(onlyPrimary, anySubject.length)}`); |
| | console.log(` both ${String(bothModes).padStart(4)} ${pct(bothModes, anySubject.length)}`); |
| | console.log(` any primary collection ${String(onlyPrimary + bothModes).padStart(4)} ${pct(onlyPrimary + bothModes, anySubject.length)}`); |
| |
| // -- 6. official API vs scraping, from tool names | // -- 6. official API vs scraping, from tool names |
| console.log(' residue sample (first 15 phenomena):'); | console.log(' residue sample (first 15 phenomena):'); |
| for (const r of unmatched.slice(0, 15)) console.log(` [${r.venue} ${r.year}] ${(r.detection[0] || {}).phenomenon}`); | for (const r of unmatched.slice(0, 15)) console.log(` [${r.venue} ${r.year}] ${(r.detection[0] || {}).phenomenon}`); |
| | |
| | // -- 16. cross-page reconciliation. |
| | // design:website_selection reports 463 papers for Alexa, folded over the 1,153 |
| | // papers that sampled the WEB. This page reports 412 for role=ranking over all |
| | // papers that stated a population. A subset cannot exceed its superset, so the |
| | // two folds must be measuring different things — and they are. Printed here so |
| | // neither page looks broken to a reader holding both. |
| | console.log(`\n## Cross-page reconciliation: how many papers name "Alexa" in a stated population source?`); |
| | { |
| | const stated = rows.filter((r) => (r.population || []).some((p) => p.sourceList && !isSentinel(p.sourceList))); |
| | const WEB_UNITS = new Set(['websites', 'domains', 'web-pages', 'urls']); |
| | const isWeb = (r) => (r.population || []).some((p) => p.sourceList && !isSentinel(p.sourceList) && WEB_UNITS.has(p.unit)); |
| | const anyAlexa = stated.filter((r) => r.population.some((p) => p.sourceList && /alexa/i.test(p.sourceList))); |
| | console.log(` papers with >=1 stated population sourceList ${stated.length}`); |
| | console.log(` ... of which at least one tuple has a web unit ${stated.filter(isWeb).length}`); |
| | console.log(` name "Alexa" in any stated sourceList (no role filter) ${anyAlexa.length}`); |
| | console.log(` ... and have a web-unit population tuple ${anyAlexa.filter(isWeb).length} <- comparable to design:website_selection's 463`); |
| | const rankingRole = [...perPaperRoles.entries()].filter(([, f]) => f.has('Amazon') && f.get('Amazon').has('ranking')).length; |
| | console.log(` assigned role=ranking by plat_fold.mjs (what this page publishes) ${rankingRole}`); |
| | console.log(' The gap is the role rule: Alexa-as-skill-store strings are diverted to'); |
| | console.log(' role=subject, and a string naming several lists is tagged per platform.'); |
| | console.log(' Neither number is wrong; they answer different questions.'); |
| | } |
| </file> | </file> |
| |
| ['2024/CCS/modern-problems-require-modern-solutions-community-developed-techniques-for-onli', '137 videos and 4,297 comments on TikTok'], | ['2024/CCS/modern-problems-require-modern-solutions-community-developed-techniques-for-onli', '137 videos and 4,297 comments on TikTok'], |
| ['2025/USENIX/darkgram-a-large-scale-analysis-of-cybercriminal-activity-channels-on-telegram', 'now replaced by the Meta Content Library'], | ['2025/USENIX/darkgram-a-large-scale-analysis-of-cybercriminal-activity-channels-on-telegram', 'now replaced by the Meta Content Library'], |
| | // Added 2026-08-27 after review found the '1% streaming API' quote attributed |
| | // to the wrong NDSS 2021 paper. hagen2021_numbers contains no occurrence of |
| | // "Twitter" at all; this is the paper the quote belongs to. |
| | ['2021/NDSS/to-err-is-human-characterizing-the-threat-of-unintended-urls-in-social-media', '1% streaming API that Twitter provides to vetted researchers'], |
| | ['2021/NDSS/to-err-is-human-characterizing-the-threat-of-unintended-urls-in-social-media', 'all the numbers that we presented in this paper are lower bounds'], |
| | // Added after review found the cross-platform blocking range omitted TikTok, |
| | // which is at the TOP of it, not absent from it. |
| | ['2025/IMC/exploration-of-the-dynamics-of-buy-and-sale-of-social-media-accounts', 'TikTok and Instagram demonstrated the highest'], |
| | ['2025/IMC/exploration-of-the-dynamics-of-buy-and-sale-of-social-media-accounts', 'TikTok 1,700 816'], |
| | // Added 2026-08-27 after the generic reviewer pointed out that the header |
| | // sentence claimed to cover "every figure the page attributes to a paper" |
| | // while many attributed figures were in no check at all. These close the gap. |
| | ['2026/PETS/a-year-under-the-dsa-ad-transparencys-uneven-landscape', '98.9% of all explanation texts cite only the main targeting form'], |
| | ['2026/NDSS/hey-there-you-are-using-whatsapp-enumerating-three-billion-accounts-for-security-and-privacy', 'roughly two-thirds of the images (66 %) contain detectable human faces'], |
| | ['2026/NDSS/hey-there-you-are-using-whatsapp-enumerating-three-billion-accounts-for-security-and-privacy', '245 countries'], |
| | ['2019/WWW/auditing-offline-data-brokers-via-facebooks-advertising-platform', 'Australia (81.3%), and for the U.K. (74.4%)'], |
| | ['2026/PETS/banned-books-analysis-of-censorship-on-amazon-com', '8,965 out of the 796,081'], |
| | // Column splice: the running sentence is interleaved. The paper's own |
| | // contiguous statement of the same fact is in its contributions list. |
| | ['2023/IMC/flocking-to-mastodon-tracking-the-great-twitter-migration', 'The top 25% most populous instances contain 96% of the users'], |
| | ['2023/IMC/flocking-to-mastodon-tracking-the-great-twitter-migration', '2,879 unique Mastodon instances'], |
| | ['2025/IMC/exploration-of-the-dynamics-of-buy-and-sale-of-social-media-accounts', 'Facebook 649 37 5.70'], |
| | ['2026/WWW/does-this-button-work-investigating-youtubes-ineffective-user-controls', '22,722'], |
| | ['2024/IMC/beyond-the-guidelines-assessing-metas-political-ad-moderation-in-the-eu', '29.5 million'], |
| | // "63.2 bn" on the page is a rounding of this table value; check the value. |
| | ['2026/NDSS/hey-there-you-are-using-whatsapp-enumerating-three-billion-accounts-for-security-and-privacy', '63,170,000,000'], |
| | ['2024/WWW/tiktok-and-the-art-of-personalization-investigating-exploration-and-exploitation', '4.9M'], |
| ]; | ]; |
| |
| |
| The family sets are **regex candidate sets**, so they were audited against the papers' own full text with ''scripts/platforms_audit.mjs'', which prints whitespace-collapsed windows around each platform-name match. Samples are deterministic (every //k//-th row of the ''%%--list%%'' output, which is sorted by year then venue), so the audit is reproducible. | The family sets are **regex candidate sets**, so they were audited against the papers' own full text with ''scripts/platforms_audit.mjs'', which prints whitespace-collapsed windows around each platform-name match. Samples are deterministic (every //k//-th row of the ''%%--list%%'' output, which is sorted by year then venue), so the audit is reproducible. |
| | |
| | **These samples are small** — //n// = 7, 15, 16 and 11. At //n// = 15 the 95% interval around 73% is roughly ±20 points. The precisions below are coarse corrections on a candidate set, not measurements, and the content page now says so too. |
| |
| ==== TikTok — full audit, all 7 candidates ==== | ==== TikTok — full audit, all 7 candidates ==== |
| ===== Quotes spot-checked against the source ===== | ===== Quotes spot-checked against the source ===== |
| |
| Every figure the content page attributes to a paper is listed in ''scripts/platforms_quotecheck.mjs'' with the phrase that must appear in that paper's ''paper.cols.txt''. Whitespace is collapsed on both sides first. | Every figure and every quoted phrase the content page attributes to a paper is listed in ''scripts/platforms_quotecheck.mjs'' with the text that must appear in that paper's ''paper.cols.txt''. Whitespace is collapsed on both sides first. |
| | |
| | <WRAP important> |
| | **That sentence was not true when this page was first published, and the generic reviewer caught it.** The list then held 26 checks and covered the //numbered// figures but not every attributed figure: the 98.9% YouTube explanation share, the 66% face-detection rate, 81.3%/74.4%, 1.1% of 796,081, the Mastodon 96% and 2,879, the per-platform blocking rates, 22,722, 29.5 M ads and 63,170,000,000 were all attributed on the page and checked by nobody. It now holds **42** checks and the sentence is true. The lesson generalises past this page: a quote-check list built from "the figures I published" misses everything a reader would also want to trust, and the number guard cannot see quoted phrases at all. |
| | </WRAP> |
| |
| <file text platforms_quotecheck-output.txt> | <file text platforms_quotecheck-output.txt> |
| PASS CCS 2024 "137 videos and 4,297 comments on TikTok" | PASS CCS 2024 "137 videos and 4,297 comments on TikTok" |
| PASS USENIX 2025 "now replaced by the Meta Content Library" | PASS USENIX 2025 "now replaced by the Meta Content Library" |
| | PASS NDSS 2021 "1% streaming API that Twitter provides to vetted researchers" |
| | PASS NDSS 2021 "all the numbers that we presented in this paper are lower bounds" |
| | PASS IMC 2025 "TikTok and Instagram demonstrated the highest" |
| | PASS IMC 2025 "TikTok 1,700 816" |
| | PASS PETS 2026 "98.9% of all explanation texts cite only the main targeting form" |
| | PASS NDSS 2026 "roughly two-thirds of the images (66 %) contain detectable human faces" |
| | PASS NDSS 2026 "245 countries" |
| | PASS WWW 2019 "Australia (81.3%), and for the U.K. (74.4%)" |
| | PASS PETS 2026 "8,965 out of the 796,081" |
| | PASS IMC 2023 "The top 25% most populous instances contain 96% of the users" |
| | PASS IMC 2023 "2,879 unique Mastodon instances" |
| | PASS IMC 2025 "Facebook 649 37 5.70" |
| | PASS WWW 2026 "22,722" |
| | PASS IMC 2024 "29.5 million" |
| | PASS NDSS 2026 "63,170,000,000" |
| | PASS WWW 2024 "4.9M" |
| |
| 26 pass, 0 fail, of 26 checks | 42 pass, 0 fail, of 42 checks |
| </file> | </file> |
| |
| After those replacements: **26 checks, 26 pass, 0 fail.** | After those replacements: **26 checks, 26 pass, 0 fail.** |
| |
| One further note on the corpus text: some IMC volumes in ''paper.cols.txt'' include the **published reviewer comments** ("Reviewer #2 Strengths: ..."). A keyword probe over full text can therefore hit a reviewer's sentence rather than the authors'. Two of the early rate-limit examples in the probe output are exactly that. It does not change the paper counts materially at this scale, but a probe designed to measure what //authors// say would need to strip those sections. | One further note on the corpus text: some IMC volumes in ''paper.cols.txt'' include the **published reviewer comments** ("Reviewer #2 Strengths: ..."). A keyword probe over full text can therefore hit a reviewer's sentence rather than the authors'. Two of the early rate-limit examples in the probe output are exactly that. Quantified rather than waved away: **36 of the 5,869** ''paper.cols.txt'' files contain the string ''%%Reviewer #%%'', and **9 of the 897** platform-subject papers do. So the contamination is real and bounded at about 1% of the population every probe on this page ran over. A probe designed to measure what //authors// say would still need to strip those sections. |
| |
| ===== External sources ===== | ===== External sources ===== |
| - **A quote was attributed to the wrong paper.** The //Denominator problem// bullet on sampled streams cited ''%%hagen2021_numbers%%'' (NDSS 2021, //All the Numbers are US//) for ''%%"the 1% streaming API that Twitter provides to vetted researchers"%%''. That paper is about WhatsApp and Signal phone-number enumeration and contains **no occurrence of the word "Twitter" at all**. The quote belongs to a **different NDSS 2021 paper**, //To Err.Is Human: Characterizing the Threat of Unintended URLs in Social Media//, which the author had read in the same probe output and confused with it. Found by the citations reviewer. Fixed by adding ''%%kaleli2021_human%%'' to the bibliography and re-citing; two quote checks for it were added to ''%%platforms_quotecheck.mjs%%''. This is the worst class of error on the page: a quote check that only verifies "does this phrase exist somewhere in the corpus" would have passed it, and the one here passed because the phrase was never in the check list at all. **Every quote on a page belongs in the check list, not just the numbered figures.** | - **A quote was attributed to the wrong paper.** The //Denominator problem// bullet on sampled streams cited ''%%hagen2021_numbers%%'' (NDSS 2021, //All the Numbers are US//) for ''%%"the 1% streaming API that Twitter provides to vetted researchers"%%''. That paper is about WhatsApp and Signal phone-number enumeration and contains **no occurrence of the word "Twitter" at all**. The quote belongs to a **different NDSS 2021 paper**, //To Err.Is Human: Characterizing the Threat of Unintended URLs in Social Media//, which the author had read in the same probe output and confused with it. Found by the citations reviewer. Fixed by adding ''%%kaleli2021_human%%'' to the bibliography and re-citing; two quote checks for it were added to ''%%platforms_quotecheck.mjs%%''. This is the worst class of error on the page: a quote check that only verifies "does this phrase exist somewhere in the corpus" would have passed it, and the one here passed because the phrase was never in the check list at all. **Every quote on a page belongs in the check list, not just the numbered figures.** |
| - **A cross-platform range omitted the platform at the top of it.** The page reported blocking efficacy "//from 5.02% (YouTube) to 46.41% (Instagram)//" across {[beluri2025_exploration]}'s five platforms — leaving out TikTok, which is at or above Instagram. The paper's own summary is ''%%"TikTok and Instagram demonstrated the highest detection efficacy at 48%, whereas YouTube and Facebook showed the lowest efficacy at just 5%"%%''; its Table 8 gives TikTok 816 of 1,700, and the efficacy cell for that row is lost to a column splice, which is how it came to be dropped. Found by the citations reviewer. Fixed, with the paper's prose figure quoted rather than a recomputed one. | - **A cross-platform range omitted the platform at the top of it.** The page reported blocking efficacy "//from 5.02% (YouTube) to 46.41% (Instagram)//" across {[beluri2025_exploration]}'s five platforms — leaving out TikTok, which is at or above Instagram. The paper's own summary is ''%%"TikTok and Instagram demonstrated the highest detection efficacy at 48%, whereas YouTube and Facebook showed the lowest efficacy at just 5%"%%''; its Table 8 gives TikTok 816 of 1,700, and the efficacy cell for that row is lost to a column splice, which is how it came to be dropped. Found by the citations reviewer. Fixed, with the paper's prose figure quoted rather than a recomputed one. |
| | - **A true finding was framed so it invited a false inference.** The headline box's "official researcher routes are almost entirely absent from this literature" is correct as a count — the generic reviewer re-ran it with wider probes and the absolutes held — but a fresh reader would have concluded that the programmes do not work, when part of the absence is mechanical: 2023-and-later launches, a provisional 2025–2026 slice, and the venues that publish most such work (ICWSM, CHI, FAccT, communications journals) excluded by construction. That caveat existed on the page, 280 lines below the claim. Moved into the box. |
| | - **Three pages counted the same names and got different numbers, and none of them said so.** [[design:website_selection]] reports 463 papers for Alexa; this page reported 412; [[design:user_studies]] publishes 139 / 92 / 279 for Mechanical Turk against this page's 179. A subset appearing to exceed its superset is a signal that two folds disagree, not that one is broken. Reconciled and now printed by the report script: **486** papers name Alexa in a stated population source at all, **464** of those have a web-unit population tuple — which is what [[design:website_selection]] measures — and 412 is what survives our role rule. The content page carries the reconciliation and does not claim its fold is the better one. |
| | - **A schema label was published as an audited one.** The ''%%crawlConfig.authentication%%'' split (145 / 22 / 4 / 0) was stated as what papers "state", when [[programming:registration]] warns on this exact field that ''%%crawlConfig%%'' carries one evidence quote for the whole object so the label cannot be checked against it, and [[privacy:consent]]'s complete audit found **279 of 313 ''no-interaction'' labels unsupported (89.1%)**, alongside **7 of 36 (19.4%)** false positives among labels claiming interaction. We did not audit the 145; the page now says so. This is the finding most likely to move a number if someone does the audit. |
| - **No accidental exposure.** No credentials, participant data or unpublished material was written to the wiki. The only non-public thing touched was ''.env'' for the JSON-RPC credentials, read by ''scripts/dw.mjs''. | - **No accidental exposure.** No credentials, participant data or unpublished material was written to the wiki. The only non-public thing touched was ''.env'' for the JSON-RPC credentials, read by ''scripts/dw.mjs''. |
| |
| Four reviewers, each given the page text, the report scripts and their unedited output, and these notes, and each told explicitly that the author's context may not be exhaustive. | Four reviewers, each given the page text, the report scripts and their unedited output, and these notes, and each told explicitly that the author's context may not be exhaustive. |
| |
| Four reviewers, each given the page text, the report scripts and their unedited output, and these notes, and each told explicitly that the author's context may not be exhaustive. Three focused passes ran in parallel first; the generic pass ran afterwards against the corrected page. Rejections are recorded as fully as fixes — they are the only record of whether a reviewer earned its slot. | Three focused passes ran in parallel first; the generic pass ran afterwards against the corrected page. Rejections are recorded as fully as fixes — they are the only record of whether a reviewer earned its slot. |
| |
| ==== Reviewer 1 — figures against the scripts (Claude Sonnet) ==== | ==== Reviewer 1 — figures against the scripts (Claude Sonnet) ==== |
| ==== Reviewer 4 — generic, no checklist (Claude Fable) ==== | ==== Reviewer 4 — generic, no checklist (Claude Fable) ==== |
| |
| //Still running as this revision is saved; its findings are added in the same sitting.// | The generic pass was run against the corrected page, after the three focused ones. It produced 14 findings; 11 were accepted, 3 partly. It is the only reviewer that found anything about **framing** rather than about facts, and it earned its slot. |
| | |
| | ^ Finding ^ Verdict ^ Action ^ |
| | | The headline "almost entirely absent from this literature" is numerically right (it re-ran the probes wider and the absolutes held) but invites the inference that the programmes do not work; the mechanical explanation sat 280 lines below | **accepted** | caveat moved into the box | |
| | | The ''%%crawlConfig.authentication%%'' split republishes a schema read that [[programming:registration]] explicitly warns against on this field, with no audit and no caveat | **accepted — the highest-value finding** | caveat carried inline, with the sibling field's measured 89.1% unsupported-label rate (and 19.4% interacting-label false-positive rate); the 145 remain unaudited and the page says so | |
| | | Cross-page count conflicts: Alexa 412 here vs 463 on [[design:website_selection]] (a subset exceeding its superset), and Mechanical Turk 179 here vs 139 / 92 / 279 on [[design:user_studies]] | **accepted** | reconciled in the report script and on the page: 486 name Alexa at all, 464 with a web-unit population, 412 after the role rule | |
| | | This page's quote-check header claimed to cover "every figure the content page attributes to a paper" while ~12 attributed figures were in no check at all, and the embedded script and output were stale | **accepted** | 16 checks added (42 total, all passing), both ''%%<file>%%'' blocks regenerated, and the overclaim recorded above rather than quietly fixed | |
| | | "The top two rows are largely app-store work" — row 2 is Meta, not app-store work | **accepted** | rewritten to rows 1 and 7 | |
| | | Two pointers send the reader to [[design:mobile_and_app_measurement]] for skill-store and on-device-extraction material that page does not contain | **accepted** | reworded to say what that page does and does not cover | |
| | | Google is rank 1 with 323 papers and gets no route, no read-first and no pointer; and the page is silent on what Reddit currently offers | **accepted** | both named as explicit gaps — search/ads auditing in Open Questions, Reddit as a "we could not establish this" bullet | |
| | | "statistically indistinguishable from the 33.8% baseline" — no test was run, and the platform papers are inside that baseline | **accepted** | reworded to "essentially at the baseline", with both caveats stated | |
| | | "roughly twice as likely" does not name which baseline it is twice of; against crawling papers (11.5%) the ratio is ~1.3× | **accepted** | both baselines now named | |
| | | "Three practical consequences:" followed by four bullets | **accepted** | trivial fix | |
| | | ''%%TikTok-Api%%'' is the unofficial scraper library, on a page whose axis is official-versus-unofficial routes | **accepted** | row relabelled | |
| | | "61.5% of platform-subject papers name no instrument" converts absence-from-extracted-tool-lists into a claim about what papers state | **accepted** | softened to an upper bound, with the un-audited 552 stated | |
| | | The early routes table and the closing currency table duplicate each other row-for-row | **partly accepted** | the early table's status column reduced to one word and the evidence left to the closing section; the two tables are kept, because one is a route inventory and the other a dated verdict, and a skimmer needs the first | |
| | | The TikTok no-page justification ("four papers is a paragraph") reads as the page marking its own homework | **accepted** | rewritten to rest on venue scope: the TikTok literature is mostly outside these seven venues, so a page built from this corpus would misrepresent the field | |
| | | Smaller rigour points: a duplicated sentence in this log; a negative claim ("no academic tier") with no stated search method; an unquantified reassurance about reviewer-comment contamination; audit precisions quoted as bare points at //n// = 7–16 | **all accepted** | sentence deleted; the tier claim scoped to the pages an applicant is sent to; contamination quantified (36 of 5,869 files, 9 of 897 platform papers); a small-//n// caveat added here and on the page | |
| | | Confirmations: the framing claim //is// delivered (organised by route and denominator, not by vendor); length and section order match the house pattern; the read-first quartet is right for the stated reader; the box's zero/once absolutes survive wider independent probes | no action | — | |
| | |
| | One thing the generic reviewer reported that is **not** a page defect: the local checkout's ''pages/start.txt'' and ''pages/design.txt'' are stale relative to the live wiki. The live pages were updated in this sitting; the stale files are old local copies and were never the source of a save. |
| |
| |
| |
| <bibtex bibliography></bibtex> | <bibtex bibliography></bibtex> |
| | |
| | ===== ConsentAction audit propagation, 2026-09-17 ===== |
| | |
| | **What was wrong.** The page used the 19.4% interacting-side ''consentAction'' false-positive rate as its warning about the neighbouring authentication labels, even though the complete audit found a much larger no-interaction-side failure. |
| | |
| | **Command and real output.** ''node scripts/platforms_report.mjs'' → exit 0; its platform figures were unchanged. ''node scripts/consent_action_noninteraction_audit.mjs'' → **279/313 = 89.1%** of no-interaction labels unsupported; the interacting-side comparison remains **7/36 = 19.4%** false positives. |
| | |
| | **What changed.** [[design:platforms]] and this provenance page now cite the stronger 89.1% result alongside 19.4%. The **145-paper authentication split was not audited**; it is filed as a separate drain item. |
| | |
| | **Review.** One **model: 'sonnet'** figures-vs-script pass re-ran ''platforms_report.mjs'' and checked the authentication paragraph and the adjacent route/platform figures. No mismatch was found. No citation pass was needed: no citekey or quoted claim changed. |
| | |
| | **Rejected finding.** The 89.1% consentAction result was not used as evidence that authentication labels are wrong; it only motivates auditing the 145-paper authentication split next. |
| | |
| | ===== Audit of the 145 crawlConfig.authentication='none' platform papers, 2026-09-22 ===== |
| | |
| | **What was wrong.** [[design:platforms]] published the ''%%crawlConfig.authentication%%'' split for the 260 platform-subject papers with a crawl configuration — **145 (55.8%)** ''%%none%%'', 22 ''%%account-registration%%'', 4 ''%%manual-login%%'', 0 ''%%automated-login%%'' / ''%%sso%%'' — with the caveat that these were schema labels and that the 145 had not been audited, and filed the audit as the next item. This is that audit. The shape was right and the schema counts are unchanged; what was missing is that ''%%none%%'' is overwhelmingly an //inference from silence//, and that a measurable minority of the labels are contradicted by the paper. |
| | |
| | **Why the evidence quote cannot decide this.** ''%%crawlConfig%%'' is an object carrying **one** ''%%evidence.quote%%'' for statefulness, browsers, headless, consent action, interaction depth, authentication, languages and repeat visits together ([[programming:registration]], [[privacy:consent]]). So the audit did not read the quote as evidence; it read the **paper text** — ''%%paper.cols.txt%%'' and the raw ''%%paper.txt%%'' pypdf render, whitespace collapsed and de-hyphenated, because either rendering alone splices sentences. The sibling ''%%consentAction%%'' result (279 of 313 ''%%no-interaction%%'' labels unsupported, 89.1%) was **not** used as evidence that authentication labels are wrong; see the rejected finding below. |
| | |
| | **Command and real output.** |
| | |
| | <code> |
| | $ node scripts/platforms_report.mjs # unchanged, the schema split the page publishes |
| | ## The login wall: crawlConfig.authentication |
| | platform-subject papers with a crawlConfig object (260) |
| | none 145 55.8% |
| | not-stated 89 34.2% |
| | account-registration 22 8.5% |
| | manual-login 4 1.5% |
| | -> states a value: 171/260 = 65.8% |
| | |
| | $ node scripts/auth145_audit.mjs --verdicts |
| | ## Audit of the 145 platform-subject papers labelled crawlConfig.authentication='none' |
| | (denominator: the 145, not the 260 with a crawlConfig object and not the corpus) |
| | contradicted 6 4.1% |
| | affirmed 12 8.3% |
| | ambiguous 8 5.5% |
| | silent 119 82.1% |
| | total 145 |
| | </code> |
| | |
| | **How the candidate set was built, and what its recall rests on.** The 145 were re-derived inside ''%%scripts/auth145_audit.mjs%%'' with the same four signals ''%%platforms_report.mjs%%'' uses, and the script **throws** if the platform-subject-with-crawlConfig population is not 260 or the ''%%none%%'' count is not 145. Five paired probes then ran over the paper text (''%%--ctx%%''), each with a tight and a loose form, and the script **throws** if any tight probe returns more matches than its loose form in the same file — the first version did, on //Cookies That Give You Away//, which is why the guard exists. Stage 3 (''%%--buckets%%'') sorted all 145 into three candidate sets: papers with a first-person register/login sentence (**64**), papers with an explicit no-authentication sentence (**11**), and the residue where neither fires (**70**). **This sentence was wrong when first published, and a figures review caught it.** It said the pool read by hand was stage 3's A and C sets, 75 papers. It was not: the pool is the **union** of stage 2's strong first-person list and stage 3's A and C sets, and the review found the proof — //A View into YouTube View Fraud// carries a verdict below while sitting in stage 3's **B residue**, because stage 2 surfaced it and stage 3 did not. The script now computes that union itself and ''%%--verdicts%%'' **throws** if any adjudicated paper falls outside it (mutation-tested: disabling the stage 2 term makes it throw on exactly that paper). Its real output: **97 of 145** papers in the pool, **26** given a verdict, **71** read and left silent, and **48** that never surfaced a sentence to read. So the 119 **silent** are 71 papers read and found to say nothing about their own authentication posture plus 48 that no probe reached — not, as first published, 119 papers none of which were read. Recall therefore rests on the C probe being loose (it fires on any ''%%our ... account%%''), which is why it returned 64 candidates of which most were topical noise about the papers' //subjects// rather than their methods. |
| | |
| | **What changed.** [[design:platforms]]'s logged-out-surface bullet now carries the audit result in place of "We did not audit these 145". The schema counts (145 / 22 / 4 / 0, and the 55.8%) are **unchanged** — this is a precision audit published alongside the labels, in the same style as the Meta / Twitter-X / TikTok / Amazon subject-fold audits higher up this page. No number was hand-edited and no label was rewritten in ''%%extractions.jsonl%%''. |
| | |
| | **Two of the six read as contradicting their own object.** //A Large-scale Investigation into Geodifferences in Mobile Apps// (USENIX Security 2022) has a ''%%crawlConfig%%'' quote about the Selenium privacy-policy scraper while the study's actual download apparatus is ten Google Pixel 2 phones, one Google account per country, with an apk scraper that "requires the device's ID and Google account credentials". //Experimental Analyses of the Physical Surveillance Risks in Client-Side Content Scanning// (NDSS 2024) is scored ''%%none%%'' while its own ''%%crawlConfig.evidence.quote%%'' reads "we scrape user photographs from Instagram using an alias account" — the label contradicts the quote it was stored with. |
| | |
| | ==== scripts/auth145_audit.mjs ==== |
| | |
| | The committed file, verbatim. |
| | |
| | <file javascript auth145_audit.mjs> |
| | // Audit of the 145 platform-subject papers labelled crawlConfig.authentication='none'. |
| | // |
| | // Filed by design:platforms itself: "We did not audit these 145. Treat the |
| | // authentication shape as real and the precise share as unverified." |
| | // |
| | // The label cannot be checked against its own evidence quote, because |
| | // crawlConfig carries ONE evidence quote for the whole object (see |
| | // programming:registration). So this script does not read the quote as |
| | // evidence. It reads the PAPER TEXT. |
| | // |
| | // node scripts/auth145_audit.mjs # probe + candidate list |
| | // node scripts/auth145_audit.mjs --ctx <n> # printed context window (default 260) |
| | // node scripts/auth145_audit.mjs --control # random sample of non-firing papers |
| | // |
| | // The paper population is re-derived with the same four signals |
| | // platforms_report.mjs uses; the counts are asserted against that script's |
| | // published output (260 with a crawlConfig object, 145 labelled 'none') and |
| | // this script throws if either moves. |
| | |
| | import fs from 'node:fs'; |
| | import path from 'node:path'; |
| | import { loadExtractions, isSentinel, dataRoot } from './lib.mjs'; |
| | import { tagsOf, roleOf, familiesOf, ANY_PLATFORM } from './plat_fold.mjs'; |
| | |
| | const argv = process.argv.slice(2); |
| | const has = (f) => argv.includes(f); |
| | const val = (f, d) => (argv.indexOf(f) >= 0 ? argv[argv.indexOf(f) + 1] : d); |
| | const CTX = Number(val('--ctx', 260)); |
| | const pct = (n, d) => `${((100 * n) / d).toFixed(1)}%`; |
| | |
| | const rows = loadExtractions(); |
| | const key = (r) => `${r.venue}/${r.year}/${r.slug}`; |
| | const SIGNALS = { |
| | title: (r) => [r.title], |
| | sourceList: (r) => (r.population || []).map((p) => p.sourceList), |
| | phenomenon: (r) => (r.detection || []).map((d) => d.phenomenon), |
| | tool: (r) => |
| | (r.tools || []) |
| | .filter((t) => t.usedOrMentioned === 'used' || t.usedOrMentioned === 'produced') |
| | .map((t) => t.name), |
| | }; |
| | |
| | const perPaper = new Map(); |
| | for (const r of rows) { |
| | const k = key(r); |
| | const fams = new Map(); |
| | for (const [sig, get] of Object.entries(SIGNALS)) { |
| | for (const raw of get(r)) { |
| | if (!raw || isSentinel(raw)) continue; |
| | const s = String(raw); |
| | if (sig === 'sourceList' && (r.participants || []).length > 0) continue; |
| | for (const { family, role } of tagsOf(s)) { |
| | if (role !== 'subject') continue; |
| | if (!fams.has(family)) fams.set(family, new Set()); |
| | fams.get(family).add(sig); |
| | } |
| | } |
| | } |
| | if (fams.size) perPaper.set(k, fams); |
| | } |
| | const byKey = new Map(rows.map((r) => [key(r), r])); |
| | const anySubject = [...perPaper.keys()].map((k) => byKey.get(k)); |
| | const platCrawled = anySubject.filter((r) => r.crawlConfig !== null); |
| | const none = platCrawled.filter((r) => r.crawlConfig.authentication === 'none'); |
| | |
| | // Guard: this script's population must be platforms_report.mjs's population. |
| | if (platCrawled.length !== 260) |
| | throw new Error(`population drift: platform-subject papers with a crawlConfig object = ${platCrawled.length}, platforms_report.mjs publishes 260`); |
| | if (none.length !== 145) |
| | throw new Error(`population drift: authentication='none' = ${none.length}, platforms_report.mjs publishes 145`); |
| | |
| | // ------------------------------------------------------------------ full text |
| | // paper.cols.txt de-columns but leaves ~a quarter of pages spliced; paper.txt |
| | // is the raw pypdf render. Read BOTH and probe the union, whitespace collapsed, |
| | // because a PDF line break inside a phrase silently undercounts. |
| | const collapse = (s) => s.replace(//g, '').replace(/-\s*\n\s*/g, '').replace(/\s+/g, ' '); |
| | function textOf(r) { |
| | const dir = path.join(dataRoot(), 'fulltext', String(r.year), r.venue, r.slug); |
| | const out = []; |
| | for (const f of ['paper.cols.txt', 'paper.txt', 'paper.norm.txt']) { |
| | const p = path.join(dir, f); |
| | if (!fs.existsSync(p)) continue; |
| | const raw = fs.readFileSync(p, 'latin1'); |
| | if (raw.length === 0) continue; |
| | out.push({ file: f, text: collapse(raw) }); |
| | } |
| | return out; |
| | } |
| | |
| | // ------------------------------------------------------------------- probes |
| | // Two widths per concern. The tight probe must return a subset of the loose |
| | // one; the script asserts tight <= loose per paper and prints both. |
| | const PROBES = [ |
| | ['created-account', /\b(creat(?:e|ed|ing)|register(?:ed|ing)?|set up|made)\s+(?:\w+\s+){0,4}(accounts?|profiles?|personas?|sock[- ]?puppets?|bot accounts?)\b/i, |
| | /(account|profile|persona|sock)/i], |
| | ['logged-in', /\b(logged[- ]?in|log(?:ged)?\s+into|sign(?:ed|ing)?[- ]in(?:to)?|authenticated (?:session|crawl|user|request)|login(?:s|ed)? (?:to|into)|after (?:the )?login)\b/i, |
| | /(log|sign|authenticat)/i], |
| | ['credentials', /\b(credentials?|username and password|password(?:ed)? account|session cookies?|auth(?:entication)? tokens?|access tokens?|OAuth)\b/i, |
| | /(credential|password|cookie|token|oauth|username)/i], |
| | ['login-wall', /\b(login[- ]wall|behind a login|requires? (?:a )?(?:login|an account|sign[- ]?up)|logged[- ]?out (?:view|surface|user))\b/i, |
| | /(log|account|sign)/i], |
| | ['sso', /\b(single sign[- ]on|SSO|sign in with (?:Google|Facebook|Apple)|social login|identity provider)\b/i, |
| | /(sso|sign|single|identity provider)/i], |
| | ]; |
| | |
| | const results = []; |
| | for (const r of none) { |
| | const texts = textOf(r); |
| | const hits = {}; |
| | let anyText = texts.length > 0 && texts.some((t) => t.text.length > 2000); |
| | for (const [name, tight, loose] of PROBES) { |
| | let nT = 0, nL = 0, sample = null; |
| | for (const { text } of texts) { |
| | const mT = text.match(new RegExp(tight.source, 'gi')) || []; |
| | const mL = text.match(new RegExp(loose.source, 'gi')) || []; |
| | if (mT.length > mL.length) |
| | throw new Error(`probe ${name} on ${key(r)} in ${'' + text.length}: tight ${mT.length} > loose ${mL.length} (loose must contain tight)`); |
| | nT = Math.max(nT, mT.length); |
| | nL = Math.max(nL, mL.length); |
| | if (!sample && mT.length) { |
| | const i = text.search(new RegExp(tight.source, 'i')); |
| | sample = text.slice(Math.max(0, i - CTX / 2), i + CTX).trim(); |
| | } |
| | } |
| | hits[name] = { tight: nT, loose: nL, sample }; |
| | } |
| | results.push({ r, hits, anyText, nfiles: texts.length }); |
| | } |
| | |
| | const fired = results.filter((x) => Object.values(x.hits).some((h) => h.tight > 0)); |
| | const quiet = results.filter((x) => !Object.values(x.hits).some((h) => h.tight > 0)); |
| | const notext = results.filter((x) => !x.anyText); |
| | |
| | console.log(`## Population`); |
| | console.log(` platform-subject papers with a crawlConfig object : ${platCrawled.length}`); |
| | console.log(` of those, crawlConfig.authentication = 'none' : ${none.length}`); |
| | console.log(` full text readable for : ${results.length - notext.length}/${none.length}`); |
| | if (notext.length) for (const x of notext) console.log(` NO TEXT ${key(x.r)}`); |
| | |
| | console.log(`\n## Probe totals over the 145 (papers, not matches)`); |
| | for (const [name] of PROBES) { |
| | const t = results.filter((x) => x.hits[name].tight > 0).length; |
| | const l = results.filter((x) => x.hits[name].loose > 0).length; |
| | console.log(` ${name.padEnd(16)} tight ${String(t).padStart(3)} loose ${String(l).padStart(3)}`); |
| | } |
| | console.log(`\n any tight probe fires : ${fired.length}/${none.length}`); |
| | console.log(` no tight probe fires : ${quiet.length}/${none.length}`); |
| | |
| | if (has('--control')) { |
| | // Deterministic sample of the quiet papers, for the false-negative side. |
| | const s = quiet.slice().sort((a, b) => key(a.r).localeCompare(key(b.r))); |
| | const pick = []; |
| | for (let i = 0; i < s.length; i += Math.max(1, Math.floor(s.length / 20))) pick.push(s[i]); |
| | console.log(`\n## Control: ${pick.length} papers where no tight probe fired`); |
| | for (const x of pick) console.log(` ${key(x.r)} ${x.r.title}`); |
| | process.exit(0); |
| | } |
| | |
| | console.log(`\n## Candidates (tight probe fired) — ${fired.length}`); |
| | for (const x of fired.sort((a, b) => key(a.r).localeCompare(key(b.r)))) { |
| | const names = Object.entries(x.hits).filter(([, h]) => h.tight > 0).map(([n, h]) => `${n}:${h.tight}`); |
| | console.log(`\n--- ${key(x.r)}`); |
| | console.log(` title: ${x.r.title}`); |
| | console.log(` probes: ${names.join(' ')}`); |
| | console.log(` crawlConfig.evidenceQuote: ${JSON.stringify(x.r.crawlConfig.evidenceQuote || null)}`); |
| | for (const [n, h] of Object.entries(x.hits)) if (h.sample) console.log(` [${n}] ${h.sample}`); |
| | } |
| | |
| | // ---------------------------------------------------------- stage 2: --method |
| | // A tight probe fires on any mention of accounts or logins, including the |
| | // paper's SUBJECT (spam accounts, credential theft). What decides the |
| | // authentication label is what the AUTHORS did. Stage 2 keeps only sentences |
| | // that carry a first-person / own-apparatus subject AND an auth term, and |
| | // prints them in full for hand reading. Run over all 145, not just the 82, |
| | // because sentence co-occurrence is a different probe, not a narrowing of |
| | // stage 1. |
| | if (has('--method')) { |
| | const FIRST = /\b(we|our|us|the authors?|this (?:paper|study|work)|i )\b/i; |
| | const APPARATUS = /\b(crawler?s?|scraper?s?|bots?|browser|instrumentation|framework|pipeline|measurement|collection|harvest)\b/i; |
| | const AUTH = /\b(account|log ?in|logged|sign(?:ed|ing)? ?in|authenticat|credential|password|session cookie|API key|access token|OAuth|sock ?puppet|persona|profile we|registered)\b/i; |
| | let printed = 0; |
| | for (const x of results.sort((a, b) => key(a.r).localeCompare(key(b.r)))) { |
| | const seen = new Set(); |
| | const keep = []; |
| | for (const { text } of textOf(x.r)) { |
| | for (const sent of text.split(/(?<=[.!?])\s+(?=[A-Z(])/)) { |
| | if (sent.length < 30 || sent.length > 700) continue; |
| | if (!AUTH.test(sent)) continue; |
| | if (!FIRST.test(sent) && !APPARATUS.test(sent)) continue; |
| | const norm = sent.toLowerCase().replace(/[^a-z0-9]/g, '').slice(0, 90); |
| | if (seen.has(norm)) continue; |
| | seen.add(norm); |
| | keep.push(sent.trim()); |
| | } |
| | } |
| | console.log(`\n=== ${key(x.r)}`); |
| | console.log(` ${x.r.title}`); |
| | console.log(` quote: ${JSON.stringify(x.r.crawlConfig.evidenceQuote || null)}`); |
| | if (!keep.length) console.log(' (no first-person auth sentence)'); |
| | for (const s of keep.slice(0, 14)) console.log(` * ${s}`); |
| | printed++; |
| | } |
| | console.log(`\n-- ${printed} papers printed`); |
| | } |
| | |
| | // ------------------------------------------------------- stage 3: --buckets |
| | // `none` and `not-stated` are BOTH available in the schema, so `none` is an |
| | // affirmative claim: the crawler did not log in or register. Stage 3 sorts the |
| | // 145 by what the paper actually says about its own crawl: |
| | // A affirmed — the paper states it did not authenticate |
| | // C contradicted — the paper describes registering or logging in to collect data |
| | // B silent — neither; `none` was inferred from the paper saying nothing |
| | // A and C are candidate sets, hand-read one by one; B is the residue and is |
| | // printed in full so it can be counted, not assumed. |
| | if (has('--buckets')) { |
| | // "does not take into account" / "accounts for 75% of the market" swamp any |
| | // probe that treats the bare word `account` as an authentication term: all 20 |
| | // hits of the first version of AFFIRM were that. AFFIRM therefore requires a |
| | // login/authentication term, and excludes the two idioms explicitly. |
| | const IDIOM = /(take[ns]? |taking |taken )?into account|accounts? for\b|on account of/i; |
| | const AFFIRM = [ |
| | /\b(without|never|not|no|did not|does not|neither|refrain\w* from)\b[^.]{0,90}\b(logg?(?:ed|ing)? ?in(?:to)?|log ?in|sign(?:ed|ing)? ?in|authenticat\w*|login wall|creat\w+ an account|registering|registration)\b/i, |
| | /\b(logged[- ]?out|unauthenticated|without (?:an? )?(?:account|login)|publicly available without)\b/i, |
| | /\bdo(?:es)? not (?:require|need)\b[^.]{0,40}\b(login|logging|account|authenticat\w*)\b/i, |
| | ]; |
| | const CONTRA = [ |
| | /\b(we|our)\b[^.]{0,120}?\b(creat\w+|register\w+|set ?up|sign(?:ed)? up|purchas\w+)\b[^.]{0,60}?\b(account|profile|persona|sock ?puppet)s?\b/i, |
| | /\b(we|our)\b[^.]{0,120}?\b(logg?ed (?:in|into)|log(?:ged)? in to|signed in(?:to)?|authenticat\w+)\b/i, |
| | /\bour\b[^.]{0,40}\b(accounts?|credentials?|logins?)\b/i, |
| | /\b(account|login) credentials?\b[^.]{0,60}\b(crawler|scraper|tool|script|download)/i, |
| | ]; |
| | const bucket = { A: [], B: [], C: [] }; |
| | for (const x of results) { |
| | const sents = []; |
| | for (const { text } of textOf(x.r)) |
| | for (const s of text.split(/(?<=[.!?])\s+(?=[A-Z(])/)) if (s.length >= 30 && s.length <= 700) sents.push(s); |
| | const a = sents.filter((s) => !IDIOM.test(s) && AFFIRM.some((p) => p.test(s))); |
| | const c = sents.filter((s) => CONTRA.some((p) => p.test(s))); |
| | x.aHits = a; x.cHits = c; |
| | if (c.length) bucket.C.push(x); |
| | else if (a.length) bucket.A.push(x); |
| | else bucket.B.push(x); |
| | } |
| | console.log(`\n## Stage 3 candidate buckets over the 145 (probe output, NOT the audit result)`); |
| | console.log(` C paper has a first-person register/login sentence : ${bucket.C.length}`); |
| | console.log(` A paper has an explicit no-authentication sentence : ${bucket.A.length}`); |
| | console.log(` B residue, neither probe fires : ${bucket.B.length}`); |
| | console.log(` total ${bucket.A.length + bucket.B.length + bucket.C.length} (must be 145)`); |
| | if (bucket.A.length + bucket.B.length + bucket.C.length !== 145) throw new Error('buckets do not sum to 145'); |
| | for (const [name, set] of Object.entries(bucket)) { |
| | console.log(`\n######## BUCKET ${name} (${set.length})`); |
| | for (const x of set.sort((p, q) => key(p.r).localeCompare(key(q.r)))) { |
| | console.log(`\n=== ${key(x.r)}\n ${x.r.title}`); |
| | console.log(` quote: ${JSON.stringify(x.r.crawlConfig.evidence ? x.r.crawlConfig.evidence.quote : null)}`); |
| | const seen = new Set(); |
| | for (const s of [...x.cHits.map((v) => ['C', v]), ...x.aHits.map((v) => ['A', v])].slice(0, 40)) { |
| | const n = s[1].toLowerCase().replace(/[^a-z0-9]/g, '').slice(0, 80); |
| | if (seen.has(n)) continue; seen.add(n); |
| | if (seen.size > 8) break; |
| | console.log(` ${s[0]}> ${s[1]}`); |
| | } |
| | } |
| | } |
| | } |
| | |
| | // ------------------------------------------------------- stage 4: --verdicts |
| | // The audit result. Every verdict below was reached by READING the paper, not |
| | // by reading crawlConfig.evidence.quote — that quote covers the whole |
| | // crawlConfig object and cannot adjudicate this field (programming:registration). |
| | // |
| | // The candidate set that was read is stage 3's A and C buckets (the papers with |
| | // any first-person sentence about accounts, logins or authentication, either |
| | // direction). Papers in neither bucket are SILENT by construction: they contain |
| | // no first-person statement about their own authentication posture, so `none` |
| | // there is an inference from the paper saying nothing, not a reading of it. |
| | // |
| | // affirmed the paper states its collection was unauthenticated |
| | // contradicted the paper describes registering or using accounts FOR the |
| | // collection the crawlConfig object describes |
| | // ambiguous the study used accounts, but the crawl the evidence quote |
| | // describes plausibly did not; a reasonable person could score |
| | // this either way |
| | // silent no statement either way (the residue; not individually read) |
| | const VERDICTS = { |
| | // ---- contradicted (6) |
| | 'NDSS/2012/warningbird-detecting-suspicious-urls-in-twitter-stream': |
| | ['contradicted', 'Collection is the Twitter Streaming API under the authors’ own authenticated accounts: "To collect tweets, we use Twitter Streaming APIs. Our accounts have the Spritzer access role".'], |
| | 'WWW/2013/i-know-the-shortened-urls-you-clicked-on-twitter-inference-attack-using-public-c': |
| | ['contradicted', '"To perform the attack, we create monitoring accounts that monitor messages from all followings of a target user" — and "We created a Twitter account and followed 29 Twitter users". The accounts are the collection apparatus.'], |
| | 'USENIX/2022/a-large-scale-investigation-into-geodifferences-in-mobile-apps': |
| | ['contradicted', '"We set up these phones with one Google account per country" and "The apk scraper requires the device’s ID and Google account credentials". The crawlConfig quote describes only the Selenium privacy-policy sub-crawl.'], |
| | 'USENIX/2022/exploring-the-unchartered-space-of-container-registry-typosquatting': |
| | ['contradicted', '"We register all of our accounts legally through Docker Hub and Google Container Registry"; pull counts are read from the authors’ own Docker Hub and Google Cloud Console, and 20,000 AWS accounts were spawned from a registered pilot account.'], |
| | 'NDSS/2024/experimental-analyses-of-the-physical-surveillance-risks-in-client-side-content-scanning': |
| | ['contradicted', 'The crawlConfig evidence quote ITSELF says "we scrape user photographs from Instagram using an alias account". The label contradicts its own quote.'], |
| | 'USENIX/2025/no-way-to-sign-out-unpacking-non-compliance-with-google-plays-app-account-deleti': |
| | ['contradicted', '"We registered around 300 accounts in total throughout the study. Each app was assigned two registered accounts"; "we manually registered, deleted, relogged in, and re-registered accounts".'], |
| | // ---- affirmed (12) |
| | 'IMC/2010/detecting-and-characterizing-social-spam-campaigns': |
| | ['affirmed', '"Because access to regional networks is unauthenticated, we chose 8 regional networks … as targets for data collection."'], |
| | 'WWW/2012/serf-and-turf-crowdturfing-for-fun-and-profit': |
| | ['affirmed', '"All information on both sites is publicly available, and neither site employs security measures to prevent crawling." The accounts the authors created were for a separate end-to-end campaign experiment, not the crawl.'], |
| | 'USENIX/2014/towards-detecting-anomalous-user-behavior-in-online-social-networks': |
| | ['affirmed', '"We did not seek or receive any account credentials of any Facebook user."'], |
| | 'IEEE-SP/2018/the-spyware-used-in-intimate-partner-violence': |
| | ['affirmed', '"We did not set up any user profile, and performed each search from a new browser session." Accounts were created only during manual app analysis, not for the search crawl.'], |
| | 'WWW/2019/measuring-political-personalization-of-google-news-search': |
| | ['affirmed', '"we do not create or log-in to a Firefox account during training"; "We also do not create or log-in to a Google account".'], |
| | 'USENIX/2020/the-tools-and-tactics-used-in-intimate-partner-surveillance-an-analysis-of-onlin': |
| | ['affirmed', '"Our data is from publicly available fora accessible on the Internet without authentication."'], |
| | 'USENIX/2022/behind-the-tube-exploitative-monetization-of-content-on-youtube': |
| | ['affirmed', '"We considered only sites publicly accessible … and did not require registration or prior site activity to view information."'], |
| | 'IEEE-SP/2024/no-easy-way-out-the-effectiveness-of-deplatforming-an-extremist-forum-to-suppres': |
| | ['affirmed', '"we refrain from scraping private and protected posts behind the login wall due to safety and legality concerns."'], |
| | 'USENIX/2025/lost-in-the-mists-of-time-expirations-in-dns-footprints-of-mobile-apps': |
| | ['affirmed', '"We did not attempt to bypass any login or splash screens of apps."'], |
| | 'PETS/2026/banned-books-analysis-of-censorship-on-amazon-com': |
| | ['affirmed', '"at no point in our methodology do we sign into any Amazon account"; "all data we collected was available publicly without being logged in".'], |
| | 'PETS/2026/the-tcf-doesn-t-really-a-a-id-automatic-privacy-analysis-and-legal-compliance-of': |
| | ['affirmed', '"Our Selenium instance was not logged in our Google account while crawling" — which is also the crawlConfig evidence quote.'], |
| | 'USENIX/2026/bridges-to-self-silent-web-to-app-tracking-on-mobile-via-localhost': |
| | ['affirmed', '"We did not use authenticated user accounts and limited interactions to what was necessary to observe default script behavior."'], |
| | // ---- ambiguous (8) |
| | 'CCS/2010/spam-the-underground-on-140-characters-or-less': |
| | ['ambiguous', 'The URL crawler in the evidence quote is unauthenticated, but the Twitter monitoring used a "fake account to act as a spam trap".'], |
| | 'CCS/2022/ready-raider-one-exploring-the-misuse-of-cloud-gaming-services': |
| | ['ambiguous', '"We registered all of our accounts legally through cloud gaming services" — but the crawlConfig quote is the Steam game-database crawler, which plausibly needed none.'], |
| | 'CCS/2023/dont-leak-your-keys-understanding-measuring-and-exploiting-the-appsecret-leaks-i': |
| | ['ambiguous', '"we first registered two accounts, a victim account and an attack account" — for a single proof-of-concept attack, not for the keyword-seeded mini-program crawl in the quote.'], |
| | 'USENIX/2018/o-single-sign-off-where-art-thou-an-empirical-analysis-of-single-sign-on-account': |
| | ['ambiguous', 'The 1M-site SSO-prevalence crawl only detects login/signup pages, but the RP takeover study "create[s] a new account using SSO" for each website.'], |
| | 'USENIX/2022/ovrseen-auditing-network-traffic-and-privacy-policies-in-oculus-vr': |
| | ['ambiguous', 'The webstore crawl in the quote is unauthenticated; the network-traffic collection that is the paper’s main measurement used "our test OVR account".'], |
| | 'WWW/2022/a-view-into-youtube-view-fraud': |
| | ['ambiguous', '"we generated 20 API keys (the maximum number of keys allowed by Google per account)" — an API credential tied to a Google account, which the schema’s "logged in or registered accounts" does not clearly cover.'], |
| | 'PETS/2024/privacy-policies-on-the-fediverse-a-case-study-of-mastodon-instances': |
| | ['ambiguous', 'The 351-policy Selenium download was unauthenticated; one account was created afterwards to check a single instance’s cookie claim.'], |
| | 'USENIX/2025/darkgram-a-large-scale-analysis-of-cybercriminal-activity-channels-on-telegram': |
| | ['ambiguous', '"we used the official Telegram API to collect 64,801 posts" — Telegram API access is bound to an account, but the paper never says so.'], |
| | }; |
| | |
| | // The candidate pool that was actually hand-read is the UNION of two probes, |
| | // not stage 3 alone. Stage 2 (--method) prints, for all 145, the sentences that |
| | // carry a first-person/own-apparatus subject AND an auth term; the "strong" |
| | // subset of that list is what was read. Stage 3 (--buckets) adds its A and C |
| | // sets. A figures review caught that WWW/2022/a-view-into-youtube-view-fraud |
| | // carries a verdict while sitting in stage 3's B residue — it was surfaced by |
| | // stage 2, not stage 3. The pool is therefore computed here, and --verdicts |
| | // THROWS if any adjudicated paper is outside it, so the script can no longer |
| | // claim a candidate-discovery process it does not reproduce. |
| | const STRONG = [ |
| | /\b(we|our)\b[^.]{0,140}?\b(creat\w*|register\w*|set up|signed up|sign up|made|use[d]?|operat\w*|control\w*|maintain\w*)\b[^.]{0,60}?\b(account|profile|persona|sock ?puppet|bot account)s?\b/i, |
| | /\b(we|our)\b[^.]{0,140}?\b(logged[- ]?in|log ?in|logged into|signed in|authenticat\w*|credential\w*|session cookie|access token|api key|oauth|bearer token)\b/i, |
| | /\b(logged[- ]?in|authenticated)\b[^.]{0,60}\b(crawl\w*|session|browser|account|user|state)\b/i, |
| | /\b(without|not)\b[^.]{0,40}\b(log ?in|logging in|account|authenticat\w*)\b/i, |
| | ]; |
| | function stage2Strong(x) { |
| | for (const { text } of textOf(x.r)) |
| | for (const sent of text.split(/(?<=[.!?])\s+(?=[A-Z(])/)) { |
| | if (sent.length < 30 || sent.length > 700) continue; |
| | if (STRONG.some((p) => p.test(sent))) return true; |
| | } |
| | return false; |
| | } |
| | |
| | if (has('--verdicts')) { |
| | const keys145 = new Set(none.map(key)); |
| | for (const k of Object.keys(VERDICTS)) if (!keys145.has(k)) throw new Error(`verdict for a paper that is not in the 145: ${k}`); |
| | // Recompute the candidate pool and assert every verdict came out of it. |
| | const AF = [ |
| | /\b(without|never|not|did not|does not|neither|refrain\w* from)\b[^.]{0,90}\b(logg?(?:ed|ing)? ?in(?:to)?|log ?in|sign(?:ed|ing)? ?in|authenticat\w*|login wall|creat\w+ an account|registering|registration)\b/i, |
| | /\b(logged[- ]?out|unauthenticated|without (?:an? )?(?:account|login)|publicly available without)\b/i, |
| | ]; |
| | const CO = [ |
| | /\b(we|our)\b[^.]{0,120}?\b(creat\w+|register\w+|set ?up|sign(?:ed)? up|purchas\w+)\b[^.]{0,60}?\b(account|profile|persona|sock ?puppet)s?\b/i, |
| | /\b(we|our)\b[^.]{0,120}?\b(logg?ed (?:in|into)|log(?:ged)? in to|signed in(?:to)?|authenticat\w+)\b/i, |
| | /\bour\b[^.]{0,40}\b(accounts?|credentials?|logins?)\b/i, |
| | ]; |
| | const IDIOM2 = /(take[ns]? |taking |taken )?into account|accounts? for\b|on account of/i; |
| | const pool = new Set(); |
| | for (const x of results) { |
| | let hit = stage2Strong(x); |
| | if (!hit) |
| | for (const { text } of textOf(x.r)) { |
| | for (const sent of text.split(/(?<=[.!?])\s+(?=[A-Z(])/)) { |
| | if (sent.length < 30 || sent.length > 700) continue; |
| | if (CO.some((p) => p.test(sent)) || (!IDIOM2.test(sent) && AF.some((p) => p.test(sent)))) { hit = true; break; } |
| | } |
| | if (hit) break; |
| | } |
| | if (hit) pool.add(key(x.r)); |
| | } |
| | for (const k of Object.keys(VERDICTS)) |
| | if (!pool.has(k)) throw new Error(`adjudicated paper outside the candidate pool the script computes: ${k}`); |
| | console.log(`\n## Candidate pool actually read (stage 2 strong list UNION stage 3 buckets A and C)`); |
| | console.log(` papers in the pool : ${pool.size}/145`); |
| | console.log(` of those, given a verdict : ${Object.keys(VERDICTS).length}`); |
| | console.log(` of those, read and left silent: ${pool.size - Object.keys(VERDICTS).length}`); |
| | console.log(` never surfaced a sentence : ${145 - pool.size}`); |
| | const counts = { contradicted: 0, affirmed: 0, ambiguous: 0, silent: 0 }; |
| | const rowsOut = []; |
| | for (const r of none) { |
| | const v = VERDICTS[key(r)]; |
| | const verdict = v ? v[0] : 'silent'; |
| | counts[verdict]++; |
| | if (v) rowsOut.push([verdict, key(r), r.title, v[1]]); |
| | } |
| | const total = Object.values(counts).reduce((a, b) => a + b, 0); |
| | if (total !== 145) throw new Error(`verdicts sum to ${total}, not 145`); |
| | console.log(`\n## Audit of the 145 platform-subject papers labelled crawlConfig.authentication='none'`); |
| | console.log(` (denominator: the 145, not the 260 with a crawlConfig object and not the corpus)`); |
| | for (const k of ['contradicted', 'affirmed', 'ambiguous', 'silent']) |
| | console.log(` ${k.padEnd(13)} ${String(counts[k]).padStart(3)} ${pct(counts[k], 145)}`); |
| | console.log(` total ${total}`); |
| | console.log(`\n## Per-paper evidence for every non-silent verdict`); |
| | for (const k of ['contradicted', 'affirmed', 'ambiguous']) |
| | for (const [v, kk, t, why] of rowsOut.filter((x) => x[0] === k)) |
| | console.log(`\n [${v}] ${kk}\n ${t}\n ${why}`); |
| | } |
| | |
| | </file> |
| | |
| | ==== The 26 adjudicated papers ==== |
| | |
| | Generated from ''%%node scripts/auth145_audit.mjs --verdicts%%''. The other 119 of the 145 are **silent**: no first-person sentence about the study's own authentication posture in either direction. |
| | |
| | ^ Verdict ^ Paper ^ Deciding sentence, from the paper text ^ |
| | | **contradicted** | //WarningBird: Detecting Suspicious URLs in Twitter Stream// (NDSS 2012) | Collection is the Twitter Streaming API under the authors’ own authenticated accounts: "To collect tweets, we use Twitter Streaming APIs. Our accounts have the Spritzer access role". | |
| | | **contradicted** | //I know the shortened URLs you clicked on Twitter: inference attack using public click analytics and Twitter metadata// (WWW 2013) | "To perform the attack, we create monitoring accounts that monitor messages from all followings of a target user" — and "We created a Twitter account and followed 29 Twitter users". The accounts are the collection apparatus. | |
| | | **contradicted** | //A Large-scale Investigation into Geodifferences in Mobile Apps// (USENIX 2022) | "We set up these phones with one Google account per country" and "The apk scraper requires the device’s ID and Google account credentials". The crawlConfig quote describes only the Selenium privacy-policy sub-crawl. | |
| | | **contradicted** | //Exploring the Unchartered Space of Container Registry Typosquatting// (USENIX 2022) | "We register all of our accounts legally through Docker Hub and Google Container Registry"; pull counts are read from the authors’ own Docker Hub and Google Cloud Console, and 20,000 AWS accounts were spawned from a registered pilot account. | |
| | | **contradicted** | //Experimental Analyses of the Physical Surveillance Risks in Client-Side Content Scanning// (NDSS 2024) | The crawlConfig evidence quote ITSELF says "we scrape user photographs from Instagram using an alias account". The label contradicts its own quote. | |
| | | **contradicted** | //No Way to Sign Out? Unpacking Non-Compliance with Google Play's App Account Deletion Requirements// (USENIX 2025) | "We registered around 300 accounts in total throughout the study. Each app was assigned two registered accounts"; "we manually registered, deleted, relogged in, and re-registered accounts". | |
| | | **affirmed** | //Detecting and characterizing social spam campaigns// (IMC 2010) | "Because access to regional networks is unauthenticated, we chose 8 regional networks … as targets for data collection." | |
| | | **affirmed** | //Serf and turf: crowdturfing for fun and profit// (WWW 2012) | "All information on both sites is publicly available, and neither site employs security measures to prevent crawling." The accounts the authors created were for a separate end-to-end campaign experiment, not the crawl. | |
| | | **affirmed** | //Towards Detecting Anomalous User Behavior in Online Social Networks// (USENIX 2014) | "We did not seek or receive any account credentials of any Facebook user." | |
| | | **affirmed** | //Measuring Political Personalization of Google News Search// (WWW 2019) | "we do not create or log-in to a Firefox account during training"; "We also do not create or log-in to a Google account". | |
| | | **affirmed** | //The Tools and Tactics Used in Intimate Partner Surveillance: An Analysis of Online Infidelity Forums// (USENIX 2020) | "Our data is from publicly available fora accessible on the Internet without authentication." | |
| | | **affirmed** | //Behind the Tube: Exploitative Monetization of Content on YouTube// (USENIX 2022) | "We considered only sites publicly accessible … and did not require registration or prior site activity to view information." | |
| | | **affirmed** | //No Easy Way Out: the Effectiveness of Deplatforming an Extremist Forum to Suppress Hate and Harassment// (IEEE-SP 2024) | "we refrain from scraping private and protected posts behind the login wall due to safety and legality concerns." | |
| | | **affirmed** | //Banned Books: Analysis of Censorship on Amazon.com// (PETS 2026) | "at no point in our methodology do we sign into any Amazon account"; "all data we collected was available publicly without being logged in". | |
| | | **affirmed** | //The TCF doesn’t really A(A)ID – Automatic Privacy Analysis and Legal Compliance of TCF-based Android Applications// (PETS 2026) | "Our Selenium instance was not logged in our Google account while crawling" — which is also the crawlConfig evidence quote. | |
| | | **affirmed** | //The Spyware Used in Intimate Partner Violence// (IEEE-SP 2018) | "We did not set up any user profile, and performed each search from a new browser session." Accounts were created only during manual app analysis, not for the search crawl. | |
| | | **affirmed** | //Lost in the Mists of Time: Expirations in DNS Footprints of Mobile Apps// (USENIX 2025) | "We did not attempt to bypass any login or splash screens of apps." | |
| | | **affirmed** | //Bridges to Self: Silent Web-to-App Tracking on Mobile via Localhost// (USENIX 2026) | "We did not use authenticated user accounts and limited interactions to what was necessary to observe default script behavior." | |
| | | **ambiguous** | //@spam: the underground on 140 characters or less// (CCS 2010) | The URL crawler in the evidence quote is unauthenticated, but the Twitter monitoring used a "fake account to act as a spam trap". | |
| | | **ambiguous** | //O Single Sign-Off, Where Art Thou? An Empirical Analysis of Single Sign-On Account Hijacking and Session Management on the Web// (USENIX 2018) | The 1M-site SSO-prevalence crawl only detects login/signup pages, but the RP takeover study "create[s] a new account using SSO" for each website. | |
| | | **ambiguous** | //Ready Raider One: Exploring the Misuse of Cloud Gaming Services// (CCS 2022) | "We registered all of our accounts legally through cloud gaming services" — but the crawlConfig quote is the Steam game-database crawler, which plausibly needed none. | |
| | | **ambiguous** | //OVRseen: Auditing Network Traffic and Privacy Policies in Oculus VR// (USENIX 2022) | The webstore crawl in the quote is unauthenticated; the network-traffic collection that is the paper’s main measurement used "our test OVR account". | |
| | | **ambiguous** | //A View into YouTube View Fraud// (WWW 2022) | "we generated 20 API keys (the maximum number of keys allowed by Google per account)" — an API credential tied to a Google account, which the schema’s "logged in or registered accounts" does not clearly cover. | |
| | | **ambiguous** | //Don't Leak Your Keys: Understanding, Measuring, and Exploiting the AppSecret Leaks in Mini-Programs// (CCS 2023) | "we first registered two accounts, a victim account and an attack account" — for a single proof-of-concept attack, not for the keyword-seeded mini-program crawl in the quote. | |
| | | **ambiguous** | //Privacy Policies on the Fediverse: A Case Study of Mastodon Instances// (PETS 2024) | The 351-policy Selenium download was unauthenticated; one account was created afterwards to check a single instance’s cookie claim. | |
| | | **ambiguous** | //DarkGram: A Large-Scale Analysis of Cybercriminal Activity Channels on Telegram// (USENIX 2025) | "we used the official Telegram API to collect 64,801 posts" — Telegram API access is bound to an account, but the paper never says so. | |
| | |
| | **Review.** One **model: 'sonnet'** figures-vs-script pass re-ran ''%%platforms_report.mjs%%'', ''%%report_registration.mjs%%'' and ''%%scripts/auth145_audit.mjs%%'' in all four modes, and checked every figure in the rewritten bullet plus the figures either side of it (the 383/208/175 existing-dataset split above it, the 897 denominator, the 34.2% not-stated share below it). Every verdict count, percentage and table row matched. It raised **two findings, both accepted**: (1) the candidate-pool sentence above misdescribed which papers were read, fixed above with a new script-computed guard; (2) [[programming:interaction]] enumerated the 857 web crawls as 553 + 204 + 81 = **838**, silently dropping the 19 papers with no ''%%crawlConfig%%'' object that ''%%report_registration.mjs%%'' prints as a sixth bucket — that page now carries the 19 and the sum reconciles. It also flagged that the ''%%kumar2022_investigation%%'' quote on [[design:platforms]] was ellipsis-joined across two sentences; it has been replaced with the contiguous sentence the audit script itself verifies. One **model: 'sonnet'** citations pass ran, because the bullet added two ''%%{[citekey]}%%'' markers and several verbatim quotes. It verified both citekeys against the papers' author lists, titles, venues and years, and **all 26** table rows verbatim against the paper text, reading the PDF directly where a column-spliced render made a quote look non-contiguous and stripping NUL bytes from WarningBird's text where plain ''%%grep%%'' silently returns nothing. It independently confirmed from ''%%extractions.jsonl%%'' that the Instagram //"alias account"// line really is that record's own ''%%crawlConfig.evidence.quote%%'' under an ''%%authentication: none%%'' label. It found nothing wrong. |
| | |
| | **Rejected finding.** That the 89.1% ''%%consentAction%%'' unsupported-label rate is evidence the authentication labels are wrong. It is not, and it was not used as such: it is a different field with a different extraction failure mode, and the authentication verdicts above each rest on a sentence from the paper in question. The 89.1% is the reason the audit was prioritised, not an input to any verdict on this page. |
| | |
| | **Deliberately not concluded.** That the corrected split is 139 / 28 / 4 / 0. Six labels being contradicted does not license re-deriving the published counts by hand: the corrected value for each of the six is itself a judgement (WarningBird's authenticated Streaming API accounts are not obviously ''%%account-registration%%'' rather than ''%%automated-login%%''), and the **8 ambiguous** papers would move the result either way depending on whether "the crawler" means the study's apparatus or the specific crawl the evidence quote names. The schema counts stay; the audit sits beside them. |
| |
