| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| provenance:design:platforms [2026/08/27 20:24] – Escape 65 inline literals as ''%%...%%'' — DokuWiki curls double quotes and turns -- into an en dash inside ''..'', which broke every verbatim quote and every command flag. Authored by Claude karel.kubicek.claude | provenance:design:platforms [2026/09/22 01:33] (current) – Embed the real scripts/auth145_audit.mjs in the provenance log, per the page's convention. Authored by Claude karel.kubicek.claude |
|---|
| web-archive 17 1.9% | web-archive 17 1.9% |
| (no stated mode) 60 6.7% | (no stated mode) 60 6.7% |
| | EXCLUSIVE split of the same 897 papers: |
| | existing-dataset ONLY (no primary collection) 208 23.2% |
| | primary collection ONLY (no existing dataset) 453 50.5% |
| | both 175 19.5% |
| | any primary collection 628 70.0% |
| |
| ## Did they use the platform's official API, or scrape? (tools[].name over the 897 platform-subject papers) | ## Did they use the platform's official API, or scrape? (tools[].name over the 897 platform-subject papers) |
| [CCS 2012] malware infection | [CCS 2012] malware infection |
| [IMC 2012] Facebook gifting application activity | [IMC 2012] Facebook gifting application activity |
| | |
| | ## Cross-page reconciliation: how many papers name "Alexa" in a stated population source? |
| | papers with >=1 stated population sourceList 5492 |
| | ... of which at least one tuple has a web unit 1143 |
| | name "Alexa" in any stated sourceList (no role filter) 486 |
| | ... and have a web-unit population tuple 464 <- comparable to design:website_selection's 463 |
| | assigned role=ranking by plat_fold.mjs (what this page publishes) 412 |
| | The gap is the role rule: Alexa-as-skill-store strings are diverted to |
| | role=subject, and a string naming several lists is tagged per platform. |
| | Neither number is wrong; they answer different questions. |
| </file> | </file> |
| |
| const noMode = anySubject.filter((r) => (r.temporal || []).every((t) => !t.mode || isSentinel(t.mode))).length; | const noMode = anySubject.filter((r) => (r.temporal || []).every((t) => !t.mode || isSentinel(t.mode))).length; |
| console.log(` (no stated mode) ${String(noMode).padStart(4)} ${pct(noMode, anySubject.length)}`); | console.log(` (no stated mode) ${String(noMode).padStart(4)} ${pct(noMode, anySubject.length)}`); |
| | // temporal.mode is MULTI-VALUED, so "383 existing-dataset" is not "383 papers |
| | // that collected nothing": 175 of them also name a primary-collection mode. |
| | // Publishing the multi-valued count as an exclusive one overstates dataset |
| | // reuse and understates primary collection. Exclusive split: |
| | const PRIMARY = new Set(['live-crawl', 'active-probing', 'passive-collection']); |
| | let bothModes = 0; |
| | let onlyExisting = 0; |
| | let onlyPrimary = 0; |
| | for (const r of anySubject) { |
| | const m = new Set((r.temporal || []).map((t) => t.mode).filter((x) => x && !isSentinel(x))); |
| | const ex = m.has('existing-dataset'); |
| | const pr = [...m].some((x) => PRIMARY.has(x)); |
| | if (ex && pr) bothModes += 1; |
| | else if (ex) onlyExisting += 1; |
| | else if (pr) onlyPrimary += 1; |
| | } |
| | console.log(` EXCLUSIVE split of the same ${anySubject.length} papers:`); |
| | console.log(` existing-dataset ONLY (no primary collection) ${String(onlyExisting).padStart(4)} ${pct(onlyExisting, anySubject.length)}`); |
| | console.log(` primary collection ONLY (no existing dataset) ${String(onlyPrimary).padStart(4)} ${pct(onlyPrimary, anySubject.length)}`); |
| | console.log(` both ${String(bothModes).padStart(4)} ${pct(bothModes, anySubject.length)}`); |
| | console.log(` any primary collection ${String(onlyPrimary + bothModes).padStart(4)} ${pct(onlyPrimary + bothModes, anySubject.length)}`); |
| |
| // -- 6. official API vs scraping, from tool names | // -- 6. official API vs scraping, from tool names |
| console.log(' residue sample (first 15 phenomena):'); | console.log(' residue sample (first 15 phenomena):'); |
| for (const r of unmatched.slice(0, 15)) console.log(` [${r.venue} ${r.year}] ${(r.detection[0] || {}).phenomenon}`); | for (const r of unmatched.slice(0, 15)) console.log(` [${r.venue} ${r.year}] ${(r.detection[0] || {}).phenomenon}`); |
| | |
| | // -- 16. cross-page reconciliation. |
| | // design:website_selection reports 463 papers for Alexa, folded over the 1,153 |
| | // papers that sampled the WEB. This page reports 412 for role=ranking over all |
| | // papers that stated a population. A subset cannot exceed its superset, so the |
| | // two folds must be measuring different things — and they are. Printed here so |
| | // neither page looks broken to a reader holding both. |
| | console.log(`\n## Cross-page reconciliation: how many papers name "Alexa" in a stated population source?`); |
| | { |
| | const stated = rows.filter((r) => (r.population || []).some((p) => p.sourceList && !isSentinel(p.sourceList))); |
| | const WEB_UNITS = new Set(['websites', 'domains', 'web-pages', 'urls']); |
| | const isWeb = (r) => (r.population || []).some((p) => p.sourceList && !isSentinel(p.sourceList) && WEB_UNITS.has(p.unit)); |
| | const anyAlexa = stated.filter((r) => r.population.some((p) => p.sourceList && /alexa/i.test(p.sourceList))); |
| | console.log(` papers with >=1 stated population sourceList ${stated.length}`); |
| | console.log(` ... of which at least one tuple has a web unit ${stated.filter(isWeb).length}`); |
| | console.log(` name "Alexa" in any stated sourceList (no role filter) ${anyAlexa.length}`); |
| | console.log(` ... and have a web-unit population tuple ${anyAlexa.filter(isWeb).length} <- comparable to design:website_selection's 463`); |
| | const rankingRole = [...perPaperRoles.entries()].filter(([, f]) => f.has('Amazon') && f.get('Amazon').has('ranking')).length; |
| | console.log(` assigned role=ranking by plat_fold.mjs (what this page publishes) ${rankingRole}`); |
| | console.log(' The gap is the role rule: Alexa-as-skill-store strings are diverted to'); |
| | console.log(' role=subject, and a string naming several lists is tagged per platform.'); |
| | console.log(' Neither number is wrong; they answer different questions.'); |
| | } |
| </file> | </file> |
| |
| ['2024/CCS/modern-problems-require-modern-solutions-community-developed-techniques-for-onli', '137 videos and 4,297 comments on TikTok'], | ['2024/CCS/modern-problems-require-modern-solutions-community-developed-techniques-for-onli', '137 videos and 4,297 comments on TikTok'], |
| ['2025/USENIX/darkgram-a-large-scale-analysis-of-cybercriminal-activity-channels-on-telegram', 'now replaced by the Meta Content Library'], | ['2025/USENIX/darkgram-a-large-scale-analysis-of-cybercriminal-activity-channels-on-telegram', 'now replaced by the Meta Content Library'], |
| | // Added 2026-08-27 after review found the '1% streaming API' quote attributed |
| | // to the wrong NDSS 2021 paper. hagen2021_numbers contains no occurrence of |
| | // "Twitter" at all; this is the paper the quote belongs to. |
| | ['2021/NDSS/to-err-is-human-characterizing-the-threat-of-unintended-urls-in-social-media', '1% streaming API that Twitter provides to vetted researchers'], |
| | ['2021/NDSS/to-err-is-human-characterizing-the-threat-of-unintended-urls-in-social-media', 'all the numbers that we presented in this paper are lower bounds'], |
| | // Added after review found the cross-platform blocking range omitted TikTok, |
| | // which is at the TOP of it, not absent from it. |
| | ['2025/IMC/exploration-of-the-dynamics-of-buy-and-sale-of-social-media-accounts', 'TikTok and Instagram demonstrated the highest'], |
| | ['2025/IMC/exploration-of-the-dynamics-of-buy-and-sale-of-social-media-accounts', 'TikTok 1,700 816'], |
| | // Added 2026-08-27 after the generic reviewer pointed out that the header |
| | // sentence claimed to cover "every figure the page attributes to a paper" |
| | // while many attributed figures were in no check at all. These close the gap. |
| | ['2026/PETS/a-year-under-the-dsa-ad-transparencys-uneven-landscape', '98.9% of all explanation texts cite only the main targeting form'], |
| | ['2026/NDSS/hey-there-you-are-using-whatsapp-enumerating-three-billion-accounts-for-security-and-privacy', 'roughly two-thirds of the images (66 %) contain detectable human faces'], |
| | ['2026/NDSS/hey-there-you-are-using-whatsapp-enumerating-three-billion-accounts-for-security-and-privacy', '245 countries'], |
| | ['2019/WWW/auditing-offline-data-brokers-via-facebooks-advertising-platform', 'Australia (81.3%), and for the U.K. (74.4%)'], |
| | ['2026/PETS/banned-books-analysis-of-censorship-on-amazon-com', '8,965 out of the 796,081'], |
| | // Column splice: the running sentence is interleaved. The paper's own |
| | // contiguous statement of the same fact is in its contributions list. |
| | ['2023/IMC/flocking-to-mastodon-tracking-the-great-twitter-migration', 'The top 25% most populous instances contain 96% of the users'], |
| | ['2023/IMC/flocking-to-mastodon-tracking-the-great-twitter-migration', '2,879 unique Mastodon instances'], |
| | ['2025/IMC/exploration-of-the-dynamics-of-buy-and-sale-of-social-media-accounts', 'Facebook 649 37 5.70'], |
| | ['2026/WWW/does-this-button-work-investigating-youtubes-ineffective-user-controls', '22,722'], |
| | ['2024/IMC/beyond-the-guidelines-assessing-metas-political-ad-moderation-in-the-eu', '29.5 million'], |
| | // "63.2 bn" on the page is a rounding of this table value; check the value. |
| | ['2026/NDSS/hey-there-you-are-using-whatsapp-enumerating-three-billion-accounts-for-security-and-privacy', '63,170,000,000'], |
| | ['2024/WWW/tiktok-and-the-art-of-personalization-investigating-exploration-and-exploitation', '4.9M'], |
| ]; | ]; |
| |
| |
| The family sets are **regex candidate sets**, so they were audited against the papers' own full text with ''scripts/platforms_audit.mjs'', which prints whitespace-collapsed windows around each platform-name match. Samples are deterministic (every //k//-th row of the ''%%--list%%'' output, which is sorted by year then venue), so the audit is reproducible. | The family sets are **regex candidate sets**, so they were audited against the papers' own full text with ''scripts/platforms_audit.mjs'', which prints whitespace-collapsed windows around each platform-name match. Samples are deterministic (every //k//-th row of the ''%%--list%%'' output, which is sorted by year then venue), so the audit is reproducible. |
| | |
| | **These samples are small** — //n// = 7, 15, 16 and 11. At //n// = 15 the 95% interval around 73% is roughly ±20 points. The precisions below are coarse corrections on a candidate set, not measurements, and the content page now says so too. |
| |
| ==== TikTok — full audit, all 7 candidates ==== | ==== TikTok — full audit, all 7 candidates ==== |
| | **genuine** | IEEE S&P 2024 //A Picture is Worth 500 Labels// {[west2024_picture]} | extracted and evaluated the on-device ML models shipped in the TikTok app | | | **genuine** | IEEE S&P 2024 //A Picture is Worth 500 Labels// {[west2024_picture]} | extracted and evaluated the on-device ML models shipped in the TikTok app | |
| | **genuine** | TheWebConf 2024 //TikTok and the Art of Personalization// {[vombatkere2024_tiktok]} | five sock-puppet accounts driving the feed, plus ''TikTok-Api'' | | | **genuine** | TheWebConf 2024 //TikTok and the Art of Personalization// {[vombatkere2024_tiktok]} | five sock-puppet accounts driving the feed, plus ''TikTok-Api'' | |
| | **genuine** | CCS 2024 //Modern problems require modern solutions// {[simko2024_modern]} | collected 120 TikTok videos and 4,297 comments; verified in full text | | | **genuine** | CCS 2024 //Modern problems require modern solutions// {[simko2024_modern]} | collected social-media video and comment data from TikTok and YouTube. The paper's own wording is ''%%"the qualitative analysis of 137 videos and 4,297 comments on TikTok (n = 120) and YouTube (n = 17)"%%'' — so **137 videos in total, of which 120 are TikTok**, and the 4,297 comments are across both platforms. An earlier version of this row said "120 TikTok videos and 4,297 comments", which silently reassigned the combined comment count to TikTok alone; corrected after review. | |
| | **genuine** | IEEE S&P 2026 //Setting the Course, but Forgetting to Steer// {[karnam2026_setting]} | GDPR right-of-access requests to TikTok from sock-puppet accounts | | | **genuine** | IEEE S&P 2026 //Setting the Course, but Forgetting to Steer// {[karnam2026_setting]} | GDPR right-of-access requests to TikTok from sock-puppet accounts | |
| | no — topic only | PETS 2023 //Creative beyond TikToks// | interview and diary study of adolescents; TikTok is the subject matter, not the measured system | | | no — topic only | PETS 2023 //Creative beyond TikToks// | interview and diary study of adolescents; TikTok is the subject matter, not the measured system | |
| **Precision 11/15 = 73%.** All four false positives are 2022 or later, which is why the content page warns that the recent Twitter/X counts are inflated relative to the earlier ones, and states the trend as a change in **composition** rather than in size. | **Precision 11/15 = 73%.** All four false positives are 2022 or later, which is why the content page warns that the recent Twitter/X counts are inflated relative to the earlier ones, and states the trend as a change in **composition** rather than in size. |
| |
| ==== Meta — every 10th of 151 = 15 audited ==== | ==== Meta — every 10th of 151 = 16 audited ==== |
| |
| **Genuine (10):** CCS 2010 //Detecting and characterizing social spam campaigns// (3.5 M users' wall messages) · USENIX 2012 //MyPageKeeper// · CCS 2017 //walk2friends// (Instagram check-ins) · TheWebConf 2018 //Tagvisor// (239 k Instagram posts) · TheWebConf 2019 //Auditing Offline Data Brokers via Facebook's Advertising Platform// {[venkatadri2019_auditing]} · USENIX 2020 //DELF// {[cohn2020_delf]} (Facebook-authored, own systems) · NDSS 2021 //All the Numbers are US// {[hagen2021_numbers]} (WhatsApp crawling) · TheWebConf 2023 //The Thin Ideology of Populist Advertising// {[capozzi2023_thin]} (Meta Ad Library, 45 k campaigns) · USENIX 2024 //The Imitation Game// {[acharya2024_imitation]} · IMC 2025 //Buy and Sale of Social Media Accounts// {[beluri2025_exploration]}. | **Genuine (11):** CCS 2010 //Detecting and characterizing social spam campaigns// (3.5 M users' wall messages) · USENIX 2012 //MyPageKeeper// · CCS 2017 //walk2friends// (Instagram check-ins) · TheWebConf 2018 //Tagvisor// (239 k Instagram posts) · TheWebConf 2019 //Auditing Offline Data Brokers via Facebook's Advertising Platform// {[venkatadri2019_auditing]} · USENIX 2020 //DELF// {[cohn2020_delf]} (Facebook-authored, own systems) · NDSS 2021 //All the Numbers are US// {[hagen2021_numbers]} (WhatsApp crawling) · TheWebConf 2023 //The Thin Ideology of Populist Advertising// {[capozzi2023_thin]} (Meta Ad Library, 45 k campaigns) · USENIX 2024 //The Imitation Game// {[acharya2024_imitation]} · IMC 2025 //Buy and Sale of Social Media Accounts// {[beluri2025_exploration]} · TheWebConf 2026 //Longitudinal Trends in Global Climate Change Discourse on Facebook// {[biswas2026_longitudinal]} (299,329 Facebook posts via CrowdTangle). |
| |
| **False positives (5):** TheWebConf 2013 //Google+ or Google-?// (measures Google+; Facebook only in the framing) · CCS 2015 //Perplexed Messengers from the Cloud// (the Facebook client app is one of many analysed; the study is push clouds) · USENIX 2022 //Pre-hijacked accounts// (Facebook as an identity-provider example) · PETS 2023 //Creative beyond TikToks// (Facebook in related work) · TheWebConf 2025 //Cross-Modal Transfer from Memes to Videos// (Facebook Hateful Memes benchmark). | **False positives (5):** TheWebConf 2013 //Google+ or Google-?// (measures Google+; Facebook only in the framing) · CCS 2015 //Perplexed Messengers from the Cloud// (the Facebook client app is one of many analysed; the study is push clouds) · USENIX 2022 //Pre-hijacked accounts// (Facebook as an identity-provider example) · PETS 2023 //Creative beyond TikToks// (Facebook in related work) · TheWebConf 2025 //Cross-Modal Transfer from Memes to Videos// (Facebook Hateful Memes benchmark). |
| |
| **Precision 10/15 = 67%.** | **Precision 11/16 = 69%.** |
| | |
| | <WRAP important> |
| | **This row was wrong on first publication and was corrected by review.** It read "every 10th of 151 = **15** audited ... precision **10/15 = 67%**". A stride of 10 over 151 rows yields **16** samples (indices 0, 10, … 150), not 15, and the sixteenth — TheWebConf 2026 //Longitudinal Trends in Global Climate Change Discourse on Facebook// — was silently dropped. It is unambiguously genuine: the content page cites it two sections earlier for having collected 299,329 Facebook posts through CrowdTangle. So the audited sample is 16 and the precision is 11/16. TikTok (stride 1 over 7), Twitter/X (stride 10 over 142 → 15) and Amazon (stride 7 over 72 → 11) were re-checked and are correct. Found by the figures reviewer; the author had counted the sample by eye from a truncated terminal listing rather than from ''%%wc -l%%''. |
| | </WRAP> |
| |
| ==== Amazon — every 7th of 72 = 11 audited ==== | ==== Amazon — every 7th of 72 = 11 audited ==== |
| ===== Quotes spot-checked against the source ===== | ===== Quotes spot-checked against the source ===== |
| |
| Every figure the content page attributes to a paper is listed in ''scripts/platforms_quotecheck.mjs'' with the phrase that must appear in that paper's ''paper.cols.txt''. Whitespace is collapsed on both sides first. | Every figure and every quoted phrase the content page attributes to a paper is listed in ''scripts/platforms_quotecheck.mjs'' with the text that must appear in that paper's ''paper.cols.txt''. Whitespace is collapsed on both sides first. |
| | |
| | <WRAP important> |
| | **That sentence was not true when this page was first published, and the generic reviewer caught it.** The list then held 26 checks and covered the //numbered// figures but not every attributed figure: the 98.9% YouTube explanation share, the 66% face-detection rate, 81.3%/74.4%, 1.1% of 796,081, the Mastodon 96% and 2,879, the per-platform blocking rates, 22,722, 29.5 M ads and 63,170,000,000 were all attributed on the page and checked by nobody. It now holds **42** checks and the sentence is true. The lesson generalises past this page: a quote-check list built from "the figures I published" misses everything a reader would also want to trust, and the number guard cannot see quoted phrases at all. |
| | </WRAP> |
| |
| <file text platforms_quotecheck-output.txt> | <file text platforms_quotecheck-output.txt> |
| PASS CCS 2024 "137 videos and 4,297 comments on TikTok" | PASS CCS 2024 "137 videos and 4,297 comments on TikTok" |
| PASS USENIX 2025 "now replaced by the Meta Content Library" | PASS USENIX 2025 "now replaced by the Meta Content Library" |
| | PASS NDSS 2021 "1% streaming API that Twitter provides to vetted researchers" |
| | PASS NDSS 2021 "all the numbers that we presented in this paper are lower bounds" |
| | PASS IMC 2025 "TikTok and Instagram demonstrated the highest" |
| | PASS IMC 2025 "TikTok 1,700 816" |
| | PASS PETS 2026 "98.9% of all explanation texts cite only the main targeting form" |
| | PASS NDSS 2026 "roughly two-thirds of the images (66 %) contain detectable human faces" |
| | PASS NDSS 2026 "245 countries" |
| | PASS WWW 2019 "Australia (81.3%), and for the U.K. (74.4%)" |
| | PASS PETS 2026 "8,965 out of the 796,081" |
| | PASS IMC 2023 "The top 25% most populous instances contain 96% of the users" |
| | PASS IMC 2023 "2,879 unique Mastodon instances" |
| | PASS IMC 2025 "Facebook 649 37 5.70" |
| | PASS WWW 2026 "22,722" |
| | PASS IMC 2024 "29.5 million" |
| | PASS NDSS 2026 "63,170,000,000" |
| | PASS WWW 2024 "4.9M" |
| |
| 26 pass, 0 fail, of 26 checks | 42 pass, 0 fail, of 42 checks |
| </file> | </file> |
| |
| After those replacements: **26 checks, 26 pass, 0 fail.** | After those replacements: **26 checks, 26 pass, 0 fail.** |
| |
| One further note on the corpus text: some IMC volumes in ''paper.cols.txt'' include the **published reviewer comments** ("Reviewer #2 Strengths: ..."). A keyword probe over full text can therefore hit a reviewer's sentence rather than the authors'. Two of the early rate-limit examples in the probe output are exactly that. It does not change the paper counts materially at this scale, but a probe designed to measure what //authors// say would need to strip those sections. | One further note on the corpus text: some IMC volumes in ''paper.cols.txt'' include the **published reviewer comments** ("Reviewer #2 Strengths: ..."). A keyword probe over full text can therefore hit a reviewer's sentence rather than the authors'. Two of the early rate-limit examples in the probe output are exactly that. Quantified rather than waved away: **36 of the 5,869** ''paper.cols.txt'' files contain the string ''%%Reviewer #%%'', and **9 of the 897** platform-subject papers do. So the contamination is real and bounded at about 1% of the population every probe on this page ran over. A probe designed to measure what //authors// say would still need to strip those sections. |
| |
| ===== External sources ===== | ===== External sources ===== |
| | Current VLOP designation list, including WhatsApp | ''https://digital-strategy.ec.europa.eu/en/policies/list-designated-vlops-and-vloses'' | list includes ''%%"WhatsApp Ireland Ltd."%%'', ''%%"X Internet Unlimited Company(XIUC)"%%'', ''%%"Meta Platforms Ireland Limited (MPIL)"%%'', ''%%"TikTok Technology Limited"%%'', ''%%"Amazon EU S.à.r.l."%%'' | ''%%"Information updated on 24 July 2026"%%'' | | | Current VLOP designation list, including WhatsApp | ''https://digital-strategy.ec.europa.eu/en/policies/list-designated-vlops-and-vloses'' | list includes ''%%"WhatsApp Ireland Ltd."%%'', ''%%"X Internet Unlimited Company(XIUC)"%%'', ''%%"Meta Platforms Ireland Limited (MPIL)"%%'', ''%%"TikTok Technology Limited"%%'', ''%%"Amazon EU S.à.r.l."%%'' | ''%%"Information updated on 24 July 2026"%%'' | |
| | ''help.crowdtangle.com'' is gone | DNS | ''net::ERR_NAME_NOT_RESOLVED'' | 2026-08-27 | | | ''help.crowdtangle.com'' is gone | DNS | ''net::ERR_NAME_NOT_RESOLVED'' | 2026-08-27 | |
| | | X pay-per-use launched 6 Feb 2026; Basic and Pro remain for existing subscribers; Owned Reads added 20 Apr 2026 at %%$0.001%% | ''%%https://docs.x.com/changelog%%'' | ''%%"Today, we officially launched X API Pay-Per-Use pricing"%%''; ''%%"Basic and Pro plans remain available, and existing subscribers can opt in to Pay-Per-Use."%%''; ''%%"Effective April 20, 2026, we introduced new Owned Reads pricing at $0.001 per resource"%%'' | entries dated ''%%Feb 6, 2026%%'' and ''%%Apr 16, 2026%%'' | |
| | | Reg. (EU) 2025/2050 was published in the OJ on 9 October 2025, so it has been in force since 29 October 2025 | the same EUR-Lex record | ''%%"OJ L, 2025/2050, 9.10.2025"%%'' | n/a | |
| | | The DSA Data Access Portal accepts applications from 29 October 2025 and publishes a register of vetted-researcher projects | ''%%https://data-access.dsa.ec.europa.eu/home%%'' (headless browser) | ''%%"You can send applications as of 29 October 2025."%%''; ''%%"Find out more about ongoing research projects conducted by vetted researchers who have access to data under Article 40 of the Digital Services Act."%%'' | none shown | |
| | | Pushshift live access is moderator-only, by Reddit approval, for moderation use only | ''%%https://support.reddithelp.com/hc/en-us/articles/16470271632404-Pushshift-Access-Request%%'' (headless browser; Cloudflare-walled to ''%%curl%%'') | ''%%"which will be reinstated for verified Reddit moderators"%%''; ''%%"Each moderator will also need explicit approval from Reddit, and the use of Pushshift will be limited to moderation use cases only."%%'' | ''%%"Updated 1 year ago"%%'' | |
| | | TikTok's non-academic EU pathway is a beta | ''%%https://developers.tiktok.com/products/research-api/%%'' | ''%%"We are currently beta testing this service with select researchers in the US, UK, Switzerland, Norway, Iceland and Liechtenstein."%%'' — the sentence sits immediately after the EU not-for-profit clause | none shown | |
| |
| Two of these pages are JavaScript-only and were fetched with Playwright's own Chromium rather than ''curl'' (''transparency.meta.com'', ''developers.facebook.com''; ''curl'' returns an error page for the second). The DSA data-access portal at ''https://data-access.dsa.ec.europa.eu/'' renders nothing without JavaScript — the page title ''%%"DSA - Data Access Portal"%%'' is all that could be confirmed, and the content page says only that the portal exists. | Two of these pages are JavaScript-only and were fetched with Playwright's own Chromium rather than ''curl'' (''transparency.meta.com'', ''developers.facebook.com''; ''curl'' returns an error page for the second). The DSA data-access portal at ''https://data-access.dsa.ec.europa.eu/'' renders nothing without JavaScript — the page title ''%%"DSA - Data Access Portal"%%'' is all that could be confirmed, and the content page says only that the portal exists. |
| | ICPSR/SOMAR compute fees for Meta Content Library access from January 2026 | no primary source found | | | ICPSR/SOMAR compute fees for Meta Content Library access from January 2026 | no primary source found | |
| | Google Ads Transparency Center researcher API | no primary Google page describes one; the page makes no claim | | | Google Ads Transparency Center researcher API | no primary Google page describes one; the page makes no claim | |
| | | **X has appealed the €120 M DSA fine** (reported as filed at the EU General Court on 16 February 2026) | reported by the currency reviewer from secondary press; no case number could be confirmed from a primary Curia document. **Not published** — the page presents the decision as the Commission published it and does not claim finality either way. This is the single most likely thing on the page to go out of date first | |
| | | **DSA Art. 40(12) has been enforced directly by researchers in German courts** — Democracy Reporting International v X, reported as LG Berlin II, 6 February 2025 (41 O 140/25), and a Kammergericht order of 17 February 2026 granting API access to 30 June 2026 and holding that Art. 40(12) confers an individually enforceable right | reported by the currency reviewer; we tried to verify it. ''%%democracy-reporting.org%%'' is Cloudflare-walled even to a headless browser, the Columbia Global Freedom of Expression search did not surface the case, and the cited German legal blog's February 2026 archive does not contain it. **Not published.** If true it is the most important thing missing from the page, because it would mean the "no worked example of Art. 40" claim holds only for //published papers//, not for practice. What would close it: the Kammergericht order itself, or DRI's case page fetched from a network Cloudflare does not challenge | |
| | Amazon Conditions of Use anti-scraping clause | ''amazon.com/gp/help/...'' returned HTTP 503 twice | | | Amazon Conditions of Use anti-scraping clause | ''amazon.com/gp/help/...'' returned HTTP 503 twice | |
| |
| * **Why platform-subject papers state authentication state //less// often** (65.8% vs 72.1% of all papers with a crawl configuration) even though the login wall matters more to them. The difference is 6.3 points on 260 papers; it could be noise. Not published as a finding, only as the two figures. | * **Why platform-subject papers state authentication state //less// often** (65.8% vs 72.1% of all papers with a crawl configuration) even though the login wall matters more to them. The difference is 6.3 points on 260 papers; it could be noise. Not published as a finding, only as the two figures. |
| * **Whether the Twitter/X decline is caused by the API closing.** The share was already down to 1.9% in 2016–2018, years before the restriction. The page says so explicitly rather than drawing the tempting line. | * **Whether the Twitter/X decline is caused by the API closing.** The share was already down to 1.9% in 2016–2018, years before the restriction. The page says so explicitly rather than drawing the tempting line. |
| | * **Whether Art. 40(12) has already been enforced through national courts.** A reviewer reported a German line of cases (Democracy Reporting International v X) that would mean the route has been exercised in practice even though no paper here uses it. We could not verify it from a primary document — see the unverified table above. This is the biggest known hole in the page. |
| * **The real cost of a metered-API study.** No paper in the corpus publishes its API spend. The arithmetic on the page (1 M posts = %%$5,000%%) is ours, from the vendor's published unit price, and is labelled as such. | * **The real cost of a metered-API study.** No paper in the corpus publishes its API spend. The arithmetic on the page (1 M posts = %%$5,000%%) is ours, from the vendor's published unit price, and is labelled as such. |
| * **Whether reused corpora still describe the platform.** 42.7% of platform papers use an existing dataset; nobody in the corpus tests whether conclusions from a 2019 Twitter corpus hold for X in 2026. Listed as an open question. | * **Whether reused corpora still describe the platform.** 383 of 897 platform papers name an existing dataset, and 208 name one with no primary collection of their own; nobody in the corpus tests whether conclusions from a 2019 Twitter corpus hold for X in 2026. Listed as an open question. |
| |
| ===== Mistakes and near-misses in this run ===== | ===== Mistakes and near-misses in this run ===== |
| - **The bibliography cache served a stale parse.** After appending 24 entries, the first render resolved only some citekeys. Fixed with ''?purge=true'' on ''literature:bibliography'' and then on ''design:platforms''; verified by counting %%<dt>%% entries in the rendered reference list (26) against distinct %%{[key]}%% markers on the page (26). | - **The bibliography cache served a stale parse.** After appending 24 entries, the first render resolved only some citekeys. Fixed with ''?purge=true'' on ''literature:bibliography'' and then on ''design:platforms''; verified by counting %%<dt>%% entries in the rendered reference list (26) against distinct %%{[key]}%% markers on the page (26). |
| - **LaTeX accent escapes render as HTML entities.** ''Beno\^{i}t'' came out as ''Benoˆit'' in the reference list. The five new entries with diacritics were rewritten in UTF-8. Pre-existing entries elsewhere in the bibliography have the same problem and were left alone. | - **LaTeX accent escapes render as HTML entities.** ''Beno\^{i}t'' came out as ''Benoˆit'' in the reference list. The five new entries with diacritics were rewritten in UTF-8. Pre-existing entries elsewhere in the bibliography have the same problem and were left alone. |
| | - **A multi-valued field was published as an exclusive one.** The first draft read ''%%383 (42.7%) worked from an existing dataset rather than collecting anything themselves — a higher share than live collection (346, 38.6%)%%''. Both halves were wrong: ''%%temporal.mode%%'' is multi-valued, so 175 of the 383 //also// name a primary-collection mode, only **208** name an existing dataset and nothing else, and **628 (70.0%)** name some primary collection. Caught by the author before review, by asking what the exclusive split was; ''%%platforms_report.mjs%%'' now prints it so the next reader cannot make the same mistake. |
| | - **An audit sample was miscounted by one, in the direction that lowered the precision.** A stride of 10 over Meta's 151 candidates gives 16 samples, not 15; the sixteenth was dropped and is genuine, so the published precision was 67% instead of 69%. The author counted the sample from a truncated terminal listing instead of ''%%wc -l%%''. Found by review, not by the number guard — which passed, because "67" happens to appear in an unrelated per-year denominator elsewhere in the report. Explicit ALLOW entries for all four audit precisions have since been added to ''%%scripts/check_page_numbers.mjs%%'' so the next change to them is actually guarded rather than passing by coincidence. |
| | - **A combined figure was reassigned to one platform.** This log said CCS 2024 "collected 120 TikTok videos and 4,297 comments"; the paper says 137 videos in total, of which 120 are TikTok, and the 4,297 comments span TikTok //and// YouTube. Found by review. The quote-check script had the right phrase all along and passed — the error was in the prose beside it, which is exactly the gap the number guard cannot see. |
| | - **A vendor fact was stale within the same page.** The first version said "the Basic tier the paper worked around no longer exists". X's own changelog says the opposite: pay-per-use launched 6 February 2026 and "Basic and Pro plans remain available" to existing subscribers. Found by the currency reviewer. The lesson is the page's own: an access-route claim needs a dated primary source even when it is only an aside. |
| | - **A quote was attributed to the wrong paper.** The //Denominator problem// bullet on sampled streams cited ''%%hagen2021_numbers%%'' (NDSS 2021, //All the Numbers are US//) for ''%%"the 1% streaming API that Twitter provides to vetted researchers"%%''. That paper is about WhatsApp and Signal phone-number enumeration and contains **no occurrence of the word "Twitter" at all**. The quote belongs to a **different NDSS 2021 paper**, //To Err.Is Human: Characterizing the Threat of Unintended URLs in Social Media//, which the author had read in the same probe output and confused with it. Found by the citations reviewer. Fixed by adding ''%%kaleli2021_human%%'' to the bibliography and re-citing; two quote checks for it were added to ''%%platforms_quotecheck.mjs%%''. This is the worst class of error on the page: a quote check that only verifies "does this phrase exist somewhere in the corpus" would have passed it, and the one here passed because the phrase was never in the check list at all. **Every quote on a page belongs in the check list, not just the numbered figures.** |
| | - **A cross-platform range omitted the platform at the top of it.** The page reported blocking efficacy "//from 5.02% (YouTube) to 46.41% (Instagram)//" across {[beluri2025_exploration]}'s five platforms — leaving out TikTok, which is at or above Instagram. The paper's own summary is ''%%"TikTok and Instagram demonstrated the highest detection efficacy at 48%, whereas YouTube and Facebook showed the lowest efficacy at just 5%"%%''; its Table 8 gives TikTok 816 of 1,700, and the efficacy cell for that row is lost to a column splice, which is how it came to be dropped. Found by the citations reviewer. Fixed, with the paper's prose figure quoted rather than a recomputed one. |
| | - **A true finding was framed so it invited a false inference.** The headline box's "official researcher routes are almost entirely absent from this literature" is correct as a count — the generic reviewer re-ran it with wider probes and the absolutes held — but a fresh reader would have concluded that the programmes do not work, when part of the absence is mechanical: 2023-and-later launches, a provisional 2025–2026 slice, and the venues that publish most such work (ICWSM, CHI, FAccT, communications journals) excluded by construction. That caveat existed on the page, 280 lines below the claim. Moved into the box. |
| | - **Three pages counted the same names and got different numbers, and none of them said so.** [[design:website_selection]] reports 463 papers for Alexa; this page reported 412; [[design:user_studies]] publishes 139 / 92 / 279 for Mechanical Turk against this page's 179. A subset appearing to exceed its superset is a signal that two folds disagree, not that one is broken. Reconciled and now printed by the report script: **486** papers name Alexa in a stated population source at all, **464** of those have a web-unit population tuple — which is what [[design:website_selection]] measures — and 412 is what survives our role rule. The content page carries the reconciliation and does not claim its fold is the better one. |
| | - **A schema label was published as an audited one.** The ''%%crawlConfig.authentication%%'' split (145 / 22 / 4 / 0) was stated as what papers "state", when [[programming:registration]] warns on this exact field that ''%%crawlConfig%%'' carries one evidence quote for the whole object so the label cannot be checked against it, and [[privacy:consent]]'s complete audit found **279 of 313 ''no-interaction'' labels unsupported (89.1%)**, alongside **7 of 36 (19.4%)** false positives among labels claiming interaction. We did not audit the 145; the page now says so. This is the finding most likely to move a number if someone does the audit. |
| - **No accidental exposure.** No credentials, participant data or unpublished material was written to the wiki. The only non-public thing touched was ''.env'' for the JSON-RPC credentials, read by ''scripts/dw.mjs''. | - **No accidental exposure.** No credentials, participant data or unpublished material was written to the wiki. The only non-public thing touched was ''.env'' for the JSON-RPC credentials, read by ''scripts/dw.mjs''. |
| | |
| | ===== Observations for whoever maintains the shared bibliography ===== |
| | |
| | Found incidentally while checking this page's citations. **None of these keys are used by this page**; they are recorded because a key-string collision check does not find them. |
| | |
| | ^ Duplicate pair ^ Note ^ |
| | | ''%%lerner2016internet%%'' / ''%%lerner2016_internet%%'' | same paper under two keys | |
| | | ''%%bouhoula2024automated%%'' / ''%%bouhoula2024_automated%%'' | same | |
| | | ''%%fouad2022my%%'' / ''%%fouad2022_cookie%%'' | same | |
| | | ''%%bottger2025_regional%%'' / ''%%boettger2025_regional%%'' | same, transliteration variant | |
| | | ''%%ahmad2026_ipfp%%'' / ''%%ahmad2026_more%%'' | same | |
| | |
| | Also: ''%%biswas2026_longitudinal%%'''s DOI ''%%10.1145/3774904.3793026%%'' did **not** resolve at doi.org or via the CrossRef API on 2026-08-27, although the title and author list check out against the ACM listing. Almost certainly registration lag on a TheWebConf 2026 paper rather than a wrong DOI, but it is unverified as of this run. |
| | |
| | And a corpus trap worth knowing: ''%%data/fulltext/2024/IMC/beyond-the-guidelines-.../paper.cols.txt%%'' contains a **null byte**, so plain ''%%grep%%'' treats the file as binary and reports no match for a phrase that is present. Use ''%%grep -a%%'', or read it from a language runtime. Every script on this page reads the file through Node, so none of the published figures were affected — but a hand-check with ''%%grep%%'' would have produced a false negative. Same family as the known corpus under-count trap. |
| |
| ===== Review log ===== | ===== Review log ===== |
| Four reviewers, each given the page text, the report scripts and their unedited output, and these notes, and each told explicitly that the author's context may not be exhaustive. | Four reviewers, each given the page text, the report scripts and their unedited output, and these notes, and each told explicitly that the author's context may not be exhaustive. |
| |
| //Reviews are running as this page is first saved; the log is filled in below in the same sitting.// | Three focused passes ran in parallel first; the generic pass ran afterwards against the corrected page. Rejections are recorded as fully as fixes — they are the only record of whether a reviewer earned its slot. |
| | |
| | ==== Reviewer 1 — figures against the scripts (Claude Sonnet) ==== |
| | |
| | ^ Finding ^ Verdict ^ Action ^ |
| | | Meta audit sample is 16, not 15 — a stride of 10 over 151 rows yields indices 0…150 — and the dropped 16th ({[biswas2026_longitudinal]}) is genuine, so precision is 11/16 = 69%, not 10/15 = 67% | **accepted**, reproduced with ''%%wc -l%%'' | fixed on both pages; the dropped paper added to the genuine list; TikTok, Twitter/X and Amazon strides re-checked and correct | |
| | | This log said CCS 2024 collected "120 TikTok videos and 4,297 comments"; the paper says 137 videos total (120 TikTok + 17 YouTube) with the comments spanning both | **accepted** | fixed, with the paper's own sentence quoted | |
| | | The number guard passes the three other audit precisions only **by coincidence** — "57" matches inside an unrelated WhatsApp quote, "67" inside a per-year denominator, "73" inside the artifact table — so a future change to them would pass silently | **accepted**, and the more useful of its findings | explicit ALLOW entries added for all four audit precisions, each naming its numerator and denominator | |
| | | All four committed ''%%*-output.txt%%'' files reproduce byte-for-byte, including a full re-run of the 7-minute full-text probe; ''%%check_page_numbers.mjs%%'' returns OK; every figure in //Measured results you can cite// verified independently against ''%%paper.cols.txt%%'' and ''%%population.n%%'' | confirmation, no action | — | |
| | |
| | ==== Reviewer 2 — citations and quotes (Claude Sonnet) ==== |
| | |
| | ^ Finding ^ Verdict ^ Action ^ |
| | | The ''%%"1% streaming API that Twitter provides to vetted researchers"%%'' quote was attributed to {[hagen2021_numbers]}, which contains **no occurrence of "Twitter"**; it belongs to a different NDSS 2021 paper | **accepted — the most serious finding of the review** | re-cited to {[kaleli2021_human]}, new bibliography entry, two quote checks added | |
| | | The cross-platform blocking range omitted TikTok, which is at the top of it, not absent from it | **accepted** | fixed in both places, quoting the paper's own 48% summary | |
| | | ''%%biswas2026_longitudinal%%'''s DOI does not resolve at doi.org or CrossRef | **accepted as an observation**, not a defect — title and authors check out, almost certainly registration lag | recorded above rather than changing the entry | |
| | | Five duplicate-key pairs elsewhere in the shared bibliography, none used by this page | **accepted as an observation** | recorded above for the bibliography's maintainer | |
| | | ''%%beyond-the-guidelines%%'''s ''%%paper.cols.txt%%'' contains a null byte, so plain ''%%grep%%'' silently reports no match | **accepted as an observation** | recorded above; no published figure affected, since every script reads through Node | |
| | | All 26 content-page and 17 provenance-page citekeys resolve; all 24 new entries' authors, titles, years and DOIs verified, including the eight fetched by hand from landing pages; all 13 footnoted URLs fetched and their quotes confirmed, including the JS-only Meta and DSA pages | confirmation, no action | — | |
| | |
| | ==== Reviewer 3 — external currency (Claude Sonnet) ==== |
| | |
| | ^ Finding ^ Verdict ^ Action ^ |
| | | The page said "the Basic tier the paper worked around no longer exists". X's changelog says pay-per-use launched 6 February 2026 and "Basic and Pro plans remain available" to existing subscribers | **accepted** | rewritten, with both changelog entries quoted and dated; the point that the model changed twice inside 2026 is now made explicitly | |
| | | Pushshift was labelled "current in practice". Reddit's own moderator page says access is reinstated only for approved moderators and "limited to moderation use cases only" | **accepted** | re-labelled historical-for-researchers in three places, with the primary quote; the page now says a paper citing Pushshift is citing a historical dump | |
| | | TikTok's non-academic EU pathway is a beta, not open | **accepted** | the beta sentence is now quoted rather than the eligibility asserted | |
| | | The OJ publication date (9 October 2025) and entry into force (29 October 2025) could be closed off, and the DSA portal states applications open from 29 October 2025 and publishes a register of vetted-researcher projects | **accepted**, re-verified by the author against EUR-Lex and the portal | both dates and the register are now on the page; the register is named as where the missing worked example will first appear | |
| | | X has appealed the €120 M fine (reported as filed 16 February 2026) | **not published** — no primary Curia document could be found | recorded in the unverified table | |
| | | DSA Art. 40(12) has been enforced directly by researchers in German courts (Democracy Reporting International v X) | **not published** — three independent verification attempts failed; see the unverified table | recorded, with what would close it. If true it is the largest hole in the page | |
| | | All 13 previously verified URLs still live with their quotes unchanged; YouTube, Meta and Amazon claims unchanged | confirmation, no action | — | |
| | |
| | ==== Reviewer 4 — generic, no checklist (Claude Fable) ==== |
| | |
| | The generic pass was run against the corrected page, after the three focused ones. It produced 14 findings; 11 were accepted, 3 partly. It is the only reviewer that found anything about **framing** rather than about facts, and it earned its slot. |
| | |
| | ^ Finding ^ Verdict ^ Action ^ |
| | | The headline "almost entirely absent from this literature" is numerically right (it re-ran the probes wider and the absolutes held) but invites the inference that the programmes do not work; the mechanical explanation sat 280 lines below | **accepted** | caveat moved into the box | |
| | | The ''%%crawlConfig.authentication%%'' split republishes a schema read that [[programming:registration]] explicitly warns against on this field, with no audit and no caveat | **accepted — the highest-value finding** | caveat carried inline, with the sibling field's measured 89.1% unsupported-label rate (and 19.4% interacting-label false-positive rate); the 145 remain unaudited and the page says so | |
| | | Cross-page count conflicts: Alexa 412 here vs 463 on [[design:website_selection]] (a subset exceeding its superset), and Mechanical Turk 179 here vs 139 / 92 / 279 on [[design:user_studies]] | **accepted** | reconciled in the report script and on the page: 486 name Alexa at all, 464 with a web-unit population, 412 after the role rule | |
| | | This page's quote-check header claimed to cover "every figure the content page attributes to a paper" while ~12 attributed figures were in no check at all, and the embedded script and output were stale | **accepted** | 16 checks added (42 total, all passing), both ''%%<file>%%'' blocks regenerated, and the overclaim recorded above rather than quietly fixed | |
| | | "The top two rows are largely app-store work" — row 2 is Meta, not app-store work | **accepted** | rewritten to rows 1 and 7 | |
| | | Two pointers send the reader to [[design:mobile_and_app_measurement]] for skill-store and on-device-extraction material that page does not contain | **accepted** | reworded to say what that page does and does not cover | |
| | | Google is rank 1 with 323 papers and gets no route, no read-first and no pointer; and the page is silent on what Reddit currently offers | **accepted** | both named as explicit gaps — search/ads auditing in Open Questions, Reddit as a "we could not establish this" bullet | |
| | | "statistically indistinguishable from the 33.8% baseline" — no test was run, and the platform papers are inside that baseline | **accepted** | reworded to "essentially at the baseline", with both caveats stated | |
| | | "roughly twice as likely" does not name which baseline it is twice of; against crawling papers (11.5%) the ratio is ~1.3× | **accepted** | both baselines now named | |
| | | "Three practical consequences:" followed by four bullets | **accepted** | trivial fix | |
| | | ''%%TikTok-Api%%'' is the unofficial scraper library, on a page whose axis is official-versus-unofficial routes | **accepted** | row relabelled | |
| | | "61.5% of platform-subject papers name no instrument" converts absence-from-extracted-tool-lists into a claim about what papers state | **accepted** | softened to an upper bound, with the un-audited 552 stated | |
| | | The early routes table and the closing currency table duplicate each other row-for-row | **partly accepted** | the early table's status column reduced to one word and the evidence left to the closing section; the two tables are kept, because one is a route inventory and the other a dated verdict, and a skimmer needs the first | |
| | | The TikTok no-page justification ("four papers is a paragraph") reads as the page marking its own homework | **accepted** | rewritten to rest on venue scope: the TikTok literature is mostly outside these seven venues, so a page built from this corpus would misrepresent the field | |
| | | Smaller rigour points: a duplicated sentence in this log; a negative claim ("no academic tier") with no stated search method; an unquantified reassurance about reviewer-comment contamination; audit precisions quoted as bare points at //n// = 7–16 | **all accepted** | sentence deleted; the tier claim scoped to the pages an applicant is sent to; contamination quantified (36 of 5,869 files, 9 of 897 platform papers); a small-//n// caveat added here and on the page | |
| | | Confirmations: the framing claim //is// delivered (organised by route and denominator, not by vendor); length and section order match the house pattern; the read-first quartet is right for the stated reader; the box's zero/once absolutes survive wider independent probes | no action | — | |
| | |
| | One thing the generic reviewer reported that is **not** a page defect: the local checkout's ''pages/start.txt'' and ''pages/design.txt'' are stale relative to the live wiki. The live pages were updated in this sitting; the stale files are old local copies and were never the source of a save. |
| | |
| |
| ===== Related ===== | ===== Related ===== |
| |
| <bibtex bibliography></bibtex> | <bibtex bibliography></bibtex> |
| | |
| | ===== ConsentAction audit propagation, 2026-09-17 ===== |
| | |
| | **What was wrong.** The page used the 19.4% interacting-side ''consentAction'' false-positive rate as its warning about the neighbouring authentication labels, even though the complete audit found a much larger no-interaction-side failure. |
| | |
| | **Command and real output.** ''node scripts/platforms_report.mjs'' → exit 0; its platform figures were unchanged. ''node scripts/consent_action_noninteraction_audit.mjs'' → **279/313 = 89.1%** of no-interaction labels unsupported; the interacting-side comparison remains **7/36 = 19.4%** false positives. |
| | |
| | **What changed.** [[design:platforms]] and this provenance page now cite the stronger 89.1% result alongside 19.4%. The **145-paper authentication split was not audited**; it is filed as a separate drain item. |
| | |
| | **Review.** One **model: 'sonnet'** figures-vs-script pass re-ran ''platforms_report.mjs'' and checked the authentication paragraph and the adjacent route/platform figures. No mismatch was found. No citation pass was needed: no citekey or quoted claim changed. |
| | |
| | **Rejected finding.** The 89.1% consentAction result was not used as evidence that authentication labels are wrong; it only motivates auditing the 145-paper authentication split next. |
| | |
| | ===== Audit of the 145 crawlConfig.authentication='none' platform papers, 2026-09-22 ===== |
| | |
| | **What was wrong.** [[design:platforms]] published the ''%%crawlConfig.authentication%%'' split for the 260 platform-subject papers with a crawl configuration — **145 (55.8%)** ''%%none%%'', 22 ''%%account-registration%%'', 4 ''%%manual-login%%'', 0 ''%%automated-login%%'' / ''%%sso%%'' — with the caveat that these were schema labels and that the 145 had not been audited, and filed the audit as the next item. This is that audit. The shape was right and the schema counts are unchanged; what was missing is that ''%%none%%'' is overwhelmingly an //inference from silence//, and that a measurable minority of the labels are contradicted by the paper. |
| | |
| | **Why the evidence quote cannot decide this.** ''%%crawlConfig%%'' is an object carrying **one** ''%%evidence.quote%%'' for statefulness, browsers, headless, consent action, interaction depth, authentication, languages and repeat visits together ([[programming:registration]], [[privacy:consent]]). So the audit did not read the quote as evidence; it read the **paper text** — ''%%paper.cols.txt%%'' and the raw ''%%paper.txt%%'' pypdf render, whitespace collapsed and de-hyphenated, because either rendering alone splices sentences. The sibling ''%%consentAction%%'' result (279 of 313 ''%%no-interaction%%'' labels unsupported, 89.1%) was **not** used as evidence that authentication labels are wrong; see the rejected finding below. |
| | |
| | **Command and real output.** |
| | |
| | <code> |
| | $ node scripts/platforms_report.mjs # unchanged, the schema split the page publishes |
| | ## The login wall: crawlConfig.authentication |
| | platform-subject papers with a crawlConfig object (260) |
| | none 145 55.8% |
| | not-stated 89 34.2% |
| | account-registration 22 8.5% |
| | manual-login 4 1.5% |
| | -> states a value: 171/260 = 65.8% |
| | |
| | $ node scripts/auth145_audit.mjs --verdicts |
| | ## Audit of the 145 platform-subject papers labelled crawlConfig.authentication='none' |
| | (denominator: the 145, not the 260 with a crawlConfig object and not the corpus) |
| | contradicted 6 4.1% |
| | affirmed 12 8.3% |
| | ambiguous 8 5.5% |
| | silent 119 82.1% |
| | total 145 |
| | </code> |
| | |
| | **How the candidate set was built, and what its recall rests on.** The 145 were re-derived inside ''%%scripts/auth145_audit.mjs%%'' with the same four signals ''%%platforms_report.mjs%%'' uses, and the script **throws** if the platform-subject-with-crawlConfig population is not 260 or the ''%%none%%'' count is not 145. Five paired probes then ran over the paper text (''%%--ctx%%''), each with a tight and a loose form, and the script **throws** if any tight probe returns more matches than its loose form in the same file — the first version did, on //Cookies That Give You Away//, which is why the guard exists. Stage 3 (''%%--buckets%%'') sorted all 145 into three candidate sets: papers with a first-person register/login sentence (**64**), papers with an explicit no-authentication sentence (**11**), and the residue where neither fires (**70**). **This sentence was wrong when first published, and a figures review caught it.** It said the pool read by hand was stage 3's A and C sets, 75 papers. It was not: the pool is the **union** of stage 2's strong first-person list and stage 3's A and C sets, and the review found the proof — //A View into YouTube View Fraud// carries a verdict below while sitting in stage 3's **B residue**, because stage 2 surfaced it and stage 3 did not. The script now computes that union itself and ''%%--verdicts%%'' **throws** if any adjudicated paper falls outside it (mutation-tested: disabling the stage 2 term makes it throw on exactly that paper). Its real output: **97 of 145** papers in the pool, **26** given a verdict, **71** read and left silent, and **48** that never surfaced a sentence to read. So the 119 **silent** are 71 papers read and found to say nothing about their own authentication posture plus 48 that no probe reached — not, as first published, 119 papers none of which were read. Recall therefore rests on the C probe being loose (it fires on any ''%%our ... account%%''), which is why it returned 64 candidates of which most were topical noise about the papers' //subjects// rather than their methods. |
| | |
| | **What changed.** [[design:platforms]]'s logged-out-surface bullet now carries the audit result in place of "We did not audit these 145". The schema counts (145 / 22 / 4 / 0, and the 55.8%) are **unchanged** — this is a precision audit published alongside the labels, in the same style as the Meta / Twitter-X / TikTok / Amazon subject-fold audits higher up this page. No number was hand-edited and no label was rewritten in ''%%extractions.jsonl%%''. |
| | |
| | **Two of the six read as contradicting their own object.** //A Large-scale Investigation into Geodifferences in Mobile Apps// (USENIX Security 2022) has a ''%%crawlConfig%%'' quote about the Selenium privacy-policy scraper while the study's actual download apparatus is ten Google Pixel 2 phones, one Google account per country, with an apk scraper that "requires the device's ID and Google account credentials". //Experimental Analyses of the Physical Surveillance Risks in Client-Side Content Scanning// (NDSS 2024) is scored ''%%none%%'' while its own ''%%crawlConfig.evidence.quote%%'' reads "we scrape user photographs from Instagram using an alias account" — the label contradicts the quote it was stored with. |
| | |
| | ==== scripts/auth145_audit.mjs ==== |
| | |
| | The committed file, verbatim. |
| | |
| | <file javascript auth145_audit.mjs> |
| | // Audit of the 145 platform-subject papers labelled crawlConfig.authentication='none'. |
| | // |
| | // Filed by design:platforms itself: "We did not audit these 145. Treat the |
| | // authentication shape as real and the precise share as unverified." |
| | // |
| | // The label cannot be checked against its own evidence quote, because |
| | // crawlConfig carries ONE evidence quote for the whole object (see |
| | // programming:registration). So this script does not read the quote as |
| | // evidence. It reads the PAPER TEXT. |
| | // |
| | // node scripts/auth145_audit.mjs # probe + candidate list |
| | // node scripts/auth145_audit.mjs --ctx <n> # printed context window (default 260) |
| | // node scripts/auth145_audit.mjs --control # random sample of non-firing papers |
| | // |
| | // The paper population is re-derived with the same four signals |
| | // platforms_report.mjs uses; the counts are asserted against that script's |
| | // published output (260 with a crawlConfig object, 145 labelled 'none') and |
| | // this script throws if either moves. |
| | |
| | import fs from 'node:fs'; |
| | import path from 'node:path'; |
| | import { loadExtractions, isSentinel, dataRoot } from './lib.mjs'; |
| | import { tagsOf, roleOf, familiesOf, ANY_PLATFORM } from './plat_fold.mjs'; |
| | |
| | const argv = process.argv.slice(2); |
| | const has = (f) => argv.includes(f); |
| | const val = (f, d) => (argv.indexOf(f) >= 0 ? argv[argv.indexOf(f) + 1] : d); |
| | const CTX = Number(val('--ctx', 260)); |
| | const pct = (n, d) => `${((100 * n) / d).toFixed(1)}%`; |
| | |
| | const rows = loadExtractions(); |
| | const key = (r) => `${r.venue}/${r.year}/${r.slug}`; |
| | const SIGNALS = { |
| | title: (r) => [r.title], |
| | sourceList: (r) => (r.population || []).map((p) => p.sourceList), |
| | phenomenon: (r) => (r.detection || []).map((d) => d.phenomenon), |
| | tool: (r) => |
| | (r.tools || []) |
| | .filter((t) => t.usedOrMentioned === 'used' || t.usedOrMentioned === 'produced') |
| | .map((t) => t.name), |
| | }; |
| | |
| | const perPaper = new Map(); |
| | for (const r of rows) { |
| | const k = key(r); |
| | const fams = new Map(); |
| | for (const [sig, get] of Object.entries(SIGNALS)) { |
| | for (const raw of get(r)) { |
| | if (!raw || isSentinel(raw)) continue; |
| | const s = String(raw); |
| | if (sig === 'sourceList' && (r.participants || []).length > 0) continue; |
| | for (const { family, role } of tagsOf(s)) { |
| | if (role !== 'subject') continue; |
| | if (!fams.has(family)) fams.set(family, new Set()); |
| | fams.get(family).add(sig); |
| | } |
| | } |
| | } |
| | if (fams.size) perPaper.set(k, fams); |
| | } |
| | const byKey = new Map(rows.map((r) => [key(r), r])); |
| | const anySubject = [...perPaper.keys()].map((k) => byKey.get(k)); |
| | const platCrawled = anySubject.filter((r) => r.crawlConfig !== null); |
| | const none = platCrawled.filter((r) => r.crawlConfig.authentication === 'none'); |
| | |
| | // Guard: this script's population must be platforms_report.mjs's population. |
| | if (platCrawled.length !== 260) |
| | throw new Error(`population drift: platform-subject papers with a crawlConfig object = ${platCrawled.length}, platforms_report.mjs publishes 260`); |
| | if (none.length !== 145) |
| | throw new Error(`population drift: authentication='none' = ${none.length}, platforms_report.mjs publishes 145`); |
| | |
| | // ------------------------------------------------------------------ full text |
| | // paper.cols.txt de-columns but leaves ~a quarter of pages spliced; paper.txt |
| | // is the raw pypdf render. Read BOTH and probe the union, whitespace collapsed, |
| | // because a PDF line break inside a phrase silently undercounts. |
| | const collapse = (s) => s.replace(//g, '').replace(/-\s*\n\s*/g, '').replace(/\s+/g, ' '); |
| | function textOf(r) { |
| | const dir = path.join(dataRoot(), 'fulltext', String(r.year), r.venue, r.slug); |
| | const out = []; |
| | for (const f of ['paper.cols.txt', 'paper.txt', 'paper.norm.txt']) { |
| | const p = path.join(dir, f); |
| | if (!fs.existsSync(p)) continue; |
| | const raw = fs.readFileSync(p, 'latin1'); |
| | if (raw.length === 0) continue; |
| | out.push({ file: f, text: collapse(raw) }); |
| | } |
| | return out; |
| | } |
| | |
| | // ------------------------------------------------------------------- probes |
| | // Two widths per concern. The tight probe must return a subset of the loose |
| | // one; the script asserts tight <= loose per paper and prints both. |
| | const PROBES = [ |
| | ['created-account', /\b(creat(?:e|ed|ing)|register(?:ed|ing)?|set up|made)\s+(?:\w+\s+){0,4}(accounts?|profiles?|personas?|sock[- ]?puppets?|bot accounts?)\b/i, |
| | /(account|profile|persona|sock)/i], |
| | ['logged-in', /\b(logged[- ]?in|log(?:ged)?\s+into|sign(?:ed|ing)?[- ]in(?:to)?|authenticated (?:session|crawl|user|request)|login(?:s|ed)? (?:to|into)|after (?:the )?login)\b/i, |
| | /(log|sign|authenticat)/i], |
| | ['credentials', /\b(credentials?|username and password|password(?:ed)? account|session cookies?|auth(?:entication)? tokens?|access tokens?|OAuth)\b/i, |
| | /(credential|password|cookie|token|oauth|username)/i], |
| | ['login-wall', /\b(login[- ]wall|behind a login|requires? (?:a )?(?:login|an account|sign[- ]?up)|logged[- ]?out (?:view|surface|user))\b/i, |
| | /(log|account|sign)/i], |
| | ['sso', /\b(single sign[- ]on|SSO|sign in with (?:Google|Facebook|Apple)|social login|identity provider)\b/i, |
| | /(sso|sign|single|identity provider)/i], |
| | ]; |
| | |
| | const results = []; |
| | for (const r of none) { |
| | const texts = textOf(r); |
| | const hits = {}; |
| | let anyText = texts.length > 0 && texts.some((t) => t.text.length > 2000); |
| | for (const [name, tight, loose] of PROBES) { |
| | let nT = 0, nL = 0, sample = null; |
| | for (const { text } of texts) { |
| | const mT = text.match(new RegExp(tight.source, 'gi')) || []; |
| | const mL = text.match(new RegExp(loose.source, 'gi')) || []; |
| | if (mT.length > mL.length) |
| | throw new Error(`probe ${name} on ${key(r)} in ${'' + text.length}: tight ${mT.length} > loose ${mL.length} (loose must contain tight)`); |
| | nT = Math.max(nT, mT.length); |
| | nL = Math.max(nL, mL.length); |
| | if (!sample && mT.length) { |
| | const i = text.search(new RegExp(tight.source, 'i')); |
| | sample = text.slice(Math.max(0, i - CTX / 2), i + CTX).trim(); |
| | } |
| | } |
| | hits[name] = { tight: nT, loose: nL, sample }; |
| | } |
| | results.push({ r, hits, anyText, nfiles: texts.length }); |
| | } |
| | |
| | const fired = results.filter((x) => Object.values(x.hits).some((h) => h.tight > 0)); |
| | const quiet = results.filter((x) => !Object.values(x.hits).some((h) => h.tight > 0)); |
| | const notext = results.filter((x) => !x.anyText); |
| | |
| | console.log(`## Population`); |
| | console.log(` platform-subject papers with a crawlConfig object : ${platCrawled.length}`); |
| | console.log(` of those, crawlConfig.authentication = 'none' : ${none.length}`); |
| | console.log(` full text readable for : ${results.length - notext.length}/${none.length}`); |
| | if (notext.length) for (const x of notext) console.log(` NO TEXT ${key(x.r)}`); |
| | |
| | console.log(`\n## Probe totals over the 145 (papers, not matches)`); |
| | for (const [name] of PROBES) { |
| | const t = results.filter((x) => x.hits[name].tight > 0).length; |
| | const l = results.filter((x) => x.hits[name].loose > 0).length; |
| | console.log(` ${name.padEnd(16)} tight ${String(t).padStart(3)} loose ${String(l).padStart(3)}`); |
| | } |
| | console.log(`\n any tight probe fires : ${fired.length}/${none.length}`); |
| | console.log(` no tight probe fires : ${quiet.length}/${none.length}`); |
| | |
| | if (has('--control')) { |
| | // Deterministic sample of the quiet papers, for the false-negative side. |
| | const s = quiet.slice().sort((a, b) => key(a.r).localeCompare(key(b.r))); |
| | const pick = []; |
| | for (let i = 0; i < s.length; i += Math.max(1, Math.floor(s.length / 20))) pick.push(s[i]); |
| | console.log(`\n## Control: ${pick.length} papers where no tight probe fired`); |
| | for (const x of pick) console.log(` ${key(x.r)} ${x.r.title}`); |
| | process.exit(0); |
| | } |
| | |
| | console.log(`\n## Candidates (tight probe fired) — ${fired.length}`); |
| | for (const x of fired.sort((a, b) => key(a.r).localeCompare(key(b.r)))) { |
| | const names = Object.entries(x.hits).filter(([, h]) => h.tight > 0).map(([n, h]) => `${n}:${h.tight}`); |
| | console.log(`\n--- ${key(x.r)}`); |
| | console.log(` title: ${x.r.title}`); |
| | console.log(` probes: ${names.join(' ')}`); |
| | console.log(` crawlConfig.evidenceQuote: ${JSON.stringify(x.r.crawlConfig.evidenceQuote || null)}`); |
| | for (const [n, h] of Object.entries(x.hits)) if (h.sample) console.log(` [${n}] ${h.sample}`); |
| | } |
| | |
| | // ---------------------------------------------------------- stage 2: --method |
| | // A tight probe fires on any mention of accounts or logins, including the |
| | // paper's SUBJECT (spam accounts, credential theft). What decides the |
| | // authentication label is what the AUTHORS did. Stage 2 keeps only sentences |
| | // that carry a first-person / own-apparatus subject AND an auth term, and |
| | // prints them in full for hand reading. Run over all 145, not just the 82, |
| | // because sentence co-occurrence is a different probe, not a narrowing of |
| | // stage 1. |
| | if (has('--method')) { |
| | const FIRST = /\b(we|our|us|the authors?|this (?:paper|study|work)|i )\b/i; |
| | const APPARATUS = /\b(crawler?s?|scraper?s?|bots?|browser|instrumentation|framework|pipeline|measurement|collection|harvest)\b/i; |
| | const AUTH = /\b(account|log ?in|logged|sign(?:ed|ing)? ?in|authenticat|credential|password|session cookie|API key|access token|OAuth|sock ?puppet|persona|profile we|registered)\b/i; |
| | let printed = 0; |
| | for (const x of results.sort((a, b) => key(a.r).localeCompare(key(b.r)))) { |
| | const seen = new Set(); |
| | const keep = []; |
| | for (const { text } of textOf(x.r)) { |
| | for (const sent of text.split(/(?<=[.!?])\s+(?=[A-Z(])/)) { |
| | if (sent.length < 30 || sent.length > 700) continue; |
| | if (!AUTH.test(sent)) continue; |
| | if (!FIRST.test(sent) && !APPARATUS.test(sent)) continue; |
| | const norm = sent.toLowerCase().replace(/[^a-z0-9]/g, '').slice(0, 90); |
| | if (seen.has(norm)) continue; |
| | seen.add(norm); |
| | keep.push(sent.trim()); |
| | } |
| | } |
| | console.log(`\n=== ${key(x.r)}`); |
| | console.log(` ${x.r.title}`); |
| | console.log(` quote: ${JSON.stringify(x.r.crawlConfig.evidenceQuote || null)}`); |
| | if (!keep.length) console.log(' (no first-person auth sentence)'); |
| | for (const s of keep.slice(0, 14)) console.log(` * ${s}`); |
| | printed++; |
| | } |
| | console.log(`\n-- ${printed} papers printed`); |
| | } |
| | |
| | // ------------------------------------------------------- stage 3: --buckets |
| | // `none` and `not-stated` are BOTH available in the schema, so `none` is an |
| | // affirmative claim: the crawler did not log in or register. Stage 3 sorts the |
| | // 145 by what the paper actually says about its own crawl: |
| | // A affirmed — the paper states it did not authenticate |
| | // C contradicted — the paper describes registering or logging in to collect data |
| | // B silent — neither; `none` was inferred from the paper saying nothing |
| | // A and C are candidate sets, hand-read one by one; B is the residue and is |
| | // printed in full so it can be counted, not assumed. |
| | if (has('--buckets')) { |
| | // "does not take into account" / "accounts for 75% of the market" swamp any |
| | // probe that treats the bare word `account` as an authentication term: all 20 |
| | // hits of the first version of AFFIRM were that. AFFIRM therefore requires a |
| | // login/authentication term, and excludes the two idioms explicitly. |
| | const IDIOM = /(take[ns]? |taking |taken )?into account|accounts? for\b|on account of/i; |
| | const AFFIRM = [ |
| | /\b(without|never|not|no|did not|does not|neither|refrain\w* from)\b[^.]{0,90}\b(logg?(?:ed|ing)? ?in(?:to)?|log ?in|sign(?:ed|ing)? ?in|authenticat\w*|login wall|creat\w+ an account|registering|registration)\b/i, |
| | /\b(logged[- ]?out|unauthenticated|without (?:an? )?(?:account|login)|publicly available without)\b/i, |
| | /\bdo(?:es)? not (?:require|need)\b[^.]{0,40}\b(login|logging|account|authenticat\w*)\b/i, |
| | ]; |
| | const CONTRA = [ |
| | /\b(we|our)\b[^.]{0,120}?\b(creat\w+|register\w+|set ?up|sign(?:ed)? up|purchas\w+)\b[^.]{0,60}?\b(account|profile|persona|sock ?puppet)s?\b/i, |
| | /\b(we|our)\b[^.]{0,120}?\b(logg?ed (?:in|into)|log(?:ged)? in to|signed in(?:to)?|authenticat\w+)\b/i, |
| | /\bour\b[^.]{0,40}\b(accounts?|credentials?|logins?)\b/i, |
| | /\b(account|login) credentials?\b[^.]{0,60}\b(crawler|scraper|tool|script|download)/i, |
| | ]; |
| | const bucket = { A: [], B: [], C: [] }; |
| | for (const x of results) { |
| | const sents = []; |
| | for (const { text } of textOf(x.r)) |
| | for (const s of text.split(/(?<=[.!?])\s+(?=[A-Z(])/)) if (s.length >= 30 && s.length <= 700) sents.push(s); |
| | const a = sents.filter((s) => !IDIOM.test(s) && AFFIRM.some((p) => p.test(s))); |
| | const c = sents.filter((s) => CONTRA.some((p) => p.test(s))); |
| | x.aHits = a; x.cHits = c; |
| | if (c.length) bucket.C.push(x); |
| | else if (a.length) bucket.A.push(x); |
| | else bucket.B.push(x); |
| | } |
| | console.log(`\n## Stage 3 candidate buckets over the 145 (probe output, NOT the audit result)`); |
| | console.log(` C paper has a first-person register/login sentence : ${bucket.C.length}`); |
| | console.log(` A paper has an explicit no-authentication sentence : ${bucket.A.length}`); |
| | console.log(` B residue, neither probe fires : ${bucket.B.length}`); |
| | console.log(` total ${bucket.A.length + bucket.B.length + bucket.C.length} (must be 145)`); |
| | if (bucket.A.length + bucket.B.length + bucket.C.length !== 145) throw new Error('buckets do not sum to 145'); |
| | for (const [name, set] of Object.entries(bucket)) { |
| | console.log(`\n######## BUCKET ${name} (${set.length})`); |
| | for (const x of set.sort((p, q) => key(p.r).localeCompare(key(q.r)))) { |
| | console.log(`\n=== ${key(x.r)}\n ${x.r.title}`); |
| | console.log(` quote: ${JSON.stringify(x.r.crawlConfig.evidence ? x.r.crawlConfig.evidence.quote : null)}`); |
| | const seen = new Set(); |
| | for (const s of [...x.cHits.map((v) => ['C', v]), ...x.aHits.map((v) => ['A', v])].slice(0, 40)) { |
| | const n = s[1].toLowerCase().replace(/[^a-z0-9]/g, '').slice(0, 80); |
| | if (seen.has(n)) continue; seen.add(n); |
| | if (seen.size > 8) break; |
| | console.log(` ${s[0]}> ${s[1]}`); |
| | } |
| | } |
| | } |
| | } |
| | |
| | // ------------------------------------------------------- stage 4: --verdicts |
| | // The audit result. Every verdict below was reached by READING the paper, not |
| | // by reading crawlConfig.evidence.quote — that quote covers the whole |
| | // crawlConfig object and cannot adjudicate this field (programming:registration). |
| | // |
| | // The candidate set that was read is stage 3's A and C buckets (the papers with |
| | // any first-person sentence about accounts, logins or authentication, either |
| | // direction). Papers in neither bucket are SILENT by construction: they contain |
| | // no first-person statement about their own authentication posture, so `none` |
| | // there is an inference from the paper saying nothing, not a reading of it. |
| | // |
| | // affirmed the paper states its collection was unauthenticated |
| | // contradicted the paper describes registering or using accounts FOR the |
| | // collection the crawlConfig object describes |
| | // ambiguous the study used accounts, but the crawl the evidence quote |
| | // describes plausibly did not; a reasonable person could score |
| | // this either way |
| | // silent no statement either way (the residue; not individually read) |
| | const VERDICTS = { |
| | // ---- contradicted (6) |
| | 'NDSS/2012/warningbird-detecting-suspicious-urls-in-twitter-stream': |
| | ['contradicted', 'Collection is the Twitter Streaming API under the authors’ own authenticated accounts: "To collect tweets, we use Twitter Streaming APIs. Our accounts have the Spritzer access role".'], |
| | 'WWW/2013/i-know-the-shortened-urls-you-clicked-on-twitter-inference-attack-using-public-c': |
| | ['contradicted', '"To perform the attack, we create monitoring accounts that monitor messages from all followings of a target user" — and "We created a Twitter account and followed 29 Twitter users". The accounts are the collection apparatus.'], |
| | 'USENIX/2022/a-large-scale-investigation-into-geodifferences-in-mobile-apps': |
| | ['contradicted', '"We set up these phones with one Google account per country" and "The apk scraper requires the device’s ID and Google account credentials". The crawlConfig quote describes only the Selenium privacy-policy sub-crawl.'], |
| | 'USENIX/2022/exploring-the-unchartered-space-of-container-registry-typosquatting': |
| | ['contradicted', '"We register all of our accounts legally through Docker Hub and Google Container Registry"; pull counts are read from the authors’ own Docker Hub and Google Cloud Console, and 20,000 AWS accounts were spawned from a registered pilot account.'], |
| | 'NDSS/2024/experimental-analyses-of-the-physical-surveillance-risks-in-client-side-content-scanning': |
| | ['contradicted', 'The crawlConfig evidence quote ITSELF says "we scrape user photographs from Instagram using an alias account". The label contradicts its own quote.'], |
| | 'USENIX/2025/no-way-to-sign-out-unpacking-non-compliance-with-google-plays-app-account-deleti': |
| | ['contradicted', '"We registered around 300 accounts in total throughout the study. Each app was assigned two registered accounts"; "we manually registered, deleted, relogged in, and re-registered accounts".'], |
| | // ---- affirmed (12) |
| | 'IMC/2010/detecting-and-characterizing-social-spam-campaigns': |
| | ['affirmed', '"Because access to regional networks is unauthenticated, we chose 8 regional networks … as targets for data collection."'], |
| | 'WWW/2012/serf-and-turf-crowdturfing-for-fun-and-profit': |
| | ['affirmed', '"All information on both sites is publicly available, and neither site employs security measures to prevent crawling." The accounts the authors created were for a separate end-to-end campaign experiment, not the crawl.'], |
| | 'USENIX/2014/towards-detecting-anomalous-user-behavior-in-online-social-networks': |
| | ['affirmed', '"We did not seek or receive any account credentials of any Facebook user."'], |
| | 'IEEE-SP/2018/the-spyware-used-in-intimate-partner-violence': |
| | ['affirmed', '"We did not set up any user profile, and performed each search from a new browser session." Accounts were created only during manual app analysis, not for the search crawl.'], |
| | 'WWW/2019/measuring-political-personalization-of-google-news-search': |
| | ['affirmed', '"we do not create or log-in to a Firefox account during training"; "We also do not create or log-in to a Google account".'], |
| | 'USENIX/2020/the-tools-and-tactics-used-in-intimate-partner-surveillance-an-analysis-of-onlin': |
| | ['affirmed', '"Our data is from publicly available fora accessible on the Internet without authentication."'], |
| | 'USENIX/2022/behind-the-tube-exploitative-monetization-of-content-on-youtube': |
| | ['affirmed', '"We considered only sites publicly accessible … and did not require registration or prior site activity to view information."'], |
| | 'IEEE-SP/2024/no-easy-way-out-the-effectiveness-of-deplatforming-an-extremist-forum-to-suppres': |
| | ['affirmed', '"we refrain from scraping private and protected posts behind the login wall due to safety and legality concerns."'], |
| | 'USENIX/2025/lost-in-the-mists-of-time-expirations-in-dns-footprints-of-mobile-apps': |
| | ['affirmed', '"We did not attempt to bypass any login or splash screens of apps."'], |
| | 'PETS/2026/banned-books-analysis-of-censorship-on-amazon-com': |
| | ['affirmed', '"at no point in our methodology do we sign into any Amazon account"; "all data we collected was available publicly without being logged in".'], |
| | 'PETS/2026/the-tcf-doesn-t-really-a-a-id-automatic-privacy-analysis-and-legal-compliance-of': |
| | ['affirmed', '"Our Selenium instance was not logged in our Google account while crawling" — which is also the crawlConfig evidence quote.'], |
| | 'USENIX/2026/bridges-to-self-silent-web-to-app-tracking-on-mobile-via-localhost': |
| | ['affirmed', '"We did not use authenticated user accounts and limited interactions to what was necessary to observe default script behavior."'], |
| | // ---- ambiguous (8) |
| | 'CCS/2010/spam-the-underground-on-140-characters-or-less': |
| | ['ambiguous', 'The URL crawler in the evidence quote is unauthenticated, but the Twitter monitoring used a "fake account to act as a spam trap".'], |
| | 'CCS/2022/ready-raider-one-exploring-the-misuse-of-cloud-gaming-services': |
| | ['ambiguous', '"We registered all of our accounts legally through cloud gaming services" — but the crawlConfig quote is the Steam game-database crawler, which plausibly needed none.'], |
| | 'CCS/2023/dont-leak-your-keys-understanding-measuring-and-exploiting-the-appsecret-leaks-i': |
| | ['ambiguous', '"we first registered two accounts, a victim account and an attack account" — for a single proof-of-concept attack, not for the keyword-seeded mini-program crawl in the quote.'], |
| | 'USENIX/2018/o-single-sign-off-where-art-thou-an-empirical-analysis-of-single-sign-on-account': |
| | ['ambiguous', 'The 1M-site SSO-prevalence crawl only detects login/signup pages, but the RP takeover study "create[s] a new account using SSO" for each website.'], |
| | 'USENIX/2022/ovrseen-auditing-network-traffic-and-privacy-policies-in-oculus-vr': |
| | ['ambiguous', 'The webstore crawl in the quote is unauthenticated; the network-traffic collection that is the paper’s main measurement used "our test OVR account".'], |
| | 'WWW/2022/a-view-into-youtube-view-fraud': |
| | ['ambiguous', '"we generated 20 API keys (the maximum number of keys allowed by Google per account)" — an API credential tied to a Google account, which the schema’s "logged in or registered accounts" does not clearly cover.'], |
| | 'PETS/2024/privacy-policies-on-the-fediverse-a-case-study-of-mastodon-instances': |
| | ['ambiguous', 'The 351-policy Selenium download was unauthenticated; one account was created afterwards to check a single instance’s cookie claim.'], |
| | 'USENIX/2025/darkgram-a-large-scale-analysis-of-cybercriminal-activity-channels-on-telegram': |
| | ['ambiguous', '"we used the official Telegram API to collect 64,801 posts" — Telegram API access is bound to an account, but the paper never says so.'], |
| | }; |
| | |
| | // The candidate pool that was actually hand-read is the UNION of two probes, |
| | // not stage 3 alone. Stage 2 (--method) prints, for all 145, the sentences that |
| | // carry a first-person/own-apparatus subject AND an auth term; the "strong" |
| | // subset of that list is what was read. Stage 3 (--buckets) adds its A and C |
| | // sets. A figures review caught that WWW/2022/a-view-into-youtube-view-fraud |
| | // carries a verdict while sitting in stage 3's B residue — it was surfaced by |
| | // stage 2, not stage 3. The pool is therefore computed here, and --verdicts |
| | // THROWS if any adjudicated paper is outside it, so the script can no longer |
| | // claim a candidate-discovery process it does not reproduce. |
| | const STRONG = [ |
| | /\b(we|our)\b[^.]{0,140}?\b(creat\w*|register\w*|set up|signed up|sign up|made|use[d]?|operat\w*|control\w*|maintain\w*)\b[^.]{0,60}?\b(account|profile|persona|sock ?puppet|bot account)s?\b/i, |
| | /\b(we|our)\b[^.]{0,140}?\b(logged[- ]?in|log ?in|logged into|signed in|authenticat\w*|credential\w*|session cookie|access token|api key|oauth|bearer token)\b/i, |
| | /\b(logged[- ]?in|authenticated)\b[^.]{0,60}\b(crawl\w*|session|browser|account|user|state)\b/i, |
| | /\b(without|not)\b[^.]{0,40}\b(log ?in|logging in|account|authenticat\w*)\b/i, |
| | ]; |
| | function stage2Strong(x) { |
| | for (const { text } of textOf(x.r)) |
| | for (const sent of text.split(/(?<=[.!?])\s+(?=[A-Z(])/)) { |
| | if (sent.length < 30 || sent.length > 700) continue; |
| | if (STRONG.some((p) => p.test(sent))) return true; |
| | } |
| | return false; |
| | } |
| | |
| | if (has('--verdicts')) { |
| | const keys145 = new Set(none.map(key)); |
| | for (const k of Object.keys(VERDICTS)) if (!keys145.has(k)) throw new Error(`verdict for a paper that is not in the 145: ${k}`); |
| | // Recompute the candidate pool and assert every verdict came out of it. |
| | const AF = [ |
| | /\b(without|never|not|did not|does not|neither|refrain\w* from)\b[^.]{0,90}\b(logg?(?:ed|ing)? ?in(?:to)?|log ?in|sign(?:ed|ing)? ?in|authenticat\w*|login wall|creat\w+ an account|registering|registration)\b/i, |
| | /\b(logged[- ]?out|unauthenticated|without (?:an? )?(?:account|login)|publicly available without)\b/i, |
| | ]; |
| | const CO = [ |
| | /\b(we|our)\b[^.]{0,120}?\b(creat\w+|register\w+|set ?up|sign(?:ed)? up|purchas\w+)\b[^.]{0,60}?\b(account|profile|persona|sock ?puppet)s?\b/i, |
| | /\b(we|our)\b[^.]{0,120}?\b(logg?ed (?:in|into)|log(?:ged)? in to|signed in(?:to)?|authenticat\w+)\b/i, |
| | /\bour\b[^.]{0,40}\b(accounts?|credentials?|logins?)\b/i, |
| | ]; |
| | const IDIOM2 = /(take[ns]? |taking |taken )?into account|accounts? for\b|on account of/i; |
| | const pool = new Set(); |
| | for (const x of results) { |
| | let hit = stage2Strong(x); |
| | if (!hit) |
| | for (const { text } of textOf(x.r)) { |
| | for (const sent of text.split(/(?<=[.!?])\s+(?=[A-Z(])/)) { |
| | if (sent.length < 30 || sent.length > 700) continue; |
| | if (CO.some((p) => p.test(sent)) || (!IDIOM2.test(sent) && AF.some((p) => p.test(sent)))) { hit = true; break; } |
| | } |
| | if (hit) break; |
| | } |
| | if (hit) pool.add(key(x.r)); |
| | } |
| | for (const k of Object.keys(VERDICTS)) |
| | if (!pool.has(k)) throw new Error(`adjudicated paper outside the candidate pool the script computes: ${k}`); |
| | console.log(`\n## Candidate pool actually read (stage 2 strong list UNION stage 3 buckets A and C)`); |
| | console.log(` papers in the pool : ${pool.size}/145`); |
| | console.log(` of those, given a verdict : ${Object.keys(VERDICTS).length}`); |
| | console.log(` of those, read and left silent: ${pool.size - Object.keys(VERDICTS).length}`); |
| | console.log(` never surfaced a sentence : ${145 - pool.size}`); |
| | const counts = { contradicted: 0, affirmed: 0, ambiguous: 0, silent: 0 }; |
| | const rowsOut = []; |
| | for (const r of none) { |
| | const v = VERDICTS[key(r)]; |
| | const verdict = v ? v[0] : 'silent'; |
| | counts[verdict]++; |
| | if (v) rowsOut.push([verdict, key(r), r.title, v[1]]); |
| | } |
| | const total = Object.values(counts).reduce((a, b) => a + b, 0); |
| | if (total !== 145) throw new Error(`verdicts sum to ${total}, not 145`); |
| | console.log(`\n## Audit of the 145 platform-subject papers labelled crawlConfig.authentication='none'`); |
| | console.log(` (denominator: the 145, not the 260 with a crawlConfig object and not the corpus)`); |
| | for (const k of ['contradicted', 'affirmed', 'ambiguous', 'silent']) |
| | console.log(` ${k.padEnd(13)} ${String(counts[k]).padStart(3)} ${pct(counts[k], 145)}`); |
| | console.log(` total ${total}`); |
| | console.log(`\n## Per-paper evidence for every non-silent verdict`); |
| | for (const k of ['contradicted', 'affirmed', 'ambiguous']) |
| | for (const [v, kk, t, why] of rowsOut.filter((x) => x[0] === k)) |
| | console.log(`\n [${v}] ${kk}\n ${t}\n ${why}`); |
| | } |
| | |
| | </file> |
| | |
| | ==== The 26 adjudicated papers ==== |
| | |
| | Generated from ''%%node scripts/auth145_audit.mjs --verdicts%%''. The other 119 of the 145 are **silent**: no first-person sentence about the study's own authentication posture in either direction. |
| | |
| | ^ Verdict ^ Paper ^ Deciding sentence, from the paper text ^ |
| | | **contradicted** | //WarningBird: Detecting Suspicious URLs in Twitter Stream// (NDSS 2012) | Collection is the Twitter Streaming API under the authors’ own authenticated accounts: "To collect tweets, we use Twitter Streaming APIs. Our accounts have the Spritzer access role". | |
| | | **contradicted** | //I know the shortened URLs you clicked on Twitter: inference attack using public click analytics and Twitter metadata// (WWW 2013) | "To perform the attack, we create monitoring accounts that monitor messages from all followings of a target user" — and "We created a Twitter account and followed 29 Twitter users". The accounts are the collection apparatus. | |
| | | **contradicted** | //A Large-scale Investigation into Geodifferences in Mobile Apps// (USENIX 2022) | "We set up these phones with one Google account per country" and "The apk scraper requires the device’s ID and Google account credentials". The crawlConfig quote describes only the Selenium privacy-policy sub-crawl. | |
| | | **contradicted** | //Exploring the Unchartered Space of Container Registry Typosquatting// (USENIX 2022) | "We register all of our accounts legally through Docker Hub and Google Container Registry"; pull counts are read from the authors’ own Docker Hub and Google Cloud Console, and 20,000 AWS accounts were spawned from a registered pilot account. | |
| | | **contradicted** | //Experimental Analyses of the Physical Surveillance Risks in Client-Side Content Scanning// (NDSS 2024) | The crawlConfig evidence quote ITSELF says "we scrape user photographs from Instagram using an alias account". The label contradicts its own quote. | |
| | | **contradicted** | //No Way to Sign Out? Unpacking Non-Compliance with Google Play's App Account Deletion Requirements// (USENIX 2025) | "We registered around 300 accounts in total throughout the study. Each app was assigned two registered accounts"; "we manually registered, deleted, relogged in, and re-registered accounts". | |
| | | **affirmed** | //Detecting and characterizing social spam campaigns// (IMC 2010) | "Because access to regional networks is unauthenticated, we chose 8 regional networks … as targets for data collection." | |
| | | **affirmed** | //Serf and turf: crowdturfing for fun and profit// (WWW 2012) | "All information on both sites is publicly available, and neither site employs security measures to prevent crawling." The accounts the authors created were for a separate end-to-end campaign experiment, not the crawl. | |
| | | **affirmed** | //Towards Detecting Anomalous User Behavior in Online Social Networks// (USENIX 2014) | "We did not seek or receive any account credentials of any Facebook user." | |
| | | **affirmed** | //Measuring Political Personalization of Google News Search// (WWW 2019) | "we do not create or log-in to a Firefox account during training"; "We also do not create or log-in to a Google account". | |
| | | **affirmed** | //The Tools and Tactics Used in Intimate Partner Surveillance: An Analysis of Online Infidelity Forums// (USENIX 2020) | "Our data is from publicly available fora accessible on the Internet without authentication." | |
| | | **affirmed** | //Behind the Tube: Exploitative Monetization of Content on YouTube// (USENIX 2022) | "We considered only sites publicly accessible … and did not require registration or prior site activity to view information." | |
| | | **affirmed** | //No Easy Way Out: the Effectiveness of Deplatforming an Extremist Forum to Suppress Hate and Harassment// (IEEE-SP 2024) | "we refrain from scraping private and protected posts behind the login wall due to safety and legality concerns." | |
| | | **affirmed** | //Banned Books: Analysis of Censorship on Amazon.com// (PETS 2026) | "at no point in our methodology do we sign into any Amazon account"; "all data we collected was available publicly without being logged in". | |
| | | **affirmed** | //The TCF doesn’t really A(A)ID – Automatic Privacy Analysis and Legal Compliance of TCF-based Android Applications// (PETS 2026) | "Our Selenium instance was not logged in our Google account while crawling" — which is also the crawlConfig evidence quote. | |
| | | **affirmed** | //The Spyware Used in Intimate Partner Violence// (IEEE-SP 2018) | "We did not set up any user profile, and performed each search from a new browser session." Accounts were created only during manual app analysis, not for the search crawl. | |
| | | **affirmed** | //Lost in the Mists of Time: Expirations in DNS Footprints of Mobile Apps// (USENIX 2025) | "We did not attempt to bypass any login or splash screens of apps." | |
| | | **affirmed** | //Bridges to Self: Silent Web-to-App Tracking on Mobile via Localhost// (USENIX 2026) | "We did not use authenticated user accounts and limited interactions to what was necessary to observe default script behavior." | |
| | | **ambiguous** | //@spam: the underground on 140 characters or less// (CCS 2010) | The URL crawler in the evidence quote is unauthenticated, but the Twitter monitoring used a "fake account to act as a spam trap". | |
| | | **ambiguous** | //O Single Sign-Off, Where Art Thou? An Empirical Analysis of Single Sign-On Account Hijacking and Session Management on the Web// (USENIX 2018) | The 1M-site SSO-prevalence crawl only detects login/signup pages, but the RP takeover study "create[s] a new account using SSO" for each website. | |
| | | **ambiguous** | //Ready Raider One: Exploring the Misuse of Cloud Gaming Services// (CCS 2022) | "We registered all of our accounts legally through cloud gaming services" — but the crawlConfig quote is the Steam game-database crawler, which plausibly needed none. | |
| | | **ambiguous** | //OVRseen: Auditing Network Traffic and Privacy Policies in Oculus VR// (USENIX 2022) | The webstore crawl in the quote is unauthenticated; the network-traffic collection that is the paper’s main measurement used "our test OVR account". | |
| | | **ambiguous** | //A View into YouTube View Fraud// (WWW 2022) | "we generated 20 API keys (the maximum number of keys allowed by Google per account)" — an API credential tied to a Google account, which the schema’s "logged in or registered accounts" does not clearly cover. | |
| | | **ambiguous** | //Don't Leak Your Keys: Understanding, Measuring, and Exploiting the AppSecret Leaks in Mini-Programs// (CCS 2023) | "we first registered two accounts, a victim account and an attack account" — for a single proof-of-concept attack, not for the keyword-seeded mini-program crawl in the quote. | |
| | | **ambiguous** | //Privacy Policies on the Fediverse: A Case Study of Mastodon Instances// (PETS 2024) | The 351-policy Selenium download was unauthenticated; one account was created afterwards to check a single instance’s cookie claim. | |
| | | **ambiguous** | //DarkGram: A Large-Scale Analysis of Cybercriminal Activity Channels on Telegram// (USENIX 2025) | "we used the official Telegram API to collect 64,801 posts" — Telegram API access is bound to an account, but the paper never says so. | |
| | |
| | **Review.** One **model: 'sonnet'** figures-vs-script pass re-ran ''%%platforms_report.mjs%%'', ''%%report_registration.mjs%%'' and ''%%scripts/auth145_audit.mjs%%'' in all four modes, and checked every figure in the rewritten bullet plus the figures either side of it (the 383/208/175 existing-dataset split above it, the 897 denominator, the 34.2% not-stated share below it). Every verdict count, percentage and table row matched. It raised **two findings, both accepted**: (1) the candidate-pool sentence above misdescribed which papers were read, fixed above with a new script-computed guard; (2) [[programming:interaction]] enumerated the 857 web crawls as 553 + 204 + 81 = **838**, silently dropping the 19 papers with no ''%%crawlConfig%%'' object that ''%%report_registration.mjs%%'' prints as a sixth bucket — that page now carries the 19 and the sum reconciles. It also flagged that the ''%%kumar2022_investigation%%'' quote on [[design:platforms]] was ellipsis-joined across two sentences; it has been replaced with the contiguous sentence the audit script itself verifies. One **model: 'sonnet'** citations pass ran, because the bullet added two ''%%{[citekey]}%%'' markers and several verbatim quotes. It verified both citekeys against the papers' author lists, titles, venues and years, and **all 26** table rows verbatim against the paper text, reading the PDF directly where a column-spliced render made a quote look non-contiguous and stripping NUL bytes from WarningBird's text where plain ''%%grep%%'' silently returns nothing. It independently confirmed from ''%%extractions.jsonl%%'' that the Instagram //"alias account"// line really is that record's own ''%%crawlConfig.evidence.quote%%'' under an ''%%authentication: none%%'' label. It found nothing wrong. |
| | |
| | **Rejected finding.** That the 89.1% ''%%consentAction%%'' unsupported-label rate is evidence the authentication labels are wrong. It is not, and it was not used as such: it is a different field with a different extraction failure mode, and the authentication verdicts above each rest on a sentence from the paper in question. The 89.1% is the reason the audit was prioritised, not an input to any verdict on this page. |
| | |
| | **Deliberately not concluded.** That the corrected split is 139 / 28 / 4 / 0. Six labels being contradicted does not license re-deriving the published counts by hand: the corrected value for each of the six is itself a judgement (WarningBird's authenticated Streaming API accounts are not obviously ''%%account-registration%%'' rather than ''%%automated-login%%''), and the **8 ambiguous** papers would move the result either way depending on whether "the crawler" means the study's apparatus or the specific crawl the evidence quote names. The schema counts stay; the audit sits beside them. |
| |
