Measure The Web

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Provenance: design:user_studies ======

Working notes behind [[design:user_studies]] — every query with its population and denominator, the report script and its unedited output, the fold and its complete residue, the quotes checked against ''paper.cols.txt'', the external sources verified and the ones rejected, and the review log. Corpus-level caveats: [[literature:corpus]].

//Contemporaneous.// Written during the run that produced the content page, 2026-08-27, not reconstructed afterwards.

===== 1. What this page is backing =====

^ Item ^ Value ^
| Content page | [[design:user_studies]] — **new page**, created 2026-08-27 |
| Report script | ''scripts/report_user_studies.mjs'' → ''scripts/report_user_studies-output.txt'' |
| Fold | ''scripts/us_fold.mjs'' — CROWD_FAMILIES (multi-label), COMP_FAMILIES (single-label), LARGE_N hand map, ''isPoster'' |
| Regions | ''scripts/geo.mjs'' (existing) |
| Live platform check | ''pages/platform_status.py'' — exits 1 on a missing needle |
| Data | ''data/extract/run1/extractions.jsonl'', 5,859 papers, 7 venues, 2010–2026 |
| Bibliography additions | 5 new keys in ''pages/bib_additions_user_studies.bib'' |
| Reused keys | ''wei2024_solk'', ''klemmer2025_transparency'', ''zeber2020representativeness'', ''berke2025_demographics'', ''utz2019_informed'', ''ramulu2025_ethics'' |
| Previous figures | **none on a content page.** The item brief (965 / MTurk 120 / Prolific 27 / compensation 53.0% / regions 47.8%) was computed on the 4,322-paper corpus and was not reused. |

===== 2. Scope =====

==== Does a page already exist? ====

No. ''core.getPageInfo'' for ''design:user_studies'' on 2026-08-27: requested page does not exist. ''?do=export_raw'' returns the HTML error page (that is not an empty wiki page). ''start'' already links ''[[Design:User studies]]''. Creating, not extending.

[[design:automated_measurements]] already treats this as the sibling top-level Design branch and still said "red link until written" in its local copy; that gloss is updated in the same sitting.

==== Why this page rather than broadening a neighbour ====

  * **Chosen: ''design:user_studies''.** The item specified it. ''start'' already promised it as the other top-level Design branch. Neighbours that mention humans ([[practices:ethics]], [[statistics:study_preregistration]], [[statistics:interrater_agreement]], [[privacy:fingerprinting]]) own ethics / prereg / annotation / uniqueness, not recruitment.
  * **Rejected: putting this under ''practices:ethics''.** That page already measures the participants-vs-crawls ethics split. Adding MTurk/Prolific/n/compensation would bury the design question inside the review-board question.
  * **Rejected: an HCI textbook page.** CHI, SOUPS, CSCW are absent. The audience is a web-measurement PhD student who now needs humans, not a CHI methods course.
  * **Rejected: unioning ''studyTypes'' ''user-study'' (1,149) with ''participants[]'' (1,357).** They disagree both ways (232 / 24). The page population is ''participants[]'' non-empty, which is OVERVIEW.md's ''humanSubjects''.

==== Reachability ====

''start'' already linked here. A one-line gloss is added: Mechanical Turk shuts 30 September 2026.

===== 3. Populations =====

^ Name ^ Definition ^ N ^
| ''all'' | every extracted paper | 5,859 |
| **''humanSubjects''** | ''participants[]'' length > 0 — **page population** | **1,357** |
| ''user-study'' type | ''studyTypes'' includes ''user-study'' | 1,149 |
| ''interview-or-survey'' type | | 755 |
| ''unit=human-participants'' | | 979 |
| ''crawled'' | | 1,120 |
| hybrid | humanSubjects ∩ crawled | 149 |
| ''humanAnnotation'' | | 3,318 |
| crowdworker annotators | ''annotatorType == crowdworkers'' | 73 |
| crowdworker annotators NOT in humanSubjects | **the annotation trap** | 33 |

Sentinels are never answers. Papers, never tuples. ''paperCounts()'' in ''scripts/lib.mjs''.

===== 4. Running it =====

<code bash>
cd /workspace/artifacts/wiki
node scripts/report_user_studies.mjs > scripts/report_user_studies-output.txt
node scripts/report_user_studies.mjs --residue   # 25 unnamed other-crowd papers
uv run python pages/platform_status.py
node scripts/check_wrap.mjs pages/design_user_studies.txt
node scripts/check_tables.mjs pages/design_user_studies.txt
node scripts/check_page_numbers.mjs pages/design_user_studies.txt scripts/report_user_studies-output.txt
node scripts/check_attributions.mjs pages/design_user_studies.txt pages/bib_additions_user_studies.bib pages/literature_bibliography.txt
</code>

Unedited report output, 2026-08-27, exit 0, LARGE_N map and set agreed:

<file - report_user_studies-output.txt>
# design/user_studies — dataset queries
Corpus: 7 venues, 2010–2026, 5859 papers with full text.
2025 is thin at the edges and 2026 is provisional (CCS/IMC 2026 not yet held; IEEE S&P and WWW 2026 abstracts absent from OpenAlex).
Missing paper.cols.txt: 4 corpus-wide, 1 of humanSubjects.


## 1. POPULATIONS

Population                          Definition                                     Papers  Share of 5859
----------------------------------  ---------------------------------------------  ------  -------------
all                                 extraction records                             5859    100%
empirical                           isEmpirical === true                           5118    87.4%
crawled                             crawlConfig or studyTypes automated-web-crawl  1120    19.1%
**humanSubjects**                   participants[] non-empty — PAGE POPULATION     1357    23.2%
studyTypes user-study               studyTypes includes 'user-study'               1149    19.6%
studyTypes interview-or-survey      studyTypes includes 'interview-or-survey'      755     12.9%
either studyType                    user-study OR interview-or-survey              1328    22.7%
population.unit human-participants  any population[].unit == 'human-participants'  979     16.7%
humanAnnotation                     humanAnnotation[] non-empty                    3318    56.6%
crowdworker annotators              annotatorType == 'crowdworkers'                73      1.2%

### Signal disagreement (not unioned; humanSubjects is the page N)

Overlap                                            Papers
-------------------------------------------------  ------
humanSubjects ∩ user-study type                    1125
humanSubjects only (no user-study type)            232
user-study type only (no participants[])           24
humanSubjects ∩ interview-or-survey type           744
humanSubjects ∩ either type                        1294
either type only (no participants[])               34
humanSubjects ∩ unit=human-participants            930
unit=human only (no participants[])                49
humanSubjects only (no unit=human)                 427
humanSubjects ∩ crawled                            149
crawled, no participants                           971
humanSubjects ∩ annotated                          919
crowdworker-annot ∩ humanSubjects                  40
crowdworker-annot NOT in humanSubjects (THE TRAP)  33

**Page population is humanSubjects = 1357.** studyTypes user-study (1149) disagrees in both directions: 232 papers have participants[] but not the type, 24 have the type but an empty participants[]. The type-only papers are crawls, telemetry, or crowd *annotation* the extractor filed as a study type. They are listed under --list type-only.

### Annotation trap: crowdworkers labelling data

annotatorType over 3318 annotated papers (multi-label, papers not tuples):
annotatorType     Papers  Share of 3318
----------------  ------  -------------
authors           2848    85.8%
domain-experts    340     10.2%
mixed             109     3.3%
crowdworkers      73      2.2%
students          60      1.8%
hired-annotators  35      1.1%
legal-experts     5       0.2%

73 papers have annotatorType crowdworkers. 40 also have participants[]; **33 do not** — those are hired labelers, not human subjects. Conflating them double-counts Mechanical Turk.

## 2. VENUE AND YEAR (population: humanSubjects, N=1357)

Venue            humanSubjects  all papers  Share of venue
---------------  -------------  ----------  --------------
PoPETs           215            510         42.2%
IEEE S&P         208            767         27.1%
USENIX Security  367            1410        26.0%
NDSS             144            701         20.5%
TheWebConf       170            843         20.2%
CCS              186            990         18.8%
IMC              67             638         10.5%

PoPETs is the user-study venue in this corpus as a share of its own output. IMC is the least.

Window      all papers  humanSubjects  share of window
----------  ----------  -------------  ---------------
2010–2013   511         84             16.4%
2014–2017   769         155            20.2%
2018–2021   1439        329            22.9%
2022–2024   1955        486            24.9%
2025–2026*  1185        303            25.6%

* 2025–2026 is provisional.

Per calendar year (2026 is not a complete year):
Year  papers  humanSubjects  share
----  ------  -------------  -----
2010  119     18             15.1%
2011  116     12             10.3%
2012  151     29             19.2%
2013  125     25             20.0%
2014  166     31             18.7%
2015  190     37             19.5%
2016  182     35             19.2%
2017  231     52             22.5%
2018  254     48             18.9%
2019  402     92             22.9%
2020  404     83             20.5%
2021  379     106            28.0%
2022  546     117            21.4%
2023  719     198            27.5%
2024  690     171            24.8%
2025  770     198            25.7%
2026  415     105            25.3%

## 3. STUDY KIND (population: humanSubjects, N=1357, multi-label)

studyKind       Papers  Share of 1357
--------------  ------  -------------
survey          511     37.7%
lab-study       373     27.5%
interview       349     25.7%
field-study     229     16.9%
usability-test  99      7.3%
other           79      5.8%
focus-group     26      1.9%
diary-study     1       0.1%

stated 1357, sentinel-only 0, silent 0.
Shares do not sum to 100%: a paper can run a survey and an interview.

## 4. RECRUITMENT ENUM (schema, population: humanSubjects, N=1357)

recruitment (schema enum)  Papers  Share of 1357  Sentinel?
-------------------------  ------  -------------  ---------
other-crowd-platform       193     14.2%
professional-network       189     13.9%
university-pool            174     12.8%
mechanical-turk            139     10.2%
social-media               119     8.8%
snowball                   91      6.7%
in-person                  68      5.0%
prolific                   61      4.5%
panel-company              42      3.1%

stated 903 (66.5%), sentinel-only 454 (33.5%), silent 0.
**Sentinel-only is the finding: one humanSubjects paper in three does not name how people were recruited.**

## 5. CROWD PLATFORMS, RECOVERED (quotes + full text, population: humanSubjects)

other-crowd-platform papers: 193. Named a recovered family (quote or full text): 168. Named nothing: 25.
humanSubjects papers with missing paper.cols.txt during recovery: 1.

Platform                            Schema enum  Quote recovery  Quote+fulltext  Share of 1357 (fulltext)
----------------------------------  -----------  --------------  --------------  ------------------------
Prolific                            61           102             194             14.3%
Mechanical Turk                     139          92              279             20.6%
CloudResearch                       —            1               10              0.7%
CrowdFlower / Figure Eight / Appen  —            3               18              1.3%
Clickworker                         —            3               6               0.4%
Microworkers                        —            1               2               0.1%
Respondi                            —            3               4               0.3%
Wenjuanxing / Credamo               —            5               7               0.5%
Upwork / Freelancer                 —            11              31              2.3%
Toloka                              —            2               2               0.1%

Mechanical Turk undercount: schema 139 vs quote+fulltext 279 (schema is 49.8% of recovered).
Prolific undercount: schema 61 vs quote+fulltext 194 (schema is 31.4% of recovered).
Union MTurk∪Prolific recovered: 409 of 1357 (30.1%).
Papers recovered as both MTurk and Prolific: 64.

Qualtrics full-text (NOT a recruitment family — it is the survey tool): 124 of 1357 humanSubjects, 134 of 5859 corpus. missing 1.
CloudResearch (TurkPrime) recovered: 10 of 1357.

Unnamed other-crowd residue: 25 papers. Pass --residue to list them.

### Schema vs recovered, over time (humanSubjects in each window)

Window      HS   schema MTurk  recovered MTurk  schema Prolific  recovered Prolific  schema other-crowd
----------  ---  ------------  ---------------  ---------------  ------------------  ------------------
2010–2013   84   14            23               0                0                   5
2014–2017   155  20            42               0                3                   13
2018–2021   329  60            104              2                27                  42
2022–2024   486  38            86               25               94                  91
2025–2026*  303  7             24               34               70                  42

The recovered series is the one to date methods from. Schema Prolific is 0 until 2019 in both; recovered Prolific leads schema in every window after that because other-crowd-platform ate it.

### Recovered MTurk vs Prolific, per calendar year

Year  humanSubjects  recovered MTurk  recovered Prolific  MTurk share of HS  Prolific share of HS
----  -------------  ---------------  ------------------  -----------------  --------------------
2010  18             2                0                   11.1%              0.0%
2011  12             5                0                   41.7%              0.0%
2012  29             10               0                   34.5%              0.0%
2013  25             6                0                   24.0%              0.0%
2014  31             6                0                   19.4%              0.0%
2015  37             13               2                   35.1%              5.4%
2016  35             10               1                   28.6%              2.9%
2017  52             13               0                   25.0%              0.0%
2018  48             16               5                   33.3%              10.4%
2019  92             29               6                   31.5%              6.5%
2020  83             27               6                   32.5%              7.2%
2021  106            32               10                  30.2%              9.4%
2022  117            24               22                  20.5%              18.8%
2023  198            34               33                  17.2%              16.7%
2024  171            28               39                  16.4%              22.8%
2025  198            19               46                  9.6%               23.2%
2026  105            5                24                  4.8%               22.9%

## 6. OTHER SCHEMA RECRUITMENT CHANNELS, OVER TIME

Window      HS   university-pool  professional-network  social-media  snowball  in-person  panel-company
----------  ---  ---------------  --------------------  ------------  --------  ---------  -------------
2010–2013   84   13               6                     3             0         6          3
2014–2017   155  31               21                    9             6         10         8
2018–2021   329  41               40                    24            10        22         9
2022–2024   486  53               76                    56            44        19         12
2025–2026*  303  36               46                    27            31        11         10

## 7. SAMPLE SIZE n (paper-level maximum across tuples)

n stated at paper level: 1284 of 1357 (94.6%). Unstated: 73.
min 1, p25 20, median 43, p75 210, p90 820, p99 13000, max 253238367.

Bucket (paper-level max n)  Papers  Share of 1284 with n stated
--------------------------  ------  ---------------------------
1–19                        319     24.8%
20–49                       340     26.5%
50–99                       141     11.0%
100–499                     295     23.0%
500–999                     84      6.5%
1,000–9,999                 87      6.8%
≥10,000                     18      1.4%

Under 50: 659 of 1284 (51.3%).

LARGE_N map covers 18 papers with n>=10000. Guard: map and set agree.
Role              Papers  What it is
----------------  ------  ------------------------------------------------------------
in-product        7       Existing users of Facebook/Chrome/a live site, not recruited
donation          4       Opt-in telemetry, extension, or data-donation panel
panel             3       Commissioned survey panel (Kantar, Cint, ISP customers)
university-field  1       A captive institutional population (phishing sim, campus)
recruited-crowd   3       Actually recruited on MTurk or Prolific at this scale

The 18:
  253238367  in-product  WWW/2012/the-role-of-social-networks-in-information-diffusion  Random sample of Facebook users who visited Aug–Oct 2010. Not recruited.
  697212  in-product  USENIX/2021/driving-2fa-adoption-at-scale-optimizing-two-factor-authentication-notification  Facebook 2FA prompt experiment on existing US users.
  288895  donation  USENIX/2025/i-can-tell-your-secrets-inferring-privacy-attributes-from-mini-app-interaction-h  AliPay users who agreed to share mini-app history; plus 31 vendor interviews.
  85535  in-product  WWW/2022/what-does-perception-bias-on-social-networks-tell-us-about-friend-count-satisfac  Facebook in-product survey on friend-count satisfaction.
  82890  in-product  CCS/2019/un-informed-consent-studying-gdpr-consent-notices-in-the-field  Real visitors of a German e-commerce site; 110-person voluntary survey on top.
  52000  donation  WWW/2020/the-representativeness-of-automated-web-crawls-as-a-surrogate-for-human-browsing  Firefox Pioneer / opt-in telemetry, 52k en-US users.
  50000  in-product  CCS/2014/increasing-security-sensitivity-with-social-proof-a-large-scale-experimental-con  Facebook announcement experiment, US English users; 2,814 survey on top.
  50000  panel  IEEE-SP/2021/sok-hate-harassment-and-the-changing-landscape-of-online-abuse  Opt-in panels, stratified by country/age/gender, 22 countries.
  29809  university-field  USENIX/2025/phishing-attacks-against-password-manager-browser-extensions  Deceptive phishing simulation of a university population; 3,547 survey.
  22722  donation  WWW/2026/does-this-button-work-investigating-youtubes-ineffective-user-controls  Crowdsourced browser-extension panel, 567M recommendations.
  20895  panel  CCS/2018/predicting-impending-exposure-to-malicious-content-from-user-behavior  Invited 600,000 eligible cellular customers in Japan; raffle.
  16829  in-product  USENIX/2024/sok-or-solk-on-the-quantitative-study-of-sociodemographic-factors-and-computer-s  Facebook in-feed recruitment, 16 countries. The SoK is the reason to read it.
  13109  in-product  NDSS/2024/dont-interrupt-me-a-large-scale-study-of-on-device-permission-prompt-quieting-in-chrome  Chrome in-product surveys; telemetry of 20M permission prompts is separate.
  13000  donation  CCS/2014/private-by-design-advertising-meets-the-real-world  Privad Firefox prototype, opt-in.
  12461  recruited-crowd  PETS/2025/how-unique-is-whose-web-browser-the-role-of-demographics-in-browser-fingerprinti  Prolific, US, gender-balanced, $0.60 / ~2 min. Consented attributes from 8,400.
  12351  panel  USENIX/2025/digital-security-perceptions-and-practices-around-the-world-a-weird-versus-non-w  Kantar online survey, 12 countries, WEIRD vs non-WEIRD.
  12000  recruited-crowd  IEEE-SP/2012/guess-again-and-again-and-again-measuring-password-strength-by-simulating-passwo  Mechanical Turk password study, 2010–2011, 25–70 cents. Plus 280 university.
  11145  recruited-crowd  WWW/2021/an-experimental-study-to-understand-user-experience-and-perception-bias-occurred  MTurk, US young adults, plus 10 interviews.

Genuinely recruited crowd samples with n>=10,000: 3 (Kelley/Bonneau MTurk 2012; a 2021 MTurk fact-check study; Berke et al. Prolific 2025).
Median n excluding the 18 giants: 41 (N=1266). Including them the median is 43 because they sit in the tail.

## 8. COMPENSATION (population: humanSubjects, N=1357)

Compensation string present on any tuple: 746 of 1357 (55.0%).
Compensation kind  Papers  Share of 1357
-----------------  ------  -------------
unpaid             84      6.2%
course-credit      6       0.4%
raffle             35      2.6%
hourly             57      4.2%
gift-card          149     11.0%
stated-amount      371     27.3%
Unmapped compensation strings: 139 tuples. Sentinel written into the field: 1 tuples. Papers with no compensation string at all: 611.

## 9. REGIONS (folded through geo.mjs, population: humanSubjects)

At least one non-sentinel region: 665 of 1357 (49.0%).

Countries (a paper naming two countries counts in both rows; shares of humanSubjects):
Country         Papers  Share of 1357
--------------  ------  -------------
United States   416     30.7%
Germany         104     7.7%
United Kingdom  89      6.6%
China           55      4.1%
India           46      3.4%
Canada          26      1.9%
Australia       25      1.8%
Netherlands     22      1.6%
Switzerland     22      1.6%
Pakistan        21      1.5%
France          20      1.5%
Italy           19      1.4%
Spain           14      1.0%
Austria         13      1.0%
Japan           13      1.0%
Brazil          11      0.8%
Poland          11      0.8%
Turkey          11      0.8%
Kenya           10      0.7%
South Africa    9       0.7%

United States (folded): 416 of 1357 (30.7%). Of the 665 that state a region: 62.6%.
Region-level strings (Europe, Asia, …): North America 28; Europe 61; Asia 30; Africa 18; Middle East 3; Americas 2; South America 14; Oceania 11.
Multi-country / worldwide claims: 28 papers.
Unmapped region strings: 64. Distinct unmapped: 45.

## 10. ETHICS REVIEW (same predicates as report_ethics.mjs, population: humanSubjects)

States a review outcome: 1022 of 1357 (75.3%).
Approved or exempt: 896 of 1357 (66.0%).
Silent (none-mentioned 277 + no record 58): 335 (24.7%).
These three numbers must match practices:ethics. If they move, that page moved too.

Window      humanSubjects  states review  %
----------  -------------  -------------  -----
2010–2013   84             24             28.6%
2014–2017   155            91             58.7%
2018–2021   329            230            69.9%
2022–2024   486            414            85.2%
2025–2026*  303            263            86.8%

## 11. HYBRID: crawled AND recruited (N=149)

149 of 1357 humanSubjects also crawled (11.0%); 149 of 1120 crawled papers also recruited (13.3%).
studyKind on the hybrid slice:
studyKind       Papers  Share of 149
--------------  ------  ------------
survey          57      38.3%
field-study     34      22.8%
lab-study       29      19.5%
interview       23      15.4%
usability-test  19      12.8%
other           11      7.4%
focus-group     1       0.7%

## 12. PLATFORMS MEASURED (population: humanSubjects)

platforms[]           Papers  Share of 1357
--------------------  ------  -------------
offline               496     36.6%
other-online-service  458     33.8%
web                   393     29.0%
mobile                333     24.5%
iot                   117     8.6%
offline (lab/interview) is the modal platform. This page is not only web-user studies; it is user studies in these seven venues.

## 13. POSTERS AND SHORT RECORDS

isPoster (slug poster- OR title Poster: OR pages<=4): 35 of 1357.
pages<=4 alone: 35. poster- slug or Poster: title: 14.
Dropping posters/short: humanSubjects 1357 → 1322. Schema MTurk 139 → 138. The recruitment ranking does not reorder.

## 14. PER-PAPER FIGURES USED ON THE PAGE


PETS/2025/how-unique-is-whose-web-browser-the-role-of-demographics-in-browser-fingerprinti
  title: How Unique is Whose Web Browser? The role of demographics in browser fingerprinting among US users
  studyTypes: user-study, interview-or-survey, mobile-app-analysis
  humanSubjects: true
  PART survey/prolific/n=12461/comp="$0.60 for an estimated 2 minute survey"/regions=["United States"]
    quote: "Participants were recruited via the online platform Prolific with a male/female gender balance and were offered $0.60 for an estimated 2 minute survey."

IEEE-SP/2012/guess-again-and-again-and-again-measuring-password-strength-by-simulating-passwo
  title: Guess Again (and Again and Again): Measuring Password Strength by Simulating Password-Cracking Algorithms.
  studyTypes: user-study, existing-dataset-analysis
  humanSubjects: true
  PART survey/mechanical-turk/n=12000/comp="25–55 cents for the first part and 50–70 cents for the second part"/regions=[]
    quote: "From August 2010 to January 2011, we advertised a two-part study on Mechanical Turk, paying between 25 and 55 cents for the first part and between 50 and 70 cents for the second part."
  PART survey/university-pool/n=280/comp=null/regions=[]
    quote: "To get real-world measures of password-related behavior, we surveyed users of Carnegie Mellon University's email system"

USENIX/2020/a-comprehensive-quality-evaluation-of-security-and-privacy-advice-on-the-web
  title: A Comprehensive Quality Evaluation of Security and Privacy Advice on the Web
  studyTypes: automated-web-crawl, user-study, interview-or-survey, manual-audit
  humanSubjects: true
  PART survey/mechanical-turk/n=50/comp="$0.25 or $0.50"/regions=[]
    quote: "We recruited 50 participants from Amazon Mechanical Turk (AMT) to write search queries for security advice."
  PART survey/panel-company/n=1586/comp="In accordance with their agreement with Cint"/regions=["United States"]
    quote: "We recruited a total of 1,586 users in June 2019 to evaluate the actionability and comprehensibility of our security advice."
  PART survey/professional-network/n=41/comp="$1 for each piece of advice they evaluated"/regions=[]
    quote: "Ultimately, 41 qualified experts evaluated our security advice."

WWW/2020/the-representativeness-of-automated-web-crawls-as-a-surrogate-for-human-browsing
  title: The Representativeness of Automated Web Crawls as a Surrogate for Human Browsing.
  studyTypes: automated-web-crawl, user-study, existing-dataset-analysis
  humanSubjects: true
  PART field-study/not-stated/n=52000/comp=null/regions=[]
    quote: "Approximately 52,000 users participated, all of whom were using Firefox versions 67 or 68 in the en-US localization at the time of data collection."

USENIX/2024/sok-or-solk-on-the-quantitative-study-of-sociodemographic-factors-and-computer-s
  title: SoK (or SoLK?): On the Quantitative Study of Sociodemographic Factors and Computer Security Behaviors
  studyTypes: existing-dataset-analysis, manual-audit
  humanSubjects: true
  PART survey/not-stated/n=16829/comp=null/regions=["16 countries"]
    quote: "Respondents were recruited through both web and mobile interfaces via a message at the top of their social media feeds."

IEEE-SP/2025/transparency-in-usable-privacy-and-security-research-scholars-perspectives-pract
  title: Transparency in Usable Privacy and Security Research: Scholars' Perspectives, Practices, and Recommendations.
  studyTypes: interview-or-survey, manual-audit, existing-dataset-analysis
  humanSubjects: true
  PART interview/professional-network/n=24/comp="$25 or equivalent in local currency; optional, accepted by 16 participants"/regions=["USA","Germany","UK"]
    quote: "We offered participants a compensation of $25 (or equivalent in local currency) as an Amazon voucher or PayPal payment. As recent studies with researchers had no consistent compensation approach, we decided to make the compensation optional."

WWW/2020/attention-please-your-attention-check-questions-in-survey-studies-can-be-automat
  title: Attention Please: Your Attention Check Questions in Survey Studies Can Be Automatically Answered.
  studyTypes: existing-dataset-analysis, manual-audit, system-or-defence-proposal
  humanSubjects: false

USENIX/2025/digital-security-perceptions-and-practices-around-the-world-a-weird-versus-non-w
  title: Digital Security Perceptions and Practices Around the World: A WEIRD versus Non-WEIRD Comparison
  studyTypes: user-study, interview-or-survey
  humanSubjects: true
  PART survey/panel-company/n=12351/comp="Approximately A C2.51–A C5.25 depending on country; exact rates undisclosed"/regions=["Germany","Israel","Italy","Poland","Sweden","United Kingdom","United States","China","India","Mexico","Saudi Arabia","South Africa"]
    quote: "To collect responses from people living in the 12 countries, we commissioned Kantar, a reputable full-service provider of online surveys."

## Z. EXTERNAL FIGURES (not from the corpus; sourced on the page)

These must appear in the report so check_page_numbers.mjs can see them, and each has a primary source on the page.
  2026-09-30  MTurk permanent close date, mturk.com/acceptable-use-policy, read 2026-08-27
  2026-07-30  MTurk closed to new customers (requester.mturk.com banner), still on that page 2026-08-27
  21  years MTurk operated (2005–2026), Amazon launch year + close year
  2005  MTurk launch year
  6  Prolific absolute minimum £6 / hour, researcher-help.prolific.com, read 2026-08-27
  8  Prolific absolute minimum $8 / hour, same page
  9  Prolific recommended £9 / hour
  12  Prolific recommended $12 / hour
  33.3  Prolific academic platform fee percent
  42.8  Prolific corporate platform fee percent
  24  Klemmer et al. interviews (IEEE S&P 2025) — also in the extraction as n=24
  151  Wei et al. SoK papers reviewed (USENIX 2024) — paper figure, extraction n=16829 is the Facebook data
  47  Wei et al. focused-on subset
  16  Wei et al. countries
  374  Redmiles et al. unique recommended behaviors
  1264  Redmiles et al. advice documents
  1586  Redmiles et al. user evaluators
  41  Redmiles et al. expert evaluators
  8400  Berke et al. consented browser-attribute subset of 12461
  0.60  Berke et al. Prolific reward USD
  2  Berke et al. estimated minutes
  25  Kelley et al. lower-end cents, first part
  55  Kelley et al. upper-end cents, first part
  50  Kelley et al. lower-end cents, second part
  70  Kelley et al. upper-end cents, second part
  280  Kelley et al. university-pool follow-up
  52000  Zeber et al. Firefox users (extraction n=52000)
  30  Zeber million-visit figure is in the summary, not n
  67  Zeber Firefox version 67
  68  Zeber Firefox version 68
  70.00  Prolific pricing-page example: $70.00 participant rewards
  100  Prolific pricing-page example: $100 corporate total
  21.3  practices:ethics crawlOnly states-review share; must match that page
  4322  previous corpus size, named as stale
  965  stale item-brief humanSubjects count (4,322-paper corpus)
  53.0  stale item-brief compensation stated rate
  47.8  stale item-brief regions stated rate
  CHI  venue absent from the corpus
  SOUPS  venue absent from the corpus
  6.00  Prolific absolute minimum £6.00, pricing page
  8.00  Prolific absolute minimum $8.00, pricing page
  45.0  complement of compensation stated 55.0
  51.0  complement of region stated 49.0

</file>

===== 5. Folds and residue =====

==== CROWD_FAMILIES ====

Ordered regexes in ''scripts/us_fold.mjs''. Multi-label. Qualtrics is deliberately not a family (survey tool; 124 of 1,357 full-text hits).

Schema is a **lower bound**; quote+fulltext is an **upper bound** (related-work citations fire). Schema MTurk 139 vs recovered 279 (49.8%). Schema Prolific 61 vs recovered 194 (31.4%).

**Unnamed other-crowd-platform residue (25 of 193).** These papers have recruitment ''other-crowd-platform'' and match no CROWD_FAMILIES regex in the quote or in ''paper.cols.txt'':

<code>
WWW/2013/your-browsing-behavior-for-a-big-mac-economics-of-personal-information-online
CCS/2016/build-it-break-it-fix-it-contesting-secure-development
IEEE-SP/2016/i-think-theyre-trying-to-tell-me-something-advice-sources-and-selection-for-digi
IEEE-SP/2019/tap-n-ghost-a-compilation-of-novel-attack-techniques-against-smartphone-touchscr
NDSS/2019/understanding-open-ports-in-android-applications-discovery-diagnosis-and-security-assessment
WWW/2019/message-distortion-in-information-cascades
WWW/2019/quality-sensitive-training-social-advertisement-generation-by-leveraging-user-cl
WWW/2020/dynamic-composition-for-conversational-domain-exploration
PETS/2021/unveiling-web-fingerprinting-in-the-wild-via-code-mining-and-machine-learning
PETS/2021/digital-inequality-through-the-lens-of-self-disclosure
WWW/2021/short-is-the-road-that-leads-from-fear-to-hate-fear-speech-in-indian-whatsapp-gr
IEEE-SP/2020/ask-the-experts-what-should-be-on-an-iot-privacy-and-security-label
USENIX/2022/how-do-you-not-lose-friends-synthesizing-a-design-space-of-social-controls-for-s
IMC/2023/estimating-webrtc-video-qoe-metrics-without-using-application-headers
IMC/2023/rovista-measuring-and-analyzing-the-route-origin-validation-rov-in-rpki
NDSS/2023/browser-permission-mechanisms-demystified
WWW/2023/multiview-representation-learning-from-crowdsourced-triplet-comparisons
USENIX/2023/multiview-finding-blind-spots-in-access-deny-issues-diagnosis
WWW/2023/visual-aware-testing-and-debugging-for-web-performance-optimization
PETS/2024/tailoring-digital-privacy-education-interventions-for-older-adults-a-comparative
IEEE-SP/2024/shedding-light-on-cvss-scoring-inconsistencies-a-user-centric-study-on-evaluatin
USENIX/2024/understanding-the-security-and-privacy-implications-of-online-toxic-content-on-r
IEEE-SP/2025/sorry-for-bugging-you-so-much-exploring-developers-behavior-towards-privacy-comp
IEEE-SP/2026/searching-for-a-farang-collective-security-among-women-in-pattaya-thailand
PETS/2025/my-data-or-our-data-a-comparative-study-of-collaborative-family-apps-and-parents
</code>

==== COMP_FAMILIES ====

Single-label, ordered unpaid → course-credit → raffle → hourly → gift-card → stated-amount. Unmapped compensation strings: **139 tuples**. Sentinel written into the field: 1 tuple. Papers with no compensation string: 611. Pass ''--residue'' after the crowd list; compensation residue is printed when that flag is on (same flag, later section). Not listed here in full — 139 free-text strings, reproducible.

==== LARGE_N ====

Every paper with paper-level max n ≥ 10,000, with a role, keyed by ''venue/year/slug''. The report **exits 1** if the map and the set diverge. 18 papers, 5 roles (in-product 7, donation 4, panel 3, university-field 1, recruited-crowd 3). Facebook 253238367 is in-product, not recruited.

==== isPoster ====

''slug'' starts with ''poster-'' OR title starts ''Poster:'' OR ''pages<=4'' → 35 of 1,357. Slug/title-only is 14. Dropping them does not reorder recruitment.

===== 6. Quote checks =====

Checked 2026-08-27 against ''paper.cols.txt'' (whitespace-normalised). Column splices and footnote markers are recorded, not silently "fixed" in the quote.

^ Paper ^ Quote or needle ^ Verdict ^
| Kelley et al. {[kelley2012_guess]} | paying between 25 and 55 cents … 50 and 70 cents | **Exact in .cols** after "two-part"→"twopart" hyphen splice. Page uses the hyphenated extraction quote, which is the paper's text. |
| Berke et al. {[berke2025_demographics]} | $0.60 for an estimated 2 minute survey; 8,400; 12,461 | **Present.** The recruitment sentence is spliced by a footnote and a section heading (''Prolific4'', ''3.1 Ethics and informed consent''). Numbers 8,400 and 12,461 are intact. |
| Redmiles et al. {[redmiles2020_comprehensive]} | 50 AMT participants; 1,586 users; 41 qualified experts; 374 unique; 1,264 documents | **Exact** for the three recruitment quotes. 374 / 1,264 / 1,586 are present and column-spliced in the abstract. |
| Zeber et al. {[zeber2020representativeness]} | Approximately 52,000 users … Firefox versions 67 or 68 in the en-US localization | **Exact.** Also "30 million site visits across 50,000 users" in the same paper — the extraction n is 52,000; the page quotes that, and separately names 30 million visits. |
| Wei et al. {[wei2024_solk]} | 151 papers; 47 papers; 16,829 users … 16 countries | **Exact.** |
| Klemmer et al. {[klemmer2025_transparency]} | $25 (or equivalent in local currency) | **Exact.** |
| Herbert et al. {[herbert2025_digital]} | commissioned Kantar … 12 countries | **Exact.** |
| Utz et al. {[utz2019_informed]} | 82,890 real website visitors of a German e-commerce website; 110 responses | **Exact.** |
| Pei et al. {[pei2020_attention]} | leverage deep learning techniques to pass attention check questions automatically | **Exact** (abstract). Paper is **not** humanSubjects. |

Redmiles authors: ''out/authors.json'' had injected **"San Diego"** as an author (Sean Kross / UC San Diego affiliation). Corrected from the USENIX landing page people-field before bibgen: Elissa M. Redmiles, Noel Warford, Amritha Jayanti, Aravind Koneru, Sean Kross, Miraida Morales, Rock Stevens, Michelle L. Mazurek. Landing: [[https://www.usenix.org/conference/usenixsecurity20/presentation/redmiles]].

Herbert authors from [[https://www.usenix.org/conference/usenixsecurity25/presentation/herbert]]: Franziska Herbert, Collins W. Munyendo, Jonas Hielscher, Steffen Becker, Yixin Zou. bibgen matched.

===== 7. External sources =====

==== Verified, 2026-08-27 ====

^ Claim ^ Primary source ^ How ^
| MTurk permanently closes 30 September 2026 | [[https://www.mturk.com/acceptable-use-policy]] | ''curl'' + browser UA; needles "September 30, 2026" and "permanently close" present. Script exits 1 if they vanish. |
| MTurk no longer accepting new customers | [[https://requester.mturk.com/]] | Same, needle present. In force since 30 July 2026 (banner still there). |
| Prolific absolute min £6 / $8, recommended £9 / $12 | [[https://researcher-help.prolific.com/en/articles/445266-how-much-should-i-pay-participants]] | Present in Intercom HTML. |
| Prolific £6.00 / $8.00, academic fee 33.3% of participant rewards, corporate 42.8% | [[https://researcher-help.prolific.com/en/articles/445239-what-is-your-pricing]] | Needles must be the **phrase** "33.3% of participant rewards", not "33.3" — a CSS gradient contains "333.38%". |
| CloudResearch exists | [[https://www.cloudresearch.com/]] | HTTP 200, title "CloudResearch: Survey Platform & Participant Recruitment". |

''pages/platform_status.py'' output, same sitting (exit 0):

<code>
ok      mturk-aup  https://www.mturk.com/acceptable-use-policy
         'September 30, 2026' -> 'er-headline">Mechanical Turk will close September 30, 2026.</div>\n      <div class="maintenance-banner-body">We regularly evaluate our programs, tools, and services and make adjust'
         'permanently close' -> 'r 30, 2026. Amazon Mechanical Turk will permanently close on September 30, 2026. For Workers and Requesters currently using the service, please visit our <a href="/help">FAQ</a> pa'
ok      mturk-requester  https://requester.mturk.com/
         'no longer accepting new customers' -> '9888;&#xFE0F; Amazon Mechanical Turk is no longer accepting new customers. We recommend existing customers migrate to a third-party solution. More information is available <a href='
ok      prolific-pricing  https://researcher-help.prolific.com/en/articles/445239-what-is-your-pricing
         '£6.00 / $8.00' -> 'ft"><p><b>Absolute minimum allowed:</b> £6.00 / $8.00 per hour</p></div></li></ul></div><div class="intercom-interblocks-paragraph no-margin intercom-interblocks-align-left"><p>You'
         '33.3% of participant rewards' -> 'rticipant rewards (corporate customers) 33.3% of participant rewards (academia and non-profits) Example: If you pay $70.00 to participants, your total cost will be $100 (corporate)'
         '42.8% of participant rewards' -> 'ercentage on top of participant rewards 42.8% of participant rewards (corporate customers) 33.3% of participant rewards (academia and non-profits) Example: If you pay $70.00 to par'
ok      prolific-pay  https://researcher-help.prolific.com/en/articles/445266-how-much-should-i-pay-participants
         '£6' -> '><p>We enforce a minimum hourly rate of £6 / $8, and recommend paying at least £9 / $12 per hour. However, the right reward for your study depends on several factors, including the'
         '$8' -> 'e enforce a minimum hourly rate of £6 / $8, and recommend paying at least £9 / $12 per hour. However, the right reward for your study depends on several factors, including the effo'
         '£9' -> ' £6 / $8, and recommend paying at least £9 / $12 per hour. However, the right reward for your study depends on several factors, including the effort required, the participants you’'
         '$12' -> ' $8, and recommend paying at least £9 / $12 per hour. However, the right reward for your study depends on several factors, including the effort required, the participants you’re ta'
ok      cloudresearch  https://www.cloudresearch.com/
         'CloudResearch' -> 'application/xml" title="Sitemap"><title>CloudResearch: Survey Platform & Participant Recruitment</title><link rel="canonical" href="https://www.cloudresearch.com/"><meta name="desc'
ok      5/5 checks
</code>

==== Rejected ====

  * **A regex needle containing ''$''.** ''re.search("$8", text)'' is end-of-string, not a dollar sign. First platform_status run reported Prolific missing $8 while the HTML contained "£6.00 / $8.00". Search is now a literal ''in''.
  * **Needle "33.3" on the pricing page.** Hits a CSS ''radial-gradient(333.38%…)''. Replaced with "33.3% of participant rewards".
  * **MTurk homepage marketing copy** as a source for the close date. The homepage still sells the product; the AUP is the primary source. The page does not cite the homepage.
  * **"MTurk alternatives 2026" listicles.** Not used. CloudResearch is named only because the corpus recovers it (10 papers) and the homepage is live.
  * **PETS Berke landing for a new bib key.** Parser failed. Reused existing ''berke2025_demographics''.
  * **''fetch_authors.py'' for Redmiles without reading the landing page.** It invented the author "San Diego". Corrected by hand.
  * **Carrying 965 / 53.0% / 47.8% from the item brief.** Those are the 4,322-paper numbers. Re-derived: 1,357 / 55.0% / 49.0%.
  * **Unioning ''user-study'' type with ''participants[]''.** Disagrees both ways.
  * **Copying Kelley 2010–2011 25–70¢ as current pay.** Historical, and below Prolific's 2026 floor.
  * **Calling Kelley 2012 a Kelley/Bonneau paper.** Bonneau is not an author. An earlier LARGE_N note said "Kelley/Bonneau"; that comment was corrected in the report's Z block. The paper is Kelley, Komanduri, Mazurek, Shay, Vidas, Bauer, Christin, Cranor, López.

==== What could not be established ====

  * Whether CloudResearch is **the** MTurk successor. They do not say that on the homepage this sitting fetched. The page says "a named MTurk-adjacent alternative" and does not certify a migration path.
  * CHI / SOUPS / CSCW recruitment rates. Those venues are not in the corpus. Said loudly, not estimated.
  * A 2026-complete year trend. 2026 is provisional; the recovered Prolific-over-MTurk crossover is dated from **2024**, which is a complete year, with 2025–2026 as supporting and labelled provisional.
  * Fair-pay rates for university pools and professional networks. No primary source analogous to Prolific's published floor.
  * Mozilla Rally's shutdown **announcement** (ethics page already recorded there is none). This page only restates that Rally is dead, pointing at [[practices:ethics]].

===== 8. Judgement calls =====

  * Page population is ''participants[]'', not the study-type tag, because the type-only 24 include crowd annotation and crawls.
  * Recovered crowd platforms are printed **beside** the schema, never instead of it, because full-text is an upper bound.
  * n≥10,000 is a hand map of roles, not a "large survey" count, because 7 of 18 are in-product.
  * Ethics 75.3 / 66.0 / 24.7 are copied from the same predicates as ''report_ethics.mjs'' and must stay equal to [[practices:ethics]]. This page does not re-explain venue ethics bodies.
  * The live script is part of the page because the MTurk close date will otherwise rot inside prose.

===== 9. The run =====

^ Item ^ Value ^
| Date | 2026-08-27 |
| Corpus | 5,859 extracted papers, extract/run1 |
| Item | drain item 166, ''design:user_studies (new)'', run 66 |
| Runner | cursor-drain-user-studies (Cursor, not Claude Code / drain-sandbox.sh) |
| Reviewers | three focused + one generic, all **GPT 5.6 Luna medium** (''gpt-5.6-luna-medium''), not the fable/sonnet split in tasks.md |

===== 10. Review log =====

Frozen drafts: ''out/freeze_user_studies/''. Reviewers were handed the frozen page, the report script and its output, and this provenance page. They were told the author's context may not be exhaustive. The content page was not edited while they ran. All four passes: **GPT 5.6 Luna medium** (''gpt-5.6-luna-medium'').

==== Focused pass 1 — figures vs script ====

^ # ^ Sev ^ Finding ^ Decision ^
| 1 | MEDIUM | Embedded ''platform_status.py'' snippets had been trimmed (requester emoji entity stripped; Prolific ''$8''/''£9'' snippets truncated differently from a fresh run). Needles still passed 5/5. | **Accepted.** Replaced the %%<code>%% block with the script's stdout from a fresh 2026-08-27 run. Added ''9888'' to the report Z block so the HTML entity in the requester snippet does not fail ''check_page_numbers''. |

No other figure, complement, ethics, schema/recovered, or LARGE_N mismatch.

==== Focused pass 2 — citations and quotes ====

No findings. All eleven citekeys resolve; table-row attributions match first authors; quoted sentences located in ''paper.cols.txt'' (with recorded splices for Kelley hyphen and Berke footnote).

==== Focused pass 3 — external currency ====

No findings. Re-fetched MTurk AUP, requester banner, both Prolific help articles, CloudResearch homepage, both USENIX landings (HTTP 200), Rally still archived. Fresh ''platform_status.py'' exit 0, 5/5. Disagreed with pass 1 only on whether snippet drift counted: pass 3 treated 5/5 success as enough; pass 1 wanted byte-identical stdout. Pass 1's bar is the one applied.

==== Generic pass — no checklist ====

^ # ^ Sev ^ Finding ^ Decision ^
| 1 | MEDIUM | ''humanSubjects'' reads as adjudicated human-subjects research rather than an extraction signal. | **Accepted in part.** One sentence added: it is ''participants[]'' non-empty, not a hand-adjudicated IRB determination, and n≥10,000 includes visitors and donated panels. Not renamed throughout — the name is OVERVIEW.md's. |
| 2 | HIGH | "One human-subjects paper in three does not name how people were recruited" overstates: could be extraction failure. | **Rejected as HIGH, accepted as wording.** Sentinel-only is the site's standard finding for this field. Changed to "in the extraction, one … has no named recruitment channel." The 454/1,357 figure is unchanged. |
| 3 | MEDIUM | 2024 Prolific-over-MTurk crossover is full-text upper bound presented as history. | **Accepted.** Now states the crossover is on the recovered-full-text series; quote-only already has Prolific 102 vs MTurk 92 overall; schema-only is later. |
| 4 | MEDIUM | "venues will reject on ethics with the technical review untouched" is an absolute policy claim. | **Accepted.** Replaced with a pointer to [[practices:ethics]] and that page's "six of the seven" caveat. |
| 5 | LOW | Qualtrics 124 presented more firmly than crowd-platform recovery. | **Accepted.** Now "full-text hits, upper bound". |
| 6 | LOW | "Preregistration already normal on this side" is uncited here. | **Accepted.** Softened to a pointer: counts live on [[statistics:study_preregistration]], not re-derived here. |