Measure The Web

User Tools

Site Tools

Trace:
design:user_studies

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
design:user_studies [2026/08/27 14:23] – Refresh platform_status stdout to match a fresh 5/5 run. Authored by Claude. karel.kubicek.claudedesign:user_studies [2026/08/27 14:26] (current) – Apply generic-review wording: extraction proxy, recovered-crossover bound, ethics pointer, Qualtrics upper bound. Authored by Claude. karel.kubicek.claude
Line 39: Line 39:
 | annotatorType ''crowdworkers'' | crowdworkers as **labelers** | 73 | 1.2% | | annotatorType ''crowdworkers'' | crowdworkers as **labelers** | 73 | 1.2% |
  
-**Every figure below is on ''humanSubjects'' (N=1,357) unless it says otherwise.**+**Every figure below is on ''humanSubjects'' (N=1,357) unless it says otherwise.** That is an extraction signal — ''participants[]'' non-empty — not a hand-adjudicated IRB determination. A paper can have participants the extractor missed, and a ''participants[]'' tuple can be a visitor log or a donated panel rather than a recruited sample (see n≥10,000 below).
  
 The type tag and the participants array disagree in both directions: **1,125** papers have both, **232** have ''participants[]'' but no ''user-study'' type, **24** have the type and an empty ''participants[]''. The type-only papers are crawls, telemetry, or crowd //annotation// the extractor filed as a study type. Unioning the two would invent a population that is neither. The item brief's "965 papers" was computed on the old 4,322-paper corpus and is stale; the current count is **1,357**. The type tag and the participants array disagree in both directions: **1,125** papers have both, **232** have ''participants[]'' but no ''user-study'' type, **24** have the type and an empty ''participants[]''. The type-only papers are crawls, telemetry, or crowd //annotation// the extractor filed as a study type. Unioning the two would invent a population that is neither. The item brief's "965 papers" was computed on the old 4,322-paper corpus and is stale; the current count is **1,357**.
Line 70: Line 70:
 ===== How the Field Recruits ===== ===== How the Field Recruits =====
  
-The schema has a recruitment enum. **903 of 1,357 (66.5%)** state a channel; **454 (33.5%)** produce only a sentinel. Sentinel-only is the finding: one human-subjects paper in three does not name how people were recruited. Sentinels are never counted as a channel.+The schema has a recruitment enum. **903 of 1,357 (66.5%)** state a channel; **454 (33.5%)** produce only a sentinel. Sentinel-only is the finding: in the extraction, one human-subjects paper in three has no named recruitment channel. Sentinels are never counted as a channel.
  
 ^ recruitment (schema enum) ^ Papers ^ Share of 1,357 ^ ^ recruitment (schema enum) ^ Papers ^ Share of 1,357 ^
Line 99: Line 99:
 **Schema is a lower bound. Full-text recovery is an upper bound.** Schema MTurk is 49.8% of the recovered-MTurk set; schema Prolific is 31.4% of recovered-Prolific. Union of recovered MTurk∪Prolific: **409 of 1,357 (30.1%)**; 64 papers hit both. Of the 193 ''other-crowd-platform'' papers, 168 name one of the families above and **25 name nothing recoverable** — listed in full on the provenance page. **Schema is a lower bound. Full-text recovery is an upper bound.** Schema MTurk is 49.8% of the recovered-MTurk set; schema Prolific is 31.4% of recovered-Prolific. Union of recovered MTurk∪Prolific: **409 of 1,357 (30.1%)**; 64 papers hit both. Of the 193 ''other-crowd-platform'' papers, 168 name one of the families above and **25 name nothing recoverable** — listed in full on the provenance page.
  
-Qualtrics full-text hits **124 of 1,357** human-subjects papers. It is the survey **tool**, not a recruitment family, and it is not in the table.+Qualtrics full-text hits **124 of 1,357** human-subjects papers. That is an **upper bound** (the same related-work trap as the crowd-platform recovery). It is the survey **tool**, not a recruitment family, and it is not in the table.
  
 ==== Which platform is current ==== ==== Which platform is current ====
Line 112: Line 112:
 | 2025–2026* | 303 | 7 | 24 | 34 | 70 | | 2025–2026* | 303 | 7 | 24 | 34 | 70 |
  
-Per calendar year, recovered Prolific **overtakes** recovered MTurk in **2024** (39 vs 28), and the gap is wide in the provisional 2025–2026 slice (46 vs 19 in 2025; 24 vs 5 in 2026). Schema-only, the crossover is later: 2025–2026* schema Prolific 34 vs schema MTurk 7. Report both bounds. After 30 September 2026 the MTurk column is historical.+Per calendar year, recovered Prolific **overtakes** recovered MTurk in **2024** (39 vs 28), and the gap is wide in the provisional 2025–2026 slice (46 vs 19 in 2025; 24 vs 5 in 2026). That crossover is on the recovered-full-text series — an **upper bound** on both platforms, because a related-work citation fires. Quote-only recovery already has Prolific ahead overall (102 vs 92). Schema-only, the crossover is later: 2025–2026* schema Prolific 34 vs schema MTurk 7. Report both bounds. After 30 September 2026 the MTurk column is historical.
  
 University-pool, professional-network, social-media, snowball, in-person and panel-company are all still live channels. Professional-network is the second schema enum (189) and grew in 2022–2024 (76 of 486). Panel-company is small in the schema (42) and is how the n≥10,000 **survey** giants actually get people — Kantar {[herbert2025_digital]}, Cint {[redmiles2020_comprehensive]}. University-pool, professional-network, social-media, snowball, in-person and panel-company are all still live channels. Professional-network is the second schema enum (189) and grew in 2022–2024 (76 of 486). Panel-company is small in the schema (42) and is how the n≥10,000 **survey** giants actually get people — Kantar {[herbert2025_digital]}, Cint {[redmiles2020_comprehensive]}.
Line 202: Line 202:
 ===== What to Report ===== ===== What to Report =====
  
-A reviewer reconstructing your sample should not have to guess. Of 1,357 human-subjects papers in these venues, **33.5%** do not name the recruitment channel, **45.0%** do not state compensation, **51.0%** do not state a region. Do not be those papers.+A reviewer reconstructing your sample should not have to guess. Of 1,357 human-subjects papers in these venues, **33.5%** have no named recruitment channel in the extraction, **45.0%** do not state compensation, **51.0%** do not state a region. Do not be those papers.
  
   - **Who was recruited, through what channel, on which platform.** "Online" is a sentinel. Name Prolific, the university pool, Kantar, the in-product surface, or the professional list. If they were visitors of a live site rather than recruits, say that — it changes what n means {[utz2019_informed]}.   - **Who was recruited, through what channel, on which platform.** "Online" is a sentinel. Name Prolific, the university pool, Kantar, the in-product surface, or the professional list. If they were visitors of a live site rather than recruits, say that — it changes what n means {[utz2019_informed]}.
Line 208: Line 208:
   - **What they were paid, in an amount a reader can convert to an hourly rate.** Prolific's absolute floor on 2026-08-27 is £6 / $8 per hour; recommended is £9 / $12. Kelley et al.'s 25–70 cents {[kelley2012_guess]} is a 2010–2011 fact, not a template. If compensation was optional, say how many took it {[klemmer2025_transparency]}.   - **What they were paid, in an amount a reader can convert to an hourly rate.** Prolific's absolute floor on 2026-08-27 is £6 / $8 per hour; recommended is £9 / $12. Kelley et al.'s 25–70 cents {[kelley2012_guess]} is a 2010–2011 fact, not a template. If compensation was optional, say how many took it {[klemmer2025_transparency]}.
   - **Where they were, at least at country level.** 62.6% of papers that state a region name the United States. If your sample is WEIRD, Wei et al. {[wei2024_solk]} is the paper a reviewer will cite back at you. If it is not, Herbert et al. {[herbert2025_digital]} is the current existence proof that a 12-country panel is possible.   - **Where they were, at least at country level.** 62.6% of papers that state a region name the United States. If your sample is WEIRD, Wei et al. {[wei2024_solk]} is the paper a reviewer will cite back at you. If it is not, Herbert et al. {[herbert2025_digital]} is the current existence proof that a 12-country panel is possible.
-  - **The ethics determination**, not a vibe. The venues that carry this literature will reject on ethics with the technical review untouched — see [[Practices:Ethics]].+  - **The ethics determination**, not a vibe. What the seven venues require, and that an ethics problem can stop a paper independently of the technical review, is [[Practices:Ethics]] — including the "six of the seven" caveat for TheWebConf.
   - **Attention checks, and what they actually show.** Pei et al. {[pei2020_attention]} demonstrated they can be passed automatically. Report the check, the fail rate, and that a passed check is not proof of attention.   - **Attention checks, and what they actually show.** Pei et al. {[pei2020_attention]} demonstrated they can be passed automatically. Report the check, the fail rate, and that a passed check is not proof of attention.
   - **If you also crawled: which claims come from the crawl and which from the people.** Zeber et al. {[zeber2020representativeness]} is the reason the two are not interchangeable.   - **If you also crawled: which claims come from the crawl and which from the people.** Zeber et al. {[zeber2020representativeness]} is the reason the two are not interchangeable.
-  - **Preregistration** if you have hypotheses about people — already normal on this side of the house, almost absent on the crawl side. See [[Statistics:Study preregistration]]. Tests, corrections, and clustered/paired designs: [[Statistics:Hypothesis testing]], [[Statistics:Pvalue corrections]]. Open coding: [[Statistics:Interrater agreement]].+  - **Preregistration** if you have hypotheses about people. Almost all of this corpus's preregistrations sit in participant studies rather than crawls — the counts are on [[Statistics:Study preregistration]], not re-derived here. Tests, corrections, and clustered/paired designs: [[Statistics:Hypothesis testing]], [[Statistics:Pvalue corrections]]. Open coding: [[Statistics:Interrater agreement]].
  
 ===== Live check: the platforms this page named ===== ===== Live check: the platforms this page named =====
design/user_studies.txt · Last modified: by karel.kubicek.claude
Except where otherwise noted, content on this wiki is licensed under the following license: CC BY-NC-SA 4.0
CC BY-NC-SA 4.0 Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki