| Next revision | Previous revision |
| provenance:design:user_studies [2026/08/27 14:18] – Provenance log for design:user_studies. Authored by Claude. karel.kubicek.claude | provenance:design:user_studies [2026/08/27 14:26] (current) – Log generic GPT 5.6 Luna medium pass accept/reject. Authored by Claude. karel.kubicek.claude |
|---|
| ===== 10. Review log ===== | ===== 10. Review log ===== |
| |
| Frozen drafts: ''out/freeze_user_studies/''. Reviewers were handed the frozen page, the report script and its output, and this provenance page. They were told the author's context may not be exhaustive. The content page was not edited while they ran. | Frozen drafts: ''out/freeze_user_studies/''. Reviewers were handed the frozen page, the report script and its output, and this provenance page. They were told the author's context may not be exhaustive. The content page was not edited while they ran. All four passes: **GPT 5.6 Luna medium** (''gpt-5.6-luna-medium''). |
| |
| (Filled after the four passes.) | ==== Focused pass 1 — figures vs script ==== |
| | |
| | ^ # ^ Sev ^ Finding ^ Decision ^ |
| | | 1 | MEDIUM | Embedded ''platform_status.py'' snippets had been trimmed (requester emoji entity stripped; Prolific ''$8''/''£9'' snippets truncated differently from a fresh run). Needles still passed 5/5. | **Accepted.** Replaced the %%<code>%% block with the script's stdout from a fresh 2026-08-27 run. Added ''9888'' to the report Z block so the HTML entity in the requester snippet does not fail ''check_page_numbers''. | |
| | |
| | No other figure, complement, ethics, schema/recovered, or LARGE_N mismatch. |
| | |
| | ==== Focused pass 2 — citations and quotes ==== |
| | |
| | No findings. All eleven citekeys resolve; table-row attributions match first authors; quoted sentences located in ''paper.cols.txt'' (with recorded splices for Kelley hyphen and Berke footnote). |
| | |
| | ==== Focused pass 3 — external currency ==== |
| | |
| | No findings. Re-fetched MTurk AUP, requester banner, both Prolific help articles, CloudResearch homepage, both USENIX landings (HTTP 200), Rally still archived. Fresh ''platform_status.py'' exit 0, 5/5. Disagreed with pass 1 only on whether snippet drift counted: pass 3 treated 5/5 success as enough; pass 1 wanted byte-identical stdout. Pass 1's bar is the one applied. |
| | |
| | ==== Generic pass — no checklist ==== |
| | |
| | ^ # ^ Sev ^ Finding ^ Decision ^ |
| | | 1 | MEDIUM | ''humanSubjects'' reads as adjudicated human-subjects research rather than an extraction signal. | **Accepted in part.** One sentence added: it is ''participants[]'' non-empty, not a hand-adjudicated IRB determination, and n≥10,000 includes visitors and donated panels. Not renamed throughout — the name is OVERVIEW.md's. | |
| | | 2 | HIGH | "One human-subjects paper in three does not name how people were recruited" overstates: could be extraction failure. | **Rejected as HIGH, accepted as wording.** Sentinel-only is the site's standard finding for this field. Changed to "in the extraction, one … has no named recruitment channel." The 454/1,357 figure is unchanged. | |
| | | 3 | MEDIUM | 2024 Prolific-over-MTurk crossover is full-text upper bound presented as history. | **Accepted.** Now states the crossover is on the recovered-full-text series; quote-only already has Prolific 102 vs MTurk 92 overall; schema-only is later. | |
| | | 4 | MEDIUM | "venues will reject on ethics with the technical review untouched" is an absolute policy claim. | **Accepted.** Replaced with a pointer to [[practices:ethics]] and that page's "six of the seven" caveat. | |
| | | 5 | LOW | Qualtrics 124 presented more firmly than crowd-platform recovery. | **Accepted.** Now "full-text hits, upper bound". | |
| | | 6 | LOW | "Preregistration already normal on this side" is uncited here. | **Accepted.** Softened to a pointer: counts live on [[statistics:study_preregistration]], not re-derived here. | |
| |
