Measure The Web

User Tools

Site Tools

Trace:
practices:ethics

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
practices:ethics [2026/08/18 23:09] – Rewrite from notes stub into a full page: measured reporting rates (the participants-vs-crawls split), whether a crawl needs review and what to do when no board exists, what the seven venues require in 2026 including the generative-AI policies, harm from karel.kubicek.claudepractices:ethics [2026/09/17 10:45] (current) – Clarify consentAction schema statistic; Authored by Claude karel.kubicek.claude
Line 55: Line 55:
 | 2025–2026* | 1,019 | 467 | 45.8% | 303 | 263 | 86.8% | 166 | 50 | 30.1% | | 2025–2026* | 1,019 | 467 | 45.8% | 303 | 263 | 86.8% | 166 | 50 | 30.1% |
  
-<wrap todo>* 2025–2026 is provisional: CCS 2026 and IMC 2026 have not been held, and IEEE S&P 2026 and TheWebConf 2026 abstracts are not in OpenAlex, so those venue-years are under-represented by construction. See [[Literature:Corpus]].</wrap>+<WRAP todo>* 2025–2026 is provisional: CCS 2026 and IMC 2026 have not been held, and IEEE S&P 2026 and TheWebConf 2026 abstracts are not in OpenAlex, so those venue-years are under-represented by construction. See [[Literature:Corpus]].</WRAP>
  
 In the last complete window, 85.2% of participant studies and 28.8% of participant-free crawls state an outcome. Put the two series on the same axis and the lag is measurable: **participant-free crawls reach in 2025–2026 (30.1%) a reporting rate participant studies had already passed before 2014** (28.6% in 2010–2013, 58.7% in 2014–2017). If you are writing a crawl paper, you are in the second column, and the second column is where the reviewers' patience is currently running out. In the last complete window, 85.2% of participant studies and 28.8% of participant-free crawls state an outcome. Put the two series on the same axis and the lag is measurable: **participant-free crawls reach in 2025–2026 (30.1%) a reporting rate participant studies had already passed before 2014** (28.6% in 2010–2013, 58.7% in 2014–2017). If you are writing a crawl paper, you are in the second column, and the second column is where the reviewers' patience is currently running out.
Line 148: Line 148:
 | TheWebConf '26* | Encouraged | Via ACM's human-participants policy | Not named | Not required | | TheWebConf '26* | Encouraged | Via ACM's human-participants policy | Not named | Not required |
  
-<wrap todo>* Six of the seven rows are the currently open cycle. TheWebConf is not: ''www2027.thewebconf.org'' does not resolve and ''thewebconf.org'' lists WWW 2027 (Dublin) with no call for papers yet, so that row is the **last published** cycle, WWW 2026, which was held in April 2026.</wrap>+<WRAP todo>* Six of the seven rows are the currently open cycle. TheWebConf is not: ''www2027.thewebconf.org'' does not resolve and ''thewebconf.org'' lists WWW 2027 (Dublin) with no call for papers yet, so that row is the **last published** cycle, WWW 2026, which was held in April 2026.</WRAP>
  
 Four things to take from it. Four things to take from it.
Line 194: Line 194:
 Two things this means for a measurement paper specifically. If you used an LLM **as an instrument** — to classify cookies, label banners, read privacy policies, decide whether a site is in scope — that is a methods decision and a disclosure, not a writing aid, and IEEE S&P's phrasing ("as an object of study") is the only one that clearly covers it. And if the model touched anything you collected, the ethics section and the AI section now interact: sending crawled content, participant text or vulnerability details to a hosted model is a disclosure of that data to a third party, and none of these policies says so for you. Two things this means for a measurement paper specifically. If you used an LLM **as an instrument** — to classify cookies, label banners, read privacy policies, decide whether a site is in scope — that is a methods decision and a disclosure, not a writing aid, and IEEE S&P's phrasing ("as an object of study") is the only one that clearly covers it. And if the model touched anything you collected, the ethics section and the AI section now interact: sending crawled content, participant text or vulnerability details to a hosted model is a disclosure of that data to a third party, and none of these policies says so for you.
  
-<wrap todo>The corpus cannot yet say how often LLM-based measurement pipelines disclose this: ''tools[]'' has an ''llm'' category, but there is no schema field for "was AI use disclosed", and the venue requirements are newer than most of the corpus. A full-text sweep would be the way in.</wrap>+<WRAP todo>The corpus cannot yet say how often LLM-based measurement pipelines disclose this: ''tools[]'' has an ''llm'' category, but there is no schema field for "was AI use disclosed", and the venue requirements are newer than most of the corpus. A full-text sweep would be the way in.</WRAP>
  
 ===== Harm from crawling, and what the field does about it ===== ===== Harm from crawling, and what the field does about it =====
Line 252: Line 252:
 | ''humanAnnotation.agreementMetric'' | 3,318 papers that coded data by hand | 512 | 15.4% | | ''humanAnnotation.agreementMetric'' | 3,318 papers that coded data by hand | 512 | 15.4% |
 | ''crawlConfig.statefulness'' | 1,120 crawling papers | 219 | 19.6% | | ''crawlConfig.statefulness'' | 1,120 crawling papers | 219 | 19.6% |
-| ''crawlConfig.consentAction'' | 1,120 crawling papers | 349 | 31.2% |+| ''crawlConfig.consentAction'' (field populated; schema, not an audited paper claim) | 1,120 crawling papers | 349 | 31.2% | 
 + 
 +The consent row is a schema-completeness statistic: a populated non-sentinel field does not by itself show that the paper made the corresponding claim. The 2026-09-05 audit found **55 of 1,120 crawling papers (4.9%)** whose full text states a consent action; it also found **279 of 313 (89.1%)** ''no-interaction'' defaults unsupported. The two populations answer different questions.
  
 The reason it is unresolved rather than merely unreported is that the file does not answer the question a researcher has. `robots.txt` is a directive to *automated indexers*; a research crawl that loads a page once, in the way a browser would, and never republishes the content is not obviously the addressee, and a strict reading excludes exactly the pages a measurement is about (a `Disallow: /` on a site's tracking-heavy subpages removes the finding). There is no consensus in this literature, and inventing one here would be dishonest. What is not defensible is silence: **decide, do the same thing throughout the crawl, and write the sentence.** If you honour it, say what share of your sample it removed, because that is a bias in your denominator, not a footnote. If you do not, say why and what you did instead — most commonly a rate limit strict enough that the file's purpose (protecting the server) is served by other means. The reason it is unresolved rather than merely unreported is that the file does not answer the question a researcher has. `robots.txt` is a directive to *automated indexers*; a research crawl that loads a page once, in the way a browser would, and never republishes the content is not obviously the addressee, and a strict reading excludes exactly the pages a measurement is about (a `Disallow: /` on a site's tracking-heavy subpages removes the finding). There is no consensus in this literature, and inventing one here would be dishonest. What is not defensible is silence: **decide, do the same thing throughout the crawl, and write the sentence.** If you honour it, say what share of your sample it removed, because that is a bias in your denominator, not a footnote. If you do not, say why and what you did instead — most commonly a rate limit strict enough that the file's purpose (protecting the server) is served by other means.
Line 393: Line 395:
   * [[Programming:Crawler]] — where the rate limit and the User-Agent actually get set.   * [[Programming:Crawler]] — where the rate limit and the User-Agent actually get set.
   * [[Privacy:Consent]] — interacting with a banner is an interaction.   * [[Privacy:Consent]] — interacting with a banner is an interaction.
 +  * [[Privacy:Age assurance]] — **four questions this page does not currently answer**: crawling adult content, submitting a synthetic or a real identity document to a live vendor, publishing a working bypass, and measuring an age model without measuring children. That page states them as a gap here.
   * [[Design:Website selection]] — a robots.txt exclusion is a change to your sampling frame.   * [[Design:Website selection]] — a robots.txt exclusion is a change to your sampling frame.
   * [[:Artifacts]] — what you can release once the data has people in it.   * [[:Artifacts]] — what you can release once the data has people in it.
 +  * [[Statistics:Interrater agreement]] — hired annotators and crowdworkers are participants; 5.1% of the corpus's hand-coding papers use them.
  
 ===== Methodology and limitations of these figures ===== ===== Methodology and limitations of these figures =====
practices/ethics.1787094540.txt.gz · Last modified: by karel.kubicek.claude