| Both sides previous revisionPrevious revision | |
| provenance:programming:filter_lists [2026/08/29 13:25] – Record that privacy:requests' script had already solved the ground-truth fold correctly with a purpose-built regex, so reusing the shared fold for that one field was the regression. Authored by Claude karel.kubicek.claude | provenance:programming:filter_lists [2026/08/29 13:38] (current) – Record the precision correction in full: the two ambiguous-family signals, the two false positives they cost in testing, all seven printed exclusions, why the residue could not have caught this (it answers 'what did I miss', not 'what did I wrongly admit' karel.kubicek.claude |
|---|
| |
| ^ ''kind'' ^ Meaning ^ Qualifies? ^ Papers ^ | ^ ''kind'' ^ Meaning ^ Qualifies? ^ Papers ^ |
| | ''list'' | a rule set you can download | **yes** | 198 | | | ''list'' | a rule set you can download | **yes**, if the tuple's own context is not off-topic | 194 | |
| | ''engine'' | code that evaluates rules (''adblockparser'', ''adblock-rust'', ''tracker-radar-collector'') | no | 30 across 6 families | | | ''engine'' | code that evaluates rules (''adblockparser'', ''adblock-rust'', ''tracker-radar-collector'') | no | 30 across 6 families | |
| | ''heuristic'' | a classifier shipped inside a blocker that is not a list (Privacy Badger) | no | — | | | ''heuristic'' | a classifier shipped inside a blocker that is not a list (Privacy Badger) | no | — | |
| |
| ^ Query ^ Population ^ Denominator ^ Result ^ | ^ Query ^ Population ^ Denominator ^ Result ^ |
| | papers using or producing a filter list | whole corpus | 5,859 | **198** (3.4%) | | | papers using or producing a filter list | whole corpus | 5,859 | **194** (3.3%) | |
| | … named in ''tools[]'' | population | 198 | 178 | | | … named in ''tools[]'' | population | 194 | 177 | |
| | … named in a ''classification[]'' step | population | 198 | 167 | | | … named in a ''classification[]'' step | population | 194 | 163 | |
| | … in both | population | 198 | 147 | | | … in both | population | 194 | 146 | |
| | … in ''tools[]'' only | population | 198 | 31 | | | … in ''tools[]'' only | population | 194 | 31 | |
| | named a list but only ''mentioned''/''compared'' | whole corpus | 5,859 | 18 (excluded) | | | named a list but only ''mentioned''/''compared'' | whole corpus | 5,859 | 19 (excluded) | |
| | | named a list, used it off-topic (malware / piracy) | whole corpus | 5,859 | 7 tuples, 5 papers (excluded) | |
| |
| **Why ''mentioned'' and ''compared'' are excluded.** A paper that beats EasyList is not a paper that adopted it. The 18 excluded are 8.4% of the 216 a raw name search would score as adoption. | **Why ''mentioned'' and ''compared'' are excluded.** A paper that beats EasyList is not a paper that adopted it. The 19 excluded are 8.9% of the 213 a raw name search would score as adoption. A further **five papers are excluded on the tuple's context rather than its name** — see //When a list name is right but the use is off-topic// below. |
| |
| **Difference from [[Privacy:Requests]].** That page's "S1" signal reports **197** where this reports **198**, with per-list counts differing by one or two. Both scripts share ''req_fold.mjs'', so the fold is identical; the difference is the membership rule. ''report_requests.mjs'' builds S1 inside a population that is then intersected with a request-classification task fold and reported after that page's own exclusions; ''report_filter_lists.mjs'' takes every paper the fold fires on, with no topic restriction. **Neither is wrong.** The delta is published on both pages rather than reconciled, because the honest lesson is that a population definition moves a count by a percent or two and belongs in the provenance. | **Difference from [[Privacy:Requests]].** That page's "S1" signal reports **197** where this reports **194**. Both scripts share ''req_fold.mjs'', so the fold is identical; the difference is the membership rule. ''report_requests.mjs'' builds S1 inside a population that is then intersected with a request-classification task fold and reported after that page's own exclusions; ''report_filter_lists.mjs'' takes every paper the fold fires on with no topic restriction, but then applies a tuple-context test that page does not have, which removes five papers whose list name is right and whose use is a malware or piracy blacklist. **Neither is wrong.** The delta is published on both pages rather than reconciled, because the honest lesson is that a population definition moves a count by a percent or two and belongs in the provenance. |
| |
| ===== Every query on the page ===== | ===== Every query on the page ===== |
| |
| ^ Figure on the page ^ Population ^ Denominator ^ Value ^ | ^ Figure on the page ^ Population ^ Denominator ^ Value ^ |
| | used or produced a filter list | corpus | 5,859 | 198 (3.4%) | | | used or produced a filter list | corpus | 5,859 | 194 (3.3%) | |
| | EasyList | list-using papers | 198 | 112 (56.6%) | | | EasyList | list-using papers | 194 | 112 (57.7%) | |
| | EasyPrivacy | list-using papers | 198 | 71 (35.9%) | | | EasyPrivacy | list-using papers | 194 | 71 (36.6%) | |
| | Disconnect | list-using papers | 198 | 48 (24.2%) | | | Disconnect | list-using papers | 194 | 48 (24.7%) | |
| | Ghostery / WhoTracks.me | list-using papers | 198 | 34 (17.2%) | | | Ghostery / WhoTracks.me | list-using papers | 194 | 34 (17.5%) | |
| | hosts-file lists | list-using papers | 198 | 27 (13.6%) | | | hosts-file lists | list-using papers | 194 | 24 (12.4%) | |
| | ''web-request'' target | papers with a list-backed classification step | 167 | 101 (60.5%) | | | ''web-request'' target | papers with a list-backed classification step | 163 | 101 (62.0%) | |
| | ''domain'' target | papers with a list-backed classification step | 167 | 53 (31.7%) | | | ''domain'' target | papers with a list-backed classification step | 163 | 51 (31.3%) | |
| | ''cookie'' target | papers with a list-backed classification step | 167 | 8 (4.8%) | | | ''cookie'' target | papers with a list-backed classification step | 163 | 8 (4.9%) | |
| | "coarser than the list" group | papers with a list-backed classification step | 167 | 53 (31.7%) | | | "coarser than the list" group | papers with a list-backed classification step | 163 | 51 (31.3%) | |
| | "objects a list was not written to label" group | papers with a list-backed classification step | 167 | 30 (18.0%) | | | "objects a list was not written to label" group | papers with a list-backed classification step | 163 | 28 (17.2%) | |
| | ''groundTruthSource'' names a filter list | **whole corpus** | 5,859 | 96 | | | papers in more than one granularity group | papers with a list-backed classification step | 163 | 17 | |
| | … within the list-using population | list-using papers | 198 | 88 (44.4%) | | | ''groundTruthSource'' names a filter list | **whole corpus** | 5,859 | 92 | |
| | … where the list is named generically, not by name | papers with a list ground truth | 96 | 13 | | | … within the list-using population | list-using papers | 194 | 86 (44.3%) | |
| | learned classifier with a filter-list ground truth | whole corpus | 5,859 | 16 papers, 22 tuples | | | … where the list is named generically, not by name | papers with a list ground truth | 92 | 13 | |
| | … of the list-using papers that ran any learned classifier | list-using papers with a learned classifier | 62 | 15 (24.2%) | | | learned classifier with a filter-list ground truth | whole corpus | 5,859 | 15 papers, 20 tuples | |
| | ''validation == not-applicable'' | papers with a list-backed classification step | 167 | 69 (41.3%) | | | … of the list-using papers that ran any learned classifier | list-using papers with a learned classifier | 60 | 15 (25.0%) | |
| | ''validation == none-reported'' | papers with a list-backed classification step | 167 | 61 (36.5%) | | | ''validation == not-applicable'' | papers with a list-backed classification step | 163 | 68 (41.7%) | |
| | ''validation == manual-validation'' | papers with a list-backed classification step | 167 | 34 (20.4%) | | | ''validation == none-reported'' | papers with a list-backed classification step | 163 | 60 (36.8%) | |
| | ''validation == comparison-to-other-method'' | papers with a list-backed classification step | 167 | 14 (8.4%) | | | ''validation == manual-validation'' | papers with a list-backed classification step | 163 | 34 (20.9%) | |
| | **papers whose validation is ONLY sentinels** | papers with a list-backed classification step | 167 | **119 (71.3%)** | | | ''validation == comparison-to-other-method'' | papers with a list-backed classification step | 163 | 12 (7.4%) | |
| | papers with at least one real validation value | papers with a list-backed classification step | 167 | 48 (28.7%) | | | **papers whose validation is ONLY sentinels** | papers with a list-backed classification step | 163 | **117 (71.8%)** | |
| | papers in more than one granularity group | papers with a list-backed classification step | 167 | 17 | | | papers with at least one real validation value | papers with a list-backed classification step | 163 | 46 (28.2%) | |
| | any version or date on the list tool | papers naming a list in ''tools[]'' | 178 | 53 (29.8%) | | | any version or date on the list tool | papers naming a list in ''tools[]'' | 177 | 53 (29.9%) | |
| | a version that identifies the **rules** | papers naming a list in ''tools[]'' | 178 | **4 (2.2%)** | | | a version that identifies the **rules** | papers naming a list in ''tools[]'' | 177 | **4 (2.3%)** | |
| | PETS share | PETS papers in corpus | 510 | 55 (10.8%) | | | PETS share | PETS papers in corpus | 510 | 55 (10.8%) | |
| | NDSS share | NDSS papers in corpus | 701 | 10 (1.4%) | | | NDSS share | NDSS papers in corpus | 701 | 10 (1.4%) | |
| | 2018–2021 rate | corpus papers 2018–2021 | 1,439 | 58 → 40.3 per 1,000 | | | 2018–2021 rate | corpus papers 2018–2021 | 1,439 | 57 → 39.6 per 1,000 | |
| | 2025–2026 rate //(provisional)// | corpus papers 2025–2026 | 1,185 | 39 → 32.9 per 1,000 | | | 2025–2026 rate //(provisional)// | corpus papers 2025–2026 | 1,185 | 38 → 32.1 per 1,000 | |
| | papers about regional lists | **whole corpus** | 5,859 | 2 | | | papers about regional lists | **whole corpus** | 5,859 | 2 | |
| | ''adblockparser'' users | whole corpus | 5,859 | 9 | | | ''adblockparser'' users | whole corpus | 5,859 | 9 | |
| === 1. POPULATION ======================================================= | === 1. POPULATION ======================================================= |
| corpus 5859 | corpus 5859 |
| USED or PRODUCED a filter list == POPULATION 198 3.4% of corpus | USED or PRODUCED a filter list == POPULATION 194 3.3% of corpus |
| ... named in tools[] 178 | ... named in tools[] 177 |
| ... named in classification[].resourceName 167 | ... named in classification[].resourceName 163 |
| ... in both 147 | ... in both 146 |
| ... tools[] only (instrument, never a labelling step) 31 | ... tools[] only (instrument, never a labelling step) 31 |
| named a list but only MENTIONED / COMPARED (excluded) 18 | named a list but only MENTIONED / COMPARED (excluded) 19 |
| |
| |
| === 2. WHICH LISTS, FOLDED (denominator: the population above) ========= | === 2. WHICH LISTS, FOLDED (denominator: the population above) ========= |
| denominator: 198 papers that used or produced a filter list | denominator: 194 papers that used or produced a filter list |
| (a paper naming several lists is counted under each, so shares do not sum to 100%) | (a paper naming several lists is counted under each, so shares do not sum to 100%) |
| list family papers share spellings unfolded | list family papers share spellings unfolded |
| EasyList 112 56.6% 28 92 | EasyList 112 57.7% 28 92 |
| EasyPrivacy 71 35.9% 22 58 | EasyPrivacy 71 36.6% 22 58 |
| Disconnect 48 24.2% 23 26 | Disconnect 48 24.7% 23 26 |
| Ghostery / WhoTracks.me 34 17.2% 9 20 | Ghostery / WhoTracks.me 34 17.5% 9 20 |
| hosts-file lists (hpHosts, AdAway, MoaAB, Pi-hole, NoTrack, …) 27 13.6% 33 3 | Adblock Plus (lists shipped with it) 26 13.4% 8 20 |
| Adblock Plus (lists shipped with it) 26 13.1% 8 20 | hosts-file lists (hpHosts, AdAway, MoaAB, Pi-hole, NoTrack, …) 24 12.4% 32 3 |
| uBlock Origin (lists) 17 8.6% 11 13 | uBlock Origin (lists) 17 8.8% 11 13 |
| DuckDuckGo Tracker Radar (list) 15 7.6% 9 8 | DuckDuckGo Tracker Radar (list) 15 7.7% 9 8 |
| unnamed / aggregated filter lists 10 5.1% 10 1 | unnamed / aggregated filter lists 9 4.6% 9 1 |
| AdGuard 8 4.0% 13 4 | AdGuard 8 4.1% 13 4 |
| EasyList annoyance / anti-adblock variants 7 3.5% 8 2 | EasyList annoyance / anti-adblock variants 7 3.6% 8 2 |
| anti-adblock scripts and services 3 1.5% 4 1 | anti-adblock scripts and services 3 1.5% 4 1 |
| cryptomining lists (NoCoin, CoinBlockerLists, MinerBlock) 3 1.5% 4 3 | cryptomining lists (NoCoin, CoinBlockerLists, MinerBlock) 3 1.5% 4 3 |
| |
| === 3. WHAT THE LIST IS USED TO LABEL (denominator: papers with >=1 list-backed classification tuple) | === 3. WHAT THE LIST IS USED TO LABEL (denominator: papers with >=1 list-backed classification tuple) |
| denominator: 167 papers | denominator: 163 papers |
| web-request 101 60.5% | web-request 101 62.0% |
| domain 53 31.7% | domain 51 31.3% |
| cookie 8 4.8% | cookie 8 4.9% |
| website-category 7 4.2% | website-category 5 3.1% |
| javascript 5 3.0% | javascript 5 3.1% |
| other 3 1.8% | other 3 1.8% |
| fingerprinting-script 3 1.8% | fingerprinting-script 3 1.8% |
| |
| -- grouped by granularity | -- grouped by granularity |
| The list's own unit — a URL pattern evaluated with request context 101 60.5% | The list's own unit — a URL pattern evaluated with request context 101 62.0% |
| Coarser than the list — a hostname or eTLD+1; path, resource-type and $third-party options are discarded 53 31.7% | Coarser than the list — a hostname or eTLD+1; path, resource-type and $third-party options are discarded 51 31.3% |
| Objects a filter list was not written to label — the list supplies provenance, not a rule match 30 18.0% | Objects a filter list was not written to label — the list supplies provenance, not a rule match 28 17.2% |
| granularity-fold residue (targets in no group): (none) | granularity-fold residue (targets in no group): (none) |
| papers in MORE THAN ONE granularity group (so shares exceed 100%) 17 of 167 | papers in MORE THAN ONE granularity group (so shares exceed 100%) 17 of 163 |
| |
| -- the papers behind each non-request target, so the claim is checkable | -- the papers behind each non-request target, so the claim is checkable |
| 2018 USENIX/who-left-open-the-cookie-jar-a-comprehensive-evaluation-of-third-party-cookie-po [blocklist] "EasyList and EasyPrivacy" | 2018 USENIX/who-left-open-the-cookie-jar-a-comprehensive-evaluation-of-third-party-cookie-po [blocklist] "EasyList and EasyPrivacy" |
| 2019 IMC/a-longitudinal-analysis-of-the-ads-txt-standard [curated-database] "WhoTracksMe" | 2019 IMC/a-longitudinal-analysis-of-the-ads-txt-standard [curated-database] "WhoTracksMe" |
| 2019 IMC/shamfinder-an-automated-framework-for-detecting-idn-homographs [curated-database] "hpHosts" | |
| 2019 PETS/4-years-of-eu-cookie-law-results-and-lessons-learned [curated-database] "Ghostery" | 2019 PETS/4-years-of-eu-cookie-law-results-and-lessons-learned [curated-database] "Ghostery" |
| 2020 PETS/missed-by-filter-lists-detecting-unknown-third-party-trackers-with-invisible-pix [curated-database] "Disconnect" | 2020 PETS/missed-by-filter-lists-detecting-unknown-third-party-trackers-with-invisible-pix [curated-database] "Disconnect" |
| 2023 IEEE-SP/protected-or-porous-a-comparative-analysis-of-threat-detection-capability-of-iot [curated-database] "EasyList" | 2023 IEEE-SP/protected-or-porous-a-comparative-analysis-of-threat-detection-capability-of-iot [curated-database] "EasyList" |
| 2023 USENIX/a-study-of-multi-factor-and-risk-based-authentication-availability [curated-database] "Disconnect.me tracker list" | 2023 USENIX/a-study-of-multi-factor-and-risk-based-authentication-availability [curated-database] "Disconnect.me tracker list" |
| 2023 USENIX/trident-towards-detecting-and-mitigating-web-based-social-engineering-attacks [curated-database] "Categorical BlockList" | |
| 2023 PETS/privacy-rarely-considered-exploring-considerations-in-the-adoption-of-third-part [curated-database] "WhoTracks.me" | 2023 PETS/privacy-rarely-considered-exploring-considerations-in-the-adoption-of-third-part [curated-database] "WhoTracks.me" |
| 2024 IMC/diffaudit-auditing-privacy-practices-of-online-services-for-children-and-adolesc [curated-database] "DuckDuckGo Tracker Radar" | 2024 IMC/diffaudit-auditing-privacy-practices-of-online-services-for-children-and-adolesc [curated-database] "DuckDuckGo Tracker Radar" |
| website-category: | website-category: |
| 2015 PETS/an-automated-approach-for-complementing-ad-blockers-blacklists [blocklist] "Adblock Plus EasyList and EasyPrivacy" | 2015 PETS/an-automated-approach-for-complementing-ad-blockers-blacklists [blocklist] "Adblock Plus EasyList and EasyPrivacy" |
| 2014 USENIX/automatically-detecting-vulnerable-websites-before-they-turn-malicious [curated-database] "hpHosts" | |
| 2019 WWW/cookie-synchronization-everything-you-always-wanted-to-know-but-were-afraid-to-a [blocklist] "EasyList" | 2019 WWW/cookie-synchronization-everything-you-always-wanted-to-know-but-were-afraid-to-a [blocklist] "EasyList" |
| 2023 CCS/you-call-this-archaeology-evaluating-web-archives-for-reproducible-web-security [blocklist] "Disconnect" | 2023 CCS/you-call-this-archaeology-evaluating-web-archives-for-reproducible-web-security [blocklist] "Disconnect" |
| 2024 PETS/generalizable-active-privacy-choice-designing-a-graphical-user-interface-for-glo [curated-database] "Disconnect Tracker Protection lists" | 2024 PETS/generalizable-active-privacy-choice-designing-a-graphical-user-interface-for-glo [curated-database] "Disconnect Tracker Protection lists" |
| 2025 IEEE-SP/only-as-strong-as-the-weakest-link-on-the-security-of-brokered-single-sign-on-on [curated-database] "Tracker Radar Entity List" | 2025 IEEE-SP/only-as-strong-as-the-weakest-link-on-the-security-of-brokered-single-sign-on-on [curated-database] "Tracker Radar Entity List" |
| 2025 WWW/welcome-to-the-dark-side-analyzing-the-revenue-flows-of-fraud-in-the-online-ad-e [blocklist] "NextDNS Piracy Blocklist" | |
| javascript: | javascript: |
| 2017 PETS/towards-seamless-tracking-free-web-improved-detection-of-trackers-via-one-class [blocklist] "EasyList" | 2017 PETS/towards-seamless-tracking-free-web-improved-detection-of-trackers-via-one-class [blocklist] "EasyList" |
| |
| === 4. THE LIST AS GROUND TRUTH (denominator: whole corpus — this is not restricted to the population) | === 4. THE LIST AS GROUND TRUTH (denominator: whole corpus — this is not restricted to the population) |
| papers whose classification.groundTruthSource names a filter list 96 | papers whose classification.groundTruthSource names a filter list 92 |
| ... as a share of the list-using population 88 of 198 = 44.4% | ... as a share of the list-using population 86 of 194 = 44.3% |
| |
| -- by target (papers) | -- by target (papers) |
| web-request 45 46.9% | web-request 45 48.9% |
| domain 36 37.5% | domain 35 38.0% |
| javascript 8 8.3% | javascript 8 8.7% |
| cookie 6 6.3% | cookie 6 6.5% |
| other 6 6.3% | other 5 5.4% |
| website-category 3 3.1% | website-category 2 2.2% |
| fingerprinting-script 2 2.1% | fingerprinting-script 2 2.2% |
| ip-address 1 1.0% | privacy-policy 1 1.1% |
| privacy-policy 1 1.0% | consent-notice 1 1.1% |
| consent-notice 1 1.0% | network-traffic 1 1.1% |
| network-traffic 1 1.0% | email-message 1 1.1% |
| email-message 1 1.0% | |
| |
| -- by list (papers) | -- by list (papers) |
| EasyList 48 50.0% | EasyList 48 52.2% |
| EasyPrivacy 37 38.5% | EasyPrivacy 37 40.2% |
| Disconnect 20 20.8% | Disconnect 19 20.7% |
| hosts-file lists (hpHosts, AdAway, MoaAB, Pi-hole, NoTrack, …) 13 13.5% | named generically, no list named 13 14.1% |
| named generically, no list named 13 13.5% | hosts-file lists (hpHosts, AdAway, MoaAB, Pi-hole, NoTrack, …) 10 10.9% |
| Ghostery / WhoTracks.me 9 9.4% | Ghostery / WhoTracks.me 9 9.8% |
| DuckDuckGo Tracker Radar (list) 6 6.3% | DuckDuckGo Tracker Radar (list) 6 6.5% |
| EasyList annoyance / anti-adblock variants 4 4.2% | EasyList annoyance / anti-adblock variants 4 4.3% |
| uBlock Origin (lists) 4 4.2% | uBlock Origin (lists) 4 4.3% |
| unnamed / aggregated filter lists 2 2.1% | unnamed / aggregated filter lists 2 2.2% |
| Adblock Plus (lists shipped with it) 1 1.0% | Adblock Plus (lists shipped with it) 1 1.1% |
| AdGuard 1 1.0% | AdGuard 1 1.1% |
| Acceptable Ads exception list 1 1.0% | Acceptable Ads exception list 1 1.1% |
| |
| -- LEARNED classifiers (supervised-ml / unsupervised-ml / llm) whose ground truth is a filter list | -- LEARNED classifiers (supervised-ml / unsupervised-ml / llm) whose ground truth is a filter list |
| distinct papers 16 (22 tuples) | distinct papers 15 (20 tuples) |
| 2015 PETS/an-automated-approach-for-complementing-ad-blockers-blacklists target=website-category gt="Manual labels and Adblock Plus ABPexact labels" | 2015 PETS/an-automated-approach-for-complementing-ad-blockers-blacklists target=website-category gt="Manual labels and Adblock Plus ABPexact labels" |
| 2017 IMC/the-ad-wars-retrospective-measurement-and-analysis-of-anti-adblock-filter-lists target=javascript gt="372 filter-list-matched anti-adblocking scripts plus remaining unmatched scripts" | 2017 IMC/the-ad-wars-retrospective-measurement-and-analysis-of-anti-adblock-filter-lists target=javascript gt="372 filter-list-matched anti-adblocking scripts plus remaining unmatched scripts" |
| 2018 IMC/characterizing-the-internet-host-population-using-deep-learning-a-universal-and target=ip-address gt="hpHosts, PhishTank, and MPDNS labels" | |
| 2018 PETS/nomoads-effective-and-efficient-cross-app-mobile-ad-blocking target=web-request gt="EasyList and manually created mobile-ad rules" | 2018 PETS/nomoads-effective-and-efficient-cross-app-mobile-ad-blocking target=web-request gt="EasyList and manually created mobile-ad rules" |
| 2019 CCS/adversarial-perceptual-ad-blocking-meets-adversarial-machine-learning target=other gt="filter-list-labeled screenshots" | 2019 CCS/adversarial-perceptual-ad-blocking-meets-adversarial-machine-learning target=other gt="filter-list-labeled screenshots" |
| |
| -- denominator for that: papers running a LEARNED classifier, that also use a list | -- denominator for that: papers running a LEARNED classifier, that also use a list |
| list-using papers running any learned classifier 62 | list-using papers running any learned classifier 60 |
| ... of which a filter list is the stated ground truth 15 24.2% | ... of which a filter list is the stated ground truth 15 25.0% |
| ... of which some OTHER source is the stated ground truth 47 | ... of which some OTHER source is the stated ground truth 45 |
| |
| |
| === 5. VALIDATION OF LIST-BACKED CLASSIFICATION (denominator: papers with >=1 list-backed classification tuple) | === 5. VALIDATION OF LIST-BACKED CLASSIFICATION (denominator: papers with >=1 list-backed classification tuple) |
| denominator: 167 papers (a paper with several tuples appears in several rows) | denominator: 163 papers (a paper with several tuples appears in several rows) |
| not-applicable [SENTINEL — not an answer] 69 41.3% | not-applicable [SENTINEL — not an answer] 68 41.7% |
| none-reported [SENTINEL — not an answer] 61 36.5% | none-reported [SENTINEL — not an answer] 60 36.8% |
| manual-validation 34 20.4% | manual-validation 34 20.9% |
| comparison-to-other-method 14 8.4% | comparison-to-other-method 12 7.4% |
| |
| -- paper-level, because the rows above overlap | -- paper-level, because the rows above overlap |
| papers whose list-backed validation is ONLY sentinels 119 71.3% | papers whose list-backed validation is ONLY sentinels 117 71.8% |
| papers with at least one real validation value 48 28.7% | papers with at least one real validation value 46 28.2% |
| papers appearing in more than one row above 11 | papers appearing in more than one row above 11 |
| |
| |
| === 6. WHICH VERSION OF THE LIST (denominator: papers naming a list in tools[], where a version field exists) | === 6. WHICH VERSION OF THE LIST (denominator: papers naming a list in tools[], where a version field exists) |
| papers naming a filter list in tools[] 178 | papers naming a filter list in tools[] 177 |
| ... with any non-null tools[].version 53 29.8% | ... with any non-null tools[].version 53 29.9% |
| ... silent 125 70.2% | ... silent 124 70.1% |
| ... whose version identifies the RULES (a list version or commit, not a year) 4 2.2% | ... whose version identifies the RULES (a list version or commit, not a year) 4 2.3% |
| |
| -- the version strings that identify the rules | -- the version strings that identify the rules |
| venue corpus pop share of venue | venue corpus pop share of venue |
| PETS 510 55 10.8% | PETS 510 55 10.8% |
| IMC 638 36 5.6% | IMC 638 35 5.5% |
| WWW 843 31 3.7% | WWW 843 30 3.6% |
| USENIX 1410 24 1.7% | |
| CCS 990 23 2.3% | CCS 990 23 2.3% |
| | USENIX 1410 22 1.6% |
| IEEE-SP 767 19 2.5% | IEEE-SP 767 19 2.5% |
| NDSS 701 10 1.4% | NDSS 701 10 1.4% |
| period corpus pop per 1,000 | period corpus pop per 1,000 |
| 2010-2013 511 6 11.7 | 2010-2013 511 6 11.7 |
| 2014-2017 769 25 32.5 | 2014-2017 769 24 31.2 |
| 2018-2021 1439 58 40.3 | 2018-2021 1439 57 39.6 |
| 2022-2024 1955 70 35.8 | 2022-2024 1955 69 35.3 |
| 2025-2026 (provisional) 1185 39 32.9 | 2025-2026 (provisional) 1185 38 32.1 |
| |
| -- per year, for the currency claim (same provisional caveat) | -- per year, for the currency claim (same provisional caveat) |
| 2012 corpus 151 pop 1 0.7% pop papers with any llm classification: 0 | 2012 corpus 151 pop 1 0.7% pop papers with any llm classification: 0 |
| 2013 corpus 125 pop 2 1.6% pop papers with any llm classification: 0 | 2013 corpus 125 pop 2 1.6% pop papers with any llm classification: 0 |
| 2014 corpus 166 pop 4 2.4% pop papers with any llm classification: 0 | 2014 corpus 166 pop 3 1.8% pop papers with any llm classification: 0 |
| 2015 corpus 190 pop 4 2.1% pop papers with any llm classification: 0 | 2015 corpus 190 pop 4 2.1% pop papers with any llm classification: 0 |
| 2016 corpus 182 pop 7 3.8% pop papers with any llm classification: 0 | 2016 corpus 182 pop 7 3.8% pop papers with any llm classification: 0 |
| 2017 corpus 231 pop 10 4.3% pop papers with any llm classification: 0 | 2017 corpus 231 pop 10 4.3% pop papers with any llm classification: 0 |
| 2018 corpus 254 pop 13 5.1% pop papers with any llm classification: 0 | 2018 corpus 254 pop 13 5.1% pop papers with any llm classification: 0 |
| 2019 corpus 402 pop 13 3.2% pop papers with any llm classification: 0 | 2019 corpus 402 pop 12 3.0% pop papers with any llm classification: 0 |
| 2020 corpus 404 pop 18 4.5% pop papers with any llm classification: 0 | 2020 corpus 404 pop 18 4.5% pop papers with any llm classification: 0 |
| 2021 corpus 379 pop 14 3.7% pop papers with any llm classification: 0 | 2021 corpus 379 pop 14 3.7% pop papers with any llm classification: 0 |
| 2022 corpus 546 pop 22 4.0% pop papers with any llm classification: 0 | 2022 corpus 546 pop 22 4.0% pop papers with any llm classification: 0 |
| 2023 corpus 719 pop 26 3.6% pop papers with any llm classification: 0 | 2023 corpus 719 pop 25 3.5% pop papers with any llm classification: 0 |
| 2024 corpus 690 pop 22 3.2% pop papers with any llm classification: 2 | 2024 corpus 690 pop 22 3.2% pop papers with any llm classification: 2 |
| 2025 corpus 770 pop 31 4.0% pop papers with any llm classification: 4 | 2025 corpus 770 pop 30 3.9% pop papers with any llm classification: 4 |
| 2026 corpus 415 pop 8 1.9% pop papers with any llm classification: 1 | 2026 corpus 415 pop 8 1.9% pop papers with any llm classification: 1 |
| |
| off-topic: censorship blocklist 3 | off-topic: censorship blocklist 3 |
| off-topic: certificate / key revocation blocklist 2 | off-topic: certificate / key revocation blocklist 2 |
| | |
| | -- EXCLUDED BY TUPLE CONTEXT: the list name is right, the use is off-topic |
| | 2014 USENIX/automatically-detecting-vulnerable-websites-before-they-turn-malicious resourceName "hpHosts" gt "hpHosts blacklist" |
| | We also discarded an additional 421 sites found in the DNS-BH [2], Google SafeBrowsing [14], and hpHosts [23] blacklists. |
| | 2018 IMC/characterizing-the-internet-host-population-using-deep-learning-a-universal-and gt "hpHosts, PhishTank, and MPDNS labels" |
| | we use reported malicious hosts (as labels) to train a series of supervised models that assign reputation scores to arbitrary hosts |
| | 2018 IMC/characterizing-the-internet-host-population-using-deep-learning-a-universal-and gt "hpHosts, PhishTank, and MPDNS labels" |
| | We thus use 10-D embeddings and train k-NN models that return the percentage of malicious hosts within the most 50 similar training samples |
| | 2019 IMC/shamfinder-an-automated-framework-for-detecting-idn-homographs resourceName "hpHosts" gt "hpHosts blacklist" |
| | malicious domains | To check whether the detected IDN homographs have been used for malicious purposes, we leveraged three different sources of blacklists, hpHosts [17], Google Safe Browsing (GSB) [14 |
| | 2023 USENIX/trident-towards-detecting-and-mitigating-web-based-social-engineering-attacks tools[] "Categorical BlockList" |
| | Provided malicious URL labels for ground truth. | We choose three additional services for identifying whether a website is malicious or not, a categorical BlockList [31] on Github |
| | 2023 USENIX/trident-towards-detecting-and-mitigating-web-based-social-engineering-attacks resourceName "Categorical BlockList" |
| | Malware, Scam, Abuse, Phishing, and Fraud | A URL malicious if it falls in the buckets of Malware, Scam, Abuse, Phishing, and Fraud in the BlockList |
| | 2025 WWW/welcome-to-the-dark-side-analyzing-the-revenue-flows-of-fraud-in-the-online-ad-e resourceName "NextDNS Piracy Blocklist" |
| | torrent and warez categories | We utilize NextDNS' Piracy Blocklist [56] and focus on 1,395 websites in the "torrent" and "warez" categories. |
| | tuples excluded by context 7 |
| |
| -- groundTruthSource residue: strings that look like a blocklist and are NOT counted as one | -- groundTruthSource residue: strings that look like a blocklist and are NOT counted as one |
| distinct groundTruthSource strings excluded 68 | distinct groundTruthSource strings excluded 69 |
| 6 Google Safe Browsing blacklist | 6 Google Safe Browsing blacklist |
| 2 Spamhaus blacklist | 2 Spamhaus blacklist |
| 1 new binaries, blacklisted domains, and deviation from benign program profiles | 1 new binaries, blacklisted domains, and deviation from benign program profiles |
| 1 blacklist entries | 1 blacklist entries |
| | 1 physical ECU access and disconnection |
| 1 manually curated blacklists | 1 manually curated blacklists |
| 1 blacklist patterns and Wappalyzer family mapping | 1 blacklist patterns and Wappalyzer family mapping |
| </code> | </code> |
| |
| None of those matched, so the papers folded to nothing and dropped out of the ground-truth counts. **The first published version of this page therefore said 88 papers where the answer is 96, and printed an eleven-paper "who's-who of the reference baselines" that omitted AdGraph and WebGraph** — the two most famous filter-list-labelled classifiers in the field. It also silently disagreed with [[Privacy:Requests]], which names both — **because that page's script had already got this right**: ''report_requests.mjs'' does not reuse the shared fold for ''groundTruthSource'' at all, it carries a purpose-built ''GT_LIST_RX''. Reusing a shared fold is usually the right instinct and is why the two pages agree about what "EasyList" means; it was the wrong instinct for **this one field**, and the sibling page's apparent duplication was the correct design. | None of those matched, so the papers folded to nothing and dropped out of the ground-truth counts. **The first published version of this page therefore said 88 papers where the answer is 92, and printed an eleven-paper "who's-who of the reference baselines" that omitted AdGraph and WebGraph** — the two most famous filter-list-labelled classifiers in the field. It also silently disagreed with [[Privacy:Requests]], which names both — **because that page's script had already got this right**: ''report_requests.mjs'' does not reuse the shared fold for ''groundTruthSource'' at all, it carries a purpose-built ''GT_LIST_RX''. Reusing a shared fold is usually the right instinct and is why the two pages agree about what "EasyList" means; it was the wrong instinct for **this one field**, and the sibling page's apparent duplication was the correct design. |
| |
| The fix is a second, **unanchored** regex applied **only to ''groundTruthSource''**: | The fix is a second, **unanchored** regex applied **only to ''groundTruthSource''**: |
| **Lesson for the next page, stated plainly: a fold's residue must be surveyed over every field the fold is applied to.** Surveying it over two of three fields produced a page that passed every mechanical check and was wrong about its central claim. | **Lesson for the next page, stated plainly: a fold's residue must be surveyed over every field the fold is applied to.** Surveying it over two of three fields produced a page that passed every mechanical check and was wrong about its central claim. |
| |
| ==== 4. Regional lists — ''REGIONAL_RX'' in ''fl_fold.mjs'' ==== | ==== 4. When a list name is right but the use is off-topic — ''TUPLE_OFFTOPIC_RX'' / ''SECURITY_FEED_RX'' ==== |
| | |
| | <WRAP important> |
| | **A second correction from the same review round, and it moved the headline population.** After the ground-truth fold was widened, a re-review dumped all the papers it now admitted and read them. Three were wrong, and the fold could not have known: **the list name was right and the //use// was not.** |
| | </WRAP> |
| | |
| | Some families are genuinely ambiguous artefacts. hpHosts shipped an advertising/tracking file **and** a malware file. "Categorical BlockList" is a bucket name. NextDNS publishes a piracy list. Three papers named one of these and were doing security work: |
| | |
| | <code> |
| | 2014 USENIX automatically-detecting-vulnerable-websites-before-they-turn-malicious |
| | gt "hpHosts blacklist" |
| | "We also discarded an additional 421 sites found in the DNS-BH, Google SafeBrowsing, and hpHosts blacklists." |
| | 2018 IMC characterizing-the-internet-host-population-using-deep-learning |
| | gt "hpHosts, PhishTank, and MPDNS labels" -> reputation scores for malicious hosts |
| | 2019 IMC shamfinder-an-automated-framework-for-detecting-idn-homographs |
| | gt "hpHosts blacklist" taxonomy "malicious domains" |
| | </code> |
| | |
| | The fix consults the **tuple's own context** — its ''taxonomy'', its ''targetDetail'', its ''purpose'' and its verbatim evidence quote — and only for the ambiguous families. EasyList, EasyPrivacy, Disconnect, Ghostery and Tracker Radar are unambiguous advertising/tracking artefacts, and a tracking paper that happens to mention malware must not be dropped for saying so. |
| | |
| | Two signals, because one was not enough: |
| | |
| | * ''TUPLE_OFFTOPIC_RX'' — malware, malicious, phishing, drive-by, spam, fraud, scam, piracy, botnet, ransomware, typosquatting, homograph. |
| | * ''SECURITY_FEED_RX'' — the list is named in the same sentence as DNS-BH, Google Safe Browsing, PhishTank, Symantec DeepSight, Spamhaus, CleanMX, VirusTotal, URLhaus, OpenPhish or the APWG. **The 2014 paper above contains none of the vocabulary words**; it was caught only by the company it keeps. |
| | |
| | Two things were deliberately kept **out** of the regex, each after it cost a false positive in testing: |
| | |
| | * ''\bmalware(?!bytes)'' — //"MalwareBytes' hpHosts ATS file"// is a company name, in a paper whose taxonomy is literally "advertising and tracking services". Matching it would have dropped a correct row. |
| | * **Cryptomining is not off-topic.** NoCoin and MinerBlock are Adblock-syntax lists that ship inside ad blockers; a cryptojacking paper using them is using a filter list. |
| | |
| | **Every exclusion is printed** in the report output under //EXCLUDED BY TUPLE CONTEXT//: 7 tuples across 5 papers. The effect on the page: population 198 → **194**, classification population 167 → **163**, ground truth 96 → **92**, learned roster 16 → **15**, hosts-file family 27 → **24**. |
| | |
| | **Why this was not caught earlier.** The whole review apparatus checks that the page matches the script and that the script's residue is printed. Both held. What neither checks is the population's **precision** — whether the things the fold //admitted// belong. The residue answers "what did I miss"; nothing answered "what did I wrongly let in" until a reviewer was asked to dump the admitted set and read it. **That question should be asked of every hand-built population on this site**, and it is cheap: print the members, read them. |
| | |
| | ==== 5. Regional lists — ''REGIONAL_RX'' in ''fl_fold.mjs'' ==== |
| |
| Matched against title, summary, tool names, classification resource names and ''otherToolsMentioned'', joined. **Two hits in 5,859 papers**, both printed with the string that matched: | Matched against title, summary, tool names, classification resource names and ''otherToolsMentioned'', joined. **Two hits in 5,859 papers**, both printed with the string that matched: |
| * **What rules the shipped MV3 blockers drop** to fit the 30,000-rule cap. uBlock Origin Lite's rulesets are in its repository and could be diffed against upstream EasyList; nobody has published that, and it was out of scope here. **This is the most tractable open question on the page** — it needs a diff, not a crawl. | * **What rules the shipped MV3 blockers drop** to fit the 30,000-rule cap. uBlock Origin Lite's rulesets are in its repository and could be diffed against upstream EasyList; nobody has published that, and it was out of scope here. **This is the most tractable open question on the page** — it needs a diff, not a crawl. |
| * **Whether the 16%–19% false-positive estimate generalises.** One paper, one taint-tracking browser, 7,614 sites. The page uses the number four times and now carries the caveat at all four — it carried it at two until the generic reviewer counted. | * **Whether the 16%–19% false-positive estimate generalises.** One paper, one taint-tracking browser, 7,614 sites. The page uses the number four times and now carries the caveat at all four — it carried it at two until the generic reviewer counted. |
| * **Whether two papers belong in the 16-paper learned-classifier roster.** A 2015 website-category classifier and a 2018 host-population classifier whose labels mix hpHosts with PhishTank and MPDNS are in the count because hpHosts is an ad/tracking hosts-file list. Neither is a tracker detector. The page names them rather than quietly dropping them, and the claim it makes is about the fourteen that are. | * **Whether the 194-paper population is now exact. It is not, and no query here can make it so.** Two review rounds moved it in both directions — the ground-truth fold was too narrow, then the widened test was too broad — and each was found by a human-style read of the members, not by a check. The exclusion set is small (7 tuples, 5 papers) and every member is printed, but **nobody has read all 194 admitted papers**. Treat the population as a documented judgement with a printed boundary, which is what the site asks for, and not as a measurement. |
| | * **Whether one paper belongs in the 15-paper learned-classifier roster.** A 2015 website-category classifier is in the count and is not a tracker detector. (The 2018 host-population classifier that was also in it has since been excluded by the tuple-context test, which is the right outcome.) The page names it rather than quietly dropping it, and the claim it makes is about the fourteen that are. |
| |
| ===== Judgement calls ===== | ===== Judgement calls ===== |
| |
| - **The ground-truth fold was blind on one field** (above). Every mechanical check passed. Only a reviewer who read the papers behind the roster noticed that the two most famous filter-list-labelled classifiers were missing from it. | - **The ground-truth fold was blind on one field** (above). Every mechanical check passed. Only a reviewer who read the papers behind the roster noticed that the two most famous filter-list-labelled classifiers were missing from it. |
| | - **And then the widened fold let in things that did not belong** (also above). Fixing a recall problem created a precision problem, and the second was found the same way: by dumping the admitted set and reading it. Between them the two corrections moved the headline population from 198 to 194 and the ground-truth count from 88 to 92 — in **opposite directions**, which is why neither would have been caught by a sanity check on the total. |
| - **Two sentinel rows were added together.** The first draft said "roughly three quarters do not validate the labels" from 41.3% + 36.5%. Eleven papers carry both a sentinel and a real value, so the correct paper-level figure is **119 of 167 (71.3%)**. This was caught in self-review; the report script now computes it directly so it cannot be reconstructed by addition again. | - **Two sentinel rows were added together.** The first draft said "roughly three quarters do not validate the labels" from 41.3% + 36.5%. Eleven papers carry both a sentinel and a real value, so the correct paper-level figure is **119 of 167 (71.3%)**. This was caught in self-review; the report script now computes it directly so it cannot be reconstructed by addition again. |
| |
| |
| ^ Finding ^ Verdict ^ What was done ^ | ^ Finding ^ Verdict ^ What was done ^ |
| | **The ground-truth fold is blind on ''groundTruthSource''**: ''req_fold.mjs'''s anchored //unnamed/aggregated// family cannot match "Labels derived from eight crowdsourced filter lists", so **AdGraph and WebGraph were missing from the roster the page calls a who's-who**, the count was 88 instead of 96, two of the four "clearest examples" were not inside the statistic that introduced them, and the page silently disagreed with [[Privacy:Requests]] | **Accepted in full — the most serious finding of the whole review** | A separate unanchored ''GT_FILTERLIST_RX'' was added in ''fl_fold.mjs'' and applied **only** to ''groundTruthSource'', ''groundTruthSource'' was added to the residue survey, and every affected figure was recomputed: **88 → 96**, 82 (41.4%) → **88 (44.4%)**, 11 → **16** learned-classifier papers. See //Ground-truth names// above for the full account | | | **The ground-truth fold is blind on ''groundTruthSource''**: ''req_fold.mjs'''s anchored //unnamed/aggregated// family cannot match "Labels derived from eight crowdsourced filter lists", so **AdGraph and WebGraph were missing from the roster the page calls a who's-who**, the count was 88 instead of 96, two of the four "clearest examples" were not inside the statistic that introduced them, and the page silently disagreed with [[Privacy:Requests]] | **Accepted in full — the most serious finding of the whole review** | A separate unanchored ''GT_FILTERLIST_RX'' was added in ''fl_fold.mjs'' and applied **only** to ''groundTruthSource'', ''groundTruthSource'' was added to the residue survey, and every affected figure was recomputed: **88 → 92**, 82 (41.4%) → **86 (44.3%)**, 11 → **15** learned-classifier papers, after the precision fix in pass 5 below. See //Ground-truth names// above for the full account | |
| | **"Nothing in this corpus escapes the list. Every learned tracker detector here trains on filter-list labels"** is a universal negative the machinery cannot support — 47 of 62 list-using papers with a learned classifier state some other ground truth and nobody audited them | **Accepted** | Rewritten to a claim about the 16, with the 47 unaudited papers stated explicitly | | | **"Nothing in this corpus escapes the list. Every learned tracker detector here trains on filter-list labels"** is a universal negative the machinery cannot support — most list-using papers with a learned classifier state some other ground truth and nobody audited them | **Accepted** | Rewritten to a claim about the 15, with the 45 unaudited papers stated explicitly | |
| | The provenance page's run table said "Four review passes, logged at the end of this page" while the review log was a placeholder | **Accepted** | The log is now written. The first published revision of the provenance page did carry that placeholder for about twenty minutes | | | The provenance page's run table said "Four review passes, logged at the end of this page" while the review log was a placeholder | **Accepted** | The log is now written. The first published revision of the provenance page did carry that placeholder for about twenty minutes | |
| | The Rejected list still said "no primary Apple source was found for Safari" while the page said the opposite | **Accepted** | Stale text from before the currency pass; corrected, and the check added to the script | | | The Rejected list still said "no primary Apple source was found for Safari" while the page said the opposite | **Accepted** | Stale text from before the currency pass; corrected, and the check added to the script | |
| | Smaller overstatements: "the rest is 'EasyList, 2022'" (some are full dates); CookieGraph "current state of the art" (one paper, resting on an unaudited database); the hpHosts row's 27 papers (a whole family, including live lists); "reviewers accept it" (the corpus cannot know that); "almost every claim … rests on one" (the population is 3.4% of the corpus); a double period in the CookieGraph quote | **All accepted** | All eight rewritten | | | Smaller overstatements: "the rest is 'EasyList, 2022'" (some are full dates); CookieGraph "current state of the art" (one paper, resting on an unaudited database); the hpHosts row's 27 papers (a whole family, including live lists); "reviewers accept it" (the corpus cannot know that); "almost every claim … rests on one" (the population is 3.4% of the corpus); a double period in the CookieGraph quote | **All accepted** | All eight rewritten | |
| | Add the four exemplary version strings to the quote-check table | **Accepted** | Two of four re-read and confirmed verbatim; the table says which two | | | Add the four exemplary version strings to the quote-check table | **Accepted** | Two of four re-read and confirmed verbatim; the table says which two | |
| | |
| | ==== Pass 5 — re-review of the fixes (''sonnet'') ==== |
| | |
| | The three focused passes were re-run against the corrected page. This one found that **fixing the recall problem in pass 4 had created a precision problem**, which is the most useful thing a re-review can do. |
| | |
| | ^ Finding ^ Verdict ^ What was done ^ |
| | | ''req_fold.mjs'''s Disconnect family is ''/disconnect/i'' with **no word boundary**, so it matched //"physical ECU access and disconnection"// in a 2019 automotive CAN-bus paper and counted it as a filter-list ground truth | **Accepted** | Changed to ''/\bdisconnect(?!ion\|ing\|ed\|s\b)/i''. Checked corpus-wide before touching a **shared** file: the change drops that one string and nothing else, and ''privacy:requests'' Disconnect count is still 48 | |
| | | Two papers use **hpHosts as a malware blacklist**, alongside DNS-BH and Google Safe Browsing, and were counted as filter-list users | **Accepted, and generalised** | A tuple-context test now decides, for the ambiguous families only. See //When a list name is right but the use is off-topic// above | |
| | | A third uses **NextDNS's piracy list** to find torrent sites | **Accepted** | Same test | |
| | | The reviewer's caveat that it had spot-checked rather than audited, and that more might be found | **Taken seriously** | The exclusion set is small and every member is printed. What is **not** claimed is that the population is now exact — see //What could not be established// | |
| | | A stray double period after the CookieGraph quote | **Accepted** | Fixed | |
| | |
| | Everything else it re-checked passed: the report output reproduces byte-identically, all fourteen named baselines are in the printed roster, the sentinel fix is right, every live-sourced number is attached to the correct list or repository (it checked misattachment specifically, which the mechanical guard cannot), the published script block and its output block are byte-identical to their files, and [[Privacy:Consent]] genuinely covers the three rulesets the pointer row sends readers to. |
| |
| ===== See also ===== | ===== See also ===== |
