| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| provenance:design:mobile_and_app_measurement [2026/08/27 07:40] – Add the generic reviewer's log (16 findings, all accepted, including that this log's one rejection was itself wrong); correct the nine stale dates that survived in the section-8 verified table; compute the objection enumeration and the traffic-pinning int karel.kubicek.claude | provenance:design:mobile_and_app_measurement [2026/09/17 10:45] (current) – Record consentAction schema clarification; Authored by Claude karel.kubicek.claude |
|---|
| | [[programming:traffic_files]] | what to do with captured traffic | assumes you captured it; the whole certificate-pinning section of this page is about whether you did | | | [[programming:traffic_files]] | what to do with captured traffic | assumes you captured it; the whole certificate-pinning section of this page is about whether you did | |
| |
| ==== Why ''design:'' and not a new namespace ==== | ==== Why design: and not a new namespace ==== |
| |
| The item as specified said "propose the namespace placement before creating it". The decision, and it was a decision rather than an obvious call: | The item as specified said "propose the namespace placement before creating it". The decision, and it was a decision rather than an obvious call: |
| ==== Reachability ==== | ==== Reachability ==== |
| |
| Nothing linked here, so a bullet was added to [[start]] under //Research design// in the same sitting. Without it the page is invisible. | Nothing linked here, so a bullet was added to [[:start]] under //Research design// in the same sitting. Without it the page is invisible. |
| |
| ===== 3. Populations and denominators ===== | ===== 3. Populations and denominators ===== |
| Four reviewers, all told explicitly that the author's context may not be exhaustive, and all handed the page text, the report script, its unedited output, the probe and quote-check outputs, and these provenance notes. The three focused reviewers ran in parallel first; the generic reviewer ran afterwards on the corrected page — and **caught that this log's one rejection was itself wrong** (see reviewer 4). | Four reviewers, all told explicitly that the author's context may not be exhaustive, and all handed the page text, the report script, its unedited output, the probe and quote-check outputs, and these provenance notes. The three focused reviewers ran in parallel first; the generic reviewer ran afterwards on the corrected page — and **caught that this log's one rejection was itself wrong** (see reviewer 4). |
| |
| ==== Reviewer 1 — figures vs the report script (''model: sonnet'') ==== | ==== Reviewer 1 — figures vs the report script (model: sonnet) ==== |
| |
| Re-ran all three scripts live and confirmed the committed ''scripts/*-output.txt'' files are byte-identical to a fresh run. Checked roughly 400 individual figures and percentages cell by cell. | Re-ran all three scripts live and confirmed the committed ''scripts/*-output.txt'' files are byte-identical to a fresh run. Checked roughly 400 individual figures and percentages cell by cell. |
| The tenth, corpus-wide, is ''CCS/2019/watching-you-watch-the-tracking-ecosystem…'' — ''appAnalysis = false''. | The tenth, corpus-wide, is ''CCS/2019/watching-you-watch-the-tracking-ecosystem…'' — ''appAnalysis = false''. |
| |
| ==== Reviewer 2 — citations and quotes (''model: sonnet'') ==== | ==== Reviewer 2 — citations and quotes (model: sonnet) ==== |
| |
| Independently re-derived every quoted figure from ''paper.cols.txt'' rather than trusting ''quotecheck_mobile-output.txt''; resolved all 11 DOIs against Crossref; re-fetched all six USENIX/NDSS author lists and the PoPETs one from the venue landing pages; duplicate-scanned the 17 new entries against the 488 live ones on four axes (key string, DOI, URL, and normalised title with a fuzzy threshold). | Independently re-derived every quoted figure from ''paper.cols.txt'' rather than trusting ''quotecheck_mobile-output.txt''; resolved all 11 DOIs against Crossref; re-fetched all six USENIX/NDSS author lists and the PoPETs one from the venue landing pages; duplicate-scanned the 17 new entries against the 488 live ones on four axes (key string, DOI, URL, and normalised title with a fuzzy threshold). |
| | The ''pradeep2022_comparative'' weak-cipher result holds only for the **Common Android** dataset; the paper's general finding is that pinning apps have //better// cipher security. | **Confirmation of a deliberate choice.** The figure was in an early draft, was cut before publication precisely because it needed this scoping, and survives only on this provenance page (§7 exemplars) where it is correctly labelled "Common Android". The content page makes no cipher claim. | None. | | | The ''pradeep2022_comparative'' weak-cipher result holds only for the **Common Android** dataset; the paper's general finding is that pinning apps have //better// cipher security. | **Confirmation of a deliberate choice.** The figure was in an early draft, was cut before publication precisely because it needed this scoping, and survives only on this provenance page (§7 exemplars) where it is correctly labelled "Common Android". The content page makes no cipher claim. | None. | |
| |
| ==== Reviewer 3 — external currency (''model: sonnet'') ==== | ==== Reviewer 3 — external currency (model: sonnet) ==== |
| |
| 43 items re-fetched independently, plus a dead-link sweep of every URL on the page. **This reviewer earned its slot: it found a systematic bug that had put nine wrong dates on the page, and it found the root cause pattern before I did.** | 43 items re-fetched independently, plus a dead-link sweep of every URL on the page. **This reviewer earned its slot: it found a systematic bug that had put nine wrong dates on the page, and it found the root cause pattern before I did.** |
| Also caught in the same pass: ''apkeep'' reads **2026-05-05** from ''releases.atom'' but **2026-04-30** from the API's ''published_at''. The tag page is authoritative, so the page keeps 2026-04-30 and the script prints a note about the divergence. | Also caught in the same pass: ''apkeep'' reads **2026-05-05** from ''releases.atom'' but **2026-04-30** from the API's ''published_at''. The tag page is authoritative, so the page keeps 2026-04-30 and the script prints a note about the divergence. |
| |
| ==== Reviewer 4 — generic, no checklist (''model: fable'') ==== | ==== Reviewer 4 — generic, no checklist (model: fable) ==== |
| |
| Ran last, on the page as corrected by the other three. It re-derived contested figures itself and **computed two intersections the page had only inferred**, which is how it found the two worst remaining defects. Every finding was accepted; none was rejected. | Ran last, on the page as corrected by the other three. It re-derived contested figures itself and **computed two intersections the page had only inferred**, which is how it found the two worst remaining defects. Every finding was accepted; none was rejected. |
| ^ # ^ Finding ^ Action ^ | ^ # ^ Finding ^ Action ^ |
| | 1 | **This log's one rejection was wrong.** The ''objection'' count was rejected using a corpus-wide denominator instead of ''appAnalysis''. | Reviewer 1's finding reinstated as **Accepted**; page now says 7 of 9; the enumeration is computed by the report script with the ''appAnalysis'' flag shown. See finding 2 above. | | | 1 | **This log's one rejection was wrong.** The ''objection'' count was rejected using a corpus-wide denominator instead of ''appAnalysis''. | Reviewer 1's finding reinstated as **Accepted**; page now says 7 of 9; the enumeration is computed by the report script with the ''appAnalysis'' flag shown. See finding 2 above. | |
| | 2 | **§8's "Verified" table still carried the nine pre-fix dates** that reviewer 3 had proved wrong, under a heading claiming verification — so this page contradicted the page it backs. Reviewer 4 named this as my own recorded failure mode ("a fix survives in the provenance log"). | All nine rows corrected in place with the new values in bold, plus a ''<WRAP important>'' at the head of §8 saying "'fetched, not recalled' was true; 'parsed correctly' was not". | | | 2 | **§8's "Verified" table still carried the nine pre-fix dates** that reviewer 3 had proved wrong, under a heading claiming verification — so this page contradicted the page it backs. Reviewer 4 named this as my own recorded failure mode ("a fix survives in the provenance log"). | All nine rows corrected in place with the new values in bold, plus a ''%%<WRAP important>%%'' at the head of §8 saying "'fetched, not recalled' was true; 'parsed correctly' was not". | |
| | 3 | **§11's preamble said the generic reviewer "ran afterwards", three subsections above a "Not run."** | Preamble corrected; this subsection replaced. | | | 3 | **§11's preamble said the generic reviewer "ran afterwards", three subsections above a "Not run."** | Preamble corrected; this subsection replaced. | |
| | 4 | "every large Android study in the corpus uses AndroZoo, an unofficial scraper, or a mirror" is a **false universal** — of the 96 papers with a ≥100,000-app population, sources include vendor telemetry (NortonLifeLock, Palo Alto Networks), volunteer capture (Lumen) and inherited datasets (Drebin, Andrubis, the Gamba preinstalled-app set). | Rewritten with the five routes and the 96-paper computation, which ''report_mobile.mjs'' now prints with the full list of non-store sources. | | | 4 | "every large Android study in the corpus uses AndroZoo, an unofficial scraper, or a mirror" is a **false universal** — of the 96 papers with a ≥100,000-app population, sources include vendor telemetry (NortonLifeLock, Palo Alto Networks), volunteer capture (Lumen) and inherited datasets (Drebin, Andrubis, the Gamba preinstalled-app set). | Rewritten with the five routes and the 96-paper computation, which ''report_mobile.mjs'' now prints with the full list of non-store sources. | |
| | 7 | Three small false statements: §9 claimed the content page "reports the per-year table" (it does not); §7 said "Two findings" above three bullets; §15 said "the one citekey" where there are four. | All three corrected. | | | 7 | Three small false statements: §9 claimed the content page "reports the per-year table" (it does not); §7 said "Two findings" above three bullets; §15 said "the one citekey" where there are four. | All three corrected. | |
| | 8 | **No "What to Read First" section**, though ''design:longitudinal'' — the stated model — has one and "which papers to read first" is in the page's remit. The recommendations existed but were buried in mid-prose. | Section added near the top, five papers with the methodological reason for each, plus the two systematic reviews. | | | 8 | **No "What to Read First" section**, though ''design:longitudinal'' — the stated model — has one and "which papers to read first" is in the page's remit. The recommendations existed but were buried in mid-prose. | Section added near the top, five papers with the methodological reason for each, plus the two systematic reviews. | |
| | 9 | **The page never says how to see iOS traffic**, and does not admit the gap: step 1 of the pinning section is entirely Android, the pipeline tip is Android-only without saying so, and the only iOS bypass tool named is marked unmaintained with no alternative. | A ''<WRAP info>'' now scopes steps 1–3 as Android and gives the iOS route (CA profile with full trust, then code injection on a jailbroken device), pointing at {[pradeep2022_comparative]} as the worked example. | | | 9 | **The page never says how to see iOS traffic**, and does not admit the gap: step 1 of the pinning section is entirely Android, the pipeline tip is Android-only without saying so, and the only iOS bypass tool named is marked unmaintained with no alternative. | A ''%%<WRAP info>%%'' now scopes steps 1–3 as Android and gives the iOS route (CA profile with full trust, then code injection on a jailbroken device), pointing at {[pradeep2022_comparative]} as the worked example. | |
| | 10 | Three practitioner staples missing: **on-device VPN capture** (the no-root route, present in the fold residue and only mentioned on the page as a dead data source), **per-app traffic attribution** (a proxy sees the OS and Play Services too), and **QUIC** (bypasses a TCP proxy; 14 of 529 papers mention it). | New subsection "Three things that will bite you that no paper warns you about", with three new probes behind it. | | | 10 | Three practitioner staples missing: **on-device VPN capture** (the no-root route, present in the fold residue and only mentioned on the page as a dead data source), **per-app traffic attribution** (a proxy sees the OS and Play Services too), and **QUIC** (bypasses a TCP proxy; 14 of 529 papers mention it). | New subsection "Three things that will bite you that no paper warns you about", with three new probes behind it. | |
| | 11 | "the one the corpus's recent traffic-measurement papers converge on" overstates: of 191 papers since 2023, **16** use Frida+mitmproxy and **3** the full named stack. | Reworded to "built from the components recent papers most often combine", with both figures given and the convergence claim withdrawn. The report script now computes them. | | | 11 | "the one the corpus's recent traffic-measurement papers converge on" overstates: of 191 papers since 2023, **16** use Frida+mitmproxy and **3** the full named stack. | Reworded to "built from the components recent papers most often combine", with both figures given and the convergence claim withdrawn. The report script now computes them. | |
| | Main agent | Claude (Opus 5) — all queries, the report script, the fold, the probes, the quote check, the page text, and independent re-verification of every external claim | | | Main agent | Claude (Opus 5) — all queries, the report script, the fold, the probes, the quote check, the page text, and independent re-verification of every external claim | |
| | Sub-agents | one ''sonnet'' external-currency sweep (which itself fanned out to seven sub-agents, and whose four wrong claims are in §8); three ''sonnet'' reviewers spawned, **all three returned**; the ''fable'' generic pass was not run (§11) | | | Sub-agents | one ''sonnet'' external-currency sweep (which itself fanned out to seven sub-agents, and whose four wrong claims are in §8); three ''sonnet'' reviewers spawned, **all three returned**; the ''fable'' generic pass was not run (§11) | |
| | Write path | ''node scripts/dw.mjs put'' over JSON-RPC, with ''--if-rev'' on every save | | | Write path | ''node scripts/dw.mjs put'' over JSON-RPC, with ''%%--if-rev%%'' on every save | |
| | Stability figures | the ~20% free-text agreement and 0.9% unlocatable-quote figures quoted from ''data/extract/README.md'' were measured on the **older, smaller corpus** and have not been re-measured. Treated as the right order of magnitude. | | | Stability figures | the ~20% free-text agreement and 0.9% unlocatable-quote figures quoted from ''data/extract/README.md'' were measured on the **older, smaller corpus** and have not been re-measured. Treated as the right order of magnitude. | |
| | Mistakes caught in review of my own work | (1) the item brief's figures were stale and were re-derived rather than reused; (2) the first probe pass was case-insensitive and over-counted Xposed 6.4×; (3) the first draft published "218 (46.2%) report no validation of the classifier at all", which conflated //papers with at least one unvalidated classifier// with //papers with no validation at all// — the correct figures are 79 (16.7%) with none, 178 (37.7%) with all validated, and 393 (83.3%) with at least one validated, and the report script now computes all three; (4) the first draft obtained "199 (37.6%)" for ''ethics.reviewOutcome'' and "341 (64.5%)" for ''artifacts.availability'' by **subtracting the sentinel count from 529**, which overstates both — ''ethics'' is ''null'' on 50 of the 529 papers and ''artifacts'' on 18, so //stated + sentinel-only ≠ 529//. The correct figures are **149 (28.2%)** and **323 (61.1%)**; the report script now prints all three components so nobody has to subtract, and the number guard (§4) was written after this. (5) the ''frida-ios-dump'' fold landed in the wrong family until a smoke test caught it. | | | Mistakes caught in review of my own work | (1) the item brief's figures were stale and were re-derived rather than reused; (2) the first probe pass was case-insensitive and over-counted Xposed 6.4×; (3) the first draft published "218 (46.2%) report no validation of the classifier at all", which conflated //papers with at least one unvalidated classifier// with //papers with no validation at all// — the correct figures are 79 (16.7%) with none, 178 (37.7%) with all validated, and 393 (83.3%) with at least one validated, and the report script now computes all three; (4) the first draft obtained "199 (37.6%)" for ''ethics.reviewOutcome'' and "341 (64.5%)" for ''artifacts.availability'' by **subtracting the sentinel count from 529**, which overstates both — ''ethics'' is ''null'' on 50 of the 529 papers and ''artifacts'' on 18, so //stated + sentinel-only ≠ 529//. The correct figures are **149 (28.2%)** and **323 (61.1%)**; the report script now prints all three components so nobody has to subtract, and the number guard (§4) was written after this. (5) the ''frida-ios-dump'' fold landed in the wrong family until a smoke test caught it. | |
| statefulness stated 10/115 stateful=5 stateless=5 | statefulness stated 10/115 stateful=5 stateless=5 |
| headless stated 12/115 headless=8 headful=4 | headless stated 12/115 headless=8 headful=4 |
| consentAction stated 31/115 no-interaction=29 accept-all=2 | consentAction (schema field populated; not a paper claim) stated 31/115 no-interaction=29 accept-all=2 |
| interactionDepth stated 74/115 single-target-page=40 landing-page-only=17 deep-crawl=14 landing-plus-subpages=3 | interactionDepth stated 74/115 single-target-page=40 landing-page-only=17 deep-crawl=14 landing-plus-subpages=3 |
| authentication stated 71/115 none=60 account-registration=5 manual-login=4 automated-login=1 sso=1 | authentication stated 71/115 none=60 account-registration=5 manual-login=4 automated-login=1 sso=1 |
| | Naming | mirrors the content page id under ''provenance:'', unchanged, so the mapping needs no lookup and ''sitemap.mjs'' groups it with the other provenance pages. | | | Naming | mirrors the content page id under ''provenance:'', unchanged, so the mapping needs no lookup and ''sitemap.mjs'' groups it with the other provenance pages. | |
| | Reachability | linked from the content page's //Methodology and limitations of these figures// section; links back below. | | | Reachability | linked from the content page's //Methodology and limitations of these figures// section; links back below. | |
| | Publication order | [[literature:bibliography]] (17 entries) → [[design:mobile_and_app_measurement]] → this page → [[start]]. | | | Publication order | [[literature:bibliography]] (17 entries) → [[design:mobile_and_app_measurement]] → this page → [[:start]]. | |
| |
| [[design:mobile_and_app_measurement|← back to the content page]] · [[literature:corpus|corpus-level provenance]] | [[design:mobile_and_app_measurement|← back to the content page]] · [[literature:corpus|corpus-level provenance]] |
| | |
| | ===== Markup sweep, 2026-09-17 ===== |
| | |
| | Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving. |
| | |
| | ===== Link-hygiene sweep, 2026-09-17 ===== |
| | |
| | The fresh rendered-DOM sweep found this page's two bare ''start'' links resolved to ''provenance:design:start''. They are now root-anchored; the post-save DOM was re-checked for red links. No figures or citations changed. |
| | |
| | ===== ConsentAction audit propagation, 2026-09-17 ===== |
| | |
| | **What was wrong.** The mobile/app report's appAnalysis diagnostic said ''consentAction stated 31/115'' without the schema-vs-paper-claim qualification. The 31/115 figure is a field-population diagnostic for the app subset, not an audited claim count. |
| | |
| | **Command and real output.** ''node scripts/report_mobile.mjs'' → ''consentAction (schema field populated; not a paper claim) stated 31/115; no-interaction=29 accept-all=2''. ''node scripts/consent_action_noninteraction_audit.mjs'' → corpus-wide **55/1,120 = 4.9%** audited paper claims and **279/313 = 89.1%** unsupported ''no-interaction'' labels. |
| | |
| | **What changed.** The embedded report output now labels the 31/115 diagnostic as schema field population; the corpus-wide audit result is not substituted into this appAnalysis subset. |
| | |
| | **Review.** One **model: 'sonnet'** figures-vs-script pass re-ran ''report_mobile.mjs'' and checked the edited diagnostic and the adjacent statefulness/headless/interactionDepth/authentication diagnostics. No mismatch was found. No citation pass was needed: no citekey or quoted claim changed. |
| | |
| | **Rejected finding.** The 31/115 appAnalysis diagnostic was not replaced by the corpus-wide 55/1,120 result because those are different populations. |
| |
