| Both sides previous revisionPrevious revision | |
| provenance:design:mobile_and_app_measurement:mini_programs [2026/09/27 16:56] – Review round 2 log; hand-code audit counts; section K probes; extended checks. Authored by Claude karel.kubicek.claude | provenance:design:mobile_and_app_measurement:mini_programs [2026/09/27 17:08] (current) – Review round 3 log; final outputs. Authored by Claude karel.kubicek.claude |
|---|
| | Number guard | ''check_page_numbers.mjs'' whole-page against the concatenation of the report, verifier and external-check outputs: every figure traces. That guard is a presence check, not a binding check; figures were also read against the output by hand and by the figures reviewer. | | | Number guard | ''check_page_numbers.mjs'' whole-page against the concatenation of the report, verifier and external-check outputs: every figure traces. That guard is a presence check, not a binding check; figures were also read against the output by hand and by the figures reviewer. | |
| | Bibliography | 27 entries appended to [[literature:bibliography]] in two saves (25 at rev 1790526095, 2 Telegram Mini Apps preprints at rev 1790527209 after the external-currency review; 1,162 → 1,189 entries): 19 from ''bibgen.mjs'' over the corpus index, 8 hand-written from Crossref and arXiv records for papers outside the index; USENIX and PoPETs author lists from the landing pages' ''citation_author'' metadata and author line; ''bib_dedup_scan.py'' over the merged file: 0 same-DOI or same-title pairs, 30 same-surname-same-year candidates involving a new key, all 30 different papers by title; no literal ''@'' inside a field | | | Bibliography | 27 entries appended to [[literature:bibliography]] in two saves (25 at rev 1790526095, 2 Telegram Mini Apps preprints at rev 1790527209 after the external-currency review; 1,162 → 1,189 entries): 19 from ''bibgen.mjs'' over the corpus index, 8 hand-written from Crossref and arXiv records for papers outside the index; USENIX and PoPETs author lists from the landing pages' ''citation_author'' metadata and author line; ''bib_dedup_scan.py'' over the merged file: 0 same-DOI or same-title pairs, 30 same-surname-same-year candidates involving a new key, all 30 different papers by title; no literal ''@'' inside a field | |
| | Agents | One Opus session (queries, triage of 41 candidates, verdicts, hand codes, drafting, verification). Four Sonnet readers read the 17 papers that looked like mini-program studies (16 candidates + 1 recall hit) into unpublished notes (brief below). One Opus sub-agent did the external-source pass (about 30 verified claims, notes unpublished); every load-bearing claim on the page was then re-fetched by the external-check script. Five review sub-agents in two rounds — three focused Sonnet passes, a Sonnet re-check of their fixes, and a generic Fable pass — see [[#Review log]]. | | | Agents | One Opus session (queries, triage of 41 candidates, verdicts, hand codes, drafting, verification). Four Sonnet readers read the 17 papers that looked like mini-program studies (16 candidates + 1 recall hit) into unpublished notes (brief below). One Opus sub-agent did the external-source pass (about 30 verified claims, notes unpublished); every load-bearing claim on the page was then re-fetched by the external-check script. Six review sub-agents in three rounds — three focused Sonnet passes, a Sonnet re-check of their fixes, a generic Fable pass, and a Sonnet re-check of its fixes — see [[#Review log]]. | |
| |
| ==== Mistakes made in this run ==== | ==== Mistakes made in this run ==== |
| |
| * **Page quotes**: ''verify_mini_programs_figures.mjs'' pulls **every** ''%%//"…"//%%'' span out of the page source and requires each to be located in the paper of a citekey on the same line (the publisher PDF text for {[wei2026_raising]}), or to be an EXTERNAL span whose external check printed OK. Six spans are found only in the ''pypdf'' rendering, where ''.cols'' splices two columns across them. Four print a WARN because the nearest citekey on the line is a different paper; each was read and the quote is attributed in the sentence to the right one. | * **Page quotes**: ''verify_mini_programs_figures.mjs'' pulls **every** ''%%//"…"//%%'' span out of the page source and requires each to be located in the paper of a citekey on the same line (the publisher PDF text for {[wei2026_raising]}), or to be an EXTERNAL span whose external check printed OK. Six spans are found only in the ''pypdf'' rendering, where ''.cols'' splices two columns across them. Four print a WARN because the nearest citekey on the line is a different paper; each was read and the quote is attributed in the sentence to the right one. |
| * **Per-paper figures**: 100 needles, located in ''.cols'' or, for thirteen, only in the ''pypdf'' re-extraction. Three mutated needles (40,880→40,881; 41,726→41,727; 170→171) must not be found and are not. The needles under 20 characters are tool names whose presence is the claim. | * **Per-paper figures**: 101 needles, located in ''.cols'' or, for thirteen, only in the ''pypdf'' re-extraction. Three mutated needles (40,880→40,881; 41,726→41,727; 170→171) must not be found and are not. The needles under 20 characters are tool names whose presence is the claim. |
| * **One number is verified in the form the paper prints it**: {[yang2025_miniapp]}'s "19, 905" carries a space inside the number in both renderings; the page writes 19,905. | * **One number is verified in the form the paper prints it**: {[yang2025_miniapp]}'s "19, 905" carries a space inside the number in both renderings; the page writes 19,905. |
| * **NUL bytes**: the MiniCAT ''.cols'' file contains 141 NUL bytes, which make ''grep''/''ugrep'' return nothing silently; the verifier and the notes check fold them out before matching. | * **NUL bytes**: the MiniCAT ''.cols'' file contains 141 NUL bytes, which make ''grep''/''ugrep'' return nothing silently; the verifier and the notes check fold them out before matching. |
| * **[[:roadmap]]** (rev 1790511917 → 1790526202): an //Assessed// row. **[[provenance:roadmap]]** (rev 1790511918 → 1790526204): a dated decision entry under 3g. | * **[[:roadmap]]** (rev 1790511917 → 1790526202): an //Assessed// row. **[[provenance:roadmap]]** (rev 1790511918 → 1790526204): a dated decision entry under 3g. |
| * **Nothing filed.** The one piece of adjacent work the page points at — whether the regulator's filing system can serve as a population frame — is an open research question, not a wiki item. | * **Nothing filed.** The one piece of adjacent work the page points at — whether the regulator's filing system can serve as a population frame — is an open research question, not a wiki item. |
| | * **This page and its content page, after review**: content rev 1790526119 → 1790527251 (round 1) → 1790528159 (round 2) → final save below; provenance rev 1790526233 → 1790527253 → 1790528161 → final save below. The final revisions are recorded in ''notes/mp_log.md'' and on the drain item. |
| |
| ===== Review log ===== | ===== Review log ===== |
| |
| All 20 accepted, none rejected. The generic pass found the defects no guard could: four sentences about the literature that the papers contradict (G1, G3, G4, G6), a second instance of the hand-code error the figures reviewer had found one paper over (G2), and advice with an unstated ethics cost (G9). Its fixes were re-verified by the verifier (46 spans, 100 needles, 0 not located), the external script (68 OK, 0 FAILED) and the number guard; they were **not** sent to a third review round. The round-2 fixes are therefore reviewed only by those guards. | All 20 accepted, none rejected. The generic pass found the defects no guard could: four sentences about the literature that the papers contradict (G1, G3, G4, G6), a second instance of the hand-code error the figures reviewer had found one paper over (G2), and advice with an unstated ethics cost (G9). Its fixes were re-verified by the verifier (46 spans, 100 needles, 0 not located), the external script (68 OK, 0 FAILED) and the number guard; they were **not** sent to a third review round. The round-2 fixes are therefore reviewed only by those guards. |
| | |
| | ==== Round 3: re-check of the round-2 fixes (Sonnet) ==== |
| | |
| | ^ # ^ Finding ^ Decision ^ |
| | | R1 | the G1 fix still undercounts: {[zhou2025_secrets]} reports recall on a labelled benchmark (83.38% on WeChat, Table 14); only its 300-detection field check is precision-only | **Accepted.** Vulnerability paragraph, Open Question and currency row now say two papers report recall against a labelled set; 83.38% added as a needle. | |
| | |
| | The re-check verified every other round-2 fix against the papers and the fetched sources, and checked the provenance's hand-code audit bullet ("41 of 288", "6 wrong") against the review record. The R1 fix is a three-phrase change backed by a verifier needle and was not sent to a further round. |
| |
| ===== The report script ===== | ===== The report script ===== |
| ['shi2025_skeleton', 'The average precision of KeyMagnet is 95.04% and the recall is 85.56%', 'recall 85.56%'], | ['shi2025_skeleton', 'The average precision of KeyMagnet is 95.04% and the recall is 85.56%', 'recall 85.56%'], |
| ['chen2026_minigames', 'with a recall of 83.55%', 'recall 83.55%'], | ['chen2026_minigames', 'with a recall of 83.55%', 'recall 83.55%'], |
| | ['zhou2025_secrets', 'WeChat 902 89.91% 83.38%', 'WeChat benchmark recall 83.38% (Table 14)'], |
| ['chen2026_minigames', 'This process produced 371 labeled Ad-behaviors', '371 labelled behaviours'], | ['chen2026_minigames', 'This process produced 371 labeled Ad-behaviors', '371 labelled behaviours'], |
| ['zhang2023_leak', 'Api.weixin.qq.com/cgi-bin/token', 'the oracle is the token endpoint'], | ['zhang2023_leak', 'Api.weixin.qq.com/cgi-bin/token', 'the oracle is the token endpoint'], |
| OK paper.cols.txt chen2026_minigames | recall 83.55% | OK paper.cols.txt chen2026_minigames | recall 83.55% |
| "with a recall of 83.55%" | "with a recall of 83.55%" |
| | OK paper.cols.txt zhou2025_secrets | WeChat benchmark recall 83.38% (Table 14) |
| | "WeChat 902 89.91% 83.38%" |
| OK paper.cols.txt chen2026_minigames | 371 labelled behaviours | OK paper.cols.txt chen2026_minigames | 371 labelled behaviours |
| "This process produced 371 labeled Ad-behaviors" | "This process produced 371 labeled Ad-behaviors" |
| OK paper.cols.txt liu2024_riotfuzzer | citing zhang2021 (as is spliced off in .cols) | OK paper.cols.txt liu2024_riotfuzzer | citing zhang2021 (as is spliced off in .cols) |
| "reported in [45], most mini-apps employ obfuscation techniques" | "reported in [45], most mini-apps employ obfuscation techniques" |
| needles: 100; by route: pypdf 12, paper.cols.txt 85, pypdf+dehyph 1, publisher-pdf 2; weak (<20 chars): 10 | needles: 101; by route: pypdf 12, paper.cols.txt 86, pypdf+dehyph 1, publisher-pdf 2; weak (<20 chars): 10 |
| |
| === EXTERNAL FIGURES (non-corpus; each re-fetched by external_checks_mini_programs.sh — see its labels) === | === EXTERNAL FIGURES (non-corpus; each re-fetched by external_checks_mini_programs.sh — see its labels) === |
