| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| provenance:programming:crawler:llm_agents [2026/09/17 07:24] – Markup rendering sweep: wrap escapes, CLI flag typography, and heading quotes. Authored by Claude. karel.kubicek.claude | provenance:programming:crawler:llm_agents [2026/09/21 13:43] (current) – Quote-check refresh 2026-09-21: regenerated llm_agents_quotecheck output block; 9 below threshold -> 2 rescued + 7 below in both. Authored by Claude karel.kubicek.claude |
|---|
| exact: 3 | exact: 3 |
| partial: 3 (>=60% of 5-word windows) | partial: 3 (>=60% of 5-word windows) |
| below: 9 <- read these by hand | rescued from the PDF: 2 |
| | below threshold in BOTH renderings: 7 <- read these by hand |
| no full text on disk: 0 | no full text on disk: 0 |
| |
| IEEE-SP/2026/investigating-the-impact-of-dark-patterns-on-llm-based-web-agents | IEEE-SP/2026/investigating-the-impact-of-dark-patterns-on-llm-based-web-agents |
| VisualWebArena: 21% of 14 windows — "we evaluate four commercial agents (Skyvern, DoBrowser, BrowserUse, and Agent-E) and two academic web agents (WebArena and VisualWebArena)" | VisualWebArena: 21% of 14 windows — "we evaluate four commercial agents (Skyvern, DoBrowser, BrowserUse, and Agent-E) and two academic web agents (WebArena and VisualWebArena)" |
| WWW/2026/mcp-vs-rag-vs-nlweb-vs-html-a-comparison-of-the-effectiveness-and-efficiency-of | |
| AgentLab: 44% of 9 windows — "The agent uses the AgentLab library and is executed within the BrowserGym framework." | |
| WWW/2026/mcp-vs-rag-vs-nlweb-vs-html-a-comparison-of-the-effectiveness-and-efficiency-of | |
| BrowserGym: 44% of 9 windows — "The agent uses the AgentLab library and is executed within the BrowserGym framework." | |
| USENIX/2026/the-state-of-passkeys-studying-the-adoption-and-security-of-passkeys-on-the-web | USENIX/2026/the-state-of-passkeys-studying-the-adoption-and-security-of-passkeys-on-the-web |
| Browser Use: 30% of 10 windows — "We built a prototype tool using Playwright and the agentic LLM-based Browser Use framework." | Browser Use: 30% of 10 windows — "We built a prototype tool using Playwright and the agentic LLM-based Browser Use framework." |
| </code> | </code> |
| |
| **All nine below-threshold quotes were then read by hand against | **Re-run 2026-09-21 with the PDF fallback: 2 of the 9 are rescued, 7 are below threshold in both renderings.** See //Quote-check refresh, 2026-09-21// at the foot of this page. The two rescued are the WWW/2026 //MCP vs RAG vs NLWeb vs HTML// pair (''AgentLab'', ''BrowserGym''), which is exactly right — their failure is a column splice, and an independent ''pypdf'' rendering does not splice that sentence. The other seven fail because the extraction dropped bracketed citation markers from its own string, and no rendering of the PDF can repair that. |
| | |
| | **All nine of the then-below-threshold quotes were read by hand against |
| ''data/fulltext/<year>/<venue>/<slug>/paper.cols.txt'', and all nine are present in the | ''data/fulltext/<year>/<venue>/<slug>/paper.cols.txt'', and all nine are present in the |
| source.** Each fails for one of two mechanical reasons — the extraction dropped bracketed | source.** Each fails for one of two mechanical reasons — the extraction dropped bracketed |
| | 2 | **"the versions the other two papers used cannot be recovered at all" is false.** Both release artefacts; nobody looked in the lock files. | **Accepted.** The supported claim is "neither paper //states// a version". Reworded, and it now points at the artefacts. | | | 2 | **"the versions the other two papers used cannot be recovered at all" is false.** Both release artefacts; nobody looked in the lock files. | **Accepted.** The supported claim is "neither paper //states// a version". Reworded, and it now points at the artefacts. | |
| | 3 | **SeeAct and Mind2Web are not Ersoy's subjects, and SeeAct is nowhere in the corpus** — zero probe matches across all 5,869 files. | **Accepted.** The table row was split: WebArena/VisualWebArena keeps the Ersoy attribution; Mind2Web and SeeAct are marked as used by no paper in this corpus, with the probe counts. | | | 3 | **SeeAct and Mind2Web are not Ersoy's subjects, and SeeAct is nowhere in the corpus** — zero probe matches across all 5,869 files. | **Accepted.** The table row was split: WebArena/VisualWebArena keeps the Ersoy attribution; Mind2Web and SeeAct are marked as used by no paper in this corpus, with the probe counts. | |
| | 4 | **"run-to-run variance" is promised three times and retracted once.** Ramesh measured pass@k on benchmark tasks; nobody measured prevalence spread on a site sample, which Open Questions says. | **Accepted.** All three mentions — the lead WRAP, [[programming:crawler]]'s new section and [[start]] — now say "on benchmark tasks" or "what nobody has measured", and the lead points at Open Questions. | | | 4 | **"run-to-run variance" is promised three times and retracted once.** Ramesh measured pass@k on benchmark tasks; nobody measured prevalence spread on a site sample, which Open Questions says. | **Accepted.** All three mentions — the lead WRAP, [[programming:crawler]]'s new section and [[:start]] — now say "on benchmark tasks" or "what nobody has measured", and the lead points at Open Questions. | |
| | 5 | **The headline count and the "same wire" claim both break on the fifth paper.** It says "a named agent framework", but the fifth is a vendor computer-use API the page's own table calls "not a framework"; and the intro's "still drive Chromium through one of the control channels" is exactly what is unknown for that paper. | **Accepted.** The headline reads "a named agent framework or a vendor computer-use API"; the intro now separates the open-source frameworks (which do use a known channel) from the vendor APIs (screenshots and coordinates, wire supplied by you). [[programming:crawler]]'s mirror sentence got the same split. | | | 5 | **The headline count and the "same wire" claim both break on the fifth paper.** It says "a named agent framework", but the fifth is a vendor computer-use API the page's own table calls "not a framework"; and the intro's "still drive Chromium through one of the control channels" is exactly what is unknown for that paper. | **Accepted.** The headline reads "a named agent framework or a vendor computer-use API"; the intro now separates the open-source frameworks (which do use a known channel) from the vendor APIs (screenshots and coordinates, wire supplied by you). [[programming:crawler]]'s mirror sentence got the same split. | |
| | 6 | "Three things that table is for" is followed by five. | **Accepted.** Two bullets had been added after the count was written. Now five. | | | 6 | "Three things that table is for" is followed by five. | **Accepted.** Two bullets had been added after the count was written. Now five. | |
| |
| Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving. | Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving. |
| | |
| | ===== Link-hygiene sweep, 2026-09-17 ===== |
| | |
| | The fresh rendered-DOM sweep found this page's bare ''start'' link resolved to ''provenance:programming:crawler:start''. It is now root-anchored; the post-save DOM was re-checked for red links. No figures or citations changed. |
| | |
| | ===== Quote-check refresh, 2026-09-21 ===== |
| | |
| | The 2026-09-04 ''cols''-vs-PDF audit on [[:provenance:literature:corpus]] showed that 73.1% of evidence quotes that cannot be located in ''paper.cols.txt'' **are** present in an independent ''pypdf'' rendering of the same ''paper.pdf''. ''llm_agents_quotecheck.mjs'' already carried the fallback; what was stale was this page's copy of its output. ''scripts/llm_agents_quotecheck-output.txt'' was regenerated by re-running the checker and §6's block replaced from it — the block was byte-identical to the old artifact before the re-run, and is byte-identical to the new one after it. |
| | |
| | ^ Figure ^ Was ^ Is ^ Why ^ |
| | | quotes checked | 15 | 15 | population unchanged | |
| | | exact | 3 | 3 | unchanged | |
| | | partial (≥60% of 5-word windows) | 3 | 3 | unchanged | |
| | | rescued from the PDF | — | **2** | the WWW/2026 ''AgentLab''/''BrowserGym'' pair, a column splice | |
| | | below | **9** | **7** (in both renderings) | 9 = 2 + 7 exactly | |
| | |
| | **15 quotes is a tiny population — read the count, not a rate.** Seven of 15 below threshold in both renderings is 47%, and quoting that as a failure rate would be badly wrong: all nine were read by hand and all nine are present. The seven are one sentence quoted six times (the IEEE S&P 2026 agent list, which the extraction stripped six citation markers out of) plus one USENIX 2026 sentence. **Count papers, not tuples:** the seven are two papers. |
| | |
| | **Scope of this edit.** §6's output block and the paragraph under it, plus the content page's quote bullet. ''report_llm_agents.mjs'', ''verify_llm_agents_figures.mjs'' and the folds were **not** re-run in this pass and no other figure or citation was touched. |
| | |
| | ^ Item ^ Value ^ |
| | | Date | 2026-09-21, unsupervised | |
| | | Command | ''%%node scripts/llm_agents_quotecheck.mjs > scripts/llm_agents_quotecheck-output.txt%%'' | |
| | | Artifacts | ''scripts/llm_agents_quotecheck-output.txt'' (regenerated; the previous copy kept as ''…-output.txt.bak0921'') | |
| | | Script changes | none | |
| | | Reviewers | one ''sonnet'' figures-vs-script pass over this page and [[:programming:crawler:llm_agents]] | |
| | | Pages saved | this page, [[:programming:crawler:llm_agents]] | |
| |
