| Both sides previous revisionPrevious revision | |
| provenance:programming:crawler:llm_agents [2026/09/17 08:13] – Qualify root start link; Authored by Claude karel.kubicek.claude | provenance:programming:crawler:llm_agents [2026/09/21 13:43] (current) – Quote-check refresh 2026-09-21: regenerated llm_agents_quotecheck output block; 9 below threshold -> 2 rescued + 7 below in both. Authored by Claude karel.kubicek.claude |
|---|
| exact: 3 | exact: 3 |
| partial: 3 (>=60% of 5-word windows) | partial: 3 (>=60% of 5-word windows) |
| below: 9 <- read these by hand | rescued from the PDF: 2 |
| | below threshold in BOTH renderings: 7 <- read these by hand |
| no full text on disk: 0 | no full text on disk: 0 |
| |
| IEEE-SP/2026/investigating-the-impact-of-dark-patterns-on-llm-based-web-agents | IEEE-SP/2026/investigating-the-impact-of-dark-patterns-on-llm-based-web-agents |
| VisualWebArena: 21% of 14 windows — "we evaluate four commercial agents (Skyvern, DoBrowser, BrowserUse, and Agent-E) and two academic web agents (WebArena and VisualWebArena)" | VisualWebArena: 21% of 14 windows — "we evaluate four commercial agents (Skyvern, DoBrowser, BrowserUse, and Agent-E) and two academic web agents (WebArena and VisualWebArena)" |
| WWW/2026/mcp-vs-rag-vs-nlweb-vs-html-a-comparison-of-the-effectiveness-and-efficiency-of | |
| AgentLab: 44% of 9 windows — "The agent uses the AgentLab library and is executed within the BrowserGym framework." | |
| WWW/2026/mcp-vs-rag-vs-nlweb-vs-html-a-comparison-of-the-effectiveness-and-efficiency-of | |
| BrowserGym: 44% of 9 windows — "The agent uses the AgentLab library and is executed within the BrowserGym framework." | |
| USENIX/2026/the-state-of-passkeys-studying-the-adoption-and-security-of-passkeys-on-the-web | USENIX/2026/the-state-of-passkeys-studying-the-adoption-and-security-of-passkeys-on-the-web |
| Browser Use: 30% of 10 windows — "We built a prototype tool using Playwright and the agentic LLM-based Browser Use framework." | Browser Use: 30% of 10 windows — "We built a prototype tool using Playwright and the agentic LLM-based Browser Use framework." |
| </code> | </code> |
| |
| **All nine below-threshold quotes were then read by hand against | **Re-run 2026-09-21 with the PDF fallback: 2 of the 9 are rescued, 7 are below threshold in both renderings.** See //Quote-check refresh, 2026-09-21// at the foot of this page. The two rescued are the WWW/2026 //MCP vs RAG vs NLWeb vs HTML// pair (''AgentLab'', ''BrowserGym''), which is exactly right — their failure is a column splice, and an independent ''pypdf'' rendering does not splice that sentence. The other seven fail because the extraction dropped bracketed citation markers from its own string, and no rendering of the PDF can repair that. |
| | |
| | **All nine of the then-below-threshold quotes were read by hand against |
| ''data/fulltext/<year>/<venue>/<slug>/paper.cols.txt'', and all nine are present in the | ''data/fulltext/<year>/<venue>/<slug>/paper.cols.txt'', and all nine are present in the |
| source.** Each fails for one of two mechanical reasons — the extraction dropped bracketed | source.** Each fails for one of two mechanical reasons — the extraction dropped bracketed |
| |
| The fresh rendered-DOM sweep found this page's bare ''start'' link resolved to ''provenance:programming:crawler:start''. It is now root-anchored; the post-save DOM was re-checked for red links. No figures or citations changed. | The fresh rendered-DOM sweep found this page's bare ''start'' link resolved to ''provenance:programming:crawler:start''. It is now root-anchored; the post-save DOM was re-checked for red links. No figures or citations changed. |
| | |
| | ===== Quote-check refresh, 2026-09-21 ===== |
| | |
| | The 2026-09-04 ''cols''-vs-PDF audit on [[:provenance:literature:corpus]] showed that 73.1% of evidence quotes that cannot be located in ''paper.cols.txt'' **are** present in an independent ''pypdf'' rendering of the same ''paper.pdf''. ''llm_agents_quotecheck.mjs'' already carried the fallback; what was stale was this page's copy of its output. ''scripts/llm_agents_quotecheck-output.txt'' was regenerated by re-running the checker and §6's block replaced from it — the block was byte-identical to the old artifact before the re-run, and is byte-identical to the new one after it. |
| | |
| | ^ Figure ^ Was ^ Is ^ Why ^ |
| | | quotes checked | 15 | 15 | population unchanged | |
| | | exact | 3 | 3 | unchanged | |
| | | partial (≥60% of 5-word windows) | 3 | 3 | unchanged | |
| | | rescued from the PDF | — | **2** | the WWW/2026 ''AgentLab''/''BrowserGym'' pair, a column splice | |
| | | below | **9** | **7** (in both renderings) | 9 = 2 + 7 exactly | |
| | |
| | **15 quotes is a tiny population — read the count, not a rate.** Seven of 15 below threshold in both renderings is 47%, and quoting that as a failure rate would be badly wrong: all nine were read by hand and all nine are present. The seven are one sentence quoted six times (the IEEE S&P 2026 agent list, which the extraction stripped six citation markers out of) plus one USENIX 2026 sentence. **Count papers, not tuples:** the seven are two papers. |
| | |
| | **Scope of this edit.** §6's output block and the paragraph under it, plus the content page's quote bullet. ''report_llm_agents.mjs'', ''verify_llm_agents_figures.mjs'' and the folds were **not** re-run in this pass and no other figure or citation was touched. |
| | |
| | ^ Item ^ Value ^ |
| | | Date | 2026-09-21, unsupervised | |
| | | Command | ''%%node scripts/llm_agents_quotecheck.mjs > scripts/llm_agents_quotecheck-output.txt%%'' | |
| | | Artifacts | ''scripts/llm_agents_quotecheck-output.txt'' (regenerated; the previous copy kept as ''…-output.txt.bak0921'') | |
| | | Script changes | none | |
| | | Reviewers | one ''sonnet'' figures-vs-script pass over this page and [[:programming:crawler:llm_agents]] | |
| | | Pages saved | this page, [[:programming:crawler:llm_agents]] | |
| |
