Differences

This shows you the differences between two versions of the page.

--- provenance:programming:crawler:llm_agents [2026/08/29 06:37] – Provenance for programming:crawler:llm_agents: every query with its denominator, the new tool_fold family and its residue, the wide/tight full-text probe, quote adjudication, external currency, judgement calls and four review passes. Authored by Claude karel.kubicek.claude
+++ provenance:programming:crawler:llm_agents [2026/09/21 13:43] (current) – Quote-check refresh 2026-09-21: regenerated llm_agents_quotecheck output block; 9 below threshold -> 2 rescued + 7 below in both. Authored by Claude karel.kubicek.claude
@@ Line 439: / Line 439: @@
 stretching back to 2010.
-The invocation matters: without ''--dropped'' the script prints the summary and the
+The invocation matters: without ''%%--dropped%%'' the script prints the summary and the
 surviving papers but not the residue, which is the half worth reading.
@@ Line 587: / Line 587: @@
   exact:   3
   partial: 3  (>=60% of 5-word windows)
-  below:   9   <- read these by hand
+  rescued from the PDF: 2
+  below threshold in BOTH renderings: 7   <- read these by hand
   no full text on disk: 0
@@ Line 603: / Line 604: @@
 IEEE-SP/2026/investigating-the-impact-of-dark-patterns-on-llm-based-web-agents
    VisualWebArena: 21% of 14 windows — "we evaluate four commercial agents (Skyvern, DoBrowser, BrowserUse, and Agent-E) and two academic web agents (WebArena and VisualWebArena)"
-WWW/2026/mcp-vs-rag-vs-nlweb-vs-html-a-comparison-of-the-effectiveness-and-efficiency-of
-   AgentLab: 44% of 9 windows — "The agent uses the AgentLab library and is executed within the BrowserGym framework."
-WWW/2026/mcp-vs-rag-vs-nlweb-vs-html-a-comparison-of-the-effectiveness-and-efficiency-of
-   BrowserGym: 44% of 9 windows — "The agent uses the AgentLab library and is executed within the BrowserGym framework."
 USENIX/2026/the-state-of-passkeys-studying-the-adoption-and-security-of-passkeys-on-the-web
    Browser Use: 30% of 10 windows — "We built a prototype tool using Playwright and the agentic LLM-based Browser Use framework."
 </code>
-**All nine below-threshold quotes were then read by hand against
+**Re-run 2026-09-21 with the PDF fallback: 2 of the 9 are rescued, 7 are below threshold in both renderings.** See //Quote-check refresh, 2026-09-21// at the foot of this page. The two rescued are the WWW/2026 //MCP vs RAG vs NLWeb vs HTML// pair (''AgentLab'', ''BrowserGym''), which is exactly right — their failure is a column splice, and an independent ''pypdf'' rendering does not splice that sentence. The other seven fail because the extraction dropped bracketed citation markers from its own string, and no rendering of the PDF can repair that.
+**All nine of the then-below-threshold quotes were read by hand against
 ''data/fulltext/<year>/<venue>/<slug>/paper.cols.txt'', and all nine are present in the
 source.** Each fails for one of two mechanical reasons — the extraction dropped bracketed
@@ Line 1017: / Line 1016: @@
 | 3 | **The guard recipe in §3.1 depended on a stale ''out/new/report_crawler.txt''**, which predates this run's fold change, so following it flags seven correct figures (181, 16.2%, 199, 205, 318, 28.4%, 74) as unaccounted. | **Accepted.** §3.1 now regenerates every backing file, ''report_crawler.txt'' first, before running the guard, and runs the guard against [[programming:crawler]] as well. |
 | 4 | The run log says "six keys already existed"; the page cites 11 keys of which 4 are new, so it is seven. | **Accepted.** Corrected in §12. |
-| 5 | ''llm_agent_fulltext_probe.mjs'' needs ''--dropped'' to reproduce the committed output; §5 showed no invocation. | **Accepted.** §5 now shows the command and says what the flag is for. |
+| 5 | ''llm_agent_fulltext_probe.mjs'' needs ''%%--dropped%%'' to reproduce the committed output; §5 showed no invocation. | **Accepted.** §5 now shows the command and says what the flag is for. |
 The same pass reports as clean: the fold tested against **all 19,384 distinct ''tools[].name'' strings** in the corpus (exactly the 14 the page lists match; no Playwright or Puppeteer name is stolen despite the family's precedence; every look-alike including ''AgentDojo'', ''SWE-Agent'' and ''CogAgent'' folds to residue); independent recomputation of 9-of-5,859 and of the 177-paper LLM-classification table with its year breakdown; byte-for-byte reproduction of three committed outputs; a hand re-check of every bounded-wildcard match in the figure verifier, finding no new instance of the 23,000-character bridging bug; and no cross-population arithmetic.
@@ Line 1043: / Line 1042: @@
 | 2 | **"the versions the other two papers used cannot be recovered at all" is false.** Both release artefacts; nobody looked in the lock files. | **Accepted.** The supported claim is "neither paper //states// a version". Reworded, and it now points at the artefacts. |
 | 3 | **SeeAct and Mind2Web are not Ersoy's subjects, and SeeAct is nowhere in the corpus** — zero probe matches across all 5,869 files. | **Accepted.** The table row was split: WebArena/VisualWebArena keeps the Ersoy attribution; Mind2Web and SeeAct are marked as used by no paper in this corpus, with the probe counts. |
-| 4 | **"run-to-run variance" is promised three times and retracted once.** Ramesh measured pass@k on benchmark tasks; nobody measured prevalence spread on a site sample, which Open Questions says. | **Accepted.** All three mentions — the lead WRAP, [[programming:crawler]]'s new section and [[start]] — now say "on benchmark tasks" or "what nobody has measured", and the lead points at Open Questions. |
+| 4 | **"run-to-run variance" is promised three times and retracted once.** Ramesh measured pass@k on benchmark tasks; nobody measured prevalence spread on a site sample, which Open Questions says. | **Accepted.** All three mentions — the lead WRAP, [[programming:crawler]]'s new section and [[:start]] — now say "on benchmark tasks" or "what nobody has measured", and the lead points at Open Questions. |
 | 5 | **The headline count and the "same wire" claim both break on the fifth paper.** It says "a named agent framework", but the fifth is a vendor computer-use API the page's own table calls "not a framework"; and the intro's "still drive Chromium through one of the control channels" is exactly what is unknown for that paper. | **Accepted.** The headline reads "a named agent framework or a vendor computer-use API"; the intro now separates the open-source frameworks (which do use a known channel) from the vendor APIs (screenshots and coordinates, wire supplied by you). [[programming:crawler]]'s mirror sentence got the same split. |
 | 6 | "Three things that table is for" is followed by five. | **Accepted.** Two bullets had been added after the count was written. Now five. |
@@ Line 1068: / Line 1067: @@
 <bibtex bibliography></bibtex>
+===== Markup sweep, 2026-09-17 =====
+Mechanical rendering repair only: a fresh live raw/XHTML export of 188 pages was checked with ''check_wrap.mjs'' and ''check_typography.mjs''. Affected plugin tags, CLI flags and heading markup were repaired; no figures or substantive prose were changed. The resulting source and rendered DOM were re-checked after saving.
+===== Link-hygiene sweep, 2026-09-17 =====
+The fresh rendered-DOM sweep found this page's bare ''start'' link resolved to ''provenance:programming:crawler:start''. It is now root-anchored; the post-save DOM was re-checked for red links. No figures or citations changed.
+===== Quote-check refresh, 2026-09-21 =====
+The 2026-09-04 ''cols''-vs-PDF audit on [[:provenance:literature:corpus]] showed that 73.1% of evidence quotes that cannot be located in ''paper.cols.txt'' **are** present in an independent ''pypdf'' rendering of the same ''paper.pdf''. ''llm_agents_quotecheck.mjs'' already carried the fallback; what was stale was this page's copy of its output. ''scripts/llm_agents_quotecheck-output.txt'' was regenerated by re-running the checker and §6's block replaced from it — the block was byte-identical to the old artifact before the re-run, and is byte-identical to the new one after it.
+^ Figure ^ Was ^ Is ^ Why ^
+| quotes checked | 15 | 15 | population unchanged |
+| exact | 3 | 3 | unchanged |
+| partial (≥60% of 5-word windows) | 3 | 3 | unchanged |
+| rescued from the PDF | — | **2** | the WWW/2026 ''AgentLab''/''BrowserGym'' pair, a column splice |
+| below | **9** | **7** (in both renderings) | 9 = 2 + 7 exactly |
+**15 quotes is a tiny population — read the count, not a rate.** Seven of 15 below threshold in both renderings is 47%, and quoting that as a failure rate would be badly wrong: all nine were read by hand and all nine are present. The seven are one sentence quoted six times (the IEEE S&P 2026 agent list, which the extraction stripped six citation markers out of) plus one USENIX 2026 sentence. **Count papers, not tuples:** the seven are two papers.
+**Scope of this edit.** §6's output block and the paragraph under it, plus the content page's quote bullet. ''report_llm_agents.mjs'', ''verify_llm_agents_figures.mjs'' and the folds were **not** re-run in this pass and no other figure or citation was touched.
+^ Item ^ Value ^
+| Date | 2026-09-21, unsupervised |
+| Command | ''%%node scripts/llm_agents_quotecheck.mjs > scripts/llm_agents_quotecheck-output.txt%%'' |
+| Artifacts | ''scripts/llm_agents_quotecheck-output.txt'' (regenerated; the previous copy kept as ''…-output.txt.bak0921'') |
+| Script changes | none |
+| Reviewers | one ''sonnet'' figures-vs-script pass over this page and [[:programming:crawler:llm_agents]] |
+| Pages saved | this page, [[:programming:crawler:llm_agents]] |