Atlas of the Present Atlas · v0.1

As of 28 October 2026

Dossier 10

Measurement

v3.3-draft · unchecked

1. As of

Date
Version
v3.3-draft
Author / model
Atlas generator
Reviewer
unchecked (Josef)

2. In one sentence

What model benchmarks do not see for children (Jiao, Chen, Mishra, Murali, Atkinson, Raj, Afroogh; arXiv:2610.34705, submitted 2026-10-26, E2, abstract only): the authors code 1,331 publicly reported child-related AI incidents against a child-centred harm taxonomy and find that harm concentrates in representational, dignity-related and psychological categories and is coded most often alongside deployment-context, governance and misuse failures rather than model defects; harm to an individual child is frequently linked to population-level harm that reaches children who never interacted with the system — publicly reported incidents are not an incidence statistic. A stable aggregate score is not a stable verdict (Yorke; University College London arXiv:2610.30512, E2): on 500 fixed Arena-Hard v2.0 prompts (GPT-4.1 mini vs o3-mini), GPT-4.1 judges under eight instruction setups at temperature zero, each repeated; running the same default setup twice unchanged moves the aggregate by about one point but flips win/tie/loss on 21.2% of prompts, and only 131 of 496 prompts (26.4%) keep their outcome across all 17 runs; label-only instead of reasoning lowers the score by 4.09 points and changes 41.6%, a different judge model (Gemini) changes 38.3% at almost the same aggregate — judge agreement kappa 0.44 against 0.76 between unchanged runs. Loss-of-control evidence under audit (Uuk, Koivula, Pacchiardi, Gipiškis, Caroli, Horton arXiv:2610.25241, E2): 19 studies on eight properties (incl. strategic deception, self-preservation, autonomous replication; models up to mid-2025) rated for construct, content and external validity — overall verdict “weakly plausible” (state of evidence, not a probability); no study reaches high external validity, only one clears the thresholds, and that one against situational awareness; rater agreement Fleiss κ 0.24. Clinical claim vs evidence (Feng arXiv:2610.14395, E2): 177 multimodal-vs-unimodal studies (81 from a systematic review, 96 from a probability sample of 926 PubMed records 2025–2026) — multimodal wins in 88.6% (median AUROC gain 0.042); with a separate model per data source −0.030; without the strongest source's model the gain would rise from 0.046 to 0.108; decision utility ~22%, prospective 0; usefulness claims 84.0% / est. 87.7%. Hidden Capabilities (Draguns, Motwani, Douglas, Schroeder de Witt arXiv:2610.13928, E2): digital lockers embed executable neural nets in host weights — chemistry reaction proxy: encrypted under SFT 50.9% / LAT 52.9% vs password-locked 92.8% / 91.7%; correct trigger 73.5% vs 92.7%; GCG 16-bit 52.7%; host ~2.07B dense parameters / 38 layers; subset-parity password-locked 100% under SFT, compiled stays at chance. MemoryAgentBench audit (Pakhomov & Nijkamp arXiv:2610.09193, E2): the rule stated in the prompt, “newest statement wins”, run as a frozen zero-learning resolver, solves 642/800 = 80.25% of the conflict-resolution split (held-out 74.5%); 67 gold answers are reachable only via overwritten statements — there Gemini 2.5 Pro / Flash / BM25 agent reach 10.4 / 11.9 / 6.0% instead of 84.7 / 82.6 / 41.6%. Agents-as-Systems (Wiedmann, Girrbach, Schmid, Akata arXiv:2610.01618, E1): agents as configurable systems — 4 AI4S tasks, 5 config axes, ~18,240 trajectories; among genuine attempts ~54% of score variance = run-to-run noise; Information rank 1 on every gap_positive task (outranks Model/Budget); prompted Verification last; oracle tool lifts score+calibration; 99.8% use weaker AstroSage on gap_negative. VISTA (Han et al., E1): Opus RHAE 40.68→100.00. HELM/Arena/SWE-bench — instruments, not thermometers.

Established now · E1 / E2

3. What works today

Children and AI: harm in 1,331 reported incidents tracks deployment, governance and misuse more than model defects (Jiao et al., abstract)

  1. E2

    Jiao, Chen, Mishra, Murali, Atkinson, Raj, Afroogh (arXiv:2610.34705v1, submitted 2026-10-26, cs.AI/cs.CY; only the abstract was opened, 2026-10-28) report: AI systems are increasingly present in children's everyday environments (education, entertainment, social networks, toys), and their safety is typically assessed at the model level — through benchmarks, red-teaming and content filtering. Most AI harm taxonomies lack child-specific evidence and under-specify developmental harm and the stakeholder asymmetries unique to minors. Building on prior taxonomies, the authors construct a child-centred harm taxonomy mapped to sociotechnical failure modes and curate and code 1,331 publicly reported child-related AI incidents (internal and external harm types and the failure mode implicated in each). Finding per the abstract: harm concentrates in representational, dignity-related and psychological categories and co-occurs most often with deployment-context, governance and misuse failures rather than model defects. Harm to an individual child is also frequently linked to population-level harm, reaching children who never interacted with the system. Privacy-minimised annotations, a codebook and documentation are, per the abstract, publicly available (KidAID).

    Jiao et al. child-related AI incidents. Safe by Benchmark, Unsafe in Practice: Lessons from 1331 Child-Related AI Incidents. https://arxiv.org/abs/2610.34705. As of arXiv:2610.34705v1 (submitted 2026-10-26; cs.AI, cs.CY; per comment 17 pp., 10 figures; annotations/codebook github.com/The-Responsible-AI-Initiative/KidAID not opened); abstract via export.arxiv.org API only, PDF not opened; affiliations not stated in the abstract record. Checked 2026-10-28. Type: Incident analysis (arXiv; abstract only; 1,331 publicly reported incidents coded).

LLM-as-judge: aggregate stable, individual verdicts not — an unchanged rerun flips 21.2% of prompts (Yorke)

  1. E2

    Yorke (University College London; arXiv:2610.30512v1, submitted 2026-10-21, 18 pp., NeurIPS 2026 workshop “JUDGe”, non-archival, PDF opened 2026-10-24): 500 prompts of the Arena-Hard v2.0 hard split with fixed responses (GPT-4.1 mini candidate, o3-mini baseline); GPT-4.1 (pinned gpt-4.1-2025-04-14) judges at temperature zero under eight instruction setups (2×2×2: Arena-Hard vs accuracy-prioritising criteria; reasoning vs label-only; no scope vs coding framing), each twice, the default three times — 17 runs, 17,000 calls, 16,996 parsed. Running the same default setup twice unchanged shifts the aggregate by about one point (52.82 / 54.02 / 53.52) but flips win/tie/loss on 21.2% of prompts (95% interval 18.4–24.1); of 496 prompts complete in every run only 131 (26.4%) keep their outcome across all 17.

    Yorke LLM-judge stability. Behind Aggregate Scores: What Systematically Varying Arena-Hard’s LLM Judge Reveals. https://arxiv.org/abs/2610.30512. As of arXiv:2610.30512v1 (submitted 2026-10-21; cs.CL; 18 pp.; NeurIPS 2026 JUDGe workshop, non-archival); PDF research/pdfs/arXiv2610.30512.pdf; code github.com/Yorkel/llm-judge-release not opened. Checked 2026-10-24. Type: Paper (arXiv; PDF opened; 500 prompts, 17,000 judge calls).
  2. E2

    Same source, what moves the judge: label-only instead of reasoning lowers the aggregate by 4.09 points and changes 41.6% of verdicts (28.2 percentage points above rerun noise), almost all movement into ties (tie rate 18.5% → 43.5%); accuracy-prioritising criteria change 26.9% at just −0.49 points (+6.7 pp), coding framing +3.8. A different judge model (May runs, once each): Gemini moves the aggregate by +0.18 but changes 38.3%; Claude −3.97 and 35.0%; weighted Cohen kappa GPT-4.1↔Gemini 0.435, ↔Claude 0.549, unchanged default runs with each other 0.763. Instability sits at the decision boundary: the rerun changes 43.4% of zero-margin prompts but 5.5% at the largest margin. In one verifiable case (a coding prompt with three worked examples) the default judge gave a decisive win to the submission that fails all three, while accuracy-prioritising criteria and both judge models preferred the one that passes.

    Yorke LLM-judge stability. Behind Aggregate Scores: What Systematically Varying Arena-Hard’s LLM Judge Reveals. https://arxiv.org/abs/2610.30512. As of arXiv:2610.30512v1 (submitted 2026-10-21; cs.CL; 18 pp.; NeurIPS 2026 JUDGe workshop, non-archival); PDF research/pdfs/arXiv2610.30512.pdf; code github.com/Yorkel/llm-judge-release not opened. Checked 2026-10-24. Type: Paper (arXiv; PDF opened; 500 prompts, 17,000 judge calls).

Loss-of-control evidence: 19 studies, overall verdict “weakly plausible”, none with high external validity (Uuk et al.)

  1. E2

    Uuk, Koivula, Pacchiardi, Gipiškis, Caroli, Horton (Future of Life Institute, KU Leuven, LSE, ETH Zurich, Cambridge, AI Standards Lab, Vilnius, RAND; arXiv:2610.25241v1 of 2026-10-18, previously SSRN 2026-05-28, PDF opened 2026-10-20): the trigger is the EU Code of Practice for general-purpose AI, which lists loss of control as one of four systemic risks that must be assessed. 19 empirical studies (models up to mid-2025) on eight properties from threat models, each rated independently by three authors for construct, content and external validity (framework after Salaudeen et al. 2025); 25 study-property pairs. Result (Table 2): strategic deception 7 assessments (4 weakly plausible, 2 speculative, 1 weakly implausible) → weakly plausible; situational awareness 6 → weakly implausible; autonomous replication 2 conflicting → speculative; agency, goal misgeneralisation, persuasion, reward hacking, self-preservation → weakly plausible. Overall verdict “weakly plausible” — meaning strength and direction of evidence, not a probability. No study reaches high external validity; only one study clears the thresholds for a stronger judgment, and in the negative direction (against situational awareness).

    Uuk et al. loss-of-control evidence. Assessing the Empirical Evidence for Loss of Control from Agentic General-Purpose AI. https://arxiv.org/abs/2610.25241. As of arXiv:2610.25241v1 (2026-10-18; cs.CY; 42 pp.; previously SSRN 2026-05-28); PDF research/pdfs/arXiv2610.25241.pdf. Checked 2026-10-20. Type: Evidence audit (arXiv; PDF opened; preprint).
  2. E2

    Same source, appendix: agreement of the three raters across 75 individual scores 60.0% unanimous, 38.7% by majority; Fleiss κ = 0.24 (“fair”), because 78.7% of all ratings fall in the middle category “proceed with caution”; only 2 of 75 items with full-range disagreement; Gwet AC1 = 0.67. Sensitivity (Table 3): strict raters make almost all properties “speculative”, generous ones lift agency and deception to “highly plausible” — strategic deception alone stays at least weakly plausible in every scenario.

    Uuk et al. loss-of-control evidence. Assessing the Empirical Evidence for Loss of Control from Agentic General-Purpose AI. https://arxiv.org/abs/2610.25241. As of arXiv:2610.25241v1 (2026-10-18; cs.CY; 42 pp.; previously SSRN 2026-05-28); PDF research/pdfs/arXiv2610.25241.pdf. Checked 2026-10-20. Type: Evidence audit (arXiv; PDF opened; preprint).

Multimodal clinical AI: the claim runs ahead of the evidence (Feng)

  1. E2

    Feng (Harvard T.H. Chan School of Public Health; arXiv:2610.14395v1, submitted 2026-10-09; PDF opened 2026-10-16): audit of 177 studies comparing multimodal (e.g. imaging + clinical data) with unimodal prediction models — 81 from a systematic review (97 studies, “multimodal wins in 91%”), 96 from a stratified probability sample of 926 PubMed records 2025–2026 (design-weighted estimates). Multimodal beats the strongest unimodal comparator in 85.2% (69/81) and an estimated 89.3%; pooled 88.6% (95% CI 83.5–93.7); pooled median AUROC gain 0.042 (IQR 0.020–0.100).

    Feng multimodal clinical claims. Clinical value claims in multimodal AI outpace supporting evidence. https://arxiv.org/abs/2610.14395. As of arXiv:2610.14395v1 (submitted 2026-10-09); PDF research/pdfs/arXiv2610.14395.pdf. Checked 2026-10-16. Type: Meta-audit (arXiv; PDF opened; preprint).
  2. E2

    Studies that fit a separate model for every data source find smaller gains (adjusted −0.030; 95% CI −0.056 to −0.005). In 78 completely benchmarked studies, leaving out the strongest source's model would have raised the mean gain from 0.046 to 0.108. Complete benchmarking fell from 69.1% to an estimated 46.0% (−23.1 pp). Decision utility (decision curves or similar) an estimated ~22%; external validation of the gain 28.4% / 20.9%; calibration 16.0% / 26.5%; no study evaluated prospectively. Even so, 84.0% and an estimated 87.7% claim potential clinical usefulness or improved care — including 83.6% / 85.9% of studies with neither decision-utility nor prospective evaluation. Second blinded coder (36 studies): 88.6–100% agreement, κ 0.89 on benchmarking.

    Feng multimodal clinical claims. Clinical value claims in multimodal AI outpace supporting evidence. https://arxiv.org/abs/2610.14395. As of arXiv:2610.14395v1 (submitted 2026-10-09); PDF research/pdfs/arXiv2610.14395.pdf. Checked 2026-10-16. Type: Meta-audit (arXiv; PDF opened; preprint).

Hidden Capabilities: audits without the trigger are not enough (Draguns et al.)

  1. E2

    Draguns, Motwani, Douglas, Schroeder de Witt (Contramont Research / University of Oxford / ACS Research; arXiv:2610.13928v1 of 2026-10-09; PDF/HTML opened 2026-10-13): digital lockers (Keccak/SHA-3-based) encrypt the weights of an executable neural net and embed them in host weights (SwiGLU/RMSNorm). Chemistry proxy (USPTO, 64-bit Morgan fingerprints, ternary MLP 64→8→16→2): Table 2 — encrypted under SFT (500 steps) 50.9±1.0% and LAT 52.9±3.9% vs password-locked 92.8±0.9% / 91.7±0.7%; correct trigger 73.5±2.6% vs 92.7±0.2%; GCG at 16-bit trigger 52.7±1.4% (at 8 bits exhaustion = trigger performance). Random-network control under SFT 50.6% / LAT 53.6% — no observed benefit from the trained payload without the trigger.

    Draguns et al. Hidden Capabilities. AI Models Can Provably Hide Arbitrary Capabilities. https://arxiv.org/abs/2610.13928. As of arXiv:2610.13928v1 (2026-10-09); PDF research/pdfs/HideCapabilities_arXiv2610.13928.pdf. Checked 2026-10-13. Type: Paper (arXiv; PDF+HTML opened).
  2. E2

    Compiled chemistry host: 38 layers, ≈2.07×10⁹ dense parameters, residual width 10,266 (hash subcircuits ≈95% of parameters). Subset parity: password-locked reaches 100% under SFT, the compiled model stays at chance; noise elicits neither. Proof-of-concept / model organisms — not a turnkey attack on frontier LLMs; presence of compiled circuits can be anomalous; full integration into pretrained models is future work per the authors.

    Draguns et al. Hidden Capabilities. AI Models Can Provably Hide Arbitrary Capabilities. https://arxiv.org/abs/2610.13928. As of arXiv:2610.13928v1 (2026-10-09); PDF research/pdfs/HideCapabilities_arXiv2610.13928.pdf. Checked 2026-10-13. Type: Paper (arXiv; PDF+HTML opened).

MemoryAgentBench: a lookup rule solves 80% (Pakhomov & Nijkamp)

  1. E2

    Pakhomov & Nijkamp (Salesforce AI Research; arXiv:2610.09193v1 of 2026-10-06, NeurIPS 2026 workshop “Interpreting Agent Behavior”; PDF opened 2026-10-08): the conflict-resolution split of MemoryAgentBench (ICLR 2026) is read as a test of “selective forgetting” — numbered fact lists, later statements contradict earlier ones. The rule is stated in the prompt itself: the newest statement wins. Run as a zero-learning resolver (35 parser patterns, no LLM calls), developed on one of the four lists and frozen before the rest: 642/800 = 80.25% correct (95% CI 77.3–83.0; 32 abstentions), on the three held-out lists 447/600 = 74.5%. Single-hop 94.0%, multi-hop 66.5%; at 262K 88/36 (SH/MH) — above every main-table system of the benchmark’s v1 (at most 60/7; per the authors a reading aid, not a leaderboard: whole list vs chunking).

    Pakhomov & Nijkamp MemoryAgentBench audit. Bookkeeping, Composition, or Unreachable Gold? Reading MemoryAgentBench’s Conflict-Resolution Scores Against a Frozen Last-Write Resolver. https://arxiv.org/abs/2610.09193. As of arXiv:2610.09193v1 (2026-10-06; NeurIPS 2026 Workshop IAB); PDF research/pdfs/MemoryAgentBench_LastWrite_arXiv2610.09193.pdf. Checked 2026-10-08. Type: Paper (arXiv; PDF opened; workshop).
  2. E2

    67 of the 158 resolver failures (42.4%) have a released gold answer reachable only via overwritten statements (example from the dataset: statement 158 “The capital of India is New Delhi.”, later statement 341 “… is Grosseto.”, gold New Delhi) — a third of the multi-hop questions at 262K. Gemini 2.5 Pro (overall 75.0%), Gemini 2.5 Flash and a re-implemented BM25 agent reach 84.7 / 82.6 / 41.6% on the items the rule solves, but 10.4 / 11.9 / 6.0% on those 67. Cross-check: first write instead of last loses 526 of the 642 solved items — so overwriting is really tested, but as bookkeeping.

    Pakhomov & Nijkamp MemoryAgentBench audit. Bookkeeping, Composition, or Unreachable Gold? Reading MemoryAgentBench’s Conflict-Resolution Scores Against a Frozen Last-Write Resolver. https://arxiv.org/abs/2610.09193. As of arXiv:2610.09193v1 (2026-10-06; NeurIPS 2026 Workshop IAB); PDF research/pdfs/MemoryAgentBench_LastWrite_arXiv2610.09193.pdf. Checked 2026-10-08. Type: Paper (arXiv; PDF opened; workshop).

Agents-as-Systems: configuration > single score (Wiedmann et al.)

  1. E1

    Wiedmann, Girrbach, Schmid, Akata (TUM/MCML / Helmholtz Munich / Inria–ENS; arXiv:2610.01618v1, PDF opened 2026-10-04): agents as configurable systems — 4 AI4S tasks (coding agent must find and operate a specialist model); 5 config axes (Information, Reasoning, Verification, Budget, Model); release of benchmark + 18,240 trajectories.

    Wiedmann et al. Agents-as-Systems. Agents Are Systems, Not Models: Rethinking Agentic Evaluation. https://arxiv.org/abs/2610.01618. As of arXiv:2610.01618v1 (2026-10-01); PDF research/pdfs/AgentsAsSystems_arXiv2610.01618.pdf. Checked 2026-10-04. Type: Paper (arXiv; PDF opened).
  2. E1

    Among genuine attempts ~54% of score variance = run-to-run noise (not config). Information has the largest axis effect — rank 1 on every gap_positive task (Table 3); outranks Model and Budget; prompted Verification mean rank 5.00 (last). Axes interact: more Budget helps G only with enough Information / a capable Model. Oracle tool lifts score and calibration; prompted Verification barely changes verification behavior. On gap_negative mmlu-astronomy, 99.8% of runs use weaker AstroSage (correct is to decline). Config choice moves the score — instrument, not thermometer.

    Wiedmann et al. Agents-as-Systems. Agents Are Systems, Not Models: Rethinking Agentic Evaluation. https://arxiv.org/abs/2610.01618. As of arXiv:2610.01618v1 (2026-10-01); PDF research/pdfs/AgentsAsSystems_arXiv2610.01618.pdf. Checked 2026-10-04. Type: Paper (arXiv; PDF opened).

VISTA: visual harness lifts RHAE (Han et al.)

  1. E1

    Han, Hu, Qiu, Wu, He (MIT; arXiv:2610.02200v1, PDF opened 2026-10-02): VISTA — visual harness with direct observation and lossless visual memory; the model actively retrieves past frames and reorganizes visual input while reasoning (no program synthesis for ARC-AGI-3).

    Han et al. VISTA. VISTA: A Visual Harness for Reasoning in an Interactive World. https://arxiv.org/abs/2610.02200. As of arXiv:2610.02200v1 (2026-10-01); PDF research/pdfs/VISTA_arXiv2610.02200.pdf. Checked 2026-10-02. Type: Paper (arXiv; PDF opened).
  2. E1

    ARC-AGI-3 public 25 games: Claude Opus 5.0 + VISTA RHAE 100.00 (xhigh) — all 25; 7,302 actions (−57.4% vs human reference 17,135). Official baseline without VISTA: Opus 5.0 high RHAE 40.68; GPT-5.6 Sol max 13.33. Sol+VISTA RHAE 99.00. Extension GameWorld (34) / AI GameStore (10): same-model minimal harness beaten. Harness choice moves the score — instrument, not thermometer.

    Han et al. VISTA. VISTA: A Visual Harness for Reasoning in an Interactive World. https://arxiv.org/abs/2610.02200. As of arXiv:2610.02200v1 (2026-10-01); PDF research/pdfs/VISTA_arXiv2610.02200.pdf. Checked 2026-10-02. Type: Paper (arXiv; PDF opened).

OSWorld-Science: science desktop ≠ solved (Dai et al.)

  1. E1

    Dai, Qi, Liu, Li, Chen, Dou et al.; Liu (corr.) (Tsinghua / UCLA / RIKEN / Waterloo / CMU / Yale et al.; arXiv:2609.39903v1, PDF opened 2026-10-01): OSWorld-Science — 146 tasks / 7 domains / 14 software configs; 12 VLMs under shared PromptAgent harness; artifact-based graders with partial credit; VOID = 0.

    Dai et al. OSWorld-Science. OSWorld-Science: A Benchmark of Computer Use Agents for Learning and Using Scientific Software. https://arxiv.org/abs/2609.39903. As of arXiv:2609.39903v1 (2026-09-30); PDF research/pdfs/OSWorldScience_arXiv2609.39903.pdf. Checked 2026-10-01. Type: Paper (arXiv; PDF opened).
  2. E1

    Mean task score: Claude Fable 5.1 73.7% ($6.5/task); Opus 5 57.9% ($6.7); GPT-6 Astra 57.4% ($7.7); GPT-5.6 sol 50.1% ($2.79). Intro: strongest agent ~70% avg / ~40% hardest set. Chemistry harder: Fable ~44%, Opus ~39%, sol ~20%, eight models ≤~6%. Trajectory audit ~1500 runs: only 402 (26.3%) full credit; 667/1100 non-full without graded artifact. Bench ≠ live-lab autonomy.

    Dai et al. OSWorld-Science. OSWorld-Science: A Benchmark of Computer Use Agents for Learning and Using Scientific Software. https://arxiv.org/abs/2609.39903. As of arXiv:2609.39903v1 (2026-09-30); PDF research/pdfs/OSWorldScience_arXiv2609.39903.pdf. Checked 2026-10-01. Type: Paper (arXiv; PDF opened).

VLA Accel Audit: bench bugs distort acceleration eval (Chen et al.)

  1. E1

    Chen, Zhou, Shi, Li, Feng, Gu (Shanghai Jiao Tong; arXiv:2609.37771v1, PDF opened 2026-09-30): audit across 7 manipulation benchmarks (incl. RoboTwin, LIBERO-Plus, VLABench) — 22 bugs (task consistency/initialization/reproducibility) + 4 design limitations; trajectories vs checker acceptance regions.

    Chen et al. VLA Accel Audit. Faster and Better? Benchmark Bugs and Design Limitations Distort the Evaluation of Vision-Language-Action Acceleration. https://arxiv.org/abs/2609.37771. As of arXiv:2609.37771v1 (2026-09-29); PDF research/pdfs/VLAAccelAudit_arXiv2609.37771.pdf. Checked 2026-09-30. Type: Paper (arXiv; PDF opened).
  2. E1

    Bug fixes reverse rankings: LIBERO-PRO Spatial-Task baseline SR 0.4%→53.2% (last→first, +52.8 pp); RoboTwin place shoe baseline 65→74 vs DP-Cache-Fast 68→69 (3 pp behind → 5 pp ahead). Design mass correction place fan: baseline from 21 pp behind DP-Cache-Fast to 5 pp ahead (40/61→38/33). Acceleration gains can be bench artifacts.

    Chen et al. VLA Accel Audit. Faster and Better? Benchmark Bugs and Design Limitations Distort the Evaluation of Vision-Language-Action Acceleration. https://arxiv.org/abs/2609.37771. As of arXiv:2609.37771v1 (2026-09-29); PDF research/pdfs/VLAAccelAudit_arXiv2609.37771.pdf. Checked 2026-09-30. Type: Paper (arXiv; PDF opened).

MechBench: phenomenal law ≠ mechanism (Yu et al.)

  1. E1

    Yu, Zhang, Cheng, Ding, Li (Tsinghua / CAS / XJTU; arXiv:2609.35515v1, PDF opened 2026-09-29): MechBench — separates phenomenal-law recovery SA(P) from mechanism recovery SA(M) via mechanism probes and controlled mutations; Full-set 512 instances / Core-set 80.

    Yu et al. MechBench. MechBench: Can AI Scientific Agents Discover Mechanisms Beyond Phenomenal Laws?. https://arxiv.org/abs/2609.35515. As of arXiv:2609.35515v1 (2026-09-28); PDF research/pdfs/MechBench_arXiv2609.35515.pdf. Checked 2026-09-29. Type: Paper (arXiv; PDF opened).
  2. E1

    Codex + GPT-5.6-sol Core-set: SA(P) 35.00%, SA(M) 13.75%; conditional mechanism failure among correct P: 64.29%. Full-set: SA(P) 16.02%, SA(M) 7.81%; 1-(M|P)=57.32%. Gold-P Direct-Ask (law given): SA(M) still 45.00% Core / 49.02% Full (<50%). Gap widens with mutation count; mechanism fails faster than phenomenal.

    Yu et al. MechBench. MechBench: Can AI Scientific Agents Discover Mechanisms Beyond Phenomenal Laws?. https://arxiv.org/abs/2609.35515. As of arXiv:2609.35515v1 (2026-09-28); PDF research/pdfs/MechBench_arXiv2609.35515.pdf. Checked 2026-09-29. Type: Paper (arXiv; PDF opened).

Benchmarks are instruments, not thermometers

  1. E1

    HELM taxonomizes scenarios (task × domain × language) and measures 7 metrics (accuracy, calibration, robustness, fairness, bias, toxicity, efficiency) on 16 core scenarios, 87.5% of 112 pairs (98/112). 30 models, 42 scenarios, 4,939 runs. Before: models evaluated on 17.9% of core scenarios on average; after 96.0%. E1 for the construct and 2022 campaign; E0 for 2026 frontier scores.

    Liang et al. Holistic Evaluation of Language Models. https://arxiv.org/html/2211.09110. As of 2022-11 / TMLR 2023. Checked 2026-08-28. Type: Paper.
  2. E1

    Landing 2026: Classic, Lite, Capabilities, Instruct, Audio, VHELM, Safety, Finance, MedHELM, Long Context and others exist. E1 for the instrument ecosystem; result tables not opened.

    Liang et al. Holistic Evaluation of Language Models. https://arxiv.org/html/2211.09110. As of 2022-11 / TMLR 2023. Checked 2026-08-28. Type: Paper.HELM landing. Holistic Evaluation of Language Models (HELM) Leaderboards. https://crfm.stanford.edu/helm/. As of page as of 2026-08-28. Checked 2026-08-28. Type: Official site.
  3. E1

    What accuracy hides (HELM findings 2022): instruction-tuning leads accuracy/robustness/fairness despite 10× smaller scale. Accuracy–calibration is scenario-dependent. NarrativeQA TNLG v2 72.6% → 38.9% under robustness perturbations. Prompting: wild swings 30% → 80%. Multiple-choice adaptation: OPT 175B HellaSwag 79.1% (separate 0-shot) vs. 30.2% (joint 5-shot). Scale predicts accuracy within families; across families for no scenario. E1 for the 2022 snapshot.

    Liang et al. Holistic Evaluation of Language Models. https://arxiv.org/html/2211.09110. As of 2022-11 / TMLR 2023. Checked 2026-08-28. Type: Paper.

Four instruments: construct and limit

  1. E2

    Arena: anonymous side-by-side battles, Bradley-Terry MLE (not online Elo). Paper Jan 2024: ~240k votes, ~90k users, >50 models. Crowd–expert agreement 72.8–83.1%. User base primarily LLM hobbyists; helpfulness, not safety. Live ranks 2026: E0 (board not opened).

    Chiang et al. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. https://arxiv.org/html/2403.04132. As of 2024-03-07 (ICML 2024). Checked 2026-08-28. Type: Paper.
  2. E2

    Style control Aug 2024: length coefficient 0.249 — dominant. After control: GPT-4o-mini and Grok-2-mini fall; Claude 3.5 Sonnet/Opus, Llama-3.1-405B rise. Observational, not causal. Length is part of human preference, not only noise.

    Li, Angelopoulos, Chiang. Does style matter? Disentangling style and substance in Chatbot Arena. https://www.lmsys.org/blog/2024-08-28-style-control/. As of 2024-08-29. Checked 2026-08-28. Type: Official blog.
  3. E1

    SWE-bench: 2,294 issues, 12 Python repos. % resolved = all FAIL_TO_PASS and PASS_TO_PASS green. Claude 2 + BM25: 1.96% resolved; oracle retrieval 4.80%. Apply rates ≫ resolve. Longer context lowers resolve (1.96 → 1.22% at 50k). SWE-Verified / Pro / + E0 in this log (OpenAI timeout).

    Jimenez et al. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. https://arxiv.org/html/2310.06770. As of 2023-10 / ICLR 2024. Checked 2026-08-28. Type: Paper.
  4. E2

    LiveCodeBench arXiv HTML: 511 problems May 2023–May 2024. Time window: DeepSeek-Instruct-33B drops after Aug 2023; GPT-4o after Nov 2023; Codestral 36.5% → 28.3%. HumanEval overfitting: DS-Ins-1.3B 59.8% HumanEval+ vs. 26.3% LCB-Easy. GPT-4-Turbo-2024-04-09 code-gen total 41.1 Pass@1; hard 5.1. Live board 2026 E0.

    Jain et al. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. https://arxiv.org/html/2403.07974. As of 2024-03 (HTML: 511 problems May 2023–May 2024). Checked 2026-08-28. Type: Paper.
  5. E1

    Fact sheets — what is measured, what is not.

    Four instruments. Construct and opened limits only. No 2026 ranking.
    InstrumentMeasuresDoes notNoise
    HELMScenarios × 7 metrics, 5-shotAgents, non-EN core, CoT 2022Prompt/MC, auto-summarization
    ArenaPairwise human preference, liveSafety, facts, executionLength/markdown, no GT
    SWE-benchPatch per repo testsDesign, review, multi-languageRetrieval, apply, 12 repos
    LiveCodeBenchContest code, time-filteredRepo SE, other languagesn after cutoff, LeetCode leak
    Liang et al. Holistic Evaluation of Language Models. https://arxiv.org/html/2211.09110. As of 2022-11 / TMLR 2023. Checked 2026-08-28. Type: Paper.Chiang et al. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. https://arxiv.org/html/2403.04132. As of 2024-03-07 (ICML 2024). Checked 2026-08-28. Type: Paper.Jimenez et al. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. https://arxiv.org/html/2310.06770. As of 2023-10 / ICLR 2024. Checked 2026-08-28. Type: Paper.Jain et al. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. https://arxiv.org/html/2403.07974. As of 2024-03 (HTML: 511 problems May 2023–May 2024). Checked 2026-08-28. Type: Paper.

Truth vs preference: TruthfulQA

  1. E1

    TruthfulQA = 817 questions, 38 categories, zero-shot, adversarial against imitative falsehoods. Human (internet, ~2 min/question): 94% true, 87% true+informative. Best model in the paper (GPT-3-175B, helpful prompt): 58% true, 21% true+informative; 42% false+informative (human 6%). Inverse scaling: larger models in the same family often less truthful (GPT-Neo/J 6B 17% less truthful than a 60× smaller one). UnifiedQA more truthful but less informative. Multiple-choice: no model significantly above chance; larger often worse.

    Lin, Hilton, Evans. TruthfulQA: Measuring How Models Mimic Human Falsehoods. https://arxiv.org/html/2109.07958. As of 2021-09-08. Checked 2026-08-29. Type: Paper (NeurIPS).
  2. E2

    Inverse scaling is explained in the paper as imitative falsehoods (better learning of the training distribution), with control questions (edits) that improve with size, and paraphrases that hold the trend. That is 2021 GPT-3/J/2/T5, not the 2026 frontier. Appendix B.3 (newer Anthropic/InstructGPT/WebGPT/Gopher models) shows progress and a return to positive scaling at the largest, but still a gap to humans — E2 for these appendix figures (external eval, opened in the same HTML).

    Lin, Hilton, Evans. TruthfulQA: Measuring How Models Mimic Human Falsehoods. https://arxiv.org/html/2109.07958. As of 2021-09-08. Checked 2026-08-29. Type: Paper (NeurIPS).

MMLU: saturation is ceiling plus noise

  1. E1

    MMLU: 57 tasks, 15,908 questions. GPT-3 175B few-shot 43.9%. MTurk 34.5%; expert estimate ~89.8%. Entropy–accuracy negatively correlated (r=−0.43 zero-shot) — no signal for verbatim Q&A memorization in the GPT-3 era.

    Hendrycks et al. Measuring Massive Multitask Language Understanding. https://arxiv.org/html/2009.03300. As of 2020-09 / ICLR 2021. Checked 2026-08-28. Type: Paper.
  2. E2

    MMLU-Pro: frontier clusters on MMLU 86–87% (GPT-4 86.4% as a citation; GPT-4 report not opened here). MMLU-Pro: 12,032 questions, mean 9.47 options. GPT-4o 72.6% CoT vs. 53.5% direct. Gap GPT-4o vs. GPT-4-Turbo ~1 pp MMLU → ~9 pp MMLU-Pro. Prompt sensitivity peak 10.98% → 3.74%. Saturation ≠ expert parity.

    Wang et al. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. https://arxiv.org/html/2406.01574. As of 2024-06 / NeurIPS 2024. Checked 2026-08-28. Type: Paper.

Contamination: mem ≠ expl; little verbatim on four open models

  1. E1

    Magar: mem (MLM seen−unseen) ≠ expl (task seen−unseen). SST-5, 200 duplicates: mem ~60%, expl ~40%. mem 15% can yield expl <1%. Early contamination → high expl; late + LR-decay → expl ~0 with still-high mem. BERT scale, not frontier LLM.

    Magar & Schwartz. Data Contamination: From Memorization to Exploitation. https://arxiv.org/html/2203.08242. As of 2022-03 / ACL 2022. Checked 2026-08-28. Type: Paper.
  2. E1

    Oren: exchangeability test, FPR guarantee. Audit Llama2-7B, Mistral-7B, Pythia-1.4B, GPT-2 XL: no significant verbatim contamination (exception Mistral × ARC-Easy). Limit: order memorization only, not paraphrases. E1 method, E2 audit.

    Oren et al. Proving Test Set Contamination in Black-Box Language Models. https://arxiv.org/html/2310.17623. As of 2023-10 / ICLR 2024. Checked 2026-08-28. Type: Paper.
  3. E2

    Sainz: guideline / raw-text / annotation contamination; after deployment (API logs). ChatGPT/WizardCoder regenerate CoNLL-2003 train start (appendix). GSM8K body now opened as source 18 (E1 paper); GPT-4 figures from the Technical Report in Sainz remain E3 second-order (report not opened here).

    Sainz et al. NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark. https://arxiv.org/html/2310.18018. As of 2023-10 / Findings EMNLP 2023. Checked 2026-08-28. Type: Position paper.

Metric artefacts and unstable ranks

  1. E1

    Schaeffer: nonlinear metrics (accuracy, exact match, multiple choice grade) produce sharpness from smooth per-token error. >92% of hand-annotated BIG-Bench emergences under MC grade or exact string match. “nothing … cannot display emergent abilities” — only the published claims are suspect. Not a refutation of every possible emergence.

    Schaeffer et al. Are Emergent Abilities of Large Language Models a Mirage?. https://arxiv.org/html/2304.15004. As of 2023-04 / NeurIPS 2023. Checked 2026-08-28. Type: Paper.
  2. E1

    Wei et al. (arXiv:2206.07682, TMLR 2022, body opened): emergence definition — an ability is emergent if absent in smaller models but present in larger ones; not predictable by extrapolating a scaling curve (near-random until a threshold, then a phase transition).

    Wei et al. Emergent. Emergent Abilities of Large Language Models. https://arxiv.org/pdf/2206.07682. As of 2022-06 / TMLR 2022. Checked 2026-09-06. Type: Paper (arXiv; TMLR).
  3. E1

    Wei Table 1 / Fig. 2: few-shot examples — e.g. 3-digit arithmetic ~2×10²² FLOPs / 13B GPT-3; MMLU ~3×10²³ / 175B GPT-3; WiC only at ~2.5×10²⁴ / 540B PaLM. BIG-Bench: >200 tasks; dozens still near-random even at largest GPT-3/PaLM (paper).

    Wei et al. Emergent. Emergent Abilities of Large Language Models. https://arxiv.org/pdf/2206.07682. As of 2022-06 / TMLR 2022. Checked 2026-09-06. Type: Paper (arXiv; TMLR).
  4. E1

    Wei limits the claim: no fixed universal threshold; scale not the only factor; not a proof that all abilities emerge. Survey/definition — contrast with Schaeffer stays open; no reconciliation here.

    Wei et al. Emergent. Emergent Abilities of Large Language Models. https://arxiv.org/pdf/2206.07682. As of 2022-06 / TMLR 2022. Checked 2026-09-06. Type: Paper (arXiv; TMLR).
  5. E2

    Alzahrani: mini-perturbations on MMLU shift ranks by up to 8 places (11 models). Llama-2-7b zero-shot: correct-at-A 66.36% vs. correct-at-D 23.37% (Δ 43 pp). In-context cheating: 5-shot with correct answer 92–99%; with wrong → collapse.

    Alzahrani et al. When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards. https://arxiv.org/html/2402.01781. As of 2024-02. Checked 2026-08-28. Type: Paper.
  6. E2

    Bowman & Dahl 2021: GLUE/SuperGLUE saturates while failing CheckList/McCoy. 80→81% needs thousands of items. Saturation as a design fault, not mastery.

    Bowman & Dahl. What Will it Take to Fix Benchmarking in Natural Language Understanding?. https://arxiv.org/html/2104.02145. As of 2021-04 / NAACL 2021. Checked 2026-08-28. Type: Position paper.

Graduate-level Google-proof Q&A: GPQA

  1. E1

    GPQA (Rein et al., arXiv:2311.12022, body opened): 448 expert-written MCQs (main set) in biology/physics/chemistry; Extended 546; Diamond 198.

    Rein et al. GPQA. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. https://arxiv.org/pdf/2311.12022. As of 2023-11-20. Checked 2026-09-05. Type: Paper (arXiv; COLM).
  2. E1

    Experts (PhD domain) ~65% accuracy on Extended (74% when discounting clear mistakes / post-hoc agreement analysis).

    Rein et al. GPQA. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. https://arxiv.org/pdf/2311.12022. As of 2023-11-20. Checked 2026-09-05. Type: Paper (arXiv; COLM).
  3. E1

    Skilled non-experts with web access ~34% (avg ≥30 min); questions designed to be Google-proof.

    Rein et al. GPQA. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. https://arxiv.org/pdf/2311.12022. As of 2023-11-20. Checked 2026-09-05. Type: Paper (arXiv; COLM).
  4. E1

    Strongest GPT-4 few-shot CoT baseline ~39% (chance 25%).

    Rein et al. GPQA. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. https://arxiv.org/pdf/2311.12022. As of 2023-11-20. Checked 2026-09-05. Type: Paper (arXiv; COLM).
  5. E1

    Diamond: both experts agree and ≤1/3 non-experts correct (highest-quality subset).

    Rein et al. GPQA. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. https://arxiv.org/pdf/2311.12022. As of 2023-11-20. Checked 2026-09-05. Type: Paper (arXiv; COLM).

Discovery vs execution: TruthInsightBench

  1. E1

    TruthInsightBench (arXiv:2609.05079v2, PDF opened): 40 blind tasks from 40 peer-reviewed studies across 10 scientific domains; agent gets a neutral objective + frozen data; source conclusions/analysis paths withheld.

    TruthInsightBench. TruthInsightBench: An Evidence-Grounded Benchmark for Automated Evaluation of Open-Ended Scientific Discovery Agents. https://arxiv.org/abs/2609.05079. As of arXiv:2609.05079v2 (2026-09-08); PDF research/pdfs/TruthInsightBench_arXiv2609.05079.pdf. Checked 2026-09-16. Type: Paper (arXiv; PDF opened).
  2. E1

    Fixed LLM judge scores evidentiary maturity of the agent’s own claims along six dimensions (29 artifact-grounded items); aggregation automated/deterministic, no per-instance human grading.

    TruthInsightBench. TruthInsightBench: An Evidence-Grounded Benchmark for Automated Evaluation of Open-Ended Scientific Discovery Agents. https://arxiv.org/abs/2609.05079. As of arXiv:2609.05079v2 (2026-09-08); PDF research/pdfs/TruthInsightBench_arXiv2609.05079.pdf. Checked 2026-09-16. Type: Paper (arXiv; PDF opened).
  3. E1

    Four coding agents on one frozen base model: total scores form a narrow plateau 58.4–60.3 of 100 (Table 4: Claude Code 60.27; OpenScience 59.03; Codex CLI 58.81; DeepSeek Harness 58.40); no pairwise separation at 0.05 (smallest p = 0.118).

    TruthInsightBench. TruthInsightBench: An Evidence-Grounded Benchmark for Automated Evaluation of Open-Ended Scientific Discovery Agents. https://arxiv.org/abs/2609.05079. As of arXiv:2609.05079v2 (2026-09-08); PDF research/pdfs/TruthInsightBench_arXiv2609.05079.pdf. Checked 2026-09-16. Type: Paper (arXiv; PDF opened).
  4. E1

    Paper: evidence auditability and novelty comparatively strong; controls, robustness, falsifiability and cross-dataset generalization largely absent — bottleneck is scientific judgment, not coding; genuine discovery remains out of reach.

    TruthInsightBench. TruthInsightBench: An Evidence-Grounded Benchmark for Automated Evaluation of Open-Ended Scientific Discovery Agents. https://arxiv.org/abs/2609.05079. As of arXiv:2609.05079v2 (2026-09-08); PDF research/pdfs/TruthInsightBench_arXiv2609.05079.pdf. Checked 2026-09-16. Type: Paper (arXiv; PDF opened).

SWE-Gate: functional pass ≠ review constraint

  1. E1

    SWE-Gate (He et al., arXiv:2609.04167v1, PDF opened 2026-09-19): repository-level benchmark with 303 repair instances across 75 open-source Python repos; derives review constraints from real PR review comments; separate functional and constraint tests plus non-compliant and gold patches.

    He et al. SWE-Gate. SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents. https://arxiv.org/abs/2609.04167. As of arXiv:2609.04167v1 (2026-09-03); PDF research/pdfs/SWE-Gate_arxiv2609.04167.pdf. Checked 2026-09-19. Type: Paper (arXiv; PDF opened).
  2. E1

    Four LLM backends under a common coding-agent scaffold (Constraint-Provided): among 644 functionally successful repairs, 221 fail the review constraints (hidden failure rate 34.3%); only 423 pass both suites. Functional-only evaluation overestimates acceptable integration.

    He et al. SWE-Gate. SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents. https://arxiv.org/abs/2609.04167. As of arXiv:2609.04167v1 (2026-09-03); PDF research/pdfs/SWE-Gate_arxiv2609.04167.pdf. Checked 2026-09-19. Type: Paper (arXiv; PDF opened).
  3. E1

    Table 2 (+C): GPT-5.5 FSR 74.9% / CFR 70.5% / JSR 52.8%; GPT-5.4-mini 61.7 / 64.2 / 39.6; DeepSeek-V4-Flash 66.7 / 64.4 / 42.9; GPT-4o-mini 9.2 / 46.4 / 4.3. Constraint text raises JSR for all four models (GPT-5.5 +11.5 pp); FSR does not rise.

    He et al. SWE-Gate. SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents. https://arxiv.org/abs/2609.04167. As of arXiv:2609.04167v1 (2026-09-03); PDF research/pdfs/SWE-Gate_arxiv2609.04167.pdf. Checked 2026-09-19. Type: Paper (arXiv; PDF opened).

AutoTuneBench: speedup headline ≠ trustworthy measurement

  1. E1

    Chen (harnets.ai; arXiv:2609.18123v1, PDF opened 2026-09-22): AutoTuneBench — measurement protocol for LLM-agent auto-tuning of serving engines (vLLM/SGLang). Four failure modes from a pilot corpus of 619 model calls: strawman baselines, non-transferable absolutes, task saturation, infrastructure defects impersonating science. Protocol as code, ingest validator, anti-cheat outside the agent modification surface, pre-registered readouts, external anchoring.

    Chen AutoTuneBench. AutoTuneBench: Trustworthy Measurement for Agent Auto-Tuning of LLM Serving Engines. https://arxiv.org/abs/2609.18123. As of arXiv:2609.18123v1 (2026-09-16); PDF research/pdfs/AutoTuneBench_arXiv2609.18123.pdf. Checked 2026-09-22. Type: Paper (arXiv; PDF opened).
  2. E1

    Honest measurement: best gelu tuning kernel 10.6× against a naive strawman, but 2.03× against the honest baseline the search started from; same config 1.174× on one machine and 1.0049× on another; pre-registered on/off A/B nulls at a shared wall (2.4840 vs 2.4957 ms); KernelBench Level-1: 51% of tasks correct, median speedup 1.0001× over PyTorch eager (misses 1.2× bar, recorded as a finding).

    Chen AutoTuneBench. AutoTuneBench: Trustworthy Measurement for Agent Auto-Tuning of LLM Serving Engines. https://arxiv.org/abs/2609.18123. As of arXiv:2609.18123v1 (2026-09-16); PDF research/pdfs/AutoTuneBench_arXiv2609.18123.pdf. Checked 2026-09-22. Type: Paper (arXiv; PDF opened).
  3. E1

    SGLang flag-space search: best committed ngram speculative decoding 1.2561× requests/s over its own default (extension 1.2753×); vLLM pilot ~1.01× — stakes are engine-dependent. Held-out device: GB10-certified 11.03× gelu reads 0.79× against a fresh Mac baseline. All GPU measurements on one DGX Spark/GB10 — paper ≠ fleet census.

    Chen AutoTuneBench. AutoTuneBench: Trustworthy Measurement for Agent Auto-Tuning of LLM Serving Engines. https://arxiv.org/abs/2609.18123. As of arXiv:2609.18123v1 (2026-09-16); PDF research/pdfs/AutoTuneBench_arXiv2609.18123.pdf. Checked 2026-09-22. Type: Paper (arXiv; PDF opened).

PartHackBench: partial credit can inflate at matched progress (Yang et al.)

  1. E1

    Yang, Xie, Xiong (Georgia Tech; arXiv:2609.29578v1, PDF opened 2026-09-27): PartHackBench — a private certifier admits pairs only when trajectories match component-wise in current-state predicate satisfaction and standardized agent attribution; score inflation Δhack = f(A)−f(H) is measured only afterward. PB-CSTE: 6 dev + 18 sealed held-out tasks.

    Yang et al. PartHackBench. PartHackBench: Certified Equal-Progress Stress Tests for Partial-Credit Tool-Agent Evaluation. https://arxiv.org/abs/2609.29578. As of arXiv:2609.29578v1 (2026-08-27); PDF research/pdfs/PartHackBench_arXiv2609.29578.pdf. Checked 2026-09-27. Type: Paper (arXiv; PDF opened).
  2. E1

    Historical Milestone: matched adversaries 15/18; mean Δhack .252 (95% CI [.154, .351]); conditional ASR 10/15 (Δhack > .10); E2E yield 10/18; rollback detection 0/14. Current-state CSPS / Lightweight CAPE: Δhack = 0 and rollback 14/14 (by construction). DeepSeek transfer mean .022 (ASR 1/15); IDJ .017 (ASR 1/15). Evaluator-targeted: DeepSeek .069 / IDJ .048. PB-CSTE lab ≠ production eval.

    Yang et al. PartHackBench. PartHackBench: Certified Equal-Progress Stress Tests for Partial-Credit Tool-Agent Evaluation. https://arxiv.org/abs/2609.29578. As of arXiv:2609.29578v1 (2026-08-27); PDF research/pdfs/PartHackBench_arXiv2609.29578.pdf. Checked 2026-09-27. Type: Paper (arXiv; PDF opened).

Physics Audit: expert re-grading shows bench defects (Ansari et al.)

  1. E1

    Ansari, Sun, Liu, Jabbour et al. (Yale / Jump Trading / Cambridge / USC; arXiv:2609.13009v1, PDF opened 2026-09-25): expert audit of six physics benches (UGPhysics, PHYBench, PRISM-Physics, HLE-Physics, CMT-Benchmark, CritPt). Most audited “incorrect” cases are benchmark/grader defects, not model physics errors.

    Ansari et al. Physics Audit. How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks. https://arxiv.org/abs/2609.13009. As of arXiv:2609.13009v1 (2026-09-11); PDF research/pdfs/PhysicsExpertRegrade_arXiv2609.13009.pdf. Checked 2026-09-25. Type: Paper (arXiv; PDF opened).
  2. E1

    GPT-5.6-Sol mean@4: HLE-Physics 47.3%→78.7%; CMT-Benchmark 61.0%→87.2%. CritPt: pre-audit mean@5 32.3% (70 challenges) → corrected mean@4 87.5% / pass@4 94.4% on 54 of 56 audited (retained) challenges.

    Ansari et al. Physics Audit. How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks. https://arxiv.org/abs/2609.13009. As of arXiv:2609.13009v1 (2026-09-11); PDF research/pdfs/PhysicsExpertRegrade_arXiv2609.13009.pdf. Checked 2026-09-25. Type: Paper (arXiv; PDF opened).
  3. E1

    HLE-Physics audit sampling (Table 2): of 98 audited cases, 86 (87.76%) benchmark error, 4 (4.08%) grader, 8 (8.16%) model error. Authors: current benches substantially understate frontier ability on well-posed physics problems.

    Ansari et al. Physics Audit. How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks. https://arxiv.org/abs/2609.13009. As of arXiv:2609.13009v1 (2026-09-11); PDF research/pdfs/PhysicsExpertRegrade_arXiv2609.13009.pdf. Checked 2026-09-25. Type: Paper (arXiv; PDF opened).

Claimed · E3

4. What is claimed, not shown

Model-level evaluations necessary but not sufficient for child safety (author reading)

  1. E3

    Jiao et al.: model-level evaluations, while necessary, cannot by themselves anticipate the harm pathways most often documented for children. They recommend mitigations beyond model-level safeguards such as content filtering: age assurance, consent verification for depiction, safety-by-design UX constraints, incident disclosure norms, and auditing regimes targeting developmental risk. Author reading from the abstract; the effectiveness of these measures is not shown by this.

    Jiao et al. child-related AI incidents. Safe by Benchmark, Unsafe in Practice: Lessons from 1331 Child-Related AI Incidents. https://arxiv.org/abs/2610.34705. As of arXiv:2610.34705v1 (submitted 2026-10-26; cs.AI, cs.CY; per comment 17 pp., 10 figures; annotations/codebook github.com/The-Responsible-AI-Initiative/KidAID not opened); abstract via export.arxiv.org API only, PDF not opened; affiliations not stated in the abstract record. Checked 2026-10-28. Type: Incident analysis (arXiv; abstract only; 1,331 publicly reported incidents coded).

Those who use individual verdicts should prespecify setups, repeat them and report both levels (author reading)

  1. E3

    Yorke: a stable aggregate score does not establish that individual verdicts are repeatable; anyone using judge verdicts as preference labels, for safety review, regression tests or release decisions should run a small, prespecified set of plausible judge setups alongside the main one, repeat each unchanged, and report the aggregate and prompt-level change rate per setup together with its test-retest noise. A methodological recommendation, not measured as an effect.

    Yorke LLM-judge stability. Behind Aggregate Scores: What Systematically Varying Arena-Hard’s LLM Judge Reveals. https://arxiv.org/abs/2610.30512. As of arXiv:2610.30512v1 (submitted 2026-10-21; cs.CL; 18 pp.; NeurIPS 2026 JUDGe workshop, non-archival); PDF research/pdfs/arXiv2610.30512.pdf; code github.com/Yorkel/llm-judge-release not opened. Checked 2026-10-24. Type: Paper (arXiv; PDF opened; 500 prompts, 17,000 judge calls).

“Weakly plausible” is no reason for inaction; evidence for the most dangerous properties stays structurally limited (author reading)

  1. E3

    Uuk et al. (mostly Future of Life Institute): given potentially irreversible consequences, policy should invest steadily in evaluation capacity — realistic, deployment-proximate test environments, post-deployment monitoring, independent third-party evaluators — and prioritise behavioural deception tests over self-reports; high external validity for the most dangerous properties may be unachievable under responsible research, and empirical assessments systematically lag the capability frontier. The authors’ policy conclusions, not part of the measurement.

    Uuk et al. loss-of-control evidence. Assessing the Empirical Evidence for Loss of Control from Agentic General-Purpose AI. https://arxiv.org/abs/2610.25241. As of arXiv:2610.25241v1 (2026-10-18; cs.CY; 42 pp.; previously SSRN 2026-05-28); PDF research/pdfs/arXiv2610.25241.pdf. Checked 2026-10-20. Type: Evidence audit (arXiv; PDF opened; preprint).

Feng: usefulness claims should name their evidence level (author reading)

  1. E3

    Author proposal: journals could require that claims of clinical usefulness name the evidence level reached, and reserve “improved care” for studies that measured care or outcomes. A normative proposal, not a measured effect.

    Feng multimodal clinical claims. Clinical value claims in multimodal AI outpace supporting evidence. https://arxiv.org/abs/2610.14395. As of arXiv:2610.14395v1 (submitted 2026-10-09); PDF research/pdfs/arXiv2610.14395.pdf. Checked 2026-10-16. Type: Meta-audit (arXiv; PDF opened; preprint).

Hidden Capabilities: elicitation audits ≠ absence (author reading)

  1. E3

    Author reading: failure of SFT/LAT/GCG on these model organisms shows a limit of elicitation-based capability audits under adversarial weight manipulation — not evidence that such lockers occur in production frontier models.

    Draguns et al. Hidden Capabilities. AI Models Can Provably Hide Arbitrary Capabilities. https://arxiv.org/abs/2610.13928. As of arXiv:2610.13928v1 (2026-10-09); PDF research/pdfs/HideCapabilities_arXiv2610.13928.pdf. Checked 2026-10-13. Type: Paper (arXiv; PDF+HTML opened).

Pakhomov & Nijkamp: item split, not aggregate (author reading)

  1. E3

    Authors: a score on this split can be read only per item, not as an aggregate; read multi-hop scores against the share of ambiguous gold answers per length tier. “Ambiguous gold” is explicitly operational (relative to the frozen parser graph), not a claim that the gold labels are wrong; re-adjudication of the 67 items with the maintainers is pending.

    Pakhomov & Nijkamp MemoryAgentBench audit. Bookkeeping, Composition, or Unreachable Gold? Reading MemoryAgentBench’s Conflict-Resolution Scores Against a Frozen Last-Write Resolver. https://arxiv.org/abs/2610.09193. As of arXiv:2610.09193v1 (2026-10-06; NeurIPS 2026 Workshop IAB); PDF research/pdfs/MemoryAgentBench_LastWrite_arXiv2610.09193.pdf. Checked 2026-10-08. Type: Paper (arXiv; PDF opened; workshop).

Snapshots 2024, not frontier 2026

  1. E2

    MMLU-Pro CoT May 2024: GPT-4o 72.6; Gemini-1.5-Pro 69.0; Claude-3-Opus 68.5; GPT-4-Turbo 63.7; Llama-3-70B-Instruct 56.2. One eval harness.

    Wang et al. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. https://arxiv.org/html/2406.01574. As of 2024-06 / NeurIPS 2024. Checked 2026-08-28. Type: Paper.
  2. E2

    Style-control rank shifts Aug 2024 (chatgpt-4o-latest stays #1; grok-2-mini 6→18) are one blog snapshot, not 2026.

    Li, Angelopoulos, Chiang. Does style matter? Disentangling style and substance in Chatbot Arena. https://www.lmsys.org/blog/2024-08-28-style-control/. As of 2024-08-29. Checked 2026-08-28. Type: Official blog.
  3. E2

    HELM 2022 winners: text-davinci-002, Anthropic-LM 52B, TNLG v2 530B. 2022 models ≠ 2026 frontier.

    Liang et al. Holistic Evaluation of Language Models. https://arxiv.org/html/2211.09110. As of 2022-11 / TMLR 2023. Checked 2026-08-28. Type: Paper.

Math instruments: GSM8K and MATH

  1. E1

    GSM8K (Cobbe et al., arXiv:2110.14168, body opened): 8.5K human-written grade-school math word problems; 7.5K train / 1K test; 2–8 steps; elementary arithmetic (+−×÷); <2% estimated breaking errors. Calculator annotations at train/test.

    Cobbe et al. GSM8K. Training Verifiers to Solve Math Word Problems. https://arxiv.org/pdf/2110.14168. As of 2021-11. Checked 2026-09-08. Type: Paper (arXiv).
  2. E1

    6B finetuned with intermediate steps: 20.6% test solve; without intermediate steps: 5.2%. Verification (ranking many completions) improves substantially over finetuning and scales better with more data than the finetuning baseline. Naive log-linear extrapolation in the paper: ~10¹⁶ parameters for 80% solve on full GSM8K train.

    Cobbe et al. GSM8K. Training Verifiers to Solve Math Word Problems. https://arxiv.org/pdf/2110.14168. As of 2021-11. Checked 2026-09-08. Type: Paper (arXiv).
  3. E1

    MATH (Hendrycks et al., arXiv:2103.03874, body opened): 12,500 competition math problems; 7,500 train / 5,000 test; 7 subjects; difficulty 1–5 (AoPS). Large LMs in the paper: 3.0%–6.9% accuracy; easiest level up to ~15%. Humans: CS PhD ~40%; three-time IMO gold ~90%.

    Hendrycks et al. MATH. Measuring Mathematical Problem Solving With the MATH Dataset. https://arxiv.org/pdf/2103.03874. As of 2021-03. Checked 2026-09-08. Type: Paper (arXiv).
  4. E1

    GPT-2 1.5B + AMPS pretrain: 6.9% overall (Table 2); 0.1B: 5.4%. AMPS: >100k Khan + >5M Mathematica, 23 GB. Training on step-by-step raises relative accuracy by 10% vs Q&A only; generating solutions at test time decreased accuracy vs immediate answer. Scaling extrapolation: ~10³⁵ params for 40% (impractical).

    Hendrycks et al. MATH. Measuring Mathematical Problem Solving With the MATH Dataset. https://arxiv.org/pdf/2103.03874. As of 2021-03. Checked 2026-09-08. Type: Paper (arXiv).
  5. E1

    Two instruments, not one: GSM8K measures grade-school word problems; MATH measures competition math. 2021 paper snapshots — no claim about 2026 frontier scores.

    Cobbe et al. GSM8K. Training Verifiers to Solve Math Word Problems. https://arxiv.org/pdf/2110.14168. As of 2021-11. Checked 2026-09-08. Type: Paper (arXiv).Hendrycks et al. MATH. Measuring Mathematical Problem Solving With the MATH Dataset. https://arxiv.org/pdf/2103.03874. As of 2021-03. Checked 2026-09-08. Type: Paper (arXiv).

BBH: answer-only underestimates multi-step

  1. E1

    BBH (Suzgun et al., arXiv:2210.09261, body opened): 23 BIG-Bench tasks where prior LM evals did not beat the average human-rater. Avg human-rater BBH all 67.7; max human-rater 94.4; best prior BIG-Bench 50.9 (0/23 above avg).

    Suzgun et al. BBH. Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. https://arxiv.org/pdf/2210.09261. As of 2022-10 / Findings ACL 2023. Checked 2026-09-09. Type: Paper (arXiv; ACL Findings).
  2. E1

    Table 2: Codex (code-davinci-002) CoT 73.9 (+16.7 vs answer-only 56.6) — 17/23 above avg human (answer-only 5/23). PaLM 540B CoT 65.2 (10/23); InstructGPT text-davinci-002 CoT 68.4 (15/23).

    Suzgun et al. BBH. Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. https://arxiv.org/pdf/2210.09261. As of 2022-10 / Findings ACL 2023. Checked 2026-09-09. Type: Paper (arXiv; ACL Findings).
  3. E1

    Authors: few-shot without CoT systematically underestimates multi-step BBH; CoT can unlock emergent task performance on otherwise flat scaling curves. Codex CoT > avg human by >6%, but >20% behind max human-rater.

    Suzgun et al. BBH. Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. https://arxiv.org/pdf/2210.09261. As of 2022-10 / Findings ACL 2023. Checked 2026-09-09. Type: Paper (arXiv; ACL Findings).

HumanEval: functional correctness as an instrument

  1. E1

    HumanEval (Chen et al., arXiv:2107.03374; body opened 2026-09-12): 164 handwritten Python problems from docstrings; mean ~7.7 unit tests; functional correctness via unit tests (not BLEU). pass@k as an unbiased estimator with n≥k (paper: n=200, k≤100).

    Chen et al. HumanEval. Evaluating Large Language Models Trained on Code. https://arxiv.org/pdf/2107.03374. As of arXiv 2021-07. Checked 2026-09-12. Type: Paper (arXiv).
  2. E1

    Codex-12B pass@1 ≈28.8% (abstract/Fig.; Table often 28.81); GPT-3 ≈0%; GPT-J 6B pass@1 ≈11.4–11.62%. With 100 samples: ~70.2% of problems solved (abstract); Codex-S supervised pass@1 37.7%, pass@100 with oracle selection 77.5%. BLEU distributions of correct vs incorrect samples overlap — BLEU ≠ functional correctness.

    Chen et al. HumanEval. Evaluating Large Language Models Trained on Code. https://arxiv.org/pdf/2107.03374. As of arXiv 2021-07. Checked 2026-09-12. Type: Paper (arXiv).
  3. E0

    Limit: interview-like single functions, not multi-file/SWE. 2026 live boards and contamination of the 164 tasks not re-measured here (E0).

    Chen et al. HumanEval. Evaluating Large Language Models Trained on Code. https://arxiv.org/pdf/2107.03374. As of arXiv 2021-07. Checked 2026-09-12. Type: Paper (arXiv).

EvalPlus / HumanEval+: test-suite inflation

  1. E1

    EvalPlus (Liu et al., arXiv:2305.01210; body opened 2026-09-13): automatic test generation (LLM seeds + type-aware mutation) augments programming benchmarks. Focus here: HumanEval → HumanEval+. 164 tasks; mean tests 9.6 → 764.1 (~80× unique tests).

    Liu et al. EvalPlus. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. https://arxiv.org/pdf/2305.01210. As of arXiv 2023-05 (v3 2023-10-30). Checked 2026-09-13. Type: Paper (arXiv).
  2. E1

    HumanEval+ lowers pass@k across evaluated LLMs by up to 23.1% (pass@1⋆) / 19.3% (pass@1) / 24.9% (pass@10) / 28.9% (pass@100). Table 3: GPT-4 pass@1⋆ 88.4 → 76.2; ChatGPT pass@1⋆ 73.2 → 63.4 (pass@1 71.6→62.5; pass@10 90.5→82.1; pass@100 96.2→91.1).

    Liu et al. EvalPlus. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. https://arxiv.org/pdf/2305.01210. As of arXiv 2023-05 (v3 2023-10-30). Checked 2026-09-13. Type: Paper (arXiv).
  3. E1

    18 defects (11% of problems) found in the original HumanEval ground truth and re-implemented for HumanEval+. HumanEval+-Mini: ~47× fewer tests with similar effectiveness on the studied models. MBPP+ is not reported with numbers in this body (future work) — not claimed here.

    Liu et al. EvalPlus. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. https://arxiv.org/pdf/2305.01210. As of arXiv 2023-05 (v3 2023-10-30). Checked 2026-09-13. Type: Paper (arXiv).
  4. E0

    2023 paper snapshot ≠ 2026 live EvalPlus boards (E0). HumanEval+/EvalPlus ≠ SWE-bench / multi-file engineering.

    Liu et al. EvalPlus. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. https://arxiv.org/pdf/2305.01210. As of arXiv 2023-05 (v3 2023-10-30). Checked 2026-09-13. Type: Paper (arXiv).

Constrained · Limit

5. Bottleneck and limit

Child-related AI incidents: public reports are a selection, not a rate; coding is judgement; abstract only

  1. E2

    Jiao et al. 2026: the 1,331 incidents are publicly reported cases — what gets reported depends on media attention, visibility and reporting channels; “most often” describes the distribution in public reporting (the abstract itself says “most prevalent in public reporting”), not the actual incidence of harm or a per-use risk. Rare or hard-to-see model defects may be under-represented. Harm types and failure modes are author-coded categories; “co-occurs” is co-occurrence, not proof of causation. The abstract gives no per-category shares, no definition of incident sources or time window, no country breakdown and no inter-coder agreement. Preprint without peer review; only the abstract via the arXiv API was read (no PDF, figures or affiliations in the record); KidAID data and codebook not checked. Do not net against benchmark scores or loss-of-control evidence (source 36).

    Jiao et al. child-related AI incidents. Safe by Benchmark, Unsafe in Practice: Lessons from 1331 Child-Related AI Incidents. https://arxiv.org/abs/2610.34705. As of arXiv:2610.34705v1 (submitted 2026-10-26; cs.AI, cs.CY; per comment 17 pp., 10 figures; annotations/codebook github.com/The-Responsible-AI-Initiative/KidAID not opened); abstract via export.arxiv.org API only, PDF not opened; affiliations not stated in the abstract record. Checked 2026-10-28. Type: Incident analysis (arXiv; abstract only; 1,331 publicly reported incidents coded).

LLM-judge study: one benchmark, one model pair near 50%, one main judge — measures change, not correctness

  1. E2

    Yorke 2026: one benchmark (Arena-Hard v2.0), one model pair chosen deliberately near 50% (hence many close calls), one main judge (GPT-4.1); Claude and Gemini only once each in May with different provider code and settings, so not a clean model effect. Temperature zero does not make hosted inference deterministic. Each instruction contrast is a whole package, not an isolated wording. No ground truth except four checkable prompts — the study measures how much verdicts move, not which setup is more correct. The two-order sign rule turns “tie plus slight lean” into a win; counting ±0.5 as a tie lowers the default rate only from 21.2% to 19.3%. Concurrent work (Ayyagari 2026) found 0.13–9.7% — the 21.2% here is higher, partly because of the near-50% pair. Non-archival workshop preprint; code not opened.

    Yorke LLM-judge stability. Behind Aggregate Scores: What Systematically Varying Arena-Hard’s LLM Judge Reveals. https://arxiv.org/abs/2610.30512. As of arXiv:2610.30512v1 (submitted 2026-10-21; cs.CL; 18 pp.; NeurIPS 2026 JUDGe workshop, non-archival); PDF research/pdfs/arXiv2610.30512.pdf; code github.com/Yorkel/llm-judge-release not opened. Checked 2026-10-24. Type: Paper (arXiv; PDF opened; 500 prompts, 17,000 judge calls).

Loss-of-control audit: subjective rubric, selected corpus, one model generation behind

  1. E2

    Uuk et al. 2026: the scores are judgments by three authors using a fixed question list (κ 0.24); studies were partly selected for their influence in the safety debate; the evidence runs to mid-2025 and is, per the authors, one model generation behind the frontier; properties are rated individually, their interplay is not modelled; passive and delegated forms of loss of control are excluded. “Weakly plausible” is a verdict on evidence quality, neither an all-clear nor a risk probability. Preprint.

    Uuk et al. loss-of-control evidence. Assessing the Empirical Evidence for Loss of Control from Agentic General-Purpose AI. https://arxiv.org/abs/2610.25241. As of arXiv:2610.25241v1 (2026-10-18; cs.CY; 42 pp.; previously SSRN 2026-05-28); PDF research/pdfs/arXiv2610.25241.pdf. Checked 2026-10-20. Type: Evidence audit (arXiv; PDF opened; preprint).

Feng audit: a literature sample, not clinical practice

  1. E2

    Single-author audit with 20% second coding; PubMed frame (an Embase audit estimates ~73 further studies; the rise in “improved care” claims 3.7% → 13.7% is then no longer significant); AUROC for both models in only 127 studies; full text missing for 18 records; observational associations. Says nothing about whether individual multimodal models help clinically — only that the published evidence mostly does not show it.

    Feng multimodal clinical claims. Clinical value claims in multimodal AI outpace supporting evidence. https://arxiv.org/abs/2610.14395. As of arXiv:2610.14395v1 (submitted 2026-10-09); PDF research/pdfs/arXiv2610.14395.pdf. Checked 2026-10-16. Type: Meta-audit (arXiv; PDF opened; preprint).

Hidden Capabilities: PoC ≠ frontier sabotage

  1. E2

    Small PyTorch model organisms, chemistry classification and subset parity — not a full embedding in Gemma 3 270M; O(n²) residual-feature overhead; cryptographic assumptions (random oracle / min-entropy). Bench ≠ supply-chain attack on production LLMs.

    Draguns et al. Hidden Capabilities. AI Models Can Provably Hide Arbitrary Capabilities. https://arxiv.org/abs/2610.13928. As of arXiv:2610.13928v1 (2026-10-09); PDF research/pdfs/HideCapabilities_arXiv2610.13928.pdf. Checked 2026-10-13. Type: Paper (arXiv; PDF+HTML opened).

MemoryAgentBench audit: one split ≠ memory in general

  1. E2

    One split, one rule, one parser; only four fact lists (intervals conditional on those four), and the held-out lists share the development list’s templates. No memory systems with learned or structured stores tested — for them the reinterpretation holds only by analogy. Gemini one run per item, outcomes only, no traces; the BM25 agent is an approximate re-implementation (top-10 chunks match the closest official variant on 134/800 items). 13 parser-scope labels assigned LLM-assisted (GPT-5.6). Workshop paper, funded by Salesforce, no independent replication. No verdict on conflict resolution in general: probes that defeat “newest wins” exist (StateMemBench, per the paper).

    Pakhomov & Nijkamp MemoryAgentBench audit. Bookkeeping, Composition, or Unreachable Gold? Reading MemoryAgentBench’s Conflict-Resolution Scores Against a Frozen Last-Write Resolver. https://arxiv.org/abs/2610.09193. As of arXiv:2610.09193v1 (2026-10-06; NeurIPS 2026 Workshop IAB); PDF research/pdfs/MemoryAgentBench_LastWrite_arXiv2610.09193.pdf. Checked 2026-10-08. Type: Paper (arXiv; PDF opened; workshop).

Agents-as-Systems: AI4S grid ≠ universal agent eval

  1. E1

    AI4S bench measures coding agents with specialist tools under a controlled config grid — no warrant for desktop/open-world AGI. ~54% noise holds among genuine attempts; prompted Verification ≠ oracle tool; specialist refusal (AstroSage) describes this setup, not every tool regime.

    Wiedmann et al. Agents-as-Systems. Agents Are Systems, Not Models: Rethinking Agentic Evaluation. https://arxiv.org/abs/2610.01618. As of arXiv:2610.01618v1 (2026-10-01); PDF research/pdfs/AgentsAsSystems_arXiv2610.01618.pdf. Checked 2026-10-04. Type: Paper (arXiv; PDF opened).

VISTA: public set / system score ≠ private AGI

  1. E1

    RHAE measures harness+model on the 25 public ARC-AGI-3 games; private set not opened. GameWorld/AI GameStore are further visual games — no warrant for open desktop/lab AGI or that every multimodal model matches without VISTA.

    Han et al. VISTA. VISTA: A Visual Harness for Reasoning in an Interactive World. https://arxiv.org/abs/2610.02200. As of arXiv:2610.02200v1 (2026-10-01); PDF research/pdfs/VISTA_arXiv2610.02200.pdf. Checked 2026-10-02. Type: Paper (arXiv; PDF opened).

VLA Accel Audit: audited tasks ≠ live robotics

  1. E1

    Audited tasks/methods are a subset; sim≠real remains (masses, smoothness transfer); fixes as pending PRs / released revisions — no warrant for all VLA acceleration claims outside the audited benches.

    Chen et al. VLA Accel Audit. Faster and Better? Benchmark Bugs and Design Limitations Distort the Evaluation of Vision-Language-Action Acceleration. https://arxiv.org/abs/2609.37771. As of arXiv:2609.37771v1 (2026-09-29); PDF research/pdfs/VLAAccelAudit_arXiv2609.37771.pdf. Checked 2026-09-30. Type: Paper (arXiv; PDF opened).

MechBench: symbolic elimination ≠ lab discovery

  1. E1

    MechBench measures mechanism discovery in a synthetic symbolic-elimination subset of science (controlled mutations, harness/wall-clock budgets). ≠ lab discovery / peer review; SA(P)/SA(M) describe the instrument, not open research.

    Yu et al. MechBench. MechBench: Can AI Scientific Agents Discover Mechanisms Beyond Phenomenal Laws?. https://arxiv.org/abs/2609.35515. As of arXiv:2609.35515v1 (2026-09-28); PDF research/pdfs/MechBench_arXiv2609.35515.pdf. Checked 2026-09-29. Type: Paper (arXiv; PDF opened).

What a score is not

  1. E1

    Leaderboard live 2026 frontier scores on GPQA not opened here (E0 for 2026 live boards). GPQA measures hard Q&A for scalable oversight, not software engineering (contrast SWE-bench, already in the dossier).

    Rein et al. GPQA. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. https://arxiv.org/pdf/2311.12022. As of 2023-11-20. Checked 2026-09-05. Type: Paper (arXiv; COLM).
  2. E0

    2026 live leaderboards for GSM8K and MATH not opened here (E0).

    Cobbe et al. GSM8K. Training Verifiers to Solve Math Word Problems. https://arxiv.org/pdf/2110.14168. As of 2021-11. Checked 2026-09-08. Type: Paper (arXiv).Hendrycks et al. MATH. Measuring Mathematical Problem Solving With the MATH Dataset. https://arxiv.org/pdf/2103.03874. As of 2021-03. Checked 2026-09-08. Type: Paper (arXiv).
  3. E0

    2026 live leaderboards / replications for BBH not opened here (E0).

    Suzgun et al. BBH. Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. https://arxiv.org/pdf/2210.09261. As of 2022-10 / Findings ACL 2023. Checked 2026-09-09. Type: Paper (arXiv; ACL Findings).
  4. E1

    HumanEval ≠ SWE-bench / multi-file engineering: single functions from docstrings; pass@k with oracle ≠ deployment pass@1. 2021 paper snapshot ≠ 2026 live boards (E0 for live scores).

    Chen et al. HumanEval. Evaluating Large Language Models Trained on Code. https://arxiv.org/pdf/2107.03374. As of arXiv 2021-07. Checked 2026-09-12. Type: Paper (arXiv).
  5. E1

    HumanEval+ hardens the instrument (more tests, fixed ground truth) — still measures single functions, not multi-file SWE. Live boards 2026 E0.

    Liu et al. EvalPlus. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. https://arxiv.org/pdf/2305.01210. As of arXiv 2023-05 (v3 2023-10-30). Checked 2026-09-13. Type: Paper (arXiv).
  6. E1

    BBH avg human-rater ≠ max human-rater; CoT score ≠ “reasoning solved”. 2022/23 paper snapshot ≠ 2026 frontier.

    Suzgun et al. BBH. Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. https://arxiv.org/pdf/2210.09261. As of 2022-10 / Findings ACL 2023. Checked 2026-09-09. Type: Paper (arXiv; ACL Findings).
  7. E1

    GSM8K grade-school ≠ competition MATH ≠ GPQA graduate Q&A — three different instruments; one score does not replace the others.

    Cobbe et al. GSM8K. Training Verifiers to Solve Math Word Problems. https://arxiv.org/pdf/2110.14168. As of 2021-11. Checked 2026-09-08. Type: Paper (arXiv).Hendrycks et al. MATH. Measuring Mathematical Problem Solving With the MATH Dataset. https://arxiv.org/pdf/2103.03874. As of 2021-03. Checked 2026-09-08. Type: Paper (arXiv).Rein et al. GPQA. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. https://arxiv.org/pdf/2311.12022. As of 2023-11-20. Checked 2026-09-05. Type: Paper (arXiv; COLM).
  8. E1

    Paper snapshots (Cobbe 2021 / Hendrycks 2021) ≠ 2026 frontier — E1 only for opened paper measurements.

    Cobbe et al. GSM8K. Training Verifiers to Solve Math Word Problems. https://arxiv.org/pdf/2110.14168. As of 2021-11. Checked 2026-09-08. Type: Paper (arXiv).Hendrycks et al. MATH. Measuring Mathematical Problem Solving With the MATH Dataset. https://arxiv.org/pdf/2103.03874. As of 2021-03. Checked 2026-09-08. Type: Paper (arXiv).
  9. E2

    Saturation ≠ mastery: MMLU plateau 86–87% with MMLU-Pro 72.6% (27 pp of ceiling); HELM summarization metrics do not discriminate; Bowman: score near ceiling with CheckList fail.

    Wang et al. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. https://arxiv.org/html/2406.01574. As of 2024-06 / NeurIPS 2024. Checked 2026-08-28. Type: Paper.Liang et al. Holistic Evaluation of Language Models. https://arxiv.org/html/2211.09110. As of 2022-11 / TMLR 2023. Checked 2026-08-28. Type: Paper.Bowman & Dahl. What Will it Take to Fix Benchmarking in Natural Language Understanding?. https://arxiv.org/html/2104.02145. As of 2021-04 / NAACL 2021. Checked 2026-08-28. Type: Position paper.
  10. E1

    Contamination does not explain all gains: Magar expl only under duplication/timing; Oren little verbatim on 4 open models; Hendrycks no entropy coupling. LCB time-window drops can coexist — do not net them against each other.

    Magar & Schwartz. Data Contamination: From Memorization to Exploitation. https://arxiv.org/html/2203.08242. As of 2022-03 / ACL 2022. Checked 2026-08-28. Type: Paper.Oren et al. Proving Test Set Contamination in Black-Box Language Models. https://arxiv.org/html/2310.17623. As of 2023-10 / ICLR 2024. Checked 2026-08-28. Type: Paper.Hendrycks et al. Measuring Massive Multitask Language Understanding. https://arxiv.org/html/2009.03300. As of 2020-09 / ICLR 2021. Checked 2026-08-28. Type: Paper.Jain et al. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. https://arxiv.org/html/2403.07974. As of 2024-03 (HTML: 511 problems May 2023–May 2024). Checked 2026-08-28. Type: Paper.
  11. E2

    Arena ≠ capability: pairwise pref, diverse prompts, BT CIs. Not: safety, facts, production, ground truth. Style control is an interpretation, not “true” strength.

    Chiang et al. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. https://arxiv.org/html/2403.04132. As of 2024-03-07 (ICML 2024). Checked 2026-08-28. Type: Paper.Li, Angelopoulos, Chiang. Does style matter? Disentangling style and substance in Chatbot Arena. https://www.lmsys.org/blog/2024-08-28-style-control/. As of 2024-08-29. Checked 2026-08-28. Type: Official blog.
  12. E1

    Arena measures preference (helpfulness); TruthfulQA measures avoidance of imitative falsehood. These are different instruments. An Arena rank is not a truth rank.

    Chiang et al. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. https://arxiv.org/html/2403.04132. As of 2024-03-07 (ICML 2024). Checked 2026-08-28. Type: Paper.Lin, Hilton, Evans. TruthfulQA: Measuring How Models Mimic Human Falsehoods. https://arxiv.org/html/2109.07958. As of 2021-09-08. Checked 2026-08-29. Type: Paper (NeurIPS).
  13. E1

    SWE % resolved ≠ SE ability: execution-based, real issues. Not: retrieval-independent, multi-language, design. 2023 SOTA 2%.

    Jimenez et al. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. https://arxiv.org/html/2310.06770. As of 2023-10 / ICLR 2024. Checked 2026-08-28. Type: Paper.
  14. E1

    Emergence not refuted: many claims are metric-induced. Schaeffer does not rule out real emergence. HELM finding 25 (scale threshold ≥50B on accuracy) remains compatible.

    Schaeffer et al. Are Emergent Abilities of Large Language Models a Mirage?. https://arxiv.org/html/2304.15004. As of 2023-04 / NeurIPS 2023. Checked 2026-08-28. Type: Paper.Liang et al. Holistic Evaluation of Language Models. https://arxiv.org/html/2211.09110. As of 2022-11 / TMLR 2023. Checked 2026-08-28. Type: Paper.
  15. E0

    SWE-Verified dead as a frontier measure: OpenAI page timeout. 59.4% flawed tests = E0. Model names from aggregator snippets (GPT-5.2, Claude Opus 4.5, Gemini 3 Flash) not in the log.

Discovery score ≠ discovery

  1. E1

    TruthInsightBench plateau ~58–60/100 and LLM judge: instrument for evidentiary maturity of own claims, not a thermometer of scientific discovery; one frozen base model; paper does not claim auto-scientist maturity.

    TruthInsightBench. TruthInsightBench: An Evidence-Grounded Benchmark for Automated Evaluation of Open-Ended Scientific Discovery Agents. https://arxiv.org/abs/2609.05079. As of arXiv:2609.05079v2 (2026-09-08); PDF research/pdfs/TruthInsightBench_arXiv2609.05079.pdf. Checked 2026-09-16. Type: Paper (arXiv; PDF opened).

Functional pass ≠ review gate / product SE

  1. E1

    SWE-Gate: 221/644 hidden failures show that repo tests alone do not measure review acceptance. Benchmark on Python repos and four backends — no warrant for product SE, multi-language deploy, or 2026 live SWE-Verified boards.

    He et al. SWE-Gate. SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents. https://arxiv.org/abs/2609.04167. As of arXiv:2609.04167v1 (2026-09-03); PDF research/pdfs/SWE-Gate_arxiv2609.04167.pdf. Checked 2026-09-19. Type: Paper (arXiv; PDF opened).

AutoTuneBench: one-box protocol ≠ fleet tune

  1. E1

    AutoTuneBench measures agent tuning under a frozen protocol on one GB10 node. Strawman speedups and KernelBench median 1.0001× show measurement integrity, not that every serving fleet gains 1.26×. Cross-box only as ratios; held-out 0.79× warns against device transfer.

    Chen AutoTuneBench. AutoTuneBench: Trustworthy Measurement for Agent Auto-Tuning of LLM Serving Engines. https://arxiv.org/abs/2609.18123. As of arXiv:2609.18123v1 (2026-09-16); PDF research/pdfs/AutoTuneBench_arXiv2609.18123.pdf. Checked 2026-09-22. Type: Paper (arXiv; PDF opened).

PartHackBench: PB-CSTE lab ≠ production eval

  1. E1

    PartHackBench measures evaluator robustness in a deterministic PB-CSTE environment (18 held-out; no networks/stochasticity). White-box prompted attacks (K=3) ≠ spontaneous exploit rate in production; current-state invariance holds for the certified components, not for all score axes.

    Yang et al. PartHackBench. PartHackBench: Certified Equal-Progress Stress Tests for Partial-Credit Tool-Agent Evaluation. https://arxiv.org/abs/2609.29578. As of arXiv:2609.29578v1 (2026-08-27); PDF research/pdfs/PartHackBench_arXiv2609.29578.pdf. Checked 2026-09-27. Type: Paper (arXiv; PDF opened).

Physics Audit: corrected subsets ≠ open research

  1. E1

    Corrected scores are on retained/repaired subsets (not identical question sets); closed-ended text-only (tools for some models) ≠ open research physics; near-saturation on well-posed problem-set physics ≠ AGI / lab modeling.

    Ansari et al. Physics Audit. How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks. https://arxiv.org/abs/2609.13009. As of arXiv:2609.13009v1 (2026-09-11); PDF research/pdfs/PhysicsExpertRegrade_arXiv2609.13009.pdf. Checked 2026-09-25. Type: Paper (arXiv; PDF opened).

6. Actors and incentives

Who builds the instrument

  1. E1

    Stanford CRFM runs HELM as a living benchmark; the 2022 paper is the campaign, the 2026 landing is the ecosystem.

    Liang et al. Holistic Evaluation of Language Models. https://arxiv.org/html/2211.09110. As of 2022-11 / TMLR 2023. Checked 2026-08-28. Type: Paper.HELM landing. Holistic Evaluation of Language Models (HELM) Leaderboards. https://crfm.stanford.edu/helm/. As of page as of 2026-08-28. Checked 2026-08-28. Type: Official site.
  2. E2

    LMSYS / LMArena: preference board plus style control. Live 2026 not opened.

    Chiang et al. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. https://arxiv.org/html/2403.04132. As of 2024-03-07 (ICML 2024). Checked 2026-08-28. Type: Paper.Li, Angelopoulos, Chiang. Does style matter? Disentangling style and substance in Chatbot Arena. https://www.lmsys.org/blog/2024-08-28-style-control/. As of 2024-08-29. Checked 2026-08-28. Type: Official blog.
  3. E1

    Princeton/NLP group (SWE-bench) and Berkeley/MIT et al. (LiveCodeBench) supply execution evals with their own confounds.

    Jimenez et al. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. https://arxiv.org/html/2310.06770. As of 2023-10 / ICLR 2024. Checked 2026-08-28. Type: Paper.Jain et al. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. https://arxiv.org/html/2403.07974. As of 2024-03 (HTML: 511 problems May 2023–May 2024). Checked 2026-08-28. Type: Paper.

7. State of the dispute

Emergence, contamination, one score

  1. E1

    Schaeffer vs. Wei: both bodies opened (sources 7 and 17). Wei: unpredictable scale thresholds; Schaeffer: many claims metric-induced. No reconciliation in this entry. HELM scale threshold remains.

    Schaeffer et al. Are Emergent Abilities of Large Language Models a Mirage?. https://arxiv.org/html/2304.15004. As of 2023-04 / NeurIPS 2023. Checked 2026-08-28. Type: Paper.Wei et al. Emergent. Emergent Abilities of Large Language Models. https://arxiv.org/pdf/2206.07682. As of 2022-06 / TMLR 2022. Checked 2026-09-06. Type: Paper (arXiv; TMLR).Liang et al. Holistic Evaluation of Language Models. https://arxiv.org/html/2211.09110. As of 2022-11 / TMLR 2023. Checked 2026-08-28. Type: Paper.
  2. E2

    Oren “little evidence for pervasive contamination” on four open models vs. LCB performance drops on post-cutoff contest problems. Both can be true.

    Oren et al. Proving Test Set Contamination in Black-Box Language Models. https://arxiv.org/html/2310.17623. As of 2023-10 / ICLR 2024. Checked 2026-08-28. Type: Paper.Jain et al. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. https://arxiv.org/html/2403.07974. As of 2024-03 (HTML: 511 problems May 2023–May 2024). Checked 2026-08-28. Type: Paper.

8. Open questions

  1. OpenAI SWE-Verified withdrawal 2026-02-23 fetch again — timeout.
  2. SWE-Bench Pro (Deng 2025) and SWE-bench+ (Aleithan) not opened in this run.
  3. Chen HumanEval 2021 opened 2026-09-12 (source 21, E1); Liu EvalPlus/HumanEval+ 2023 opened 2026-09-13 (source 22, E1); MBPP+ figures absent from opened body; live boards 2026 E0.
  4. Wei 2022 Emergent Abilities — opened 2026-09-06 (source 17).
  5. GSM8K (Cobbe) and MATH (Hendrycks) — opened 2026-09-08 (sources 18–19); BBH (Suzgun) opened 2026-09-09 (source 20); 2026 live boards still E0.
  6. HLE, LiveBench, SimpleQA; GPQA body opened 2026-09-05 (source 16); 2026 live boards still E0.
  7. HELM Classic/Capabilities live tables 2026.
  8. Arena.ai / LMArena live board 2026.
  9. LiveCodeBench leaderboard + ICLR 2025 (600+ problems).
  10. GPT-4 Technical Report (MMLU 86.4% only via Wang).

9. Changes

  • v3.3-draft2026-10-28: Jiao et al. child-related AI incidents arXiv:2610.34705 (source 38, E2, abstract only) — 1,331 publicly reported incidents coded against a child-centred taxonomy; harm concentrated in representational, dignity-related, psychological categories; co-occurs mostly with deployment-context, governance and misuse failures rather than model defects; individual often linked to population-level harm; preprint, abstract warrant.
  • v3.2-draft2026-10-24: Yorke Arena-Hard LLM judge arXiv:2610.30512 (source 37, E2, PDF) — 500 fixed prompts, 17 runs; unchanged rerun moves aggregate ~1 point, flips 21.2% of prompts; only 131/496 (26.4%) stable across all 17; label-only −4.09 points / 41.6%; Gemini 38.3% at almost the same aggregate; kappa 0.44 vs 0.76; verifiable case: default judge crowns the failing submission.
  • v3.1-draft2026-10-20: Uuk et al. loss-of-control evidence arXiv:2610.25241 (source 36, E2, PDF) — 19 studies, 8 properties, 25 pairs × 3 validity dimensions; overall weakly plausible; deception 7 assessments weakly plausible, situational awareness weakly implausible, replication speculative; no high external validity; κ 0.24 / AC1 0.67; models up to mid-2025.
  • v2.9-draft2026-10-13: Hidden Capabilities Draguns et al. arXiv:2610.13928 (source 34, E2, PDF/HTML) — digital lockers; chemistry SFT 50.9% vs 92.8%; trigger 73.5%; GCG 16-bit 52.7%; ~2.07B parameters; PoC ≠ frontier sabotage.
  • v2.8-draft2026-10-08: MemoryAgentBench audit Pakhomov & Nijkamp arXiv:2610.09193 (source 33, E2, PDF) — rule “newest wins” 80.25% / held-out 74.5%; 67 items only via overwritten facts; models 10.4–11.9% there vs 82.6–84.7% elsewhere; one split ≠ memory in general.
  • v2.7-draft2026-10-04: Agents-as-Systems Wiedmann et al. arXiv:2610.01618 (source 32, E1, PDF) — ~54% run noise; Information rank 1; 18,240 trajectories; Verification last; oracle > prompt; AI4S ≠ universal.
  • v2.6-draft2026-10-02: VISTA Han et al. arXiv:2610.02200 (source 31, E1, PDF) — Opus RHAE 40.68→100.00; −57.4% actions; Sol+VISTA 99.00; public ≠ private.
  • v2.5-draft2026-10-01: OSWorld-Science Dai et al. arXiv:2609.39903 (source 30, E1, PDF) — 146 tasks / 12 VLMs; Fable 5.1 73.7%; chemistry ~44%; full credit 26.3%; bench ≠ live lab.
  • v2.4-draft2026-09-30: VLA Accel Audit Chen et al. arXiv:2609.37771 (source 29, E1, PDF) — 7 benches / 22 bugs + 4 design limits; Spatial-Task 0.4%→53.2%; place shoe 65→74; place fan 40/61→38/33; sim≠real.
  • v2.3-draft2026-09-29: MechBench Yu et al. arXiv:2609.35515 (source 28, E1, PDF) — Core SA(P) 35.00% / SA(M) 13.75%; 1-(M|P) 64.29%; Full 16.02%/7.81%; Gold-P Direct-Ask SA(M) 45%/49%; symbolic ≠ lab discovery.
  • v2.2-draft2026-09-27: PartHackBench Yang et al. arXiv:2609.29578 (source 27, E1, PDF) — historical mean Δhack .252; ASR 10/15; rollback 0/14; current-state 0; DeepSeek .022 / IDJ .017; PB-CSTE ≠ production.
  • v2.1-draft2026-09-25: Physics Audit Ansari et al. arXiv:2609.13009 (source 26, E1, PDF) — HLE-Physics 47.3%→78.7%; CMT 61.0%→87.2%; CritPt 32.3%→87.5% / pass@4 94.4% (54/56); HLE audit 86/98 benchmark errors; subsets ≠ open research.
  • v2.0-draft2026-09-22: AutoTuneBench Chen arXiv:2609.18123 (source 25, E1, PDF) — strawman 10.6×→2.03×; KernelBench L1 51% / median 1.0001×; SGLang ngram 1.256×; held-out 0.79×; measurement ≠ fleet.
  • v1.9-draft2026-09-19: SWE-Gate He et al. arXiv:2609.04167 (source 24, E1, PDF) — 303 instances / 75 repos; 221/644 functionally correct fail constraints (HFR 34.3%); GPT-5.5 JSR 52.8%; functional pass ≠ review gate.
  • v1.8-draft2026-09-16: TruthInsightBench arXiv:2609.05079v2 (source 23, E1, PDF) — 40 blind tasks / 10 domains; plateau 58.4–60.3/100; bottleneck judgment ≠ coding; discovery out of reach.
  • v1.7-draft2026-09-13: EvalPlus/HumanEval+ (Liu et al., arXiv:2305.01210, source 22, E1) opened — 9.6→764.1 tests (~80×); pass@k drops up to 19.3–28.9%; GPT-4 88.4→76.2; ChatGPT 73.2→63.4; 18 ground-truth defects (11%); MBPP+ not in body; live boards E0.
  • v1.6-draft2026-09-12: HumanEval (Chen et al., arXiv:2107.03374, source 21, E1) opened — 164 problems, ~7.7 tests; Codex-12B pass@1 ≈28.8%; GPT-J ≈11.4–11.62%; Codex-S 37.7% / oracle pass@100 77.5%; BLEU ≠ correctness; live boards/contamination E0.
  • v1.5-draft2026-09-09: BBH (Suzgun et al., arXiv:2210.09261, source 20, E1) opened — answer-only underestimates multi-step; Codex CoT 73.9 / 17 of 23 above avg human; live boards E0.
  • v1.4-draft2026-09-08: GSM8K (Cobbe et al., arXiv:2110.14168, source 18, E1) and MATH (Hendrycks et al., arXiv:2103.03874, source 19, E1) opened — math instruments; live boards 2026 E0; Sainz note on GSM8K body updated.
  • v1.3-draft2026-09-06: Wei et al. Emergent Abilities (arXiv:2206.07682, source 17, E1) opened — definition + Table 1 examples; contrast with Schaeffer without reconciliation.
  • v1.2-draft2026-09-05: GPQA (Rein et al., arXiv:2311.12022 PDF, source 16) opened — 448/546/198; experts ~65% (74% post-hoc); non-experts ~34%; GPT-4 few-shot CoT ~39%; Diamond filter. Live boards 2026 still E0.
  • v1.1-draft2026-08-29: TruthfulQA (arXiv:2109.07958 HTML) opened; inverse scaling 2021 as an instrument, not as 2026 SOTA.
  • v1.0-draftFirst version from the 2026-08-28 verification log.

10. Sources

No. Source As of Checked Grade
1 Liang, P.; Bommasani, R.; Lee, T.; Tsipras, D.; Soylu, D.; Yasunaga, M.; et al.. Holistic Evaluation of Language Models. https://arxiv.org/html/2211.09110. Type: Paper. E1
2 Chiang, W.-L.; Zheng, L.; Sheng, Y.; Angelopoulos, A. N.; Li, T.; Li, D.; et al.. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. https://arxiv.org/html/2403.04132. Type: Paper. E2
3 Jimenez, C. E.; Yang, J.; Wettig, A.; Yao, S.; Pei, K.; Press, O.; Narasimhan, K.. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. https://arxiv.org/html/2310.06770. Type: Paper. E1
4 Jain, N.; Han, K.; Gu, A.; Li, W.; Yan, F.; Zhang, T.; et al.. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. https://arxiv.org/html/2403.07974. Type: Paper. E2
5 Hendrycks, D.; Burns, C.; Basart, S.; Zou, A.; Mazeika, M.; Song, D.; Steinhardt, J.. Measuring Massive Multitask Language Understanding. https://arxiv.org/html/2009.03300. Type: Paper. E1
6 Wang, Y.; Ma, X.; Zhang, G.; Ni, Y.; Chandra, A.; Guo, S.; et al.. MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark. https://arxiv.org/html/2406.01574. Type: Paper. E2
7 Schaeffer, R.; Miranda, B.; Koyejo, S.. Are Emergent Abilities of Large Language Models a Mirage?. https://arxiv.org/html/2304.15004. Type: Paper. E1
8 Magar, I.; Schwartz, R.. Data Contamination: From Memorization to Exploitation. https://arxiv.org/html/2203.08242. Type: Paper. E1
9 Sainz, O.; Campos, J. A.; García-Ferrero, I.; Etxaniz, J.; Lopez de Lacalle, O.; Agirre, E.. NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark. https://arxiv.org/html/2310.18018. Type: Position paper. E2
10 Oren, Y.; Meister, N.; Chatterji, N.; Ladhak, F.; Hashimoto, T.. Proving Test Set Contamination in Black-Box Language Models. https://arxiv.org/html/2310.17623. Type: Paper. E1
11 Bowman, S. R.; Dahl, G. E.. What Will it Take to Fix Benchmarking in Natural Language Understanding?. https://arxiv.org/html/2104.02145. Type: Position paper. E2
12 Alzahrani, N.; Alyahya, H.; Alnumay, Y.; Alrashed, S.; Alsubaie, S.; et al.. When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards. https://arxiv.org/html/2402.01781. Type: Paper. E2
13 Li, T.; Angelopoulos, A. N.; Chiang, W.-L.. Does style matter? Disentangling style and substance in Chatbot Arena. https://www.lmsys.org/blog/2024-08-28-style-control/. Type: Official blog. E2
14 Stanford CRFM. Holistic Evaluation of Language Models (HELM) Leaderboards. https://crfm.stanford.edu/helm/. Type: Official site. E1
15 Lin, S.; Hilton, J.; Evans, O.. TruthfulQA: Measuring How Models Mimic Human Falsehoods. https://arxiv.org/html/2109.07958. Type: Paper (NeurIPS). E1
16 Rein, D.; Hou, B. L.; Stickland, A. C.; Petty, J.; Pang, R. Y.; Dirani, J.; Michael, J.; Bowman, S. R.. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. https://arxiv.org/pdf/2311.12022. Type: Paper (arXiv; COLM). E1
17 Wei, J.; Tay, Y.; Bommasani, R.; Raffel, C.; Zoph, B.; Borgeaud, S.; Yogatama, D.; Bosma, M.; Zhou, D.; Metzler, D.; Chi, E. H.; Hashimoto, T.; Vinyals, O.; Liang, P.; Dean, J.; Fedus, W.. Emergent Abilities of Large Language Models. https://arxiv.org/pdf/2206.07682. Type: Paper (arXiv; TMLR). E1
18 Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; Hesse, C.; Schulman, J.. Training Verifiers to Solve Math Word Problems. https://arxiv.org/pdf/2110.14168. Type: Paper (arXiv). E1
19 Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; Steinhardt, J.. Measuring Mathematical Problem Solving With the MATH Dataset. https://arxiv.org/pdf/2103.03874. Type: Paper (arXiv). E1
20 Suzgun, M.; Scales, N.; Schärli, N.; Gehrmann, S.; Tay, Y.; Chung, H. W.; Chowdhery, A.; Le, Q.; Chi, E.; Zhou, D.; Wei, J.. Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. https://arxiv.org/pdf/2210.09261. Type: Paper (arXiv; ACL Findings). E1
21 Chen, M.; Tworek, J.; Jun, H.; Yuan, Q.; et al. (OpenAI). Evaluating Large Language Models Trained on Code. https://arxiv.org/pdf/2107.03374. Type: Paper (arXiv). E1
22 Liu, J.; Xia, C. S.; Wang, Y.; Zhang, L.. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. https://arxiv.org/pdf/2305.01210. Type: Paper (arXiv). E1
23 TruthInsight-AI (collective authorship; correspondence TruthInsightBench@163.com). TruthInsightBench: An Evidence-Grounded Benchmark for Automated Evaluation of Open-Ended Scientific Discovery Agents. https://arxiv.org/abs/2609.05079. Type: Paper (arXiv; PDF opened). E1
24 He, X.; Wang, Y.; Liu, M.; Chen, J.; Zhang, H.; Li, G.. SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents. https://arxiv.org/abs/2609.04167. Type: Paper (arXiv; PDF opened). E1
25 Chen, L. (harnets.ai). AutoTuneBench: Trustworthy Measurement for Agent Auto-Tuning of LLM Serving Engines. https://arxiv.org/abs/2609.18123. Type: Paper (arXiv; PDF opened). E1
26 Ansari, A.; Sun, H.; Liu, A. Z.; Jabbour, M.; et al. (Yale; Jump Trading; Cambridge; USC). How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks. https://arxiv.org/abs/2609.13009. Type: Paper (arXiv; PDF opened). E1
27 Yang, H.; Xie, Z.; Xiong, S. (Georgia Institute of Technology). PartHackBench: Certified Equal-Progress Stress Tests for Partial-Credit Tool-Agent Evaluation. https://arxiv.org/abs/2609.29578. Type: Paper (arXiv; PDF opened). E1
28 Yu, Z.; Zhang, J.; Cheng, J.; Ding, J.; Li, Y. (Tsinghua University; Chinese Academy of Sciences; Beijing Zhongguancun Academy; Xi'an Jiaotong University). MechBench: Can AI Scientific Agents Discover Mechanisms Beyond Phenomenal Laws?. https://arxiv.org/abs/2609.35515. Type: Paper (arXiv; PDF opened). E1
29 Chen, Q.; Zhou, K.; Shi, N.; Li, Z.; Feng, Y.; Gu, J. (Shanghai Jiao Tong University). Faster and Better? Benchmark Bugs and Design Limitations Distort the Evaluation of Vision-Language-Action Acceleration. https://arxiv.org/abs/2609.37771. Type: Paper (arXiv; PDF opened). E1
30 Dai, D.; Qi, H.; Liu, L.; Li, Y.; Chen, B.; Dou, Z.; et al.; Liu, T. (corr.) (Tsinghua; UCLA; RIKEN AIP; Waterloo; CMU; Yale; et al.). OSWorld-Science: A Benchmark of Computer Use Agents for Learning and Using Scientific Software. https://arxiv.org/abs/2609.39903. Type: Paper (arXiv; PDF opened). E1
31 Han, Q.; Hu, K.; Qiu, L.; Wu, C.; He, K. (MIT). VISTA: A Visual Harness for Reasoning in an Interactive World. https://arxiv.org/abs/2610.02200. Type: Paper (arXiv; PDF opened). E1
32 Wiedmann, L.; Girrbach, L.; Schmid, C.; Akata, Z. (Technical University of Munich, MCML; Helmholtz Munich; Inria, ENS, CNRS, PSL). Agents Are Systems, Not Models: Rethinking Agentic Evaluation. https://arxiv.org/abs/2610.01618. Type: Paper (arXiv; PDF opened). E1
33 Pakhomov, E.; Nijkamp, E. (Salesforce AI Research). Bookkeeping, Composition, or Unreachable Gold? Reading MemoryAgentBench’s Conflict-Resolution Scores Against a Frozen Last-Write Resolver. https://arxiv.org/abs/2610.09193. Type: Paper (arXiv; PDF opened; workshop). E2
34 Draguns, A.; Motwani, S. R.; Douglas, R.; Schroeder de Witt, C. (Contramont Research; University of Oxford; ACS Research). AI Models Can Provably Hide Arbitrary Capabilities. https://arxiv.org/abs/2610.13928. Type: Paper (arXiv; PDF+HTML opened). E2
35 Feng, J. (Department of Health Policy and Management, Harvard T.H. Chan School of Public Health). Clinical value claims in multimodal AI outpace supporting evidence. https://arxiv.org/abs/2610.14395. Type: Meta-audit (arXiv; PDF opened; preprint). E2
36 Uuk, R. (Future of Life Institute; KU Leuven; LSE); Koivula, S. (FLI; ETH Zurich); Pacchiardi, L. (University of Cambridge); Gipiškis, R. (FLI; AI Standards Lab; Vilnius University); Caroli, L. (independent); Horton, K. (RAND). Assessing the Empirical Evidence for Loss of Control from Agentic General-Purpose AI. https://arxiv.org/abs/2610.25241. Type: Evidence audit (arXiv; PDF opened; preprint). E2
37 Yorke, L. (University College London). Behind Aggregate Scores: What Systematically Varying Arena-Hard’s LLM Judge Reveals. https://arxiv.org/abs/2610.30512. Type: Paper (arXiv; PDF opened; 500 prompts, 17,000 judge calls). E2
38 Jiao, J.; Chen, K.; Mishra, S.; Murali, A.; Atkinson, D.; Raj, S.; Afroogh, S.. Safe by Benchmark, Unsafe in Practice: Lessons from 1331 Child-Related AI Incidents. https://arxiv.org/abs/2610.34705. Type: Incident analysis (arXiv; abstract only; 1,331 publicly reported incidents coded). E2

11. Uncertainty log

Overall uncertainty of this entry, bound to the verification log of 2026-08-28 plus TruthfulQA 2026-08-29, GPQA 2026-09-05, Wei 2026-09-06, GSM8K/MATH 2026-09-08, BBH 2026-09-09, HumanEval 2026-09-12 (source 21) EvalPlus/HumanEval+ 2026-09-13 (source 22), TruthInsightBench 2026-09-16 (source 23), SWE-Gate 2026-09-19 (source 24), AutoTuneBench 2026-09-22 (source 25) Physics Audit 2026-09-25 (source 26) PartHackBench 2026-09-27 (source 27) and MechBench 2026-09-29 (source 28) plus VLA Accel Audit 2026-09-30 (source 29), OSWorld-Science 2026-10-01 (source 30) VISTA 2026-10-02 (source 31), Agents-as-Systems 2026-10-04 (source 32) and MemoryAgentBench audit 2026-10-08 (source 33, E2) and Hidden Capabilities 2026-10-13 (source 34, E2) and Feng clinical claims 2026-10-16 (source 35, E2) and Uuk et al. loss-of-control evidence 2026-10-20 (source 36, E2) and Yorke LLM-judge stability 2026-10-24 (source 37, E2) and Jiao et al. child-related AI incidents 2026-10-28 (source 38, E2, abstract). 38 openings (37 full text, 1 abstract only). Not used as warrant: aggregators. OpenAI SWE-Verified page: fetch timeout, no warrant.

  • Established (layer 1): 1,331 child-related AI incidents: harm mainly representational/dignity/psychological, co-occurring with deployment, governance, misuse failures rather than model defects (38, E2, abstract); LLM judge aggregate stable, individual verdicts not — rerun flips 21.2%, label-only 41.6% (37, E2); loss-of-control evidence 19 studies weakly plausible, no high external validity, κ 0.24 (36, E2); HELM/Arena/SWE-bench/LCB constructs; TruthfulQA; GPQA (E1); Wei emergence definition (E1) beside Schaeffer; GSM8K/MATH paper measurements (E1); BBH/Suzgun CoT vs answer-only (E1); HumanEval/Chen instrument + Codex pass@k 2021 (E1); EvalPlus/HumanEval+ test inflation + pass@k drops (E1); SWE-Gate 221/644 HFR (E1); AutoTuneBench strawman/KernelBench/SGLang (E1); Physics Audit HLE/CMT/CritPt corrections (26, E1); PartHackBench historical Δhack .252 / current-state 0 (27, E1); VLA Accel Audit 22 bugs/4 design limits (29, E1); MechBench Core SA(P)/SA(M) 35.00%/13.75% (28, E1); OSWorld-Science (30, E1); VISTA RHAE 40.68→100 (31, E1); Agents-as-Systems ~54% run noise / Information rank 1 (32, E1); MMLU-Pro; contamination; Alzahrani; Arena style control.
  • Claimed (layer 2): model-level evaluations alone insufficient for child safety; age assurance, disclosure, audits (38, E3); prespecify judge setups, repeat, report both levels (37, E3); Uuk et al. evaluation capacity rather than inaction (36, E3); MMLU-Pro 72.6% GPT-4o; Arena 240k votes Jan 2024; style-control shifts Aug 2024; LCB contamination drops.
  • Constrained (layer 3): child-related AI incidents publicly reported ≠ incidence, co-occurrence ≠ causation, abstract only (38); LLM-judge study one benchmark/pair near 50%, measures change ≠ correctness (37); loss-of-control audit subjective rubric/corpus to mid-2025 ≠ risk probability (36); live leaderboards 2026 not opened (incl. GSM8K/MATH/BBH/HumanEval); SWE-Verified timeout; HumanEval/HumanEval+ bodies E1, live scores/contamination 2026 E0; HLE live board E0 (audit body E1); GPQA/GSM8K/MATH bodies E1, live scores E0; AutoTuneBench GB10 single-box ≠ fleet; Physics Audit subsets ≠ open research / AGI; PartHackBench PB-CSTE ≠ production eval; VLA Accel Audit subset/sim≠real/pending PRs; MechBench symbolic ≠ lab discovery; VISTA public set/system score ≠ private AGI; Agents-as-Systems AI4S grid ≠ universal agent eval.

Not opened (not a warrant)

  • OpenAI Why we no longer evaluate SWE-bench Verified — timeout; 59.4% E0.
  • OpenAI Introducing SWE-bench Verified.
  • Aleithan SWE-bench+; Deng SWE-Bench Pro (sister log, E0 here).
  • Chen HumanEval opened 2026-09-12 (source 21); Liu EvalPlus/HumanEval+ opened 2026-09-13 (source 22); MBPP+ figures not in opened body; Wei Emergent 2022 opened (source 17); GSM8K/MATH opened 2026-09-08 (sources 18–19); BBH opened 2026-09-09 (source 20); BIG-bench full suite still broader than BBH; HLE (GPQA body opened, source 16).
  • LMSYS / LMArena live leaderboard 2026; HELM live result tables.
  • LiveCodeBench live leaderboard / ICLR 2025.
  • GPT-4 Technical Report; Dubois AlpacaEval-LC; Dynabench.