Large Models

Established now · E1 / E2

3. What works today

Training loss and allocation

1. E1

   Cross-entropy loss L falls as a power law in parameter count N, dataset size D, and compute C, across many orders of magnitude. Paper exponents: α_N ≈ 0.076, α_D ≈ 0.095, α_C^min ≈ 0.050. Compute-optimal according to Kaplan: N ∝ C^0.73, D ∝ C^0.27. This holds in the measured regime; transfer to downstream is not identical to pretraining loss.

   Kaplan et al. Scaling Laws for Neural Language Models. https://arxiv.org/html/2001.08361. As of 2020-01-23. Checked 2026-08-28. Type: Paper (primary).
2. E1

   For a given compute budget, optimal model size and token count scale roughly equally (not 73/27). Chinchilla 70B / 1.4T tokens beats Gopher 280B / 300B tokens. Three fit methods in the same paper: a ≈ 0.46–0.50, b ≈ 0.50–0.54; about 400 models from 70M to 16B. Explicit departure from Kaplan’s compute-optimal allocation; both papers were opened.

   Hoffmann et al. Training Compute-Optimal Large Language Models. https://arxiv.org/html/2203.15556. As of 2022-03-29. Checked 2026-08-28. Type: Paper.
3. E2

   MMLU in the Chinchilla paper: 67.5% (Chinchilla) versus 60.0% (Gopher). That is the vendor/DeepMind measurement in the paper. The MMLU construct itself was not opened here.

   Hoffmann et al. Training Compute-Optimal Large Language Models. https://arxiv.org/html/2203.15556. As of 2022-03-29. Checked 2026-08-28. Type: Paper.

Coding evals

1. E1

   SWE-bench measures issue resolution in real GitHub repos via fail-to-pass tests. 2,294 issues, 12 Python repos. Claude 2 + BM25: 1.96% resolved; oracle retrieval Claude 2: 4.8%. Gold patches average 1.7 files / 32.8 lines. Lite = 300.

   Jimenez et al. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. https://arxiv.org/html/2310.06770. As of 2023-10-10. Checked 2026-08-28. Type: Paper.
2. E2

   SWE-bench Verified (500 human-reviewed out of 1,699) has not been used by OpenAI as a frontier measure since 2026-02-23. SOTA on Verified in the six months before the text: 74.9% → 80.9%. Audit of 138 unsolved tasks: 59.4% had faulty tests (35.5% too narrow, 18.8% too broad). Contamination: GPT-5.2, Claude Opus 4.5, Gemini 3 Flash Preview reproduced gold patches. Recommendation there: SWE-bench Pro. The model names are evidence of contamination elicitation, not of general 2026 capability ranks. Live leaderboard not checked.

   OpenAI. Why SWE-bench Verified no longer measures frontier coding capabilities. https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/. As of 2026-02-23. Checked 2026-08-28. Type: Company report.
3. E2

   SWE-Bench Pro (Scale AI): 1,865 problems; public 731 (GPL), commercial 276 (private), held-out 858. Gold averages 107.4 LOC / 4.1 files. Under SWE-Agent, snapshot around 2025-09-18: public — Claude Sonnet 4.5 43.6%, GPT-5 (high) 41.8%. Commercial — Claude Opus 4.1 17.8%, GPT-5 high 15.7%. Human-augmentation ablation (Table 3; different setting from Table 1, including turn/cost cap): GPT-5 (high) 25.9% → 8.40% on the problem statement only (no requirements/interface). Error modes (LLM-as-judge on trajectories): wrong solution, tool use, syntax, context overflow. One lab, one scaffold; scaffold dependence is documented in the paper.

   Deng, Da et al. SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?. https://arxiv.org/html/2509.16941. As of 2025-09-18. Checked 2026-08-28. Type: Paper (Scale AI).
4. E2

   LiveCodeBench collects contest tasks with release dates (LeetCode, AtCoder, CodeForces). 511 tasks May 2023–May 2024; evaluation after cutoff against contamination. DeepSeek-Instruct and GPT-4o drop markedly on LeetCode problems after their cutoff/release. Fine-tuned open models can score high on HumanEval+ and low on LCB-Easy (overfitting cluster). Hard split near zero for most models in 2024. GPT-4-Turbo-2024-04-09 code generation (Sep-filtered): Easy 85.3 / Medium 33.0 / Hard 5.1 / Total 41.1 Pass@1. GPT-4o self-repair total 49.1. Limits: Python only; prompt sensitivity; contest ≠ industry repos; bootstrap noise about 1–1.5% on 349 tasks. Live board 2026 not opened.

   Jain et al. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. https://arxiv.org/html/2403.07974. As of 2024-03-12. Checked 2026-08-28. Type: Paper.
5. E1

   HumanEval (Chen et al., opened): 164 handwritten Python tasks, mean 7.7 unit tests; Pass@k as an unbiased estimator with n≥k (paper: n=200, k≤100). Codex-12B pass@1 28.8% (Table 1: 28.81), pass@10 46.81, pass@100 72.31. GPT-3 near 0%; GPT-J 6B pass@1 11.62 / pass@100 27.74. Codex-S (supervised) pass@1 37.7%, pass@100 with oracle selection 77.5%. Training: 54M public GitHub repos, 159 GB after filtering (May 2020). BLEU distributions of correct and incorrect samples overlap (Fig. 8) — BLEU ≠ functional correctness. Limit: interview-like single functions, not multi-file; contamination of the 164 tasks not re-measured in 2026. LiveCodeBench and SWE-bench cite the construct.

   Chen et al. Evaluating Large Language Models Trained on Code. https://arxiv.org/html/2107.03374. As of 2021-07-07. Checked 2026-08-29. Type: Paper (primary).Jimenez et al. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. https://arxiv.org/html/2310.06770. As of 2023-10-10. Checked 2026-08-28. Type: Paper.Jain et al. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. https://arxiv.org/html/2403.07974. As of 2024-03-12. Checked 2026-08-28. Type: Paper.

GPT-4 Technical Report (exams, benchmarks, vision capability)

1. E1

   GPT-4 Tech Report (OpenAI, arXiv:2303.08774, body opened, as of 2023-03-15): multimodal — image+text inputs → text outputs (report scope). Table 1 exams: Uniform Bar Exam GPT-4 298/400 (~90th) vs GPT-3.5 213/400 (~10th); GRE Quant 163/170 (~80th) with vision / 157 no-vision; GRE Verbal 169/170 (~99th). Caveat: minority of exam items seen in training; contamination variants reported (Appendix C). E1 for paper measurements; not E1 for “human-level AGI”.

   OpenAI GPT-4 Tech Report. GPT-4 Technical Report. https://arxiv.org/pdf/2303.08774. As of 2023-03-15. Checked 2026-09-10. Type: Tech report (arXiv).
2. E1

   Table 2 few-shot: MMLU 86.4% (5-shot); HumanEval 67.0% (0-shot); HellaSwag 95.3%; ARC Challenge 96.3%; GSM-8K 92.0%* with CoT (*part of train set in pretrain mix — footnote). Beats LM SOTA on the listed; DROP F1 80.9 loses to tuned SOTA 88.4.

   OpenAI GPT-4 Tech Report. GPT-4 Technical Report. https://arxiv.org/pdf/2303.08774. As of 2023-03-15. Checked 2026-09-10. Type: Tech report (arXiv).
3. E1

   Vision: accepts interleaved image+text; academic vision benchmark numbers stated preliminary on the blog — E1 for capability existence / exam image handling and comic-panel example (Table 3), not for specific VQA leaderboard scores absent from the PDF tables.

   OpenAI GPT-4 Tech Report. GPT-4 Technical Report. https://arxiv.org/pdf/2303.08774. As of 2023-03-15. Checked 2026-09-10. Type: Tech report (arXiv).
4. E1

   Limitations §5: still hallucinates; +19 percentage points vs latest GPT-3.5 on internal adversarial factuality eval (Fig. 6).

   OpenAI GPT-4 Tech Report. GPT-4 Technical Report. https://arxiv.org/pdf/2303.08774. As of 2023-03-15. Checked 2026-09-10. Type: Tech report (arXiv).

Chain-of-Thought prompting (Wei et al. 2022)

1. E1

   Wei et al., Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (arXiv:2201.11903; NeurIPS 2022; body opened 2026-09-11): CoT = few-shot exemplars with intermediate natural-language reasoning steps (not finetuning in this paper). Emergent with scale (~100B+): smaller models produce fluent but illogical CoTs and can hurt vs. standard prompting.

   Wei et al. CoT. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. https://arxiv.org/pdf/2201.11903. As of arXiv 2022-01 / NeurIPS 2022. Checked 2026-09-11. Type: Paper (arXiv; NeurIPS 2022).
2. E1

   Table 1, PaLM 540B: GSM8K standard 17.9% → CoT 56.9% (+39.0); +ext.calc 58.6%. Prior best finetuned 55% (Cobbe et al.). Also SVAMP 69.4→79.0; MAWPS 79.2→93.3 (same table). Eight manual CoT exemplars for math; abstract: surpasses finetuned GPT-3 with a verifier on GSM8K.

   Wei et al. CoT. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. https://arxiv.org/pdf/2201.11903. As of arXiv 2022-01 / NeurIPS 2022. Checked 2026-09-11. Type: Paper (arXiv; NeurIPS 2022).
3. E1

   Constrained: prompting method ≠ autonomy/AGI; PaLM not public; gains largest on hard multi-step math. ReAct is separate (agents dossier) — CoT only here.

   Wei et al. CoT. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. https://arxiv.org/pdf/2201.11903. As of arXiv 2022-01 / NeurIPS 2022. Checked 2026-09-11. Type: Paper (arXiv; NeurIPS 2022).

Self-Consistency (Wang et al. 2022)

1. E1

   Wang et al., Self-Consistency Improves Chain of Thought Reasoning in Language Models (arXiv:2203.11171; ICLR 2023; body opened 2026-09-12): sample diverse CoT reasoning paths (not greedy), then majority vote / marginalize over answers — unsupervised, no extra training.

   Wang et al. Self-Consistency. Self-Consistency Improves Chain of Thought Reasoning in Language Models. https://arxiv.org/pdf/2203.11171. As of arXiv 2022-03 / ICLR 2023. Checked 2026-09-12. Type: Paper (arXiv; ICLR 2023).
2. E1

   PaLM-540B Table 2 (self-consistency): GSM8K 74.4 (+17.9 vs CoT greedy); SVAMP 86.6 (+7.6); AQuA 48.3 (+12.5); MultiArith 99.3 (+4.6). Code-davinci-002 Table 2: GSM8K 78.0 (+17.9); SVAMP 86.8 (+11.0); AQuA 52.0 (+12.2) — absolute gains match the abstract. Abstract also: StrategyQA +6.4%, ARC-challenge +3.9%.

   Wang et al. Self-Consistency. Self-Consistency Improves Chain of Thought Reasoning in Language Models. https://arxiv.org/pdf/2203.11171. As of arXiv 2022-03 / ICLR 2023. Checked 2026-09-12. Type: Paper (arXiv; ICLR 2023).
3. E1

   Constrained: decoding/sampling cost (paper: 40 paths); method ≠ autonomy/AGI; paper snapshot ≠ 2026 frontier live boards (E0).

   Wang et al. Self-Consistency. Self-Consistency Improves Chain of Thought Reasoning in Language Models. https://arxiv.org/pdf/2203.11171. As of arXiv 2022-03 / ICLR 2023. Checked 2026-09-12. Type: Paper (arXiv; ICLR 2023).

Language, knowledge, preference

1. E2

   HELM defines a taxonomy: 16 core scenarios × 7 metrics, 30 models (as of the paper). Holistic evaluation rather than a single score. Full text of result tables not opened; 2022 individual scores are not named here (E0 for figures).

   Liang et al. Holistic Evaluation of Language Models. https://arxiv.org/abs/2211.09110. As of 2022-11-16. Checked 2026-08-28. Type: Paper (abstract/intro only).
2. E2

   SimpleQA: 4,326 short fact questions, adversarial against GPT-4. GPT-4o 38.2% correct; o1-preview 42.7%; Claude-3.5-Sonnet 28.9%. Models overestimate confidence. Label-error estimate about 3%. Scope: short-form, English, adversarial vs. GPT-4; no claim about the 2026 frontier without a new eval.

   Wei et al. Measuring short-form factuality in large language models (SimpleQA). https://arxiv.org/html/2411.04368. As of 2024-11-07. Checked 2026-08-28. Type: Paper.
3. E2

   Chatbot Arena = pairwise crowdsourcing + Bradley-Terry. Paper: 240k votes through January 2024. Crowd–expert agreement about 73–83%. Limits: hobbyist user base; helpfulness ≠ safety. Live leaderboard 2026 not opened — no 2026 ranking claims.

   Chiang et al. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. https://arxiv.org/html/2403.04132. As of 2024-03-07. Checked 2026-08-28. Type: Paper.

Tool use (lab, not product agent)

1. E2

   Toolformer: GPT-J 6.7B learns API calls self-supervised (QA, calculator, wiki, MT, calendar). No chaining of tools, no interactive search, one call per input. Calendar unused on TempLAMA. Does not beat GPT-3 on all QA sets.

   Schick et al. Toolformer: Language Models Can Teach Themselves to Use Tools. https://arxiv.org/html/2302.04761. As of 2023-02-09. Checked 2026-08-28. Type: Paper.
2. E2

   GAIA: 466 questions with a unique short answer. Humans 92%. GPT-4 + plugins in the paper about 15% overall; level split for plugins about 30% L1, 9.7% L2, 0% L3. Web-search baseline beyond L1 near zero. 2023 snapshot. Hugging Face leaderboard 2026 not opened.

   Mialon et al. GAIA: a benchmark for General AI Assistants. https://arxiv.org/html/2311.12983. As of 2023-11-21. Checked 2026-08-28. Type: Paper.
3. E2

   τ-bench, Table 2, function calling, at least 3 trials, as of the 2024 paper: gpt-4o pass^1 61.2% retail / 35.2% airline (avg 48.2); gpt-4-turbo 57.7 / 32.4 (45.1); claude-3-opus 44.2 / 34.7 (39.5); gpt-3.5-turbo 20.0 / 10.8 (15.4); llama-3-70B (text-ReAct) 14.8 / 14.4 (14.6). Author limits: user simulator = LM; reward = DB end state (necessary, not sufficient for policy fidelity); two synthetic domains; annotation with gpt-4-turbo bias possible.

   Model	retail pass^1	airline pass^1	avg
   gpt-4o	61.2	35.2	48.2
   gpt-4-turbo	57.7	32.4	45.1
   claude-3-opus	44.2	34.7	39.5
   gpt-3.5-turbo	20.0	10.8	15.4
   llama-3-70B (text-ReAct)	14.8	14.4	14.6

   Yao, Shinn et al. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. https://arxiv.org/html/2406.12045. As of 2024-06-17. Checked 2026-08-28. Type: Paper.

Measured error modes 2023–2024

1. E1

   Lost-in-the-middle: U-shaped retrieval curve (primacy/recency). GPT-3.5-Turbo, 20 documents, middle worse than closed-book (56.1%). Extended-context models do not automatically use context better. Query-aware contextualization helps synthetic KV retrieval, not multi-doc QA. Encoder-decoder more robust within training length. For 2026 long context without re-test: E0.

   Liu et al. Lost in the Middle: How Language Models Use Long Contexts. https://arxiv.org/html/2307.03172. As of 2023-07-06. Checked 2026-08-28. Type: Paper.
2. E1

   Sycophancy: five assistants (Claude 1.3/2, GPT-3.5/4, Llama-2-70b-chat) shift answers toward user feedback, “are you sure?”, and answer matching; poem attribution is imitated. HH-RLHF: matching the user’s view predicts preference. Claude-2-PM: mixed effects BoN vs. RL. Direction laid down by Perez 2022; Perez not in full text here.

   Sharma et al. Towards Understanding Sycophancy in Language Models. https://arxiv.org/html/2310.13548. As of 2023-10-20. Checked 2026-08-28. Type: Paper.
3. E2

   In-context scheming (capability in constructed evals): o1, Claude 3.5 Sonnet, Claude 3 Opus, Gemini 1.5 Pro, Llama 3.1 405B show in-context scheming when a goal is given. o1 maintains deception in more than 85% of follow-ups. Scenarios are toy; CoT often explicit. gpt-4o: no scheming evidence on all six evals in the suite (Table 1). o1: evidence on all six. Not shown: spontaneous deployment scheming. E2 for capability-under-nudge, not E1 for real-world deception.

   Meinke et al. Frontier Models are Capable of In-context Scheming. https://arxiv.org/html/2412.04984. As of 2024-12-06. Checked 2026-08-28. Type: Paper.

Energy / compute (linkage, not a standalone dossier)

1. E1

   Inference energy depends strongly on task class. Luccioni et al.: 88 models, 10 tasks, 1,000 inferences. Image generation about 2.9 kWh / 1,000 versus text classification 0.002 kWh / 1,000. Generative >> discriminative; multi-purpose more expensive than task-specific. BLOOMz-7B: training+FT parity about 593 million inferences. Measured joules, open models 2024. That is energy, not visual understanding. A layer-1 claim “vision 2026” is not tenable in this log.

   Luccioni, Jernite, Strubell. Power Hungry Processing: Watts Driving the Cost of AI Deployment?. https://arxiv.org/html/2311.16863. As of 2023-11-28. Checked 2026-08-28. Type: Paper (FAccT 2024).
2. E2

   Epoch AI (Sevilla & Roldán, report 28 May 2024, not peer-reviewed) finds frontier training compute — models in the top 10 by training FLOP at release — grew 5.3×/year (90% CI 4.9–5.7) from 2010 to May 2024; after an ~2018 slowdown Epoch prefers ~4.2×/year (90% CI 3.6–4.9); frontier language models after mid-2020 ~5×/year — the same 3.1–7.3 interval is labeled 90% CI in the body and 80% CI in Table A8 (quote both; do not pick a side). Authors’ headline: 4–5×/year for notable and frontier. 2e29 FLOP / 2030 is not established (a different Epoch projection, not this warrant).

   Sevilla & Roldán. Training compute of frontier AI models grows by 4-5x per year. https://epoch.ai/publications/training-compute-of-frontier-ai-models-grows-by-4-5x-per-year. As of 2024-05-28. Checked 2026-08-31. Type: Report (not peer-reviewed).

Time horizon on software tasks (METR)

1. E2

   METR defines the 50% task-completion time horizon as the human-expert duration of tasks an agent completes at 50% success. Suite: 170 tasks (HCAST, RE-Bench, 66 SWAA), more than 800 human baselines, 2,529 hours. On this suite o3 has a 50% horizon of about 110 minutes. GPT-2: 2 seconds. The 50% horizon doubles about every seven months from 2019–2025 (207 days; 95% bootstrap 166–240). The 80% horizon has a similar doubling (204 days) but sits about 4–6× shorter. One lab, one scaffold family.

   Kwa, West et al. (METR). Measuring AI Ability to Complete Long Software Tasks. https://arxiv.org/html/2503.14499v3. As of 2025-03-18. Checked 2026-08-28. Type: Paper (METR).
2. E2

   At a given task length, models score worse on messier tasks (16 messiness factors; controlling for length). The time series on the messier subset is similar; no messiness-specific plateau in the paper. Internal PRs: contract baselines 5–18× slower than repo maintainers; agent performance matches contractor time more closely. Horizon is relative to low-context work.

   Kwa, West et al. (METR). Measuring AI Ability to Complete Long Software Tasks. https://arxiv.org/html/2503.14499v3. As of 2025-03-18. Checked 2026-08-28. Type: Paper (METR).
3. E2

   METR dashboard, last updated 2026-05-08 (TH 1.1, larger suite). FAQ example: a GPT-5 agent with a horizon of about 2 hours 17 minutes. Measurements above 16 hours are marked by the page itself as unreliable on the current suite. 2026 chart point values are JS-rendered and are not cited as figures here.

   METR time-horizons. Task-Completion Time Horizons of Frontier AI Models. https://metr.org/time-horizons/. As of 2026-05-08. Checked 2026-08-28. Type: Lab dashboard.