A field guide to the science of evaluating AI systems for safety — how the discipline developed, what it has found, and what remains unresolved. The handbook spans the full arc: from MMLU's attempt to measure general knowledge, through DeepMind's framework for dangerous capabilities, to Apollo Research's discovery that frontier models will deceive evaluators to avoid shutdown. Each chapter covers one paper or framework, but the emphasis is on the chain of problems and responses that connects them. Classical benchmarks revealed capability gaps; dangerous-capability evals revealed what capable models could do to the world; behavioral evals revealed that models may strategically hide those capabilities. Governance frameworks are now trying to catch up.
- Students: Read sequentially from Part I. Each chapter is self-contained but builds on the ones before it.
- Researchers: Jump to the Synthesis sections for open problems and cross-cutting themes. The governance chapters (16–18) map the most tractable near-term work.
- Project ideas: Every chapter ends with concrete reproduction and extension projects, rated by difficulty.
- Quick reference: See Comparison Tables for a cross-paper summary by org, technique, and threat model.
| Chapter | Primary Source | Year |
|---|---|---|
| Ch. 1 — Why Evals Matter: The Epistemic Challenge | Synthesis chapter | — |
| Ch. 2 — What Is an Eval? Taxonomy and Typology | Liang et al. (HELM), Stanford CRFM | 2022 |
| Ch. 3 — A Science of Evals | Raji et al.; Burnell et al. | 2021, 2023 |
| Ch. 4 — Responsible Scaling Policies: Linking Evals to Deployment | Anthropic RSP | 2023 |
| Chapter | Primary Source | Year |
|---|---|---|
| Ch. 5 — MMLU and HELM: Knowledge and Holistic Evaluation | Hendrycks et al.; Liang et al. | 2020, 2022 |
| Ch. 6 — BIG-bench and Frontier Probing | Srivastava et al. (BIG-bench) | 2022 |
| Ch. 7 — Agent Benchmarks: From Tool Use to Autonomous Tasks | Yang et al. (SWE-bench); Zhou et al. (WebArena) | 2023, 2024 |
| Chapter | Primary Source | Year |
|---|---|---|
| Ch. 8 — The DeepMind Framework: Dangerous Capability Evaluation | Shevlane et al., DeepMind | 2023 |
| Ch. 9 — Biosecurity and CBRN Uplift | Anthropic; RAND biosecurity studies | 2023–2024 |
| Ch. 10 — Cybersecurity Capability Evaluation | Phuong et al., DeepMind | 2024 |
| Ch. 11 — Autonomous Replication and Adaptation | METR (formerly ARC Evals) | 2024 |
| Chapter | Primary Source | Year |
|---|---|---|
| Ch. 12 — In-Context Scheming: Frontier Models and Strategic Deception | Apollo Research | 2024 |
| Ch. 13 — Evaluation Awareness: When Models Know They're Being Tested | Greenblatt et al. (Anthropic); Apollo Research | 2024 |
| Ch. 14 — Deliberative Alignment and Anti-Scheming Training | Anthropic | 2024 |
| Ch. 15 — Loss of Control: From Threat Model to Evaluation | Ngo et al.; Cotra | 2022, 2022 |
| Chapter | Primary Source | Year |
|---|---|---|
| Ch. 16 — Red Teaming: Structured Adversarial Testing | Ganguli et al., Anthropic | 2022 |
| Ch. 17 — White-Box Access: Mechanistic Evaluation | Synthesis chapter | — |
| Ch. 18 — International Governance and Evaluation Ecosystems | UK AISI; Seoul AI Safety Summit | 2023–2024 |
| Section | Contents |
|---|---|
| Timeline | How the field evolved from capability measurement to behavioral safety |
| Themes | Five cross-cutting tensions across all papers |
| Open Problems | Collected research gaps, grouped by cluster |
| Research Directions | Concrete project ideas with difficulty ratings |
2016–2019 "Can we measure capabilities?"
MMLU, static knowledge benchmarks, scaling-law era
↓
2020–2022 "What can't they do?"
BIG-bench, failure-mode hunting, behavioral diversity
↓
2022–2023 "What could they do if misused?"
DeepMind dangerous-capability framework, CBRN uplift studies
↓
2023–2024 "Are they deceptive?"
Apollo Research scheming, evaluation awareness, alignment faking
↓
2024–2025 "Can governance catch up?"
RSPs, UK AISI, Seoul Summit, international evaluation ecosystems
This handbook covers eighteen papers and frameworks selected for field-changing impact. The evaluation literature now spans thousands of papers across dozens of labs. The selection criteria: works that introduced a new class of question, changed how evaluators think about what models can do, or established institutional infrastructure that others now depend on. Coverage of adversarial robustness, calibration, and general NLP benchmarking is deliberately light — those areas have their own extensive literature. The handbook focuses on the thread that leads from capability measurement to safety-relevant behavioral evaluation to governance.