The Anatomy of an Eval

What an Eval Is

An evaluation — an "eval" — is a test that applies grading logic to an AI system's output to measure success. This guide focuses on automated evals: tests that run during development without real users.

The simplest eval has three parts: a prompt, a response, and a grader. That's a single-turn eval. The moment your system uses tools, modifies state, or takes multiple turns to reach a goal, you've moved into multi-turn or agent evals — and the surface area for mistakes explodes.

Frontier models also do unexpectedly creative things. Opus 4.5 once solved a τ2-Bench flight-booking problem by discovering a loophole in the policy — it failed the eval as written but delivered a better outcome for the user. Good eval design has to anticipate this.

Key Definitions

Every conversation about evals uses this vocabulary. Learn it once.

• Task — a single test case with defined inputs and success criteria. Sometimes called a "problem."
• Trial — one attempt at a task. Multiple trials of the same task give you a success-rate distribution.
• Grader — the logic that scores agent performance. A task can have several graders combined.
• Transcript (also trace or trajectory) — the complete record of a trial: outputs, tool calls, reasoning, and interactions.
• Outcome — the final state of the environment at the end of a trial. For a support agent, "was the refund actually processed?"
• Evaluation harness — the infrastructure that runs the evals end-to-end.
• Agent harness (or scaffold) — the system that turns a model into an agent: tool use, memory, planning loops. Evaluation tests the harness and the model together.
• Evaluation suite — a collection of tasks measuring a specific capability or behavior.

Three Grader Types

Graders fall into three families. Most production suites use a mix.

Code-based graders are deterministic checks: string match, regex, fuzzy match, binary tests (fail-to-pass, pass-to-pass), static analysis (lint, type, security), outcome verification, tool-call verification, transcript analysis (turns taken, tokens used).

• Strengths: fast, cheap, objective, reproducible, easy to debug.
• Weaknesses: brittle to valid variations, weak on nuance, limited for subjective tasks.

Model-based graders (LLM-as-judge) score with rubrics, natural-language assertions, pairwise comparison, reference-based evaluation, or multi-judge consensus.

• Strengths: flexible, scales, captures nuance, handles open-ended output.
• Weaknesses: non-deterministic, more expensive, requires human calibration to trust.

Human graders — SME review, crowdsourced judgment, spot-check sampling, A/B testing, inter-annotator agreement — are the gold standard. Reserve them for calibration and ambiguous domains.

Capability vs Regression

Evals serve two different roles in the development loop. Both are needed; confusing them will mislead you.

Capability evals ask "what can this agent do well?" Their job is to surface what doesn't work yet. They should start at low pass rates — a hill to climb. A capability eval at 100% has nothing more to teach you.

Regression evals ask "does this agent still handle the things it used to?" They should sit near 100% pass and protect you from backsliding when you change the prompt, swap the model, or tweak the harness.

This is why you want both in CI. Capability evals tell you which optimization moves the frontier forward. Regression evals catch the side effects of those moves on everything else.

Why Build Them

Early teams iterate by hand. You dogfood the agent, you fix bugs as users report them, and life is fine. The wheels come off at scale: users report regressions and you can't verify improvements except by trial and error. That's the inflection point.

Without evals, debugging is purely reactive: wait for complaints, reproduce manually, fix, hope nothing else regresses. With evals, the payoff compounds:

• You can specify what success means — two engineers reading the same spec resolve edge-case ambiguity through the eval suite.
• You can adopt new models in days, not weeks. Teams without evals face weeks of vibes-testing.
• Baselines and regression tests come free — latency, token usage, cost per task, error rates, all tracked on static tasks.
• Evals become the highest-bandwidth channel between product and research — they define the metric researchers can optimize against.

Tap any card to reveal the definition. Or open the full deck to filter and shuffle across all guides.