Zero to One: A Roadmap

Collect Tasks: Steps 0–3

The first hard problem is filling the suite with the right tasks. Anthropic's playbook compresses to four steps before you even think about graders.

Step 0 — Start early. Teams delay evals waiting for hundreds of tasks. The reality: 20–50 simple tasks from real failures work well. Early agent development shows clear impact from small samples; mature agents need larger suites. Take the 80/20 approach.

Step 1 — Start with manual checks. Begin with behaviors you already verify by hand before each release. Already in production? Mine the bug tracker and support queue. Converting real failures into test cases ensures the suite reflects actual usage.

Step 2 — Write unambiguous tasks with reference solutions. A good task is one two domain experts can independently grade pass/fail and agree. Could you pass the task yourself? If not, the spec needs work. Build a reference solution — a known working output that passes every grader. This proves the task is solvable and that your graders are wired correctly.

Step 3 — Build balanced problem sets. Test cases where behavior should happen and cases where it shouldn't. Only testing the positive case creates overtriggering. Claude.ai web search took many rounds to balance: queries that need search (weather) plus queries that don't (Apple founder).

Design Harness and Graders: Steps 4–5

Step 4 — Build a robust harness with a stable environment. The agent should run roughly like production. The environment shouldn't introduce noise. Each trial starts clean — no shared state, no leftover files, no cached data. Shared state can artificially inflate performance: in internal evals, Claude once gained an unfair advantage by reading git history from previous trials.

Step 5 — Design graders thoughtfully. Choose deterministic graders where possible, model-based where necessary for flexibility, human graders judiciously for validation.

• Don't rigidly check tool-call order. Agents find valid unanticipated paths. Grade what they produced, not the trail they took.
• Build in partial credit. A support agent that identified the problem and verified the customer but failed the refund is meaningfully better than one that failed immediately. Score the success continuum.
• Calibrate model judges to humans. Compare LLM judge scores to a small expert-labeled set. Give the judge an out — "return Unknown if insufficient information." Score rubric dimensions separately rather than all at once.
• Resist bypasses. Tasks should require actually solving the problem, not exploiting loopholes.

Transcripts and Saturation: Steps 6–7

Step 6 — Read the transcripts. You can't know whether graders work without reading actual run records. When tasks fail, transcripts reveal whether the agent made a real mistake or the grader rejected a valid solution. Failures should feel fair: clear what the agent got wrong and why. When scores stop climbing, confirm it's the agent — not the eval — that's stuck.

Step 7 — Watch for eval saturation. Eval saturation happens when agents pass every solvable task. Evals at 100% protect against regressions but provide no improvement signal. SWE-Bench Verified started this year around 30%; frontier models now sit above 80% saturation. As evals saturate, big capability jumps look like small score gains.

Qodo was initially unimpressed by Opus 4.5 because their one-shot coding evals couldn't capture the longer, more complex task gains. They built new agentic eval frameworks and the real picture came back into focus. When your suite saturates, build a harder one — don't accept the score at face value.

Eval-Driven Development: Step 8

Step 8 — Keep suites healthy long-term. Eval suites are living artifacts that need ongoing attention and clear ownership. Anthropic's most effective pattern: a dedicated evals team owning core infrastructure while domain experts and product teams contribute tasks and run evaluations.

For AI product teams, owning evals should be as routine as maintaining unit tests. Teams waste weeks on features that "work" in early testing but fail unstated expectations. Well-designed evals surface those expectations before launch. Defining tasks stress-tests whether requirements are concrete enough to build.

The strongest practice is eval-driven development: write the eval defining a planned capability before the agent can fulfill it. Iterate until pass rates climb. Build features that are bets on future model capabilities — low-pass capability evals make those bets visible. When new models drop, your suite reveals which bets paid off.

Swiss Cheese Coverage

Automated evaluations run thousands of tasks before any user sees a deployment. But automated evals are only one slice of the picture. Complete coverage stacks several methods, each catching what the others miss.

• Automated evals — fast iteration, fully reproducible, no user impact, run on every commit. Risk: false confidence if they don't match real usage.
• Production monitoring — reveals real user behavior at scale and catches issues synthetic evals miss. Risk: reactive; problems reach users first.
• A/B testing — measures actual user outcomes like retention or task completion. Risk: slow to reach significance; needs traffic.
• User feedback — surfaces unanticipated problems with real examples. Risk: sparse, self-selected, severity-skewed.
• Manual transcript review — builds intuition for failure modes, catches subtle quality issues. Risk: doesn't scale.
• Systematic human studies — gold-standard grading by trained raters. Risk: expensive and slow; reserve for calibration.

The Swiss-cheese analogy from safety engineering: any single layer has holes, but stacked layers stop most failures. Most effective teams combine automated evals for fast iteration, production monitoring for ground truth, and periodic human review for calibration.

Several frameworks help teams skip the build-from-scratch work: Harbor for containerized agent environments and trial scaling (Terminal-Bench 2.0 ships through Harbor); Braintrust for offline eval plus observability plus experiment tracking; LangSmith for tracing and dataset management in the LangChain ecosystem; Langfuse as the open-source self-hosted alternative; Arize Phoenix (open-source) and AX (SaaS) for LLM tracing and online/offline evals. Frameworks accelerate the plumbing — they're only as good as the tasks and graders you run through them.

Tap any card to reveal the definition. Or open the full deck to filter and shuffle across all guides.