Code first, model second, human last — and never trust one tier alone. The atoms in Guide 01 need stratified graders, evidence rules, and an "unknown" exit before they're production-ready.
Stratify graders
Code-based blocks hard errors. Model-based handles semantics. Human calibrates the model. Anthropic's three-tier system is the cost/reliability backbone.
Three rubric kinds
Outcome (did the world change?), Answer (was the response right?), Trajectory (were tools chosen and ordered right?). Outcome judges; Trace diagnoses.
Allow "unknown"
If the trace is missing evidence, force the Judge to return unknown. Otherwise it hallucinates a "complete" verdict and writes a wrong fix into your assets.
Grader stratification — code first, model second, human last
Anthropic's eval framework names three grader types, in this exact cost-and-reliability order:
Grader type
What it's good for
Cost / reliability
Code-based
JSON validity, schema check, state assertions, unit tests, sandbox runs, exact string match.
Cheap, fast, reproducible. Brittle to valid variation.
Expensive. Slow. The gold standard for everything else to align to.
The point of stratification is not "use all three." It's that each layer catches what the cheaper layer below it cannot, so you don't pay LLM costs to check what a regex could and you don't pay human-review costs on cases an LLM Judge already gets right.
Stratification is a cost-and-reliability decision, not a sophistication contest. The cheapest grader that gives a trustworthy signal wins the slot. LLM Judge is reserved for what code can't enumerate.
Deterministic checks come before semantic judgment
If a question can be answered by a rule, an enum, a regex, a sandbox, or a database query, never let an LLM judge it. Turning a deterministic check into a probabilistic one is one of the most common rubric design mistakes — and the most expensive one.
Examples that should always be code-based, not model-based:
Question
Should be answered by
Why not LLM Judge
Is the JSON valid?
A JSON parser.
It's a binary check with a free, perfect oracle.
Did the order status actually update?
A database query.
The truth lives in the DB, not the trace.
Did the code pass tests?
A test runner.
"Looks like it should pass" is not passing.
Is the amount under the spend limit?
A threshold comparison.
Probabilistic check on a deterministic constraint is risky.
Did the tool return an error?
The tool result schema.
The error field is right there.
LLM Judge's value is in places where rules can't enumerate — semantic quality, claim support, user-goal completion. Spend its budget there, not on checks a regex already handles.
"Let the LLM Judge do everything — it's smart enough." A smart Judge handling JSON validity is still a slow, expensive, non-deterministic judge handling JSON validity. You've replaced a parser with a coin flip.
Three rubric kinds — Outcome, Answer, Trajectory
An Agent product is harder to evaluate than a chatbot because the user only sees one final message, but the business risk lives in the middle steps. Three rubric kinds split this cleanly:
Rubric kind
Asks the question
Catches
Outcome Rubric
Did the end-state in the world actually match the target?
"Agent said done but record not updated"; missing files; failed tests; missing required conclusions.
Answer Rubric
Was the final expression correct, complete, grounded, and constraint-compliant?
Hallucinations, missing citations, format violations, off-topic responses.
Trajectory Rubric
Were tool choice, parameters, dependency order, exception recovery, cost, and turn count reasonable?
Tool misuse, parameter blindness, redundant calls, runaway cost — risks invisible in the final answer.
You don't need all three for every product. A pure RAG-style QA bot rarely needs a Trajectory Rubric. A multi-step bookings Agent rarely gets away without one.
A user reporting "the answer was great" doesn't mean the trajectory was safe. The booking went through and the email looked nice — but the Agent also called the refund API twice while it was at it. Trajectory rubrics surface that.
Outcome is the main judge, Trace is the diagnostic
A common over-correction once teams add Trajectory Rubrics is to write the "correct path" as a must-pass assertion: "the Agent must call A, then B, then C in this exact order." This is almost always wrong.
Hard-coded paths punish valid new approaches the model might find. They make the eval brittle, and they conflate "the Agent took a path I didn't expect" with "the Agent failed."
The healthier relationship: Outcome is the main judge. Trace is the diagnostic evidence. Only paths that represent explicit safety or business-hard-constraints should be hard-coded as must-pass assertions — e.g., "must check inventory before charging the card."
Anthropic states this directly in their agent eval guidance: "There is a common instinct to check that agents followed very specific steps like a sequence of tool calls in the right order — it's often better to grade what the agent produced, not the path it took."
"If we don't lock the path, the agent will do something crazy." It can — but most "crazy" paths are actually better paths in disguise. The safety net is must-pass assertions on the few steps that actually matter (auth, payment, data deletion), plus Outcome verification — not a play-by-play script.
The trajectory-level failure modes you'll see in production
TRAJECT-Bench (He et al., 2025) decomposed tool-use evaluation into three sub-dimensions — tool selection, argument correctness, dependency order — and identified four named failure patterns. Each maps to a different fix.
Failure mode
What you see in the trace
Fix points at
Similar tool confusion
Two tools have overlapping capabilities; the Agent picks the worse one (e.g., "Spotify: Search" vs a sibling).
Tool called more times than needed — "related but unhelpful" calls, or unrelated hallucinated calls.
Stopping rule, cost guardrail, planner — also a training-data smell.
Intent inference failure
Indirect user goal misread; Agent solves a different problem than asked.
Prompt clarity, task router, intent classifier.
The paper also showed model performance dropping sharply at the transition from short to mid-length trajectories — across Claude-4, Gemini-2.5-pro, o4-mini, DeepSeek, Qwen3-235B-A22B, and Kimi-k2. Scaling tool use is a universal challenge, in the authors' words. If your Agent works on 3-step demos and breaks on 7-step real flows, you're looking at this transition.
Tool-chain failures look identical to the user — the answer is just wrong. But the fix differs by failure mode. Naming the mode is what turns a "tool bug" into a routable engineering task.
Every dimension needs an evidence prerequisite
The most dangerous failure mode in eval design isn't the Judge being wrong. It's the Judge being confidently wrong because evidence was missing and it filled the gap with a guess.
Every atomic dimension should have an entry condition — a check that the evidence needed to judge it actually exists in the trace. If the evidence is missing, the dimension returns "unknown," not "pass" and not "fail."
Dimension
Evidence prerequisite
If missing →
groundedness
retrieval_context field present and non-empty
return unknown
outcome_state
expected_state defined, or state-check tool result
return unknown
argument_correctness
Tool call log with arguments captured
return unknown
answer_relevance
Original user goal captured
return unknown
dependency_order
Tool call sequence with timestamps; declared dependencies
return unknown
This is the same idea as "garbage in, garbage out." If your trace doesn't capture the user's original ask, no Judge can score "answer relevance" honestly — it can only guess what the user wanted. Force the system to say "I can't tell" instead of guessing.
UNKNOWN is a first-class output
LLMs are extremely good at producing plausible-looking explanations. That's exactly the trait that makes them dangerous Judges when evidence is missing — they will fabricate a "complete" verdict instead of admitting the trace is incomplete.
If your Rubric doesn't have unknown / insufficient_evidence as a first-class output, the Judge will paper over evidence gaps with common-sense judgments. Those judgments then get written into your asset layer as "fixes" for problems that didn't exist.
An excellent eval Agent says "insufficient evidence, can't judge" often. The output looks less polished than one that always answers, but the rate of incorrect asset writebacks drops sharply. The self-evolution loop's worst enemy is not "missed bad samples" — it's "fabricated verdicts that became assets."
"An eval that says 'unknown' a lot looks unfinished — leadership won't accept it." That's a communication problem, not an eval problem. Reframe "unknown" as a coverage report — "X% of traces don't capture the evidence needed to judge groundedness" is the most actionable engineering ask you can hand to a team.
# Sketch of an evidence-gated rubric checkif retrieval_context isNoneor len(retrieval_context) == 0:
return {"groundedness": "unknown",
"reason": "no retrieval_context captured in trace"}
return judge.score_groundedness(answer, retrieval_context)
pass@k vs pass^k — handling non-determinism
Agents are non-deterministic. The same task run twice can succeed once and fail once. Two metrics name the two product realities you'll need to choose between:
Metric
Asks
Use when
pass@k
Probability of at least one success in k attempts.
Creative or exploratory tasks (e.g., code generation with retry; copywriting). Best-of-k is the product experience.
pass^k
Probability ALL k attempts succeed.
Reliability-critical tasks (payments, refunds, data deletion). A single bad attempt is a real-world incident.
The same Agent can look excellent on pass@5 and disastrous on pass^5. Both are honest descriptions of the same model — they describe different product surfaces. Picking the wrong one will systematically over- or under-estimate ship-readiness.
PMs working on reliability-critical flows (payments, account changes, anything an incident report would cover) should always report pass^k alongside pass@k. A 95% pass@1 sounds great until you do the math on 1 in 20 bookings being wrong.
Quiz — Applied
1. You're designing a rubric for an Agent that books appointments via a database. Which of these checks should be code-based, not LLM-Judge-based?
Database state has a deterministic oracle. Letting an LLM judge whether a record exists is replacing a SELECT statement with a guess.
2. Your Agent runs a 7-step bookings flow. The user says it worked. Which rubric kind would still catch "called the refund API twice in the middle"?
Outcome only sees the end-state ("booking made"). Answer only sees the user-facing text. Trajectory is the only rubric that inspects intermediate tool calls, which is where redundant or wrong middle-step calls show up.
3. The trace for a failed run is missing the original user goal. Your rubric scores groundedness=PASS and answer_relevance=PASS. What's the likely problem?
Answer relevance cannot be honestly scored without knowing what the user asked. A well-designed rubric forces "unknown" in this case; one that returns "pass" is filling the gap with a guess.
4. You write a Trajectory Rubric assertion: "Agent must call check_inventory, then validate_payment, then create_order, in this exact order." What's the risk?
Outcome is the main judge; Trace is the diagnostic. Hard-coded sequences punish valid alternatives. Only safety-critical steps (auth, payment, deletion) deserve must-pass assertions.
5. Two tools — "search_inventory" and "lookup_product" — have overlapping descriptions. The Agent keeps picking the wrong one. By TRAJECT-Bench's taxonomy, this is which failure mode?
Similar tool confusion is overlapping/distinguishable-only-by-fine-print tools. The fix is sharper tool descriptions and a better router, not a smarter model.
6. Your payments Agent has 95% pass@1 in eval. Leadership wants to ship. What additional metric should you report?
pass@k metrics describe creative-task quality; pass^k describes reliability. For payments, the right framing is "what fraction of single attempts works perfectly" — which exposes the 1-in-20 problem.
7. Which of these is the WRONG argument for adding "unknown" as a first-class rubric output?
Three of these are real reasons. "Dashboard cosmetics" is the wrong reason — and the most common objection when introducing unknown. Reframe it as a coverage report.
8. The Agent calls the same lookup tool 4 times in a row with slightly different arguments before settling on the answer. By TRAJECT-Bench, this is which mode, and where does the fix go?
Redundant calling is over-use of correct tools. The fix points at stopping conditions, cost guardrails, and the planner — and TRAJECT-Bench notes it's also a training-data smell that careful tool training can reduce.
0 / 8 correct
Flashcards — Applied
02 · Applied
Grader stratification
tap to reveal →
Anthropic's three-tier eval system: code-based for deterministic checks, model-based for semantic judgment, human for calibration. Each tier catches what the cheaper tier below it cannot.
← tap to flip back
02 · Applied
Code-based grader
tap to reveal →
Deterministic check using rules, schemas, sandboxes, or DB queries. Cheap, fast, reproducible. Use whenever the question has a non-probabilistic oracle.
← tap to flip back
02 · Applied
Model-based grader
tap to reveal →
LLM-as-Judge. Flexible, handles open-ended quality and semantic claims. Non-deterministic and biased — requires human calibration to be trustworthy.
← tap to flip back
02 · Applied
Human grader
tap to reveal →
Expert review or crowdsourced judgment. Expensive and slow but the gold standard. Reserved for calibration sets, gray-area cases, and taxonomy review — not bulk grading.
← tap to flip back
02 · Applied
Outcome Rubric
tap to reveal →
Asks: did the end-state in the world actually match the target? Verifies files written, records updated, tests passing, reports containing required conclusions. The main judge.
← tap to flip back
02 · Applied
Answer Rubric
tap to reveal →
Asks: was the final expression correct, complete, grounded, constraint-compliant? Catches hallucinations, missing citations, format violations, off-topic responses.
← tap to flip back
02 · Applied
Trajectory Rubric
tap to reveal →
Asks: were tool choice, parameters, dependency order, exception recovery, cost, and turn count reasonable? Catches risks invisible in the final answer. Acts as diagnostic evidence.
← tap to flip back
02 · Applied
Trace (transcript)
tap to reveal →
The complete record of a trial: outputs, tool calls, reasoning, intermediate results, costs. The substrate every rubric reads from. If the trace is incomplete, the Judge is forced to guess.
← tap to flip back
02 · Applied
Similar tool confusion
tap to reveal →
TRAJECT-Bench failure mode. Overlapping tool capabilities confuse the agent into picking the worse option. Fix: sharper tool descriptions, router improvements, disambiguation examples.
← tap to flip back
02 · Applied
Parameter-blind selection
tap to reveal →
TRAJECT-Bench failure mode. Right tool, wrong arguments — wrong values, formats, or units. Fix: tighter schemas, validators, enums, parameter examples in tool descriptions.
← tap to flip back
02 · Applied
Redundant tool calling
tap to reveal →
TRAJECT-Bench failure mode. Tool called more times than needed, or with unrelated hallucinated calls. Fix: stopping rules, cost guardrails, planner — also a training-data smell.
← tap to flip back
02 · Applied
Evidence prerequisite
tap to reveal →
Entry condition for an atomic dimension. Before scoring, check that the evidence needed exists in the trace; if not, return "unknown." Prevents Judge hallucination on missing data.
← tap to flip back
02 · Applied
UNKNOWN (insufficient evidence)
tap to reveal →
First-class rubric output for when the trace doesn't contain enough evidence to judge. The single most important defense against fabricated verdicts being written into asset fixes.
← tap to flip back
02 · Applied
pass@k
tap to reveal →
Probability of at least one success in k attempts. Use for creative or exploratory tasks where best-of-k retries are part of the product experience.
← tap to flip back
02 · Applied
pass^k
tap to reveal →
Probability that ALL k attempts succeed. Use for reliability-critical flows like payments or data deletion — any single bad attempt is a real-world incident.
← tap to flip back
02 · Applied
Must-pass assertion
tap to reveal →
Hard-constraint check on a specific step that must happen. Reserve for safety / business hard-rules (auth, payment, data deletion) — never for full path lockdown.