A bad answer is the symptom. The broken layer is the bug. Until your eval points at a layer, you're just rating quality — not driving fixes.
If you remember nothing else
A rubric isn't a scoring prompt — it's a five-step protocol that turns "user complained" into "team X ships fix Y."
Why total scores fail
A 3-out-of-5 quality score routes nowhere. Atomic dimensions route the same complaint to retrieval, tools, prompts, workflow, or memory — five different fixes.
Don't trust one LLM Judge
MT-Bench measured Claude-v1 favoring whichever answer came first 75% of the time. Judges are biased; calibrate them like you'd calibrate any production signal.
Rubric is a system protocol, not a scoring prompt
Most teams treat a rubric as a prompt for the LLM Judge: "Rate this answer 1–5 on accuracy, completeness, and helpfulness." That works as a quality check. It does not work as a self-improvement loop.
A rubric that drives system improvement binds five things together: what evidence the Judge is allowed to read, how scores are decomposed, how low scores are categorized into failure types, what asset each failure type points at, and how the rubric itself stays calibrated over time.
The Rubric's job is not to produce a number. It is to compress a messy trace into a labeled failure that points at exactly one asset to fix — a prompt, a tool description, a retrieval query, a workflow guardrail, or a human review queue.
Picture a bug ticket without a repro step or a stack trace. You can rate it "high priority" all day; no one knows whose code to open. A rubric without these five parts is that ticket.
The same bad answer hides five different bugs
Agent failures look identical from the user's seat: the answer was wrong, the order didn't go through, the report was incomplete. But behind "wrong" sit five separate broken layers, each with its own owner and fix.
Broken layer
What it looks like in the trace
Who fixes it
Retrieval
The answer cites nothing because the search returned nothing relevant.
Retriever / chunking
Tool
The right material was found, but the model called the wrong API or filled bad arguments.
Tool schema / validator
Prompt
Sources were found and read, but the model ignored them and ad-libbed.
System prompt / output schema
Workflow
The high-risk step had no verification gate, so a hallucination flowed through.
Workflow graph / planner
Memory
User preferences were re-asked or violated because nothing wrote them to long-term store.
Memory / personalization
If your eval only says "the answer was wrong," all five owners shrug. If your eval names the broken layer, exactly one team picks it up.
Better prompts will fix it. Most reflexive responses to a bad eval are "let's adjust the system prompt." But if the failure was a tool argument or a missing memory write, the prompt is exactly the wrong place to look — and you'll keep losing the same case.
A total score can't route a fix
The most common rubric mistake is collapsing many criteria into one number — "Rate this 1–5 on accuracy, completeness, clarity, and helpfulness." Humans do this naturally; a self-improving system can't.
When the system reads back "3," it has no idea whether to change retrieval, tools, prompts, or workflow. So it does the cheapest thing: it tweaks the prompt. That's why most "AI quality" review cycles drift into prompt fiddling and the same failures keep coming back.
Each rubric dimension should answer exactly one question, and each question should map to exactly one fix direction. The more you collapse dimensions, the more you collapse fixes into "change the prompt again."
Imagine if your only bug-tracker field were "severity 1–5." Engineering would still ship fixes, but they'd be guessing at scope. The "component" field is what makes triage cheap. Atomic rubric dimensions are the component field of your eval system.
The atomic dimension table
A practical knowledge-intensive Agent rubric breaks into at least ten atomic dimensions. Each is one binary or short scale, each maps to one asset.
Dimension
What it checks
Fix points at
answer_relevance
Does the response actually answer the user's goal?
Intent routing, task router
groundedness
Are key claims supported by retrieved context or tool results?
Retrieval, citation, evidence constraint
context_recall
Was the necessary information actually retrieved?
Retriever, query rewrite, chunking
context_precision
Is the retrieved context clean and well-ranked?
Rerank, filter, context compression
instruction_following
Were explicit constraints and format rules respected?
Prompt, output schema
tool_selection
Did the agent pick the right tool for the job?
Tool descriptions, router
argument_correctness
Are tool arguments complete, legal, and task-aligned?
Schema, validator, parameter hints
dependency_order
Did tool calls satisfy the required before/after order?
Workflow graph, planner
outcome_state
Did the end-state in the world actually become the target state?
Verifier, sandbox, end-state check
interaction_efficiency
Are turn count, tool calls, and cost in the normal range?
Stopping rule, cost guardrail
You don't need all ten on day one. A useful starter set for a RAG-style Agent is answer_relevance, groundedness, context_recall, context_precision. For a tool-using Agent, add tool_selection, argument_correctness, dependency_order, outcome_state.
Notice each row has a verb in the third column. "Fix points at" is the whole reason for atomicity. If a dimension can't point at a verb you'd actually do this sprint, drop it.
G-Eval — turn fuzzy criteria into structured evaluation
"Rate this answer 1–5 on coherence" is a famously bad prompt. The model's score collapses to the mode (everything gets 3), the variance is low, and the correlation with human judgments is weak. G-Eval (Yang Liu et al., 2023) gave the standard fix.
The trick is two-stage. First, ask the model to generate its own chain-of-thought evaluation steps from the criteria — turning a one-line definition into a numbered procedure. Then have it score each criterion by form-filling against those steps. Final score uses token probabilities to make the output continuous rather than collapsing to a single integer.
Evaluation criteria→Auto chain-of-thought steps→Form-filling against steps→Probability-weighted score
G-Eval reached Spearman 0.514 with human summarization scores — a large jump over previous reference-free metrics. The reason is structure, not model power: the same model with "rate 1–5" loses on the same task.
"We need a smarter model to judge well." G-Eval's lesson is the opposite — the same model becomes a much better judge the moment you structure the criteria into explicit steps and let it form-fill rather than guess a number.
LLM Judge biases you'll trip over
If you let one LLM grade your Agent in production with no calibration, you will ship the Judge's biases. The MT-Bench paper (Zheng et al., NeurIPS 2023) measured three that hit almost every team.
Bias
How big
How to defend
Position bias
Claude-v1 favored the first answer 75% of the time. GPT-3.5 50%. GPT-4 only when consistency dropped to 65%.
Swap A/B order; only count agreements across both orderings.
Verbosity bias
"Repetitive list" attack: Claude-v1 fooled 91.3% of the time, GPT-3.5 91.3%, GPT-4 only 8.7%.
Length-normalize. Test with a known-padding attack in CI.
Self-enhancement bias
GPT-4 favored its own answers by +10% win rate. Claude-v1 by +25%.
Never use the same model family as Judge and Generator in regression-critical lanes.
The good news from the same paper: GPT-4 as Judge reaches 80%+ agreement with human experts — about the same as expert-to-expert agreement. The bad news is the only way you know your Judge is in the safe regime is by running a calibration set against humans.
"LLM Judge is objective." It isn't. It has predictable, measurable, and large biases. Treating it as ground truth instead of an instrument-needing-calibration is how Judges drift unnoticed.
The RAGAS split — retrieval vs. generation
For RAG-style Agents, the most useful pre-built dimension split is from RAGAS. It carves RAG quality into four numbers along two failure axes:
Metric
Owns which failure axis
Routes the fix to
context_recall
Was the necessary material in the retrieved set at all?
Retrieval team
context_precision
Is the retrieved set clean and well-ordered?
Retrieval team
faithfulness
Did the model use the retrieved material honestly?
Generator team
answer_relevancy
Does the answer address the user's actual question?
Generator team
The reason RAGAS doesn't just publish "RAG score" is that retrieval failures and generation failures need different fixes. If you only know "RAG is at 3/5," you don't know whether to invest a sprint in your retriever or in your prompts. If you know context_recall is 0.4 but faithfulness is 0.95, the retriever is starved and the generator is doing its job — fix the retriever.
Splitting one quality number into four metrics turns evaluation into diagnosis. A diagnosis tells you which doctor to call. A score tells you to feel bad.
Quiz — Foundations
1. Your Agent gets a 2/5 from the eval Judge on a customer interaction. Which of the following is the most useful next action?
A total score doesn't tell you which layer broke. The dimension breakdown is what routes the fix to retrieval, tools, prompt, workflow, or memory.
2. Which of these is NOT part of the minimum rubric closed loop?
The five steps are Evidence Contract → Score Dimensions → Failure Taxonomy → Remediation Target → Calibration Loop. Each is a system protocol element, not a comms artifact.
3. Your RAG Agent's customer says the answer was wrong. Which atomic dimension would tell you the retriever failed to surface the source material at all?
context_recall asks "was the necessary information actually retrieved?" A low score routes to the retriever, query rewrite, or chunking — not to the prompt.
4. The material was retrieved correctly but the model ignored it and hallucinated. Which dimension catches this specifically?
Groundedness / faithfulness asks whether key claims are actually supported by the retrieved context. A retrieval hit + hallucinated answer is a classic faithfulness failure that routes to the generator team.
5. In the MT-Bench study, what was the measured position-bias rate for Claude-v1 — the percentage of cases where it favored whichever answer came first?
Claude-v1 was biased toward the first answer in 75% of pairs. GPT-3.5 was at 50%. GPT-4 was the only judge with greater than 60% consistency across both orderings.
6. Why do practitioners say you should not use the same model family as both the Judge and the Generator?
Self-enhancement bias is real and measurable. It biases the Judge toward its own generation patterns and silently inflates win-rate on regression suites.
7. What's G-Eval's main contribution over naive "rate this 1–5" prompts?
G-Eval's value is structural: criteria → CoT steps → form-filling → token-probability scoring. The same model becomes a much more reliable judge when the procedure is structured.
8. Which of these is TRUE about LLM-as-a-Judge?
MT-Bench measured GPT-4 reaching 80%+ agreement with humans, matching human-to-human agreement. That's why LLM-as-a-Judge is viable — but only with bias mitigations layered on.
9. Your team proposes a single "Agent Quality Score" rolled up from 7 sub-criteria for the company dashboard. What's the main product risk?
Total scores collapse routing. A dashboard number is fine as a top-line indicator, but the actionable layer is always the atomic dimension — that's the field that says "fix the retriever" instead of "fix the prompt for the 12th time."
0 / 9 correct
Flashcards — Foundations
01 · Foundations
answer_relevance
tap to reveal →
Atomic rubric dimension. Does the response actually address the user's goal, or does it answer a different question? A low score routes to intent recognition and task routing.
← tap to flip back
01 · Foundations
groundedness
tap to reveal →
Are the key claims in the answer actually supported by retrieved context or tool results? A low score routes to retrieval, citation logic, or evidence constraints in the prompt.
← tap to flip back
01 · Foundations
context_recall
tap to reveal →
Was the necessary information actually retrieved at all? Low scores route to retriever, query rewrite, or chunking — never to the prompt.
← tap to flip back
01 · Foundations
context_precision
tap to reveal →
Is the retrieved context clean and ranked well? Low scores point at rerankers, filtering, and context compression — the noise side of retrieval, not the recall side.
← tap to flip back
01 · Foundations
tool_selection
tap to reveal →
Did the agent pick the right tool from a set with overlapping capabilities? Low scores route to tool descriptions and the router — not to the model's intelligence.
← tap to flip back
01 · Foundations
argument_correctness
tap to reveal →
Are tool arguments complete, legally formatted, and consistent with the task? Low scores route to schemas, validators, enums, and parameter examples in the tool description.
← tap to flip back
01 · Foundations
dependency_order
tap to reveal →
Did tool calls happen in the required before/after order? Low scores route to workflow graphs and the planner — the structural constraints between steps.
← tap to flip back
01 · Foundations
outcome_state
tap to reveal →
Did the end-state in the world actually become the target state — was the record updated, the file written, the test passing? Verified by code, not by LLM judgment.
← tap to flip back
01 · Foundations
interaction_efficiency
tap to reveal →
Are turn count, tool calls, and cost within normal range? An efficiency anomaly is often the first signal of a loop or redundant-call failure mode.
← tap to flip back
01 · Foundations
Evidence Contract
tap to reveal →
Step 1 of the rubric closed loop. The rule that says which fields the Judge is allowed to read, and which fields force a return of "unknown" when missing.
← tap to flip back
01 · Foundations
Score Dimensions
tap to reveal →
Step 2 of the rubric closed loop. Quality is broken into atomic, independently scorable dimensions so each low score can route to one fix direction.
← tap to flip back
01 · Foundations
Failure Taxonomy
tap to reveal →
Step 3 of the rubric closed loop. Low scores get categorized into actionable failure types — not just "bad" but "retrieval miss" or "tool argument error."
← tap to flip back
01 · Foundations
Remediation Target
tap to reveal →
Step 4 of the rubric closed loop. Each failure type points at exactly one asset — prompt, tool, workflow, memory, retriever, model, or human review.
← tap to flip back
01 · Foundations
Calibration Loop
tap to reveal →
Step 5 of the rubric closed loop. The mechanism that keeps the Judge honest — human spot-checks, calibration sets, bias attacks, drift alerts.
← tap to flip back
01 · Foundations
G-Eval
tap to reveal →
Structured LLM-as-Judge method (Liu et al., 2023). Auto-generates chain-of-thought evaluation steps from criteria, scores by form-filling, uses token probability for continuous output.
← tap to flip back
01 · Foundations
Position bias
tap to reveal →
LLM Judge bias toward whichever option appears first. MT-Bench: Claude-v1 75%, GPT-3.5 50%, GPT-4 lowest. Defend by swapping order and only counting answers stable across both.
← tap to flip back
01 · Foundations
Verbosity bias
tap to reveal →
LLM Judge bias toward longer answers. "Repetitive list" attack fooled Claude-v1 and GPT-3.5 91.3%, GPT-4 only 8.7%. Defend by length-normalizing and CI-attacking.
← tap to flip back
01 · Foundations
Self-enhancement bias
tap to reveal →
LLM Judges favor their own model family's outputs. GPT-4 +10% win-rate boost on its own. Claude-v1 +25%. Never let Judge and Generator share a family in regression-critical lanes.
← tap to flip back
01 · Foundations02 · Applied
faithfulness (RAGAS)
tap to reveal →
RAGAS metric. Did the generator faithfully use the retrieved context, or did it ignore/contradict it? The generator-side counterpart to context_recall — same symptom, different fix.