Rubrics age. Treat them as system assets, not scoring prompts. The discipline that separates a one-shot eval from a self-evolving system is the loop that keeps rubrics honest.
Context before checklist
Generic rubrics pass dangerous solutions and fail valid ones. Build the rubric from each task's context first, then judge — pay one upfront cost, save downstream wrong fixes.
Rubrics age
When humans keep overturning the Judge, when "unknown" rates climb, when a dimension goes perma-pass — the rubric is broken, not the Agent. Treat rubric changes like system changes.
Judge → repairable signal
The Rubric's customer is system improvement, not the score. If a low score can't turn into a fix target, it's a quality review — not part of self-evolution.
Generic checklists misjudge specific tasks
The reason "rate this Agent 1–5 on accuracy, helpfulness, clarity" doesn't survive contact with complex tasks is that the right judgment depends on context the rubric doesn't carry. A code Agent's "correct" depends on the repo structure, related functions, and test constraints. A business-flow Agent's "correct" depends on tool interfaces, state machines, and rule boundaries. A knowledge-task Agent's "correct" depends on the available sources and the user's specific goal.
When the rubric runs without first collecting this context, two failure modes dominate:
Failure
What happens
PM-visible symptom
False positive
A dangerous solution passes because the checklist didn't know about the missing edge case.
Production incident — "the eval said pass."
False negative
A valid creative solution fails because it took a path the generic checklist didn't anticipate.
Eng frustration — "the eval is brittle."
"A bigger, more general checklist will catch more cases." It won't. Adding rows to a generic rubric adds noise without adding context-specific judgment. The only fix is to derive criteria from the task at hand.
The contextual rubric chain
High-quality Agentic Rubrics aren't templated. They grow from the context of each task before any scoring happens. The chain looks like:
Context Gathering Agent→Rubric Builder→Trace Inspector→Failure Attribution→Eval Case Generator
The first half builds the evaluation basis — user goal, business rules, available tools, ground truth, end-state, similar historical failures, user preferences, known failure modes. The second half applies the Rubric to the current trace. This costs one extra upfront context-collection round per task, but it dramatically reduces the rate of incorrect "fixes" being written into your assets.
Stage
What it produces
Why it exists
Context Gathering Agent
User goal, business rules, available tools, ground truth, end-state, historical failures.
Without context, judgment is guessing.
Rubric Builder
Task-specific criteria with evidence prerequisites and remediation targets.
Generic checklists misjudge — this is the cure.
Trace Inspector
Structured reading of the trace: tool calls, intermediates, end-state.
Reads what the Judge needs to score.
Failure Attribution
Maps low scores to broken layer + remediation target.
The tradeoff is not "spend more on one eval Agent or skip it." It's "spend once on understanding the task, or spend repeatedly on incorrect 'fixes' that don't stick." The upfront cost compounds in the opposite direction from the downstream cost.
Agentic Rubrics — context-built criteria for code Agents
The Agentic Rubrics for SWE Agents paper (Raghavendra et al., 2026) gave the contextual-rubric idea a concrete shape for software engineering tasks. Instead of a one-size checklist for "is this patch good," they generate per-instance rubrics from the repo plus the issue, scoring patches on:
Rubric axis
What it checks
Why per-instance matters
File / Method targeting
Was the right code surface changed?
Depends on the repo structure.
Spec Alignment
Does the patch match the issue's stated intent?
Depends on the issue text.
Integrity
Does the change leave related invariants intact?
Depends on the function neighborhood.
Runtime
Does it run cleanly under the project's test/build constraints?
Depends on project-specific test setup.
One striking number from their flakiness study: when rubrics were generated by Sonnet-4.5 with atomic, self-contained criteria, only 2% of rubric items were flaky across 5 independent Judge runs. Qwen3-32B-generated rubrics hit 9%. The paper attributes the gap to atomic, self-contained instructions reducing the scope for Judge interpretation.
A well-built rubric is more reliable than a stronger Judge. The single biggest determinant of grading consistency in their experiments wasn't the Judge model — it was whether the rubric's items were atomic and self-contained. The same lesson as Guide 01's atomicity rule, measured in a SWE setting.
When the rubric itself is broken — aging signals
Run an Agent long enough and the bad samples will expose problems in your eval, not just your Agent. The Rubric is itself a system asset, and it's affected by business shifts, tool changes, model upgrades, and user-expectation drift. Four aging signals are worth setting alerts on:
Signal
What it tells you
What to change
Humans keep overturning the Judge
Criteria are vague or mismatched with how humans actually grade now.
Trace schema is incomplete — the system can't capture what the rubric needs.
Extend the trace schema, not the rubric.
A class of low scores can't map to a fix
Failure taxonomy is too coarse for the new failure mode.
Add a new failure type and remediation target.
A dimension is perma-pass
It's saturated; either the system genuinely solved it (turn into regression suite) or the dimension is too easy.
Convert to regression suite OR raise difficulty / specificity.
"Eval drift means the Agent is degrading." Not necessarily. Eval drift can also mean the rubric is what's stale — the business changed, the tool changed, or human standards moved. Always check the rubric's calibration before declaring an Agent regression.
Treat rubric changes like system changes
Once you accept that the Rubric is a system asset, the discipline that keeps it healthy mirrors what you already do for production code. Rubric changes shouldn't go straight to production any more than untested DB migrations should.
Human correction / production incident / Judge disagreement↓Rubric bug report↓Candidate rubric patch↓Calibration set replay↓Bias tests (position swap, length attack, self-enhancement check)↓Shadow evaluation (run in parallel with live rubric)↓Rubric version release
Why this matters: a rubric that gets stricter overnight starts failing reasonable samples; one that gets looser starts passing bad ones; one with a shifted attribution starts pointing at the wrong team to fix things. Versioning the rubric and running it in shadow first is what prevents these regressions from being silent.
Shadow evaluation — the safest way to ship a rubric change
Shadow mode runs the candidate rubric in parallel with the live rubric on real traffic for a fixed window. Both produce scores. Only the live rubric drives asset writebacks. The candidate's outputs are logged and compared.
Before releasing the candidate as the new live rubric, four metrics must look better on the calibration set than the current rubric:
Metric
What it measures
Direction
Human-agreement rate
% of judgments where the rubric agrees with the human gold label.
↑ better
False positive rate
% of bad samples the rubric incorrectly let through.
↓ better
False negative rate
% of good samples the rubric incorrectly flagged.
↓ better
Remediation-pointing accuracy
% of failures where the named asset was actually the right one to fix.
↑ better
This is the same pattern as feature flags + canary deploys for product features. The Rubric is the production code of your eval system. Shipping a Rubric change without shadow mode is shipping a DB migration without staging.
The conclusion — Judge → repairable signal
The Rubric's ultimate customer is not the score. It is system improvement. The Rubric that finds bad samples but can't explain which layer to fix is still just a quality review. The Rubric that compresses trace evidence into failure types, regression samples, and candidate patches becomes the hub of self-evolution.
A useful test of any rubric you propose: when a sample fails, can a teammate use the rubric's output to file a ticket assigned to exactly one team, with exactly one suggested fix? If yes, the rubric is part of the self-evolution loop. If no, it's still just a scoreboard.
Five chained "because → so" statements summarize the whole field — and they map back to everything in Guides 01–03:
Because…
So the rubric must…
Agent failures happen in a system chain, not in text.
Define evidence fields before judging.
Evidence will be missing in real traces.
Allow "unknown" as a first-class output.
A low score has to drive a fix.
Have dimensions that map to asset layers.
Judges are biased; standards age.
Run replays, bias tests, and shadow evaluations.
Tasks are context-specific.
Build per-instance criteria, not generic checklists.
Make that chain a protocol — not a prompt — and your eval Agent stops being a critic and starts being part of the engineering team.
Quiz — Advanced
1. Your team uses one generic "rate this 1–5 on accuracy, completeness, helpfulness" rubric across three different Agents (code review, refund processing, customer Q&A). Why is this likely producing bad fixes?
Generic checklists pass dangerous solutions (false positives) and fail valid ones (false negatives) because they don't carry task context. The fix is per-instance rubrics from a Context Gathering stage.
2. Which stage in the contextual rubric chain is missing from this list: Rubric Builder → Trace Inspector → Failure Attribution → Eval Case Generator?
The chain starts with a Context Gathering Agent that collects user goal, rules, tools, ground truth, and known failures. Without it, the Rubric Builder works from generic templates.
3. In the Agentic Rubrics for SWE paper, Sonnet-4.5-generated rubrics showed only 2% flakiness across 5 independent Judge runs. The authors attribute the low flakiness to which property of the rubric items?
Atomicity wins again. Self-contained items leave less room for the Judge to interpret differently across runs, so grading determinism rises.
4. Humans keep overturning your Judge's verdicts on the same dimension. Which of these is the right first action?
Human-overrules-Judge is the canonical aging signal. The fix is in the rubric (criteria language + decomposition + calibration), not the model.
5. A rubric dimension has been at 100% pass for 6 weeks. What does this most likely mean, and what should you do?
A perma-pass dimension isn't discriminating anymore. The healthy responses are either to "lock in" the win as a regression test or to raise the bar so the dimension produces signal again.
6. You want to release a new version of a rubric criterion that you believe is stricter than the current one. What's the safest release process?
Rubric changes are system changes. Replay + bias tests + shadow mode is the canary-deploy equivalent for eval systems.
7. Which of the following metrics is NOT one you'd track when running a shadow rubric evaluation against the live rubric?
Cost matters for budget but doesn't tell you whether the candidate rubric is a better judge. The four release-gate metrics are agreement, FP, FN, and remediation accuracy.
8. What is the single best test for whether a Rubric is part of the self-evolution loop vs. just a scoreboard?
A repairable signal is the whole point. If the output routes to one team and one fix, the eval Agent has joined engineering. If it just produces scores, it's a quality review.
0 / 8 correct
Flashcards — Advanced
03 · Advanced
Context Gathering Agent
tap to reveal →
First stage of the contextual rubric chain. Collects user goal, business rules, available tools, ground truth, end-state, and known failure modes before any judging happens.
← tap to flip back
03 · Advanced
Rubric Builder
tap to reveal →
Generates task-specific criteria from gathered context. Each criterion comes with an evidence prerequisite and a remediation target. Replaces generic checklists.
← tap to flip back
03 · Advanced
Trace Inspector
tap to reveal →
Structured reader of the trace: tool calls, intermediate states, end-state, costs. Feeds the Judge the evidence each rubric dimension needs.
← tap to flip back
03 · Advanced
Failure Attribution
tap to reveal →
Maps a low score on a dimension to a broken layer (retrieval / tool / prompt / workflow / memory) and a specific remediation target. Turns "bad" into "fix this asset."
← tap to flip back
03 · Advanced
Eval Case Generator
tap to reveal →
Final stage. Packages today's failure into a reusable replay sample plus a candidate fix sketch. Today's incident becomes tomorrow's regression test.
← tap to flip back
03 · Advanced
Agentic Rubrics for SWE
tap to reveal →
Raghavendra et al., 2026. Generates per-instance code rubrics from repo + issue. Scores patches on file targeting, spec alignment, integrity, runtime. Sonnet-4.5 rubrics had only 2% flakiness.
← tap to flip back
03 · Advanced
Rubric aging signals
tap to reveal →
Four canonical signals that the rubric (not the Agent) is broken: humans overturning the Judge, persistent unknown rates, low scores that can't map to fixes, perma-pass dimensions.
← tap to flip back
03 · Advanced
Calibration set
tap to reveal →
A small, hand-graded set of representative samples (good + bad) used to measure whether a Judge or rubric matches human judgment. The "staging environment" for eval changes.
← tap to flip back
03 · Advanced
Bias tests
tap to reveal →
Adversarial probes for known LLM Judge weaknesses: position swap, repetitive-list verbosity attack, self-enhancement probe. Run on every candidate rubric or Judge model change.
← tap to flip back
03 · Advanced
Shadow evaluation
tap to reveal →
Run a candidate rubric in parallel with the live rubric on real traffic. Only release if human-agreement, false-positive rate, false-negative rate, and remediation-pointing accuracy all improve.
← tap to flip back
03 · Advanced
Rubric version release
tap to reveal →
Treat each rubric change like a code release: bug report → candidate patch → calibration set replay → bias tests → shadow eval → version pin. Rolling forward and back become safe operations.
← tap to flip back
03 · Advanced
Repairable signal
tap to reveal →
The Rubric's actual output goal. Not a number; a failure type + asset target + candidate fix that a teammate can use to file a ticket with one owner and one suggested action.