Drill the Atoms

Raghavendra et al., 2026. Generates per-instance code rubrics from repo + issue. Scores patches on file targeting, spec alignment, integrity, runtime. Sonnet-4.5 rubrics had only 2% flakiness.

Asks: was the final expression correct, complete, grounded, constraint-compliant? Catches hallucinations, missing citations, format violations, off-topic responses.

Atomic rubric dimension. Does the response actually address the user's goal, or does it answer a different question? A low score routes to intent recognition and task routing.

Are tool arguments complete, legally formatted, and consistent with the task? Low scores route to schemas, validators, enums, and parameter examples in the tool description.

Adversarial probes for known LLM Judge weaknesses: position swap, repetitive-list verbosity attack, self-enhancement probe. Run on every candidate rubric or Judge model change.

Step 5 of the rubric closed loop. The mechanism that keeps the Judge honest — human spot-checks, calibration sets, bias attacks, drift alerts.

A small, hand-graded set of representative samples (good + bad) used to measure whether a Judge or rubric matches human judgment. The "staging environment" for eval changes.

Deterministic check using rules, schemas, sandboxes, or DB queries. Cheap, fast, reproducible. Use whenever the question has a non-probabilistic oracle.

First stage of the contextual rubric chain. Collects user goal, business rules, available tools, ground truth, end-state, and known failure modes before any judging happens.

Is the retrieved context clean and ranked well? Low scores point at rerankers, filtering, and context compression — the noise side of retrieval, not the recall side.

Was the necessary information actually retrieved at all? Low scores route to retriever, query rewrite, or chunking — never to the prompt.

Did tool calls happen in the required before/after order? Low scores route to workflow graphs and the planner — the structural constraints between steps.

Final stage of the rubric chain. Packages today's failure into a reusable replay sample plus a candidate fix sketch. Today's incident becomes tomorrow's regression test.

Step 1 of the rubric closed loop. The rule that says which fields the Judge is allowed to read, and which fields force a return of "unknown" when missing.

Entry condition for an atomic dimension. Before scoring, check that the evidence needed exists in the trace; if not, return "unknown." Prevents Judge hallucination on missing data.

RAGAS metric. Did the generator faithfully use the retrieved context, or did it ignore/contradict it? The generator-side counterpart to context_recall — same symptom, different fix.

Maps a low score on a dimension to a broken layer (retrieval / tool / prompt / workflow / memory) and a specific remediation target. Turns "bad" into "fix this asset."

Step 3 of the rubric closed loop. Low scores get categorized into actionable failure types — not just "bad" but "retrieval miss" or "tool argument error."

Structured LLM-as-Judge method (Liu et al., 2023). Auto-generates chain-of-thought evaluation steps from criteria, scores by form-filling, uses token probability for continuous output.

Anthropic's three-tier eval system: code-based for deterministic checks, model-based for semantic judgment, human for calibration. Each tier catches what the cheaper tier below it cannot.

Are the key claims in the answer actually supported by retrieved context or tool results? A low score routes to retrieval, citation logic, or evidence constraints in the prompt.

Expert review or crowdsourced judgment. Expensive and slow but the gold standard. Reserved for calibration sets, gray-area cases, and taxonomy review — not bulk grading.

Are turn count, tool calls, and cost within normal range? An efficiency anomaly is often the first signal of a loop or redundant-call failure mode.

LLM-as-Judge. Flexible, handles open-ended quality and semantic claims. Non-deterministic and biased — requires human calibration to be trustworthy.

Hard-constraint check on a specific step that must happen. Reserve for safety / business hard-rules (auth, payment, data deletion) — never for full path lockdown.

Asks: did the end-state in the world actually match the target? Verifies files written, records updated, tests passing, reports containing required conclusions. The main judge.

Did the end-state in the world actually become the target state — was the record updated, the file written, the test passing? Verified by code, not by LLM judgment.

TRAJECT-Bench failure mode. Right tool, wrong arguments — wrong values, formats, or units. Fix: tighter schemas, validators, enums, parameter examples in tool descriptions.

Probability of at least one success in k attempts. Use for creative or exploratory tasks where best-of-k retries are part of the product experience.

Probability that ALL k attempts succeed. Use for reliability-critical flows like payments or data deletion — any single bad attempt is a real-world incident.

LLM Judge bias toward whichever option appears first. MT-Bench: Claude-v1 75%, GPT-3.5 50%, GPT-4 lowest. Defend by swapping order and only counting answers stable across both.

TRAJECT-Bench failure mode. Tool called more times than needed, or with unrelated hallucinated calls. Fix: stopping rules, cost guardrails, planner — also a training-data smell.

Step 4 of the rubric closed loop. Each failure type points at exactly one asset — prompt, tool, workflow, memory, retriever, model, or human review.

The Rubric's actual output goal. Not a number; a failure type + asset target + candidate fix that a teammate can use to file a ticket with one owner and one suggested action.

Four canonical signals that the rubric (not the Agent) is broken: humans overturning the Judge, persistent unknown rates, low scores that can't map to fixes, perma-pass dimensions.

Generates task-specific criteria from gathered context. Each criterion comes with an evidence prerequisite and a remediation target. Replaces generic checklists.

Treat each rubric change like a code release: bug report → candidate patch → calibration set replay → bias tests → shadow eval → version pin. Rolling forward and back become safe operations.

Step 2 of the rubric closed loop. Quality is broken into atomic, independently scorable dimensions so each low score can route to one fix direction.

LLM Judges favor their own model family's outputs. GPT-4 +10% win-rate boost on its own. Claude-v1 +25%. Never let Judge and Generator share a family in regression-critical lanes.

Run a candidate rubric in parallel with the live rubric on real traffic. Only release if human-agreement, false-positive rate, false-negative rate, and remediation-pointing accuracy all improve.

TRAJECT-Bench failure mode. Overlapping tool capabilities confuse the agent into picking the worse option. Fix: sharper tool descriptions, router improvements, disambiguation examples.

Did the agent pick the right tool from a set with overlapping capabilities? Low scores route to tool descriptions and the router — not to the model's intelligence.

The complete record of a trial: outputs, tool calls, reasoning, intermediate results, costs. The substrate every rubric reads from. If the trace is incomplete, the Judge is forced to guess.

Structured reader of the trace: tool calls, intermediate states, end-state, costs. Feeds the Judge the evidence each rubric dimension needs.

Asks: were tool choice, parameters, dependency order, exception recovery, cost, and turn count reasonable? Catches risks invisible in the final answer. Acts as diagnostic evidence.

First-class rubric output for when the trace doesn't contain enough evidence to judge. The single most important defense against fabricated verdicts being written into asset fixes.

LLM Judge bias toward longer answers. "Repetitive list" attack fooled Claude-v1 and GPT-3.5 91.3%, GPT-4 only 8.7%. Defend by length-normalizing and CI-attacking.