25 multiple-choice questions covering the seven principles, the three rubric kinds, the LLM Judge biases, and the operating loop. Live score tracks as you go.
0
Answered
0
Correct
0%
Score
01 · FoundationsWhy Rubrics Are a Product Decision0 / 9 correct
1. Your Agent gets a 2/5 from the eval Judge on a customer interaction. Which of the following is the most useful next action?
A total score doesn't tell you which layer broke. The dimension breakdown is what routes the fix to retrieval, tools, prompt, workflow, or memory.
2. Which of these is NOT part of the minimum rubric closed loop?
The five steps are Evidence Contract → Score Dimensions → Failure Taxonomy → Remediation Target → Calibration Loop.
3. Your RAG Agent's customer says the answer was wrong. Which atomic dimension tells you the retriever failed to surface the source material at all?
context_recall asks "was the necessary information actually retrieved?" Low scores route to retriever, query rewrite, or chunking.
4. The material was retrieved correctly but the model ignored it and hallucinated. Which dimension catches this specifically?
Groundedness / faithfulness asks whether key claims are supported by retrieved context. A retrieval hit + hallucinated answer is a classic faithfulness failure.
5. In the MT-Bench study, what was the measured position-bias rate for Claude-v1 — the percentage of cases where it favored whichever answer came first?
Claude-v1 was biased toward the first answer in 75% of pairs. GPT-3.5 was at 50%. GPT-4 was the only judge with >60% consistency.
6. Why do practitioners say you should not use the same model family as both the Judge and the Generator?
Self-enhancement bias is measurable. It biases the Judge toward its own generation patterns and silently inflates win-rate on regression suites.
7. What's G-Eval's main contribution over naive "rate this 1–5" prompts?
G-Eval's value is structural: criteria → CoT steps → form-filling → token-probability scoring. The same model becomes a much more reliable judge with structure.
8. Which of these is TRUE about LLM-as-a-Judge?
MT-Bench measured GPT-4 at 80%+ agreement with humans — matching human-to-human. That's why LLM-as-a-Judge is viable, with bias mitigations layered on.
9. Your team proposes a single "Agent Quality Score" rolled up from 7 sub-criteria for the company dashboard. What's the main product risk?
Total scores collapse routing. The actionable layer is always the atomic dimension — that's the field that says "fix the retriever" instead of "fix the prompt again."
02 · AppliedBuilding Your Rubric0 / 8 correct
1. You're designing a rubric for an Agent that books appointments via a database. Which of these checks should be code-based, not LLM-Judge-based?
Database state has a deterministic oracle. Letting an LLM judge whether a record exists replaces a SELECT statement with a guess.
2. Your Agent runs a 7-step bookings flow. The user says it worked. Which rubric kind would still catch "called the refund API twice in the middle"?
Outcome only sees the end-state. Answer only sees user-facing text. Trajectory is the only rubric that inspects intermediate tool calls.
3. The trace for a failed run is missing the original user goal. Your rubric scores groundedness=PASS and answer_relevance=PASS. What's the likely problem?
Answer relevance cannot be honestly scored without knowing what the user asked. Without an evidence prerequisite, the Judge fills the gap with a guess.
4. You write a Trajectory Rubric assertion: "Agent must call check_inventory, then validate_payment, then create_order, in this exact order." What's the risk?
Outcome is the main judge; Trace is the diagnostic. Hard-coded sequences punish valid alternatives. Only safety-critical steps deserve must-pass.
5. Two tools — "search_inventory" and "lookup_product" — have overlapping descriptions. The Agent keeps picking the wrong one. By TRAJECT-Bench's taxonomy, this is which failure mode?
Similar tool confusion is overlapping tools distinguishable only by fine print. Fix: sharper descriptions, better router.
6. Your payments Agent has 95% pass@1 in eval. Leadership wants to ship. What additional metric should you report?
7. Which of these is the WRONG argument for adding "unknown" as a first-class rubric output?
Three of these are real reasons. "Dashboard cosmetics" is the wrong reason and the most common objection. Reframe unknown as a coverage report.
8. The Agent calls the same lookup tool 4 times with slightly different arguments before settling. By TRAJECT-Bench, this is which mode, and where does the fix go?
Redundant calling is over-use of correct tools. The fix points at stopping conditions, cost guardrails, and the planner.
03 · AdvancedOperating the Rubric0 / 8 correct
1. Your team uses one generic "rate this 1–5 on accuracy, completeness, helpfulness" rubric across three different Agents (code review, refund processing, customer Q&A). Why is this likely producing bad fixes?
Generic checklists both pass dangerous solutions and fail valid ones. The fix is per-instance rubrics from a Context Gathering stage.
2. Which stage in the contextual rubric chain is missing from this list: Rubric Builder → Trace Inspector → Failure Attribution → Eval Case Generator?
The chain starts with a Context Gathering Agent. Without it, the Rubric Builder works from generic templates.
3. In the Agentic Rubrics for SWE paper, Sonnet-4.5-generated rubrics showed only 2% flakiness across 5 independent Judge runs. The authors attribute the low flakiness to which property of the rubric items?
Atomicity wins again. Self-contained items leave less room for the Judge to interpret differently across runs.
4. Humans keep overturning your Judge's verdicts on the same dimension. Which of these is the right first action?
Human-overrules-Judge is the canonical aging signal. The fix is in the rubric (criteria + decomposition + calibration), not the model.
5. A rubric dimension has been at 100% pass for 6 weeks. What does this most likely mean, and what should you do?
A perma-pass dimension stops discriminating. Lock it in as regression or raise the bar so it produces signal.
6. You want to release a new version of a rubric criterion that you believe is stricter than the current one. What's the safest release process?
Rubric changes are system changes. Replay + bias tests + shadow mode is the canary-deploy equivalent.
7. Which of the following metrics is NOT one you'd track when running a shadow rubric evaluation against the live rubric?
Cost matters for budget but doesn't tell you whether the candidate is a better judge. The four release-gate metrics are agreement, FP, FN, and remediation accuracy.
8. What is the single best test for whether a Rubric is part of the self-evolution loop vs. just a scoreboard?
A repairable signal is the whole point. One team, one fix — that's when the eval Agent has joined engineering.
—%
Final Score
Guide 1 · Foundations
—
Guide 2 · Applied
—
Guide 3 · Advanced
—
Done quizzing? Drill the concepts with flashcards.All flashcards →