Every concept from all three guides, deduplicated and alphabetized. Filter by guide, shuffle the deck, or flip everything at once.
Showing 40 cards · 0 flipped
03 · Roadmap
A/B testing
tap to reveal →
Comparing variants with real user traffic. Measures actual outcomes (retention, completion) but slow to reach significance and needs traffic.
← tap to flip back
01 · Foundations
Agent harness
tap to reveal →
The scaffold that turns a model into an agent: tool wiring, memory, planning loops. Evaluation tests the harness and model together.
← tap to flip back
03 · Roadmap
Balanced problem set
tap to reveal →
An eval suite covering cases where a behavior should occur and cases where it shouldn't. Prevents one-sided over- or under-triggering.
← tap to flip back
02 · Applied
BrowseComp
tap to reveal →
Research benchmark of needle-in-haystack web questions — easy to verify once found, hard to find.
← tap to flip back
01 · Foundations02 · Applied
Capability eval
tap to reveal →
"What can this agent do well?" Starts at low pass rates and targets struggling areas. Improvement moves the score upward.
← tap to flip back
01 · Foundations
Code-based grader
tap to reveal →
A deterministic grader: string match, binary tests, static analysis, outcome or tool-call verification. Fast and cheap but brittle to valid variation.
← tap to flip back
02 · Applied
Coverage check
tap to reveal →
A grader that confirms an agent's report mentions the essential facts defined up front. Used heavily in research-agent evaluation.
← tap to flip back
03 · Roadmap
Eval framework
tap to reveal →
Off-the-shelf tooling that runs evals, manages datasets, and tracks results. Examples: Harbor, Braintrust, LangSmith, Langfuse, Arize Phoenix / AX.
← tap to flip back
03 · Roadmap
Eval saturation
tap to reveal →
When agents pass nearly every solvable task. Large capability jumps look like small score gains; time to graduate this eval and write a harder one.
← tap to flip back
03 · Roadmap
Eval-driven development
tap to reveal →
Writing the eval that defines a planned capability before the agent can pass it. Makes future-model bets visible and falsifiable.
← tap to flip back
01 · Foundations
Evaluation (eval)
tap to reveal →
A test that applies grading logic to an AI system's output to measure success. Automated evals run during development without real users.
← tap to flip back
01 · Foundations
Evaluation harness
tap to reveal →
The infrastructure that runs an evaluation suite end-to-end: spinning up environments, executing trials, aggregating scores.
← tap to flip back
01 · Foundations
Evaluation suite
tap to reveal →
A collection of tasks measuring a specific capability or behavior. Suites are the unit of work in CI and model upgrades.
← tap to flip back
01 · Foundations
Grader
tap to reveal →
Logic that scores agent performance on a task. A task can combine several graders with weighted, binary, or hybrid scoring.
← tap to flip back
02 · Applied
Groundedness check
tap to reveal →
A grader that verifies each claim in an agent's output traces back to a retrieved source. Catches confident hallucination.
← tap to flip back
01 · Foundations
Human grader
tap to reveal →
Expert or crowd judgment. The gold standard for subjective tasks, but slow and expensive. Use to calibrate LLM judges, not for every trial.
← tap to flip back
01 · Foundations
Model-based grader
tap to reveal →
LLM-as-judge grader using rubrics, NL assertions, pairwise comparison, or multi-judge consensus. Flexible and scales but needs human calibration.
← tap to flip back
01 · Foundations
Multi-turn / agent eval
tap to reveal →
Evaluation of an agent that uses tools across many turns and modifies state. Mistakes propagate, so the grader has to score the whole trajectory.
← tap to flip back
02 · Applied
OSWorld
tap to reveal →
Computer-use benchmark covering full OS control. Checks file-system state, app configs, database contents, and UI properties.
← tap to flip back
01 · Foundations
Outcome
tap to reveal →
The final state of the environment at the end of a trial — e.g. "the refund was actually processed" or "the file was created."
← tap to flip back
03 · Roadmap
Partial credit grading
tap to reveal →
Scoring multi-step tasks on a success continuum rather than pass/fail. Distinguishes "did three steps right, failed the fourth" from "failed immediately."
← tap to flip back
02 · Applied
pass@k
tap to reveal →
Probability of at least one success in k trials. Rises with k. Use when one success is enough — code gen, brainstorming, search.
← tap to flip back
02 · Applied
pass^k
tap to reveal →
Probability all k trials succeed. Falls with k. Use for customer-facing agents where consistency is essential.
← tap to flip back
03 · Roadmap
Production monitoring
tap to reveal →
Tracking metrics and errors in live systems. Ground truth on real user behavior, but reactive — problems reach users first.
← tap to flip back
01 · Foundations
Reactive debugging
tap to reveal →
The pre-eval default mode: wait for user complaints, reproduce manually, fix, hope nothing else regresses. Doesn't scale past prototyping.
← tap to flip back
03 · Roadmap
Reference solution
tap to reveal →
A known working output that passes all graders. Proves the task is solvable and verifies the graders are wired correctly.
← tap to flip back
01 · Foundations02 · Applied
Regression eval
tap to reveal →
"Does the agent still handle previous tasks?" Sits near 100% pass and protects against backsliding when prompts, models, or harnesses change.
← tap to flip back
03 · Roadmap
Robust eval harness
tap to reveal →
A harness that runs the agent production-like with a clean environment per trial — no shared state, no leftover files, no infrastructure flakiness.
← tap to flip back
02 · Applied
Simulated user
tap to reveal →
A second LLM that plays a customer persona during evaluation. Stress-tests conversational agents across extended, adversarial dialogues.
← tap to flip back
01 · Foundations
Single-turn eval
tap to reveal →
The simplest shape: one prompt, one response, one grader. Straightforward to specify and run.
← tap to flip back
02 · Applied
SWE-bench Verified
tap to reveal →
Canonical coding benchmark: real GitHub issues from Python repos, graded by the project's own tests. Frontier moved from ~40% to over 80% in a year.
← tap to flip back
03 · Roadmap
Swiss cheese model
tap to reveal →
Stacking multiple eval methods so failures slipping through one layer are caught by another. From safety engineering — no single layer is enough.
← tap to flip back
03 · Roadmap
Systematic human study
tap to reveal →
Structured grading by trained raters across many examples. Gold standard for subjective tasks; reserve for calibrating LLM judges or final-call evaluation.
← tap to flip back
01 · Foundations
Task
tap to reveal →
A single test case with defined inputs and success criteria. Also called a problem.
← tap to flip back
02 · Applied
τ-Bench / τ2-Bench
tap to reveal →
Conversational benchmark where one LLM plays a user persona and the agent navigates realistic multi-turn scenarios.
← tap to flip back
02 · Applied
Tool-choice eval
tap to reveal →
A check that the agent picked the right tool for the context — e.g. DOM extraction for Wikipedia vs screenshots for a busy shopping site.
← tap to flip back
01 · Foundations
Transcript
tap to reveal →
Complete record of a trial: outputs, tool calls, reasoning, interactions. Also called a trace or trajectory.
← tap to flip back
03 · Roadmap
Transcript review
tap to reveal →
Reading actual run records to verify what the agent did and whether graders rejected valid work. The highest-leverage habit in eval maintenance.
← tap to flip back
01 · Foundations
Trial
tap to reveal →
One attempt at a task. Multiple trials of the same task produce a success-rate distribution.
← tap to flip back
02 · Applied
WebArena
tap to reveal →
Browser-agent benchmark using URL and page-state checks plus backend verification that the intended data change actually happened.