How evaluation actually looks for coding, conversational, research, and computer-use agents — and how to think about non-determinism when the same agent runs many times.
Coding agents write, test, and debug code. Of the four agent types this is the most well-trodden ground because software evaluation is unusually clean: does the code run, and does it pass tests?
The canonical benchmark is SWE-bench Verified — GitHub issues from Python repositories, graded by the project's own test suite. A solution passes only if it fixes failing tests without breaking existing ones. The frontier moved from ~40% to over 80% in one year.
A production-grade coding eval typically stacks: deterministic_tests for required unit tests, static_analysis for lint/type/security, state_check for side effects (e.g. "did the security log entry get written?"), tool_calls verification, and a final llm_rubric on code quality. Tracked metrics include turns, tool calls, total tokens, and latency.
Conversational agents work in support, sales, coaching, and similar domains. Unlike chatbots, they maintain state, use tools, and take mid-conversation actions like processing refunds or filing tickets.
Success here is multidimensional: did the ticket actually get resolved (state check)? Did the agent finish in under 10 turns (transcript constraint)? Was the tone appropriate (LLM rubric)? Each dimension uses a different grader type.
The trick most teams miss is the simulated user. A second LLM plays a customer persona, stress-testing the agent across extended, sometimes adversarial conversations. τ-Bench and τ2-Bench formalize this, simulating multi-turn interactions in realistic scenarios.
A typical conversational task layers llm_rubric assertions on empathy and clarity, state_check on backend outcomes, tool_calls required (verify identity → process refund → send confirmation), and a max_turns ceiling on the transcript.
Research agents gather, synthesize, and analyze information, producing answers or reports. Unlike coding, there's no unit test to lean on — quality depends entirely on context. Market scans, due-diligence reports, and scientific syntheses each demand different standards.
Three problems make this hard. Experts disagree on what counts as comprehensive. Ground truth shifts as the web changes underneath you. Longer outputs create more opportunities for hidden mistakes.
The benchmark to know is BrowseComp — needle-in-a-haystack questions across the open web, easy to verify once found but hard to find. The grading strategy stacks layers:
Computer-use agents drive software through human interfaces — screenshots, clicks, keyboard, scrolling — rather than APIs. They work with anything that has a GUI, from design tools to legacy enterprise software.
Evaluation needs a real or sandboxed environment to check intended outcomes. WebArena tests browser tasks with URL and page-state checks, plus backend verification that the intended data modification actually happened. OSWorld extends this to full OS control, checking file-system state, application configs, database contents, and UI properties.
Browser agents face a tokens-vs-latency trade-off worth understanding. DOM-based interactions are fast but token-heavy. Screenshot-based interactions are slower but token-efficient. Summarizing a Wikipedia article benefits from DOM text extraction; finding a laptop case on Amazon benefits from screenshots, because the full DOM is huge.
Agent behavior varies between runs. Each task has its own success rate. What you actually measure is how often the agent succeeds — and there are two ways to summarize that.
pass@k measures the probability of at least one success in k attempts. As k grows, pass@k rises. A 50% pass@1 means the model gets it right half the time on the first try. In coding, first-try success matters most. In other situations, multiple proposals are fine if any one of them works.
pass^k measures the probability that all k trials succeed. As k grows, pass^k falls — demanding consistency across more trials is harder. A 75% per-trial success rate over 3 trials is (0.75)³ ≈ 42%. This is the metric you want for customer-facing agents where every call has to work.
Tap any card to reveal the definition. Or open the full deck to filter and shuffle across all guides.