Evals

Evals run fixed cases against an agent and record whether its turns, tools, events, and output meet a contract. Files under the project-root evals/ directory define cases with @cursor/bdk/evals; the runner discovers their IDs, executes them, and reports every assertion. See the Evals guide for the regression workflow.

Eval discovery and case IDs

Eval files live under the project-root evals/ directory and end in .eval.ts or .eval.js. The path under that directory becomes the eval ID:

• evals/readiness.eval.ts becomes readiness.
• evals/builds/api.eval.ts becomes builds/api.
• evals/builds/index.eval.ts becomes builds.

Export one defineEval with either test or cases, not both. A case ID appends one path segment to the file ID:

export default defineEval({  tags: ["smoke"],  cases: [    {      id: "checkout",      async test(t) {        await t.send("Review checkout PR 42.");        t.succeeded();      },    },    {      id: "search",      async test(t) {        await t.send("Review search PR 7.");        t.succeeded();      },    },  ],});

Those cases are prs/checkout and prs/search when the file is evals/prs.eval.ts. Case IDs must be unique single path segments.

A file can also export an array of defineEval calls. The runner names them with zero-padded indexes such as sql/0000. Named cases give handwritten scenarios stable IDs; arrays fit loaded datasets.

iterations repeats one datapoint from 1 to 100 times. For iterations: 3, weather/nyc expands to weather/nyc/1, weather/nyc/2, and weather/nyc/3; selecting weather/nyc runs all three.

Eval configuration

Every running suite needs evals/evals.config.ts with maxConcurrency:

import { defineEvalConfig } from "@cursor/bdk/evals";export default defineEvalConfig({  maxConcurrency: 20,  timeoutMs: 180_000,  judge: { model: "gpt-5.4-mini" },});

A case can override description, tags, timeoutMs, iterations, judge, reporters, and metadata. Case metadata merges over file-level metadata; case reporters add to the file's reporters.

Send turns in a case

await t.send(message, options?) runs one turn and waits until it finishes or fails. Several sends in one test share the session.

The returned turn exposes its assistant message, sessionId, events, ordered toolCalls, ok, and one-based index. Assertions on the turn inspect only that turn; assertions on t inspect the whole case. Run values include t.reply, t.events, t.turns, t.sessionId, t.iteration, t.iterations, and the timeout t.signal.

These options apply on the first send because they shape the session:

Use turn.expectOk() when later test steps depend on the turn succeeding. It throws when the turn failed.

Trajectory assertions

Assertions record failures without stopping the test, so one case reports every violated contract.

t.requireToolCall, t.requireInputRequest, and t.require return the matched value and stop the remaining test body when no match exists.

Tool and event matchers

Tool matchers can narrow by input, output, status, and count. Values accept literals, regular expressions, or predicates. Object literals partial-deep-match:

t.calledTool("inspect_pr", {  input: { verdict: "review" },  status: "completed",  count: 1,});

Tool status is completed, failed, pending, or rejected. Subagent matchers accept output, status, count, and callId. Event matchers accept data and count.

Expectation builders

Import builders from @cursor/bdk/evals:

matches accepts Standard Schema implementations such as Zod. normalizedSimilarity(value, expected) returns the same 0-to-1 value used by similarity.

Severity and verdicts

Every assertion returns a severity handle:

t.succeeded();t.calledTool("search").soft();t.check(t.reply, similarity("Expected answer")).atLeast(0.8);t.judge.closedQA("names the failing step").gate(1);

• .gate(threshold?) fails the case when it misses.
• .soft(threshold?) records a value without failing.
• .atLeast(threshold) records a soft bar; a miss produces a scored verdict.

Case verdicts are passed, failed, scored, or skipped. The CLI exits 1 for a failed gate. A scored result exits 0 unless --strict is set. Exit 2 means the selection matched no cases.

Judges

Use a judge for meaning that deterministic checks cannot express:

t.judge.factuality(expected).atLeast(0.8);t.judge.summarizes(expected).atLeast(0.8);t.judge.closedQA("names the root cause").gate(1);t.judge.sql(expected).atLeast(0.9);

Each grades t.reply by default; pass { on } to grade another value. Judge configuration can live in the project config, eval, case, or call, with the nearest value winning.

t.judge.model(prompt) returns a raw model reply for custom grading. Treat agent output as untrusted input: wrap it with fenceUntrusted and include EVAL_JUDGE_INJECTION_GUARD in the judge prompt.

Judge host

A judge grades by calling a Cursor model. It needs a Cursor credential (bdk login, CURSOR_API_KEY, or CURSOR_SERVICE_ACCOUNT_KEY) and a judge model. The grading session offers no tools and turns the local SDK sandbox off, including any per-user sandbox config on the machine, so the same case grades on macOS, a Linux cloud agent, and a Linux CI runner.

Without a credential, or when that credential belongs to a different backend than the SDK will call, the case is skipped and the suite continues. Without a judge model, the case fails and the assertion names where to set one.

Datasets and fixtures

Load committed JSON, JSONL, or YAML relative to the project root:

import { defineEval, includes } from "@cursor/bdk/evals";import { loadYaml } from "@cursor/bdk/evals/loaders";const rows = await loadYaml<  Array<{ task: string; prompt: string; expected: string }>>("evals/data/cases.yaml");export default rows.map(row =>  defineEval({    description: row.task,    async test(t) {      await t.send(row.prompt);      t.succeeded();      t.check(t.reply, includes(row.expected));    },  }));

Reporters and results

Built-in reporters include JUnit({ filePath, suiteName? }) and Artifacts({ dir }). Custom reporters can expose onRunStart, onEvalComplete, and onRunComplete.

JSON output contains run totals and one result per case. Results include the case ID, verdict, assertions, session ID, inputs, final text, tools, tool calls, metrics, logs, duration, metadata, tags, and any error or skip reason.

Select cases

bdk eval --dir . --listbdk eval --dir .bdk eval --dir . builds/checkoutbdk eval --dir . --tag smoke

ID filters match an exact ID and its descendants. Repeated IDs use OR; repeated tags use OR. When both are present, a case must match both groups.

Inspect run artifacts

Default run artifacts go under evals/<stamp>/ in the project state directory and include summary.json, results.jsonl, and evals/<case-id>.json. --artifacts <dir> chooses another destination; --no-artifacts disables them. This location is independent of --state-root.

Run evals against a server

The playground Evals view starts batches on its running server, and --url targets a named running server:

bdk eval --url http://127.0.0.1:3000/vulnerability-scanner --tag smokebdk eval status <eval-id> --url http://127.0.0.1:3000/vulnerability-scannerbdk eval cancel <eval-id> --url http://127.0.0.1:3000/vulnerability-scanner

Server-side batches use the target's discovered cases and return an eval ID for status or cancel. Local reporter and concurrency flags don't apply to --url runs.

Related

• Evals guide
• CLI reference
• Sessions
• Artifacts