Skip to main content

Command Palette

Search for a command to run...

BDK

Evals

An eval sends a fixed input to the real agent and checks the resulting trajectory: whether the turn succeeded, which tools ran, and what shape the answer took. Use evals to protect behavior you already understand, not to discover what the prompt should do.

Evals run real model turns and real tools. Guard external writes before running a suite against an agent that can post, merge, or deploy.

Protect a tool decision

This smoke case asks for a PR verdict, requires the read-only inspection tool, and fails if the agent tries to approve. The CLI reports all three decisions together instead of stopping at the first miss.

Use the Jev extension when host code needs a typed choice, score, or boolean before it writes.

// evals/readiness.eval.tsimport { defineEval, includes } from "@cursor/bdk/evals";export default defineEval({  tags: ["smoke"],  async test(t) {    await t.send(      "Is https://github.com/acme/checkout/pull/42 ready to approve?"    );    t.succeeded();    t.calledTool("inspect_pr");    t.notCalledTool("approve_pr");    t.check(t.reply, includes(/ready|blocked|approve/i));  },});

Evals live in the project-root evals/ directory, beside bot/. Add the required concurrency config once:

// evals/evals.config.tsimport { defineEvalConfig } from "@cursor/bdk/evals";export default defineEvalConfig({  maxConcurrency: 20,});
bdk eval --dir . --listbdk eval --dir . --tag smoke

Check answer shape or quality

Prefer a deterministic shape check when the output has a contract. The failure tells you which field was wrong, and the case remains stable when the model changes its wording.

import { matches } from "@cursor/bdk/evals";import { z } from "zod";const verdict = JSON.parse(t.reply ?? "{}");t.check(  verdict,  matches(    z.object({      ready: z.boolean(),      blockers: z.array(z.string()),    })  ));

Use a judge when correctness depends on meaning that a schema, regex, or known value cannot capture:

t.judge  .factuality("The lint step failed on src/sidebar.ts.")  .atLeast(0.8);

Judge checks are tracked scores until you give them a hard gate. Configure the judge model in evals.config.ts. The Evals reference owns grader options and severity rules.

Test a conversation

Several t.send calls in one case share a session. Assertions on a returned turn inspect only that turn, while assertions on t inspect the whole conversation.

const first = await t.send("Inspect PR 42.");first.calledTool("inspect_pr");const followUp = await t.send("Now summarize only the blockers.");followUp.usedNoTools();t.check(followUp.message, includes(/blocker/i));t.succeeded();

Pin realistic inputs

A useful eval changes only when the agent changes. Freeze canonical chat prompts, seed workspace evidence directly, and save webhook payloads instead of depending on a developer's checkout or live external state.

await t.send("Review pr/diff.patch and report blockers.", {  workspaceFiles: {    "pr/diff.patch": [      "diff --git a/app/routes/search.ts b/app/routes/search.ts",      "+res.send(`<h1>Results for ${req.query.q}</h1>`);",    ].join("\n"),  },});

For GitHub agents, snapshot a PR into committed fixtures:

bdk github replay acme/checkout#42 \  --events '*' \  --dry-run \  --out fixtures/github

Keep a small, fast smoke set for every change. Put larger datasets and drift-prone live cases behind an explicit selection. See Datasets and fixtures for loaders and case expansion.

Run and debug a regression suite

Run one case while iterating, then the smoke suite before review. A failure artifact records the input, assertions, tool calls, and final text, making it the first place to look before changing the prompt.

bdk eval --dir . readinessbdk eval --dir . --tag smokebdk eval --dir . --json

By default, results land under evals/<stamp>/ in the project state directory. Open evals/<case-id>.json inside that run directory for the failed case. This artifact location is independent of --state-root; pass --artifacts <dir> to choose another destination.

Model turns need a credential. Resolution checks CURSOR_API_KEY, CURSOR_API_KEY_FILE, CURSOR_SERVICE_ACCOUNT_KEY, then a saved bdk login. eval --list only discovers cases and needs none.

Run evals in CI

Write machine-readable output and JUnit annotations, then let a failed gate fail the job:

# CURSOR_API_KEY comes from the CI secret store.bdk eval --dir . --json --no-stream \  --junit reports/evals.xml \  --artifacts reports/evals \  > reports/evals.json

Run --tag smoke on pull requests and the full suite on a schedule when cost or latency makes every-push coverage impractical. Use soft scores for a new probabilistic benchmark until its threshold is trustworthy.

The CLI reference covers selectors, server-side batches, exit codes, and JSON output.

Keep side effects out of eval sessions

Eval sessions carry purpose: "eval". Check it at the deterministic write boundary and return the decision without performing the external action:

async execute({ findings }, ctx) {  if (ctx.session.purpose === "eval") {    return {      posted: false,      reason: "eval",      count: findings.length,    };  }  return await postFindings(findings);}

This preserves the tool call in the trajectory, so the eval can still assert that the agent chose to post.

Evals, hooks, or hillclimbing?

Use an eval to gate one fixed input. Use a hook to observe every live session for metrics, auditing, or alerts. Use hillclimbing to decide which source change improves a fixed set of inputs; every kept hillclimb change should add an eval.

Gates and scores

t.succeeded(), tool assertions, and deterministic checks are hard gates by default. A failed gate fails the case. .soft() records a measurement without blocking, while .atLeast(threshold) records a score and marks a miss as scored.

Start with decisions and output shape: the intended tool, the tempting wrong tool, and one structural check. Exact prose is rarely a stable contract.

Keep improvements with regression evals

Every kept improvement needs an eval that would have failed before the change. A tool-choice gate, bounded maxToolCalls, or output-shape check turns the improvement into a durable contract.

Never weaken an existing gate to make a new implementation pass. That moves the freeze line instead of proving the change.

Related