# Evals

An eval sends a fixed input to the real agent and checks which tools ran and what the answer looks
like. To improve an agent and lock each win with an eval, follow [Hillclimbing](https://cursor.com/docs/agent-sdk/hillclimbing.md).

Evals run real tools, so [guard any writes](https://cursor.com/docs/agent-sdk/evals.md#keep-side-effects-out-of-eval-sessions) first.

## Protect a tool decision

The case below passes only if the agent calls the inspection tool and doesn't approve.

```ts
// evals/readiness.eval.ts
import { defineEval, includes } from "@grok/agent-sdk/evals";

export default defineEval({
  tags: ["smoke"],
  async test(t) {
    await t.send("Is https://github.com/acme/checkout/pull/42 ready to approve?");

    t.succeeded();
    t.calledTool("inspect_pr");
    t.notCalledTool("approve_pr");
    t.check(t.reply, includes(/ready|blocked|approve/i));
  },
});
```

Evals live in `evals/` beside `bot/`. Set how many cases run at once:

```ts
// evals/evals.config.ts
import { defineEvalConfig } from "@grok/agent-sdk/evals";

export default defineEvalConfig({
  maxConcurrency: 20,
});
```

```bash
agent-sdk eval --dir . --list
agent-sdk eval --dir . --tag smoke
```

## Check answer shape or quality

When the output has a contract, check its shape:

```ts
import { matches } from "@grok/agent-sdk/evals";
import { z } from "zod";

const verdict = JSON.parse(t.reply ?? "{}");
t.check(verdict, matches(z.object({ ready: z.boolean(), blockers: z.array(z.string()) })));
```

When correctness depends on meaning, use a judge. [Judges](https://cursor.com/docs/agent-sdk/reference/evals.md#judges) covers
models and grader options.

```ts
t.judge.factuality("The lint step failed on src/sidebar.ts.").atLeast(0.8);
```

## Test a conversation

Several `t.send` calls in one case share a session. An assertion on a returned turn checks only that
turn, and an assertion on `t` checks the whole conversation.

```ts
const first = await t.send("Inspect PR 42.");
first.calledTool("inspect_pr");

const followUp = await t.send("Now summarize only the blockers.");
followUp.usedNoTools();
t.check(followUp.message, includes(/blocker/i));
t.succeeded();
```

## Pin realistic inputs

Seed the files the agent reads instead of depending on your checkout:

```ts
await t.send("Review pr/diff.patch and report blockers.", {
  workspaceFiles: {
    "pr/diff.patch": [
      "diff --git a/app/routes/search.ts b/app/routes/search.ts",
      "+res.send(`<h1>Results for ${req.query.q}</h1>`);",
    ].join("\n"),
  },
});
```

Save a GitHub pull request's webhook payloads as fixtures:

```bash
agent-sdk github replay https://github.com/acme/checkout/pull/42 \
  --events '*' --dry-run --fixtures-dir fixtures/github
```

[Datasets and fixtures](https://cursor.com/docs/agent-sdk/reference/evals.md#datasets-and-fixtures) covers loaders for larger sets.

## Run and debug a regression suite

Run one case while you iterate, then the smoke set before review:

```bash
agent-sdk eval --dir . readiness
agent-sdk eval --dir . --tag smoke
```

Each run writes a folder under `evals/` in the project state directory, or under `--artifacts <dir>`
if you pass it. Open `evals/<case-id>.json` in that folder to see a failed case's input, assertions,
tool calls, and final text. Model turns need a
[Cursor credential](https://cursor.com/docs/agent-sdk/reference/cli.md#environment-variables).

## Run evals in CI

```bash
# CURSOR_API_KEY comes from the CI secret store.
agent-sdk eval --dir . --json --no-stream \
  --junit reports/evals.xml \
  --artifacts reports/evals \
  > reports/evals.json
```

Run `--tag smoke` on pull requests and the full suite on a schedule. The
[CLI reference](https://cursor.com/docs/agent-sdk/reference/cli.md#run-evals) covers every flag.

## Keep side effects out of eval sessions

Eval sessions carry `purpose: "eval"`. Check it where the tool writes, and return the decision
without the external action:

```ts
async execute({ findings }, ctx) {
  if (ctx.session.purpose === "eval") {
    return { posted: false, reason: "eval", count: findings.length };
  }

  return await postFindings(findings);
}
```

The tool call stays in the trajectory, so the eval can still assert that the agent chose to post.


---

## Sitemap

[Overview of all docs pages](/llms.txt)