Skip to main content

Command Palette

Search for a command to run...

BDK

Hillclimbing

Hillclimbing improves an agent through controlled before-and-after comparisons. Keep the inputs fixed, name one failure, change one lever, and retain the change only when the same run gets better without crossing the freeze line.

The companion hillclimb skill can run this loop with you.

Correct a wrong tool choice

Suppose a PR reviewer calls approve_pr before inspect_pr. Pin one PR fixture, record the original tool order, then tighten the instruction that owns the decision. The improved run should inspect first and leave approval untouched.

bdk run --dir . \  --message "Is https://github.com/acme/checkout/pull/42 ready?"

Lock the result with the intended and tempting-wrong paths:

t.succeeded();t.calledTool("inspect_pr");t.notCalledTool("approve_pr");

Do not change the tool implementation, prompt, and evidence together. If tool choice improves, you should know which edit caused it.

Reduce wandering or latency

A slow run often repeats reads, searches broad directories, or asks the model to discover evidence the host already has. Compare the trajectory by tool name and wall time, then move deterministic preparation ahead of the turn or trim the evidence shape.

before: 14 tool calls, 92 seconds, 6 repeated readsafter:   5 tool calls, 31 seconds, 0 repeated reads

Keep the change only if the answer still finds the same known issue. Efficiency without a quality check can make an agent fast by making it incomplete.

Improve quality without changing the contract

For answer quality, freeze the public shape before editing: required sections, API fields, side effects, and existing evals. Compare both runs against a known-good answer or a small set of gold findings, then change the instructions or skill that owns the missing judgment.

fixture: saved checkout PR diffsuccess: report both gold findings with file pathsfreeze:  findings JSON shape; no review post; existing smoke evals

A wording change is not a win by itself. Keep it when correctness rises and every frozen behavior remains intact.

Run one measured round

Agree on four inputs before editing:

  1. Target: the agent path or slug.
  2. Fixtures: one to three prompts, saved payloads, or pinned PRs.
  3. Success: the metric or decision that should improve.
  4. Freeze line: behavior that must not move.

Then run one loop:

  1. Exercise the agent through the same surface a user would.
  2. Read the trajectory and name the dominant failure in one sentence.
  3. Change one lever.
  4. Rerun the same fixtures.
  5. Compare correctness, efficiency, and contract fit.
  6. Keep, revert, or narrow the change.

Do not replace a fixture mid-round. A moving input makes the comparison meaningless.

Choose the smallest lever

Start with the earliest place that can remove the failure:

  1. Host preparation: fetch deterministic evidence before the turn.
  2. Evidence shape: trim, order, or label what the model receives.
  3. Instructions or skills: clarify the decision procedure.
  4. Tool surface: remove or gate tools that invite wandering.
  5. Framework: change it only when the agent cannot express the fix.

Changing the host or evidence often improves both quality and latency because the model spends less time searching for ground truth.

Lock every kept win

Each retained change needs an eval that would have failed before it. Use a tool-choice assertion, maxToolCalls bound, output-shape check, or quality score that captures the measured improvement.

bdk eval --dir . --tag smoke

Never weaken an existing gate to make the round pass. The eval is the record that the improvement survived after the fixture and source change leave your working memory.

Run the loop with a coding agent

Ask the coding agent to read skills/hillclimb/SKILL.md, then give it the target, fixtures, success criterion, and freeze line. It will measure the current run, propose one change, remeasure, and ask before starting another round.

Use saved or local surfaces during a climb. Do not deploy or post to a real external channel unless that side effect is part of an explicitly approved fixture.

Related