Specifying done: acceptance criteria the agent can check itself against
The agent stops when the work looks done to it. Write acceptance criteria it can run and observe, plus the clause that stops it passing them by weakening them.
It stops when it thinks it's finished
Most often, people complain that the agent did about 80% of the job and then announced it was done, so we often finish with a feeling that it left some substantial part undone. But it's not that it quits early, it's that the agent stops once it thinks it's done and this is the behaviour the model demonstrates when it doesn't have a way to validate if what it did is right. In which case, the human becomes the last validation step and every bug lies there until spotted
But you know, every prompt contains an instruction of how long the work needs to be. Sometimes it's written, sometimes it's not, but there's always one, even if you just say "make it better" — in this case, it's like "make it better, I'll tell you when it's good". But if you don't specify that instruction, the model creates its own and this is what you can find out about once you compare the diff
We know it's all very obvious, we've been writing specs for twenty years, but you see the thing is — a specification assumes a human will eventually look. The agent spec needs to be verifiable by the agent during the session while the session is still in progress
So if we say "it'd be good if the error message was more clear", it's a great ticket description, a terrible instruction of how long the work should go on. There's no way for the agent to validate this
Checkable, reportable, and the gap between them
To make it work, we need to come up with criteria and place them into three buckets before writing the prompt:
The first bucket is about the things that you can run which exit with a code, like npm test or cargo build or npx tsc --noEmit or curl -sf localhost:8080/health. These are the things that the agent can actually check and they're also the only type of things for which we don't need a human to make sure it's really done
The second bucket is about the things that emit some output which the agent can read and compare, like running diff against specific paths or a grep that returns nothing or hitting an endpoint and verifying the status code is 201 or comparing a screenshot with a figma. These are also valid, but the thing is — they require an interpretation step from the model so it's not the perfect solution
The third bucket is about everything that can be checked only by a human, like how well-abstracted the solution is or how pleasant the API is to use or if this is what the team wanted. These are things that can't be expressed as a command
The way we define what's done is that we move things from the third bucket to the first one. And everything that's left in the third bucket is what we need to instruct the model about, otherwise it will come up with its own solution and say it's done
This is actually outlined by OpenAI too, in the Codex plan docs, they write:
"Acceptance should be phrased as behavior a human can verify ('after starting the server, navigating to http://localhost:8080/health returns HTTP 200 with body OK') rather than internal attributes ('added a HealthCheck struct')"
If you think about it, an internal attribute is sth that exists. A behaviour is sth that a system does
So if you have a truly internal change, show its impact by making some tests failing before and passing after
The done block
In a matter of a minute, you can create a completion block which looks like this:
What needs to be achieved — one sentence, a human-understandable behaviour
Checks — these are the actual commands with the expected output or return value
Protected surface — things that can't change
Exclusions — things that you will be personally evaluating
And then you compare:
This is what people usually write:
Add rate limiting to the API and make sure it works.And this is the same task, but with the completion criteria:
Add per-IP rate limiting to the public API.
Done when:
- npm test exits 0
- a new test in test/rate-limit.test.ts sends 101 requests inside a minute and
asserts the 101st response is 429
- npx tsc --noEmit is clean
Don't change:
- any existing file under test/
- anything under src/auth/
Out of scope: per-user quotas and the admin routes. Whether 100/min is the right
number is my call, not yours — don't tune it.Like for example per-IP rate-limiting on a public API. Npm test exits with 0. We create a file under tests and in it we send 101 requests to the API within a minute and assert that the status code of the 101st request is 429. Npx tsc --noEmit runs with no errors. Everything under test/ is off limits, everything under src/auth/ is off limits too. Per-user quotas and admin routes are excluded as well. Whether 100 reqs per minute is the right number is my call, not yours — don't tune it
This is almost 7 times longer, but still the cheapest minute of the session. Every line is a round-trip we won't need to make later just to find out the agent had a different understanding of what we wrote
The third part of this completion block, the protected surface, is not actually a courtesy. It's serious business and we'll show it in a second
Every check can be passed by weakening the check
You see, once you define what "done" means as a test, getting the test to pass becomes the goal. And there are two ways to make a test pass:
Do the thing
Make the test worse
By using skip markers or .only isolation or by relaxing assertions or adding @ts-ignore. Or by putting it all into a try block or by hardcoding the return value to what the test expects
It's not that the agent is lazy or tries to cheat, it just reads the instruction literally. Anthropic actually has tests for this, and system cards show trends of "hackable"-ness with regard to these criteria across releases
But that's not really about the model, it's about the setup as the setup is what we can change. If you want a criterion that survives many turns, according to the Claude Code goal-condition docs, it needs three things:
A measurable target state
An explicit check
Constraints that matter — what can't change, like not modifying any other test file
And these are the constraints which people often omit. Git actually has a lot of first-bucket criteria, let us show you
So we can pin the base commit to the current state of the repo and then run git diff --name-only --diff-filter=MD against test directory to see if there's anything modified or deleted and pipe the diff into grep -E '^+.*(.skip(|.only(|@ts-ignore|eslint-disable)' without -q — both should print nothing
# start of session: pin what "before" means
export DONE_BASE=$(git rev-parse HEAD)
# the checks must not have been made easier to pass
git diff --name-only --diff-filter=MD "$DONE_BASE" -- test/
git diff "$DONE_BASE" | grep -E '^\+.*(\.skip\(|\.only\(|@ts-ignore|eslint-disable)'And this is important because diffing against HEAD only sees uncommitted changes — a smart agent that commits its tampering can get past it. We tried, it says it's clean
And also — don't use grep -q with the pipe to diff as it exits after the first match and sends SIGPIPE to diff which under pipefail becomes 0 if diff is big enough for git to still be writing
Collapse it into one command
These are all things that can be expressed as a script. If you have multiple commands in a repo and a few guardrails, you end up with an entangled prompt, but if you create a script, the completion block boils down to this:
Run this script until it exits with 0
Don't change the script
Here's the script itself:
It runs the acceptance checks and then the guardrails
It prints either pass or fail for every line
And it also prints what each check found so the agent knows what to do
It exits with non-zero if any of the checks failed
And this is the part that we edit, four commands, the directories and the base commit
Let's run it against a repo in which the agent "fixed" a test by introducing a skip marker and suppressing a type error:
The acceptance checks passed
All the guardrails failed
And they returned not only the file but also the line that was added
Whereas if we were to just run npm test, it would have said everything's good
And here are three tamper scenarios which the naive version of this script won't be able to handle:
It commits instead of leaving it in the working tree
It creates a new file under tests and introduces the changes there
And the most subversive one — it introduces the changes in a 600-thousand-lines-long diff
But with this script, you have a standardised way to express completion criteria across all the harnesses. You can run it locally or headlessly in CI, and you define what "done" means in one place rather than scattering it across everybody's prompts
Ask for the criteria back, then for evidence
But ofc there's still a thing which we can't ask a script about — the things that go to the third bucket. So we need to include them in the prompt
What's left is two very cheap habits:
Ask the agent to list the checks it'll run and the expected output of each before making any changes
And ask for evidence instead of an opinion
Whenever you see this: we can be told that all the tests pass, and types are clean, but we have no idea if the agent actually run anything
Ask it to show you. It's really important to ask for the evidence — looking at the output takes us 10 seconds, it's faster than running the suite yourself and it's very valuable when it comes to sessions nobody watched
"Done when" is also "stop when"
But criteria can also bound the work from above: if an agent finished the task and then refactored three adjacent files, it didn't overdo it — it just had a goal without an upper limit and kept finding things that could be part of its solution
By setting an explicit completion criterion we basically say "it's okay to stop here"
And this weighs more the less you watch, as with explicit criteria several harnesses will re-check them after every turn rather than giving control back to you — and that's the part where the mechanism differs between them
Setting a condition in Claude Code
Normally there are two places where you can define criteria for a turn:
persistent ones, under the command(s) they apply to in
CLAUDE.mdone-time ones, in the prompt itself
But thanks to Claude Code it can be also used for another purpose — automatic verification of whether some condition is met — via /goal. Let's say for instance you'd like to make sure that tests are passing in test/auth, there are no errors after typecheck and you haven't modified anything except src/auth/ and test/auth/. That could look sth like this:
/goal all tests in test/auth pass, npx tsc --noEmit is clean, and no file
outside src/auth/ or test/auth/ is modifiedOnce you run it, a new turn is being created and the condition is being treated as instruction. Then, at the end of every turn the compact quick model examines the chat transcript and decides if this condition has been met. If it hasn't, it tells Claude why and what she should do in another turn instead of returning control to you.
Here's how it works:
/goal— to check the status, turns spent and the evaluator's last reason/goal clear— to finish it
The condition can contain up to 4000 characters. It's also worth to familiarise yourself with the official description, as it lists three things that are worth to keep in mind when working with goals:
single, measurable "finish line"
explicit verification step
constraints that matter
So make sure you have Claude Code version 2.1.139 (or newer; we've been using 2.1.193).
The most important thing is that the evaluator is not a bot, so it doesn't run any commands and doesn't open any files on its own — it's only basing its decision on what Claude has already written in the chat. That means that if she never shows sth, it can't be confirmed — a condition like "the code is clean now" will simply never resolve. And the sharper edge is worth thinking through yourself: if the transcript is everything the evaluator sees, then "all tests pass" typed by Claude has exactly the same shape as the actual output of the suite. So write the condition so the proof has to land in the chat rather than the summary — "all tests in test/auth pass" works precisely because running them puts the result there for the evaluator to read.
But there's another thing to keep in mind: goals don't modify the permissions, so by default it will still ask you if you want to run the command. If you'd like to change that and would like to use this feature in a fully autonomous mode, make sure to also enable auto mode.
Alternatively, you can enforce it using Stop hook — which basically just runs your script and doesn't allow the turn to finish until it does. But remember that with Claude Code it's limited to 8 consecutive blocks, after that it will automatically finish the turn. We'll have a separate chapter about hooks later in the course.
Making the report machine-readable in Codex CLI
Persistent requirements are defined in AGENTS.md, these are loaded by Codex before it does anything. What's unique about Codex is how it outputs things, not what it gets in, so you can actually define the structure of its output for every run like:
codex exec --output-schema done.schema.json \
-o result.json \
"implement per-IP rate limiting; run the checks in scripts/done.sh"If you run codex exec --help, you'll see that --output-schema takes a path to a JSON Schema that describes the structure of the model's last response. We recommend defining it like this:
{
"type": "object",
"additionalProperties": false,
"required": ["build", "tests", "types", "untouched_paths", "evidence"],
"properties": {
"build": { "type": "boolean" },
"tests": { "type": "boolean" },
"types": { "type": "boolean" },
"untouched_paths": { "type": "boolean" },
"evidence": { "type": "string" }
}
}That means a separate boolean property for each of the criterias and proof as a string. Thanks to that, whoever uses the output of Codex will be parsing actual data, not text. It's a good idea to make all these properties required and the object closed — so it's not possible to add any other properties.
The schema is read before the run starts, if you don't point to an existing file with it, you get an error saying sth like Failed to read output schema file <path>: No such file or directory (os error 2), so you don't lose a session.
The -o / --output-last-message flag, as you can see, is also about pointing to a path where the last message will be stored. What's important to know is that the schema defines only the structure of the claim, not if it's true.
It's still possible for the model to return true for any of these booleans without doing anything, so make sure to include exit codes in your loop and keep using them. The schema is about the convenience of consuming the output, not verifying the work. That's why it's important to make the proof field required as well, and actually require a real command output there, so that if sb lies — they need to be specific.
Where done lives in GitHub Copilot CLI
Let's start with copilot init — it reads the repository using read-only tools and creates .github/copilot-instructions.md file. As you can see in the tool's docs, that file describes the configuration of a lot of aspects: the build and test commands, coding conventions, project structure, tech stack etc, which basically means that this is almost a stand-alone done block. There are just two things it can't know:
which commands need to pass for the work to be considered as done
which paths you don't want to allow
So you need to define them manually. What's more, it also reads AGENTS.md file, and by running copilot --no-custom-instructions you can make sure that it doesn't load custom instructions from AGENTS.md and similar files — this way you can easily check if a specific rule comes from there.
The autopilot is the Copilot-specific thing here as it affects the price of an unverifiable criterion, like this:
copilot --mode autopilot --allow-all-tools \
-p "implement per-IP rate limiting; run ./scripts/done.sh until it exits 0"--max-autopilot-continues — as you can see in the docs, that's the maximum number of continuation messages in autopilot mode and it defaults to 5; for non-verifiable criterions that means it makes 5 guesses (charging for all of them). For verifiable criterions, that would mean 5 cycles of examining the error output and modifying the code.
--allow-all-toolsmust be used in non-interactive mode, so it's there; that means-pruns are no-click ones, so that's when the "must not change" clause stops being optionalIn Copilot, the repo-wide and path-scoped instructions are accumulated, not overwritten. As the docs say, if there's a path-scoped file and it points to the same file that is being worked on and there's also a repository-wide file — both of their instructions are applied in the
doneblock; in such scenario, their content gets merged instead of being arbitrated, so it's good to keep it in a single file
Feeding failures back in Cursor
Where are project's persistent requirements stored? In .cursor/rules files or AGENTS.md file at the root, used by Cursor as a replacement for rules file and applied to its own directory as well as subdirectories.
How it works. The stop hook is what's specific to Cursor; docs describe it as an event that is being dispatched when the agent loop has finished, with an option of automatically sending a follow-up user message in order to initiate another iteration. Hooks are defined in the project's .cursor/hooks.json file at the root, and we need to point the stop one to a script which will contain checks.
{
"version": 1,
"hooks": {
"stop": [{ "command": "./.cursor/check-done.sh" }]
}
}This script is being run and it runs the checks — if they fail it outputs the failure as the next prompt so the requirements are not just waiting for someone to act on them but rather are being worked on by an agent whenever they are not met.
#!/usr/bin/env bash
if out="$(./scripts/done.sh 2>&1)"; then
echo '{}'
else
printf '{"followup_message": %s}' "$(printf '%s' "$out" | tail -15 | python3 -c 'import json,sys; print(json.dumps(sys.stdin.read()))')"
fiSame applies to cloud agents too which also read the project's .cursor/hooks.json file, so you can define everything in one place.
If there's sth you should keep in mind it's that:
By default this solution assumes up to 5 automatic follow-ups per script; this number can be configured using
loop_limit. Just know that such a limit is rather informational than limiting — if you need more than 5 runs it means either the task is too big or the checks are not precise enough to act on.Another thing is that the docs outline which events are supported for the editor agent but doesn't cover this topic for
cursor-agentso we don't know which of them it actually emits. Our advice is to have the hook create a file in its first run so you can see if it runs when using any of these surfaces.
Enforcing the shape in Antigravity CLI
If you need to make criteria required for the Antigravity CLI you need to define them under AGENTS.md or GEMINI.md — agy reads both names. In case your repo has both of these files it has both of them in play. To use them in your code make sure to require the appropriate file in a completion block, do not treat one as a fallback for another.
If you run Antigravity from code using agy you can define a schema for the machine to enforce when it comes to the final answer that it returns. For example:
agy -p "implement per-IP rate limiting; run ./scripts/done.sh until it exits 0" \
--output-format json \
--json-schema ./done.schema.json--json-schema — an optional parameter pointing to a JSON schema (string or path) that can be used to enforce structure of the output; you can use either in-line string or path to a file in this case.
--output-format — an output format; you can use json or stream-json.
That way in headless mode the end of every run is branchable thanks to having a single boolean field per criterion and an evidence field.
Just remember that this flag is useless on its own, and this is not mentioned in the docs. If you try to use it without setting --output-format to json or stream-json you will get an error saying: Error: --json-schema can only be used when --output-format is 'json' or 'stream-json'. Also keep in mind that with stream-json the schema is only responsible for the final result, not the intermediate events. This means that you can't be sure what the machine will tell you once it starts parsing the stream to show you its progress.
If you want to make runs more "blocking" (not reporting) Antigravity comes with hooks. You can configure them using a hooks.json file in .agents/ directory or in the global configuration, or ship them as part of your plugin. Remember that it's good to check what a particular hook is responsible for before you rely on it to wait for something, especially to wait for the end of the turn. The official examples show only pre- and post-tool hooks, not end-of-turn ones.
Blocking the turn in Kimi Code CLI
The persistent rules should be kept in AGENTS.md but Kimi Code has a nice addition - the Stop hook which can be blocked. In docs it's being said that the Stop hook is called before the LLM finishes the current turn so by stopping it you can attach an additional message and make it continue the turn. In other words, it's a lighthouse to enforce a particular definition of done in the most crucial moment.
You configure the hooks as an array of hooks in ~/.kimi-code/config.toml:
event required
command required
matcher optional
timeout optional
For example, here's how the Stop hook looks like:
[[hooks]]
event = "Stop"
command = "./scripts/done.sh 1>&2 || exit 2"
timeout = 300It runs a done script which outputs something to stderr (which is redirected in the configuration above) and terminates with the code of 2 if anything goes wrong, with a timeout of 300 seconds. The behaviour depends on the exit code:
0 — lets the turn finish
2 — stops it
In the latter case whatever was printed to the stderr by the script is returned to the LLM as an explanation. So basically, by piping the output to stderr and using the exitcode of 2 for the done script we tell Kimi Code to return the output of this script as its failed checks so then it tries to complete them.
But that's the thing, these hooks are set to allow by default (which is sth you need to be aware of), in docs it says that every exit code different than 2 and every error or timeout is treated as allowing the turn to continue. So if for example the done script exits with the code of 1 on a failed check or with 127 because you don't have some dependency installed, the turn will end and the run will look like it succeeded.
That's why you should always append this || exit 2 part at the end of the command so no matter what goes wrong during its run it returns the code of 2 which is stopping the turn and proceeding with the output to stderr. Don't rely on a script's native exit code, always use this construct.
Another thing that might be problematic is that the timeouts are set to 30s by default (with an upper limit of 600s) so any serious e2e test suite will be cut short and also the timeouts will default to allow. That's why for longer suites you should always make sure to set a proper timeout value in the config, and if it's a real e2e test suite which takes longer than 10 minutes to run, you might want to make it return early from the hook running a few tests and delegate the actual run to CI.
The habit to take away
So before you hit enter, think if the agent could know it's done without you. And if not — it means that you are the check, sometimes it's okay but it's good to be aware of it rather than realising it 40 minutes later and seeing a diff you don't like
scripts/done.sh#!/usr/bin/env bash
# scripts/done.sh — the one command that answers "am I finished?"
#
# Point the agent at this instead of listing criteria in every prompt:
# "Run ./scripts/done.sh until it exits 0. Do not edit this file."
#
# Exits 0 only when every check passes. Every failing check prints what it
# found, so the agent has something to act on rather than just a red line.
set -uo pipefail
# --- edit this block for your repo ------------------------------------------
BUILD_CMD="npm run build"
TEST_CMD="npm test"
LINT_CMD="npm run lint"
TYPES_CMD="npx tsc --noEmit"
# Paths that hold the checks themselves. Editing these changes the grading, so
# the guardrails below watch them.
TEST_PATHS="test tests spec"
# What "before" means for the guardrails. HEAD only covers uncommitted work, so
# export the starting commit before you hand the task over — otherwise an agent
# that commits its own tampering walks straight through:
# export DONE_BASE=$(git rev-parse HEAD)
BASE="${DONE_BASE:-HEAD}"
# ---------------------------------------------------------------------------
FAILED=0
check() {
local name="$1" cmd="$2" output
if output="$(eval "$cmd" 2>&1)"; then
printf 'PASS %s\n' "$name"
else
printf 'FAIL %s\n' "$name"
[ -n "$output" ] && printf '%s\n' "$output" | tail -n 15 | sed 's/^/ | /'
FAILED=1
fi
}
echo "== acceptance =="
check "build" "$BUILD_CMD"
check "tests" "$TEST_CMD"
check "lint" "$LINT_CMD"
check "types" "$TYPES_CMD"
# Every line the agent added — committed, staged, unstaged and untracked — in
# one file. Never pipe a diff into `grep -q`: grep exits on the first match,
# git takes SIGPIPE, and under `pipefail` the guardrail silently passes.
ADDED="$(mktemp)"
trap 'rm -f "$ADDED"' EXIT
git diff "$BASE" | grep '^+' > "$ADDED"
while IFS= read -r f; do
[ -f "$f" ] && sed 's/^/+/' "$f" >> "$ADDED"
done < <(git ls-files --others --exclude-standard)
echo
echo "== guardrails =="
# Every check above can be passed by weakening the check. These look for that.
check "no existing test file edited or deleted" \
"! git diff --name-only --diff-filter=MD '$BASE' -- $TEST_PATHS | grep ."
check "no tests skipped or narrowed" \
"! grep -nE '^\+.*(\.skip\(|\.only\(|xit\(|xdescribe\(|@pytest\.mark\.skip|#\[ignore\])' '$ADDED'"
check "no errors suppressed" \
"! grep -nE '^\+.*(@ts-ignore|@ts-nocheck|eslint-disable|# type: ignore|#\[allow\(|// *nolint)' '$ADDED'"
echo
if [ "$FAILED" -eq 0 ]; then
echo "done: all checks pass"
else
echo "not done: fix the FAIL lines above and run this again"
fi
exit "$FAILED"