When it fails at 3am

In Claude Code

We would say that JSON mode prints single object to stdout containing 21 properties, it's the most descriptive one if you need some diagnostics. The properties are: is_error, subtype, stop_reason, terminal_reason, api_error_status, num_turns, total_cost_usd, session_id, permission_denials. The latter is an empty array in both scenarios and it's good to check during a late-night incident because it's the first thing worth eliminating (that permission was denied). Here's the example of a passing run: subtype "success", terminal_reason "completed", stop_reason "end_turn", api_error_status null.

And here's a run with the budget set to 0.001:

{ "subtype": "error_max_budget_usd", "is_error": true, "stop_reason": "tool_use",
  "terminal_reason": "budget_exhausted", "num_turns": 1, "result": null,
  "total_cost_usd": 0.0221338, "session_id": "fbe95f70-ff8f-433b-98fa-9e444eadd5ae" }

A few things to note:

• null result means the run prints null and exits 0 when you extract the .result property using jq. It means it's a string, not an error.

• The recorded cost is 0.022 vs 0.001, it's because the limit is being evaluated after getting a response, so once the response is back the money is already spent.

Regarding the session, there's a transcript under ~/.claude/projects/{working-directory-with-slashes-replaced-by-hyphens}/, with the name of the file being the same as the session_id and having .jsonl extension in the end. For instance, if your directory is /private/tmp/checkout/repo, it will be here: ~/.claude/projects/-private-tmp-checkout-repo/

In case of a failing run, there's still a transcript (it's 23kB long, next to 22kB from the passing run). And there are two useful flags:

• --session-id — to enforce a specific uuid upfront so it's easier to predict where to find the transcript

• --debug-file — to point claude where to put the debug log so you can easily upload it

Also, if you're using this in CI, don't use the --no-session-persistence flag, it even says in its help that sessions are not being written to the disk and therefore cannot be resumed — and resuming is exactly what you want in a 3am scenario.

The last thing is that if you try to run claude in headless mode with print command, it won't say anything if you put some invalid settings in .claude/settings.json. It just exits with 0 and the is_error property being false (and the stderr empty), and its help text says that it ignores files that are not properly set up.

But if you run claude doctor in the directory, it will point out that file and the problem — but it exits 0 whether it found anything or not, so gate on grepping its output, never on its status.

In Codex CLI

If you run codex like this:

codex exec --json

it outputs JSONL stream, where the first line is with run ID and the last one is with result. For example:

{"type":"thread.started","thread_id":"019fcec7-653f-7a82-9a11-22b843c40d0c"}
{"type":"turn.failed","error":{"message":"unexpected status 401 Unauthorized: ..."}}

The same thread_id is embedded in the transcript file path, which is located at $CODEX_HOME/sessions/YYYY/MM/DD/rollout-<timestamp>-<thread-id>.jsonl, so you can just search for this events to find a file to attach.

Also, even if the run fails, like with 401 above, rollout still emits a transcript for it, just in case.

And then, if you want to include the last message of the agent into the alert, you can use -o <FILE> option, as this is the cheapest thing to attach.

There are two ways to make sure that a run will report the failure properly. The first one is to pass --strict-config flag, which won't let the app to start if there's a typo in a config key, for example:

Error loading config.toml:
config.toml:2:1: unknown configuration field `modle_reasoning_effort`
  |
2 | modle_reasoning_effort = "high"
  | ^^^^^^^^^^^^^^^^^^^^^^

It fails with this message, pointing at the exact line and column of the unknown field. And it returns exit code 1, which means that the run won't even get to making a model request, so it's free in terms of costs.

If you don't pass this flag, the very same file will be processed without any complaints, and the app will continue its work, with the setting being just not set. Which is not a great way of handling such a scenario.

The other option is --ephemeral flag which runs the code without creating sessions files (which is what rollout does by default). That's exactly what we don't want in case there's even a chance that the run will need to be reconstructed at some point.

Also, there are few gotchas:

If you provide prompt via argument but leave stdin open, it won't work. This is because as per its own help message piped stdin is attached as a <stdin> block, so it waits for EOF. But we don't send one, so it blocks. We tried it and there was no output on stdout for 2 minutes, until we killed the process. If we ran codex exec --json < /dev/null it took 20 seconds to finish.

Stderr even outputs message about reading extra input from stdin during a normal run, so this won't help us with identifying such a case.

Another thing is that if it receives 401, the app tries to request 5 times via WebSockets, and then via HTTPS, and then 5 more times. In total it spends 20 seconds on this which is not optimal, so make sure you set timeout accordingly to include the entire sequence.

And the last thing, codex resume --last won't work for non-interactive sessions by default. You need to pass --include-non-interactive flag in such a scenario, and CI runs are exactly this type of sessions.

In GitHub Copilot CLI

Using --output-format json it produces JSON Lines, of which the most crucial from a practical perspective is the last one, result, which contains information about what was written to disk. For example:

{"type":"result","sessionId":"29d7a277-a4d3-4c73-8323-5674bf5ed159","exitCode":0,
 "usage":{"premiumRequests":0,"totalApiDurationMs":2347,"sessionDurationMs":4588,
          "codeChanges":{"linesAdded":0,"linesRemoved":0,"filesModified":[]}}}

The codeChanges field is always there, even if nothing was changed, the no-op run yielded an empty array in filesModified. The exit code is accessible both via the JSON and from the process itself.

The other events are more about the loading part:

• session.skills_loaded — a record of all the skills with their description, source and path

• session.mcp_server_status_changed — one per server, ours changed from pending to connected

Thanks to that we have a log of what was actually loaded in the run, with zero additional costs, as it's part of a stream we kept anyway.

In case of non-interactive runs (like CI), you can use --share=./session.md to create a Markdown transcript at the end of the run. It'll print the path to it on stderr. 383B in total, with the session ID and both turns, perfect to attach to an alert.

There are also other flags that might be useful:

• --log-dir <dir> (default: ~/.copilot/logs/) — to keep the process log

• --log-level debug — to make it more verbose

• --secret-env-vars=A,B — to remove the specified env variables from the output before sharing anything

The state of a session is being stored in ~/.copilot/session-state/<session-id>/. You can also provide a custom session ID using --session-id <id> flag.

Finally, there's a --no-ask-user flag which disables ask_user tool. As per the help, it makes the agent work autonomously without asking questions. That way, you can exclude class 2 from automated runs.

But… Our run was green (returned exit code 0), while the log contained an ERROR line saying that GitHub MCP server was configured after authentication. So, it seems like the severity level in the logs doesn't always reflect the actual outcome of a run. That means that for example if you have a CI step that greps the logs for "ERROR", it'll keep failing builds that were actually successful.

In Cursor

We've been exploring the Cursor CLI in the headless JSON mode and it outputs a single result object containing:

• The duration of the run and the API call (in milliseconds)

• The result, session ID and request ID

• The token usage for input, output, cache read and cache write

{"type":"result","subtype":"success","is_error":false,"duration_ms":2840,
 "duration_api_ms":2840,"result":"ok","session_id":"7607179e-ed57-48e0-8885-8d83abee3264",
 "request_id":"f6d9e848-0943-48b2-92fd-40c967f0dd07",
 "usage":{"inputTokens":11697,"outputTokens":27,"cacheReadTokens":5248,"cacheWriteTokens":0}}

But we see that the price is not included, which is unfortunate given we have the request ID there which is what support will ask for. We mean, it's really good to have the request ID tbh.

So sessions are being saved in ~/.cursor/chats/<workspace-hash>/<chat-id>/. You can create a new chat with create-chat and resume an existing one using --resume [chatId].

The main problem is that it doesn't work in any of our directories as it stops the run because of the per-directory workspace trust check. Being a git repo is not a thing that could help either — we hit the very same block in a repo with a commit in it, and a fresh clone is exactly a directory it has never been told to trust. When you run it the first time you see this:

⚠ Workspace Trust Required
  Cursor Agent can execute code and access files in this directory.

Which doesn't work — returns exit 1 in less than a second, empty stdout (so also no JSON) and stderr with the notice above. If you were to pipe it to jq for example, it returns an empty result and exit 0, no error at all so you don't really see it's not working.

That's because we need to run it with --trust, it's mentioned in the message above but we don't like that it also suggests using --yolo or -f, which actually work (they remove the prompt), but their own help calls them force-allow-commands flags, so using one to get rid of the trust prompt is more than we want.

In Antigravity CLI

--output-format json outputs a single object so you can assert on its status property

Here's an example:

{"conversation_id":"080aa64e-98e8-4558-83d1-52b7931f9749","status":"SUCCESS","response":"ok\n",
 "duration_seconds":1.483619,"num_turns":1,
 "usage":{"input_tokens":18187,"output_tokens":29,"thinking_tokens":23,
          "cache_read_tokens":0,"total_tokens":18216}}

Other options:

• --log-file <path> redirects the CLI log to a file that you can collect

• --conversation <id> and --continue to resume a previous run, the records are stored under ~/.gemini/antigravity-cli/conversations/

What's special about it is that it's the only tool out of the six that has its unattended run limit enabled by default (--print-timeout 5m), Claude Code and Copilot CLI have spend caps as well but they're opt-in. So here, consider increasing it if you don't want to hit it

Also, keep in mind that -p is a parameter so the order of flags matters for which question you'll ask. If you do:

agy -p --output-format json "Reply with exactly: ok"

It'll actually ask LLM "--output-format" as it treats it as a prompt, so make sure to place your flags before the parameter

That's what we did and LLM was really helpful describing output-format flags across various CLIs but also ignoring our actual prompt ("Reply with exactly: ok") which got lost as an unused positional parameter so... no JSON and exit code 0

To resolve it, just prepend the flags:

agy --output-format json -p "…"

So if you see this tool running with a zero exit code, consider checking if it's not due to a JSON parsing issue as it might be just a result of calling it with an improper order

In Kimi Code CLI

The output format is either text or stream-json so you can only choose between these two, no combined-envelope json support so you need to treat it as a stream. Here's how it looks like when you run it:

{"role":"assistant","content":"ok"}
{"role":"meta","type":"session.resume_hint","session_id":"session_931d455a-...","command":"kimi -r session_931d455a-..."}

Assistant reply line and then the meta line with the session id and the command to resume it. There's also no token usage info, cost or is_error field so you can't know the status of anything. The exit code, diff and transcript are what the result envelope contains in other tools.

As for the transcript it's really easy to find thanks to an index: ~/.kimi-code/session_index.jsonl. One record per session with the session id and a path to its directory together with a working directory of this run:

{"sessionId":"session_931d455a-...","sessionDir":"/Users/you/.kimi-code/sessions/wd_repo_39f2ff01c913/session_931d455a-...","workDir":"/private/tmp/repo"}

The session directory contains state.json, logs/ and agents/ folders. The CLI's own log is located in ~/.kimi-code/logs/kimi-code.log. To download a session you can use kimi export <sessionId> command which creates a zip file with the session contents — this is what you should share. The kimi doctor command can be used to check if your config files are all good, it prints an OK line per file.

Just keep in mind that -y / --yolo flag auto-approves regular tool calls but as you can see in the help text the agent can still ask questions. If you want a fully automated run you need to use --auto instead.

Also make sure which binary is used, here's what we mean: the Kimi Code CLI we ran reports version 0.31.x and it puts everything under ~/.kimi-code/. There's also a legacy Python tool called kimi-cli (which is the thing that kimi migrate command imports from), it lives in a separate place than the binary above and doesn't use any of the paths mentioned above. To find out which binary you use run kimi --version.