Judge panels: when three agents on one answer is worth it
Three judges on one answer buys decorrelation, not intelligence. The four conditions that make a panel worth its 3x bill, and a script that tallies the votes.
The objection you should have first
Generally, you can ask three agents for sth that is hard to verify (migration plan, if bug is real, security decision etc) and proceed with majority opinion, but then you're basically buying the same thing 3 times by using the same LLM with the same prompt against the same context. It's worth taking this seriously, because the published research points both ways.
Anthropic: in their account of the multi-agent system they built to do the research, they say that they tried using multiple judges per component, but ended up with one — "a single LLM call with a single prompt outputting scores from 0.0-1.0 and a pass-fail grade was the most consistent and aligned with human judgements".
PoLL: "Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models" by Verga et al (2024) — they say that a panel of multiple small models "outperforms a single large judge, exhibits less intra-model bias due to its composition of disjoint model families", and is over seven times cheaper.
The difference between these two systems is crucial — Anthropic split judges between components of a single rubric, while PoLL used a panel of different models and in the paper they say that the main reason behind this was to reduce biases connected with using a single model family. So from our perspective, panels are not about capabilities, but about decorrelation — when you ask multiple judges you basically buy the fact that they will be able to make mistakes in different ways. If they can't, then you just spent more money.
That's why we think that if you ask a panel "is this good code?", you will get three blurry answers and a number with no meaning; panels are about reducing variance for already well-asked questions.
The four conditions
So here are the preconditions to use a panel (and if any of these aren't met, then usually it's a no):
There is no mechanical arbitrator — a test, a type-checker, a compiler or just running the thing is the most cost-effective, repeatable and objective answer you can get; if you want to ask a panel, you should ask about sth that doesn't have one (is this migration plan sensible? Is this really a bug? Does this diff support the claim?).
The cost of a mistake is high or it's being found too late — irreversible actions like data migration, public release, security decision etc; if you can do sth and find it doesn't work in 10 minutes so you do it again, then you don't need a panel.
You can achieve decorrelation — by using different model families, or asking genuinely different questions about the same answer; same model with same prompt against same context is like one judge in three costumes.
The verdict is countable — you need to have only two possible outcomes here; if you use a scale from 1 to 10, the panel doesn't work (6, 8, 7 is not helpful) but it does work for "holds" / "breaks".
The recipe
Then the process:
Freeze the candidate — create a file with the answer and its self-description in it; don't let it change during the verification process, don't improve it in the middle of it, don't tell the judges who is the author; if the thing you ask about changes during the process, then you actually ask three different things.
Define the verdict contract — create a simple type with verdict (a binary), reason (one-sentence referring to a specific line or claim) and confidence; if you can't name the two verdict values, then you didn't ask a good question yet so go back to point one. In half of the cases you can enforce this shape with JSON schema, in the other half you need to ask about it in the prompt and parse the answer.
Assign every judge their own lens — either three different perspectives on one answer (if it aligns with its claim, what input breaks it, if every claim is covered) or one perspective across three model families; lenses are the cheap version and they work, using different models is what PoLL actually measured.
Isolate every judge — text in, verdict out, no tools, no access to the repository, no shared session; if you give a judge access to your codebase they will go into it, if you show one judge another's verdict then you don't have three judges any more.
Count in code — the surrounding script is what does the counting for you; never ask a fourth LLM to summarise the three verdicts you got, you just introduce a single point of failure that you paid for the panel to eliminate and lose the disagreement that carried the value.
Report splits instead of settling them — the script is reporting unanimity as unanimity and split as split, not resolving it under the hood; 2-1 is information on its own, it means that the answer can stand one perspective but not another, exactly what a single judge would hide in a wall of text.
A lens is a prior
And here's an example. This is a function with a real bug: it says that it returns None if the input is empty, but actually throws an error because it tries to access the first element of a list even if it might be empty. We asked a panel, and they all said "breaks" and gave the same reason: the same IndexError. So, yes, it's correct, but if you ask a single judge it will cost you three times less for this one.
This time we asked about the fixed version, and the panel split 1-2; one of the judges said "breaks" and pointed out that it throws TypeError if you pass None. Which is correct, but not in line with what the function says — it never said anything about None. The reason why the panel said this is that we asked them to find edge cases, and if you ask a judge to find edge cases they will find one.
To make it more robust you need to:
Limit the tally question to what the thing actually says; you can point where the judge should look by using lenses.
Write in every prompt that "holds" is a valid verdict.
Occasionally submit a perfect answer, if the panel never returns "holds" it's a ceremony and not an evaluation.
Running the panel in Claude Code
Here's what it looks like to run one judge:
claude -p "$PROMPT" \
--model haiku \
--json-schema "$SCHEMA" \
--tools "" \
--no-session-persistence \
> "$OUT/$name.json"We use 3 flags:
--json-schema(2.1.220 calls it "JSON Schema for structured output validation") — set to return the verdict in the expected structure instead of returning a text that'd need regex to extract. Using it, with the default text output format, we get a bare object in stdout. As you can see, this way there's no preambles or any other noise — we get a raw JSON which directly goes intojq--tools ""— this is what actually does the lockdown. The docs sayUse "" to disable all toolsand that makes perfect sense given it's a print mode so the judge only sees the text you paste, so no tool is needed. Thanks to that there's no room for it to approve or deny anything--no-session-persistence— we set this one as the docs say sessions are neither being written to disk nor being resumable. This way we make sure we don't end up with 3 new sessions in the session list
In addition to the above, there's a couple more flags that are useful when it comes to the fan-out:
--model— you can set it to an alias: haiku, sonnet or opus--max-budget-usd— you can set the hard cap for what this run of print mode should cost (in case the panel is more expensive than you think)
And that's it. The outcome of all of these, three times, in parallel, is presented below:
Using Codex for the panel
To run the judge panel via Codex CLI, run one codex exec per judge like so:
printf '%s' "$SCHEMA" > "$OUT/schema.json" # once, before the loop
codex exec "$PROMPT" \
--output-schema "$OUT/schema.json" \
-o "$OUT/$name.json" \
-s read-only \
--ephemeral \
--skip-git-repo-check \
< /dev/nullThat way, Codex reads the schema from a file (not as a parameter like Claude Code does) and writes the last message to the file system on its own. That way we don't need to parse stdout anywhere in the panel.
The flags:
--output-schema <FILE>: The schema is needed, but as we provide it to Codex in a file, not inline like in Claude Code; that's why we need to write it there first-o/--output-last-message <FILE>: The place where the verdict will be saved, to make sure it's the same place the tally script reads it from so we don't need any stdout parsing-s read-only: Locks the sandbox for the judge--ephemeral: Doesn't create 3 disposable sessions on disk
And a word of caution about that last line; it's kinda a five-minute story. codex exec is a utility that consumes stdin and attaches it to the prompt in the <stdin> block. If you run it from a script where its stdin is a pipe, and this pipe remains open (like ours, because we didn't close it with < /dev/null), it keeps printing "Reading additional input from stdin..." and waiting.
And that's what happened to us — the run hung until a five-minute timeout killed it, which under fan-out means every judge at once. So make sure to redirect from /dev/null unless you're deliberately feeding the candidate that way.
The remaining parts of the artifact (lenses, schema, tally script) can be left unchanged.
Running the panel in GitHub Copilot CLI
Of the six harnesses this is the shortest judge command of the lot:
copilot -p "$PROMPT" -s < /dev/null \
| sed '/^```/d' > "$OUT/$name.json"This is how you can pass a prompt to it and make it write a file with a JSON object per judge; -s / --silent flag is mentioned as only displaying the agent's response and not stats, which is perfect for this use case in conjunction with -p flag.
However, the docs say that you need to add --allow-all-tools flag to run it non-interactively, but that is needed only if you want to run tools during the process, the judges don't; so there's no need to include any of these in this command — and if you ever feel like you need them, that means your judges aren't isolated. Instead of using --allow-all-tools or --yolo flags you should utilise --available-tools and --excluded-tools ones if you really want to give the judge access to tools.
Also important is that there's no way to enforce any schema in Copilot by providing a flag, so if you use --output-format json flag you'll be getting JSONL event stream rather than a validated verdict; therefore it's crucial to tell the judge what type of output you expect from it and handle the output carefully. The sed above removes the backticks if they're present, so what lands in the file can just be parsed with jq — and in case jq is not able to parse a verdict, it should be treated as an abstention by the tally in the artifact instead of a vote. Our run came back as bare JSON with no fence, but nothing enforced that.
Running the panel in Cursor
agent -p "$PROMPT" \
--mode ask \
--trust \
--output-format json \
< /dev/null \
| jq -r '.result' | sed '/^```/d' > "$OUT/$name.json"In the pipeline above, for every judge, we run a single print-mode agent in a given directory. That way we collect the JSON output of the agent, extract the result and strip the fences to create a file with the result per-judge.
There are a couple of things to know here, and because all the judges start from the same directory, they hit all of them at once:
The agent doesn't know the directory — in such scenario, running the agent in print mode results in a "Workspace Trust Required" error with options to use
--trust(a solution that can be used to grant access for a single directory),--yoloor-f(which basically give the agent a universal access pass)By default the
-pflag allows access to all the tools including write and shell which is not ideal for a judge. Instead we can use--mode ask(read-only question-answer mode) to get an isolated judgeTo extract the result in the pipeline we need to do it in two steps:
the verdict itself comes as a string nested in the result envelope so we need to access the
.resultfirstthis time the string was actually fenced json so we needed to strip the fences in order to get a plain object which is what the tally function expects
Running the panel in Antigravity CLI
We used another binary for the judge panel because Google announced that from 18 June 2026 Gemini CLI stopped serving Google AI Pro, Google AI Ultra and free-tier users. They still support Code Assist Standard/Enterprise licences and paid API keys, so we can use Antigravity CLI's agy instead. Just run it with one -p per judge:
agy -p "$PROMPT" \
--json-schema "$SCHEMA" \
--output-format json \
< /dev/null \
| jq -c '.structured_output' > "$OUT/$name.json"The --json-schema flag works with either an inline schema string or a path to a schema file. If you set it up the verdict will be returned as an object in the .structured_output field, so no need to remove code fences from it. The tally consumes it as-is if you provide that field, but keep in mind these two flags work together only — if you use the --json-schema flag on its own it will throw an error saying it requires the --output-format flag set to json or stream-json. Which means that if you were to use them in the fan-out setup, it would result in three of your judges dying immediately for no apparent reason.
Running the panel in Kimi Code CLI
The general flow for running the judging is to run kimi -p once for each judge; what's important here is setting it up properly so it uses an output format that can be easily consumed, for example:
kimi -p "$PROMPT" --output-format stream-json < /dev/null \
| jq -r 'select(.role == "assistant") | .content' > "$OUT/$name.json"This way the assistant's messages are saved to a separate file for every judge. The problem is that by default Kimi Code uses text-based output which is not suitable for parsing, it looks like this: lines prefixed with • , plus a "To resume this session: kimi -r session_..." line appended at the end. Piping that into jq just fails.
With --output-format stream-json you get JSONL instead, so the verdict is the .content of the assistant line and the resume hint arrives as its own meta line, which is exactly what the select above filters out.
There's no schema flag here — --output-format takes text or stream-json and nothing else — so same as with Copilot, ask for the verdict shape in the prompt and treat anything unparseable as an abstention rather than a vote. And if you want your judges to differ by more than their lens, kimi --help also lists --agent <name> and --agent-file <path> for loading an agent definition per run, plus -m/--model for the model alias.
What it costs before it says anything
And the absolute minimum is not connected with your question, but with the harness. Here's how much it costs (for four different CLIs) to ask a single-sentence question — not the same sentence in each case, so read these as orders of magnitude: 4290 cache write and 21649 cache read input tokens in Claude Code (input_tokens: 10), 14016 input tokens in Codex, 9984 of which were from the cache, 15473 in Cursor, 18327 in Antigravity. So the minimum for Claude Code on Haiku, in this case, is 1.47 cents. If you ask three judges, you pay it three times.
Choose the smallest LLM that's suitable for the task — that's what PoLL found, it's not a compromise — and keep the answers short; if you ask a judge to read your entire repository, it will cost more than it's worth.
Panel, or adversary?
The other technique we mentioned in this chapter is adversarial verification, when you ask another agent to invalidate the findings of the first one. It's a different tool. If you think that the survival of a finding against attack is in question — use an adversary; if you need multiple independent opinions, use a panel.
If you ever find yourself asking a 5-judge panel for every point in your findings list, it means you should have used an adversary and a loop (which is cheaper).
And retire the panel once the question stops needing it — the moment somebody writes the test that answers it mechanically, delete the panel and run the test.
scripts/judge-panel.sh#!/usr/bin/env bash
# Three sealed judges, one answer, one verdict each — and the shell counts the votes.
# Usage: judge-panel.sh <file containing the answer under review>
set -euo pipefail
CANDIDATE="${1:?usage: judge-panel.sh <file-with-the-answer-under-review>}"
JUDGE_MODEL="${JUDGE_MODEL:-haiku}"
OUT="$(mktemp -d)"
# Three different questions about the same answer. Three copies of one question would
# just buy the same opinion three times.
LENSES=(
"correctness|Does it do what it claims to do? Wrong logic only — not style, not naming."
"edges|What input breaks it? Empty, null, zero, duplicate, huge, concurrent. Name the input."
"evidence|Is every claim in the answer supported by what is in the answer? Flag anything asserted without support."
)
SCHEMA='{"type":"object","properties":{
"verdict":{"type":"string","enum":["holds","breaks"]},
"reason":{"type":"string"},
"confidence":{"type":"number"}},
"required":["verdict","reason","confidence"],"additionalProperties":false}'
for lens in "${LENSES[@]}"; do
name="${lens%%|*}"
question="${lens#*|}"
claude -p "You are one judge on a panel. You judge alone and you never see the other judges' verdicts.
Judge through this lens only: $question
Judge the text below and nothing else. Do not open files, do not run anything. If the lens
you were given finds nothing wrong, say it holds — a judge that never says 'holds' is worthless.
--- ANSWER UNDER REVIEW ---
$(cat "$CANDIDATE")" \
--model "$JUDGE_MODEL" \
--json-schema "$SCHEMA" \
--tools "" \
--no-session-persistence \
> "$OUT/$name.json" &
done
wait
breaks=0 holds=0 abstained=0
for f in "$OUT"/*.json; do
verdict=$(jq -r '.verdict // "unreadable"' "$f" 2>/dev/null || echo unreadable)
reason=$(jq -r '.reason // ""' "$f" 2>/dev/null || true)
printf '%-12s %-10s %s\n' "$(basename "$f" .json)" "$verdict" "$reason"
case "$verdict" in
breaks) breaks=$((breaks + 1)) ;;
holds) holds=$((holds + 1)) ;;
# A judge that didn't answer in the agreed shape didn't vote. Don't guess for it.
*) abstained=$((abstained + 1)) ;;
esac
done
echo
if [ "$abstained" -gt 0 ]; then
echo "PANEL: $breaks breaks / $holds holds, and $abstained judge(s) came back unreadable."
elif [ "$breaks" -eq 0 ] || [ "$holds" -eq 0 ]; then
echo "PANEL: unanimous — all $((breaks + holds)) say $([ "$breaks" -gt 0 ] && echo breaks || echo holds)."
else
echo "PANEL: split $breaks-$holds — read the reasons yourself. The disagreement is the finding."
fi
echo "verdicts: $OUT"