Kimi K3 vs Claude Fable 5 and Opus 4.8: a benchmark you can run yourself

Every time a new model lands — and Kimi K3 is the one everyone's poking at this week — the internet fills up with benchmark scores. Leaderboard screenshots, bar charts, a hundred hot takes. What you almost never get handed is the benchmark itself: the actual task, the actual code, the thing you could open up and run to find out whether the number means anything for the work you actually do.

A score is a final scoreboard. Useful at a glance — but you can't see how the game was played from it, and you certainly can't see whether the code that came out the other end is something you'd be happy to inherit. So we built the match you can watch: a small, honest benchmark for Kimi K3 against Claude Fable 5 and Opus 4.8. We ran each model three times — nine runs in all — read every line they wrote, and put the whole thing on GitHub so you can run it too.

The challenge

The task is deliberately close to real work, not a party trick. Each model gets a small money-handling module — a double-entry ledger — that already comes with a test suite, plus a plain senior-engineer brief: get this production-ready, and add two features (reverse a transaction, and produce a statement). Nothing about "find the bugs."

The catch is that the unit tests implemented in the project are a trap. Plenty of nasty bugs sail straight through them: a transfer that loses the money if its second half fails, amounts that drift by a fraction of a cent, an accessor that hands a caller the keys to corrupt the ledger. Passing those tests makes the thing demo. So we grade against a second set of tests the model never sees — the ones that ask whether the money is actually safe. Green on the tests you can see is the easy part. Green on the ones you can't is the real question, and it's the one a leaderboard can't ask.

Same harness, on purpose

Same setup for all three: each model works agentically inside Claude Code — opening files, editing them, running the tests, reacting to what breaks — at high effort, three separate runs each so we can see how much a model wobbles between attempts on an identical prompt.

Kimi K3 was run through the exact same harness — Claude Code — as the two Claudes, so that the harness wasn't a variable. If one model does better, we want it to be the model, not the scaffolding around it. It's worth mentioning that this measures each model as an agent inside Claude Code, which reflects what a real-world dev team is likely to actually put to use.

Results

Whether the hidden tests passed is one bit of information. What's worth looking at is how each model wrote the code — what all three got right, and where they went different ways.

The part every one of them got right

The load-bearing fix in a ledger is atomicity: a transaction has to be all-or-nothing, so that if the second half fails, the first half never happened and no money evaporates. The starter module got this wrong on purpose — it validated and wrote each line in the same loop, so a bad account halfway through left a half-transaction behind.

Every single one of the nine runs fixed it the same way, and the right way: validate the whole transaction first, then commit in a second pass that cannot throw.

// validate everything first — this part is allowed to throw
for (const line of lines) {
  const acc = accounts.get(line.accountId);
  if (acc.state !== "open") throw new Error(`account not open: ${line.accountId}`);
}
// then commit — from here nothing throws, so it's all-or-nothing
for (const line of lines) this.entries.push(makeEntry(line));

They also all reached for the same clean pattern for the new reverse feature — a compensating transaction routed back through post, so a reversal inherits the same atomicity and lifecycle checks for free instead of re-implementing them — and they all plugged the accessor leak so a caller can't reach in and mutate the ledger's history.

So the boring, settled truth is: can a frontier model fix the bug and add the feature is a solved problem. All three can, every time. If that's your bar, they're interchangeable. The differences only show up when you ask a harder question — not "does it work today" but "how will this code behave in a year, in someone else's hands."

Claude Fable 5 — the steadiest hand

Fable was the most consistent of the three, and consistency is a feature. Across all three runs it made the same load-bearing choices, and it tested its own encapsulation as an invariant — each run writes a test that tries to corrupt the ledger through a returned object and asserts it can't, so a future dev who mistakenly removes the defensive copy breaks a test immediately rather than shipping a leak. That's the difference between "did the safe thing" and "made the safe thing hard to undo."

Where the three Fable runs differed, they differed by degree of paranoia rather than direction — the most careful run reserved a naming namespace, guarded against integer overflow, and threw on unknown accounts instead of reporting a zero balance. None of them shipped a bug.

Its ceiling: money stays a bare number. Nothing at the type level stops a future caller from passing dollars where cents are expected — both are just numbers — so the safety lives in runtime checks a later edit can weaken. And its reversal linkage leans on a string prefix convention, which is the kind of thing that works beautifully until someone names a transaction the wrong way and silently can't reverse it.

Claude Opus 4.8 — the highest ceiling, and the clearest cautionary tale

Opus wrote the most self-aware code. Its atomic commit is annotated at the exact line where the invariant starts holding (// from here nothing can throw), it freezes entries at write time so they can't be mutated even in place, and its reports are engineering-grade: a per-change root-cause writeup and a "Deliberately NOT changed" section that names the things a junior would get wrong and explains why it left them. Its conservatism reads as seniority, not laziness — it ships the additive fix and documents the risky one instead of charging in.

And yet Opus is also where we found the single most instructive failure in the whole exercise. In one of its three runs, it made two individually-reasonable changes that it never reconciled with each other:

// change one: tighten post to reject non-integer amounts — sensible
if (!Number.isInteger(amount)) throw new Error("amount must be whole minor units");

// change two, in another file: leave the dollars->cents conversion unrounded — looks harmless
function toMinor(major: number) { return major * 100; } // 0.07 * 100 -> 7.000000000000001

Each is defensible on its own. Together, a transfer of 0.07 now computes a hair over 7 cents and gets rejected by the stricter check — a perfectly ordinary transfer throws an error. The kicker: the run's own tests were green, because it never wrote a test that transfers a fractional amount. Its self-authored safety net had exactly the hole its own change fell through. The other two Opus runs rounded the conversion and tested it, and were flawless. Same model, same prompt — one run threads money-correctness end to end, one leaves a landmine that looks green.

Kimi K3 — bold, stable, a little heavy-handed

Kimi wrote structurally sound code and, notably, wrote the same sound structure all three times — the atomic post, the idempotent reversal, the copy-at-the-boundary discipline are stable across every run, and a couple of runs go further than the others and reject a reused transaction id with conflicting contents, catching a class of double-spend bug the reference didn't. Stability like that, run to run, is a real quality signal.

It was also the boldest with the hard decision: two of its runs switched the money type to bigint, which is genuinely the correct way to hold exact money in JavaScript. The catch is that it committed to the choice in the middle but not at the edges — the transfer function still takes a floating-point number, and the balance reader still converts back through one, so in one run you get Number(minor) / 100 reintroducing floating-point error at the exact layer bigint was adopted to remove. A future dev reads Money = bigint, trusts it completely, and gets bitten at the seams.

And in one run Kimi over-corrected: it froze the array it hands back from getEntries, which protects the ledger — but now a caller that appends to that array for its own local use crashes with a type error. The internal state is safe; the contract quietly changed underneath everyone using it. That's the mirror image of a leak: hardening that breaks the caller instead of the invariant.

The thread running through all of it

Step back and the three of them rhyme. The core is solved; the risk has moved to the edges, and every model left a version of the same kind of footgun — the sort a future person or model trips over precisely because of how the code is written. Money as a bare number that the type system won't police. Safety that rests on a freeze sitting in a different method than the code depending on it, one refactor away from silently opening a hole. A reversal linked by a string prefix. A balance reader whose comment still promises "whole units" after the value quietly stopped being whole.

And then there's the thing three runs each made visible that a single run would have hidden: the same model, on the same prompt, gave subtly different public contracts each time. Whether a duplicate transaction throws or is a silent no-op; whether you can reverse a transaction after closing the account; whether asking for a statement on a typo'd account is an error or an empty list — all of these flipped between runs, in all three models. None of it is wrong, exactly. But it's the clearest argument in the whole exercise for a boring old discipline: if an edge behaviour is load-bearing for you, pin it with a test. Don't trust a model — any of them — to be consistent about it for free.

Did it actually hold up?

For all that, on the thing we actually graded — the hidden tests that check the money is safe — it was close to a dead heat.

Fable came through clean on all three runs. Opus and Kimi each slipped once — and it's worth noticing how they slipped, as neither was a "couldn't do it"-type failure. For Opus this was two good changes that weren't reconciled; in Kimi's case it was hardening pushed one notch too far. Both are edge-and-judgement misses, not capability misses, which is exactly what you'd expect once the core is this solidly within reach for all three.

The restraint trap

We planted one thing that has nothing to do with fixing bugs and everything to do with judgement. One function rounds money to whole units — it's lossy, but it's the documented, tested behaviour, so any caller out there depends on it. The junior move is to spot it and "fix" it, silently, breaking every one of those callers. The senior move is to leave the contract alone, flag the hazard, and add a safe alternative next to it.

Fable and Opus made the senior move, every run: kept the rounding, wrote a loud warning at the call site, and added an exact accessor beside it. Kimi was the least disciplined here — it quietly changed the rounding, and to a different result in each run, while leaving the comment still claiming "whole major units." A small thing. But how a model treats a contract it didn't write, when nobody's forcing it to, tells you a fair amount about what it'll do to yours.

So, who won?

You can read it two ways, to be honest. One reading: Fable won — cleanest across the board, no shipped bug, the most consistent, and the most likely to make its good decisions stick. The other reading: this was one task, one small ledger, and the others' slips were really small; on another problem the order could shuffle.

On another problem — exactly. This is just one benchmark, and it's a good start. If you want to work out which model is best for the kind of problem you work on, this is the shape to build on: decide what you actually need it to get right, then run far more than three attempts per model — three was just enough to show the variance is real, nowhere near enough to rank through it — across many more tasks, and varied ones, because a model that's a hair behind on a ledger might be ahead on a parser or a state machine. Spread it over a range of domains and some long-horizon work, and measure consistency as its own number rather than folding it into a pass or a fail.

Run it yourself

So it's all there — the module, the hidden tests, the runner scripts, the exact prompt — on GitHub at github.com/dsplce-co/kimi-vs-fable-vs-opus. Clone it, rip out our toy ledger, drop in a slice of your own codebase with your own hidden tests, and point it at whichever models you're weighing up. You'll learn more about how one fits your workflow in an afternoon than in a month of watching bar charts.

If you just want to get Kimi K3 running inside Claude Code first, we wrote that bit up separately.

And if you run it and it tells you something different from what it told us — brilliant. That's rather the point. Let us know.