Opus 5 vs Sonnet 5 vs Fable 5.1 vs Kimi K3: who hardens real production code?

6 models were given the same task — inheriting a small broken TS module for reserving stock against SKU catalog, bringing it to production quality and implementing 2 features without a bug list, each of them ran 3 times, 5 of them got perfect scores (15/15) in a previously hidden test suite on every run. The sixth one missed exactly 1 bug but the main difference between them lied in how much work they did and what left for the user to verify.

It's the first entry in this real-code benchmark series involving Sonnet, so it includes Sonnet 5 and Sonnet 4.6 and 4.5, showing a within-version progress rather than a cross-family comparison. The task itself, the hidden grader and every model's output are available so you can rerun it

The models (tested on 2026-09-18)

• Opus 5 — claude-opus-5

• Sonnet 5 — claude-sonnet-5

• Sonnet 4.6 — claude-sonnet-4-6

• Sonnet 4.5 — claude-sonnet-4-5

• Fable 5.1 — claude-fable-5-1

• Kimi K3 — kimi-k3

The input artefact

A 5-file TS module for reserving stock against SKU catalog, implementing the quantity by summing a ledger (append-only list of movements; negative for reservations and positive for returns) instead of using a mutable field

Starting point

The module ships with a test suite in which one of the tests is already red

The task itself

We want to have it perfect so that we can feel safe using it against an actual warehouse stock. Therefore, it's important that:

• no stock is lost, duplicated or goes below zero in case of any failure or invalid input

• all monetary calculations are exact

• operations are idempotent so they can be retried

• the module's internal state can't be altered from outside

We'd also like to have two new methods:

• cancel(orderId) — cancels a previously reserved order and returns its stock to availability

• history(sku, fromSeq, toSeq) — returns a particular SKU's movements within a sequence range (from fromSeq to toSeq)

As finding all the bugs is part of the task

The bugs we introduced (none of them labelled)

• The reserve method writes every line of an order as it goes during multiline orders, so in case of an error on the second line it leaves the first one's reservation under a non-existing order

• The getMovements() method returns the movements array which is accessible from the outside so anyone can alter it and modify stock totals by doing things like appending movements or modifying existing ones

• There's no quantity validation — it's possible to set a negative quantity which will mint stock and setting NaN will poison this particular SKU's availability for good

• Discontinued SKUs are reservable

• Re-submitting an order results in double reservation

• We use floats in the discount, so it's 899.1 cents

Grading method

We prepared a previously hidden test suite with 15 checks (9 of which target the bugs) and 6 more for the two new methods as well as edge cases. It only validates the end state — the final availability, ledger sums and if partial writes are persisted despite an error; it doesn't check exception types or implementation approach, so any pre-validation, rollback or straight rejection are equally good. It's all about the outcome (in this case — the production state of the code), not the process (the way they were reached)

Here are the two calibration points:

• our own solution: 15/15

• the initial (non-touched) broken code: 2/15

Reproducibility

The benchmark is open — you can find there a scaffold, the hidden test suite and the runs for every model with the report and the transcript. There are also two scripts you can run to rerun everything and print the results:

• matrix.sh — runs the entire matrix (all models three times)

• report.sh — runs the grader on its outputs and presents the results table

bash matrix.sh 3     # every model, three runs each
bash report.sh       # grade everything, print the table

Results

What's noticeable

Given the nature of the task and its size, it appears that on such a compact, well-outlined piece of work aiming to bring sth to production quality (in this case — making a small broken TS module reserving stock against SKU catalog fit for actual warehouse usage), correctness is a hygiene factor — 5 out of 6 (including both less pricey Sonnets and an external model Kimi K3) found all the bugs and got perfect scores on every run

A note on the timings

It's important to keep in mind that it was run on a single machine so the times are just rough estimates, and also that it was run sequentially against live APIs (Kimi via its own endpoint)

Did any model actually get a bug wrong?

Sonnet 4.5 got a perfect score on the first run but then 14 and 14, both losing the same check — idempotency; it double-reserved an order so it deducted stock twice

From the report

It was intentional as it deduplicated cancelled orders by IDs but decided that in case of already active reservations it's not clear what the user wants to achieve by doing the same thing again, so it allows it

Our take

Given that the task explicitly required the solution to be idempotent (so that it can be retried in case of a failure), this is a miss and not a natural difference. Also, it's weird as its first run was actually retry-safe

In comparison

Sonnet 4.6 and Sonnet 5 both maintain an orders registry and make every subsequent reservation of an already active order no-op, and they never failed the check

The version line

It seems that in terms of correctness, Sonnets became not only good but consistent — it's not a broader improvement but an increased reliability around one particular mandatory requirement

So if they all pass, what separates them?

Despite the fact that all of them got 15/15, there was a huge difference between them when it comes to what they left for the user after doing the work. This is something that can't be seen by the grader:

• Opus 5 and Fable 5.1 went all-in:

  • Wrote a failing test before addressing every bug

  • Replaced the linear availability scan with an O(1)-time index (which wasn't required by any of the checks)

  • Wrote 34–46 new tests

  • Delivered 1,400–2,500-word handover documents each, listing around a dozen risks

• Sonnet 4.6 was perfect too but much more conservative:

  • Did all the changes perfectly in a +60-lines diff and didn't add any new tests, saying that given we asked to preserve the code's structure it means we didn't want them to edit the test file

• Sonnet 5 was kinda between, the fastest with no misses but also saying they're not going to rewrite the availability index as it doesn't make sense in the context of the task's scope

• Kimi K3 did a good job too, also kinda in-between — around 25 new tests, detailed reports, all bugs found

Thoughts

None of these approaches is wrong, but if you were to delegate real code you should know that you might get a very thorough, well-tested, documented handover with a lot of tests or a minimal perfect patch. The grader can't tell them apart, and what's more — an untested solution can silently degrade after every subsequent change so two of the Sonnets delivered just that

Another example of a legitimate difference

How they handled cancelling an order against a discontinued SKU:

• Opus 5 decided not to return stock in such a scenario, and mentioned this as the main limitation of the solution

• Sonnet 5, Sonnet 4.6 and Fable 5.1 returned the stock as they reasoned that physical units are still there even if the SKU is no longer being sold

• Both are reasonable, and what's more — both pass the grader as it only validates the outcome (in this case — the production state of the code), not the process (the way the models got there)

So which model should you actually pick?

Knowing the above, if you were to work on a compact, well-outlined piece of work aiming to bring sth to production quality (in this case — making a small broken TS module reserving stock against SKU catalog fit for actual warehouse usage) in September 2026, and had a budget, the correctness is a hygiene factor — every current model (including both less pricey Sonnets and an external model Kimi K3) can deliver it. It's also important to know that the price doesn't correlate with the number of bugs found, so if you were to choose — choose based on what actually differed:

• Thorough, well-tested, documented handover, with a new tests and a big diff: Opus 5 and Fable 5.1

• The fastest perfect patch: Sonnet 5 (the quickest one, no misses)

• If you were keen to stay on Sonnet: 4.6 over 4.5 is a substantial improvement in terms of idempotency

Caveats

It's worth remembering that it was a single task and a single day, for a small module. A bigger and more tangled piece of code would probably make these models differ again — parity on a neat ~150-line problem doesn't generalise

The date

We state the date (and the model identifiers) up front because this is not a ranking — it's a snapshot with an attached timestamp, so the positions might change

One more thing about the hidden suite

Publishing the benchmark makes the hidden checks public too, so a future model might be trained on them and anyone re-running should assume the suite is visible rather than held out. That's fine — the point here is the method (a real task, end-state grading, the raw per-run evidence), not a leaderboard we're asking you to trust