Chinese open-source models have closed the gap on quality, matching frontier performance on easy to medium tasks while slashing costs by 60-80%. They lag by a few percent on the hardest problems, but win decisively on speed and price. This is an interactive blog post comparing .
Here at Isotopes AI, we think of Artificial Intelligence as a commodity that we use and pay top dollar for. Because of our multi-agent design approach, we have never used the same model for all the agents and periodically upgraded/switched models for them as new models came up. Thanks to the modularity of our design, we've been able to move a fraction of them at a time, while monitoring costs incurred.
Considering we started building our product when GPT4o was the best model available, it has been quite a journey over the years, but we've never talked about all the work our engineers do to measure the newest model on the block.
This exercise started as evaluation of GPT 5.5 and transitioning some of our agents from Sonnet 4.6 to GPT 5.5 due to the upcoming restrictions in the Anthropic api options. The head to head between them was lacking in perspective or depth, also we almost knew the answer before we started.
We decided to expand our evals outside of the trinity of model families we've used from the Big 3 AI vendors and take out the new chinese competition for a test drive.
For compliance reasons, we ran all open models through BaseTen[1].
The industry-standard overall comparison: quality (Y) vs cost or speed (X). No normalization — raw absolute values. Up-and-left = better value. Pick any two metrics.
Profile across axes — by question or by judge aspect (per model, quality /5), or by metrics (per arm: cost/speed/latency/tokens, normalized). Bigger polygon = better. Arms follow the header Models + Thinking selection.
Judge quality per model × question. Green = strong, red = weak.
Each cell = mean judge quality, /5 for that model × question. Green = strong (→5), red = weak (→3), grey = not in selection.
Head-to-head on every metric (current mode + thinking).
Two arms (model × thinking) head-to-head over the selected questions. Green = the better value on that metric (higher quality/approval, lower cost/tokens/latency/retries); Δ is B − A.
Ranks the selected models by the header's Score by — Judge (raw quality /5, the default), Metrics (telemetry composite), or Combined. The caption below states the current basis.
Every cost in this report is computed from raw token counts × these per-model rates ($ per 1M tokens) — so edit any rate and the whole report recomputes (leaderboard, scatter, radar, tables, verdict). Prefilled with the list price. We know you probably have discounts, so apply yours.
Wrote a 5-question analyst conversation of rising difficulty, and built hand-verified deterministic goldens for each. Each model, at thinking off and low, acts as a coding team: it writes Python, runs it in a sandbox, and critiques its own code — with retries. A fixed neutral judge (Claude Opus 4.8) then scores the team's final output against the golden. One try per cell; every signal captured per agent.
Five aspects, 1–5: output correctness, code correctness, code cleanliness, approach, robustness — against the golden's result + code. The judge allows reasonable methodological variation but penalizes wrong magnitudes, missing keys, or a wrong method.
Hugging Face and GitHub links coming soon
Everything here is computed live in your browser from one raw source: the per-cell results of the two full runs (1 try per cell). No pre-averaged number is baked into the charts — every aggregate is derived from first principles, so any filter re-computes the whole report.
The default test is self-critique (each model is its own coding team). We also ran a fixed GPT-5.5 critic for comparison. Below is how the two modes behave (all models, both thinking levels).
Judge quality is the ground-truth metric — the same neutral judge scores both modes, so it is directly comparable. The other rows describe the critique loop's behavior (approval rate, retries forced, writer cost driven), not quality.
Fable 5 was deliberately excluded. It is not offered with our ZDR account at the moment. It is also the wrong tool for the job: a frontier, long-running reasoning model is overkill and costly for this benchmark with well-scoped coding tasks, where the goal is efficient, correct code — not open-ended, long-horizon agentic reasoning. The models here span the practical price/quality range a team would actually deploy for day-to-day analytical coding.
To put metrics of different units on one radar / composite, each is scaled to 0–1. Two rules, chosen for honesty:
Consequence: on the metrics radar, quality vertices cluster near the edge (models are similar on quality) while telemetry axes fan out (cost/speed vary a lot) — a truthful picture of the trade-off.
Min-max is relative, not absolute. The best arm on an axis becomes 1.0 and the worst 0.0 by construction — so a Metrics score is standing within the shown field, not an absolute rating: a model at 1.0 is best-in-field (e.g. cheapest & fastest), not "perfect," and the last-place arm at 0.0 isn't "worthless." The endpoints are always 1 and 0 regardless of whether the real gap is 6× or 5%. Read the Judge score (raw /5) as the absolute metric; Metrics/Combined are relative lenses. A metric on which all arms are identical (e.g. thinking tokens when thinking=off) carries no information and is dropped from the composite.
| Control | What it does |
|---|---|
| Critique mode | Which critique run you are viewing — Self-critique (each model critiques its own code; the "coding team" test) or Fixed GPT-5.5 (one constant critic for all). |
| Thinking | Reasoning level: off, low, or both (averaged). |
| Models | Show/hide models. Charts re-scale to what's shown (except the metrics radar, which is normalized against the full field — see A4). |
| Questions / Difficulty | Restrict to a subset of the 5 questions; the difficulty buttons are a quick-filter (e.g. "Hard" = Q3+Q4). |
| Metrics | Which telemetry signals count in the metrics radar + the metrics/combined score. |
| Aspects | Which of the 5 judge aspects define "quality" (see A2). Deselecting an aspect re-scores the ENTIRE report. |
| Score by | What the leaderboard + KPIs rank by: Judge, Metrics, or Combined (A5). |
| Judge weight | The judge-vs-metrics split used by the Combined score. |
A fixed neutral judge (Claude Opus 4.8) scores each answer 1–5 on five aspects — output correctness, code correctness, code cleanliness, approach, robustness — against the hand-verified golden. Quality = the mean of the SELECTED aspects, averaged over the selected cells. It is recomputed from the raw aspect scores, so the Aspects filter changes quality everywhere (KPIs, leaderboard, heatmap, scatter, table).
The metrics are the actual measured signals of the writer (the agent under test), captured per call: Cost ($), TTFT (time to first token, ms), Latency (total, ms), and In / Out / Think tokens. Latency excludes retry backoff; thinking tokens are separated from output so cost isn't double-counted.
w · normalized-quality + (1−w) · metrics-composite, where w = Judge weight. Quality here is min-max-normalized (not /5) so the weight has real influence — otherwise quality's narrow ~0.85–0.95 range would let telemetry dominate despite a high judge weight. The trade-off: quality's small spread is stretched inside Combined. The two dimensions are orthogonal (quality is never double-counted). For the faithful accuracy view use Judge (raw /5).Under self-critique a lenient critic is a real weakness — but the same neutral judge scores both modes, so quality stays comparable, and critic_approved is captured as a separate per-model signal. See the full comparison below.
Every cell in the current selection. Click a header to sort.