def has_negative(values):
found = False
for value in values:
if value < 0:
found = True
return foundParsimony
Existing benchmarks measure solve rate and cost, but don't punish excess code, which makes a codebase harder to work with in the long term. Parsimony measures how much code AI models add.
It takes the patches that benchmarks already publish and counts syntax units (calls, names, literals and so on) rather than lines: net syntax units = added − removed (method).
Compare models
Change chart
How to read this
Models are ranked by mean net syntax units added on the tasks they solved, lowest first. "Net syntax units added (mean)" also includes failed attempts. Solved comes from SWE-bench Verified; tests aren't rerun. Measured is how many solved attempts could be measured.
Each model has its own solved, measured comparison tasks and peer mixture. Missing measurements are not zero; large patches can dominate the ratio. Hover over a ratio for its shared-task count and conditional 95% range, or over Measured for success coverage. These ranges do not cover missingness, peer selection or differing harness/retry policies; configurations are not controlled model-only comparisons. Ranking details and offline diagnostics.
It's an outside score, matched by model and reasoning effort, and doesn't affect the rank.
Faded models were run by me on the first few DeepSWE Python tasks (OpenAI models with pi, and the GPT-6 family with Codex, whose pi runs appear with "Show other harnesses"; Anthropic models with Claude Code) and graded with each task's own tests. They aren't verified or ranked. GPT-5.6 Sol ran both ways: my run added 0.88× the code of its published run.
Leaderboard
Swipe the table sideways for more columns.
| Rank | Model |
|---|
Missing a model? Request it. A number looks wrong? Report it. Have spare API credits? Run a model on the tasks.
Data sources
Hashes
Method
- Each patch is applied to the task's code, and both versions of every changed file are parsed. Nothing is run.
- Tests, docs and generated files are left out. Only Python is measured.
- Pass/fail comes from SWE-bench's published results; tests aren't rerun.
Limits
- Smaller isn't always better: units measure size, not readability or design.
- A model whose fixes mostly delete code can rank first with a negative net, and failed attempts that change nothing pull the all-measured mean toward zero.
- Models that write more code also tend to solve more tasks (Spearman 0.33–0.60 on the boards with 26+ models).
- Every model uses the same agent (mini-SWE-agent), but the agent still affects how much code gets written.
Add your own benchmark
Have code-size numbers from another benchmark? Download the example file, fill in your results and open it to get the same chart and table. The file stays in your browser. Format and how to share it.
Example: check for a negative number
Both return the same result.
def has_negative(values):
return any(value < 0 for value in values)