Parsimony
How much code do coding agents write to fix real issues? Research preview.
Parsimony measures published SWE-bench Verified patches in coding units: each statement, call, comparison, name and literal counts as one unit. Formatting, comments and punctuation are not counted. Patches that pass the tests score higher when they are smaller. Failed patches score zero or below.
Leaderboard
| Tier | Model | Score | 95% CI | Solved | Per solve | Median churn |
|---|
Models in the same tier are not statistically distinguishable (sensitivity report).
Score: mean over all tasks, from −25 to 100. Per solve: mean score over the tasks the model solved, where 50.5 is average. Median churn: units added plus removed per solved task.
Method
- Each patch is applied to the task's base commit and both versions of each changed file are parsed. Nothing is executed.
- On each task, a passing patch is ranked against all passing patches from these ten models: 80% on net units added, 20% on units changed.
- A failed patch scores from −25 to 0, depending on how much code it added.
- Test, docs and generated files are excluded. Only Python is measured.
Tasks
Enter a task ID. The page opens on the task where solutions differ most in size.
| Model | Result | Net units | Churn | Task score |
|---|
Limits
- Smaller is not always better. Units measure size, not readability or design.
- Scores are relative to these ten models, which all use the same agent harness (mini-SWE-agent v2.0.0).
- Pass/fail comes from SWE-bench's published results; tests are not rerun.