Parsimony
How much code do coding agents write to fix real issues? Research preview.
Parsimony measures published SWE-bench Verified patches in coding units: each statement, call, comparison, name and literal counts as one unit. Formatting, comments and punctuation are not counted. Patches that pass the tests score higher when they are smaller. Failed patches score zero or below.
Leaderboard
Population coverage
| Rank | Model |
|---|
All-task score rank (unchanged when sorting other columns): the range covering 95% of each model's ranks when the tasks are resampled 2,000 times; overlapping ranges mean the order is uncertain (sensitivity report).
Score: mean over all tasks, from −25 to 100. Per solve: mean score over the tasks the model solved, where 50.5 is average. Median churn: units added plus removed per solved task.
Click a column heading to sort; click again to reverse. Score uses the midpoint when shown as bounds; 95% CI sorts by its lower bound. Missing values stay last. Per solve and median churn describe successful patches only; sorting them does not change the all-task score.
95% CI shows score uncertainty from resampling tasks, widened for missing measurements. It helps avoid overinterpreting small score gaps; it is not a measure of code complexity and does not cover every source of bias.
Churn and complexity added
Here “complexity added” means net coding units added (code footprint), not semantic complexity. All-task points show mean footprint per measured, in-scope solve, excluding failures; the default leaderboard score still includes all tasks. Selecting a task replaces the points with its measured attempts, including failures. Missing measurements have no point. Hover over a point for its model and values.
Each task was run four times; each attempt is scored against every passing patch of the task.
Method
- Each patch is applied to the task's base commit and both versions of each changed file are parsed. Nothing is executed.
- On each task, a passing patch is ranked against all passing patches from the reference models: 80% on net units added, 20% on units changed.
- A failed patch scores from −25 to 0, depending on how much code it added.
- Test, docs and generated files are excluded. Only Python is measured.
Limits
- Smaller is not always better. Units measure size, not readability or design.
- Scores are relative to the reference models, which all use the same agent harness (mini-SWE-agent).
- Pass/fail comes from SWE-bench's published results; tests are not rerun.