Parsimony
How much code do coding agents write to fix real issues? Research preview.
Parsimony measures published SWE-bench Verified patches in coding units: each statement, call, comparison, name and literal counts as one unit. Formatting, comments and punctuation are not counted. Among patches that pass the tests, smaller footprints earn higher credit. Resolve rate is reported separately.
Leaderboard
| Model | Footprint credit ↑ | Resolve rate | Measured solves | Median churn |
|---|
Footprint credit: mean per-task credit over measured, in-scope passing patches (1–100, higher means smaller relative to the reference panel; 50.5 is the panel midpoint). Measured solves: scored / published passing attempts in this panel. Median churn: units added plus removed per measured solve.
Models solve different task subsets: this ordering is descriptive, not a same-task comparison or an overall capability ranking. Missing and out-of-scope passing patches are excluded, not assigned zero. No footprint confidence intervals or rank ranges are shown.
All-task score diagnostics (includes failures)
The original score averages all tasks, from −25 to 100. Its intervals and rank ranges describe that score, not footprint credit. Unscored tasks contribute bounds; ranks use their midpoint. Attempts of the same task are resampled together.
| Model | All-task score | 95% CI | 95% rank range |
|---|
All-task rank ranges from the sensitivity report; these do not apply to the footprint ordering above.
Method
- Each patch is applied to the task's base commit and both versions of each changed file are parsed. Nothing is executed.
- On each task, a passing patch is ranked against all passing patches from the reference models: 80% on net units added, 20% on units changed.
- A failed patch scores from −25 to 0, depending on how much code it added.
- Test, docs and generated files are excluded. Only Python is measured.
Tasks
Enter a task ID. The page opens on the task where solutions differ most in size.
Each task was run four times; each attempt is scored against every passing patch of the task.
| Model | Result | Net units | Churn | Task score |
|---|
Limits
- Smaller is not always better. Units measure size, not readability or design.
- Scores are relative to the reference models, which all use the same agent harness (mini-SWE-agent).
- Pass/fail comes from SWE-bench's published results; tests are not rerun.