Harness Leverage Across Model Size: A Ladder of Five Local Agent Harnesses from 0.75B to 4.7B
Organizations: School of Mathematics and Physics, University of Science and Technology Beijing, Beijing, China · Honor Device Co., Ltd., Beijing, China
Abstract
A harness turns a model into a system that completes tasks: it builds the prompt, exposes tools and decides when work is done. Harness comparisons hold the model fixed. Model comparisons hold the harness fixed. We vary both: five shipped harnesses, 18 artifact-producing tasks, four locally deployable open-weight models from 0.75B to 4.7B and a 35.5B anchor, with one deterministic checker scoring all 360 ladder cells. The completion-controlling harness, Mingbird, has the highest mean at every size. Of four pre-registered paired contrasts, two survive Holm correction, at 0.75B and 4.7B. Thirteen of sixteen paired intervals exclude zero under the deployment reading, with leads from 0.075 over the strongest rival to 0.819 over the weakest. The completion gate is not the carrier: disabling all six executable checks moves the mean by +0.080, +0.117, +0.003 and -0.004, every interval including zero, and only at 1.9B does disabling it create empty runs. Removing one repeat-blocking rule alone costs 0.411 at 0.75B and leaves eight of eighteen tasks with no artifact. The harness effect is a progress effect: it decides whether a run leaves anything on disk. Of 357 usable cells, 127 leave none of their declared artifacts and 85 of those exit normally, so a scorer that reads exit status credits work that was never written. Whether the effect grows or shrinks with size, four rungs cannot settle: a floor arm produces a rise, a ceiling a decline, and a cross-family rung a peak.
Figures & tables
| Role | Tag as served | Parameters | Architecture | Quantization | File | Family |
|---|---|---|---|---|---|---|
| 0.75B rung | qwen3.5:0.8b | 0.75B | dense | Q8_0 | 1.3 GB | Qwen3.5 |
| 1.9B rung | qwen3.5:2b | 1.9B | dense | Q8_0 | 2.7 GB | Qwen3.5 |
| 3.7B anchor | granite4.2:3b | 3.7B | dense | Q4_K_M | 2.2 GB | Granite 4.2 |
| 4.7B rung | qwen3.5:4b | 4.7B | dense | Q4_K_M | 3.4 GB | Qwen3.5 |
| 35.5B anchor | ornith-1.5:35b | 35.5B (3B active) | MoE | Q4_K_M | 22.6 GB | Qwen3.5-MoE |
| Harness | Version | Tool-call path | Role in the design | Stop rule |
|---|---|---|---|---|
| Mingbird | v3.1.9 (56cd23d) | native JSON plus shell tools | small-model-tuned | model-decided plus executable checks |
| pi | 1.3.0 | native JSON (bash, read, write, edit) | minimal construction kit | model-decided, no checks |
| OpenCode | 1.18.23 | native JSON | product harness | model-decided, no checks |
| goose | 1.48.0 | native JSON, MCP/ACP broker | product harness | model-decided, no checks |
| mini-swe-agent | 2.4.6 | bash actions | compact bash-only control | model-decided, step limit, no checks |
| Arm | API path | Temperature | Thinking off | Context |
|---|---|---|---|---|
| Mingbird | Ollama /api/chat , direct | AGENT_TEMP = 0 | AGENT_THINK = 0 | AGENT_CTX = 32768, compaction 8000 |
| pi | OpenAI /v1 via proxy | proxy rewrites to 0 | --thinking off | server default 32768 |
| OpenCode | OpenAI /v1 via proxy | proxy rewrites to 0 | proxy sets reasoning_effort = none | server default 32768 |
| goose | Ollama provider via proxy | GOOSE_TEMPERATURE = 0, proxy rewrites | proxy sets reasoning_effort = none | server default 32768 |
| mini-swe-agent | OpenAI /v1 via proxy | temperature = 0 in the model call | proxy sets reasoning_effort = none | server default 32768 |
| Model (B) | Mingbird | pi | OpenCode | goose | mini | Span |
|---|---|---|---|---|---|---|
| Deployment reading: a cell at the wall budget scores zero | ||||||
| 0.75 | 0.664 | 0.172 | 0.112 | 0.186 | 0.012 | 0.653 |
| 1.9 | 0.593 | 0.409 | 0.295 | 0.372 | 0.029 | 0.564 |
| 3.7 | 0.762 | 0.714 | 0.000 | 0.262 | 0.279 | 0.762 |
| 4.7 | 0.968 | 0.893 | 0.632 | 0.718 | 0.150 | 0.819 |
| Rescored reading: the archived directory of a timed-out cell is scored | ||||||
| Model (B) | Baseline | Deployment reading | Rescored reading |
|---|---|---|---|
| 0.75 | pi | +0.483 [+0.256, +0.681] | +0.325 [+0.102, +0.546] |
| OpenCode | +0.546 [+0.349, +0.731] | +0.546 [+0.349, +0.731] | |
| goose | +0.508 [+0.339, +0.682] | +0.508 [+0.334, +0.679] | |
| mini | +0.652 [+0.483, +0.812] | +0.634 [+0.466, +0.790] | |
| 1.9 | pi | +0.184 [ 0.080, +0.442] | +0.180 [ 0.006, +0.371] |
| OpenCode | +0.298 [+0.063, +0.528] | +0.397 [+0.158, +0.605] |
| Model (B) | Span | Span -mini | Goose gap | Best gap | Saturated | Unsat. gap |
|---|---|---|---|---|---|---|
| 0.75 | 0.653 | 0.533 | +0.523 | +0.461 | 0 of 15 | +0.523 |
| 1.9 | 0.601 | 0.286 | +0.246 | +0.246 | 2 of 15 | +0.284 |
| 3.7 | 0.791 | 0.791 | +0.482 | 0.061 | 1 of 15 | +0.517 |
| 4.7 | 0.782 | 0.318 | +0.262 | +0.071 | 6 of 15 | +0.436 |
| Model (B) | Full | Guards off | Text only | Guards [95% CI] | Reminder | |
|---|---|---|---|---|---|---|
| 0.75 | 0.431 | 0.351 | 0.386 | [ , ] | 0.700 | |
| 1.9 | 0.822 | 0.705 | 0.784 | [ , ] | 0.241 | |
| 3.7 † | 0.910 | 0.907 | 0.812 | [ , ] | 0.833 | |
| 4.7 | 0.980 | 0.984 | 1.000 | [ , ] | 0.785 |
| Arm | Complete | Partial | None | None: finished | None: budget | None: other |
|---|---|---|---|---|---|---|
| Mingbird | 52 | 14 | 4 | 4 | 0 | 0 |
| pi | 37 | 20 | 15 | 10 | 4 | 1 |
| OpenCode | 12 | 28 | 32 | 22 | 10 | 0 |
| goose | 22 | 23 | 26 | 15 | 8 | 3 |
| mini-swe-agent | 4 | 18 | 50 | 34 | 7 | 9 |
| all | 127 | 103 | 127 | 85 | 29 | 13 |
| Rung | Mingbird | pi | OpenCode | goose | mini-swe-agent |
|---|---|---|---|---|---|
| 0.75B | 10 | 32 | – | 27 | – |
| 1.9B | 11 | 29 | 19 | 15 | – |
| 3.7B | 6 | 10 | 195 | 64 | 70 |
| 4.7B | 5 | 6 | 11 | 7 | 72 |
| Level | bpw | Mingbird | pi | Gap [95% CI] | |
| Deployment reading | |||||
| q2_K | 3.15 | 0.029 | 0.000 | [ , ] | |
| q3_K_S | 3.63 | 0.458 | 0.195 | [ , ] | |
| Q4_K_M | 4.85 | 0.822 | 0.493 | [ , ] | |
| q6_K | 6.50 | 0.860 | 0.568 | [ , ] | |
| Deployment reading, tasks both arms solve dropped | |||||
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Task | Tier | Work type | Fixtures | Principal deliverables | Checks |
|---|---|---|---|---|---|
| HWF-01 | Retrieval | materials recent literature map | 1 | recent_review.md , citation_index.md | 4 |
| HWF-02 | Retrieval | manuscript integrity audit | 1 | revised_conclusions.md , verification_checklist.md | 4 |
| HWF-03 | Synthesis | cli feature per day report | 1 | timetrack.py , usage_day.md | 4 |
| HWF-04 | Synthesis | characterization first debugging | 2 | inventory.py , regression_notes.md | 4 |
| HWF-05 | Synthesis | quarantine cleaning pipeline | 1 | tidy_data.csv , rejected.csv , sensor_rollup.csv , … | 6 |
| HWF-06 | Synthesis | hardness microstructure analysis | 1 | fig_hardness.png , fig_ductility.png , plot_hardness.py , … | 7 |
| Check type | What it tests | Count | Share |
|---|---|---|---|
| exists | the file is present | 44 | 38% |
| contains_any | one of a list of strings appears | 25 | 21% |
| python_compiles | the file parses as Python | 18 | 15% |
| min_words | a word-count floor | 13 | 11% |
| contains_groups | several groups of strings each appear | 5 | 4% |
| structure | required sections and a word floor | 3 | 3% |
LLM Agent Harnesses
LLM agent harnesses are the software and runtime systems that organize an agent’s tools, context, state, execution, and safeguards.
See all 428 papers