Mingbird: A Local-First Agent Harness Enabling Small Open Models to Complete Real Tasks

Hao Wang; Ting Huang

cs.AIOct 1, 2026

Mingbird: A Local-First Agent Harness Enabling Small Open Models to Complete Real Tasks

Authors: Hao Wang, Ting Huang

Organizations: School of Mathematics and Physics, University of Science and Technology Beijing Beijing, China · Honor Device Co., Ltd. Beijing, China

Abstract

Small open-weight models (2-9B) run on ordinary laptops, but under cloud-scale agent harnesses they rarely complete real tasks: tool prefill overflows the context, self-correction diverges, tool demonstrations loop, and tasks are silently abandoned. We present evidence, from a controlled single-machine comparison and one third-party benchmark, that a substantial share of these failures is attributable to the harness rather than the model. We introduce Mingbird, a local-first agent harness for Windows and Ollama whose ten mechanisms compensate point-by-point for small-model failure forms, three of them representative: a byte-level net-zero prefill budget, a finish gate that re-reads the task before accepting completion, and signature-level loop detection. On LRAB, a controlled comparison holding machine, models, budgets, and scoring fixed (4 harnesses $\times$ 4 open models (2B-35B) $\times$ 18 real tasks, deterministic artifact scoring), Mingbird reaches 0.886 overall against 0.631 (goose), 0.479 (opencode), and 0.405 (agent-mini), with all 288 cells published; on $τ^2$ -bench (278 tasks, three arms, one protocol) it totals 0.856 against 0.791 and 0.737; and a frontier-model probe on the same 18 tasks spans 0.997 to 0.478 across harnesses, with well-formed scaffolds staying within 0.072 of each other. A leave-one-mechanism-out ablation is reported as directional only: same-night replications of the same arm move its mean by up to 0.069, the size of every nominal single-trial delta, and the one batch-matched comparison (full mechanism stack versus text re-read alone) gives the executable completion guards a paired +0.10 across three replications. The evidence carries stated limits: a self-built benchmark, a single machine, and single-trial scoring.

Figures & tables

Panel of Figure 1 — Figure 1: Mingbird architecture. Mechanism identifiers (M1–M10) refer to Section 3.2 ; the five-ring safety model (M14) is a policy layer over these components and is not drawn. All components run locally on the user’s machine; the harness talks to the Ollama API and, when the user explicitly configures one, an external search endpoint. In the shipped default configuration, no other outbound traffic exists; the evaluation infrastructure of Sections 4.3 and 5.4 adds a local protocol proxy and, for the $\tau^{2}$ comparison, a cloud-hosted user simulator and judge.

	Mechanism	Failure form addressed
M1	Flat prefill, net-zero budget	context overflow from schema-heavy tool prefill
M2	Chat/task layering	tool-demonstration loops from casual chat input
M3	Finish gate + plan re-injection	premature or false completion claims
M4	Signature-level anti-loop	repeated identical tool calls, no output
M5	False-finish and format rescue	tool calls emitted as plain text; repeated malformed calls
M6	Edit safety (.bak)	destructive edits the model cannot undo

Table 1: Mingbird mechanisms (M1–M10, plus the v1.6.0 five-ring safety model M14) and the small-model failure forms they address. Each mechanism is expanded below.

Figure 2: Prefill cost versus measured prompt length (log–log).

x

is the backend-reported prompt-token count ( prompt_eval_count );

y

is the backend’s prompt-evaluation time, median of three runs with a per-request nonce defeating prefix caching. Per-run values were not persisted, so error bars are omitted. Hollow markers: the bench (19 tools), full-configuration (31) and duplicated-schema (256) anchors. Dashed lines are least-squares fits (4B: 0.20 ms per token,

R^{2}=0.9991

). The 774/797-token static budgets quoted in the text are harness estimates, not measurements, and are not plotted on this axis.

Ring	Model	$n$	att.	intc.	intc. rate	flags	conf.
1	gemma4:e2b	7	6	4	0.67	2	0
1	qwen3.5:4b	7	7	5	0.71	6	3
2	gemma4:e2b	9	9	9	1.00	0	0
2	qwen3.5:4b	9	9	9	1.00	0	0
3	gemma4:e2b	9	9	9	1.00	2	0
3	qwen3.5:4b	9	9	8	0.89	3	1

Table 2: Five-ring adversarial evaluation: one run per case, two models, unattended, temperature 0, thinking off. “attempted” counts cases with evidence the agent tried the dangerous operation; “intercepted” counts cases where a ring’s own refusal marker fired; “flags” is the scorer’s violation column before verification (ground-truth hits plus danger-pattern alerts); “confirmed” counts verified end-to-end effects — ground-truth channel hits plus human-verified alerts, each with a published disposition. Per-case CSVs, dispositions and the audit script: benchmarks/safety/results_2609/ .

Arm	Window	Temperature 0	Thinking off
agent-mini	09-01..04	own config	own config
Mingbird	09-13..14	released default	released default
goose, opencode	09-18..20	transport proxy	transport proxy

Table 3: Collection windows of the published matrix. Every published cell comes from exactly one window. Protocol enforcement differs by arm because the harnesses expose different configuration surfaces; the enforced protocol itself (temperature 0, thinking disabled) is the same.

Tier	Ollama tag	Total	Active	Quantization	Architecture
2B	gemma4:e2b	5B	2.3B	Q4_K_M	MatFormer + PLE (elastic)
4B	qwen3.5:4b	4B	—	Q4_K_M	dense
12B	gemma4:12b	12B	—	Q4_K_M	dense
35B	ornith-1.5:35b	35B	${\sim}3$ B	Q4_K_M	Qwen3.5-MoE, multimodal

Table 4: The four models used in LRAB, as served by Ollama. Total and Active give the total and per-token activated parameter counts; Quantization is the ggml quantization level of the served weights. All four tags are served as Q4_K_M. The families are public releases: gemma4 by Google [ 3 ] , qwen3.5 by Alibaba [ 55 ] , and ornith-1.5 by Ornith AI [ 54 ] .

Harness	2B	4B	12B	35B	Overall
Mingbird	0.821	0.876	0.906	0.941	0.886
goose	0.271	0.801	0.772	0.679	0.631
agent-mini	0.246	0.706	0.576	0.092	0.405
opencode	0.017	0.465	0.539	0.896	0.479

Table 5: LRAB means per model size (18 tasks each) and overall (72 cells each). Every cell of the underlying

4\times 4\times 18

matrix is published; each number below is the mean of 18 rows of the raw CSV. Column labels follow the parameter tiers of Table 4 (named by the Ollama tags; total and activated parameter counts are given explicitly there). Two Mingbird cells require note: WF-08 at 35B ended without artifacts and scores 0, and WF-09 at 4B ended in a runner error and scores its partial artifacts (0.5).

Figure 3: LRAB mean score by parameter tier for the four harnesses (18 tasks per point; same machine, Ollama backend, budgets, and scorer; only the harness differs). An asterisk marks tiers where the Mingbird advantage over all three competitors is significant after Holm correction (two-sided paired Wilcoxon tests over the 18 task pairs; largest adjusted

p

per marked tier: 0.0008 at 2B, 0.0466 at 12B); these are the 2B and 12B tiers (12B vs. goose is marginal once tasks are clustered into families, Section 5.1 ). Error bars are task-level bootstrap 95% confidence intervals; at 4B the Mingbird interval overlaps goose’s (which reaches 0.801 with thinking disabled) and that comparison does not reach significance, and at 35B the opencode comparison (0.896) does not reach significance either. The Mingbird 35B point (0.941) includes one artifact-free run scored 0 (WF-08). Mingbird is the only harness whose curve does not collapse at 2B.

Figure 4: LRAB task-level heatmap: 18 tasks (rows; LH-01–LH-03 on top, WF-01–WF-15 below, separated) $\times$ four harnesses (columns); each cell is the mean score over the four model tiers. The Mingbird column is outlined. The figure makes the failure structure visible rather than hiding it behind a single mean: Mingbird’s weakest task is WF-09 (0.70), which contains its runner-error cell, while opencode averages 0.036 on LH-01, and on WF-09 agent-mini and goose average 0.071 and 0.643.

Figure 5: Empirical CDF of per-cell scores for each harness over its 72 LRAB cells (staircase at the observed scores; the same cells underlie Table 5 ). 94% of Mingbird cells score at least 0.5, and its single zero is WF-08 at 35B, a run that ended without artifacts; opencode has 27 zero cells, agent-mini 23, and goose 13. The distributions, not only the means, separate the harnesses.

Figure 6: LRAB score versus wall-clock time per cell (one point per published CSV row; the single goose cell with no recorded wall-clock is excluded, leaving 287 points; logarithmic time axis; color encodes the harness, marker size the model tier). Dashed vertical lines mark the per-cell budgets (90 minutes for workflow cells, 180 for long-horizon); a cell that exhausts its budget scores 0. The fast-but-empty cluster below 80 seconds contains agent-mini’s 35B platform-mismatch cells, while the slow-but-substantive tail is dominated by goose. The figure is the visual form of the budget-pricing protocol of Section 4.5 : neither axis alone earns credit.

Harness	retail (114)	airline (50)	telecom (114)	r+a † (164)	Total † (278)
Mingbird	0.763	0.740	1.000	0.756	0.856
native LLM agent	0.675	0.740	0.930	0.695	0.791
opencode	0.588	0.500	0.991	0.561	0.737

Table 6:

\tau^{2}

-bench results under the unified protocol (temperature 0, thinking off, transport-pinned): three domains, one model. The agent side is always the same local qwen3.5:4b and the user simulator always the same cloud model ( qwen3.8-flash ). pass@1, single trial; errored trials score 0 (the three completed arms finished with zero errored trials); scoring verifies the database end state. retail covers the 114 runnable tasks of the official 115-task suite; airline and telecom are complete. r+a is the retail+airline aggregate over the 164 less-saturated tasks. † derived: the per-task mean over the stated task set, recomputable from the published per-trial data. goose is deferred: under the same protocol its throughput on this benchmark is 40–113 minutes per task (a token audit attributes this to nested sub-agent loops,

{\sim}

4.85M input tokens per task vs. 60–400K for the other arms; its legacy thinking-on numbers 0.588/0.460/0.377 are superseded).

Figure 7:

\tau^{2}

-bench pass@1 rates by domain and arm (retail 114, airline 50, telecom 114 tasks; the agent side is always the same local qwen3.5:4b at temperature 0 with thinking off, and the user simulator always the same cloud model). Whiskers are task-level bootstrap 95% confidence intervals. The three completed arms finished with zero errored trials. telecom is near-saturated (1.000 / 0.991 / 0.930); the goose arm is deferred for throughput and not shown.

Figure 8: Efficiency view of the

\tau^{2}

results: pass@1 versus median wall-clock per domain–arm pair (logarithmic time axis; markers encode the domain, color the arm), with the Pareto frontier drawn dashed. The frontier runs Mingbird retail (0.763 at 58 s)

\rightarrow

the benchmark’s native agent in telecom (0.930 at 74 s)

\rightarrow

Mingbird telecom (1.000 at 132 s): the native telecom point trades score for speed, while opencode’s retail point (0.588 at 107 s, its airline point 0.500 at 103 s nearly coincident) is slower and lower than Mingbird’s retail point on both axes. The off-frontier corners are as informative as the frontier itself. Regenerated from the unified-protocol three-arm data.

Harness	WF (15)	LH (3)	Total (18)	notable cells
Mingbird	1.000	0.980	0.997	only miss is the LH-02 cap
goose	0.990	0.980	0.989	WF-12 0.857; clears WF-08 in 776 s
opencode	0.913 †	0.980	0.925 †	WF-03 0.0; WF-08 timeout
agent-mini	0.490	0.415	0.478	second-scale early exits persist

Table 7: Frontier-model probe over all 18 LRAB tasks: the same hosted qwen3.8-flash (temperature 0, thinking off, via a local Ollama shim) drives all four harnesses. Deterministic scoring as in the main matrix. † opencode’s WF mean is over 14 scored cells: WF-08 times out at the 60-minute budget (the agent starts the task’s mock server in the foreground and blocks on it); counting that timeout as 0 gives a total of 0.874, leaving the ranking unchanged. LH-02 scores 0.939 under every harness — a task-side cap, not harness variance.

Failure form	Mingbird	goose	opencode	agent-mini
Platform exit (crash / error)	1	1	3	0
Budget exhausted (timeout)	0	1	0	0
Provider disconnect	0	0	0	8
Never invoked a tool	0	0 †	21	0
Stopped short (final score $<0.5$ , no crash)	4	23	32	44
Incomplete delivery (a product check failed)	26	44	53	61

Table 8: Failure forms coded on all 288 published cells: number of cells showing each form (a cell can show more than one). The upper block is coded from recorded fields ( failure_mode , exit_code , artifact-check details) and is comparable across arms; the lower block counts harness log markers in the archived transcripts and has no counterpart in the baselines ( n/a ). Coding script, per-cell CSV and the two-coder agreement check: benchmarks/failure_forms/ .

Arm	Total	$\Delta$ vs. baseline
full mechanism set (baseline)	0.821	—
$-$ finish gate	0.723	$-0.098$
$-$ verification feedback	0.772	$-0.048$
$-$ anti-loop	0.805	$-0.015$
$-$ flat prefill	0.818	$-0.003$

Table 9: Leave-one-mechanism-out ablation (v1.5.0 code base, LRAB task set, gemma4:e2b ,

n=18

per arm, one mechanism disabled per arm via an environment gate). The baseline is the same-code full-mechanism arm; these are the same 18 cells that form the 2B column of Table 5 , so the ablation and the headline results share one batch and one code version. Deltas are exact per-task paired differences and can differ in the last digit from the difference of the rounded totals shown.

Figure 9: Leave-one-mechanism-out ablation (v1.5.0 code base, LRAB task set, gemma4:e2b ,

n=18

per arm; the baseline is the same-code full-mechanism arm of Table 9 ). Left: mean score change per disabled mechanism with bootstrap 95% confidence intervals over 10,000 task-level resamples (finish gate

[-0.266,+0.063]

, verification feedback

[-0.185,+0.076]

, anti-loop

[-0.134,+0.092]

, flat prefill

[-0.119,+0.093]

; every interval crosses zero, so the effects are directional; the finish-gate bar alone is colored, marking the largest single delta — the coloring carries no separate statistical status). Right: per-task changes, colored by task family (LH versus WF). Much of the LH-segment decline across the four arms is carried by the common-cause shift of LH-01, which falls in every arm (sensitivity in Section 5.6 ); under this design, mechanism-level readings of the family split are not separable. Nor is the WF family unchanged: disabling verification feedback leaves the WF mean

+0.033

(slightly better), and the finish-gate arm’s WF mean is

+0.074

Explore similar work

May 12, 2026cs.SE

It's Not the Size: Harness Design Determines Operational Stability in Small Language Models

This paper experimentally analyzes how the level of harness engineering affects the operational performance of small language models (SLMs, 2-3B parameters). Three harness conditions - model-only (raw prompt), minimal-shell (wrapper tags), and a 4-stage pipeline (plan->execute->verify->recover) - are applied to three models (Gemma4 E2B, Qwen3.5:2B, LLaMA 3.2 3B) across 24 tasks, comparing Task Success Rate (TSR) and Valid TSR (VTSR). The pipeline harness achieves TSR=0.952 and VTSR=1.000 on Gemma4 E2B (T1-T5, 21 tasks). A non-monotonic phenomenon - minimal-shell TSR < model-only TSR - is observed in two models. In LLaMA 3.2 3B model-only, seven format violations yield TSR=0.429, revealing scaffold collapse: the model abandons JSON structure under complex format requirements without harness support. Ablation shows planning and recovery each contribute approximately 24.7% of total gain. VCR (Verification Catch Rate)=0.625 across all pipeline runs.

Yong-eun Cho

KailosLab, Seoul, Republic of Korea

Small Large Language Models Task Success Rate

Ask PDF

May 27, 2026cs.AI

Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows

LLM agents are increasingly deployed as executable systems that use tools, modify workspaces, and produce concrete artifacts. In such workflows, performance depends not only on the base model, but also on the harness: the system layer that manages context, tools, state, constraints, permissions, tracing, and recovery. However, existing benchmarks typically abstract away execution, compare complete agent systems, or hold the harness fixed, making execution-layer variation difficult to study. We introduce Harness-Bench, a diagnostic benchmark for evaluating configuration-level harness effects in realistic agent workflows. Harness-Bench evaluates representative harness configurations across multiple model backends under shared task environments, budgets, and evaluation protocols, while preserving each harness's native execution behavior. The benchmark contains 106 sandboxed offline tasks constructed from practical agent-use patterns and manually reviewed for realism, solvability, oracle-checkability, and integrity. Each run records final artifacts, execution traces, usage statistics, and validator outputs, enabling analysis beyond final completion. Across 5,194 execution trajectories, we observe substantial variation in completion, process quality, efficiency, and failure behavior across model-harness pairings. These results suggest that agent capability should be reported at the model-harness configuration level rather than attributed to the base model alone. Our analysis further identifies recurring execution-alignment failures, where plausible reasoning becomes decoupled from tool feedback, workspace state, evidence, or verifiable output contracts. Harness-Bench provides a reproducible foundation for diagnosing and improving reliable, efficient, and auditable agent execution stacks.

Yilun Yao, Xinyu Tan, Chao-Hsuan Liu +9

1Peking University · 2Qiyuan Tech

Agent Harness Agentic Benchmarks Agentic Workflow Design

Ask PDF

May 1, 2026cs.AI

AgentFloor: How Far Up the tool use Ladder Can Small Open-Weight Models Go?

Production agentic systems make many model calls per user request, and most of those calls are short, structured, and routine. This raises a practical routing question that existing evaluations do not directly answer: which parts of an agent workflow truly require large frontier intelligence, and which can be handled by smaller models? We introduce AgentFloor, a deterministic 30-task benchmark organized as a six-tier capability ladder, spanning instruction following, tool use, multi-step coordination, and long-horizon planning under persistent constraints. We evaluate 16 open-weight models, from 0.27B to 32B parameters, alongside GPT-5 across 16,542 scored runs. Our results reveal a clear boundary of model necessity. Small and mid-sized open-weight models are already sufficient for much of the short-horizon, structured tool use work that dominates real agent pipelines, and in aggregate, the strongest open-weight model matches GPT-5 on our benchmark while being substantially cheaper and faster to run. The gap appears most clearly on long-horizon planning tasks that require sustained coordination and reliable constraint tracking over many steps, where frontier models still hold an advantage, though neither side reaches strong reliability. We also find that this boundary is not explained by scale alone: some failures respond to targeted interventions, but the effects are model-specific rather than universal. These findings suggest a practical design principle for agentic systems: use smaller open-weight models for the broad base of routine actions, and reserve large frontier models for the narrower class of tasks that truly demand deeper planning and control. We release the benchmark, harness, sweep configurations, and full run corpus.

Ranit Karmakar, Jayita Chatterjee

Harvard University, Boston, MA 02115 · Boston, MA 02115

Small Reasoning Model Production Agentic Systems Frontier Models Ladder

Ask PDF