Mid-Harness: Scaling Actions Between Model and Harness for Terminal Agents

Minki Kang; Ryo Hachiuma; Shaokun Zhang; Subhashree Radhakrishnan; Yonggan Fu; Jindong Jiang; Mingjie Liu; Ehsan Hosseini-Asl

cs.CLSep 30, 2026

Mid-Harness: Scaling Actions Between Model and Harness for Terminal Agents

Authors: Minki Kang, Ryo Hachiuma, Shaokun Zhang, Subhashree Radhakrishnan, Yonggan Fu, Jindong Jiang, Mingjie Liu, Ehsan Hosseini-Asl, +3 more

Organizations: NVIDIA · KAIST

Abstract

Terminal agents act through stochastic model generations, yet the ability to generate a useful action does not ensure its reliable execution. A poor command (e.g., wrong package install) can change the environment in ways that hinder subsequent progress, even when the model could generate a better alternative. We investigate whether allocating test-time compute at the model-harness boundary can improve action reliability and trajectory success, and what makes this allocation effective. To study these questions, we introduce Mid-Harness, which samples and verifies candidate actions before forwarding one for execution, while keeping the generator and harness unchanged. With a TMAX-9B generator, more action sampling yields little benefit under weak verification, whereas a capable verifier can exploit useful alternatives from the same generator. On TerminalBench-Lite, a GPT-5.6 Sol verifier raises Pass@1 from 50.00% for the base agent to 68.03% with 8 sampled actions. When the same TMAX-9B model serves as the verifier, pairwise verification performs best among the evaluated verification mechanisms. Distilling responses from the stronger verifier into TMAX-9B further improves Pass@1, while leaving the action generator unchanged. With TMAX-9B on TerminalBench-Lite, combining action and trajectory scaling reaches higher success at lower estimated token cost than generating more trajectories alone. Mid-Harness also improves performance across additional models, benchmarks, and harnesses. These findings identify action scaling as a promising target for test-time compute scaling in terminal agents.

Figures & tables

Panel of Figure 1 — Figure 1: (a) Concept figure. Mid-Harness verifies candidate actions before execution. (b) Effective action verification improves trajectory success and complements trajectory scaling. TMAX-9B [ 1 ] Pass@1 (%) on TerminalBench-Lite: base versus Mid-Harness with GPT-5.6 Sol verifier (top), results from different verification mechanisms (middle), and combining Mid-Harness with Best-of- $T$ ( $T=3$ ) [ 2 ] and SR ( $R=1$ ) [ 3 ] (bottom). Mid-Harness uses $N=8$ .

Figure 2: Verification mechanism determines the return from more action candidates. Pass@1 and Pass@3 on TerminalBench-Lite with TMAX-9B.

N=1

denotes the base agent without action scaling. For frontier verifier, we use the listwise verification due to its cost.

Figure 3: Verification mechanisms trade off trajectory success against estimated decoding latency and verifier output-token cost. Pass@1 versus parallelized output tokens (left) and total verifier output tokens (right) per run on TerminalBench-Lite with TMAX-9B. Dashed lines connect

N=4

and

N=8

within each mechanism. Triangles show decision-only pairwise verification at

N=8

. Cost estimates follow Appendix B.4 .

Verifier	Score MAE	Pairwise (%)	Verification (%)
Zero-shot	2.59	59.01	38.52
Distilled	1.05	74.58	57.79

Table 1: Distillation improves frontier-verifier agreement.

Figure 4: Distillation improves teacher agreement, but command reasoning remains difficult. (a) Offline verification agreement with the frontier verifier by episode turn, using 1,355 states with valid comparisons for both models (83.4% of states). (b) Teacher disagreements judged clear verifier failures by GPT-5.6 Terra, grouped by category.

Benchmark	Model	Harness	Base agent	Mid-Harness
Benchmark	Model	Harness	Base agent	Zero-shot	Distilled
TerminalBench-Lite	Qwen3.5-9B	Terminus-2	40.48 / 60.20	42.52 $+2.04$ / 61.22 $+1.02$	–
TerminalBench-Lite	Nemotron3.5 Lightning	Terminus-2	41.16 / 55.10	43.20 $+2.04$ / 58.16 $+3.06$	–
Terminal-Bench 2.1	TMAX-9B	Vanillux2	21.72 / 25.84	27.34 $+5.62$ / 35.96 $+10.12$	26.59 $+4.87$ / 35.96 $+10.12$
Terminal-Bench 2.1	Nemotron3 Ultra	Terminus-2	50.94 / 65.17	56.18 $+5.24$ / 66.29 $+1.12$	–
SWE-bench-Verified	TMAX-9B	Vanillux2	46.67 / 54.00	48.00 $+1.33$ / 58.00 $+4.00$	48.67 $+2.00$ / 62.00 $+8.00$

Table 2: Mid-Harness transfers across benchmarks, models, and harnesses. Pass@1 / Pass@3 over three runs per task. SWE-bench-Verified uses its Mini subset (50 tasks), and FeatureBench-Mini uses 23 CPU tasks. Both Mid-Harness variants use pairwise verification with

N=8

. Green subscripts

+\Delta

show differences against the base agent.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	$N=4$	$N=8$
Generator	TMAX-9B	TMAX-9B
Generator thinking / temperature	On / 0.8	On / 0.8
Maximum agent steps	64	64
Generator context / output limit	65,536 / 16,384	65,536 / 16,384
Verifier	TMAX-9B, zero-shot or distilled
Verifier thinking / temperature	Off / 0	Off / 0

Appendix

Table 3: Operating settings for default pairwise verification with TMAX-9B.

Setting	Value
LoRA rank / alpha	64 / 128
LoRA dropout	0.05
Learning rate	$10^{-4}$
Epochs	2
Per-device batch size	8
Gradient accumulation	1

Appendix

Table 4: Pairwise verifier distillation settings.

Figure 7: Pairwise verifier input lengths. Distribution across all verifier calls with token-usage records from 294 runs on 98 TerminalBench-Lite tasks, using the distilled TMAX-9B verifier at

N=8

K=4

. Vertical lines mark the median and 90th percentile.

Figure 8: Terminal-agent evaluation requires many sequential steps and substantial execution time. Task-level mean steps and agent execution time for TMAX-9B with Vanillux2 and

N=1

, sorted independently within each benchmark and panel. Each task averages three runs.

Verifier / configuration	$N$	Pass@1	Pass@3
Base agent	1	50.00	69.39
First-runnable proxy	8	49.66	66.33
Zero-shot listwise	4	49.32	66.33
Zero-shot listwise	8	51.02	67.35
Zero-shot pointwise	4	52.72	68.37
Zero-shot pointwise	8	52.38	67.35

Appendix

Table 5: Action-scaling performance underlying the main plots. Zero-shot verifiers use TMAX-9B. The frontier verifier is GPT-5.6 Sol.

Configuration	Pass@1	Pass@3
Base agent	50.00	69.39
Best-of- $T$ , $T=3$	55.10	–
Best-of- $T$ , $T=5$	57.14	–
Best-of- $T$ , $T=7$	59.18	–
SR, $R=1$	55.10	71.43
SR, $R=2$	56.46	70.41

Appendix

Table 6: Trajectory-scaling and composition performance underlying the main plots. Mid-Harness uses distilled pairwise verification with

N=8

. Best-of-

T

returns one output per task, so Pass@3 is undefined and shown as –.

Generator	Verifier	Pass@1		Pass@3
Generator	Verifier	Reasoning $\to$ decision-only	$\Delta$	Reasoning $\to$ decision-only	$\Delta$
TMAX-4B	Zero-shot	$41.50\to 38.10$	$-3.40$	$57.14\to 50.00$	$-7.14$
	Distilled	$43.88\to 41.50$	$-2.38$	$58.16\to 57.14$	$-1.02$
TMAX-9B	Zero-shot	$54.76\to 56.12$	$+1.36$	$71.43\to 73.47$	$+2.04$
	Distilled	$57.14\to 59.18$	$+2.04$	$75.51\to 73.47$	$-2.04$
TMAX-27B	Zero-shot	$73.13\to 71.77$	$-1.36$	$84.69\to 83.67$	$-1.02$

Appendix

Table 7: Decision-only versus reasoning-based pairwise verification at

N=8

K=4

. Pass@1 and oracle Pass@3 (%) on TerminalBench-Lite. Arrows show reasoning

\to

decision-only, and

\Delta

is decision-only minus reasoning in percentage points.

Figure 9: Decision-only action scaling and its composition with trajectory scaling. Pass@1 versus total output tokens, input tokens, and reference-priced token cost on TerminalBench-Lite with TMAX-9B. All Mid-Harness points use decision-only action verification. Compositions use the distilled verifier at

N=8

. Costs include the complete pipeline, using the accounting and rates in Appendix B.4 .

Generator	Configuration	Pass@1 (%)	Cost (USD)	Cost reduction
TMAX-4B	Zero-shot Mid-Harness	$41.50\to 38.10$	$0.125\to 0.076$	39.2%
	Distilled Mid-Harness	$43.88\to 41.50$	$0.134\to 0.071$	46.8%
TMAX-9B	Zero-shot Mid-Harness	$54.76\to 56.12$	$0.174\to 0.138$	20.9%
	Distilled Mid-Harness	$57.14\to 59.18$	$0.207\to 0.157$	24.1%
	+ Best-of- $T$ , $T=3$	$66.33\to 65.31$	$0.797\to 0.608$	23.7%
	+ SR, $R=1$	$60.20\to 65.99$	$0.399\to 0.309$	22.4%

Appendix

Table 8: End-to-end cost changes at

N=8

. Arrows compare verification with explicit reasoning to decision-only verification on TerminalBench-Lite. Costs are reference-priced dollars per final output as used in Figure 6 .

Variant	Reference	Pass@1 (%) ref. $\to$ variant	Pass@3 (%) ref. $\to$ variant	Verifier output ( $\times$ reference)
Five responses per pair	Distilled, one response	$57.14\to 53.40$	$75.51\to 71.43$	$4.99\times$
Thinking enabled	Zero-shot pairwise	$54.76\to 56.12$	$71.43\to 71.43$	$4.86\times$
Rubric scaling	Zero-shot pairwise	$54.76\to 55.10$	$71.43\to 72.45$	$2.26\times$

Appendix

Table 9: Additional verifier inference on TerminalBench-Lite. Arrows compare each variant with its indicated reference. Verifier output ratios use total completion tokens across all comparison records for both the reference and variant.

Model	Mid-Harness	$\Delta$ Pass@1	95% CI
4B	Zero-shot	+2.72	[-3.06, +8.84]
4B	Distilled	+5.10	[-0.68, +10.88]
9B	Zero-shot	+4.76	[-0.68, +10.20]
9B	Distilled	+7.14	[+1.36, +12.93]
27B	Zero-shot	+2.04	[-3.74, +7.48]
27B	Distilled	+5.10	[+0.34, +9.86]

Appendix

Table 10: Pass@1 differences from the base agent on TerminalBench-Lite. Differences and confidence intervals are in percentage points. Mid-Harness uses pairwise verification with

N=8

and

K=4

Figure 10: Action-scaling gains vary across task groups and model sizes. Pass@1 changes in percentage points relative to the base agent on TerminalBench-Lite. Z and D denote zero-shot and distilled Mid-Harness with $N=8$ and $K=4$ . Panels group tasks by domain, difficulty, and base-agent execution length. Domain and difficulty groups are shared across models, while length groups are defined separately for each model.

Model	Short	Medium	Long
4B	2–17 (34)	18–24 (37)	25–64 (27)
9B	4–15 (33)	16–24 (32)	25–64 (33)
27B	3–10 (39)	11–18 (26)	19–64 (33)

Appendix

Table 11: Base-agent execution-length groups. Each cell gives the range of task-median shell-call counts and the number of tasks in parentheses.

Figure 11: Score levels and within-pair gaps approach the teacher’s distributions. Each candidate-score distribution contains 20,394 observations, and each gap distribution contains 10,197 pairs.

Figure 12: Distillation aligns comparative scores with the teacher. Joint A/B score distributions on 10,197 common valid pairs. All panels use the same percentage color scale.

Explore similar work

Scaling Test-Time Compute for Agentic Coding
Apr 16, 2026Joongwon Kim, Wannan Yang, Kelvin Niu +13Long-Horizon Agents Coding Agents
1Meta Superintelligence Labs · University of Washington · 3New York University +4
VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks
Oct 1, 2026Caiqi Zhang, Rujun Han, Zifeng Wang +4Long-Horizon Agents Long-Horizon Task Planning
University of Cambridge · Google Cloud AI Research
Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows
May 27, 2026Yilun Yao, Xinyu Tan, Chao-Hsuan Liu +9Agent Harness Agentic Benchmarks
1Peking University · 2Qiyuan Tech

Apr 16, 2026, cs.SEJ/K move · Enter open · S save

Scaling Test-Time Compute for Agentic Coding

Joongwon Kim, Wannan Yang, Kelvin Niu, Hongming Zhang+12

1Meta Superintelligence Labs · University of Washington · 3New York University+4

Long-Horizon Agents Coding Agents Test Time Scaling Inference-Time

Open paper AskPDF arXiv

Test-time scaling has become a powerful way to improve large language models. However, existing methods are best suited to short, bounded outputs that can be directly compared, ranked or refined. Long-horizon coding agents violate this premise: each attempt produces an extended trajectory of actions, observations, errors, and partial progress taken by the agent. In this setting, the main challenge is no longer generating more attempts, but representing prior experience in a form that can be effectively selected from and reused. We propose a test-time scaling framework for agentic coding based on compact representations of rollout trajectories. Our framework converts each rollout into a structured summary that preserves its salient hypotheses, progress, and failure modes while discarding low-signal trace details. This representation enables two complementary forms of inference-time scaling. For parallel scaling, we introduce Recursive Tournament Voting (RTV), which recursively narrows a population of rollout summaries through small-group comparisons. For sequential scaling, we adapt Parallel-Distill-Refine (PDR) to the agentic setting by conditioning new rollouts on summaries distilled from prior attempts. Our method consistently improves the performance of frontier coding agents across SWE-Bench Verified and Terminal-Bench v2.0. For example, by using our method Claude-4.5-Opus improves from 70.9% to 77.6% on SWE-Bench Verified (mini-SWE-agent) and 46.9% to 59.1% on Terminal-Bench v2.0 (Terminus 1). Our results suggest that test-time scaling for long-horizon agents is fundamentally a problem of representation, selection, and reuse.

Abstract

Explore similar work

Scaling Test-Time Compute for Agentic Coding

VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks

Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows