Harness Learning Enables Generalizable Test-Time Adaptation

Alvin Zhang; Xuecheng Liu; Zixuan Wang; Fahim Tajwar; Daman Arora; Ruslan Salakhutdinov; Daniel Khashabi; Yuda Song

cs.CLSep 28, 2026

Harness Learning Enables Generalizable Test-Time Adaptation

Authors: Alvin Zhang, Xuecheng Liu, Zixuan Wang, Fahim Tajwar, Daman Arora, Ruslan Salakhutdinov, Daniel Khashabi, Yuda Song, +1 more

Abstract

A language-model agent is jointly defined by its model and its harness, the executable program that organizes model calls, tool use, and information flow. Because different tasks call for different ways of organizing these operations, the harness needs to be adapted using feedback from the task at hand. We introduce harness learning, which trains a proposer model to revise a solver's harness using execution feedback. We formulate this process as meta-learning over executable programs, with harness revisions playing the role of weight updates in gradient-based adaptation. We train the proposer with reinforcement learning, using the task performance of revised harnesses as the reward. At test time, the proposer uses feedback from successive executions on a new task to refine the harness, without performing any parameter-space update. Experiments on reasoning and multi-hop question answering show that harness learning improves revision quality and that the ability to adapt at test time transfers to unseen tasks. Policies trained on individual revisions can continue improving harnesses over multiple rounds, while the benefits of training on revision sequences vary across settings. These findings suggest a path towards continually learning agents that turn accumulated experience into generalizable improvements.

Figures & tables

Figure 1 : Harness learning. The proposer reads the task description, the current harness $h_{t-1}$ , and its execution report, and writes a code edit $y_{t}$ , here adding answer validation and a retry. Running the revised harness $h_{t}$ yields a reward and feedback for the next round. Training updates only the proposer, and at test time both models are frozen. Fig. 2 details training.

Figure 2 : Reinforcement learning for harness revision. Given input $x_{t}$ , the proposer samples $G$ edits to the parent harness $h_{t-1}$ . Each candidate $h_{i}^{\prime}$ runs on the same scoring questions, which gives its reward $r_{i}$ and a group-normalized advantage. For successive revision states, a selected harness $h_{t}$ runs on fresh feedback questions, disjoint from the scoring questions, to form the next input $x_{t+1}$ .

	Reasoning Gym	Multi-hop QA
Models	Qwen3.5-4B proposer and solver	Qwen3-4B proposer, Qwen3-8B solver
Training	SFT on 35B-teacher revisions, then RL; Multistep RL continues from Single-step RL	RL from Base; Multistep RL trains on online revision runs
Tasks	21 SFT and 5 RL families; 21 unseen families	Train on HotpotQA; MuSiQue and 2WikiMultihopQA unseen
Revision	8 independent proposals on format-varied questions; 5 rounds of one proposal, kept only if development score improves	80 independent proposals; 4 runs of 10 rounds, 8 proposals per round
Reported	Held-out score; best of eight as an oracle	Held-out exact match; best of $N$ as an oracle

Table 1: Experimental settings. Oracle best-of-

N

statistics select candidates by held-out score.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Work	Training	Adaptation
Harness-R1	Teacher SFT followed by GRPO	Edit a frozen target agent’s executable harness from failure feedback.
JIT-Agent	Teacher SFT, execution-based DPO, supervised repair, and Evo-GDPO	Generate, repair, and evolve task-conditioned harnesses using an expanding archive.
Ornith	Joint RL for scaffolding and solving; Ornith-1.5 also trains task generation	Improve harness generation together with the task-solving policy.
Harness Learning	SFT then RL on Reasoning Gym; direct RL on QA	Revise a current harness with a fixed solver; study transfer and repeated adaptation.

Appendix

Table 2: Contemporary approaches to learned harness adaptation. Columns give each method’s training signal and adaptation process. The accompanying text compares their transfer evaluations.

Setting	Value
Development instances	25
Held-out instances	75
Independent proposals, single-step	8
Revision rounds, multistep	5
Proposals per revision round	1
Maximum generation attempts per proposal	3

Appendix

Table 3: Reasoning Gym evaluation settings. Instance counts, proposal counts, and revision rounds are per task family.

Phase	Families
SFT	advanced_geometry , binary_alternation , bitwise_arithmetic , boxnet , calendar_arithmetic , color_cube_rotation , count_bits , countdown , dice , fraction_simplification , jugs , knights_knaves , largest_island , maze , mini_sudoku , prime_factorization , products , simple_equations , syllogism , tower_of_hanoi , tsumego
RL	codeio , maze , rotten_oranges , simple_geometry , tower_of_hanoi

Appendix

Table 4: Task families used for SFT and RL on Reasoning Gym. The unseen-family evaluation excludes every family used in either phase.

Setting	Value
Data collection
Teacher	Qwen3.6-35B-A3B
Development instances per context	25 (16 generated for largest_island and tsumego )
Failed examples in the report	Up to 8, with question text and trace excerpts
Teacher proposals per context	8
Retained examples	432

Appendix

Table 5: Supervised data collection and training settings.

Family	Seed parent	Stronger parent	Total
tower_of_hanoi	100	100	200
codeio	100	100	200
rotten_oranges	100	0	100
simple_geometry	100	0	100
maze	40	0	40
Total	440	200	640

Appendix

Table 6: Single-step RL context pool. Training contexts by family and parent harness.

Setting	Value
Proposer sampling and optimization
Initialization	SFT checkpoint
Candidates per context ( $G$ )	8
Contexts per update	40
Learning rate	$10^{-6}$
Updates per sampled batch	1

Appendix

Table 7: Single-step RL settings. Solver execution settings are shared between training and evaluation.

Stage	Procedure
Initialization	Start from the reported Single-step RL checkpoint (update 10).
First phase	Train for 9 updates on 62 stored parent harnesses with room for improvement.
State refresh	Advance each parent to its best sampled revision when it improves by at least 0.01, rescore all states on the same 25 development questions, and keep states at least 0.10 below their family’s best score.
Second phase	Train for 15 updates on 246 states that pass the same filter, most drawn from earlier revision runs. The final checkpoint is reported as Multistep RL.

Appendix

Table 8: Offline training on successive revision states.

Family	Exposure	Tool loop	Base	SFT	Single-step RL	Single-step RL (later)	Teacher
color_cube_rotation	SFT	0.75	0.19 / 0.80	0.75 / 0.84	0.60 / 0.84	0.76 / 0.82	0.32 / 1.00
countdown	SFT	0.55	0.30 / 0.62	0.42 / 0.68	0.38 / 0.70	0.55 / 0.66	0.28 / 1.00
codeio	RL	0.32	0.24 / 0.29	0.17 / 0.29	0.27 / 0.34	0.30 / 0.36	–
rotten_oranges	RL	0.45	0.39 / 1.00	0.21 / 0.44	0.45 / 1.00	0.44 / 1.00	1.00 / 1.00
simple_geometry	RL	0.53	0.40 / 0.68	0.34 / 0.56	0.48 / 0.58	0.55 / 0.57	0.83 / 1.00
maze	SFT+RL	0.95	0.05 / 0.36	0.68 / 0.93	0.86 / 0.95	0.87 / 0.97	0.97 / 0.99

Appendix

Table 9: Single-step revision by task family. Entries report mean / best-of-eight scores across all eight attempted proposals on 75 held-out instances per family, and Tool loop gives the score of the fixed tool-loop harness. Single-step RL uses the selected checkpoint at update 10, later is update 16, and Teacher is the 35B model. Families marked

\dagger

are excluded from the 28-family comparison ( Appendix B.1 ). Teacher averages cover 32 families overall and 27 in the 28-family comparison.

Evaluation set	Method	Attempted	Failed edits	Zero-scoring harnesses	Combined rate
Canonical	Base	120	1	45	38%
	SFT	120	0	27	23%
	Single-step RL	120	0	18	15%
	Single-step RL (later)	120	0	14	12%
Format-varied	Base	96	4	29	34%
	SFT	96	0	21	22%

Appendix

Table 10 : Proposal validity by evaluation set. Each method attempts eight proposals per family. A failed edit yields no scorable harness ( Appendix B.1 ), and a zero-scoring harness has a recorded score of zero on 25 development instances. The combined rate is the number of failed edits plus zero-scoring harnesses, divided by attempted proposals. The canonical, format-varied, and unseen (OOD) sets contain 15, 12, and 5 families, respectively.

Method	No edit	Seed	Zero	Applied	No edit	Seed	Zero	Applied
	Format-varied (11 families)				Unseen (19 families)
Base	0	0.254	0.254	0.254	5	0.304	0.274	0.304
SFT	0	0.415	0.415	0.415	0	0.400	0.400	0.400
Single-step RL	0	0.576	0.576	0.576	0	0.501	0.501	0.501
Single-step RL (later)	0	0.551	0.551	0.551	1	0.526	0.519	0.525
Multistep RL					1	0.534	0.528	0.535

Appendix

Table 11 : Sensitivity to failed-edit scoring. Columns give held-out mean scores when a failed edit keeps the seed score (Seed), scores zero (Zero), or is excluded (Applied). No edit counts proposals that produce no revised harness. Each family has eight proposals.

Selection group	Criteria
First six families	Single-step RL outperforms SFT, which outperforms Base; the best of eight RL proposals reaches the fixed tool-loop baseline; the seed leaves room for improvement.
Remaining six	The performance-ordering requirement is relaxed to both trained proposers outperforming Base.

Appendix

Table 12: Selection of additional multistep evaluation families. The criteria use the earlier single-step results.

	Canonical		Format-varied
Method	Score	Gain over Base	Score	Gain over Base
Base	0.593		0.498
SFT	0.698	$0.105\pm 0.030$	0.688	$0.189\pm 0.058$
Single-step RL	0.699	$0.106\pm 0.044$	0.725	$0.226\pm 0.051$
Multistep RL	0.701	$0.109\pm 0.038$	0.649	$0.150\pm 0.051$

Appendix

Table 13: Held-out performance after five revisions. Mean over 24 families per question format. Gains over Base are paired within families and reported as mean

\pm

one standard error.

Family	Exposure	Seed	Base		SFT		Single-step RL		Multistep RL
			r1	r5	r1	r5	r1	r5	r1	r5
basic_arithmetic	OOD	0.49	0.56	0.58	0.49	0.92	0.49	0.87	0.79	0.91
bf	OOD	0.00	0.00	0.00	0.04	0.04	0.00	0.00	0.00	0.04
color_cube_rotation	SFT	0.17	0.29	0.33	0.80	0.88	0.80	0.80	0.71	0.82
count_primes	OOD	0.16	0.16	1.00	1.00	1.00	0.96	1.00	0.97	0.97
countdown	SFT	0.57	0.57	0.57	0.57	0.72	0.57	0.60	0.67	0.68

Appendix

Table 14 : Multistep revision by task family: canonical questions. Columns r1 and r5 score the harness retained after rounds 1 and 5 on 25 development instances per family. All methods start from the same seed, so the Seed column is shared. Exposure labels follow Table 9 . Families below the dividing line form the additional evaluation set ( Appendix C.2 ) and have one run per method. Scores for the other families average repeated runs within each family. The format-varied results continue in the next panel.

Family	Exposure	Seed	Base		SFT		Single-step RL		Multistep RL
			r1	r5	r1	r5	r1	r5	r1	r5
basic_arithmetic	OOD	0.43	0.43	0.43	0.80	0.93	0.84	0.92	0.58	0.86
codeio	RL	0.28	0.28	0.28	0.33	0.35	0.29	0.40	0.31	0.38
color_cube_rotation	SFT	0.25	0.25	0.41	0.45	0.84	0.70	0.74	0.70	0.74
count_primes	OOD	0.08	0.68	0.72	0.32	0.94	0.87	0.98	0.96	0.96
countdown	SFT	0.54	0.54	0.73	0.61	0.65	0.63	0.70	0.54	0.69

Appendix

Table 15 : Multistep revision by task family: format-varied questions (continued). Columns and evaluation protocol match the canonical panel. Families below the dividing line form the additional evaluation set.

Setting	Composition-prompt RL	Harder-task RL
Training pool	600 contexts combining low-scoring cases from the single-step RL pool ( Appendix B.3 ) with additional data; includes 100 maze and 100 sokoban contexts	694 contexts from seven families listed below
Prompt	Helper-composition directive	Canonical questions without a composition directive
Reward	Eq. 6	Eq. 9 , with revised validity terms and a structural-composition bonus
Training updates	16	14
Evaluated checkpoints	10 and 16	14

Appendix

Table 16 : Additional training configurations. Both variants use 20 contexts per update, compared with 40 in the main RL run.

	Single-step (75 held-out)		Round 5 (25 development)
Method	Format-varied	19-family set	Canonical	Format-varied
Base		0.304	0.496	0.398
SFT	0.463	0.400	0.645	0.625
Single-step RL	0.506	0.501	0.660	0.649
Single-step RL (later)		0.526
Multistep RL		0.534	0.637	0.618

Appendix

Table 17 : Training variants under single-step and multistep evaluation. Single-step columns average all eight proposals on 75 held-out instances per family. Round 5 columns report the best proposal over five rounds on 25 development instances per family, using the original twelve-family sets.

Method	$n$	Structural	Functional	Most common class
Unseen families, without a directive
SFT	39	0	0	Interpreter loop (30)
Single-step RL	40	0	0	Interpreter loop (31)
Unseen families, composition directive
SFT	24	11	4	Tool loop + helper (12)
Single-step RL	32	12	3	Interpreter loop (17)

Appendix

Table 18 : Harness composition by prompt and training setting.

Method	Without directive	With directive
Single-step RL	0.506	0.461
Composition-prompt RL (update 10)	0.526	0.559

Appendix

Table 19: Held-out scores with and without the composition directive. Scores average eight proposals per family over eleven format-varied families, each evaluated on 75 held-out instances. All methods receive the same feedback reports, and all 88 proposals in each condition have recorded scores.

Method	Checkpoint	Round-five score
SFT	Initialization	0.645
Single-step RL	10	0.660
Tool-use reward	14	0.624
Combined reward	12	0.624
Combined reward	20	0.630

Appendix

Table 20: Repeated revision after composition-reward training. Round-five development scores on twelve canonical families, using the selection protocol in Appendix B.1 and 25 development instances per family.

Component	Observed failure	Consequence
Question parsing	The parser misses the goal marker.	Questions are routed to a fallback model call.
Answer extraction	The harness returns an unsolved grid.	The submitted answer is incomplete.
Tool communication	A tool message is malformed.	The tool interaction cannot execute.

Appendix

Table 21: Observed failures in composed harnesses. Examples from the checkpoint-12 inspection.

Benchmark	Retrieval corpus	Dev pool	Test	Split source
HotpotQA	5.23M Wikipedia abstracts (full-wiki)	500	300	train split, disjoint subsets
MuSiQue	21,100 paragraphs pooled from the dev questions	500	299	dev split, hop-stratified
2WikiMultihopQA	54,957 paragraphs pooled from the dev questions	500	300	dev split, type-stratified

Appendix

Table 22: Multi-hop QA evaluation data. Retrieval corpora and question pools per benchmark.

Setting	Value
Contexts
Parent harness	Seed (Single-step); state of a revision run advanced by the policy (Multistep)
Feedback / scoring questions	6 / 64, disjoint, from a 2,000-question training pool
Revision runs $\times$ depth (Multistep)	32 $\times$ 10; commit by best-of-8 scoring exact match
Contexts per phase	320
Proposer sampling and optimization

Appendix

Table 23: RL settings on multi-hop QA. Single-step and Multistep RL share all settings unless noted. Training and evaluation use the same solver execution settings.

Dataset	Training	Seed	Mean	Oracle@8	Best-of-80	$>\!$ seed	Failed
HotpotQA	Single-step RL	0.3900	0.4105	0.5399	0.5800	71.3%	1.3%
	Multistep RL	0.3900	0.4131	0.5209	0.5833	77.5%	1.3%
MuSiQue	Single-step RL	0.1438	0.1467	0.2126	0.2408	58.8%	5.0%
	Multistep RL	0.1438	0.1551	0.2034	0.2408	67.5%	2.5%
2WikiMultihopQA	Single-step RL	0.2500	0.3090	0.4746	0.5200	67.5%	1.3%
	Multistep RL	0.2500	0.2748	0.4374	0.5433	58.8%	0.0%

Appendix

Table 24: Independent single-step QA revision. Each row summarizes 80 proposals. The

>\!

seed column gives the fraction of proposals scoring strictly above the seed, and Failed gives the failed-edit rate. Oracle@8 and best-of-80 select on held-out scores.

Explore similar work

Jul 9, 2026cs.SE

TTHE: Test-Time Harness Evolution

The behavior of an LLM agent is determined not only by the underlying model, but also by its harness: the executable program that constructs context, invokes tools, verifies intermediate results, and recovers from failures. Existing approaches optimize such harnesses before deployment, searching training or development data for a fixed agent workflow that is then frozen at test time. This limits adaptation when the test distribution, failure modes, or tool interactions differ from those seen during development. We ask whether the harness can instead be optimized during evaluation itself, using only the unlabeled execution traces the agent produces on the test inputs. We introduce Test-Time Harness Evolution (TTHE), which treats the executable harness as the state of test-time adaptation. During evaluation, TTHE maintains a population of candidate harnesses and refines them through an agentic proposer that reasons over their execution traces, without gold labels or task-specific supervision; a judge then commits an improved harness from execution-derived proxy signals, and the selected program persists to govern subsequent inputs. Crucially, TTHE does not update model weights, require gold labels, or train a separate adaptation model: solver, proposers, and judge are different roles and harnesses around the same frozen LLM, so all adaptation occurs through changes to the surrounding program. Across text-to-SQL, competitive programming, software engineering, data-science coding, and agentic tool-use tasks, TTHE improves fixed ReAct-style baseline harnesses, yielding persistent, inspectable improvements rather than a pre-searched workflow or per-query retries. These results recast test-time adaptation for LLM agents as evolution over executable control programs and identify execution-derived proxy reliability as a central challenge for robust unsupervised agent improvement.

Jun Nie, Yonggang Zhang, Jun Song +5

Hong Kong Baptist University · University of Science and Technology of China · Hong Kong Generative AI Research & Development Center, The Hong Kong University of Science and Technology +1

Large Language Model Agents Agentic Evolutionary Optimization Methods

Ask PDF

Jun 12, 2026cs.AI

HarnessX: A Composable, Adaptive, and Evolvable Agent Harness Foundry

AI agent performance depends critically on the runtime harness, comprising the prompts, tools, memory, and control flow that mediate how a model observes, reasons, and acts. Yet today's harnesses remain largely hand-crafted and static: each new model or task still demands bespoke scaffolding, and the rich traces produced during execution are rarely distilled back into systematic improvement. We introduce HarnessX, a foundry for composable, adaptive, and evolvable agent harnesses. HarnessX assembles typed harness primitives via a substitution algebra, adapts them through AEGIS, a trace-driven multi-agent evolution engine grounded in an operational mirror between symbolic adaptation and reinforcement learning, and closes the harness-model loop by turning trajectories into both harness updates and model training signal. Across five benchmarks (ALFWorld, GAIA, WebShop, tau^3-Bench, and SWE-bench Verified), HarnessX yields an average gain of +14.5% (up to +44.0%), with gains largest where baselines are lowest. These results suggest that agent progress need not come from model scaling alone: composing and evolving runtime interfaces from execution feedback is an actionable and complementary lever. The complete codebase will be open-sourced in a future release.

Tingyang Chen, Shuo Lu, Kang Zhao +11

Darwin Agent Team

Agent Harness Multi-Agent Evolution

Ask PDF

Aug 19, 2026cs.LG

Harness Continual Learning: Continual Adaptation Beyond Model Parameters

Continual learning has largely been model-centric, treating model parameters as the state that changes with sequential experience. Modern agents can also adapt through a harness of prompts, memories, tools, skills, and routing rules. Because these contents jointly shape later execution, a harness update can disrupt previously reliable behavior even when the model is frozen. This raises a new question: how can an agent continually improve its state outside the model while retaining behavior acquired earlier? We formulate Harness Continual Learning (HCL), a new continual learning paradigm in which the harness evolves around a frozen foundation model, and define the resulting loss of earlier behavior as harness-level forgetting. We instantiate HCL with four execution-facing components: the Task Interface, Experience Memory, Capability Map, and Adaptive Router. We further introduce guarded harness evolution to separate update generation from state commitment. A Continual Optimizer proposes candidate harnesses from post-execution feedback, and a Continual Evaluator commits the resulting candidate harness only after checking current improvement, historical retention, and validity. Experiments on textual reasoning, multimodal perception, and open-world interaction demonstrate capability accumulation and failure recovery, with relative gains exceeding 10% over corresponding baselines in multiple settings. Component ablations assess the contribution of each harness component, while controlled retention sweeps reveal measurable harness-level forgetting and show that the stability--plasticity trade-off can be explicitly adjusted.

Borui Kang, Jinrui Gu, Junhan Lv +3

State Key Laboratory for Novel Software Technology, Nanjing University, China · University of Wollongong, Australia

Replay-Based Continual Learning Continual Learning Catastrophic Forgetting Plasticity Open-World

Ask PDF