A failed gaming trajectory reveals that an agent needs intervention, but not necessarily how its execution harness should change. Effective harness adaptation therefore requires gathering diagnostic evidence, rather than merely proposing alternative recovery rules. We present KPHarness, a framework that adapts the external execution harness of a frozen vision-language-action model through knowledge-probing co-evolution. Recovery knowledge supplies context-specific intervention hypotheses that guide active probing; probe outcomes reveal when these hypotheses apply and provide evidence for refining both recovery knowledge and later probe selection. This reciprocal process allows the harness to improve not only which interventions it applies, but also how it investigates failures. KPHarness separates fast policy execution from a slow research loop that translates accumulated knowledge and probing evidence into bounded harness revisions. Each revision is compared with the incumbent under matched sandbox conditions and accepted only when the paired evaluation shows improvement. Recoveries supported by further validation are retained for cross-task reuse, while immediate interaction feedback and delayed episode-success credit update action- and knowledge-probing policies during adaptation. Experiments on Minecraft tasks spanning mining, combat, and crafting show improved task coverage over bare policy execution, a manually specified fixed harness, and knowledge-only harness evolution with non-adaptive probing. Evaluation on previously unseen tasks further shows improved task completion with the evolved harness, knowledge bank, and probing policies held fixed,
Figures & tables
Figure 1: Knowledge–probing co-evolution around a shared frozen VLA. Auto-K proposes and validates harness modifications using accumulated knowledge, while KPHarness additionally uses probing evidence and interaction feedback to adapt probe and knowledge selection. In the illustrated sand-mining example, Bare and Auto-K fail, whereas KPHarness recovers and completes the task.
Figure 2: Automatic harness evolution through knowledge–probing co-evolution. Action probes collect evidence about ambiguous failures, while knowledge probes test which recovery strategies apply. Probe evidence and accumulated knowledge guide harness proposals, which are compared with the incumbent through paired sandbox validation. Interaction outcomes improve both probing policies, and validated recoveries become transferable knowledge, while the VLA remains frozen.
Figure 3: Training curves for cold-start supervised fine-tuning (SFT), showing decreasing loss and improving token accuracy.
Figure 4: GRPO training dynamics over 400 optimization steps on 32 interactive OpenHA training tasks in Minecraft, initialized from the SFT checkpoint at step 16,200. Pass counts and trajectory scores increase, while total interaction turns decrease.
Method
Protocol
Temp.
Overall
Task success
Rescued
Knowledge search
Adaptive probing
Mine
Kill
Craft
ACCEPT
HO-win
P-only
Replay succ.
Bare
mtv1
0
26/149 (17.4%)
9/50
17/49
0/50
0
–
–
–
–
Fixed
mtv1
0
34/149 (22.8%)
16/50
17/49
1/50
8
–
–
–
–
Bare
legacy
0
23/149 (15.4%)
8/50
15/49
0/50
0
–
–
–
–
Fixed
legacy
0
31/149 (20.8%)
13/50
18/49
0/50
8
–
–
–
–
Auto-K
legacy
0
41/149 (27.5%)
20/50
20/49
1/50
18
11
3
N/A
N/A
Table 1: Overall task coverage on the 149-task OpenHA evaluation set in Minecraft.
Method
Task success
Rescued
Success rate
Knowledge search
Adaptive probing
Mine
Kill
Craft
ACCEPT
HO-win
P-only
Replay succ.
Auto-K
4.7±0.6/30
7.7±5.1/31
3.3±3.2/49
15.7±8.0/110
14.2±7.3%
–
–
N/A
N/A
KPHarness
8.3±1.2/30
13.7±1.5/31
6.7±0.6/49
28.7±3.2/110
26.1±2.9%
14.0±3.0
6.7±2.1
2.3±1.2
14.7±3.2
Table 2: Multi-seed evaluation on the same 110 tasks failed by Bare, using the legacy protocol with temperature 0.7 .
Method
Overall
Mine
Kill
Craft
Bare
8/60
7/20
1/20
0/20
Auto-K
7/60
6/20
1/20
0/20
KPHarness
15/60
9/20
5/20
1/20
Table 3: Frozen transfer to 60 unseen tasks. All components are fixed during evaluation.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Statistic
Overall
Mine
Kill
Craft
Knowledge Search Best
28/110
8
14
6
Probing-enabled Replay
17/110
4
9
4
Search Best ∪ Probing Replay
31/110
9
15
7
ACCEPT
14
4
5
5
HO-win
5
1
2
2
Strict Probe-only
3
1
1
1
Appendix
Table 4: Search-time diagnostics on the 110 OpenHA tasks failed by Bare under the primary evaluation setting. The union counts distinct tasks solved by either Knowledge Search or probing-enabled replay.
Statistic
Count
Adaptation tasks
110
Tasks using prior-task knowledge
107
Prior-knowledge selection events
840
Distinct prior-skill–target relations
156
Cross-family reuse relations
101
Single-rollout seed-harness successes
12
Appendix
Table 5: Sequential reuse of recovery knowledge in the temperature- 0.7 adaptation run, computed from initial seed-rollout metrics before task-specific proposal and acceptance.
Method
Seed
Success
Task success
Knowledge search
Adaptive probing
Mine
Kill
Craft
ACCEPT
HO-win
P-only
Replay succ.
Auto-K
101
24/110 (21.8%)
5/30
12/31
7/49
14
5
N/A
N/A
KPHarness
101
31/110 (28.2%)
9/30
15/31
7/49
14
5
3
17
Auto-K
202
8/110 (7.3%)
5/30
2/31
1/49
–
–
N/A
N/A
KPHarness
202
30/110 (27.3%)
9/30
14/31
7/49
17
9
3
11
Auto-K
303
15/110 (13.6%)
4/30
9/31
2/49
–
–
N/A
N/A
Appendix
Table 6: Per-seed results on the 110 tasks failed by Bare under the legacy protocol with temperature 0.7 . The subset contains 30 mining, 31 combat, and 49 crafting tasks.
Method / Variant
Success
Mine
Kill
Craft
(a) Full Sandbox-mediated knowledge evolution
Auto-K
24/110 (21.8%)
5/30
12/31
7/49
KPHarness
31/110 (28.2%)
9/30
15/31
7/49
(b) Static-knowledge execution diagnostic
Probe-only
15/110 (13.6%)
5/30
6/31
4/49
Static K (no injection)
9/110 (8.2%)
2/30
6/31
1/49
Appendix
Table 7: Task success on the 110-task Bare-failure subset under legacy at temperature 0.7 . Panel (a) reports the complete search pipelines; Panel (b) reports the fixed-bank execution diagnostic at seed 101.
Variant
Probes per task
Steps to recovery
Useful probes (%)
K selections
Positive-utility K (%)
Probe-only
8.00
59.4
42.7
–
–
Static K (no injection)
7.98
50.8
41.3
1,463
0.07
Static K (no K-update)
7.98
61.8
39.6
1,454
0.00
Static K (adaptive selection)
7.93
51.5
41.7
1,428
0.14
Appendix
Table 8: Process statistics for the static-knowledge diagnostic. Positive-utility K is the percentage of knowledge selections with positive immediate utility.
Figure 5: Representative execution knowledge acquired by KPHarness . Promoted recovery strategies (green), candidate knowledge (blue), recurring qualitative patterns (orange), and context-conditioned probing knowledge (purple) connect task and failure contexts to reusable interventions across mining, combat, and crafting.
Figure 6: Qualitative comparisons on mining, combat, and crafting tasks using the same frozen VLA. In these examples, Bare and Auto-K fail, while KPHarness uses adaptive probing to identify and test applicable recovery knowledge before completing the task.
Action
Execution semantics
mouseMove(dx,dy)
Move the camera horizontally and vertically. The values dx and dy determine the direction and magnitude of camera rotation.
mouseClick(left)
Attack the targeted entity or block. In an inventory GUI, pick up, place, or collect an item stack.
mouseClick(right)
Use the selected item, place a block, interact with the targeted object, or manipulate an item in an inventory GUI.
mouseClick(middle)
Execute a middle-button mouse interaction.
keyPress(w)
Move forward.
keyPress(s)
Move backward.
Appendix
Table 9: Low-level Minecraft action space used by the frozen VLA. Multiple commands may be composed within the same action step.
Figure 7: Comparison of the two VLA prompting protocols. legacy performs stateless prediction from the current observation and task instruction, whereas mtv1 restores up to ten preceding observation–action turns. Both protocols predict four 50ms low-level actions per call. System prompts are shortened for readability; complete action semantics are given in Table 9 .
Stage
Parallel GPUs
Training/interaction budget
Environment steps
Time accounting
GPU-hours
Cold-start SFT
16
16,200 updates
–
121.3 h wall-clock
1,940.3
GRPO post-training
32
400 updates
–
≥ 379.8 h cumulative active job time
≥ 12,155.0
Knowledge search and co-evolution
8
1,861 episodes
175,013
≥ 547.5 h summed worker time
≥ 547.5
Knowledge–probing ablation
8
440 episodes
41,777
124.6 h summed worker time
≥ 124.6
Frozen transfer evaluation
8
180 episodes
16,709
6.94 h job wall-clock
55.5
Total recorded allocation
≥ 14,822.9
Appendix
Table 10: Computational cost of actor training, harness evolution, and evaluation. GPU-hours are not normalized across accelerator types. Values preceded by ≥ are conservative lower bounds reconstructed from retained artifacts.
A language-model agent is jointly defined by its model and its harness, the executable program that organizes model calls, tool use, and information flow. Because different tasks call for different ways of organizing these operations, the harness needs to be adapted using feedback from the task at hand. We introduce harness learning, which trains a proposer model to revise a solver's harness using execution feedback. We formulate this process as meta-learning over executable programs, with harness revisions playing the role of weight updates in gradient-based adaptation. We train the proposer with reinforcement learning, using the task performance of revised harnesses as the reward. At test time, the proposer uses feedback from successive executions on a new task to refine the harness, without performing any parameter-space update. Experiments on reasoning and multi-hop question answering show that harness learning improves revision quality and that the ability to adapt at test time transfers to unseen tasks. Policies trained on individual revisions can continue improving harnesses over multiple rounds, while the benefits of training on revision sequences vary across settings. These findings suggest a path towards continually learning agents that turn accumulated experience into generalizable improvements.
Long-horizon terminal agents may solve a task in one run yet fail across repeated runs. Stochastic errors can propagate through actions and context, creating a gap between capability and reliable execution. We present RiseHarness, a framework for iteratively improving agent harnesses with model weights fixed. It focuses on flipping tasks with both successful and failed runs under the same configuration, using successful runs as references for failure analysis. RiseHarness combines agent trajectories with internal harness traces from a runtime monitor to examine how model actions interact with command execution, feedback, context management, and completion decisions. A Harness Handbook links this evidence to harness behaviors and source code, helping identify recurring problems in mechanisms shared across tasks. RiseHarness then searches other harness frameworks for relevant implementations and adapts them as design references to the current model and execution protocol. Candidate changes undergo behavioral checks and repeated task evaluation; versions with measured gains seed the next iteration. This loop integrates suitable mechanisms from different frameworks into a single harness and uses new execution evidence to guide further changes. We evaluate task success and reliability across models, ablate harness execution evidence and external research, and test transfer to unseen benchmarks with the same model. RiseHarness treats a specific model as the unit of harness adaptation, studying how changes to shared interaction mechanisms can make existing capabilities more reliable across tasks.
Ruhan Wang, Yucheng Shi, Zongxia Li +6
Tencent HY LLM Frontier · Indiana University · University of Maryland, College Park +2
AI agent performance depends critically on the runtime harness, comprising the prompts, tools, memory, and control flow that mediate how a model observes, reasons, and acts. Yet today's harnesses remain largely hand-crafted and static: each new model or task still demands bespoke scaffolding, and the rich traces produced during execution are rarely distilled back into systematic improvement. We introduce HarnessX, a foundry for composable, adaptive, and evolvable agent harnesses. HarnessX assembles typed harness primitives via a substitution algebra, adapts them through AEGIS, a trace-driven multi-agent evolution engine grounded in an operational mirror between symbolic adaptation and reinforcement learning, and closes the harness-model loop by turning trajectories into both harness updates and model training signal. Across five benchmarks (ALFWorld, GAIA, WebShop, tau^3-Bench, and SWE-bench Verified), HarnessX yields an average gain of +14.5% (up to +44.0%), with gains largest where baselines are lowest. These results suggest that agent progress need not come from model scaling alone: composing and evolving runtime interfaces from execution feedback is an actionable and complementary lever. The complete codebase will be open-sourced in a future release.