Harness Policy Distillation: Aligning Agent Policies with Runtime Semantics

Junze Zhu; Weihao Chen; Zhiyang Zhou; Zhen Wu; Xinyu Dai

cs.LGOct 14, 2026

Harness Policy Distillation: Aligning Agent Policies with Runtime Semantics

Authors: Junze Zhu, Weihao Chen, Zhiyang Zhou, Zhen Wu, Xinyu Dai

Organizations: National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China

Abstract

Deploying a general-purpose model as an agent means running it inside a product-specific runtime harness that decides which actions are executable, how state advances, and when the loop may terminate. Ordinary trajectory supervision shows which action was taken but leaves the runtime semantics behind that action implicit. We introduce Harness Policy Distillation (HPD), which makes these runtime control semantics an explicit training target alongside actions to align the harness policy. HPD replays reference trajectories through the harness to recover the runtime semantics that are implicit in ordinary trajectory data, and represents them as control documents at each model-call boundary. These documents provide privileged information for a teacher to generate actions, while harness verification ensures that the resulting targets are executable under the runtime contract. We jointly train a shared model on the runtime semantics recovered from control-state privileged information and native history, thereby internalizing these semantics into the model policy. At deployment, the trained model uses the original harness interface without requiring the control document or any additional supplement. On TRAJECT-Bench with the Pi harness, Qwen3-8B trained with HPD reaches 22.98% autonomous task success and 49.23% runtime compliance, against 3.09% and 19.90% for the unadapted model. Harness-verified teacher actions substantially improve the native deployment policy, while joint control-state supervision provides further gains in task completion and runtime compliance. HPD turns executable runtime semantics into dense supervision for learning harness-compatible actions at each decision boundary.

Figures & tables

Figure 1: Harness Policy Distillation. Left: agent model–harness policy mismatches. (a) Recorded actions advance replay; AutoProbe compiles control documents; teacher-model proposals are verified and ranked by reference similarity. (b) Three views train one shared model for native action prediction, document-conditioned action prediction, and document reconstruction.

Method	Supervision source	TS	F1	Step-SR	RC	Exact	Action-Acc
Base	—	3.09	26.82	26.45	19.90	2.57	53.41
SFT	Ground truth	2.06	2.40	47.20	55.92	1.72	2.06
DPO	Preference labels	7.20	33.62	11.12	25.56	5.15	72.38
GiGPO	Task reward	4.46	29.65	26.17	21.61	3.43	54.63
GRPO	Task reward	3.43	27.59	25.79	20.41	2.74	54.35
GSPO	Task reward	7.72	37.61	13.46	28.64	6.69	54.30

Table 1: Qwen3-8B comparison on TRAJECT-Bench/Pi. Scores (%) are from pi-runtime ; TS, RC, and Exact use 583 cases. HPD-T and HPD share document-conditioned, verifier-selected teacher targets; HPD-T trains only the Harness-Replay (T) view, while HPD adds I/R. Baselines are detailed in Appendix C .

Figure 2: Zero-shot transfer to BFCL v4 from above checkpoints without specific training.

HPD variant	TS	F1	Step-SR	RC	Exact	Action-Acc
HPD-T	20.58	39.96	59.25	43.74	15.61	61.40
T $\rightarrow$ I (staged)	18.01	40.23	47.99	34.99	14.24	64.07
T $\rightarrow$ R (staged)	7.03	17.47	37.29	18.35	5.15	56.64
T+I joint (without R)	20.58	39.66	56.92	41.68	14.92	62.90
T+R joint (without I)	20.58	40.63	63.13	48.03	16.81	62.80
Full HPD (T+I+R joint)	22.98	41.22	63.79	49.23	18.70	64.95

Table 2: Qwen3-8B HPD component and schedule ablation with 20k training batches per active view. T is the replay-action view; I and R are the injected- and runtime-document views. Arrows denote consecutive phases. Scores are percentages from pi-runtime .

Figure 3: Control entry and reference-conditioned continuation. At identical harness states: (a) probability of the reference harness control token; (b)

\log_{10}

of mean full-vocabulary predictive entropy in bits; (c) geometric mean reference-token probability. Continuation statistics use reference prefixes, exclude the initial control token, and cover at most 63 positions. KDEs share a bandwidth across models within each panel and integrate to one in the plotted coordinate.

Figure 4: TS and F1 across model architectures (left two panels) and separately adapted harnesses (right two panels). All scores are percentages. The Qwen3-8B/Pi Base scores match Table 1 ; its HPD scores use the 20k-optimizer-step checkpoint.

Figure 5: Task success and final-answer F1 for the Qwen3-8B Base model, Pi-Qwen HPD, and API-accessed models in the same Pi agent and evaluation setup. Values are percentages.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Setting or diagnostic	Value
Candidates per sampling attempt	$K=2$
Attempts to full raw-trajectory coverage	5
Sampling temperature	1
Vocabulary filter	distribution_top_k=0 (full vocabulary)
First-attempt acceptance rate ( $K=2$ )	87.90%
Cumulative coverage by attempt 5	100%

Appendix

Table 3: Teacher sampling settings and recorded construction diagnostics.

Setting	Value
Accelerators	$8\times$ NVIDIA A800-SXM4-80GB (81,920 MiB each)
Host resources	64 CPU cores; 1,030,000 MB RAM
Runtime and precision	CUDA 12.8; BF16 (bfloat16)
Batch size	128
Optimizer	torch.optim.AdamW(foreach=False)
Learning rate	constant $2\times 10^{-6}$

Appendix

Table 4: Training platform and optimization settings shared by HPD and all baseline training runs.

Method	TS	F1	Step-SR	RC
Base	$2.68\pm 0.23$	$26.81\pm 0.11$	$26.54\pm 0.13$	$19.62\pm 0.15$
HPD-T	$18.20\pm 1.67$	$39.50\pm 0.66$	$56.21\pm 3.43$	$42.82\pm 0.81$
Full HPD	$\mathbf{22.14\pm 1.18}$	$\mathbf{40.79\pm 0.39}$	$\mathbf{60.58\pm 3.54}$	$\mathbf{48.26\pm 0.88}$

Appendix

Table 5: Supplementary seed-study results from pi-runtime . Values are percentages, reported as mean

\pm

standard deviation over five seeds (715–719), excluding the main-run seed 42. Run-level scores exclude cases interrupted by device crashes. HPD-T retains the full teacher-construction pipeline and trains only the native-action view.

Panel of Figure 7 — Figure 7: BFCL v4 transfer on 92 cases without retraining. Panels show (a) TS, (b) final-answer F1, and (c) Step-SR for the eight methods. All axes share the same zero-based percentage scale. HPD-T and Full HPD are blue; Full HPD is hatched. Numerical labels give the scores on this subset.

Figure 8: Full HPD optimization over 60k optimizer steps (20k training rounds). Left: 200-cycle mean loss and local 10–90% range. Right: task success and final-answer F1. Base scores are TS 3.09% and F1 26.82%; trained-checkpoint labels are rounded to one decimal.

Optimizer steps	TS	F1	Action-Acc	Step-SR	RC	Exact
40k (intermediate)	19.55	39.79	63.74	58.27	43.40	15.61
60k (completed)	22.98	41.22	64.95	63.79	49.23	18.70

Appendix

Table 6: Intermediate and completed full-HPD checkpoints evaluated by pi-runtime , indexed by optimizer steps. The 60k endpoint completes 20k training rounds. Scores are percentages.

Panel of Figure 9 — Figure 9: Single-axis sampling-weight search at 20k optimizer steps. Blue circles vary Injected; orange squares vary Runtime. Dashed lines mark the shared 1:1:1 default. Each panel shows one metric with its own y-scale and labels its maximum. Lines connect evaluated settings.

Varied view	T:I:R	TS	F1	Step-SR	RC	Exact
Default	1:1:1	16.12	39.90	52.76	40.65	13.21
Injected (0.2)	5:1:5	15.95	39.18	56.40	43.57	12.52
Injected (0.6)	5:3:5	16.64	39.48	53.93	41.34	13.55
Injected (1.4)	5:7:5	14.92	39.30	51.21	39.11	12.01
Injected (1.8)	5:9:5	16.12	39.90	49.77	39.28	12.86
Runtime (0.2)	5:5:1	17.67	39.33	53.74	40.48	14.07

Appendix

Table 7: Complete single-axis sampling-weight search. Ratios give the realized T:I:R sampling cycle; all scores are percentages. Bold marks the highest score in each metric.

Model	Variant	TS	F1	Action-Acc	Step-SR	RC	Exact
Qwen3-0.6B	Base	0.00	24.78	52.06	3.64	0.00	0.00
	HPD	4.46	25.70	54.07	24.20	16.50	2.40
Qwen3-1.7B	Base	0.00	27.00	49.90	4.53	0.00	0.00
	HPD	6.52	30.50	59.00	33.60	20.80	3.30
Qwen3-4B	Base	1.03	20.81	53.40	9.81	7.38	0.86
	HPD	15.09	38.50	59.40	48.60	38.90	11.50

Appendix

Table 8: Qwen3 model-size results. Action-Acc uses the native gold-history boundary protocol separately from end-to-end evaluation. Exact retains the historical strict trajectory-success definition, not answer-string exact match. All values are percentages.

LLM Agent Training

Large language model agent training teaches models to plan, use tools, and learn from multi-step interaction with environments.

See all 586 papers

Oct 1, 2026

Harness Annealing: Learning to Act with Less External Control

Language agents rely on external harnesses to track state, organize workflows, and verify answers. Beyond providing tools and information, these harnesses supply control decisions about what to investigate, whether to revise, and when to stop. Training on successful harness-supported trajectories can improve task performance while leaving these decisions dependent on runtime intervention. We ask whether harness-supported experience can also teach the model to make these decisions, allowing the division of control to change as the model learns. We call this objective harness internalization: learning to assume specified control responsibilities while retaining task performance after the corresponding support is withdrawn. We introduce HARNESS ANNEALING TRAINING (HAT), which combines explicit control supervision with a curriculum over teacher trajectories collected under progressively weaker harnesses. Experiments with 9B and 35B models on SWE-QA and SWE-QA-Pro evaluate every checkpoint under four deployment harnesses. Selected annealed checkpoints operating with tools alone achieve scores close to those of their respective starting checkpoints deployed with the full harness. The benefits vary with model scale and deployment configuration, and further annealing does not uniformly improve performance. These findings suggest that harness-supported experience can help reduce the runtime control required by a trained agent.

Yingxuan Yang, Huacan Chai, Ying Wen

Shanghai Jiao Tong University · University of California, Berkeley

Language Model-Based Control Agent Harness Optimization LLM Agent Training

Ask PDF

Sep 21, 2026

Harness-Zero: Harness Distillation via Agent-as-Harness

Agent harnesses, the external systems that mediate model-environment interaction, can substantially improve agent performance, but their gains remain tied to the harness at deployment. Because the best harness varies across domains, instances, and models, a general-purpose agent must either settle for a suboptimal shared harness or route among an ever-growing set of specialized ones. We therefore study agent harness distillation: using a domain- or instance-optimized harness as training-time guidance and transferring the behaviors it induces into model weights, so that its gains survive under a single fixed target harness. The challenge is that the two harnesses differ in action space and available information, so guidance from the optimized harness cannot serve directly as supervision for the target one. We introduce Harness-Zero, which enables harness distillation through agent-as-harness. Guided by the optimized harness, a harnessing agent corrects student responses before execution in the target harness's action space, turning harness guidance into training demonstrations. Fine-tuning on the resulting trajectories internalizes harness-induced behavior into the model, so the specialized harness can be removed at deployment. Our experiments spanning knowledge work, tool use, and science domains show that: (1) For frontier LLMs using the same evolved harness, agent-as-harness outperforms code-as-harness. (2) With the specialized harness removed at deployment, Harness-Zero improves the base model's macro-average task success from 23.3% to 44.3%, even exceeding the 41.7% it reaches with that harness still attached. (3) Harness-Zero recovers harness-induced behaviors absent from the base model, with 82.3% average recovery across 28 patterns in the three domains.

Haoran Ye, Yuxing Lu, Haonan Dong +2

State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · College of Future Technology, Peking University · Google +1

LLM Agent Training Knowledge Distillation LLM Agent Harnesses

Ask PDF

Oct 2, 2026

Harness-Aware Distillation for Small Language Model Agents

Language model agents are deployed with a harness, the software around the model that manages its context, tools, and feedback. When such an agent is distilled into a smaller one, the harness stays in place, so the student mainly needs the teacher-specific abilities that the harness cannot provide, such as acting correctly on harness information. Standard distillation, however, imitates the teacher's full outputs and treats the harness as part of the input. We propose Harness-Aware Distillation (HAD), which focuses distillation on what the teacher adds beyond the harness. HAD complements on-policy distillation with two components: an action preference that contrasts the same teacher's actions with and without the harness information, scored after the student's own reasoning, and a validity check that drops preference pairs whose preferred action contradicts the harness records. We show that the contrast gives the student information that imitating the teacher alone cannot provide, and HAD needs no task rewards, success labels, or future information. Across multiple long-horizon agent benchmarks and models, HAD outperforms on-policy distillation baselines with the same fixed harness. Our analysis shows that HAD enters fewer unproductive loops and recovers from errors more often than the baselines, and suggests that it adaptively keeps learnable feedback in its weights while reading state information from the harness.

Moonseok Choi, Taehong Moon, Giung Nam +1

KAIST AI · Independent researcher

LLM Agent Harnesses On-Policy Distillation

Ask PDF

Abstract

LLM Agent Training

Explore similar work

Harness Annealing: Learning to Act with Less External Control

Harness-Zero: Harness Distillation via Agent-as-Harness

Harness-Aware Distillation for Small Language Model Agents