Act First, Reason Later: Accelerating On-Policy Distillation for Multi-Turn Agents via Reference-Conditioned Inverse Dynamics

Zubin Zheng; Jiahao Wu; Shaofeng Zhang; Zhirui Zhang; Yew-Soon Ong; Shengcai Liu

cs.LGSep 29, 2026

Act First, Reason Later: Accelerating On-Policy Distillation for Multi-Turn Agents via Reference-Conditioned Inverse Dynamics

Authors: Zubin Zheng, Jiahao Wu, Shaofeng Zhang, Zhirui Zhang, Yew-Soon Ong, Shengcai Liu

Organizations: Guangdong Provincial Key Laboratory of Brain-Inspired Intelligent Computation, Department of Computer Science and Engineering, Southern University of Science and Technology · DeepCybo · Hong Kong Polytechnic University · Zhongguancun Academy ·

Abstract

On-policy distillation (OPD) trains multi-turn language agents with dense teacher supervision on student-generated responses. However, standard think-then-act rollouts require lengthy reasoning before each short action, delaying environment transitions and experience collection. Generating actions directly reduces this delay but can degrade rollout quality. To address this, we propose ActFirst-OPD, an act-first, reason-later training framework that decouples environment interaction from full-response generation. The student infers and executes actions through reference-conditioned inverse dynamics using its current interaction context and a reference next observation, and switches to autonomous next-action prediction when the resulting transition deviates from the reference trajectory. From the collected interaction contexts, the student asynchronously generates full think-then-act responses for token-level teacher supervision. Experiments across 0.6B-, 1.7B-, and 4B-parameter Qwen3 students show that ActFirst-OPD achieves average wall-clock training speedups of $2.3\times$ on ALFWorld, $1.8\times$ on WebShop, and $4.9\times$ on ScienceWorld over Vanilla OPD. It matches or exceeds all compared OPD baselines in mean task success rate across eight of nine benchmark-model settings. These results demonstrate that reasoning need not block acting during multi-turn agent distillation.

Figures & tables

Figure 1: Rollout latency and quality on ALFWorld with Qwen3-1.7B. Ranges in (a) denote the 5th–95th percentiles. (c) shows GPU utilization during the first 10 minutes of training. In (d–f), all rollout strategies use the same frozen Vanilla OPD checkpoint at each update step, with the same 3,553 training tasks, and a 30-turn limit. (d) shows selected turns from the same task at update 1.

Figure 2: Overview of ActFirst-OPD. Top: Reference-conditioned inverse dynamics generates fast actions $\tilde{a}_{t}$ for environment interaction, with autonomous next-action prediction once the rollout diverges from the reference trajectory. Bottom: The same student asynchronously generates full responses $y_{t}=(z_{t},a_{t})$ from collected interaction contexts for teacher supervision.

Qwen3 Series
Model		Method	ALFWorld
Teacher	Student		Pick	Look	Clean	Heat	Cool	Pick2	Seen		Unseen		All		Training Wall-clock Time (Speedup)
Teacher	Student		Pick	Look	Clean	Heat	Cool	Pick2	SR $\uparrow$	Round $\downarrow$	SR $\uparrow$	Round $\downarrow$	SR $\uparrow$	Round $\downarrow$	Training Wall-clock Time (Speedup)
-	0.6B	Zero-Shot	0.00	6.45	0.00	0.00	0.00	0.00	0.00	30.00	1.49	29.69	0.73	29.85	N/A
-	1.7B		40.68	16.13	13.79	12.82	8.70	4.88	17.86	27.68	17.16	27.28	17.52	27.48	N/A
-	4B		89.83	77.42	60.34	58.97	28.26	63.41	55.71	21.92	71.64	20.10	63.50	21.03	N/A

Table 1: Task performance and training cost on ALFWorld. Overall SR is reported as mean

\pm

standard deviation across three evaluation seeds. Training time is in hours, with speedup relative to Vanilla OPD for the same student size. Among OPD methods, bold and underlined values mark the best and second-best results per metric, respectively; shading highlights the shortest training time.

Qwen3 Series
Model		Method	WebShop				ScienceWorld
Teacher	Student	Method	Score $\uparrow$	SR $\uparrow$	Round	Training Wall-clock Time (Speedup)	Score $\uparrow$	SR $\uparrow$	Round	Training Wall-clock Time (Speedup)
-	0.6B	Zero-Shot	49.54	11.00	4.56	N/A	5.59	0.00	19.86	N/A
-	1.7B		51.10	15.67	5.22	N/A	22.28	6.89	19.65	N/A
-	4B		45.37	13.33	6.75	N/A	44.36	20.67	19.19	N/A
8B	-	GiGPO	71.54	43.33	4.28	N/A	55.58	35.33	15.40	N/A

Table 2: Task performance and training cost on WebShop and ScienceWorld. Score and SR are reported as mean

\pm

standard deviation across three evaluation seeds. Training time is in hours, with speedup relative to Vanilla OPD for the same benchmark and student size. Among OPD methods, bold and underlined values mark the best and second-best Score, SR, and training time, respectively; shading highlights the shortest training time.

Method	ALFWorld			WebShop			ScienceWorld
Method	Avg. SR $\uparrow$	Time (h) $\downarrow$	Rank $\downarrow$	Avg. SR $\uparrow$	Time (h) $\downarrow$	Rank $\downarrow$	Avg. SR $\uparrow$	Time (h) $\downarrow$	Rank $\downarrow$
w/o ID	$58.92_{{\color[rgb]{0.75,0,0}\downarrow 17.31}}$	2.45	4.67	$38.78_{{\color[rgb]{0.75,0,0}\downarrow 1.00}}$	1.75	3.17	$19.63_{{\color[rgb]{0.75,0,0}\downarrow 3.04}}$	2.65	3.33
w/o NAP Fallback	$75.18_{{\color[rgb]{0.75,0,0}\downarrow 1.05}}$	1.25	2.33	$36.67_{{\color[rgb]{0.75,0,0}\downarrow 3.11}}$	2.09	4.50	$\underline{19.85}_{{\color[rgb]{0.75,0,0}\downarrow 2.82}}$	2.33	3.00
Reference-Prefix Replay (50%)	$68.94_{{\color[rgb]{0.75,0,0}\downarrow 7.29}}$	1.90	4.33	$\underline{39.11}_{{\color[rgb]{0.75,0,0}\downarrow 0.67}}$	2.18	2.67	$18.89_{{\color[rgb]{0.75,0,0}\downarrow 3.78}}$	2.87	3.67
Reference-Prefix Replay (100%)	$\underline{75.87}_{{\color[rgb]{0.75,0,0}\downarrow 0.36}}$	1.11	2.00	$37.89_{{\color[rgb]{0.75,0,0}\downarrow 1.89}}$	1.66	3.17	$18.45_{{\color[rgb]{0.75,0,0}\downarrow 4.22}}$	1.73	4.00
ActFirst-OPD (Ours)	76.23	1.94	1.67	39.78	1.83	1.50	22.67	2.27	1.00

Table 3: Ablation results averaged across Qwen3-0.6B, 1.7B, and 4B. Red subscripts show SR drops from ActFirst-OPD in percentage points. Time denotes training wall-clock time. Rank averages per-size SR ranks, using average ranks for ties. Bold and underlined values mark the best and second-best results per benchmark and metric, respectively.

Figure 3: Rollout efficiency on ALFWorld with Qwen3-1.7B. (a,b) Measured rollout speedup (solid; mean

\pm

standard deviation over three paired repeats) and idealized estimates (dashed), varying

b_{\max}

T=30

in (a) and

T

b_{\max}=256

in (b). Speedup axes are linear up to

2.5

and logarithmic above. (c,d) GPU utilization for Vanilla OPD and ActFirst-OPD; annotations show full-training environment-transition throughput.

Figure 4: Rollout quality on ALFWorld. At each update, all strategies use the same frozen Qwen3-1.7B checkpoint on 3,553 training tasks with a 30-turn maximum rollout horizon.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Version	Training	Evaluation	Max. turns
ALFWorld	0.4.2	3,553	140 seen / 134 unseen	30
WebShop	64fa2a5	4,096	100	15
ScienceWorld	1.2.3	2,294	150	30

Appendix

Table 4: Benchmark configurations. Counts refer to task instances; the WebShop version denotes its base repository commit.

Benchmark	Time (h)	SFT steps	GiGPO steps	Time (h)	Candidates
	Teacher training			Candidate collection
ALFWorld	4.18	250	250	8.50	71,060
WebShop	22.05	250	2,200	4.81	40,960
ScienceWorld	18.27	—	144	19.88	22,940

Appendix

Table 5: Offline preparation costs. Times are in hours, with eight GPUs per job. Teacher training time includes all listed stages. Ten candidates are generated per task for each source model.

Algorithm and Optimization
Hyperparameter	Value
Distillation objective	Student-to-teacher reverse KL
PPO clipping	0.2
Dual-clipping constant	3.0
Optimizer	AdamW ( Loshchilov and Hutter, 2019 )
Learning rate	$10^{-6}$

Appendix

Table 6: Training hyperparameters across all benchmarks.

Evaluation Protocol
Hyperparameter	Value
Interaction mode	Think-then-act
Reference information	None
Evaluation seeds	42, 43, 44
Trajectories per task and seed	1
Evaluation batch	64 tasks

Appendix

Table 7: Evaluation hyperparameters across all benchmarks.

Workload
Parameter	Value
Tasks per batch	16
Fast-action output length	16 tokens
Full-response output length	128 or 512 tokens
Sampling temperature	1.0
Top- $p$	1.0

Appendix

Table 8: Rollout efficiency experiment settings.

Window	Method	Transitions	Window time (s)	Env. steps / s
Full training	Vanilla OPD	55,525	15,202	3.65
	ActFirst-OPD	38,616	6,796	5.68
First 10 min	Vanilla OPD	1,289	600	2.15
	ActFirst-OPD	3,803	600	6.34

Appendix

Table 9: Environment-transition throughput on ALFWorld with Qwen3-1.7B. Measurements use training runs separate from those reported in Table 1 . For each method, both windows use the same run and start when rollout collection begins. The full-training window ends when update 250 completes; the early window covers the first 600 seconds. Initialization and final shutdown are excluded.

Figure 5: Performance–time trade-off on ALFWorld for Qwen3-0.6B, 1.7B, and 4B students. SR aggregates seen and unseen tasks; shading indicates

\pm 1

standard deviation across three evaluation seeds. Stars mark the final checkpoints in Table 1 ; the dashed line marks two hours of training.

Benchmark	Student	First-action alignment (%)	Full-trajectory alignment (%)	Aligned-turn coverage (%)
ALFWorld	0.6B	84.86	32.36	18.19
	1.7B	89.38	49.75	33.59
	4B	93.39	85.63	64.05
WebShop	0.6B	49.33	11.80	18.77
	1.7B	56.89	21.24	23.50
	4B	60.85	49.87	51.77

Appendix

Table 10: Cumulative reference alignment over 250 training updates. Cumulative rates are computed by aggregating the corresponding counts across complete rollouts in rollout batches finished by update 250. First-action and full-trajectory alignment use the number of rollout tasks with reference guidance as the denominator; turn coverage uses all executed turns, including fallback turns.

Figure 6: First-action reference alignment during training. Fraction of rollout tasks with reference guidance whose first transition passes the transition consistency check.

Figure 7: Full-trajectory reference alignment during training. Fraction of rollouts with reference guidance that complete the reference trajectory and terminate without triggering fallback.

LLM Agent Training

Large language model agent training teaches models to plan, use tools, and learn from multi-step interaction with environments.

See all 590 papers

State-OPD: State-Grounded On-Policy Distillation for Multi-Turn Agents
Oct 9, 2026Zhentao Zhang, Yanzhao Zheng, Yanlin Liu +9On-Policy Distillation
Alibaba Group · Tsinghua University · Shanghai Jiao Tong University +1
Multi-Turn On-Policy Distillation with Prefix Replay
Jul 6, 2026Baohao Liao, Hanze Dong, Christof Monz +3LLM Agent Training On-Policy Distillation
1Microsoft Research · University of Amsterdam
Know When to Stop, Where to Restart: Accelerating Multi-Turn Agentic On-Policy Distillation
Sep 13, 2026Zhiyu Gui, Kexin Huang, Jia Guo +6On-Policy Distillation
University of Science and Technology of China · Ant Group

Oct 9, 2026J/K move · Enter open · S save

State-OPD: State-Grounded On-Policy Distillation for Multi-Turn Agents

Zhentao Zhang, Yanzhao Zheng, Yanlin Liu, Yuanqiang Yu+8

Alibaba Group · Tsinghua University · Shanghai Jiao Tong University+1

On-Policy Distillation

Open paper AskPDF arXiv

On-Policy Distillation (OPD) has emerged as a powerful paradigm for transferring reasoning capabilities from frontier models to smaller models, while extending its success to multi-turn agentic environments remains challenging. We study agentic OPD through interaction states, distinguishing the objective task and executed action-observation record from the internal semantic trajectory of student reasoning. Through this lens, we identify two roles of student reasoning in multi-turn OPD: prior reasoning distorts teacher supervision, while current reasoning shapes the action alternatives sampled at the same state. Specifically, at matched interaction states, removing prior student reasoning improves teacher continuation success and changes the teacher's token-level targets, with larger continuation gains at later turns. Meanwhile, resampling complete reasoning-action responses yields greater action diversity at the same state than resampling actions under a fixed reasoning trace. We introduce State-Grounded On-Policy Distillation, a framework that grounds teacher supervision in interaction states and reallocates the sampling budget to generate multiple complete reasoning-action responses at states with high response entropy, while retaining full-history student generation. Comprehensive evaluations across ALFWorld, ScienceWorld and WebShop demonstrate that State-OPD achieves the highest macro-average success rate among the evaluated methods for both Qwen3-1.7B and Qwen3-4B students. Notably, for Qwen3-4B, State-OPD raises macro-average success from 26.07% with the strongest baseline to 31.74% and surpasses its Qwen3-32B teacher on ALFWorld Seen. Further analyses reveal that State-OPD achieves higher task success with fewer supervised turn responses and substantially reduces persistent failed-action repetition; qualitative traces illustrate decision revision based on interaction feedback.

Abstract

LLM Agent Training

Explore similar work

State-OPD: State-Grounded On-Policy Distillation for Multi-Turn Agents

Multi-Turn On-Policy Distillation with Prefix Replay

Know When to Stop, Where to Restart: Accelerating Multi-Turn Agentic On-Policy Distillation