ActionOPD: Distilling Decisions, Not Tokens

Huining Yuan; Xiangmin Yi; Zelai Xu; Zixu Wang; Xinlei Chen; Xiao-Ping Zhang; Yu Wang; Chao Yu

cs.AIOct 14, 2026

ActionOPD: Distilling Decisions, Not Tokens

Authors: Huining Yuan, Xiangmin Yi, Zelai Xu, Zixu Wang, Xinlei Chen, Xiao-Ping Zhang, Yu Wang, Chao Yu

Organizations: Tsinghua University

Abstract

Distilling capable language-model agents into smaller models requires aligning the student's behavior with the teacher's. This raises a fundamental question: at which states and at what granularity should this alignment be performed? Offline imitation provides dense but token-specific supervision and can suffer from state-distribution mismatch. Token-level on-policy distillation (OPD) addresses this mismatch but typically requires teacher token probabilities and compatible tokenization. We argue that semantic actions provide a natural unit of supervision for agent distillation. We introduce ActionOPD, an action-level on-policy distillation method motivated by maximizing expected teacher-action likelihood at student-encountered states. At each such state, ActionOPD samples actions from both the student and a black-box teacher, rewarding student actions according to their semantic agreement with the sampled teacher actions. Through policy optimization, ActionOPD aligns the student's action-level behavior with the teacher's decisions without requiring reproduction of teacher-written reasoning or access to teacher token probabilities. Training Qwen3-4B and Qwen3-8B with two teachers on diverse, curated tool-use tasks yields average gains of 3.3-6.9 percentage points across four out-of-distribution benchmarks. ActionOPD scales favorably with student size and teacher capability, without requiring the teacher and student to share a model family or tokenizer. On $τ^2$ -bench, ActionOPD achieves competitive performance against offline distillation and token-level OPD. We further explore Self-ActionOPD with a same-model teacher and privileged context. These results establish ActionOPD as a promising approach to agent distillation.

Figures & tables

Figure 1: Student–teacher scaling of ActionOPD. Larger students attain higher final averages. DeepSeek-V4-Flash yields higher student averages than Qwen3.7-Flash.

Figure 2: Supervision for agent learning. Offline distillation (KD) imitates teacher tokens at teacher-induced states. Token-level OPD compares token distributions along student-generated interactions. Outcome-based RL uses terminal feedback. ActionOPD instead rewards semantic agreement with teacher actions at student-encountered states. Here, we sample only one response per state for demonstration. For implementation, we expand the sampled teacher multiset and student state group.

Figure 3: ActionOPD at one state. Student candidates are scored against a shared teacher multiset. One highest-reward action is committed. All candidates in retained groups contribute to training.

Panel of Table 1 — Table 1: Cross-benchmark transfer after 100 ActionOPD steps (%). $+$ denotes training with the named teacher. AppWorld: task/scenario success on Normal/Challenge; scenario success requires all task variants to succeed. Tau: $\mathrm{pass}^{1}/\mathrm{pass}^{4}$ . Average weights four benchmarks equally. Bold/underlined: best/second-best distinct scores per student size, including Base; ties share marks. Rankings and green average gains over Base (percentage points) use unrounded scores. Generation settings differ by role (Table 8 ).

Model	Airline		Retail		Telecom		Average
Model	Train (30)	Test (20)	Train (74)	Test (40)	Train (74)	Test (40)	Train	Test
Qwen3-32B	49.2/16.7	43.8/15.0	46.0/16.2	46.9/15.0	25.7/13.5	41.9/30.0	40.3/15.5	44.2/20.0
Base	35.0/10.0	33.8/5.0	26.7/6.8	26.9/ 10.0	17.2/4.1	26.9/7.5	26.3/6.9	29.2/7.5
KD	38.3/ 20.0	37.5/ 20.0	37.5/ 13.5	33.8/ 12.5	25.3 / 13.5	41.3 / 25.0	33.7/ 15.7	37.5/ 19.2
OPD	40.0 /13.3	41.3 / 20.0	40.2 / 10.8	37.5 / 10.0	22.6/ 10.8	40.0/ 22.5	34.3 /11.7	39.6 / 17.5
ActionOPD	42.5 / 16.7	40.0 / 15.0	42.9 / 10.8	39.4 / 12.5	28.4 / 13.5	40.6 /15.0	37.9 / 13.7	40.0 /14.2

Table 2: Tau distillation comparison:

\mathrm{pass}^{1}/\mathrm{pass}^{4}

(%). Qwen3-4B students and a Qwen3-32B teacher. Trained students are evaluated at step 50. Four trials per task. Average equally weights domains within each split. ActionOPD uses the same run as in Table 3 . Bold/underlined: best/second-best distinct scores (unrounded); teacher excluded.

Model	Airline		Retail		Telecom		Average
Model	Train (30)	Test (20)	Train (74)	Test (40)	Train (74)	Test (40)	Train	Test
Base	35.0/ 10.0	33.8 /5.0	26.7/6.8	26.9/10.0	17.2/4.1	26.9/7.5	26.3/6.9	29.2/7.5
Validity Reward	42.5 / 16.7	30.0/ 10.0	23.7/8.1	26.3/10.0	25.3 / 14.9	36.3/10.0	30.5/ 13.2	30.8/10.0
Binary Reward	35.8 / 10.0	33.8 / 10.0	35.8/8.1	36.3/ 17.5	28.4 / 13.5	40.6 / 15.0	33.3 /10.5	36.9/ 14.2
Random Commit	34.2/3.3	27.5/ 10.0	41.6 / 16.2	40.6 / 15.0	24.0/8.1	44.4 / 12.5	33.2/9.2	37.5 / 12.5
ActionOPD	42.5 / 16.7	40.0 / 15.0	42.9 / 10.8	39.4 /12.5	28.4 / 13.5	40.6 / 15.0	37.9 / 13.7	40.0 / 14.2

Table 3: Tau ablations:

\mathrm{pass}^{1}/\mathrm{pass}^{4}

(%). Qwen3-4B student, Qwen3-32B teacher; trained variants at step 50. Four trials per task. Parentheses: task counts. Average weights domains equally per split. Bold/underlined: best/second-best distinct unrounded scores, including Base; ties share marks.

Pattern / Task Context	Qwen3-4B Base	Qwen3-4B + ActionOPD
Following Policy Constraints Airline ( $2/4\rightarrow 4/4$ ). The user requests a flight change, but basic economy bookings cannot be modified.	Reads that the booking is basic economy, yet modifies the flight through the API, violating the policy. Outcome: Failure .	Explains the restriction and calls the transfer tool when the user requests escalation. Success means keeping the booking unchanged, not rebooking. Prohibited modifications: $2/4\rightarrow 0/4$ . Outcome: Success .
Tracking the Correct Target Retail ( $0/4\rightarrow 3/4$ ). Change the target order’s address and speaker. Retrieve the new address in New York from another order.	Treats the source order’s New York address as if it already belonged to the target. Changes only the speaker. The target’s address remains in Los Angeles. Outcome: Failure .	Retrieves the source address, then updates the target’s address followed by its speaker. The returned order contains both the New York address and the requested green speaker. Outcome: Success .
Completing and Verifying Actions Telecom ( $0/4\rightarrow 2/4$ ). Restore service after an overdue-bill suspension and an unseated SIM. Payment alone does not reactivate the line.	Promises restoration after payment but never calls resume_line . The user pays and reboots, but service remains disconnected and the agent transfers the case. Outcome: Failure .	Recovers from an invalid tool call, resumes the line with resume_line , and guides the user to reseat the SIM. User-side tool output confirms connection, and the evaluator verifies that the debt is cleared. Outcome: Success .

Table 4: Selected positive cases: Qwen3-4B before/after 100 ActionOPD steps with DeepSeek-V4-Flash. Fractions: successes in the first four trials (Base

\rightarrow

ActionOPD). Each pair contrasts Base failure with ActionOPD success on the same task and trial. Outcomes denote evaluation results, not training rewards.

Model	Airline		Retail		Telecom		Average
Model	Train (30)	Test (20)	Train (74)	Test (40)	Train (74)	Test (40)	Train	Test
Privileged Teacher	51.7/33.3	35.0/15.0	38.2/6.8	38.1/17.5	20.6/8.1	37.5/20.0	36.8/16.1	36.9/17.5
Base	35.0 / 10.0	33.8 / 5.0	26.7 / 6.8	26.9 / 10.0	17.2 / 4.1	26.9 / 7.5	26.3 / 6.9	29.2 / 7.5
Self-ActionOPD	44.2 / 26.7	35.0 / 10.0	35.1 / 8.1	35.6 / 10.0	19.3 / 5.4	31.3 / 10.0	32.9 / 13.4	34.0 / 10.0

Table 5: Self-ActionOPD on Tau:

\mathrm{pass}^{1}/\mathrm{pass}^{4}

(%), using Qwen3-4B. Privileged Teacher is the initial Base model given privileged context. Self-ActionOPD is evaluated at step 50 using public context only. Four trials per task. Average equally weights domains within each split. Bold/underlined: best/second-best of Base and Self-ActionOPD; ties share marks; privileged reference excluded.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	AWM	EnvScaler
Source tasks	10,000	2,550
Context-eligible	9,380	—
Deterministic integrity pass	7,562	2,495
Static feasibility: healthy	5,942	755
Environments represented in the final pool	794	49

Appendix

Table 6: Task-selection funnel for the pools used in this study. Task counts denote unique tasks remaining after each stage, not rollout attempts. The final row reports environment coverage. Context selection is an AWM-specific stage. The dash denotes that no separate corresponding stage was used for EnvScaler.

Setting	Transfer experiment	Tau ablations
Student	Qwen3-4B or Qwen3-8B	Qwen3-4B
Action teacher	Qwen3.7-Flash or DeepSeek-V4-Flash	Qwen3-32B
Training tasks	AWM/EnvScaler static-feasibility pools	Official train splits
Collection steps reported	100	50
Task starts per step	64: 59 AWM + 5 EnvScaler	24: 8 per Tau domain
Student samples $N$	4	4

Appendix

Table 7: Standard ActionOPD and ablation training configurations. Each configuration corresponds to one training run. The number of samples refers to responses at one state, not independent complete trajectories.

Generation Settings
Experiment	Role and Model	Thinking	Sampling	Maximum Response
AWM/ EnvScaler Training	Student: Qwen3-4B/8B	Enabled	$T=0.6$ ; $p=0.95$ ; $k=20$	4k
	Teacher: DeepSeek-V4-Flash	Enabled; effort max	—	4k
	Teacher: Qwen3.7-Flash	Enabled	$T=0.6$ ; $p=0.95$	8k
	Matcher: DeepSeek-V4-Flash	Disabled	$T=0$	128
	User: DeepSeek-V4-Flash	Disabled	$T=1$	—

Appendix

Table 8: Model roles and generation settings.

LLM Agent Training

Large language model agent training teaches models to plan, use tools, and learn from multi-step interaction with environments.

See all 564 papers

Interactive-Policy Distillation with Bidirectional Propose-and-Verify
Sep 29, 2026Shutong Wu, Xiwen Chen, Brendan Rappazzo +4Language Model Distillation On-Policy Distillation
Department of Computer Sciences, University of Wisconsin–Madison · Machine Learning Research, Morgan Stanley · Department of Electrical and Computer Engineering, Rutgers University–New Brunswick
PivotOPD: Learning to Recover from Pivotal Mistakes in Multi-Turn Agents
Sep 30, 2026Yinghui He, Yapei Chang, Khushi Bhardwaj +4Language Model Distillation LLM Agent Training
Princeton University · NVIDIA · University of Maryland
Beyond the Teacher: Understanding and Preserving Distillation Overshoot in Agentic On-Policy Distillation
Oct 18, 2026Jike Zhong, Ming Li, Yuxiang Lai +9On-Policy Distillation Agentic RL
University of Southern California · University of Florida · Emory University +6

Sep 29, 2026J/K move · Enter open · S save

Interactive-Policy Distillation with Bidirectional Propose-and-Verify

Shutong Wu, Xiwen Chen, Brendan Rappazzo, Daiheng Zhang+3

Department of Computer Sciences, University of Wisconsin–Madison · Machine Learning Research, Morgan Stanley · Department of Electrical and Computer Engineering, Rutgers University–New Brunswick

Language Model Distillation On-Policy Distillation

Open paper AskPDF arXiv

On-policy distillation (OPD) trains a student model on its self-generated trajectories with dense token-level teacher feedback. However, naive OPD may suffer from teacher unanchoring, where the student's reasoning trajectory drifts far from the teacher, causing the teacher to be queried on states it would hardly visit and thus provide unreliable supervision. We propose Interactive-Policy Distillation (IPD), which applies adaptive teacher intervention to the student rollout. Under a bidirectional propose-and-verify state machine, the student and teacher alternately exchange their roles as proposer and verifier, and collaboratively generate mixed-source trajectories. Then different supervisions are applied according to the source of each token. This bidirectional propose-and-verify mechanism and the source-split loss make IPD not only a more performant distillation method, but also a unified bridge between on-policy and off-policy paradigms. To make the interleaved dual-model rollouts more efficient, we also design a dedicated fused inference engine that co-hosts both models in one serving instance with separate KV caches and instantiates the state machine model to distribute, collect, and process requests. On math reasoning tasks and across multiple teacher-student model pairs, student models trained with IPD not only outperform those trained with OPD, but also demonstrate higher data efficiency. Specifically, when distilling Qwen3-30B-A3B into Qwen3-1.7B-Base, IPD brings a +3.28 mean@8 and a +3.28 best@8 benchmark-averaged accuracy improvement compared with OPD. Besides, IPD only consumes about 1/4 of the training examples and steps to outperform OPD trained on the whole training dataset for one epoch. We also investigate the impact of different loss variants and takeover / handback configurations, and demonstrate the robustness of IPD on different training data.

Abstract

LLM Agent Training

Explore similar work

Interactive-Policy Distillation with Bidirectional Propose-and-Verify

PivotOPD: Learning to Recover from Pivotal Mistakes in Multi-Turn Agents

Beyond the Teacher: Understanding and Preserving Distillation Overshoot in Agentic On-Policy Distillation