On-policy distillation (OPD) transfers specialized capabilities to language-model agents, but its cross-domain effects on interaction remain unclear. We investigate how OPD reshapes ReAct interaction patterns across mathematics, ALFWorld, and τ2-bench, focusing on reasoning, actions, and observation acquisition. General-reasoning supervision reduces mathematical tool attempts from 70.4% to 14.4% while increasing accuracy from 16.7% to 38.0%; code-interpreter supervision improves household actions and retail queries. In contrast, ALFWorld supervision reduces τ2-bench accuracy from 41.5% to 6.6%, accompanied by frequent tool calls that violate user-information dependencies. To address negative transfer using only source-domain training trajectories, we explore selective OPD guided by teacher preference changes before and after domain-specific RL. Retaining the top 95% of positions raises τ2 accuracy from 11.5% to 29.9% with a source-learning trade-off.
Figures & tables
Figure 1: Agentic OPD can change the ReAct patterns of LLM agents. Our proposed mitigation approach helps alleviate the overfitting to task-specific patterns.
Figure 2: Cross-domain accuracy changes relative to Base. Rows denote student training sources; columns denote evaluation domains. Outlined cells indicate domains included in training. Absolute accuracies are reported in Appendix 10 .
Model
Accuracy
Tool Call Rate
Correct no tool
Correct with tool
Base
16.70
70.40
2.40
14.30
ALF-OPD
38.30
11.50
30.60
7.70
CI-OPD
52.40
73.00
4.90
47.60
GR-OPD
38.00
14.40
30.10
7.90
Table 1: The statistics on CI math tasks.
Model
Accuracy
Calls
Repeat
Change
Base
12.90
12.97
4.50
90.80
CI
40.20
18.94
0.70
94.30
Table 2: Positive transfer from CI. (a) Calls are complete generated blocks per rollout; Repeat is exact repetition among adjacent actions; Change is a changed action after no-op feedback with a following action. (b) All/Retail are overall and retail task accuracy; Tool is rollout-level attempt coverage; Error is the fraction of rollouts with error feedback; Query is complete reference-query coverage on the 200 eligible retail rollouts. Calls again use a per-rollout denominator. Detailed metric definitions and denominators are provided in Appendix 9 .
Model
Accuracy
Ask–call
Dependency
Base
41.50
0.04
0.50
Teacher
46.00
0.67
0.38
ALF
6.60
94.30
55.80
ALF+ τ2
68.90
0.24
0.00
Table 3: Information acquisition in τ2 interactions. Accuracy measures task success.
Figure 3: Excerpts from trajectories for the same annotated task. The ALF student generates a user-ID request alongside a tool call using a schema-example ID. ALF+ τ2 waits for the user’s actual ID.
Model
CI
ALF
τ2
GR
CI+GR
51.30
38.60
45.30
44.50
ALF+CI
49.80
83.20
34.00
46.70
CI+ τ2
50.20
28.40
67.80
45.80
ALF+ τ2
33.80
83.40
68.90
36.70
ALF+CI+ τ2
51.10
86.90
64.40
45.00
Table 4: Teacher composition. (a) Accuracy on CI/GR, ALFWorld, and τ2 . (b) Ask–call uses request turns; error repetition uses post-error turns. Definitions and counts appear in Appendix 9 .
Figure 4: Effect of retained fraction after 20 updates on ALF accuracy, τ2 accuracy, and ask–call co-occurrence. The shaded leftmost point is an evaluation of the shared initialization from the Top95 run, before training. Trained points use their numerical retained fractions on the x-axis. Labels are offset for the closely spaced 95/99/100% points. Horizontal dashed lines show the Random95 update-20 evaluation.
Figure 5: Exclusion rates and enrichment on Top95 training contexts. Reasoning content denotes assistant text outside tool-call blocks. <tool_call> is the opening special token; payload denotes tokens inside a call. Parentheses show enrichment relative to the overall exclusion rate (dashed line).
Approach
ALF acc.
τ2 acc.
Ask–call rate
Signed- Δ Top95
76.64
29.88
24.26
Delimiter masking
76.28
4.88
99.37
Full-block masking
79.74
3.00
99.72
Negative masking
80.02
3.63
83.54
Advantage Top95
81.66
23.50
82.68
Advantage Top98
83.03
5.88
99.16
Table 5: Results of alternative masking approaches.
Method
GR acc.
CI acc.
CI tool attempts
No tool & correct
Tool & correct
Full OPD
46.02
37.65
10.41
31.63
6.02
Signed- Δ Top95
46.33
36.53
29.49
24.90
11.63
Table 6: Results of GR-to-CI transfer.
Domain
Interaction approach
GR
Text-based mathematical reasoning without an external tool-feedback loop.
CI
Reasoning interleaved with Python execution for calculation, symbolic manipulation, and verification.
ALF
Reasoning followed by an admissible environment action, which returns the next observation.
τ2
Customer communication and account-side tool use, with actions conditioned on available information and required authorization.
Table 7: Interaction approaches in the four training domains. Their feedback sources and turn requirements are described below.
Figure 6: Teacher RL learning curves. Solid curves show validation task outcomes, and dashed curves show sampled training reward. CI validation measures answer accuracy, with separate update axes for its two training stages. Squares mark the selected teacher checkpoints. Training and validation use their respective prompt pools and decoding settings.
Parameter
Teacher RL
OPD / selected-position runs
Optimizer
Megatron Adam
Megatron Adam
Learning rate
10−6
10−5
Warmup / schedule
none / constant
none / constant
Weight decay
0.01
0.01
Adam momentum
backend (0.9,0.999)
backend (0.9,0.999)
Adam epsilon
backend default; no override
backend default; no override
Table 8: Optimization settings separated from sampling and task rewards. Adam momentum and epsilon are inherited by the backend when no explicit optimizer override is provided. Configured minibatch sizes and rollout counts are listed separately because the worker expands prompt-unit minibatches by the rollout count.
Student/condition
Rows
Batch
Samples
Traj./batch
Prompt/response caps
CI
17,917
128
4
512
1,024/31,740
ALF
3,553
128
4
512
1,024/31,740
τ2
2,160
32
8
256
8,192/32,768
GR
17,917
32
4
128
8,192/32,768
CI+GR
6,000
128
4
512
8,192/32,768
ALF+CI
6,000
128
4
512
1,024/31,740
Table 9: Main OPD and selected-position launch recipes. Batch and configured PPO minibatch are prompt counts; “Samples” is rollouts per prompt. Learning rate is 10−5 and per-GPU microbatch is one. Prompt/response caps are measured in tokens.
Figure 7: Single-teacher OPD learning curves. Solid curves show source-validation outcomes, and dashed curves show sampled training outcomes. Squares mark the checkpoints used in the main analyses. CI is displayed through update 130 and GR through update 50. The τ2 validation curve contains measurements at updates 0, 25, and 50. Mathematical monitoring pools differ from the 980-trajectory behavioral evaluation subset.
Figure 8: Source-validation learning curves for teacher compositions. Each curve reports a separate training domain; squares mark the checkpoints used in the main analyses. ALF+CI and CI+ τ2 are displayed through update 80, and CI+GR through update 50. Other panels show their respective recorded ranges.
Metric
Unit or denominator
Definition
Task accuracy
Evaluation trajectories
Mean answer correctness for mathematics or binary task completion for ALFWorld and τ2 .
Tool-attempt rate
Evaluation trajectories
Fraction containing at least one generated tool-call marker.
Calls per trajectory
Evaluation trajectories
Mean number of generated spans enclosed by opening and closing tool-call markers.
Call JSON validity
Complete generated call blocks
Whether a block parses to an object with a string-valued name and dictionary-valued arguments.
Ask–call co-occurrence
Assistant outputs containing detected request-like text
Fraction of assistant outputs containing detected request-like text that also contain a generated tool call.
Dependency-violation rate
Evaluation trajectories
Fraction of trajectories containing a tool call that depends on user information or authorization not yet obtained, as identified through semantic review.
Table 10: Definitions of the evaluation metrics. Calls are reported as a mean count; rates are reported as percentages. Generated calls and JSON validity describe model outputs, while task accuracy measures final outcomes.
Model
Ask+call / Ask
Dependent / Ask+call
Affected rollouts / 800
Affected tasks / 100
Base
1/2,278
0/1
4
4
ALF teacher
14/2,081
4/14
3
3
ALF
5,022/5,328
4,411/5,022
446
98
ALF+ τ2
5/2,055
0/5
0
0
Table 11: Natural-trajectory semantic review. The first two columns report turn-level numerator/denominator pairs. Rollout coverage counts trajectories containing at least one dependency violation. Task coverage counts tasks with at least one affected trajectory among their eight samples.
Model
CI n=980
ALF n=1,096
τ2n=800
GR n=980
Base
0.167
0.129
0.415
0.424
GR
0.380
0.214
0.425
0.413
CI+GR
0.513
0.386
0.453
0.445
CI
0.524
0.402
0.510
0.444
ALF
0.383
0.855
0.066
0.417
τ2
0.262
0.229
0.611
0.367
Table 12: Complete accuracies for the selected students.
Model
Airline n=160
Retail n=320
Telecom n=320
Base
0.325
0.481
0.394
ALF teacher
0.319
0.634
0.356
τ2 teacher
0.469
0.684
0.822
CI
0.425
0.653
0.409
ALF
0.019
0.119
0.037
τ2
0.369
0.528
0.816
Table 13: Per-domain τ2 task accuracy. Teacher rows provide reference performance; the remaining rows are 4B students.
Figure 9: RL-induced log-probability changes and OPD advantages on Top95 training contexts. Curves show mean ordinary advantage, pre-domain-RL reference advantage, and signed Δ across within-rollout preference percentiles. Shading marks the excluded tail.
Figure 10: Token-occurrence densities of signed Δ against ordinary OPD advantage and pre-domain-RL reference advantage. Axes use symmetric-log scaling, and colors indicate occurrence counts.
Figure 11: Frequent excluded token types. Bars show their shares of all valid positions and excluded positions. A centered dot denotes leading whitespace.
Figure 12: Signed Δ for eight token types across their observed histories, ordered by retained fraction. Red occurrences are excluded and blue occurrences retained. Vertical jitter separates occurrences. A centered dot denotes leading whitespace.
Figure 13: Learning dynamics across retained fractions. Evaluation curves show ALF and τ2 accuracy at initialization and updates 10 and 20. Dashed curves show sampled ALF training reward at updates 1–20. All values are percentages.
Figure 14: Learning dynamics of Random95, delimiter masking, full-block masking, and negative masking. Evaluation curves report ALF and τ2 accuracy; training curves report sampled ALF task reward.
Rule
Excluded mass (%)
Retained mass (%)
Signed- Δ Top95
41.02
62.02
Advantage Top95
66.88
34.83
Advantage Top98
43.56
57.54
Absolute- Δ Top95
0.021
105.15
Table 14: Fixed-context feedback diagnostics. Excluded mass uses full-token normalization; retained mass includes the N/K rescaling. Percentages are rollout-level averages.
Figure 15: Ranking-control dynamics over 20 updates. Solid curves show ALF and τ2 evaluation accuracy at initialization and updates 10 and 20. Light dashed curves show mean ALF training reward on each sampled batch. All values are percentages.
Figure 16: GR-to-CI learning dynamics. Lines connect the scheduled evaluations at initialization and update 10 with the update-20 companion evaluation reported in Table 6 .
Figure 17: GR-prefix and full-supervision learning dynamics through 60 updates. Panels report source GR accuracy, tool-enabled mathematical accuracy, and tool-attempt rate.