Trust Is Not Enough: Influence Calibration for On-Policy Self-Distillation in Agentic RL
Authors: Qizhen Lan, Xi Xiao, Xiangchen Guan, Mengchen Fan, Moule Lin, Jung Im Choi, Lijing Zhu
Organizations: The University of Texas Health Science Center at Houston · University of Alabama at Birmingham · Amazon · Trinity College Dublin and Lero · The University of Findlay · University of Houston-Clear Lake
On-policy self-distillation (OPSD) gives language agents dense token-level supervision from a privileged self-teacher on the policy's own trajectories. Existing methods allocate this supervision mainly by teacher trust, but trust does not reveal whether emphasizing a token supports the current policy objective. We call this the trust-utility mismatch and introduce Influence Calibration for Self-Distillation (ICSD). For each supervised token, ICSD measures the first-order response of its importance-weighted RL surrogate contribution to a teacher-directed output perturbation. Batch-adaptive calibration converts this non-stationary signal into a bounded allocation weight while preserving the original auxiliary-loss mass within each action turn. These detached weights affect only the distillation loss and require no additional model pass. Across ALFWorld, WebShop, and Search-QA, ICSD improves all matched aggregate metrics over trust-only allocation under Group Relative Policy Optimization (GRPO) and Group-in-Group Policy Optimization (GiGPO), across two model families spanning 1.5B to 7B. At 7B, it reaches 96.1% ALFWorld success and a WebShop score of 93.1. Frozen-batch analyses show that ICSD reduces teacher-supported mass assigned to objective-opposed tokens from 60.1% to 37.8% and raises cosine compatibility with the RL gradient by 0.192. A companion repository is avail- able at https://github.com/lanqz7766/Influence-Calibration-for-On-Policy-Self-Distillation-in-Agentic-RL.
Figures & tables
Figure 1: Many existing OPSD allocation rules determine token weights using teacher trust alone. (a) Tokens with the same trust therefore receive the same coefficient, even when their influence on the RL objective differs. (b) ICSD retains the trust signal and uses objective influence to redistribute the same auxiliary mass.
Figure 2: Overview of ICSD. A privileged branch evaluates the student’s on-policy action tokens, while the RL objective provides token-level objective influence. ICSD calibrates this influence and uses it to redistribute the inherited trust allocation before the actor update.
Figure 3: Continuous allocation induced by ICSD on frozen GiGPO batches. Panels (a–b) show normalized coefficients over teacher-preference and objective-support ranks; darker blue denotes more mass, contours show token density, and dotted lines mark zero signals. Panel (c) reports the shift relative to SDAR among teacher-supported tokens. Whiskers are 95% action-turn bootstrap intervals over four batches.
ALFWorld
Search-QA
WebShop
Method
Pick
Look
Clean
Heat
Cool
Pick2
Avg
NQ
Triv
Pop
Hotp
2Wk
MuS
Bam
Avg
Score
Acc.
Qwen2.5-1.5B-Instruct
Skill-SD
88.9
57.1
60.0
80.0
65.0
43.5
68.0
16.7
35.4
14.8
16.7
25.2
3.9
11.2
17.7
80.6
68.8
RLSD
78.1
41.7
68.2
64.3
77.3
57.7
67.2
13.9
29.4
12.4
14.2
21.1
3.0
9.5
14.8
80.5
63.3
GRPO
84.4
41.7
77.3
50.0
68.2
50.0
65.6
16.6
34.1
15.9
16.1
24.8
3.4
58.1
24.1
72.0
60.9
+ OPSD
90.6
33.3
77.3
57.1
77.3
61.5
71.1
16.9
34.3
16.4
16.3
24.8
3.4
56.9
24.2
84.9
73.4
Table 1: Main results grouped by policy optimizer. Skill-SD and RLSD are hybrid distillation baselines; family-head rows are the RL baselines, “+” rows add auxiliary allocators, and blue rows denote ICSD. All values are percentages. Within each model block, bold and underline mark the best and second-best summary results; task-level columns are unmarked.
ALFWorld
Search-QA
WebShop
Method
Avg
Avg
Score
Acc.
Qwen2.5-1.5B-Instruct
Vanilla
5.5
12.1
17.8
5.5
Skill-Prompt*
6.2
7.7
20.8
1.6
OPSD
14.1
0.0
22.3
10.2
GiGPO+ICSD
93.8
18.2
92.2
78.1
Table 2: Summary results for extended baselines. Starred methods retain retrieved skills during validation. Supplementary Table A2 reports per-task and per-subset results.
Method
Pick
Look
Clean
Heat
Cool
Pick2
Avg
GiGPO+SDAR
96.9
66.7
100.0
78.6
81.8
96.2
89.8
GiGPO+ICSD
97.6
91.7
100.0
88.9
100.0
83.3
95.3
Table 3: Qwen3-4B ALFWorld results with GiGPO and the concise non-thinking interface.
Method
Pick
Look
Clean
Heat
Cool
Pick2
Avg
GiGPO
97.1
75.0
95.8
88.9
84.2
91.7
91.4
+ SDAR
97.2
88.9
100.0
94.1
95.0
66.7
92.2
+ Fisher magnitude
100.0
78.6
95.0
100.0
90.0
78.3
91.4
+ Sign-only
97.1
100.0
90.9
100.0
95.5
77.3
92.2
+ Influence only
93.9
88.9
92.3
100.0
95.5
83.3
93.0
+ ICSD
93.1
91.7
92.0
100.0
95.5
92.3
93.8
Table 4: Allocation ablation on ALFWorld with Qwen2.5-1.5B and GiGPO.
Figure 4: ICSD improves local update compatibility on sixteen frozen batches. (a) SDAR and ICSD cosine compatibility with the RL gradient. (b) Compatibility after tokenwise permutation of the ICSD multipliers. Diamonds and horizontal bars show means and 95% batch-bootstrap intervals.
Figure 5: Training dynamics for sensitivity and signal ablations on ALFWorld with Qwen2.5-1.5B and GiGPO. Insets enlarge steps 115–150.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
ALFWorld
WebShop
Search-QA
Train / validation batch
16 / 128
16 / 128
128 / 512
Rollouts per task
8
8
8
Maximum interaction steps
50
15
4
Maximum prompt / response length
2048 / 512
4096 / 512
4096 / 512
Actor mini-batch size
256
64
256
Actor learning rate
1×10−6
1×10−6
1×10−6
Appendix
Table A1: SDAR-aligned protocol used for ICSD training.
Figure A1: Deployed allocation across the four sign regions. ICSD moves auxiliary mass toward jointly supported tokens and away from sign-conflicting regions, while leaving the double-negative region unchanged. Bars and whiskers show mass shares and 95% action-turn bootstrap intervals over four ALFWorld batches (544,427 tokens); brackets give point differences from base trust.
Figure A2: Validation curves for matched ALFWorld allocation variants with Qwen2.5-1.5B, GiGPO, and 150 training updates under an identical training configuration.
Figure A3: Training dynamics at Qwen2.5-1.5B. Panels (a–b) show ALFWorld, panels (c–d) show WebShop, and panels (e–f) show the share of trusted auxiliary mass assigned to negative-advantage tokens. Pale lines are per-step values and dark lines are centered five-step means. This share describes allocation, not token utility.
ALFWorld
Search-QA
WebShop
Family
Method
Pick
Look
Clean
Heat
Cool
Pick2
Avg
NQ
Triv
Pop
Hotp
2Wk
MuS
Bam
Avg
Score
Acc.
Qwen2.5-1.5B-Instruct
Vanilla
11.1
0.0
6.2
0.0
0.0
4.2
5.5
10.5
25.4
17.8
8.3
14.8
2.2
6.0
12.1
17.8
5.5
Skill-Prompt*
3.4
16.7
12.9
5.3
0.0
5.0
6.2
6.2
18.1
8.7
5.4
10.1
1.5
4.0
7.7
20.8
1.6
No RL
OPSD
26.3
16.7
9.1
6.7
9.1
5.3
14.1
0.0
0.0
0.0
0.1
0.2
0.0
0.0
0.0
22.3
10.2
Skill-SD
88.9
57.1
60.0
80.0
65.0
43.5
68.0
16.7
35.4
14.8
16.7
25.2
3.9
11.2
17.7
80.6
68.8
Appendix
Table A2: Full per-task ALFWorld, per-subset Search-QA, and WebShop results corresponding to the aggregate table in the main paper. Starred methods retain retrieved skills during validation.