When Does Action Credit Need Updating?

Hongye Yang; Boxiao Huang

cs.AISep 24, 2026

When Does Action Credit Need Updating?

Authors: Hongye Yang, Boxiao Huang

Organizations: College of Computing Georgia Institute of Technology

Abstract

Tool-using agents are continually updated with new interaction data. After each policy update, however, previously estimated action credits may become stale. Recomputing them from scratch can require many additional tool calls and environment interactions, making repeated updates increasingly expensive. We ask a simple question: when does historical action credit actually need to be updated? Our key observation is that a change in action value does not necessarily imply a change in the decision. Historical credit can still be useful as long as policy-induced drift is too small to overturn the existing action ranking. Building on this idea, we introduce pairwise branch sensitivity to capture how strongly a policy update affects the downstream regions that distinguish two candidate actions. We then derive a first-order anchored credit-transport estimator that updates historical credit using old interventional trajectories, and propose a Decision-Sufficient Credit Gate (DSC-Gate) that chooses whether to reuse, transport, or resample credit. Experiments show that branch sensitivity explains credit drift substantially better than global policy distance. With sufficient historical data, credit transport reduces estimation error, while its benefit to decision making is concentrated on updates that affect action-distinguishing branches. On a fully independent test set, DSC-Gate changes mean regret by only +0.00004 relative to a gap-based gate while reducing mean new tool steps from 472 to 286, a 39.4% reduction. We observe the same pattern after a real tool-agent parameter update. Overall, our results show that agents do not need to recompute action credit after every policy update: much of the historical evidence can be reused or cheaply corrected, reducing the additional interaction required to keep action decisions up to date.

Figures & tables

Figure 1: Deciding when historical action credit must be refreshed. Left to right: after a policy update

\mu\!\to\!\pi

, old interventional trajectories

\mathcal{D}_{\mu}

collected under the previous policy are reused to compare two candidate root actions

a

and

b

. Pairwise branch sensitivity

G(a,b)

weights the update by the visitation difference

d_{\mu,a}-d_{\mu,b}

, so changes in states that both branches visit cancel in the action difference, and only changes in branch-differential states drive relative drift. First-order anchored transport then corrects the historical gap as

\widehat{c}_{T}=\widehat{c}_{R}+\widehat{\delta}

from the same trajectories, without new execution. DSC-Gate combines

\widehat{c}_{R}

\widehat{\delta}

, and empirically calibrated radii

r_{R},r_{\Delta},r_{T}

to resolve each leader-versus-competitor comparison by reuse, transport, or refresh; only refresh executes the target policy in the environment. The bars show how the gate shifts toward refresh when the update is concentrated on branch-differential states.

Figure 2: Credit drift and ranking stability. Each panel contains all 864 action pairs from the 288 L2–L3 confirmation tasks for a different update condition. Signs are oriented by old credit. Highlighted points are true ranking reversals; the annotated rate additionally requires the absolute new credit difference to exceed the prespecified relevance threshold.

Figure 3: Branch sensitivity and credit drift in L1. Panel A shows the gain in Spearman correlation from replacing global KL with pairwise KL. Panel B shows the additional gain from directed branch sensitivity over pairwise KL. Each hexagon aggregates one or more confirmation tasks; the dashed line marks zero gain.

Baseline	Baseline AUC	Difference	Simultaneous interval
Adaptive DR	0.008462	-0.001583	[-0.002651, -0.000544]
Occupancy-sensitive	0.006525	+0.000354	[-0.000147, +0.000891]
Gap-based	0.006494	+0.000385	[-0.000124, +0.000950]

Table 1: Primary L3 decision comparisons. Differences are transport-adaptive minus the baseline; lower is better. Intervals are simultaneous for the three prespecified comparisons.

Figure 4: From credit correction to decision benefit. Panel A reports the relative reduction in pairwise credit MSE from transport over direct reuse across update conditions and old-data budgets. Panel B reports the corresponding decision benefit across old-data and new-execution budgets. Numerical improvement is most useful when the update can change the action ranking.

Method	Mean regret	$P(R>0.02)$	New steps	Any refresh
DSC-Gate	0.00673	2.9%	286	38.9%
Gap-Gate	0.00669	3.0%	472	63.5%
WIS-Gate	0.00734	3.8%	521	69.4%
DR-Gate	0.00812	4.6%	558	72.6%
Reuse only	0.01284	8.1%	0	0%
Transport only	0.00803	4.7%	0	0%

Table 2: Gate results on the independent L4 test set.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Test	Mean statistic	95% interval	Result
L1-A	6.706786	[5.872131, 7.521799]	Passed
L1-B	0.7387006	[0.7298759, 0.7472724]	Passed
L1-C	-0.001264017	[-0.001395845, -0.001137836]	Passed

Appendix

Table 7

Step size	Reuse MSE	Exact first-order MSE	Sampled first-order MSE	Eligible-pair reversal rate
0.5	3.750e-04	1.443e-06	1.518e-06	0.0000%
1	1.688e-03	3.261e-05	3.288e-05	0.0000%
2	7.336e-03	8.734e-04	8.739e-04	0.0000%
4	2.180e-02	1.421e-02	1.420e-02	0.0040%
8	3.912e-02	4.105e-02	4.106e-02	0.0386%

Appendix

Table 9

Update	Old N	Reuse	Transport	DR	WIS
Small learning update	16	0.019084	0.019003	0.025226	0.018989
Small learning update	64	0.004864	0.004758	0.005483	0.004767
Small learning update	256	0.001289	0.001213	0.001199	0.001216
Moderate learning update	16	0.019444	0.019702	0.026032	0.019536
Moderate learning update	64	0.005250	0.004933	0.005670	0.004952
Moderate learning update	256	0.001639	0.001238	0.001206	0.001250

Appendix

Table 10

Old N	Estimator	Acceptance rate	Post-acceptance error	Unseen fraction	ESS/N
16	Reuse	5.74%	0.000%	23.36%	0.976
16	Transport	5.57%	0.000%	23.36%	0.976
16	DR	4.31%	0.000%	23.36%	0.976
16	WIS	4.32%	0.000%	23.36%	0.976
64	Reuse	30.43%	0.000%	5.16%	0.975
64	Transport	25.76%	0.000%	5.16%	0.975

Appendix

Table 11

Method	Primary analysis	Small learning update	Moderate learning update	Large learning update	Route-direction update
Direct reuse	0.008471	0.008571	0.008371	0.008724	0.014458
Fresh uniform	0.067821	0.068636	0.067007	0.061789	0.069343
Fresh adaptive	0.065306	0.066243	0.064370	0.059589	0.066096
Gap-based refresh	0.006494	0.006602	0.006385	0.006808	0.011174
Occupancy-sensitive refresh	0.006525	0.006621	0.006429	0.006789	0.011080
Transport uniform	0.007236	0.007241	0.007232	0.009175	0.008080

Appendix

Table 12

Population	Tasks	Gap-based refresh	Occupancy-sensitive refresh	Transport adaptive	DR adaptive
All	288	0.006067	0.006105	0.006469	0.007991
Competing routes	216	0.006494	0.006525	0.006879	0.008462
Serial controls	72	0.004789	0.004845	0.005240	0.006577
Horizon 6	96	0.006258	0.006280	0.006434	0.006647
Horizon 9	96	0.005424	0.005414	0.005989	0.008592
Horizon 12	96	0.006520	0.006620	0.006983	0.008733

Appendix

Table 13

Update	Drift MSE	Remainder MSE	$\epsilon$ -relevant reversal rate	New mean value
Small learning update	8.468e-05	3.317e-08	0.000%	0.6890
Moderate learning update	4.203e-04	8.169e-07	0.116%	0.6973
Large learning update	3.096e-03	4.660e-05	2.083%	0.7238
Route-direction update	6.257e-03	1.233e-04	4.398%	0.6876

Appendix

Table 14

Update	Old N	Gap	Occupancy	Transport	DR	Fresh uniform
Small learning update	16	Not reached	Not reached	Not reached	Not reached	Not reached
Small learning update	64	768	768	768	1536	Not reached
Small learning update	256	0	0	0	0	Not reached
Moderate learning update	16	Not reached	Not reached	Not reached	Not reached	Not reached
Moderate learning update	64	1536	1536	1536	1536	Not reached
Moderate learning update	256	0	0	0	0	Not reached

Appendix

Table 15

Comparison	Regret difference DSC - baseline	95% interval	Tool-step ratio	Simultaneous 95% interval
vs Gap-Gate	+0.00004	[-0.00010,+0.00018]	0.606	[0.550,0.668]
vs WIS-Gate	-0.00061	[-0.00089,-0.00034]	0.549	[0.500,0.604]
vs DR-Gate	-0.00139	[-0.00179,-0.00100]	0.513	[0.462,0.568]

Appendix

Table 16

Update condition	Reuse	Transport	Refresh	Mean new tool steps	Mean regret
Ordinary small or moderate	55.4%	22.8%	21.8%	151	0.00491
Branch-selective small or moderate	31.8%	12.3%	55.9%	421	0.00855
Equal-weight aggregate	43.6%	17.6%	38.9%	286	0.00673

Appendix

Table 17

Item	Count	Unit
Policy-update trajectories	1,229,204	tool steps
Largest old-trajectory set	12,293,461	tool steps
Development and calibration references	15,741,039	tool steps
Target-pool simulation	41,824,782	tool steps
Model prompts	11,454	prompts

Appendix

Table 18

Explore similar work

How Much, Then Where: Credit-Conserving Action-to-Token Allocation for Multi-Turn Agent Reinforcement Learning
Aug 7, 2026Lichao Ma, Yang Sun, Shuaitao Zhao +9Token-Level Credit Assignment Credit Assignment in RL
1Peking University · 2Meituan LongCat Interaction Team · 3Fudan University +4
TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents
Jul 15, 2026Leitian Tao, Baolin Peng, Wenlin Yao +5Credit Assignment in RL Step-Level Credit Assignment in RL
University of Wisconsin–Madison · Microsoft Research
Keep Policy Gradient in Charge: Sibling-Guided Credit Distillation for Long-Horizon Tool-Use Agents
Jun 10, 2026Tianyu Ding, Jianhong Xin, Juan Pablo De la Cruz WeinsteinToken-Level Credit Assignment Group Relative Policy Optimization
Amazon Web Services

Aug 7, 2026, cs.AIJ/K move · Enter open · S save

How Much, Then Where: Credit-Conserving Action-to-Token Allocation for Multi-Turn Agent Reinforcement Learning

Lichao Ma, Yang Sun, Shuaitao Zhao, Yangyi Fang+8

1Peking University · 2Meituan LongCat Interaction Team · 3Fudan University+4

Token-Level Credit Assignment Credit Assignment in RL Step-Level Credit Assignment in RL RL Post-Training Policy Learning Fine-Grained Credit Assignment+1

Open paper AskPDF arXiv

Credit assignment in multi-turn agent reinforcement learning operates at two levels: assigning trajectory-level credit to actions and distributing each action's credit across its tokens. In this paper, we introduce FACTOR, which separates these decisions. FACTOR uses checkpoint-calibrated TD residuals to assign per-action credits that telescope to the trajectory advantage, and feedback-conditioned teacher-student likelihood gaps to allocate each credit across the realized action tokens. Per-action normalization preserves the action-average coefficient and prevents token-level sign flips. We pair this construction with an action-mean reduction, removing the implicit dependence of an action's scalar surrogate weight on its token length. At the behavior policy and before clipping, each action's inner action-mean surrogate equals its TD credit. FACTOR consistently improves over competitive baselines across ALFWorld, WebShop, and ScienceWorld, with every environment-seed comparison favoring FACTOR and the largest gains emerging on the longest-horizon environment. The same hyperparameters transfer without retuning to a larger backbone and to a different model family. Ablations identify TD action credit as the dominant driver of the improvement, with hindsight token allocation contributing complementary gains.

Abstract

Explore similar work

How Much, Then Where: Credit-Conserving Action-to-Token Allocation for Multi-Turn Agent Reinforcement Learning

TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents

Keep Policy Gradient in Charge: Sibling-Guided Credit Distillation for Long-Horizon Tool-Use Agents