Large language models increasingly generate multiple retrieval actions jointly, while downstream systems often provide only one reward for the resulting set. Broadcasting this reward provides a valid score-function gradient estimator for the expected set-level objective, but does not determine a coalition-dependent within-set attribution. We introduce CASI (Credit Assignment via Set Intervention for Agentic Retrieval), which constructs the missing counterfactual information through snapshot-grounded set intervention: generate and retrieve once, hold replayable snapshot components fixed, and evaluate alternative subsets under common conditions. Shapley aggregation converts these marginals into within-set credit, which bounded zero-sum advantage shaping incorporates into policy optimization. For m >= 4, singleton, leave-one-out, and full-set endpoints do not generally determine the Shapley allocation; under additive set values, extra coalition evaluations are redundant. In a five-action sponsored-search setting, exhaustive auction replay over 1,200 requests reveals substantial coalition dependence. Under the training-time evaluator, CASI achieves the highest median cosine similarity to a replay-defined Shapley reference among evaluated credit rules; under a sensitivity backend, CASI does not outperform Singleton. Across five independent training runs, CASI-trained policies achieve higher held-out auction-replay set value than Singleton- or Broadcast-trained policies. In a randomized seven-day production A/B test on 10% of traffic, the CASI-trained policy yields an observed 3.51% lift in the primary advertiser-value metric over production control.
Figures & tables
Figure 1: CASI augments the broadcast rollout advantage with action-specific credit from snapshot-grounded set intervention. The base optimizer broadcasts A^(g) across the trajectory. Cached retrievals are recombined into coalitions and scored under the frozen training-side snapshot E0 , yielding coalition values R^0(S) . Shapley aggregation produces per-action credit, which bounded zero-sum advantage shaping maps to offsets zi(g) . Tokens of action qi receive Ai(g)=A^(g)+βzi(g) ; shared tokens retain A^(g) .
Figure 2: Coalition dependence and credit alignment. Panels (a)–(b) diagnose context-dependent replay structure and size-resolved evaluator agreement. Panels (c)–(d) compare credit rules against the same replay-defined Shapley reference. Because panel (b) uses size-matched references, its values are not directly comparable with panels (c)–(d).
Credit context
Sign nz
Spearman
Top action
Median cosine
∣S∣=0
0.6424
0.3521
0.3274
0.5027
∣S∣=1
0.6356
0.3321
0.3422
0.4635
∣S∣=2
0.5994
0.2539
0.3365
0.3310
∣S∣=3
0.5683
0.1749
0.3112
0.2236
∣S∣=4
0.5381
0.1117
0.2902
0.1316
CASI (Shapley)
0.6178
0.3128
0.3299
0.4237
Table 1: Size-resolved evaluator–replay agreement under the training-time evaluator. Rows ∣S∣=k compare request-centered action-credit vectors formed by averaging marginals over predecessor coalitions of size k , using the same construction for evaluator and replay. The CASI row instead compares evaluator-defined Shapley credit with the replay-defined Shapley reference and is not directly comparable across rows.
Credit rule
Median cosine ↑
CASI − rule (95% CI)
CASI
0.4237
—
Singleton
0.3877
+0.0360[0.0036,0.0677]
Banzhaf
0.3800
+0.0437[0.0083,0.0806]
LOO
0.2269
+0.1968[0.1272,0.2682]
Table 2: Alignment with the replay-defined Shapley reference under the training-time evaluator. Median cosine is the primary metric. Confidence intervals are request-level paired-bootstrap intervals for the CASI-minus-alternative difference in median cosine.
Method
Mean replay value ↑
Top-1 eCPM ↑
SFT
60.32
11.25
Broadcast
43.30 ± 4.70
11.15 ± 0.10
Singleton
55.00 ± 4.00
11.68 ± 0.09
CASI
66.70 ± 3.40
11.93 ± 0.07
Table 3: Held-out policy evaluation on 1,000 common requests; RL entries report mean ± standard deviation across five independent training runs.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Predecessor size
pr_filter
pr_rerun
All nonzero replay edges
0.5925
0.6137
∣S∣=0
0.9839
1.0000
∣S∣=1
0.6257
0.6572
∣S∣=2
0.5617
0.5819
∣S∣=3
0.5291
0.5485
∣S∣=4
0.5375
0.5348
Appendix
Table 4: Insertion-edge sign agreement with replay, stratified by predecessor size. pr_filter is the training backend; pr_rerun is a sensitivity backend.
Context
Sign all
Sign nz
Spearman
Top action
Median cosine
∣S∣=0
0.6056
0.6424
0.3521
0.3274
0.5027
∣S∣=1
0.6135
0.6356
0.3321
0.3422
0.4635
∣S∣=2
0.6003
0.5994
0.2539
0.3365
0.3310
∣S∣=3
0.5862
0.5683
0.1749
0.3112
0.2236
∣S∣=4
0.5687
0.5381
0.1117
0.2902
0.1316
Shapley / CASI
0.6029
0.6178
0.3128
0.3299
0.4237
Appendix
Table 5: Size-resolved credit fidelity for the training-time pr_filter evaluator. The ∣S∣=k rows are size-matched diagnostics; the Shapley/CASI row compares against the replay-defined Shapley reference and is not directly comparable with the size-matched rows.
Rule
Sign all
Sign nz
Spearman
Top action
Median cosine
CASI
0.6029
0.6178
0.3128
0.3299
0.4237
Singleton
0.5947
0.6193
0.3135
0.3205
0.3877
Banzhaf
0.6060
0.6149
0.3014
0.3371
0.3800
LOO
0.5875
0.5693
0.1761
0.3226
0.2269
Appendix
Table 6: Full credit-fidelity metrics against centered replay-defined Shapley credit under the training-time pr_filter backend.
Statistic
Observed
Null median
Null 95% interval
Δcosmed(CASI−LOO)
+0.2221
-0.1105
[−0.1462,−0.0813]
Δcosmed(CASI−Singleton)
+0.0359
-0.0932
[−0.1231,−0.0685]
ρ(Γctxnorm,gCASI−LOO)
-0.0270
+0.0052
[−0.0630,+0.0758]
ρ(Γctxnorm,gCASI−Singleton)
-0.0699
+0.0092
[−0.0575,+0.0756]
Appendix
Table 7: Additive correlated-error null. Observed gaps are computed on the same 870 -request subset.
Metric
Real context
Shuffled context
Auction-positive rate
40.6%
32.7%
ρ(y,V0) on both-positive requests ( n=700 )
0.374
0.351
Appendix
Table 8: Matched real-context versus shuffled-context replay.
Collaborative multi-agent large language models (LLMs) can solve complex reasoning tasks by decomposing roles, but reinforcement learning for such systems is limited by credit assignment: shared terminal rewards obscure individual contributions and can encourage free-riding. We introduce two optimizer-agnostic credit assignment methods for converting joint outcomes into agent-specific learning signals. Counterfactual Credit for Policy Optimization (CCPO) estimates an agent's marginal contribution by comparing the realized joint outcome with a counterfactual outcome where that agent is removed. Self-Evaluated Credit for Policy Optimization (SEPO) uses constrained self- and peer-evaluations as a verifier-anchored credit signal while keeping the external task outcome dominant. Both operate at the reward-construction layer rather than as policy optimizers, producing role-specific rewards or advantages for GRPO, GSPO, or REINFORCE++. We instantiate these credit signals in a sequential Think--Solve setting and evaluate them on mathematical reasoning benchmarks. Results show that explicit credit assignment often improves dual-agent reasoning, especially on MATH500 and several out-of-distribution settings, while gains vary across models and datasets. Our code is available at: https://github.com/bhai114/ccpo.
Zhongyi Li, Wan Tian, Jinju Chen +4
Beihang University · Peking University · Beijing University of Posts and Telecommunications
Retrieval is increasingly moving from one-shot matching toward interactive reasoning, where language agents iteratively inspect evidence, reformulate queries, and search again. Training such agents raises a credit-assignment challenge: executable actions such as queries or summaries can be directly evaluated by the retriever, while latent reasoning steps are not directly observable and only affect future executable actions. This asymmetry makes outcome-level reward assignment unreliable, as the same final reward may credit reasoning steps that did not actually shape retrieval success. We propose RICE-PO, a critic-free policy optimization framework that converts retrieval interactions into localized learning signals. RICE-PO selects high-uncertainty executable actions as anchors, evaluates local counterfactual branches using retrieval metrics, and propagates credit to latent reasoning steps only when reasoning-to-action influence is strong and future residual effects are stable. On BRIGHT and BEIR, RICE-PO consistently outperforms prompt-based agents and group-based RL baselines under the same retriever setting. These results show that the structure of agent-environment interaction itself can provide useful supervision for training reasoning-based retrieval agents.
Mingchen Li, Hansi Zeng, Zhuo Qian +3
University of Massachusetts, Amherst · Texas Tech University · University of Connecticut
Step-level credit evaluation can conflate numerical association with ordering and branch recovery with position-selection value. We test these distinctions in ALFWorld with two independently seeded replay banks: 27,520 continuations over 172 fixed turns. At 16 continuations per arm per bank, cross-bank replay-contrast associations are Pearson 0.736 and pooled Spearman 0.330. On 30 historical failures, A-count-selected branches at confidence-/implicit-selected positions gain 8.54 percentage points over same-position factual reruns, but their 3.33-point advantage over frozen-random positions under the same alternative-selection rule remains unresolved. Numerical association alone does not establish reliable pooled ordering; branch recovery alone does not establish better score-informed position selection. Historical judge alignment is modestly positive with an unresolved archived-control increment. Same-cohort C/D information-source comparisons and a short, sparse-exposure training pilot provide scope checks; incremental hindsight training utility remains unestablished. Findings are conditional on the tested policies, intervention support and finite budgets.
Haiyue Zhang
University of Southern California, Los Angeles, CA, USA