CASI: Credit Assignment via Set Intervention for Agentic Retrieval

Xinyi Cheng; Tianfu Zhang; Jiwen Li; Xuesen Zhang; Shihao Yu; Pengfei Zhu; Renhuan; Heda Wang

cs.AIOct 14, 2026

CASI: Credit Assignment via Set Intervention for Agentic Retrieval

Authors: Xinyi Cheng, Tianfu Zhang, Jiwen Li, Xuesen Zhang, Shihao Yu, Pengfei Zhu, Renhuan, Heda Wang

Organizations: Xiaohongshu Inc.

Abstract

Large language models increasingly generate multiple retrieval actions jointly, while downstream systems often provide only one reward for the resulting set. Broadcasting this reward provides a valid score-function gradient estimator for the expected set-level objective, but does not determine a coalition-dependent within-set attribution. We introduce CASI (Credit Assignment via Set Intervention for Agentic Retrieval), which constructs the missing counterfactual information through snapshot-grounded set intervention: generate and retrieve once, hold replayable snapshot components fixed, and evaluate alternative subsets under common conditions. Shapley aggregation converts these marginals into within-set credit, which bounded zero-sum advantage shaping incorporates into policy optimization. For m >= 4, singleton, leave-one-out, and full-set endpoints do not generally determine the Shapley allocation; under additive set values, extra coalition evaluations are redundant. In a five-action sponsored-search setting, exhaustive auction replay over 1,200 requests reveals substantial coalition dependence. Under the training-time evaluator, CASI achieves the highest median cosine similarity to a replay-defined Shapley reference among evaluated credit rules; under a sensitivity backend, CASI does not outperform Singleton. Across five independent training runs, CASI-trained policies achieve higher held-out auction-replay set value than Singleton- or Broadcast-trained policies. In a randomized seven-day production A/B test on 10% of traffic, the CASI-trained policy yields an observed 3.51% lift in the primary advertiser-value metric over production control.

Figures & tables

Figure 1: CASI augments the broadcast rollout advantage with action-specific credit from snapshot-grounded set intervention. The base optimizer broadcasts $\hat{A}^{(g)}$ across the trajectory. Cached retrievals are recombined into coalitions and scored under the frozen training-side snapshot $\mathcal{E}_{0}$ , yielding coalition values $\hat{R}_{0}(S)$ . Shapley aggregation produces per-action credit, which bounded zero-sum advantage shaping maps to offsets $z_{i}^{(g)}$ . Tokens of action $q_{i}$ receive $\widetilde{A}_{i}^{(g)}=\hat{A}^{(g)}+\beta z_{i}^{(g)}$ ; shared tokens retain $\hat{A}^{(g)}$ .

Figure 2: Coalition dependence and credit alignment. Panels (a)–(b) diagnose context-dependent replay structure and size-resolved evaluator agreement. Panels (c)–(d) compare credit rules against the same replay-defined Shapley reference. Because panel (b) uses size-matched references, its values are not directly comparable with panels (c)–(d).

Credit context	Sign nz	Spearman	Top action	Median cosine
$\|S\|=0$	0.6424	0.3521	0.3274	0.5027
$\|S\|=1$	0.6356	0.3321	0.3422	0.4635
$\|S\|=2$	0.5994	0.2539	0.3365	0.3310
$\|S\|=3$	0.5683	0.1749	0.3112	0.2236
$\|S\|=4$	0.5381	0.1117	0.2902	0.1316
CASI (Shapley)	0.6178	0.3128	0.3299	0.4237

Table 1: Size-resolved evaluator–replay agreement under the training-time evaluator. Rows

|S|=k

compare request-centered action-credit vectors formed by averaging marginals over predecessor coalitions of size

k

, using the same construction for evaluator and replay. The CASI row instead compares evaluator-defined Shapley credit with the replay-defined Shapley reference and is not directly comparable across rows.

Credit rule	Median cosine $\uparrow$	CASI $-$ rule (95% CI)
CASI	0.4237	—
Singleton	0.3877	$+0.0360\ [0.0036,0.0677]$
Banzhaf	0.3800	$+0.0437\ [0.0083,0.0806]$
LOO	0.2269	$+0.1968\ [0.1272,0.2682]$

Table 2: Alignment with the replay-defined Shapley reference under the training-time evaluator. Median cosine is the primary metric. Confidence intervals are request-level paired-bootstrap intervals for the CASI-minus-alternative difference in median cosine.

Method	Mean replay value $\uparrow$	Top-1 eCPM $\uparrow$
SFT	60.32	11.25
Broadcast	43.30 $\,\pm\,$ 4.70	11.15 $\,\pm\,$ 0.10
Singleton	55.00 $\,\pm\,$ 4.00	11.68 $\,\pm\,$ 0.09
CASI	66.70 $\,\pm\,$ 3.40	11.93 $\,\pm\,$ 0.07

Table 3: Held-out policy evaluation on

1{,}000

common requests; RL entries report mean

\pm

standard deviation across five independent training runs.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Predecessor size	pr_filter	pr_rerun
All nonzero replay edges	0.5925	0.6137
$\|S\|=0$	0.9839	1.0000
$\|S\|=1$	0.6257	0.6572
$\|S\|=2$	0.5617	0.5819
$\|S\|=3$	0.5291	0.5485
$\|S\|=4$	0.5375	0.5348

Appendix

Table 4: Insertion-edge sign agreement with replay, stratified by predecessor size. pr_filter is the training backend; pr_rerun is a sensitivity backend.

Context	Sign all	Sign nz	Spearman	Top action	Median cosine
$\|S\|=0$	0.6056	0.6424	0.3521	0.3274	0.5027
$\|S\|=1$	0.6135	0.6356	0.3321	0.3422	0.4635
$\|S\|=2$	0.6003	0.5994	0.2539	0.3365	0.3310
$\|S\|=3$	0.5862	0.5683	0.1749	0.3112	0.2236
$\|S\|=4$	0.5687	0.5381	0.1117	0.2902	0.1316
Shapley / CASI	0.6029	0.6178	0.3128	0.3299	0.4237

Appendix

Table 5: Size-resolved credit fidelity for the training-time pr_filter evaluator. The

|S|=k

rows are size-matched diagnostics; the Shapley/CASI row compares against the replay-defined Shapley reference and is not directly comparable with the size-matched rows.

Rule	Sign all	Sign nz	Spearman	Top action	Median cosine
CASI	0.6029	0.6178	0.3128	0.3299	0.4237
Singleton	0.5947	0.6193	0.3135	0.3205	0.3877
Banzhaf	0.6060	0.6149	0.3014	0.3371	0.3800
LOO	0.5875	0.5693	0.1761	0.3226	0.2269

Appendix

Table 6: Full credit-fidelity metrics against centered replay-defined Shapley credit under the training-time pr_filter backend.

Statistic	Observed	Null median	Null 95% interval
$\Delta\cos_{\rm med}(\mathrm{CASI}-\mathrm{LOO})$	+0.2221	-0.1105	$[-0.1462,-0.0813]$
$\Delta\cos_{\rm med}(\mathrm{CASI}-\mathrm{Singleton})$	+0.0359	-0.0932	$[-0.1231,-0.0685]$
$\rho(\Gamma_{\rm ctx}^{\rm norm},g^{\rm CASI-LOO})$	-0.0270	+0.0052	$[-0.0630,+0.0758]$
$\rho(\Gamma_{\rm ctx}^{\rm norm},g^{\rm CASI-Singleton})$	-0.0699	+0.0092	$[-0.0575,+0.0756]$

Appendix

Table 7: Additive correlated-error null. Observed gaps are computed on the same

870

-request subset.

Metric	Real context	Shuffled context
Auction-positive rate	40.6%	32.7%
$\rho(y,V_{0})$ on both-positive requests ( $n=700$ )	0.374	0.351

Appendix

Table 8: Matched real-context versus shuffled-context replay.

Credit Assignment

Credit assignment studies how to attribute outcomes to the actions, agents, or components that caused them.

See all 71 papers

Counterfactual Credit Policy Optimization for Multi-Agent Collaboration
Mar 23, 2026Zhongyi Li, Wan Tian, Jinju Chen +4Counterfactual Credit Assignment RL for Language Model Reasoning
Beihang University · Peking University · Beijing University of Posts and Telecommunications
RICE-PO: Turning Retrieval Interactions into Credit Signals for Reasoning Agents
May 25, 2026Mingchen Li, Hansi Zeng, Zhuo Qian +3RL for Language Model Reasoning Credit Assignment in RL
University of Massachusetts, Amherst · Texas Tech University · University of Connecticut
Credit Without Ground Truth: Claim-Matched Evaluation of Step-Level Credit with Finite Executed Replay
Aug 20, 2026Haiyue ZhangLLM Agent Evaluation Step-Level Credit Assignment in RL
University of Southern California, Los Angeles, CA, USA

Mar 23, 2026J/K move · Enter open · S save

Counterfactual Credit Policy Optimization for Multi-Agent Collaboration

Zhongyi Li, Wan Tian, Jinju Chen, Huiming Zhang+3

Beihang University · Peking University · Beijing University of Posts and Telecommunications

Counterfactual Credit Assignment RL for Language Model Reasoning Multi-Agent Collaboration

Open paper AskPDF arXiv

Collaborative multi-agent large language models (LLMs) can solve complex reasoning tasks by decomposing roles, but reinforcement learning for such systems is limited by credit assignment: shared terminal rewards obscure individual contributions and can encourage free-riding. We introduce two optimizer-agnostic credit assignment methods for converting joint outcomes into agent-specific learning signals. Counterfactual Credit for Policy Optimization (CCPO) estimates an agent's marginal contribution by comparing the realized joint outcome with a counterfactual outcome where that agent is removed. Self-Evaluated Credit for Policy Optimization (SEPO) uses constrained self- and peer-evaluations as a verifier-anchored credit signal while keeping the external task outcome dominant. Both operate at the reward-construction layer rather than as policy optimizers, producing role-specific rewards or advantages for GRPO, GSPO, or REINFORCE++. We instantiate these credit signals in a sequential Think--Solve setting and evaluate them on mathematical reasoning benchmarks. Results show that explicit credit assignment often improves dual-agent reasoning, especially on MATH500 and several out-of-distribution settings, while gains vary across models and datasets. Our code is available at: https://github.com/bhai114/ccpo.

Abstract

Credit Assignment

Explore similar work

Counterfactual Credit Policy Optimization for Multi-Agent Collaboration

RICE-PO: Turning Retrieval Interactions into Credit Signals for Reasoning Agents

Credit Without Ground Truth: Claim-Matched Evaluation of Step-Level Credit with Finite Executed Replay