Counterfactual Credit Assignment

Follow this topic Ask about this topic

Counterfactual credit assignment studies how to determine which actions, reasoning steps, tokens, or agents actually contributed to a reward, especially when feedback arrives only at the end of a task. It matters in cooperative multi-agent systems and language-model reasoning, where a shared outcome can hide useful contributions or harmful decisions. Common approaches compare paired rollouts, intervene on actions or messages, reset from intermediate states, or estimate marginal and Shapley contributions; these signals can then train policies with methods such as PPO and GRPO. A central challenge is making counterfactuals faithful and affordable: replay noise and interacting contributions can distort attribution, so reliable evaluation against executed counterfactual outcomes remains important.

Latest papers

28 papers6 papers in the last four weeks, up 100% on the four weeks before. 0.0% of all new papers.

Same Luck, Different Choice: Twin-World Credit Assignment for Agentic Reinforcement Learning
Oct 22, 2026Zeropy Surio, Tobias Engel, Mei-Lin Chou +2Step-Level Credit Assignment in RL Counterfactual Credit Assignment
University of Sheffield · Carnegie Mellon University · University of Michigan
RoleFlow: Self-Evolving Multi-Role Agentic Collaboration via Structural Adaptation
Oct 21, 2026Yi Ge, Pan Wang, Xuehao Cui +4Multi-Agent LLM Systems LLM Agent Workflow Optimization
University of Maryland, College Park · University of Pittsburgh · The University of Texas at Dallas
Learning from Revision Consequences: Hindsight Meta-Experience Distillation for Self-Improving Agents
Oct 6, 2026Qianhan Feng, Zhongzhen Huang, Yakun Zhu +2LLM Agent Skill Learning Self-Improving Agents
The Chinese University of Hong Kong · Shanghai Jiao Tong University · Shanghai Innovation Institue
Beyond Entropy: Self-Diagnostic Multi-Role Token Optimization for Video Reasoning
Oct 2, 2026Yudong Han, Yong Wang, Zaiquan Yang +4Token-Level Credit Assignment RL for VLMs
Computer Science, Beijing Institute of Technology · AMAP, Alibaba Group · City University of Hong Kong +2
VERA: Verifiable Feasibility Representations with Counterfactual Credit for Constrained Multi-Agent Control
Sep 30, 2026Bo Yin, Dongbo Li, Hongkai Chen +2Multi-Agent Control Constrained RL
Harbin Institute of Technology, Harbin, China · The Chinese University of Hong Kong, Hong Kong SAR, China
ASCT: Attentive Search over Counterfactual Trees for Credit Assignment in Agentic Reinforcement Learning
Sep 28, 2026Yang Li, Jinhan Yang, hai liu +8Agentic RL Credit Assignment in RL
University of Hong Kong · The Chinese University of Hong Kong · Jiangxi Science and Technology Normal University +1
Learning What to Skip: Counterfactual Credit Assignment for Efficient Multi-Agent LLM Workflows
Sep 25, 2026Jinfeng Xu, Zheyu Chen, Ziyue Peng +6LLM Agent Workflow Optimization Multi-Agent LLM Systems
The University of British Columbia, Canada · The Hong Kong Polytechnic University, Hong Kong · The Hong Kong University of Science and Technology, Hong Kong +4
CounterRoute: Self-Routed Reasoning via Hierarchical Counterfactual Credit Assignment
Sep 24, 2026Ruochen Jiao, Besnik Fetahu, Zhenyu Shi +1RL for Language Model Reasoning Efficient Language Model Reasoning
CREDO: Variance-Guided Rubric Evolution for Replay-Corrected Credit Assignment
Sep 21, 2026Xuchun HuCredit Assignment in RL Step-Level Credit Assignment in RL
Credit Without Ground Truth: Claim-Matched Evaluation of Step-Level Credit with Finite Executed Replay
Aug 20, 2026Haiyue ZhangLLM Agent Evaluation Step-Level Credit Assignment in RL
University of Southern California, Los Angeles, CA, USA
Not All Tokens Deserve Equal Credit: Counterfactual Sensitivity Credit Reallocation for Long-CoT Reasoning
Jul 30, 2026Qiangqiang He, Zhongheng Wu, ZiJian WangToken-Level Credit Assignment RL for Language Model Reasoning
State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · Institute of Wireless Communications Technology, Shanghai Jiao Tong University, Shanghai, China
CoRT: Counterfactual Replay for Token-Level Rubric-Guided Policy Optimization
Jul 28, 2026Bo-Wen Zhang, Junwei He, Wen Wang +5Token-Level Credit Assignment Rubric-Based RL
State Key Laboratory of Novel Software Technology, Nanjing University · School of Intelligence Science and Technology, Nanjing University · ByteDance +1
A Self-Evolving Default Action for Cooperative Tasks with Continuous Action Space
Jul 21, 2026Shuangyao HuangPolicy Gradient Credit Assignment
Xi’an Jiaotong-Liverpool University School of Internet of Things
Counterfactual Shapley Credit Assignment
Jul 18, 2026Mingxuan Li, Kai-Zhan Lee, Elias BareinboimCredit Assignment in RL Temporal Credit Assignment
Department of Computer Science, Columbia University, New York, NY, USA
CRAFT: Counterfactual Credit Assignment from Free Sibling Rollouts for Self-Distilled Agentic Reinforcement Learning
Jun 28, 2026Zibin Meng, Kani ChenAgentic RL KL-Regularized RL
The Hong Kong University of Science and Technology
Re-feeding Is Not Replaying: Measuring Replay Noise in Counterfactual Token-Credit Estimation
Jun 14, 2026Nils MattesonCounterfactual Evaluation Token-Level Credit Assignment
Northeastern University
Causal Agent Replay: Counterfactual Attribution for LLM-Agent Failures
Jun 6, 2026Jaineet ShahAgent Failure Analysis AI Agent Reliability
RICE-PO: Turning Retrieval Interactions into Credit Signals for Reasoning Agents
May 25, 2026Mingchen Li, Hansi Zeng, Zhuo Qian +3RL for Language Model Reasoning Credit Assignment in RL
University of Massachusetts, Amherst · Texas Tech University · University of Connecticut
Credit Assignment with Resets in Language Model Reasoning
May 25, 2026Ankur Samanta, Akshayaa Magesh, Ayush Jain +7RL for Language Model Reasoning Credit Assignment in RL
1Meta AI · 2Columbia University · *Work done at Meta +2
Barriers to Counterfactual Credit Attribution for Autoregressive Models
May 2, 2026Aloni Cohen, Chenhao ZhangGenerative Modeling Counterfactual Credit Assignment
Reducing Credit Assignment Variance via Counterfactual Reasoning Paths
Apr 20, 2026Fei Ding, Yongkang Zhang, Youwei Wang +1RL for Language Model Reasoning Credit Assignment in RL
Alibaba Group · Tsinghua University
COSAC: Counterfactual Credit Assignment in Sequential Cooperative Teams
Apr 20, 2026Shripad Deshmukh, Jayakumar Subramanian, Raghavendra Addanki +1Sequential MARL Credit Assignment in RL
College of Information and Computer Sciences University of Massachusetts Amherst · Adobe Research San Jose, CA
From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models
Apr 10, 2026Chenchen ZhangCredit Assignment in RL RL for Language Models
Independent Researcher
Design Conditions for Intra-Group Learning of Sequence-Level Rewards: Token Gradient Cancellation
Apr 4, 2026Fei Ding, Yongkang Zhang, youwei wang +1RL for Language Model Reasoning Step-Level Credit Assignment in RL
Alibaba Group · Tsinghua University
Marginal-Contribution Policy Gradients under Filtered Feedback for Multi-Agent LLMs
Apr 3, 2026Elai Ben-Gal, Stela TongMulti-Agent LLM Systems Multi-Agent System Optimization
Department of Mathematics Stanford University · Graduate School of Business Stanford University
Counterfactual Credit Policy Optimization for Multi-Agent Collaboration
Mar 23, 2026Zhongyi Li, Wan Tian, Jinju Chen +4Counterfactual Credit Assignment RL for Language Model Reasoning
Beihang University · Peking University · Beijing University of Posts and Telecommunications
The Trace Is the State: Exact Credit Assignment for LLM Agent Teams
Mar 6, 2026Yanjun Chen, Yirong Sun, Hanlin Wang +5Credit Assignment Counterfactual Credit Assignment
Eastern Institute of Technology · The Hong Kong Polytechnic University · Harbin Institute of Technology
Transformer-Based Multi-Agent Reinforcement Learning for Networked Systems with Long-Range Interactions
Nov 17, 2025Vidur Sinha, Muhammed Ustaomeroglu, Guannan QuCounterfactual Credit Assignment Graph Transformer
Department of Electrical and Computer Engineering, Carnegie Mellon University, Pittsburgh, USA

Oct 22, 2026J/K move · Enter open · S save

Same Luck, Different Choice: Twin-World Credit Assignment for Agentic Reinforcement Learning

Zeropy Surio, Tobias Engel, Mei-Lin Chou, Anika Raman+1

University of Sheffield · Carnegie Mellon University · University of Michigan

Step-Level Credit Assignment in RL Counterfactual Credit Assignment Agentic RL

Open paper AskPDF arXiv

Agentic reinforcement learning rewards a complete episode, yet the policy needs credit for every decision inside it. Counterfactual branching estimates this credit by re-running the episode after an alternative decision, but the branch draws its sampling noise independently of the factual episode, so the outcome difference mixes the effect of the decision with the luck of everything that follows. We observe that in agentic search the retriever, the corpus, and the grader are deterministic, so an episode is a deterministic function of the policy's decisions and its sampling noise. Twin worlds are therefore realizable: the same noise, a different decision. We introduce TWICE (TWIn-world Credit Estimation), which keys the Gumbel noise of every token by turn and line, rolls out counterfactual twins that inherit the factual episode's downstream noise, and turns their paired outcome differences into turn-level advantages for policy-gradient updates on actor-sampled trajectories; only the actor is deployed. Twin credit is unbiased for any coupling quality, its variance is that of an independent counterfactual scaled by

1-ρ

, where

ρ

is the twins' outcome correlation, and under coupling the number of rollouts needed to resolve an advantage of size

Δ

falls from

Θ(Δ^{-2})

Θ(Δ^{-1})

. A square-root rule spends the twin budget where outcomes are predicted to disagree per unit cost. On seven question-answering benchmarks, TWICE improves the average exact match of GRPO by 5.3 and 4.5 points on Qwen2.5-3B and 7B, outperforms tree-search and counterfactual-branching baselines, and uses 68% fewer auxiliary tokens than VinePPO.