Token-Level Credit Assignment

Follow this topic Ask about this topic

Token-level credit assignment studies how to turn sparse sequence-level feedback into learning signals for individual tokens, steps, or actions in language-model training. Methods include contrasting successful and failed rollouts, using uncertainty or hidden-state and attention signals, deriving token rewards from verifiers or execution traces, and coordinating token scores with critics or self-distillation. These approaches are applied to verifiable reasoning, code generation, multimodal grounding, and multi-turn tool use, where the important contribution may be concentrated in a small part of a response or trajectory. A central challenge is ensuring that dense signals genuinely predict task success: noisy teacher likelihoods, excessive reweighting, and replay variation can misdirect updates or destabilize training.

Latest papers

55 papers12 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Direction-Aware Token Credit Assignment with Reward-Model Gradients
Oct 23, 2026Yikuan Hu, Jifeng Zhu, Minle Su +6Token-Level Credit Assignment Policy Gradient Methods
Nanjing University · National University of Singapore · Tsinghua University +1
Outcome-Anchored Credit Reallocation for Agentic Knowledge Graph Question Answering
Oct 18, 2026Mingyuan Li, Yucheng Li, Xiaxia Wang +1Knowledge Graph Question Answering RL for LLM Agents
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China · Department of Computer Science, University of Oxford
P2TPO: Structure-Aligned Page-to-Token Credit Assignment for Full-Page Document Parsing
Oct 15, 2026Yong Zu, Ninghua Yang, Zezhong Tan +3Document Parsing Token-Level Credit Assignment
Qwen Business Unit, Alibaba · Xidian University
Enhancing Multimodal RLVR with Pairwise Token Contrast and Adaptive Trajectory Confidence
Oct 15, 2026Yiran Zhang, Hanbo Huang, Jielan Li +3RL for VLMs Multimodal Reasoning
Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory
Each Reward to Its Relevant Tokens: Reward-Specific Token-Level Credit Assignment for Multi-Reward RL
Oct 15, 2026Skylar Zhai, Haonan Wang, Zhining Liu +4RL for Language Models Token-Level Credit Assignment
University of Minnesota · National University of Singapore · University of Illinois Urbana-Champaign +3
Unlocking Token Rewards via Training-Free Reward Attribution
Oct 13, 2026Sitong Wu, Haoru Tan, Bin Xia +6Token-Level Credit Assignment RL for Language Model Reasoning
Chinese University of Hong Kong · University of Hong Kong · Hong Kong University of Science and Technology
CAST: Credit Alignment for Cross-Tokenizer On-Policy Distillation
Oct 9, 2026Jingru Yang, Jinhang Wu, Minjing Shi +4Cross-Tokenizer Knowledge Distillation Language Model Distillation
Taobao & Tmall Group of Alibaba · Singapore Management University
VETTA: Coordinating Turn- and Token-Level Credit Assignment for Multi-Turn LLM Agents
Oct 6, 2026Jiaju Chen, Min Yang, Jinghua Piao +4Token-Level Credit Assignment Credit Assignment in RL
University of Science and Technology of China · Zhongguancun Academy · Shandong University +2
Beyond Entropy: Self-Diagnostic Multi-Role Token Optimization for Video Reasoning
Oct 2, 2026Yudong Han, Yong Wang, Zaiquan Yang +4Token-Level Credit Assignment RL for VLMs
Computer Science, Beijing Institute of Technology · AMAP, Alibaba Group · City University of Hong Kong +2
Token-Level Video Reinforcement Learning
Oct 1, 2026Yifan Wang, Gordon Guocheng Qian, Yanyu Li +2Video Diffusion Models RL for Video Generation
Northeastern University
Reinforcing Multimodal Reasoning via Token-Level Perception-Grounded Advantage Estimation
Sep 30, 2026Zhihan Zhang, Lizi LiaoToken-Level Credit Assignment Multimodal Large Language Models
Singapore Management University Singapore, Singapore
Can We Trust the Teacher? Decoupled Credit Direction-Magnitude for Self-Distillation
Sep 28, 2026Yugu Li, Zehong Cao, Peizhen Li +3Token-Level Credit Assignment RL for Language Model Reasoning
School of CSIT, Adelaide University, Adelaide, SA 5000, Australia · CAIAA, University of North Texas, Denton, TX 76203, USA · School of EECMS, Curtin University, Bentley, WA 6102, Australia +1
PACT: From Credit Assignment to Critic Alignment
Sep 22, 2026Jiayan Fu, Hang Xu, Yong Zhang +4Reinforcement Learning Token-Level Credit Assignment
AllSpark Team
TIAO: Token Importance-Aware Policy Optimization for Text Summarization
Sep 15, 2026Qixiu Li, Chenlong Bao, Xiang Zhu +4RL for Language Models Token-Level Credit Assignment
National University of Defense Technology
Let Credit Follow Computation: Architecture-Aware Credit Transport for Large Language Model Reinforcement Learning
Aug 21, 2026Qifan Shi, Zhaolu Kang, Chenghua ZhuToken-Level Credit Assignment Credit Assignment in RL
Beyond Teacher Likelihood: Group-Calibrated On-Policy Distillation for Long-Context Reasoning
Aug 19, 2026Zhu Zhang, Jixun Wang, Xiaoang Xu +6On-Policy Distillation Knowledge Distillation
Tsinghua University · OpenBMB · Beijing University of Posts and Telecommunications
Le Critique: Privileged Value Functions for LLM Reinforcement Learning
Aug 17, 2026Siddarth Venkatraman, Matthieu Dinot, Laurence AitchisonRL for Language Models RL for Language Model Reasoning
Mistral AI · Mila – Quebec AI Institute · Université de Montréal
Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents
Aug 13, 2026Zechuan Wang, Siyuan Lu, Hongxuan Zhang +3Token-Level Credit Assignment Step-Level Credit Assignment in RL
Zhejiang University · AWorld Team, Inclusion AI · Shanghai Innovation Institute +2
Se-DPO: Self-Evolving Token Credit for Direct Preference Optimization
Aug 10, 2026Wenxiao Zhao, Shu Wang, Ying Nian WuLLM Alignment Direct Preference Optimization
University of California, Los Angeles · Shanghai AI Laboratory
Privileged Likelihood Is Not Automatically Value: Three Checks for Token Credit in On-Policy Self-Distillation
Aug 10, 2026Xuan-Phi Nguyen, Zeyu Leo Liu, Yang Li +4On-Policy Self-Distillation Token-Level Credit Assignment
Salesforce AI Research
How Much, Then Where: Credit-Conserving Action-to-Token Allocation for Multi-Turn Agent Reinforcement Learning
Aug 7, 2026Lichao Ma, Yang Sun, Shuaitao Zhao +9Token-Level Credit Assignment Credit Assignment in RL
1Peking University · 2Meituan LongCat Interaction Team · 3Fudan University +4
Agentic Reinforcement Learning with Self-Distilled Reward Shaping
Aug 4, 2026Ranxu Zhang, Guinan Chen, Chenshaodong +5Reward Shaping Token-Level Credit Assignment
Alibaba Group · University of Science and Technology of China Hefei, China · Alibaba Group Hangzhou, China
Not All Tokens Deserve Equal Credit: Counterfactual Sensitivity Credit Reallocation for Long-CoT Reasoning
Jul 30, 2026Qiangqiang He, Zhongheng Wu, ZiJian WangToken-Level Credit Assignment RL for Language Model Reasoning
State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · Institute of Wireless Communications Technology, Shanghai Jiao Tong University, Shanghai, China
CoRT: Counterfactual Replay for Token-Level Rubric-Guided Policy Optimization
Jul 28, 2026Bo-Wen Zhang, Junwei He, Wen Wang +5Token-Level Credit Assignment Rubric-Based RL
State Key Laboratory of Novel Software Technology, Nanjing University · School of Intelligence Science and Technology, Nanjing University · ByteDance +1
Right Answer, Wrong Tokens: Tail-Aware Credit Calibration for RLVR
Jul 8, 2026Xiuyi Lou, Zicheng Xu, Yu-Neng Chuang +4RL for Language Models Token-Level Credit Assignment
Johns Hopkins University · Rice University · Workato +1
ACPO: Adaptive Credit Policy Optimization via Fine-Grained Surrogate Entropy
Jul 3, 2026Zijun Xie, Yuyang You, Yongzhi Li +6Token-Level Credit Assignment RL for Language Model Reasoning
Peking University · Baidu Inc. · Kuaishou Inc.
On the Policy Gradient Foundations of Group Relative Policy Optimization: Credit Assignment, Gradient Sparsity, and Rank Collapse
Jun 28, 2026Amritansh Mishra, Supriyo Chakraborty, Berkcan KapusuzogluPolicy Gradient Token-Level Credit Assignment
Capital One
Learning from Own Solutions: Self-Conditioned Credit Assignment for Reinforcement Learning with Verifiable Rewards
Jun 17, 2026Yingyu Shan, Yuhang Guo, Zihao Cheng +7Token-Level Credit Assignment RL for Language Model Reasoning
Beijing Institute of Technology · Beihang University · Independent Researcher
BALTO: Balanced Token-Level Policy Optimization for Hallucination Mitigation
Jun 14, 2026Ning Li, Zixuan Guo, Yan Xu +7RL for Language Models LLM Hallucination Mitigation
1Shanghai Jiao Tong University · 2Tencent · 3The
Re-feeding Is Not Replaying: Measuring Replay Noise in Counterfactual Token-Credit Estimation
Jun 14, 2026Nils MattesonCounterfactual Evaluation Token-Level Credit Assignment
Northeastern University

Oct 23, 2026J/K move · Enter open · S save

Direction-Aware Token Credit Assignment with Reward-Model Gradients

Yikuan Hu, Jifeng Zhu, Minle Su, Jiaqin Yue+5

Nanjing University · National University of Singapore · Tsinghua University+1

Token-Level Credit Assignment Policy Gradient Methods Group Relative Policy Optimization Gradient-Based Attribution

Open paper AskPDF arXiv

GRPO shares a response-level advantage across tokens, leaving unresolved how strongly each token should contribute to learning. Updates induced by the shared advantage reinforce or suppress the sampled token at each position, but these updates do not always point toward increasing reward at the token level. This motivates a work-based view of credit: just as work depends on the component of force along a displacement, token credit should reflect the local reward change along the policy update direction. We introduce Direction-Aware Token Credit Assignment (DATC), which maps frozen reward-model gradients into policy logit space and allocates weights by evaluating the alignment between the reward direction and GRPO's signed sampled-action update direction, while preserving the sign and mean of the response-level advantage. DATC outperforms GRPO in all four settings across two model families.