Token-Level Credit Assignment

Overview

Token-level credit assignment studies how to turn sparse sequence-level feedback into learning signals for individual tokens, steps, or actions in language-model training. Methods include contrasting successful and failed rollouts, using uncertainty or hidden-state and attention signals, deriving token rewards from verifiers or execution traces, and coordinating token scores with critics or self-distillation. These approaches are applied to verifiable reasoning, code generation, multimodal grounding, and multi-turn tool use, where the important contribution may be concentrated in a small part of a response or trajectory. A central challenge is ensuring that dense signals genuinely predict task success: noisy teacher likelihoods, excessive reweighting, and replay variation can misdirect updates or destabilize training.

Latest papers

CardsList

60 papers6 papers in the last four weeks, down 50% on the four weeks before. 0.0% of all new papers.

  1. ACPO: Adaptive Credit Policy Optimization via Fine-Grained Surrogate Entropy

    Jul 3, 2026Zijun Xie, Yuyang You, Yongzhi Li +6Token-Level Credit AssignmentRL for Language Model Reasoning

  2. Learning from Own Solutions: Self-Conditioned Credit Assignment for Reinforcement Learning with Verifiable Rewards

    Jun 17, 2026Yingyu Shan, Yuhang Guo, Zihao Cheng +7Token-Level Credit AssignmentRL for Language Model Reasoning

  3. BALTO: Balanced Token-Level Policy Optimization for Hallucination Mitigation

    Jun 14, 2026Ning Li, Zixuan Guo, Yan Xu +7RL for Language ModelsLLM Hallucination Mitigation

  4. How Does Reasoning Flow? Tracing Attention-Induced Information Flow for Targeted RL in LLMs

    Jun 9, 2026Zhichen Dong, Yang Li, Yuhan Sun +9Token-Level Credit AssignmentRL for Language Model Reasoning

  5. PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshaping

    Jun 7, 2026Qiming Li, Tianlun Li, Xiaolong Cheng +5Token-Level Credit AssignmentVLM Reasoning

  6. TAPO: Tool-Aware Policy Optimization via Credit Transfer for Multimodal Search Agents

    Jun 4, 2026Chengqi Dong, Chuhuai Yue, Hang He +6Multimodal IRToken-Level Credit Assignment

  7. Entropy Is Not Enough: Unlocking Effective Reinforcement Learning for Visual Reasoning via Vision-Anchored Token Selection

    Jun 2, 2026Senjie Jin, Peixin Wang, Boyang Liu +8Token-Level Credit AssignmentVisual Reasoning

  8. Guidance Contrastive Token Credit Assignment for Discrete Policy Optimization

    May 28, 2026Shufan Li, Konstantinos Kallidromitis, Akash Gokul +2Token-Level Credit AssignmentRL for Language Model Reasoning

  9. OPPO: Bayesian Value Recursion for Token-Level Credit Assignment in LLM Reasoning

    May 21, 2026Yu Li, Rui Miao, Tian Lan +1Value Function EstimationToken-Level Credit Assignment

  10. DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards

    May 20, 2026Kaiyi Zhang, Wei Wu, Yankai LinPolicy GradientToken-Level Credit Assignment

  11. CEPO: RLVR Self-Distillation using Contrastive Evidence Policy Optimization

    May 19, 2026Ahmed Heakl, Abdelrahman M. Shaker, Youssef Mohamed +4Token-Level Credit AssignmentRL for Language Model Reasoning

  12. AMR-SD: Asymmetric Meta-Reflective Self-Distillation for Token-Level Credit Assignment

    May 18, 2026Zhenlin Wei, Pu Jian, Yingzhuo Deng +6Token-Level Credit AssignmentRL for Language Model Reasoning

  13. Resolving Action Bottleneck: Agentic Reinforcement Learning Informed by Token-Level Energy

    May 14, 2026Langzhou He, Junyou Zhu, Yue Zhou +7Agentic RLToken-Level Credit Assignment

  14. From Generic Correlation to Input-Specific Credit in On-Policy Self Distillation

    May 12, 2026Guobin Shen, Lei Huang, Xiang Cheng +4Reinforcement LearningOn-Policy Self-Distillation

  15. The Cancellation Hypothesis in Critic-Free RL: From Outcome Rewards to Token Credits

    May 9, 2026Tianhao Cheng, Zeyu Huang, Zihan Qiu +5Reinforcement LearningToken-Level Credit Assignment

  16. Not All Tokens Learn Alike: Attention Entropy Reveals Heterogeneous Signals in RL Reasoning

    May 8, 2026Gengyang Li, Zheng-Fan Wu, Siqi Bao +1Token-Level Credit AssignmentRL for Language Model Reasoning

  17. Structured Role-Aware Policy Optimization for Multimodal Reasoning

    May 8, 2026Bingqing Jiang, Difan ZouToken-Level Credit AssignmentMultimodal Reasoning

  18. Hidden States Know Where Reasoning Diverges: Credit Assignment via Span-Level Wasserstein Distance

    Apr 25, 2026Xinzhu Chen, Wei He, Huichuan Fan +7Token-Level Credit AssignmentGroup Relative Policy Optimization

  19. Where Hindsight Credit Can Reside: A Signed-Capacity View of Token Updates in RLVR

    Apr 13, 2026Yuhang He, Haodong Wu, Siyi Liu +7RL for Language Model ReasoningToken-Level Credit Assignment

  20. Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Models

    Apr 2, 2026Zekai Ye, Qiming Li, Xiaocheng Feng +6RL for VLMsLarge Vision-Language Models

  21. DEX-AR: A Dynamic Explainability Method for Autoregressive Vision-Language Models

    Mar 6, 2026Walid Bousselham, Angie Boggust, Hendrik Strobelt +1VLM InterpretabilityToken-Level Credit Assignment

  22. STAPO: Stabilizing Reinforcement Learning for LLMs by Silencing Rare Spurious Tokens

    Feb 17, 2026Shiqi Liu, Zeyu He, Guojian Zhan +10Reinforcement LearningRL for Language Model Reasoning