Token-Level Credit Assignment

Overview

Token-level credit assignment studies how to turn sparse sequence-level feedback into learning signals for individual tokens, steps, or actions in language-model training. Methods include contrasting successful and failed rollouts, using uncertainty or hidden-state and attention signals, deriving token rewards from verifiers or execution traces, and coordinating token scores with critics or self-distillation. These approaches are applied to verifiable reasoning, code generation, multimodal grounding, and multi-turn tool use, where the important contribution may be concentrated in a small part of a response or trajectory. A central challenge is ensuring that dense signals genuinely predict task success: noisy teacher likelihoods, excessive reweighting, and replay variation can misdirect updates or destabilize training.

Latest papers

CardsList

60 papers6 papers in the last four weeks, down 50% on the four weeks before. 0.0% of all new papers.

  1. SARGO: Agreement-Gated Reinforcement and Suppression via On-Policy Self-Guidance for Visual Hallucination Mitigation

    Nov 14, 2026Jinyu Li, Zhiming Zhang, Hao Fang +4VLM Hallucination MitigationRL for VLMs

  2. When Credit Turns Against Tools: Transition-Aware Credit Assignment in Agentic Reinforcement Learning

    Nov 8, 2026Xiangyu Lu, Zhanke Zhou, Jiazhe Ning +3Agentic RLCredit Assignment

  3. Using Experience Twice: Dynamic Process Rubrics for Privileged Token-Level Credit Assignment

    Nov 4, 2026Shumao Wu, Kongcheng Zhang, Jianqiang Huang +1Token-Level Credit AssignmentFine-Grained Credit Assignment

  4. Direction-Aware Token Credit Assignment with Reward-Model Gradients

    Oct 23, 2026Yikuan Hu, Jifeng Zhu, Minle Su +6Token-Level Credit AssignmentPolicy Gradient Methods

  5. Outcome-Anchored Credit Reallocation for Agentic Knowledge Graph Question Answering

    Oct 18, 2026Mingyuan Li, Yucheng Li, Xiaxia Wang +1Knowledge Graph Question AnsweringRL for LLM Agents

  6. P2TPO: Structure-Aligned Page-to-Token Credit Assignment for Full-Page Document Parsing

    Oct 15, 2026Yong Zu, Ninghua Yang, Zezhong Tan +3Document ParsingToken-Level Credit Assignment

  7. Enhancing Multimodal RLVR with Pairwise Token Contrast and Adaptive Trajectory Confidence

    Oct 15, 2026Yiran Zhang, Hanbo Huang, Jielan Li +3RL for VLMsMultimodal Reasoning

  8. Each Reward to Its Relevant Tokens: Reward-Specific Token-Level Credit Assignment for Multi-Reward RL

    Oct 15, 2026Skylar Zhai, Haonan Wang, Zhining Liu +4RL for Language ModelsToken-Level Credit Assignment

  9. Unlocking Token Rewards via Training-Free Reward Attribution

    Oct 13, 2026Sitong Wu, Haoru Tan, Bin Xia +6Token-Level Credit AssignmentRL for Language Model Reasoning

  10. CAST: Credit Alignment for Cross-Tokenizer On-Policy Distillation

    Oct 9, 2026Jingru Yang, Jinhang Wu, Minjing Shi +4Cross-Tokenizer Knowledge DistillationLanguage Model Distillation

  11. VETTA: Coordinating Turn- and Token-Level Credit Assignment for Multi-Turn LLM Agents

    Oct 6, 2026Jiaju Chen, Min Yang, Jinghua Piao +4Token-Level Credit AssignmentCredit Assignment in RL

  12. Beyond Entropy: Self-Diagnostic Multi-Role Token Optimization for Video Reasoning

    Oct 2, 2026Yudong Han, Yong Wang, Zaiquan Yang +4Token-Level Credit AssignmentRL for VLMs

  13. Token-Level Video Reinforcement Learning

    Oct 1, 2026Yifan Wang, Gordon Guocheng Qian, Yanyu Li +2Video Diffusion ModelsRL for Video Generation

  14. Can We Trust the Teacher? Decoupled Credit Direction-Magnitude for Self-Distillation

    Sep 28, 2026Yugu Li, Zehong Cao, Peizhen Li +3Token-Level Credit AssignmentRL for Language Model Reasoning

  15. TIAO: Token Importance-Aware Policy Optimization for Text Summarization

    Sep 15, 2026Qixiu Li, Chenlong Bao, Xiang Zhu +4RL for Language ModelsToken-Level Credit Assignment

  16. Beyond Teacher Likelihood: Group-Calibrated On-Policy Distillation for Long-Context Reasoning

    Aug 19, 2026Zhu Zhang, Jixun Wang, Xiaoang Xu +6On-Policy DistillationKnowledge Distillation

  17. Le Critique: Privileged Value Functions for LLM Reinforcement Learning

    Aug 17, 2026Siddarth Venkatraman, Matthieu Dinot, Laurence AitchisonRL for Language ModelsRL for Language Model Reasoning

  18. Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents

    Aug 13, 2026Zechuan Wang, Siyuan Lu, Hongxuan Zhang +3Token-Level Credit AssignmentStep-Level Credit Assignment in RL

  19. Se-DPO: Self-Evolving Token Credit for Direct Preference Optimization

    Aug 10, 2026Wenxiao Zhao, Shu Wang, Ying Nian WuLLM AlignmentDirect Preference Optimization

  20. How Much, Then Where: Credit-Conserving Action-to-Token Allocation for Multi-Turn Agent Reinforcement Learning

    Aug 7, 2026Lichao Ma, Yang Sun, Shuaitao Zhao +9Token-Level Credit AssignmentCredit Assignment in RL

  21. Agentic Reinforcement Learning with Self-Distilled Reward Shaping

    Aug 4, 2026Ranxu Zhang, Guinan Chen, Chenshaodong +5Reward ShapingToken-Level Credit Assignment

  22. Not All Tokens Deserve Equal Credit: Counterfactual Sensitivity Credit Reallocation for Long-CoT Reasoning

    Jul 30, 2026Qiangqiang He, Zhongheng Wu, ZiJian WangToken-Level Credit AssignmentRL for Language Model Reasoning

  23. CoRT: Counterfactual Replay for Token-Level Rubric-Guided Policy Optimization

    Jul 28, 2026Bo-Wen Zhang, Junwei He, Wen Wang +5Token-Level Credit AssignmentRubric-Based RL

  24. Right Answer, Wrong Tokens: Tail-Aware Credit Calibration for RLVR

    Jul 8, 2026Xiuyi Lou, Zicheng Xu, Yu-Neng Chuang +4RL for Language ModelsToken-Level Credit Assignment