cs.LG · 2510.23486 Copy arXiv ID · Oct 27, 2025 Save Learning to Reason Efficiently with Discounted Reinforcement Learning Authors: Alex Ayoub , Kavosh Asadi , Dale Schuurmans , Csaba Szepesvári , Karim Bouyarmane
Organizations: Amazon, University of Alberta · Amazon · University of Alberta
Abstract Large reasoning models (LRMs) often consume excessive tokens, inflating computational cost and latency. More broadly, in goal reaching sequential decision problems we often want to reach the goal quickly, and LRM reasoning can be viewed through this lens. We challenge the assumption that longer responses improve accuracy. By penalizing reasoning tokens using a discounted reinforcement learning setup (interpretable as a small token cost) and analyzing Blackwell optimality in restricted policy classes, we encourage concise yet accurate reasoning, analogous to preferring shorter successful trajectories in a stochastic shortest path problem. Experiments confirm our theoretical results that this approach shortens chains of thought while preserving accuracy.
Explore similar work May 8, 2026 · Tingcheng Bian, Yuzhe Zhang, Jing Jin +5 Large Reasoning Models Thinking
May 10, 2026 · Songtao Wei, Yi Li, Zhikai Li +7 Large Reasoning Models Efficient Inference
Aug 5, 2026 · Qiyuan Zhu, Dezhi Li, Pengyu Cheng +8 Large Reasoning Models Key-Value Cache Compression
May 8, 2026 · cs.LG J/K move · Enter open · S save
Tingcheng Bian, Yuzhe Zhang, Jing Jin, Jinchang Luo +4
1Baidu Inc. · 2Shenzhen University · 3Peking University · 4Tsinghua University · 5D-INFK, ETH Zürich
Large reasoning models (LRMs) achieve strong performance via extended chain-of-thought (CoT) reasoning, yet suffer from excessive token consumption and high inference latency. Existing reinforcement learning (RL) approaches for CoT compression rely on uniform, static length penalties that neglect model capability dynamics and problem-level difficulty variation. We propose \textbf{ExpThink}\xspace, an RL framework that addresses both dimensions through two complementary mechanisms. First, \emph{experience-guided reward shaping} tracks the shortest correct solution found so far for each problem and applies a three-tier reward: full credit for concise correct responses, discounted credit for verbose correct ones, and zero for incorrect ones. The threshold tightens automatically with model improvement, forming a self-evolving curriculum that requires no manual scheduling. Second, \emph{difficulty-adaptive advantage} replaces standard deviation normalization with correct-count normalization, yielding monotonically difficulty-scaled gradients that amplify learning on hard problems to preserve accuracy while suppressing gradients on easy ones to encourage brevity. Together, these mechanisms enforce an accuracy-first, compression-second training objective. Experiments on multiple mathematical reasoning benchmarks demonstrate that \textbf{ExpThink}\xspace reduces average response length by up to 77% while simultaneously improving accuracy, achieving up to
3 × 3\times 3 × higher accuracy-efficiency ratio (accuracy divided by average token count) than the vanilla baseline and outperforming existing RL-based compression methods on both metrics.