Bellman-Consistent Trajectory RL for LLM Arithmetic Reasoning
Authors: Lianfa Li
Organizations: Department of Computer Science, Stanford University · LREIS, Institute of Geographical Sciences and Natural Resources Research, Chinese Academy of Sciences
To address sparse terminal rewards in LLM arithmetic reasoning, we propose a Bellman-consistent trajectory RL method. Our framework introduces Trajectory Bellman Residual Minimization (TBRM) to transform sparse rewards into dense token-level signals. We integrate a Reinforce Leave-One-Out (RLOO) group strategy to reduce gradient variance of Bellman residuals and use hindsight goal relabeling to accelerate intermediate learning. Evaluated on the Countdown task, our method alleviates optimization bottlenecks and stabilizes gradient variance significantly better than GRPO. The TBRM-RLOO framework boosts Pass@1 to 0.55--0.64 and achieves a state-of-the-art Pass@16 accuracy of 76.0--84.0%, outperforming the SFT, IPO/DPO, and RLOO baselines. Our approach demonstrates exceptional single-sample efficiency and successfully circumvents standard performance plateaus via test-time scaling.
Figures & tables
Figure 1: Overview of the Bellman-consistent trajectory RL framework, detailing the coupling of Trajectory Bellman Residual Minimization (TBRM) with group-wise RLOO baseline adjustment and hindsight goal relabeling.
Figure 2: Empirical evaluation of the impact of the temperature hyperparameter β on optimization stability and reward scaling.
Figure 3: Comparison of Bellman residual loss learning curves across the four evaluated TBRM variants.
Figure 4: Roll-out reward score trajectories during training across different TBRM configurations.
Figure 5: Comprehensive Pass@k performance comparison across baseline architectures and extension models.
Figure 6: Case study comparison of reasoning path generations for a representative countdown task.