Bellman Policy Optimization

Zhuoqing Song; Haotian Xu; Xikun Zhang; Lidong Bing

cs.LGSep 14, 2026

Bellman Policy Optimization

Authors: Zhuoqing Song, Haotian Xu, Xikun Zhang, Lidong Bing

Organizations: 1Apodex US, Inc. · 2Princeton University

Abstract

Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models (LLMs). We introduce Bellman Policy Optimization (BPO), a critic-free method derived from Policy Mirror Descent (PMD). For autoregressive generation with terminal rewards, BPO uses the Bellman equations to reformulate PMD as a trajectory-level objective. The reformulation avoids estimating state values at intermediate states. We prove that it has the same unique optimal solution as the original PMD objective. We derive the practical BPO loss by approximating this objective. Its mismatch-correction weight is a smoothed ratio of complementary token probabilities. Experiments on mathematical reasoning benchmarks demonstrate the effectiveness of BPO.

Explore similar work

VIMPO: Value-Implicit Policy Optimization for LLMs
Jun 18, 2026Zhewei Kang, Aosong Feng, Sergey Levine +2Reinforcement Learning With Verifiable Reward Value Functions
LamPO: A Lambda Style Policy Optimization for Reasoning Language Models
May 20, 2026Redacted by arXivReinforcement Learning With Verifiable Reward Group Relative Policy Optimization
Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal States
May 8, 2026Yunho Choi, Jongwon Lim, Woojin Ahn +3Reinforcement Learning With Verifiable Reward Verifiable Rewards

Jun 18, 2026 · cs.LGJ/K move · Enter open · S save

VIMPO: Value-Implicit Policy Optimization for LLMs

Zhewei Kang, Aosong Feng, Sergey Levine, Dawn Song+1

Reinforcement learning with verifiable rewards has become a central tool for improving the reasoning ability of large language models, but current methods face a trade-off between simplicity and credit assignment. Group-relative methods such as GRPO avoid training a critic, but typically assign a trajectory-level advantage to every token. Actor-critic methods provide denser learning signals, but require a learned value function with its own training instability. We introduce VIMPO, a critic-free policy optimization method that derives a policy-implied value function from the optimality conditions of KL-regularized reinforcement learning. For autoregressive generation, the resulting value recurrence can be written in terms of policy-reference log-ratios and anchored by the terminal condition that no future reward remains at the end of a trajectory. This gives a simple value loss that incorporates outcome-level verifiable rewards without training a critic. The same derivation also yields a critic-free actor advantage, allowing VIMPO to separate reward incorporation through the value loss from policy improvement through a PPO-style actor update. On mathematical RLVR benchmarks, VIMPO improves over GRPO across MATH-500, AIME 2024, AIME 2025, and OlympiadBench, with especially larger gains on competition-style evaluations. Under noisy rewards, VIMPO retains a consistent advantage over GRPO, suggesting that policy-implied value optimization can provide finer credit assignment while preserving the practical simplicity of critic-free training.

Reinforcement Learning With Verifiable Reward Value Functions Vista

Open paper Ask about it PDF

Abstract

Explore similar work

VIMPO: Value-Implicit Policy Optimization for LLMs

LamPO: A Lambda Style Policy Optimization for Reasoning Language Models

Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal States