Value Function Estimation

Follow this topic Ask about this topic

Value functions estimate the expected future reward or outcome of a state or action, helping agents compare choices, evaluate policies, and assign credit to earlier decisions. Common approaches include Monte Carlo returns, temporal-difference learning, and fitted value or Q-function regression, often using neural networks and Bellman updates. Current work also studies off-policy evaluation and uncertainty, and extracts value signals from language-model and robot-policy representations; methods are tested in settings such as D4RL, Atari, MuJoCo, long-horizon control, and language-model reasoning. A central challenge is balancing low-variance learning with bias from bootstrapping, limited data coverage, and unreliable estimates under distribution shift.

Latest papers

68 papers6 papers in the last four weeks, down 14% on the four weeks before. 0.0% of all new papers.

Action-Value Calibration via Continuation Sharing for Long-Horizon Agentic Reinforcement Learning
Oct 27, 2026Yingjian Chen, Lang Zhou, Zixin Zhao +3Agentic RL Step-Level Credit Assignment in RL
Sun Yat-sen University · Shenzhen Loop Area Institute
Balanced Active Subspaces for Optimal Control via Trajectory Optimization
Oct 26, 2026Emmanuel S. Ameh, Samuel E. OttoOptimal Control Reduced-Order Modeling
Department of Chemical and Biomolecular Engineering · Department of Mechanical and Aerospace Engineering
FOCUS: Active Critic Learning for Long-Horizon Language Model Agents
Oct 26, 2026Zhengping Jiang, William Jurayj, Liaoyaqi Wang +3RL for LLM Agents Long-Horizon RL
Johns Hopkins University
Model Selection for Off-Policy Evaluation via Pairwise Bellman Validation
Oct 26, 2026Lars van der Laan, Vasilis Syrgkanis, Nathan KallusOff-Policy Evaluation Value Function Estimation
Stanford University · Netflix and Cornell University
Evaluation of Active Feature Acquisition Policies with Tabular Foundation Models
Oct 5, 2026Yuta Kobayashi, Divyam Madaan, Shalmali JoshiOff-Policy Evaluation Value Function Estimation
Department of Biomedical Informatics, Columbia University
Variance-Optimal Control Variates for Learning with Black-box Feedback
Oct 3, 2026Zihao Zhao, Shuhan Zhang, Kai WangVariance Reduction Policy Gradient Methods
School of Computational Science and Engineering Georgia Institute of Technology
Privy to the Foil: Recasting Value Estimation with a Self-Privileged Critic for RLVR
Sep 29, 2026Kun Liang, Chenming Tang, Clive Bai +5Value Function Estimation RL for Language Model Reasoning
School of Computer Science, Peking University · National Key Laboratory for Multimedia Information Processing, Peking University · Foundation Model Department, Tencent
Unlocking the Critic: Reward-Free Policy Optimization for LLM Post-Training
Sep 29, 2026Hongyang Li, Xiao Li, Caesar Wu +3Reinforcement Learning Value Function Estimation
University of Luxembourg · Seafill Open-Source Community · Université Paris-Saclay
Deep Epistemic Value Functions for Optimistic Exploration
Sep 28, 2026Leander Diaz-Bone, Marco Bagatella, Jonas Hübotter +1Value Function Estimation RL Exploration
ETH Zürich, Switzerland · Max Planck Institute for Intelligent Systems, Germany
Deep Weighted Bellman Residual Minimization for $Q^*$ Estimation
Sep 28, 2026Lican Kang, Jerry Zhijian Yang, Cheng Yuan +1Off-Policy Evaluation Value Function Estimation
Institute for Math and AI, Wuhan University, Wuhan, 430072, China · Hubei Key Laboratory of Computational Science, Wuhan University, Wuhan, 430072, China · School of Artificial Intelligence, Wuhan University, Wuhan, 430072, China +1
Limiting-Kernel Q( $λ$ ): Bridging Short and Long Horizons
Sep 23, 2026Tolga Ok, Arman Sharifi Kolarijani, Peyman Mohajerin Esfahani +1Value Function Estimation Policy Evaluation
Delft University of Technology, The Netherlands · Alpha Brain Technologies, The Netherlands · University of Toronto, Canada
Proximal Residual Value Functions for Consistent Planning and Real-Time Execution
Sep 19, 2026Harrison Waldon, Carson Eisenach, Akhil Bagaria +4Value Function Estimation Reinforcement Learning
Amazon SCOT
Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening
Sep 16, 2026Yizhuo Li, Jianhao Yan, Yun Luo +9RL for Language Models Value Function Estimation
Shanghai Jiao Tong University · Shanghai AI Laboratory · Westlake University +4
Optimal Value Inference for Reinforcement Learning
Sep 9, 2026Nan Lu, Ethan Lee, Ruicheng Ao +3Reinforcement Learning Value Function Estimation
The Surprising Effectiveness of Approximate Value Iteration in Self-Play
Sep 8, 2026Raphael Boige, Amine Boumaza, Bruno ScherrerGame-Playing Agents Self-Play RL
Université de Lorraine, CNRS, Inria, LORIA, F-54000 Nancy, France
Recursive Value Learning for Long-Horizon Offline Goal-Conditioned RL
Sep 2, 2026Hyeonseong Jeon, Youngwoon LeeLong-Horizon RL Reinforcement Learning
1Yonsei University · 2Seoul National University
Graph4BiLO: Graph Neural Network Approximation for Bilevel Mixed-Integer Linear Optimization
Aug 31, 2026Jessica D. Elrefaei, Kaixun Hua, Seungbae Kim +2Graph Neural Networks Value Function Estimation
Game-Agnostic Value Functions through Automatic JSON Feature Extraction
Aug 30, 2026Dien Nguyen, Diego Perez-LiebanaGame-Playing Agents Value Function Estimation
Queen Mary University of London - Game AI Group
Sparse Additive Off-Policy Evaluation for Reinforcement Learning with Potentially Limited Number of Trajectories
Aug 23, 2026Tuoyi Zhao, Chengchun Shi, Zhengling Qi +1Off-Policy Evaluation Reinforcement Learning
Yale University · London School of Economics and Political Science · George Washington University +1
Pandora's AI Model Routing Box: Efficient Allocation with Costly Value Estimation
Aug 20, 2026Adam Fisch, Shubhendu Trivedi, Fantine Huot +5LLM Routing Cost-Aware Inference
Google DeepMind
Self-Supervised Auxiliary Task Discovery for Stable Reinforcement Learning in Stock Trading
Aug 16, 2026Arishi Orra, Himanshu Choudhary, Manoj ThakurReinforcement Learning Representation Learning for RL
School of Mathematical and Statistical Sciences, Indian Institute of Technology Mandi, Mandi, 175005, Himachal Pradesh, India
Offline Deep Q* Estimation with Diffusion Models
Aug 14, 2026Xiaohong Chen, Yuling Jiao, Lican Kang +2Offline RL Value Function Estimation
Department of Economics, Yale University, New Haven, 06511, USA. · School of Artificial Intelligence, and Hubei Key Laboratory of Computational Science, Wuhan University, Wuhan, 430072, China. · Institute for Math and AI, Hubei Key Laboratory of Computational Science, and School of Artificial Intelligence, Wuhan University, Wuhan, 430072, China. +2
Revisiting TD Target Aggregation under Uncertainty in Q-Learning
Aug 4, 2026Lipeng Zu, Xiaonan ZhangDeep Q-Learning Value Function Estimation
Computer Science Department, Florida State University, Tallahassee, Florida, USA
ValueFormer: Stage-Aware Value Labels for Online Failure Detection and VLA Post-Training
Aug 3, 2026Inkyu Sa, Konstantin Stulov, Rajat BhageriaReward Shaping Value Function Estimation
Chef Robotics
Diffusion Policy with Behavioral Advantage Correction for Offline Reinforcement Learning
Aug 3, 2026Botao Dong, Longyang Huang, Ning Pang +1Distribution Shift Value Function Estimation
Botao Dong is with the School of Chemistry, Chemical Engineering and Biotechnology, Nanyang Technological University, Singapore 637459
General Value Functions for Remaining Useful Life and Failure-Mode Prediction
Jul 24, 2026Hao Yan, Ali Sarabi, Qing Zou +1RUL Estimation Value Function Estimation
School of Computing and Augmented Intelligence, Arizona State University
Relative Value Learning
Jul 23, 2026Marc Höftmann, Jan Robine, Stefan HarmelingValue Function Estimation Temporal-Difference Learning
Department of Computer Science, Technical University of Dortmund, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence
Generalized Kalman filter based temporal difference reinforcement learning
Jul 22, 2026Vasos Arnaoutis, Eric Lutters, Bojana RosićRL Control Reinforcement Learning
Applied Mechanics and Data Analysis, Faculty of Engineering Technology, University of Twente, Enschede, the Netherlands · Department of Design, Production and Management, Faculty of Engineering Technology, University of Twente, Enschede, the Netherlands · Digital Engineering research group, Faculty of Mechanical and Industrial Engineering, Vienna University of Technology, Vienna, Austria
Robo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learning
Jul 10, 2026Wenke Xia, Pei Ren, Wenbo Yu +10Value Function Estimation Robotic Manipulation
1Gaoling School of Artificial Intelligence, Renmin University of China · 2Beijing Innovation Center of Humanoid Robotics · 3Beijing Forestry University +4
Look Before You Leap: Distilling Tree Search into Action Evaluation for Frozen VLA Models
Jul 4, 2026Xinyi Xie, Zican Hu, Zhanyu Liu +7Value Function Estimation Test-Time Scaling
Nanjing University · Australian National University · Institute of Automation, Chinese Academy of Sciences +1

Oct 27, 2026J/K move · Enter open · S save

Action-Value Calibration via Continuation Sharing for Long-Horizon Agentic Reinforcement Learning

Yingjian Chen, Lang Zhou, Zixin Zhao, Pingfan Li+2

Sun Yat-sen University · Shenzhen Loop Area Institute

Agentic RL Step-Level Credit Assignment in RL RL for LLM Agents Value Function Estimation

Open paper AskPDF arXiv

Reinforcement learning for long-horizon agents relies on feedback from sequences of interactions, making credit assignment to individual actions difficult. Existing methods such as GiGPO compare actions at shared states to compute relative advantages, but estimate each action's value from a single sampled continuation. The resulting local credit remains sensitive to decisions made later in the trajectory. To address this problem, we propose ConCal (Continuation Calibration), a recursive continuation-value estimator that incorporates continuation evidence already present in a rollout batch. ConCal matches visits from the same task by their incoming state-action-state transitions, combines shared continuation evidence with each visit's own transition estimate, and propagates calibrated values backward to refine earlier action-value estimates. This reduces dependence on individual sampled continuations and mitigates local credit distortions without additional rollouts. Held-out return diagnostics show lower error in within-group value differences and higher agreement with reference action orderings than estimates based on individual continuations. With Qwen2.5-1.5B-Instruct, ConCal improves success rates over GiGPO by 3.39 and 5.86 percentage points on ALFWorld and WebShop, respectively. Our code is available at https://github.com/Yingjian-Chen/ConCal.