Overview

Value functions estimate the expected future reward or outcome of a state or action, helping agents compare choices, evaluate policies, and assign credit to earlier decisions. Common approaches include Monte Carlo returns, temporal-difference learning, and fitted value or Q-function regression, often using neural networks and Bellman updates. Current work also studies off-policy evaluation and uncertainty, and extracts value signals from language-model and robot-policy representations; methods are tested in settings such as D4RL, Atari, MuJoCo, long-horizon control, and language-model reasoning. A central challenge is balancing low-variance learning with bias from bootstrapping, limited data coverage, and unreliable estimates under distribution shift.

Latest papers

CardsList

73 papers6 papers in the last four weeks, down 14% on the four weeks before. 0.0% of all new papers.

  1. Approximate Dynamic Programming with Basis Function Learning

    Nov 18, 2026Mohammadreza Fathi, Chi-Guhn LeeValue Function EstimationDynamic Programming

  2. Learning Regions of Attraction for Transverse Feedback Linearization under Controller Singularities

    Nov 17, 2026Abdullah Waqar Malik, Hamza Tariq, Adeel AkhtarFeedback ControlValue Function Estimation

  3. Non-Myopic Safety Filtering with Learned Intervention Cost-to-Go

    Nov 16, 2026Johannes Autenrieb, Mark Spiller, Daniel Silvestre +2Safety FilteringControl Barrier Functions

  4. Value Forcing: Aligning Return Distributions at All Noise Levels

    Nov 1, 2026Chongyi Zheng, Zhiyuan Zhou, Qiyang Li +2Value Function EstimationSample-Efficient RL

  5. Action-Value Calibration via Continuation Sharing for Long-Horizon Agentic Reinforcement Learning

    Oct 27, 2026Yingjian Chen, Lang Zhou, Zixin Zhao +3Agentic RLStep-Level Credit Assignment in RL

  6. Balanced Active Subspaces for Optimal Control via Trajectory Optimization

    Oct 26, 2026Emmanuel S. Ameh, Samuel E. OttoOptimal ControlReduced-Order Modeling

  7. FOCUS: Active Critic Learning for Long-Horizon Language Model Agents

    Oct 26, 2026Zhengping Jiang, William Jurayj, Liaoyaqi Wang +3RL for LLM AgentsLong-Horizon RL

  8. Model Selection for Off-Policy Evaluation via Bellman Tournaments

    Oct 26, 2026Lars van der Laan, Vasilis Syrgkanis, Nathan KallusOff-Policy EvaluationValue Function Estimation

  9. Evaluation of Active Feature Acquisition Policies with Tabular Foundation Models

    Oct 5, 2026Yuta Kobayashi, Divyam Madaan, Shalmali JoshiOff-Policy EvaluationValue Function Estimation

  10. Variance-Optimal Control Variates for Learning with Black-box Feedback

    Oct 3, 2026Zihao Zhao, Shuhan Zhang, Kai WangVariance ReductionPolicy Gradient Methods

  11. Privy to the Foil: Recasting Value Estimation with a Self-Privileged Critic for RLVR

    Sep 29, 2026Kun Liang, Chenming Tang, Clive Bai +5Value Function EstimationRL for Language Model Reasoning

  12. Unlocking the Critic: Reward-Free Policy Optimization for LLM Post-Training

    Sep 29, 2026Hongyang Li, Xiao Li, Caesar Wu +3Reinforcement LearningValue Function Estimation

  13. Deep Epistemic Value Functions for Optimistic Exploration

    Sep 28, 2026Leander Diaz-Bone, Marco Bagatella, Jonas Hübotter +1Value Function EstimationRL Exploration

  14. Deep Weighted Bellman Residual Minimization for Q∗Q^* Estimation

    Sep 28, 2026Lican Kang, Jerry Zhijian Yang, Cheng Yuan +1Off-Policy EvaluationValue Function Estimation

  15. Limiting-Kernel Q(λλ): Bridging Short and Long Horizons

    Sep 23, 2026Tolga Ok, Arman Sharifi Kolarijani, Peyman Mohajerin Esfahani +1Value Function EstimationPolicy Evaluation

  16. Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening

    Sep 16, 2026Yizhuo Li, Jianhao Yan, Yun Luo +9RL for Language ModelsValue Function Estimation

  17. The Surprising Effectiveness of Approximate Value Iteration in Self-Play

    Sep 8, 2026Raphael Boige, Amine Boumaza, Bruno ScherrerGame-Playing AgentsSelf-Play RL

  18. Recursive Value Learning for Long-Horizon Offline Goal-Conditioned RL

    Sep 2, 2026Hyeonseong Jeon, Youngwoon LeeLong-Horizon RLReinforcement Learning

  19. Game-Agnostic Value Functions through Automatic JSON Feature Extraction

    Aug 30, 2026Dien Nguyen, Diego Perez-LiebanaGame-Playing AgentsValue Function Estimation

  20. Sparse Additive Off-Policy Evaluation for Reinforcement Learning with Potentially Limited Number of Trajectories

    Aug 23, 2026Tuoyi Zhao, Chengchun Shi, Zhengling Qi +1Off-Policy EvaluationReinforcement Learning

  21. Pandora's AI Model Routing Box: Efficient Allocation with Costly Value Estimation

    Aug 20, 2026Adam Fisch, Shubhendu Trivedi, Fantine Huot +5LLM RoutingCost-Aware Inference

  22. Self-Supervised Auxiliary Task Discovery for Stable Reinforcement Learning in Stock Trading

    Aug 16, 2026Arishi Orra, Himanshu Choudhary, Manoj ThakurReinforcement LearningRepresentation Learning for RL

  23. Offline Deep Q* Estimation with Diffusion Models

    Aug 14, 2026Xiaohong Chen, Yuling Jiao, Lican Kang +2Offline RLValue Function Estimation

  24. Revisiting TD Target Aggregation under Uncertainty in Q-Learning

    Aug 4, 2026Lipeng Zu, Xiaonan ZhangDeep Q-LearningValue Function Estimation

  25. Diffusion Policy with Behavioral Advantage Correction for Offline Reinforcement Learning

    Aug 3, 2026Botao Dong, Longyang Huang, Ning Pang +1Distribution ShiftValue Function Estimation