Value Function Estimation

Ask about this topic

Overview

Value functions estimate the expected future reward or outcome of a state or action, helping agents compare choices, evaluate policies, and assign credit to earlier decisions. Common approaches include Monte Carlo returns, temporal-difference learning, and fitted value or Q-function regression, often using neural networks and Bellman updates. Current work also studies off-policy evaluation and uncertainty, and extracts value signals from language-model and robot-policy representations; methods are tested in settings such as D4RL, Atari, MuJoCo, long-horizon control, and language-model reasoning. A central challenge is balancing low-variance learning with bias from bootstrapping, limited data coverage, and unreliable estimates under distribution shift.

Latest papers

CardsList

68 papers6 papers in the last four weeks, down 14% on the four weeks before. 0.0% of all new papers.

  1. Fitted Q-Evaluation without Bellman Completeness via Occupancy Weighting

    Dec 29, 2025Lars van der Laan, Nathan KallusOff-Policy EvaluationValue Function Estimation

    Department of Statistics, University of Washington · Netflix · Cornell University

  2. FlowCritic: Bridging Value Estimation with Flow Matching in Reinforcement Learning

    Oct 26, 2025Shan Zhong, Shutong Ding, He Diao +3Value Function EstimationDistributional RL

    School of Mechanical and Electrical Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China · School of Information Science and Technology, ShanghaiTech University · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore +1

  3. Regression-adjusted Monte Carlo Estimators for Shapley Values and Probabilistic Values

    Jun 13, 2025R. Teal Witter, Yurong Liu, Christopher MuscoShapley Value AttributionValue Function Estimation

    Claremont McKenna College · New York University

  4. KETCHUP: K-Step Return Estimation for Sequential Knowledge Distillation

    Apr 26, 2025Jiabin Fan, Guoqing Luo, Michael Bowling +1RL for Knowledge DistillationLanguage Model Distillation

    Dept. Computing Science, Alberta Machine Intelligence Institute (Amii), University of Alberta, Canada · Canada CIFAR AI Chair, Amii

  5. A Robust Model-Based Approach for Continuous-Time Policy Evaluation with Unknown Lévy Process Dynamics

    Apr 2, 2025Qihao Ye, Xiaochuan Tian, Yuhua ZhuReinforcement LearningValue Function Estimation

  6. Temporal difference learning adapts to incomplete coverage: an oracle inequality over subgraphs

    Nov 14, 2024Wenlong Mou, Jian QianTemporal-Difference LearningReinforcement Learning

    Department of Statistical Sciences, University of Toronto · Department of EECS, Massachusetts Institute of Technology

  7. Statistical Inference for Policy Evaluation with Temporal Difference Learning

    Oct 21, 2024Weichen Wu, Gen Li, Yuting Wei +1Parameter EstimationConfidence Region Estimation

    The Voleon Group, New York, NY, USA · Department of Statistics, The Chinese University of Hong Kong, Hong Kong SAR. · Department of Statistics and Data Science, The Wharton School, University of Pennsylvania, Philadelphia, PA, USA. +1

  8. Decision-Centered Abstractions via Orthogonal Estimation of Difference-of-Q Functions

    Jun 12, 2024Defu Cao, Angela ZhouValue Function EstimationState Abstraction

    University of Southern California