cs.LGNov 6, 2026

Future-Informed Actor-Critic

Authors: Samy Mokeddem, Gaspard Lambrechts, Adrien Bolland, Damien Ernst

Organizations: University of Liège · McGill University, Mila · École Polytechnique

Abstract

Actor-critic methods optimize a parametrized policy using a critic to estimate their policy gradient. Recent work has shown that conditioning the critic on features of future states, actions, or rewards can improve sample efficiency and even final return. However, existing methods are restricted to exogenous features (independent of actions), overlooking endogenous features (dependent on actions) that may be crucial for credit assignment. We introduce the future-informed value function, which can be conditioned on both exogenous and endogenous features of future states, actions, and rewards. Then, we express the policy gradient using this value function, yielding the future-informed policy gradient. Building on that, we propose the future-informed actor-critic algorithm. Empirical results show that this algorithm leveraging endogenous features of the future outperforms standard actor-critic algorithms in environments with noisy and sparse rewards.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Functional Critics for Off-Policy Actor-Critic: From Stability to Exploration

    Sep 26, 2025Qinxun Bai, Yuxuan Han, Yuxiao Wen +2Actor-Critic MethodsOff-Policy RL

  2. Simple Actors and Deep Critics for Scalable Reinforcement Learning

    Aug 27, 2026Guhyeon Kang, Jaehwi Lee, Minhae KwonOffline RLActor-Critic Methods

  3. Bayesian policy gradient and actor-critic algorithms

    Apr 30, 2026Mohammad Ghavamzadeh, Yaakov Engel, Michal ValkoBayesian RLTemporal-Difference Learning