Malliavin Calculus for Counterfactual Gradient Estimation in Adaptive Inverse Reinforcement Learning

Vikram Krishnamurthy; Luke Snow

cs.LGApr 1, 2026

Malliavin Calculus for Counterfactual Gradient Estimation in Adaptive Inverse Reinforcement Learning

Authors: Vikram Krishnamurthy, Luke Snow

Organizations: School of Electrical and Computer Engineering, Cornell University, Ithaca, NY 14853, USA

Abstract

Inverse reinforcement learning (IRL) recovers the loss function of a forward learner from its observed responses. Adaptive IRL aims to reconstruct the loss function of a forward learner by passively observing its gradients as it performs reinforcement learning (RL). This paper proposes a novel passive Langevin-based algorithm that achieves adaptive IRL. The key difficulty in adaptive IRL is that the required gradients in the passive algorithm are counterfactual, that is, they are conditioned on events of probability zero under the forward learner's trajectory. Therefore, naive Monte Carlo estimators are prohibitively inefficient, and kernel smoothing, though common, suffers from slow convergence. We overcome this by employing Malliavin calculus to efficiently estimate the required counterfactual gradients. We reformulate the counterfactual conditioning as a ratio of unconditioned expectations involving Malliavin quantities, thus recovering standard estimation rates. We derive the necessary Malliavin derivatives and their adjoint Skorohod integral formulations for a general Langevin structure, and provide a concrete algorithmic approach which exploits these for counterfactual gradient estimation.

Figures & tables

Symbol	Meaning
$L$	unknown loss of the forward learner
$X_{t}$ , $X_{k}$	forward Langevin state (continuous / discrete)
$\alpha$	query point; state of the inverse Langevin chain
$\hat{\nabla}L(\alpha)$	counterfactual estimate of $\nabla L$ at $\alpha$
$\beta$	inverse temperature; $\pi_{\infty}\propto e^{-\beta L}$
$\varepsilon$ , $\eta$	forward / inverse step sizes

TABLE I: Notation

Fig. 1: Estimated gradient $\widehat{L^{\prime}}(\alpha)$ versus the true gradient $L^{\prime}(\alpha)$ over a grid of target locations $\alpha$ , using Monte Carlo estimation with sample size $N=5000$ .

Fig. 2: Sequential gradient estimates $\widehat{L^{\prime}}(\alpha)$ versus the true gradient $L^{\prime}(\alpha)$ across iterations of the outer passive Langevin chain ( 29 ). Occasional outliers appear, yet the overall procedure recovers the target Gibbs distribution and hidden loss function accurately (see Figures 3 and 4 )

Fig. 3: Empirical histogram of the stationary distribution of the Langevin chain $\{\alpha_{k}\}$ ( $2000$ MCMC steps after a $300$ -step burn-in), compared with the target Gibbs density $\pi_{\beta}(\alpha)\propto e^{-\beta L(\alpha)}$ .

Fig. 4: Adaptive IRL for reconstructing the loss function $L(\cdot)$ from MCMC samples of $\exp(-\beta L(\cdot)$ . Specifically, $L(\cdot)=-\frac{1}{\beta}\log(\hat{\pi}(\cdot))$ , where $\hat{\pi}(\cdot)$ is the empirical MCMC distribution in Fig. 3 .

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

A Lecture Note on Offline RL and IRL, Part II: Foundations of Inverse Reinforcement Learning and Dynamic Discrete Choice Models
May 29, 2026Enoch Hyunwook KangReinforcement Learning Sequential Decision Making
University of Washington, Foster School of Business
Trust Region Inverse Reinforcement Learning: Explicit Dual Ascent using Local Policy Updates
May 10, 2026Anish Diwan, Davide Tateo, Christopher E. Mower +3Reinforcement Learning Policy Optimization
Technical University of Darmstadt · Robotics Institute Germany (RIG) · Lund University +3
Fast Rates for Inverse Reinforcement Learning
May 14, 2026Andreas Schlaginhaufen, Maryam KamgarpourReinforcement Learning Statistical Learning Theory
Sycamore, EPFL

May 29, 2026J/K move · Enter open · S save

A Lecture Note on Offline RL and IRL, Part II: Foundations of Inverse Reinforcement Learning and Dynamic Discrete Choice Models

Enoch Hyunwook Kang

University of Washington, Foster School of Business

Reinforcement Learning Sequential Decision Making Offline RL

Open paper AskPDF arXiv

In the forward reinforcement-learning problem, the reward is fixed and known; the learner is asked to find a good policy or value function. Here we turn the question around. Given offline data generated by an expert, can we recover the reward the expert was optimizing? This is the inverse reinforcement learning problem, and remarkably, two communities, structural econometricians studying dynamic discrete choice (DDC) and machine learners studying entropy-regularized IRL, have been working on exactly the same probabilistic model under different names. We begin by proving their equivalence. We then develop the classical identification result of Magnac and Thesmar and the classical computational paradigms that grew out of it: Rust's nested fixed-point algorithm, the conditional-choice-probability approach of Hotz and Miller, and the two temporal-difference approaches of Adusumilli and Eckardt: linear semi-gradient TD and approximate value iteration. Each route has its limits: dimensionality, transition-kernel estimation, the deadly triad, or projected fixed-point bias. We then walk through the modern ML/IRL strand: adversarial IRL, occupancy matching, IQ-Learn, and offline ML-IRL, deriving each method's actual objective and stating precisely what it does and does not identify. We close with the empirical-risk-minimization framework of Kang et al., which yields a gradient-based estimator for offline IRL/DDC.

Abstract

Reinforcement Learning

Explore similar work

A Lecture Note on Offline RL and IRL, Part II: Foundations of Inverse Reinforcement Learning and Dynamic Discrete Choice Models

Trust Region Inverse Reinforcement Learning: Explicit Dual Ascent using Local Policy Updates

Fast Rates for Inverse Reinforcement Learning