Q-based Variational Inverse Reinforcement Learning

Ondrej Bajgar; Peter Tisnikar; Alessandro Abate; Konstantinos Gatsis; Maike Osborne

cs.LGAug 17, 2026

Q-based Variational Inverse Reinforcement Learning

Authors: Ondrej Bajgar, Peter Tisnikar, Alessandro Abate, Konstantinos Gatsis, Maike Osborne

Organizations: University of Oxford · Artificial Intelligence Center, Czech Technical University in Prague · Villanova University

Abstract

The development of safe and beneficial AI requires that systems can learn and act in accordance with human preferences. However, explicitly specifying these preferences by hand is often infeasible. Inverse reinforcement learning (IRL) addresses this challenge by inferring preferences, represented as reward functions, from expert behaviour. We introduce Q-based Variational IRL (QVIRL), a novel Bayesian IRL method that recovers a posterior distribution over rewards from expert demonstrations via primarily learning a variational distribution over optimal Q-values. Unlike previous approaches, QVIRL combines scalability with uncertainty quantification, important for safety-critical applications as well as active learning. We demonstrate QVIRL's strong performance in apprenticeship learning across various tasks, including gridworlds, Lunar Lander, the Highway Environment, and two ATARI games both with static expert data and with active learning. It is the first method for Bayesian IRL that demonstrates training from raw pixel observations.

Figures & tables

Figure 1: Illustration of the QVIRL architecture. A state-action pair $(s,a)$ and a successor pair $(s^{\prime},a^{\prime})$ are fed into an encoder network that outputs the mean $\mu_{Q^{*}}$ and the standard deviation $\sigma_{Q^{*}}$ of the associated optimal Q-value $Q^{*}(s,a)$ , as well as a latent embedding $e$ . The embeddings of a set of state-action pairs can be passed to a kernel to produce their correlation matrix and, combined with individual variances, their covariance $\text{Cov}(Q^{*}(s,a),Q^{*}(s^{\prime},a^{\prime}))$ denoted by $\Sigma_{Q^{*}Q^{\prime*}}$ in the diagram. Together, they define the joint variational distribution of the optimal Q-values of a given set of state-action pairs. This distribution can then be used to deduce the distribution over the corresponding reward through the Bellman equation.

Figure 2: Illustration of posteriors recovered by ValueWalk (MCMC samples used as ground truth), AVRIL (for three values of constraint weight

\lambda)

, and QVIRL in

3

randomly selected states from our randomized

8\times 8

gridworlds.

Method	(1) $W_{1}$ (VW) $\downarrow$	(2) $\log p(r_{\mathrm{true}}\|\mathcal{D})$ $\uparrow$	(3) $r_{\mathrm{true}}\in 90\%$ CI	(4) $\pi_{\mathrm{exp}}\in 90\%$ CI	(5) TV (VW) $\downarrow$
ValueWalk	$0.000$	$-2.432\pm 0.013$	$0.888\pm 0.004$	$0.910\pm 0.003$	$0.000$
QVIRL	$0.486\pm 0.011$	$-2.437\pm 0.009$	$0.901\pm 0.004$	$0.913\pm 0.003$	$0.207\pm 0.003$
AVRIL	$1.704\pm 0.007$	$-7.568\pm 0.284$	$0.521\pm 0.006$	$0.727\pm 0.005$	$0.250\pm 0.004$

Table 1: Posterior quality on 100 random 8

\times

8 gridworlds (mean

\pm

stderr across

100

seeds). (i) log-density of the true reward under the posterior; (ii) fraction of states with

r(s)

within the posterior

90\%

credible interval; (iii) fraction of expert action probabilities

\pi(s|a)

within the posterior predictive

90\%

credible interval; and (iv) total variation distance between each method’s posterior-predictive policy and that of the ValueWalk oracle.

\uparrow

\downarrow

indicates higher/lower is better.

Policy $\pi$	$J_{r_{\mathrm{true}}}(\pi)$	$\mathbb{E}_{r\|\mathcal{D}}[J_{r}(\pi)]$	$\mathrm{CVaR}^{0.1}_{r\|\mathcal{D}}[J_{r}(\pi)]$
ValueWalk (mean)	$\phantom{-}75.4\pm 1.8$	$112.1\pm 2.8$	$\phantom{-}59.9\pm 2.8$
ValueWalk (CVaR)	$\phantom{-}75.8\pm 1.3$	$107.7\pm 3.0$	$\phantom{-}62.1\pm 2.3$
QVIRL (mean)	$\phantom{-}70.8\pm 2.1$	$106.4\pm 3.2$	$\phantom{-}49.0\pm 3.9$
QVIRL (CVaR)	$\phantom{-}66.9\pm 1.9$	$\phantom{0}89.6\pm 3.3$	$\phantom{-}49.7\pm 2.8$
AVRIL (mean)	$-22.2\pm 4.8$	$\phantom{00}5.4\pm 2.7$	$-86.3\pm 3.4$
AVRIL (CVaR)	$\phantom{-}36.7\pm 1.1$	$\phantom{0}27.1\pm 1.4$	$\phantom{-}14.0\pm 1.3$

Table 2: Apprenticeship learning on 100 random 8

\times

8 gridworlds (mean

\pm

stderr). Returns are normalised: 0 = random-policy return, 100 = optimal-policy return under the true reward.

J_{r_{\mathrm{true}}}(\pi)

: true-reward return of the apprentice policy.

\mathbb{E}_{r|\mathcal{D}}[J_{r}(\pi)]

: expected return under the ValueWalk oracle posterior.

\mathrm{CVaR}^{0.1}_{r|\mathcal{D}}[J_{r}(\pi)]

: conditional value-at-risk (10% level) of the return under the posterior, i.e. the expected return in the worst 10% cases under the ValueWalk reward posterior; the CVaR policy directly optimizes this objective under each method’s own posterior.

Figure 3: Mean episode return in apprenticeship learning with different numbers of expert demonstration trajectories in the Lunar Lander (left) and Highway (right) environments. Averaged over

10

runs, with standard errors of the mean. All methods are evaluated on discrete number of expert trajectories, but we space the points horizontally to improve plot readability.

Figure 4: Mean log-likelihood of held-out expert actions with different numbers of expert trajectories in the Lunar Lander (left) and Highway (right) environments. Averaged over

10

runs, with standard errors of the mean. All methods are evaluated on discrete numbers of expert trajectories, but we space the points horizontally to improve plot readability.

	Pong		Space Inv.
Method	Return	LL	Return	LL
BC	19.1 $\pm$ 0.8	-14.81 $\pm$ 0.49	597 $\pm$ 48	-18.67 $\pm$ 0.05
IQ-Learn	20.0 $\pm$ 0.2	-12.43 $\pm$ 2.27	740 $\pm$ 31	-8.99 $\pm$ 1.79
AVRIL	-5.2 $\pm$ 9.7	-8.79 $\pm$ 3.80	658 $\pm$ 34	-18.62 $\pm$ 0.07
QVIRL	19.7 $\pm$ 0.6	-2.09 $\pm$ 0.12	701 $\pm$ 42	-5.19 $\pm$ 1.04

Table 3: Apprenticeship learning on Atari: mean episode return and log likelihood (LL) of expert actions on held-out demonstrations (mean

\pm

standard error across seeds). QVIRL LL is computed under the probit likelihood; BC, IQ-Learn, and AVRIL LL are computed under the point likelihood.

Panel of Figure 5 — Figure 5: Active learning results on randomized gridworlds (left) and Lunar Lander (right).

Figure 6: Effect of the approximation in Eq. 3 on the reward posterior. The plot shows the MCMC posterior distribution over rewards for each state, as well as samples deduced from the MCMC posterior over rewards deduced using the approximation from the posterior over state-values. Dashed lines show Gaussians fitted to the two MCMC samples. The results coincide except in the bottom left and middle squares.

Figure 7: Comparison of the likelihood approximation equation 6 against a Monte Carlo evaluation using 10,000 Monte Carlo samples. Left: probability of

a_{1}

as a function of the mean of

Q_{1}

for 3 levels of standard deviation of

Q_{1}

. Correlation is set to 0 (the plot looks very similar for correlated values). Right: Probability of

a_{1}

as a function of the correlation between

Q_{1}

and

Q_{2}

for three levels of the mean. Standard deviation is fixed to 1. Note the different scales on y-axes of the two plots.

Panel of Figure 8 — Figure 8: Top left: Ground truth rewards for a 3x3 gridworld. The top left state is the initial state. Top right state is terminal. Top right: State-action counts in the demonstration data. Bottom left: pdf of the marginal posterior over state values recovered by QVIRL, a histogram of 2000 samples from the true posterior produced by ValueWalk, and a horizontal line indicating the point estimate recovered by AVRIL. Bottom right: pdfs of the variational posterior over reward recovered by QVIRL and AVRIL and the corresponding histogram.

Panel of Figure 10 — Figure 10: The mean true greedy action log probability difference between the demonstrator and learner for 1, 7, and 15 demonstrations, respectively.

Panel of Figure 11 — Figure 11: The ground truth rewards, demonstrations, and the posterior over the unknown right-middle reward recovered by QVIRL and Valuewalk in the illustrative gridworld.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,727 papers

Trust Region Inverse Reinforcement Learning: Explicit Dual Ascent using Local Policy Updates
May 10, 2026Anish Diwan, Davide Tateo, Christopher E. Mower +3Reinforcement Learning Policy Optimization
Technical University of Darmstadt · Robotics Institute Germany (RIG) · Lund University +3
Loop-Free Inverse Reinforcement Learning via Sequential Value Recovery with Q-Score Matching
Sep 30, 2026Yang Chen, Yitan Zhang, Michael Witbrock +1Reinforcement Learning
Shanghai Artificial Intelligence Laboratory · University of Auckland
MAVRL: Learning Reward Functions from Multiple Feedback Types with Amortized Variational Inference
Feb 16, 2026Raphaël Baur, Yannick Metz, Maria Gkoulta +3Bayesian RL
ETH AI Center, ETH Zurich, Zurich, Switzerland · Department of Computer Science, ETH Zurich, Zurich, Switzerland · Department of Informatics, University of Zurich, Zurich, Switzerland

May 10, 2026J/K move · Enter open · S save

Trust Region Inverse Reinforcement Learning: Explicit Dual Ascent using Local Policy Updates

Anish Diwan, Davide Tateo, Christopher E. Mower, Haitham Bou-Ammar+2

Technical University of Darmstadt · Robotics Institute Germany (RIG) · Lund University+3

Reinforcement Learning Policy Optimization Imitation Learning Trust Region Policy Optimization Policy Learning

Open paper AskPDF arXiv

Inverse reinforcement learning (IRL) is typically formulated as maximizing entropy subject to matching the distribution of expert trajectories. Classical (dual-ascent) IRL guarantees monotonic performance improvement but requires fully solving an RL problem each iteration to compute dual gradients. More recent adversarial methods avoid this cost at the expense of stability and monotonic dual improvement, by directly optimizing the primal problem and using a discriminator to provide rewards. In this work, we bridge the gap between these approaches by enabling monotonic improvement of the reward function and policy without having to fully solve an RL problem at every iteration. Our key theoretical insight is that a trust-region-optimal policy for a reward function update can be globally optimal for a smaller update in the same direction. This smaller update allows us to explicitly optimize the dual objective while only relying on a local search around the current policy. In doing so, our approach avoids the training instabilities of adversarial methods, offers monotonic performance improvement, and learns a reward function in the traditional sense of IRL--one that can be globally optimized to match expert demonstrations. Our proposed algorithm, Trust Region Inverse Reinforcement Learning (TRIRL), outperforms state-of-the-art imitation learning methods across multiple challenging tasks by a factor of 2.4x in terms of aggregate inter-quartile mean, while recovering reward functions that generalize to system dynamics shifts.

Abstract

Reinforcement Learning

Explore similar work

Trust Region Inverse Reinforcement Learning: Explicit Dual Ascent using Local Policy Updates

Loop-Free Inverse Reinforcement Learning via Sequential Value Recovery with Q-Score Matching

MAVRL: Learning Reward Functions from Multiple Feedback Types with Amortized Variational Inference