MAVRL: Learning Reward Functions from Multiple Feedback Types with Amortized Variational Inference
Authors: Raphaël Baur, Yannick Metz, Maria Gkoulta, Mennatallah El-Assady, Giorgia Ramponi, Thomas Kleine Buening
Organizations: ETH AI Center, ETH Zurich, Zurich, Switzerland · Department of Computer Science, ETH Zurich, Zurich, Switzerland · Department of Informatics, University of Zurich, Zurich, Switzerland
Reward learning typically relies on a single feedback type or combines multiple feedback types using manually weighted loss terms. Currently, it remains unclear how to jointly learn reward functions from heterogeneous feedback types such as demonstrations, comparisons, ratings, and stops that provide qualitatively different signals. We address this challenge by formulating reward learning from multiple feedback types as Bayesian inference over a shared latent reward function, where each feedback type contributes information through an explicit likelihood. We introduce a scalable amortized variational inference approach that learns a shared reward encoder and feedback-specific likelihood decoders and is trained by optimizing a single evidence lower bound. Our approach avoids reducing feedback to a common intermediate representation and eliminates the need for manual loss balancing. Across discrete and continuous-control benchmarks, we show that jointly inferred reward posteriors outperform single-type baselines, exploit complementary information across feedback types, and yield policies that are more robust to environment perturbations. The inferred reward uncertainty further provides interpretable signals for analyzing model confidence and consistency across feedback types.
Figures & tables
Figure 1: Visualizations of inferred reward functions from 2 demonstrations, 256 pairwise comparisons, or 128 ratings on a 10×10 grid_trap environment. The final column shows the result obtained when combining all feedback modalities.
Table 1: Normalized mean returns and mean EPIC distances ( n=10 ) for different combinations of feedback modalities across environments. Returns (top, higher is better) are normalized such that 100 corresponds to the performance of an approximately optimal policy trained on the ground-truth reward and 0 corresponds to the performance of a uniformly random policy. EPIC distances (bottom, lower is better) measure the divergence between learned and ground-truth rewards. Cells shaded in dark gray indicate the best result for the corresponding environment, and cells shaded in light gray indicate the next best result outside this margin.
Post-Hoc Avg
MCMC
MAVRL
Environment
Perf.
Perf.
W.-Time
Perf.
W.-Time
grid_sparse
94.1
100.0
154.4 s
100.0
4.5 s
grid_cliff
54.9
100.0
156.0 s
100.0
4.9 s
grid_trap
47.3
93.0
155.0 s
95.3
4.6 s
LunarLander-v3
59.9
–
–
93.5
46.5 s
Table 2: Comparison of MAVRL with non-variational baselines. Performance (Perf.) is normalized return averaged over n=10 seeds (higher is better); wall-time (W.-Time) is per reward-model training run on identical hardware. Best per row in dark gray , second-best in light gray .
Figure 2: Normalized mean returns ( n=10 ) of policies trained on rewards inferred by each method under three dynamics perturbation scenarios. Reward models and baselines are trained in the unperturbed setting and remain fixed throughout the variations. Error bars denote standard error. (left) Increasing environmental stochasticity in grid_cliff and grid_trap. (middle) Increasing ratio between pendulum handle lengths in Acrobot-v1. (right) Increasing gravity and wind-power in LunarLander-v3.
Table 3: Misspecification robustness ( n=10 ; grid_trap in App. D ). Top: single-modality models; bottom: MAVRL with full feedback set. Each row corrupts the data-side parameter of the named channel while the model assumes the well-specified value. Strike-through denotes the corrupted modality; e.g., corresponds to misspecified preferences. Base. = well-specified normalized return; Misspec. = mean ± SEM; Rat. = Misspec./Base. Best per column in dark gray , second-best in light gray .
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Table 4: Normalized mean returns and mean EPIC distances ( n=10 ) for different combinations of feedback modalities across environments under equal feedback budget . Returns (top, higher is better) are normalized such that 100 corresponds to the performance of an approximately optimal policy trained on the ground-truth reward and 0 corresponds to the performance of a uniformly random policy. EPIC distances (bottom, lower is better) measure the divergence between learned and ground-truth rewards. Cells shaded in dark gray indicate the best result for the corresponding environment (within 1% for returns), and cells shaded in light gray indicate the next best result outside this margin.
Table 5: Misspecification robustness ( n=10 ). Top: single-modality models; bottom: MAVRL with full feedback set. Each row corrupts the data-side parameter of the named channel while the model assumes the well-specified value. Strike-through denotes the corrupted modality; e.g., corresponds to misspecified preferences. Base. = well-specified normalized return; Misspec. = mean ± SEM; Rat. = Misspec./Base. Best per column in dark gray , second-best in light gray .
Figure 3: Normalized mean returns ( n=10 ) of policies trained on rewards inferred by each method under three dynamics perturbation scenarios. Reward models and baselines are trained in the unperturbed setting and remain fixed throughout the variations. Returns are normalized such that 100.0 corresponds to the performance of an approximately optimal policy trained on the ground-truth reward in the unperturbed setting and, analogously, 0.0 corresponds to the performance of a uniformly random policy. Error bars denote standard error. (a) and (b) Increasing environmental stochasticity in grid_cliff and grid_trap. (c) Increasing ratio between pendulum handle lengths in Acrobot-v1. (d) Increasing gravity and wind-power in LunarLander-v3.
Figure 4: grid_sparse : Visualizations of inferred reward functions from 2 demonstrations, 256 pairwise comparisons, or 128 ratings on a 10×10 grid_sparse environment. The final column shows the result obtained when combining all feedback modalities. (a) Joint visual encoding of mean and variance. (b) Separate visual encoding of mean and variance for the same data.
Figure 5: grid_cliff : Visualizations of inferred reward functions from 2 demonstrations, 256 pairwise comparisons, or 128 ratings on a 10×10 grid_cliff environment. The final column shows the result obtained when combining all feedback modalities. (a) Joint visual encoding of mean and variance. (b) Separate visual encoding of mean and variance for the same data.
Figure 6: grid_trap : Visualizations of inferred reward functions from 2 demonstrations, 256 pairwise comparisons, or 128 ratings on a 10×10 grid_trap environment. The final column shows the result obtained when combining all feedback modalities. (a) Joint visual encoding of mean and variance. (b) Separate visual encoding of mean and variance for the same data.