GIRL: Generative Imagination Reinforcement Learning via Information-Theoretic Hallucination Control

Prakul Sunil Hiremath

cs.LGApr 8, 2026

GIRL: Generative Imagination Reinforcement Learning via Information-Theoretic Hallucination Control

Authors: Prakul Sunil Hiremath

Organizations: Department of Computer Science and Engineering, Visvesvaraya Technological University (VTU), Belagavi, India · Aliens on Earth (AoE) Autonomous Research Group, Belagavi, India

Abstract

Model-based reinforcement learning (MBRL) improves sample efficiency by optimizing policies inside imagined rollouts, but long-horizon planning degrades when model errors compound and imagined trajectories drift off the training manifold. We introduce GIRL (Generative Imagination Reinforcement Learning), a latent world-model framework that addresses this failure mode with two key components. First, a cross-modal grounding signal derived from a frozen foundation model (DINOv2) anchors the latent transition prior to a semantically consistent embedding space, penalizing inconsistent or implausible predictions. Second, an uncertainty-adaptive trust-region bottleneck interprets the KL regularizer as the Lagrange multiplier of a constrained optimization problem, restricting imagination drift within a learned region calibrated by Expected Information Gain and a Relative Performance Loss signal. We re-derive a value-gap bound using the Performance Difference Lemma and Integral Probability Metrics, yielding a bound that remains informative as the discount factor approaches one and connects the objective to real-environment regret. Experiments across three benchmark suites, including DeepMind Control, Adroit Hand Manipulation, and Meta-World with visual distractors, show that GIRL reduces latent rollout drift by 38 to 61 percent across tasks relative to DreamerV3, improves asymptotic return, and requires fewer environment interactions on long-horizon tasks. GIRL also outperforms TD-MPC2 on sparse-reward and high-contact settings under standard evaluation metrics. A distilled-prior variant reduces inference overhead and improves computational efficiency relative to the full model.

Figures & tables

Task	Why challenging	Drift risk	Horizon
Cheetah-Run	Fast locomotion; contact errors compound	High	300
Humanoid-Walk	$\|A\|=21$ ; long-horizon balance	Very high	500
Dog-Run	Discontinuous contact dynamics	Very high	500
Acrobot-Sparse	Sparse reward; delayed signal ( $\geq 500$ steps)	High	$>500$
Finger-Turn-Hard	Precise contact; OOD initialization	Med–high	300
Cheetah-Run-D	+ visual distractors	High	300

Table 1: Diagnostic tasks. “D” denotes visual-distractor variants. Drift risk qualitatively reflects expected KL growth per 100 steps.

Method	IQM $\uparrow$	PI $\uparrow$	DFM $(1000)$ $\downarrow$
SAC	$0.41\,[0.37,0.45]$	$0.19$	—
MBPO	$0.52\,[0.48,0.56]$	$0.27$	$1.23$
DreamerV3	$0.67\,[0.63,0.71]$	$0.26$	$4.81$
TD-MPC2 †	$0.71\,[0.67,0.75]$	$0.31$	$3.47$
GIRL-NoGround	$0.72\,[0.68,0.76]$	$0.36$	$3.12$
GIRL-FixedBeta	$0.70\,[0.66,0.74]$	$0.33$	$2.89$

Table 2: Aggregate DMC results at

3\times 10^{6}

steps (10 seeds, 8 tasks). IQM and PI are reported with stratified bootstrap 95% confidence intervals. DFM

(1000)

is averaged across tasks (lower is better).

\dagger

TD-MPC2 was not evaluated on distractor tasks in the original work.

Method	IQM $\uparrow$	PI $\uparrow$	DFM $(500)$ $\downarrow$
DreamerV3	$0.44\,[0.39,0.49]$	$0.31$	$3.92$
TD-MPC2	$0.58\,[0.53,0.63]$	$0.42$	$2.81$
GIRL-NoGround	$0.55\,[0.50,0.60]$	$0.39$	$2.79$
GIRL (ProprioGIRL)	$\mathbf{0.63\,[0.58,0.68]}$	—	$\mathbf{2.28}$

Table 3: Adroit Hand Manipulation results at

3\times 10^{6}

steps (10 seeds, 3 tasks). IQM is reported with 95% confidence intervals. DFM

(500)

is averaged across tasks (lower is better).

Method	IQM $\uparrow$	PI $\uparrow$
DreamerV3	$0.61\,[0.57,0.65]$	$0.24$
TD-MPC2	$0.72\,[0.68,0.76]$	$0.35$
GIRL-NoGround	$0.69\,[0.65,0.73]$	$0.38$
GIRL-FixedBeta	$0.67\,[0.63,0.71]$	$0.36$
GIRL (full)	$\mathbf{0.79\,[0.75,0.83]}$	—

Table 4: Meta-World MT10 multi-task success rate at

5\times 10^{6}

steps (10 seeds, 10 tasks). IQM is reported with 95% confidence intervals.

Variant	IQM (all 18) $\uparrow$	IQM (distractor) $\uparrow$
GIRL (full)	$\mathbf{0.78\,[0.75,0.81]}$	$\mathbf{0.76\,[0.71,0.81]}$
GIRL-NoIntrinsic	$0.74\,[0.71,0.77]$	$0.73\,[0.68,0.78]$
GIRL-VAE	$0.69\,[0.66,0.72]$	$0.63\,[0.58,0.68]$
GIRL-NoGround	$0.68\,[0.65,0.71]$	$0.65\,[0.60,0.70]$
GIRL-FixedBeta	$0.66\,[0.63,0.69]$	$0.67\,[0.62,0.72]$
TD-MPC2	$0.68\,[0.65,0.71]$	$0.61\,[0.56,0.66]$

Table 5: Ablation results aggregated across 18 tasks (left) and distractor DMC tasks (right). IQM is reported with 95% confidence intervals (10 seeds per task).

Figure 1: Drift-Fidelity Metric (DFM

(L)

) versus imagination horizon

L

on Humanoid-Walk . GIRL exhibits near-linear drift growth across the full horizon, while DreamerV3 shows super-linear accumulation beyond

L\approx 200

. TD-MPC2 achieves lower drift at short horizons but surpasses GIRL near

L\approx 300

as accumulated bias increases. Shaded regions denote 95% bootstrap confidence intervals over 10 seeds.

Component	Time (ms)	% of DreamerV3 total
DreamerV3 (full iteration)	$312\pm 8$	100%
GIRL: DINOv2 inference	$38\pm 3$	+12.2%
GIRL: Ensemble EIG/RPL	$47\pm 4$	+15.1%
GIRL: Additional world-model	$6\pm 1$	+1.9%
GIRL: Trust-region updates	$3\pm 1$	+1.0%
GIRL total	$\mathbf{406\pm 11}$	$\mathbf{+30.1\%}$

Table 6: Wall-clock profiling per training iteration (50 sequences, 50 steps each), A100-80GB. Mean

\pm

std over 1000 iterations. “GIRL-Distill” uses the distilled DINOv2 prior (Section 5.2 ).

Figure 2: Wall-clock overhead (GIRL / DreamerV3 ratio) as a function of action dimension. GIRL-full (solid) and GIRL-Distill (dashed). At high

|A|

(Adroit), GRU/ensemble computation dominates and DINOv2 overhead shrinks to

<3\%

(distilled) or

<7\%

(full).

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Latent dim $d$	32
Recurrent state dim	512
Grounding dim $d_{g}$	128
Foundation model	DINOv2 ViT-B/14 (frozen)
MSAE window $W$	16
MSAE mask rate	0.4

Appendix

Table 7: Full GIRL hyperparameters.

Explore similar work

May 15, 2026

Mind Dreamer: Untethering Imagination via Active Causal Intervention on Latent Manifolds

Model-Based Reinforcement Learning yields sample efficiency via latent imagination, yet remains constrained by Historical Tethering: imagination is typically initialized from observed states. This creates a learning asymmetry, where the world model's manifold discovery outpaces the policy's sparse-reward optimization. We propose Mind Dreamer (MD), a framework that instantiates Active Causal Intervention to transcend Markovian continuity. MD reformulates discovery as the minimization of a global Relay Expected Free Energy. Instead of initializing from historical data, it draws initial states from an adversarial generator

s_0 \sim p_{gen}(\cdot)

, creating non-continuous latent jumps to epistemic blind spots that are physically plausible yet cognitively challenging. We derive Relay Value Function and Relay Uncertainty Function to resolve the credit assignment paradox across these spatial ruptures. Treating synthesized anchors as interventional intermediary states, these potentials propagate pragmatic and epistemic value through Bellman-style backups. Notably, we prove that uncertainty propagation across discontinuities necessitates a quadratic discount

γ^2

, establishing a formal epistemic horizon. Theoretically, MD approximates a variance-minimizing importance sampler that expands the manifold's spectral gap, reducing the hitting time to critical bottleneck states. Empirically, MD achieves a 1.67

\times

average speedup over DreamerV3 on DeepMind Control Suite, reaching 8.8

\times

in sparse-reward tasks.

Shaojun Xu, Xiaoling Zhou, Yihan Lin +4

Center for Brain-Inspired Computing Research, Department of Precision Instrument, Tsinghua University, Beijing, China · College of Computer Science and Technology, Zhejiang University, Hangzhou, China · Pen-Tung Sah Institute of Micro-Nano Science and Technology, Xiamen University, Xiamen, China +1

Reinforcement Learning Credit Assignment in RL RL Exploration Sparse-Reward RL Causal RL Model-Based RL

Ask PDF

May 6, 2026

ELVIS: Ensemble-Calibrated Latent Imagination for Long-Horizon Visual MPC

A central challenge of visual control with model-based reinforcement learning (RL) is reliable long-horizon planning: long rollouts with learned latent dynamics exhibit branching futures and multi-modal action-value distributions. In addition, compounding model errors amplified by visual occlusions make deep imagination brittle. We present ELVIS, a latent model predictive controller (MPC) designed to make long-horizon planning practical. ELVIS plans in a Dreamer-style recurrent state space model (RSSM) and replaces standard unimodal model predictive path integral (MPPI) with a Gaussian-mixture MPPI that maintains multiple coherent hypotheses over long horizons, avoiding mode averaging under branching rollouts. In parallel, ELVIS stabilizes deep imagination with a shared uncertainty-aware lambda-return: an ensemble of latent critics defines an upper-confidence-bound (UCB) score that gates a time-varying lambda, adaptively trading off bootstrapping versus look-ahead to limit compounding error during planning. The same return is used both to train an actor-critic prior from imagined rollouts and to score candidate trajectories inside GMM-MPPI, aligning RL objectives with the planner's long-horizon optimization. On fourteen DeepMind Control Suite visual tasks, ELVIS establishes state-of-the-art performance compared with TD-MPC2 and DreamerV3. Finally, ELVIS transfers zero-shot to a real-world sand-spraying task with severe occlusions, improving surface-quality metrics and demonstrating robustness beyond simulation.

Yurui Du, Pinhao Song, Yutong Hu +1

KU Leuven, Dept. Electrical Engineering, Research unit Processing Speech and Images, B-3000 Leuven, Belgium · Flanders Make at KU Leuven, B-3000 Leuven, Belgium · KU Leuven, Dept. Mechanical Engineering, Research unit Robotics, Automation and Mechatronics, B-3000 Leuven, Belgium

Visual SSMs Model Predictive Control MPPI Control Visual RL Model-Based RL

Ask PDF

Sep 29, 2026

In-Distribution Imagination for Model-Based Offline Reinforcement Learning

Model-based offline reinforcement learning (MBORL) improves sample efficiency through model-generated trajectories. However, accumulative model error can drive imagined trajectories outside the offline data distribution, leading to unrealistic synthetic data and unstable policy optimization. Many existing methods primarily control rollouts using transition-level uncertainty. We propose in-distribution imagination (IDI), a rollout control framework that estimates trajectory support in a learned representation space and adaptively truncates rollouts that leave the offline trajectory manifold. Combined with trajectory-regularized RL, an extension of entropy-regularized RL, IDI consistently improves performance in limited-data settings. Experiments show that trajectory support predicts rollout failure substantially better than transition-level uncertainty, highlighting the importance of trajectory-level rollout control in MBORL.

Mintae Kim, Koushil Sreenath

Hybrid Robotics, BAIR, UC Berkeley

Offline RL

Ask PDF