GIRL: Generative Imagination Reinforcement Learning via Information-Theoretic Hallucination Control

Prakul Sunil Hiremath

cs.LGApr 8, 2026

GIRL: Generative Imagination Reinforcement Learning via Information-Theoretic Hallucination Control

Authors: Prakul Sunil Hiremath

Organizations: Department of Computer Science and Engineering, Visvesvaraya Technological University (VTU), Belagavi, India · Aliens on Earth (AoE) Autonomous Research Group, Belagavi, India

Abstract

Model-based reinforcement learning (MBRL) improves sample efficiency by optimizing policies inside imagined rollouts, but long-horizon planning degrades when model errors compound and imagined trajectories drift off the training manifold. We introduce GIRL (Generative Imagination Reinforcement Learning), a latent world-model framework that addresses this failure mode with two key components. First, a cross-modal grounding signal derived from a frozen foundation model (DINOv2) anchors the latent transition prior to a semantically consistent embedding space, penalizing inconsistent or implausible predictions. Second, an uncertainty-adaptive trust-region bottleneck interprets the KL regularizer as the Lagrange multiplier of a constrained optimization problem, restricting imagination drift within a learned region calibrated by Expected Information Gain and a Relative Performance Loss signal. We re-derive a value-gap bound using the Performance Difference Lemma and Integral Probability Metrics, yielding a bound that remains informative as the discount factor approaches one and connects the objective to real-environment regret. Experiments across three benchmark suites, including DeepMind Control, Adroit Hand Manipulation, and Meta-World with visual distractors, show that GIRL reduces latent rollout drift by 38 to 61 percent across tasks relative to DreamerV3, improves asymptotic return, and requires fewer environment interactions on long-horizon tasks. GIRL also outperforms TD-MPC2 on sparse-reward and high-contact settings under standard evaluation metrics. A distilled-prior variant reduces inference overhead and improves computational efficiency relative to the full model.

Figures & tables

Task	Why challenging	Drift risk	Horizon
Cheetah-Run	Fast locomotion; contact errors compound	High	300
Humanoid-Walk	$\|A\|=21$ ; long-horizon balance	Very high	500
Dog-Run	Discontinuous contact dynamics	Very high	500
Acrobot-Sparse	Sparse reward; delayed signal ( $\geq 500$ steps)	High	$>500$
Finger-Turn-Hard	Precise contact; OOD initialization	Med–high	300
Cheetah-Run-D	+ visual distractors	High	300

Table 1: Diagnostic tasks. “D” denotes visual-distractor variants. Drift risk qualitatively reflects expected KL growth per 100 steps.

Method	IQM $\uparrow$	PI $\uparrow$	DFM $(1000)$ $\downarrow$
SAC	$0.41\,[0.37,0.45]$	$0.19$	—
MBPO	$0.52\,[0.48,0.56]$	$0.27$	$1.23$
DreamerV3	$0.67\,[0.63,0.71]$	$0.26$	$4.81$
TD-MPC2 †	$0.71\,[0.67,0.75]$	$0.31$	$3.47$
GIRL-NoGround	$0.72\,[0.68,0.76]$	$0.36$	$3.12$
GIRL-FixedBeta	$0.70\,[0.66,0.74]$	$0.33$	$2.89$

Table 2: Aggregate DMC results at

3\times 10^{6}

steps (10 seeds, 8 tasks). IQM and PI are reported with stratified bootstrap 95% confidence intervals. DFM

(1000)

is averaged across tasks (lower is better).

\dagger

TD-MPC2 was not evaluated on distractor tasks in the original work.

Method	IQM $\uparrow$	PI $\uparrow$	DFM $(500)$ $\downarrow$
DreamerV3	$0.44\,[0.39,0.49]$	$0.31$	$3.92$
TD-MPC2	$0.58\,[0.53,0.63]$	$0.42$	$2.81$
GIRL-NoGround	$0.55\,[0.50,0.60]$	$0.39$	$2.79$
GIRL (ProprioGIRL)	$\mathbf{0.63\,[0.58,0.68]}$	—	$\mathbf{2.28}$

Table 3: Adroit Hand Manipulation results at

3\times 10^{6}

steps (10 seeds, 3 tasks). IQM is reported with 95% confidence intervals. DFM

(500)

is averaged across tasks (lower is better).

Method	IQM $\uparrow$	PI $\uparrow$
DreamerV3	$0.61\,[0.57,0.65]$	$0.24$
TD-MPC2	$0.72\,[0.68,0.76]$	$0.35$
GIRL-NoGround	$0.69\,[0.65,0.73]$	$0.38$
GIRL-FixedBeta	$0.67\,[0.63,0.71]$	$0.36$
GIRL (full)	$\mathbf{0.79\,[0.75,0.83]}$	—

Table 4: Meta-World MT10 multi-task success rate at

5\times 10^{6}

steps (10 seeds, 10 tasks). IQM is reported with 95% confidence intervals.

Variant	IQM (all 18) $\uparrow$	IQM (distractor) $\uparrow$
GIRL (full)	$\mathbf{0.78\,[0.75,0.81]}$	$\mathbf{0.76\,[0.71,0.81]}$
GIRL-NoIntrinsic	$0.74\,[0.71,0.77]$	$0.73\,[0.68,0.78]$
GIRL-VAE	$0.69\,[0.66,0.72]$	$0.63\,[0.58,0.68]$
GIRL-NoGround	$0.68\,[0.65,0.71]$	$0.65\,[0.60,0.70]$
GIRL-FixedBeta	$0.66\,[0.63,0.69]$	$0.67\,[0.62,0.72]$
TD-MPC2	$0.68\,[0.65,0.71]$	$0.61\,[0.56,0.66]$

Table 5: Ablation results aggregated across 18 tasks (left) and distractor DMC tasks (right). IQM is reported with 95% confidence intervals (10 seeds per task).

Figure 1: Drift-Fidelity Metric (DFM

(L)

) versus imagination horizon

L

on Humanoid-Walk . GIRL exhibits near-linear drift growth across the full horizon, while DreamerV3 shows super-linear accumulation beyond

L\approx 200

. TD-MPC2 achieves lower drift at short horizons but surpasses GIRL near

L\approx 300

as accumulated bias increases. Shaded regions denote 95% bootstrap confidence intervals over 10 seeds.

Component	Time (ms)	% of DreamerV3 total
DreamerV3 (full iteration)	$312\pm 8$	100%
GIRL: DINOv2 inference	$38\pm 3$	+12.2%
GIRL: Ensemble EIG/RPL	$47\pm 4$	+15.1%
GIRL: Additional world-model	$6\pm 1$	+1.9%
GIRL: Trust-region updates	$3\pm 1$	+1.0%
GIRL total	$\mathbf{406\pm 11}$	$\mathbf{+30.1\%}$

Table 6: Wall-clock profiling per training iteration (50 sequences, 50 steps each), A100-80GB. Mean

\pm

std over 1000 iterations. “GIRL-Distill” uses the distilled DINOv2 prior (Section 5.2 ).

Figure 2: Wall-clock overhead (GIRL / DreamerV3 ratio) as a function of action dimension. GIRL-full (solid) and GIRL-Distill (dashed). At high

|A|

(Adroit), GRU/ensemble computation dominates and DINOv2 overhead shrinks to

<3\%

(distilled) or

<7\%

(full).

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Latent dim $d$	32
Recurrent state dim	512
Grounding dim $d_{g}$	128
Foundation model	DINOv2 ViT-B/14 (frozen)
MSAE window $W$	16
MSAE mask rate	0.4

Appendix

Table 7: Full GIRL hyperparameters.

Explore similar work

Mind Dreamer: Untethering Imagination via Active Causal Intervention on Latent Manifolds
May 15, 2026Shaojun Xu, Xiaoling Zhou, Yihan Lin +4Reinforcement Learning Credit Assignment in RL
Center for Brain-Inspired Computing Research, Department of Precision Instrument, Tsinghua University, Beijing, China · College of Computer Science and Technology, Zhejiang University, Hangzhou, China · Pen-Tung Sah Institute of Micro-Nano Science and Technology, Xiamen University, Xiamen, China +1
ELVIS: Ensemble-Calibrated Latent Imagination for Long-Horizon Visual MPC
May 6, 2026Yurui Du, Pinhao Song, Yutong Hu +1Visual SSMs Model Predictive Control
KU Leuven, Dept. Electrical Engineering, Research unit Processing Speech and Images, B-3000 Leuven, Belgium · Flanders Make at KU Leuven, B-3000 Leuven, Belgium · KU Leuven, Dept. Mechanical Engineering, Research unit Robotics, Automation and Mechatronics, B-3000 Leuven, Belgium
In-Distribution Imagination for Model-Based Offline Reinforcement Learning
Sep 29, 2026Mintae Kim, Koushil SreenathOffline RL
Hybrid Robotics, BAIR, UC Berkeley

May 15, 2026J/K move · Enter open · S save

Mind Dreamer: Untethering Imagination via Active Causal Intervention on Latent Manifolds

Shaojun Xu, Xiaoling Zhou, Yihan Lin, Yapeng Meng+3

Center for Brain-Inspired Computing Research, Department of Precision Instrument, Tsinghua University, Beijing, China · College of Computer Science and Technology, Zhejiang University, Hangzhou, China · Pen-Tung Sah Institute of Micro-Nano Science and Technology, Xiamen University, Xiamen, China+1

Reinforcement Learning Credit Assignment in RL RL Exploration Sparse-Reward RL Causal RL Model-Based RL

Open paper AskPDF arXiv

Model-Based Reinforcement Learning yields sample efficiency via latent imagination, yet remains constrained by Historical Tethering: imagination is typically initialized from observed states. This creates a learning asymmetry, where the world model's manifold discovery outpaces the policy's sparse-reward optimization. We propose Mind Dreamer (MD), a framework that instantiates Active Causal Intervention to transcend Markovian continuity. MD reformulates discovery as the minimization of a global Relay Expected Free Energy. Instead of initializing from historical data, it draws initial states from an adversarial generator

s_0 \sim p_{gen}(\cdot)

, creating non-continuous latent jumps to epistemic blind spots that are physically plausible yet cognitively challenging. We derive Relay Value Function and Relay Uncertainty Function to resolve the credit assignment paradox across these spatial ruptures. Treating synthesized anchors as interventional intermediary states, these potentials propagate pragmatic and epistemic value through Bellman-style backups. Notably, we prove that uncertainty propagation across discontinuities necessitates a quadratic discount

γ^2

, establishing a formal epistemic horizon. Theoretically, MD approximates a variance-minimizing importance sampler that expands the manifold's spectral gap, reducing the hitting time to critical bottleneck states. Empirically, MD achieves a 1.67

\times

average speedup over DreamerV3 on DeepMind Control Suite, reaching 8.8

\times

in sparse-reward tasks.

Abstract

Explore similar work

Mind Dreamer: Untethering Imagination via Active Causal Intervention on Latent Manifolds

ELVIS: Ensemble-Calibrated Latent Imagination for Long-Horizon Visual MPC

In-Distribution Imagination for Model-Based Offline Reinforcement Learning