Generating Critic-Guided Synthetic Experience in Latent World Models

Jorge de Freitas; Christos Ziakas; Alessandra Russo

cs.LGOct 29, 2026

Generating Critic-Guided Synthetic Experience in Latent World Models

Authors: Jorge de Freitas, Christos Ziakas, Alessandra Russo

Organizations: Imperial College London

Abstract

Learning from offline datasets is limited in scalability, as collecting new experience is often costly or unsafe in the real world. World models offer an alternative way to generate new experience through a learned dynamics model, yet learning policies in latent space is limited due to compounding errors in imagined rollouts. To this end, we propose using the world model as a local transition generator, leveraging critic guidance to choose actions inside that model to generate synthetic experiences. We study action refinement methodologies, introducing Critic-Guided Synthetic Experience (CGSE) generation in world models. Specifically, CGSE augments an existing offline dataset by sampling candidate actions using an offline policy and refining each to maximize a first-order approximation of the $Q$ -function within a trust region. We then use a frozen latent world model to imagine the future state given each updated action, and use the critic to choose the best one, appending this synthetic transition to the replay buffer. Our method requires no additional real-environment interaction or test-time planner, as it generates synthetic experience offline using the world model. On three OGBench manipulation tasks, CGSE improves an offline policy by a relative improvement of $7.1\%$ on the average success rate.

Figures & tables

Figure 1: CGSE transition generation in world model. Starting from an offline latent $z_{0}$ , the fixed BC-flow actor samples $N$ candidate first actions and FMQ-style guidance applies a one-step trust-region critic-gradient refinement to each candidate. The refined candidates are rolled out under $f_{\psi}$ , scored with $Q_{\phi}$ , and the best branch supplies only its first action $a^{\mathrm{*}}$ and first model successor $z^{\prime}_{\psi}=f_{\psi}(z_{0},a^{\mathrm{*}})$ . The resulting synthetic transition $(z_{0},a^{\mathrm{*}},z^{\prime}_{\psi})$ is appended to the replay buffer.

Method	cube-single	scene	puzzle-3x3	Average
Offline FQL	$84.0\pm 4.2$	$15.3\pm 3.5$	$45.2\pm 10.2$	$48.2\pm 1.2$
Synthetic experience generation
No critic guidance	$80.4\pm 3.2$	$9.1\pm 2.1$	$37.2\pm 3.0$	$42.2\pm 1.4$
Best-of- $N$	$86.3\pm 3.1$	$13.2\pm 3.7$	$\mathbf{50.5\pm 2.7}$	$50.0\pm 1.3$
GradPlan	$\underline{88.9\pm 0.2}$	$\mathbf{18.7\pm 1.6}$	$46.4\pm 3.5$	$\underline{51.3\pm 1.0}$
CGSE (Ours)	$\mathbf{90.7\pm 0.9}$	$\underline{17.1\pm 2.4}$	$\underline{46.9\pm 3.6}$	$\mathbf{51.6\pm 1.8}$

Table 1: Success rate (mean

\pm

std over three seeds,

250

episodes). Best per task in bold, second best underlined.

Training	Success (%)
No world model	$89.3\pm 2.1$
CGSE, re-anchoring	$91.2\pm 3.2$
CGSE, chaining	$\mathbf{90.9\pm 1.3}$

Table 2: (a) Ablation on cube-single : all rows start from the same offline checkpoint and receive

500

k additional actor updates; success (%), mean

\pm

std over three seeds. (b) Generation cost per transition.

Horizon $\Delta$ (frames)	1	5	10	25	50
Mean drift	1.77	3.40	6.10	11.80	16.54

Table 3: Mean latent prediction drift

\lVert\hat{z}_{t+\Delta}-z_{t+\Delta}\rVert_{2}

vs. rollout horizon on visual-cube-single .

Model-Based RL

Model-based reinforcement learning uses environment models to plan actions and learn policies with fewer real-world interactions.

See all 249 papers

WorldSample: Closed-loop Real-robot RL with World Modelling
Jul 2, 2026Yuquan Xue, Le Xu, Zeyi Liu +5Robot Skill Learning World Models for Robotics
1PINE Lab, School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Department of Electronic Engineering, Tsinghua University, Beijing, China · School of Automation, Central South University, Changsha, China +1
World State Generator
Sep 21, 2026Sungheon Jeong, Sanggeon Yun, Ryozo Masukawa +3LLM Agent Planning LLM Planning
University of California, Irvine · Northeastern University
CG-World: A Large-Scale World-State Dataset and Protocol for World Models
Jul 29, 2026Yiming Cai, Fangjie Yu, Meiqing Yu +3World Model Learning World Models
CGyear World Model Lab · Tsinghua University · University of New South Wales

Jul 2, 2026J/K move · Enter open · S save

WorldSample: Closed-loop Real-robot RL with World Modelling

Yuquan Xue, Le Xu, Zeyi Liu, Zhenyu Wu+4

1PINE Lab, School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Department of Electronic Engineering, Tsinghua University, Beijing, China · School of Automation, Central South University, Changsha, China+1

Robot Skill Learning World Models for Robotics Synthetic Data Generation for Robotics Robotic RL Robotic Manipulation Sample-Efficient RL

Open paper AskPDF arXiv

Reinforcement learning (RL) can overcome the demonstration-coverage limitation of imitation learning (IL) by allowing robots to improve through trial-and-error interaction beyond the states observed in demonstrations. However, deploying RL on real robots remains constrained by high interaction costs, since each physical rollout is costly and reflects only one realized action-outcome path. To address this challenge, we propose WorldSample, a physically grounded data augmentation framework for real-robot RL that closes a real-synthetic loop between physical rollouts, world-model generation, and policy improvement. Grounded on real rollouts, WorldSample generates high-fidelity synthetic transitions through a post-trained world model, which greatly lowers the visual hallucination. Specifically, rather than simply using these transitions as real-world experience, WorldSample introduces Policy-Paced Learning (PPL) to regulate the training process through sample selection and scheduling, balancing useful augmentation against value overestimation and mitigating the hallucination-induced noise. Experiments on robot manipulation tasks involving contact-rich and precise tasks show that WorldSample improves policy success rate by 28% while reducing training steps by 59% compared with baselines. Furthermore, WorldSample improves world model visual fidelity by 19.4dB in PSNR and 0.47 in SSIM over demonstration-only post-training, validating the effectiveness of the real-synthetic loop for both policy and world model performance.

Abstract

Model-Based RL

Explore similar work

WorldSample: Closed-loop Real-robot RL with World Modelling

World State Generator

CG-World: A Large-Scale World-State Dataset and Protocol for World Models