Rethinking Video Autoencoders for Interactive World Models

Adrien Ramanana Rahary; Yutong Bai; Nicolas Dufour; Patrick Pérez; David Picard

cs.CVOct 21, 2026

Rethinking Video Autoencoders for Interactive World Models

Authors: Adrien Ramanana Rahary, Yutong Bai, Nicolas Dufour, Patrick Pérez, David Picard

Organizations: Kyutai · LIGM, ENPC, IP Paris, CNRS, UGE · UC Berkeley

Abstract

Video world models predict future frames from past frames and actions. They serve as simulators in which agents train, plan or are evaluated, and that humans can control interactively. They generate in the latent space of a video autoencoder, whose compression sets their cost per frame and how long each action waits to take effect. Yet world models often reuse the autoencoders of text-to-video generators, built for offline generation with no agent waiting and, without their training data and code, not retrainable at another compression. We therefore ask how, where and how much a world model should compress. We build CADRE, an open family of causal diffusion representation autoencoders spanning multiple spatio-temporal compressions. Its members differ only in a small bottleneck on frozen features and share a diffusion decoder self-distilled to one or a few steps. We train the same world model on each autoencoder in two domains, manipulation and navigation, with two environments each. CADRE forms most of the latency-quality frontier, especially with a moving camera. Our study also leads to four findings. At strong compression, a diffusion decoder is sharper than perceptual regression. In robot manipulation, temporal compression is more effective in the autoencoder than in the world model. Manipulation and navigation call for opposite compressions: temporal compression costs little when the camera is still, and spatial compression when it moves. Surprisingly, with a still camera, coarser spatial tokens hurt motion far more than image quality. Code, models and a live demo are at https://kyutai.org/blog/2026-10-21-cadre

Figures & tables

Figure 1: The right video autoencoder lets interactive world models react fast with high image and motion quality. On CADRE, world models run at 20 fps on one H100 and react in under 230 ms, below noticeable action lag ( Shimada et al., 2009 , top left;) , and form most of the latency-quality frontier in motion fidelity (FVD) and action following (ARR; bottom left). Right: rollouts given actions, where action lag makes acting hard (juggling, folding) or dangerous (driving).

Panel of Figure 2 — Figure 2: Every frame registers an action, whose effect appears $\tau$ frames later. A latent frame is predicted only once all its $\tau$ actions are in, so a larger $\tau$ saves world model calls but delays actions.

	Environment	Dataset	Camera	Action	DoF
Manipulation	Robot	ABC ( Allshire et al., 2026 )	static	joint deltas	14
Manipulation	Human	EgoDex ( Hoque et al., 2026 )	nearly static	hand-pose deltas	102
Navigation	Driving	Physical AI AV ( NVIDIA, 2025 )	moving	speed, curvature	2
Navigation	Indoor	RealEstate10K ( Zhou et al., 2018 )	moving	camera twist	6

Table 1: Environments. One world model is trained per environment and per autoencoder.

Figure 4: (a) Diffusion against perceptual regression decoding at

2{\times}32{\times}32

: MIRA has the worst FVD but the best PSNR, a sign of blur (all metrics: Figure 13 ). (b) Rollout quality against action latency in robot manipulation: CADRE forms most of the frontier (all environments: Figure 14 ).

Figure 5: Real-time rollouts following recorded actions. From 2 s of real video (left) and its actions, CADRE world models generate all other frames, with contact in robot manipulation, cloth dynamics in human manipulation, plausible futures in driving and 3D-consistent indoor scenes.

Figure 6: Temporal compression in the autoencoder or in the world model , at the same token grid in robot manipulation. The autoencoder gives sharper frames, closer to the recorded future, and more plausible motion at all but

2{\times}32{\times}32

. Action following barely changes, except at 8 frames.

Figure 7: Temporal compression and spatial tokens. Green raises

\tau

at a fixed token size; orange at a fixed token budget, so its grid coarsens (spatial factor

s

). A static view fits eight frames in one latent at no cost; a moving view loses quality with more frames. Full results in Figure 15 .

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Autoencoder training mixture , by share of sampling weight.

Category	Source	Share (%)	Hours
Web video	Panda-70M ( Chen et al., 2024b )	14.3	10,330
	LiveCC ( Chen et al., 2025a )	7.1	15,388
	LLaVA-Video-178K ( Zhang et al., 2025b )	7.1	2,744
	WebVid ( Bain et al., 2021 )	7.1	1,506
Robot manipulation	RoboCOIN, AgiBot World, Open X-Embodiment, RoboMIND, DreamDojo GR-1 and RoboCasa GR-1 ( Wu et al., 2025b ; AgiBot-World-Contributors et al., 2025 ; O’Neill et al., 2024 ; Wu et al., 2025a ; Gao et al., 2026a ; Nasiriany et al., 2024 )	8.9	5,387
	ABC, three cameras ( Allshire et al., 2026 )	5.4	10,790

Appendix

Table 2: Sources of the autoencoder training mixture , with their share of sampling weight and their size.

	CADRE autoencoder	World model
Architecture	frozen DINOv3 ViT-B/16, conv bottleneck,	causal DiT, 16 blocks, width 2048,
	ViT decoder 24 blocks, width 1024, 16 heads	16 heads (4 key/value), 1.2B parameters
Latent	64 channels	autoencoder latent, no further patching
Objective	flow matching, MAE, LPIPS, DINO, 3D Haar, PSD	flow matching, diffusion forcing, PSD-U
Optimizer	AdamW, $\beta=(0.9,0.95)$ , no weight decay	AdamW, $\beta=(0.9,0.99)$ , weight decay 0.1
Learning rate	$2{\times}10^{-4}$ , 2k warm-up, then constant	$10^{-4}$ , 1k warm-up, then constant

Appendix

Table 3: Training hyperparameters of the autoencoders and world models.

Autoencoder	$\tau$	$t_{\text{gen}}$ (ms)	$t_{\text{dec}}$ (ms)	Latency (ms)	Max. fps
CADRE- $1{\times}48{\times}48$	1	56	22 / 44	128 / 150	12.8 / 10.0
CADRE- $2{\times}32{\times}32$	2	61	25 / 50	185 / 211	23.4 / 18.1
CADRE- $4{\times}24{\times}24$	4	65	26 / 51	291 / 317	44.0 / 34.3
CADRE- $8{\times}16{\times}16$	8	83	28 / 56	510 / 538	72.5 / 57.8
CADRE- $1{\times}16{\times}16$	1	106	22 / 45	178 / 200	7.8 / 6.7
CADRE- $2{\times}16{\times}16$	2	91	25 / 49	216 / 240	17.3 / 14.3

Appendix

Table 4: Timing on one H100 , each decoder at its fastest streaming decode. CADRE cells: one / two decoding steps. Max. fps is the sustainable rate; the display runs at 20; bold : fluid. † Non-causal, windowed.

Figure 10: Can the world model run fluidly at 20 frames per second? Maximum frame rate against denoising steps (Table 4 ); above the line, it can.

Panel of Figure 11 — Figure 11: The streaming inference loop at $\tau=4$ . Each call reads the previous latent frame and the $\tau$ actions of the chunk that just closed. Action $a_{4}$ , entered at frame 1, first shows at frame 5 (red).

	Manipulation (robot)			Manipulation (human)			Navigation (driving)			Navigation (indoor)
Autoencoder	ARR	control	gap	ARR	control	gap	ARR	control	gap	ARR	control	gap
$1{\times}48{\times}48$	0.48	$-0.99$	1.46	0.64	$-0.54$	1.07	0.90	0.48	0.35	0.87	0.10	0.65
$2{\times}32{\times}32$	0.45	$-0.97$	1.40	0.75	$-0.58$	1.19	0.89	0.47	0.37	0.91	0.08	0.70
$4{\times}24{\times}24$	0.49	$-0.84$	1.31	0.65	$-0.49$	1.03	0.67	0.40	0.24	0.45	0.01	0.38
$8{\times}16{\times}16$	0.72	$-0.73$	1.40	0.63	$-0.50$	1.03	0.86	0.51	0.29	0.79	$-0.01$	0.69
$1{\times}16{\times}16$	0.74	$-0.89$	1.58	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A

Appendix

Table 5: Action recoverability and its control at frame 96. Gap is the raw

R^{2}

against the commanded actions minus the control, against commands shifted by 48 frames. Compare gaps within an environment. CADRE rows use the two-step decode; the CADRE-

1{\times}16{\times}16

world model is trained on robot manipulation only.

	Autoencoder ( $\tau{\times}s{\times}s$ )	FID $\downarrow$	FVD $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$
CADRE	$1{\times}48{\times}48$	9.71	43.3	21.21	0.127
	$2{\times}32{\times}32$	6.87	21.7	22.86	0.093
	$4{\times}24{\times}24$	7.06	17.7	23.32	0.087
	$8{\times}16{\times}16$	5.93	14.7	24.18	0.077
	$1{\times}16{\times}16$	4.27	10.9	25.05	0.057
	$2{\times}16{\times}16$	4.77	11.5	25.12	0.059

Appendix

Table 6: Reconstruction of 3072 held-out OpenVid-HD ( Nan et al., 2025 ) clips (16 frames at

288{\times}512

), with the one-step decode for our autoencoders. No world model is involved.

Figure 12: Compression plane. Our seven autoencoders (green) and the five others, with the tokens per 16-frame clip at

288{\times}512

Figure 13: Diffusion against perceptual regression decoding , all metrics (Figure 4 a in full), at

2{\times}32{\times}32

. The CADRE decoder lowers FID and FVD in every environment at similar action following; MIRA’s higher PSNR comes with blur.

Figure 14: Rollout quality against action latency in every environment. CADRE decodes in one step (light) or two (dark).

Manipulation (robot, ABC)
	Autoencoder ( $\tau{\times}s{\times}s$ )	Tokens	FID $\downarrow$	FVD $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$	ARR $\uparrow$
CADRE	$1{\times}48{\times}48$	1056	12.15 (14.69)	180.3 (186.7)	19.41 (19.54)	0.179 (0.184)	0.48 (0.48)
	$2{\times}32{\times}32$	1152	8.92 (11.63)	95.2 (105.0)	20.78 (20.79)	0.144 (0.149)	0.45 (0.45)
	$4{\times}24{\times}24$	1056	11.46 (13.95)	83.9 (93.8)	20.66 (20.67)	0.148 (0.152)	0.49 (0.48)
	$8{\times}16{\times}16$	1152	9.34 (11.41)	40.1 (50.6)	21.33 (21.42)	0.127 (0.131)	0.72 (0.72)
	$1{\times}16{\times}16$	9216	9.11 (10.55)	46.3 (56.2)	19.76 (19.90)	0.148 (0.150)	0.74 (0.74)

Appendix

Table 7: Full results at 96 frames (4.8 s). 2048 evaluation samples per environment, with 40 context frames and 96 generated. Metrics use the 16 frames ending at frame 96, except indoor PSNR at frame 48, since at 4.8 s it mostly measures viewpoint drift. 95% confidence intervals over test episodes are about

\pm

0.2 dB PSNR (

\pm

0.4 indoors),

\pm

0.01 LPIPS and

\pm

0.05 ARR. CADRE numbers decode in two steps, with one step in parentheses. Bold is best and italic second best per environment. The

1{\times}16{\times}16

world model is trained on robot manipulation only, and human-manipulation ARR uses the 12 wrist dimensions, since the probe barely reads the finger joints.

Figure 15: Temporal compression and spatial tokens, with CADRE-

4{\times}24{\times}24

in every environment. Dotted segments touch the overlapping-kernel cell.

Figure 16: 64 s rollouts in robot manipulation : for each autoencoder, the real context frame, then generated frames. Scenes differ between rows.

Autoencoder	FID/s	FVD/s	PSNR loss (dB/s)	LPIPS/s ( $\times 10^{-3}$ )
$1{\times}48{\times}48$	0.15	0.8	0.046	1.4
$2{\times}32{\times}32$	0.19	1.2	0.060	1.6
$4{\times}24{\times}24$	0.29	1.1	0.058	1.8
$8{\times}16{\times}16$	0.25	0.9	0.059	1.6
$1{\times}16{\times}16$	0.35	1.8	0.062	2.1
$2{\times}16{\times}16$	0.24	1.5	0.065	1.8

Appendix

Table 8: Drift rates on robot manipulation , per second between 4.8 and 64 s. Lower is better.

Figure 17: Rollouts over 64 s on robot manipulation. With 256 clips, FID and FVD are biased upward; compare them only within this figure.

	Pixel difference	Pixels that change	Patches that change	Optical flow
Environment	per frame	per frame	per frame	(pixels per frame)
Manipulation (robot)	0.010	2.6%	32%	0.24
Manipulation (human)	0.009	3.6%	33%	1.27
Navigation (driving)	0.017	8.2%	48%	2.13
Navigation (indoor)	0.032	15.7%	76%	2.69

Appendix

Table 9: How much each environment moves. Averages over the 2048 evaluation samples of each environment (1706 distinct clips indoors).

Figure 18: Rollouts, robot manipulation : the same clip and recorded actions for every autoencoder, with the real future on top.

Figure 19: Rollouts, human manipulation : the same clip and recorded actions for every autoencoder, with the real future on top.

Figure 20: Rollouts, driving : the same clip and recorded actions for every autoencoder, with the real future on top.

Figure 21: Rollouts, indoor navigation : the same clip and recorded actions for every autoencoder, with the real future on top.

Figure 22: Counterfactual rollouts. From one real context (left), the world model follows the recorded actions and two counterfactual ones (green): no motion or another clip’s actions in manipulation, a constant left or right turn in indoor navigation.

Environment	$\tau=2$	$\tau=4$	$\tau=8$
Manipulation (robot)	1.17	1.34	1.81
Manipulation (human)	1.47	1.56	1.75
Navigation (driving)	1.22	1.34	1.83
Navigation (indoor)	1.08	1.18	1.58

Appendix

Table 10: Boundary excess at

\tau=2

, 4 and 8 (

2{\times}16{\times}16

4{\times}16{\times}16

8{\times}16{\times}16

;

2{\times}32{\times}32

4{\times}24{\times}24

and

8{\times}16{\times}16

in human manipulation). One means no seam.

Video World Models

Video world models generate and predict action-conditioned visual futures to simulate changing scenes for embodied agents.

See all 250 papers

MiniWorld: Democratizing the Training of Video World Models from Scratch
Aug 2, 2026Yian Zhao, Ruochong Zheng, Hongcan Guo +3Video Diffusion Models Streaming Video Generation
Peking University
BiWM: Advancing Open-Source Interactive Video World Models with Bidirectional Autoregression
Jun 8, 2026Shaohao Rui, Xiaofeng Mao, Zhanyu Zhang +5Action-Conditioned Video Generation Camera-Controlled Video Generation
LynnReal AI · Shanghai Innovation Institute · Shanghai Jiao Tong University +1
Nano World Models: A Minimalist Implementation of Future Video Prediction
May 17, 2026Siqiao Huang, Partha Kaushik, Michael Chen +5Action-Conditioned Video Generation Long-Horizon Video Generation
Tsinghua University · Carnegie Mellon University · University of Bristol +2

Aug 2, 2026J/K move · Enter open · S save

MiniWorld: Democratizing the Training of Video World Models from Scratch

Yian Zhao, Ruochong Zheng, Hongcan Guo, Yu Yan+2

Peking University

Video Diffusion Models Streaming Video Generation Autoregressive Video Generation Video World Models

Open paper AskPDF arXiv

Video world models predict future observations conditioned on historical observations and control signals, enabling long-horizon generation through autoregressive state transitions. Unlike conventional video generation models that primarily capture visual appearance and motion, video world models learn the underlying dynamics governing environment evolution under agent actions, providing a foundation for embodied AI and interactive simulation. Recent progress has largely relied on adapting pretrained video generation models through post-training or distillation. Although effective, these approaches often require complex training pipelines, substantial computational resources, and suffer from the mismatch between bidirectional pretraining and causal streaming inference. Recent studies have shown that training autoregressive video world models from scratch is feasible and scalable. However, the community still lacks a lightweight, transparent, and fully reproducible baseline trainable end-to-end with modest computational resources. We present MiniWorld, a reproducible framework for training streaming video world models from scratch. MiniWorld employs a block-causal Video Diffusion Transformer trained with Flow Matching in the latent space of a pretrained Video VAE. Building on Diffusion Forcing, it adopts a chunk-wise non-decreasing noise schedule and two-stage continued training to improve temporal modeling and stability. During inference, MiniWorld combines a rolling KV cache with pipelined asynchronous denoising for efficient streaming generation under bounded computation. The entire model can be trained within several days on a single 8-GPU server. By releasing the training and inference codebase and pretrained checkpoints, we hope MiniWorld will facilitate future research on video world modeling.

Abstract

Video World Models

Explore similar work

MiniWorld: Democratizing the Training of Video World Models from Scratch

BiWM: Advancing Open-Source Interactive Video World Models with Bidirectional Autoregression

Nano World Models: A Minimalist Implementation of Future Video Prediction