cs.CVSep 29, 2026

Waypoint-1.5: A Real-Time Video World Model for Consumer Hardware

Authors: Rajit Rajpal, Shahbuland Matiana, Liew Wei Pyn, Anmol Agarwal, Ryan Craig, Andrew Lapp, Mithun Hunsur, Sami BuGhanem, +7 more

Organizations: HuggingFace

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We present Waypoint 1.5, a real-time diffusion world model for interactive video generation on consumer-grade hardware. Unlike general video diffusion models, interactive world models (iWMs) must respond to dense user controls under strict latency and throughput constraints. Waypoint 1.5 is pre-trained on 100,000 hours of diverse, control-aligned video game data across hundreds of games, and generates playable video conditioned on full keyboard and mouse input. The model includes two resolution variants that run across a wide spectrum of consumer hardware. To characterize this unique setting, we distinguish rendered FPS, latent FPS, and control rate. We describe the data pipeline, architecture, training methodology, and runtime system behind Waypoint 1.5. We evaluate interactivity through latency and throughput. Finally, we discuss the safety and ethics considerations unique to iWMs.

Figures & tables

Figure 1 : Waypoint-1.5 teaser. Two example autoregressive rollouts generated in real time on a single consumer GPU (top and bottom rows). The 1.28B parameter model generates high-resolution video conditioned on keyboard and mouse input.
Figure 2 : Diffusion Transformer Model. (a) Training pipeline with per-frame Diffusion Forcing and synchronized controller inputs. (b) Single-stream causal DiT architecture with frame-causal attention.
Figure 3 : Diffusion Transformer Block. Each block applies frame-causal FlexAttention with a static rolling KV cache, WAN-style AdaLN noise conditioning, and MLPFusion controller input injection.
ParameterValue
Total Parameters1,281,959,992
Transformer Layers24
Query Heads / KV Heads32 / 16
Model Dimension ( dmodeld_{\mathrm{model}} )2048
MLP Expansion Ratio4×4\times
Head Dimension64
Table 1 : Waypoint-1.5 model configuration.
Figure 4 : Data Loading Pipeline Workflow. Raw 60 FPS gameplay video and 10,000 Hz controller telemetry are encoded offline into TAEHV 1.5 latents at 15 latent FPS and bucketed controller snapshots, then packed into training sequences with cross-document attention masking.
Hardware720P INT8720P BF16360P INT8360P BF16
RTX PRO 6000 Blackwell132.4087.76371.08303.72
RTX 5090111.4066.28346.64277.52
RTX 409072.8866.60275.84207.52
RTX 5070 Ti59.6839.96270.48178.56
RTX 3090 Ti43.6429.00198.16104.36
RTX 309037.3621.60188.44116.12
Table 2 : Latent frames per second (latent FPS) throughput of Waypoint-1.5 across consumer GPUs. OOM indicates the BF16 model footprint exceeds available VRAM at 720P; INT8 quantization resolves this constraint.
Figure 5 : Waypoint-1.5 autoregressive rollouts. Five example action-conditioned rollouts of generated in real time on consumer hardware (5 frames per row, left to right). Each row is an independent autoregressive sequence conditioned on keyboard and mouse input. The model maintains visual coherence and scene consistency over long horizons across diverse environments.

Explore similar work

CardsList

  1. minWM: A Full-Stack Open-Source Framework for Real-Time Interactive Video World Models

    May 28, 2026Min Zhao, Hongzhou Zhu, Bokai Yan +9Video World ModelsVideo Diffusion Models

    1ShengShu 2THU · 1ShengShu 3RUC · 1ShengShu +4

  2. Matrix-game 2.0: An open-source, real-time, and streaming interactive world model

    Aug 18, 2025Xianglong He, Chunli Peng, Zexiang Liu +16Video World ModelsInteractive Video Generation

    Skywork AI

  3. WorldMark: A Unified Benchmark Suite for Interactive Video World Models

    Apr 23, 2026Xiaojie Xu, Zhengyuan Lin, Kang He +5Video World ModelsWorld Models

    Alaya Studio, Shanda AI Research Tokyo · The University of Tokyo · Shanghai Innovation Institute