Interactive world generation requires models to follow successive commands while preserving relevant history and action consequences. We present Hy-Nomas, a video world model with an 8B diffusion backbone that supports free-form text instructions and continuous camera control across multiple interaction turns. It generates variable-length segments conditioned on current inputs and cached history. Our central design aligns structured video captions with action-aware causal attention: descriptive context remains accessible across segments, while action instructions are restricted to their intended temporal windows. A vision-language Prompt Enhancer (PE) translates user requests and multimodal interaction histories into structured conditions for the next segment. A multi-stage curriculum progressively introduces causal generation, text interaction, camera control, and training on model-generated histories, followed by diffusion distillation. In human evaluation, Hy-Nomas shows superior performance over existing world models. We release model weights, code and benchmarks, together with a lightweight adaptation workflow, to support reproducible research on interactive world generation.
Figures & tables
Figure 1: Interactive world generation with Hy-Nomas. Free-form text instructions and continuous camera control can be interleaved across successive interaction turns. At each turn, a vision-language Prompt Enhancer converts the user request and observed world history into a structured text prompt and a duration for the next video segment. Conditioned on these inputs and cached context, Hy-Nomas autoregressively generates the next segment, allowing the world to unfold through continued interaction.
Figure 2: Data pipeline. Raw videos are split, transcoded, and filtered by basic video properties and visual quality. Hierarchical clustering supports within-cluster deduplication, and a VLM assigns labels to each video from a predefined taxonomy. Video tagging adds captions and, for selected clips, camera-trajectory annotations. Across each training stage, adaptive sampling adjusts the training distribution in response to weaknesses identified from evaluations of earlier stages.
Figure 3: Structured video captioning. The VLM annotates segments sequentially, maintains subject identities, interleaves scene and camera descriptions chronologically, and records subject actions separately with timestamps. A final text-only turn reviews, translates, and validates the annotations to produce aligned English and Chinese captions. During training, camera-motion sentences and their tags are removed when camera trajectories are available and retained otherwise.
Figure 5: Overview of the model architecture. A structured multi-segment prompt, a video clip, and a camera trajectory are encoded into text, video, and camera tokens, respectively. Text and video tokens are processed by N=54 dual-stream MMDiT blocks. Within each block, segment-wise causal attention with 3D RoPE allows tokens to attend to the current and preceding segments, while diffusion timestep embeddings modulate both streams. The two streams pass through separate MLPs, and camera cross-attention further conditions the video stream on camera tokens. During autoregressive generation, keys and values associated with both text and video tokens are stored in the KV cache as context for subsequent segments. The 3D VAE decoder maps denoised video latents to RGB frames. Purple, blue, and green denote the text, video, and camera streams, respectively.
Component
Configuration
Video autoencoder
DCAE-V, 32× spatial and 4× temporal compression
Latent representation
32 channels; patch size 1×1×1
Text encoder
Qwen2.5-VL-7B, width 3,584, maximum length 1,280
Transformer backbone
54 dual-stream MMDiT blocks
Hidden width / MLP width
2,048 / 8,192
Self-attention
16 heads, 128 dimensions per head
Table 1: Summary of primary architectural settings.
Figure 6: Comparison of segment-wise and action-aware causal attention. (a) Tokens attend to all video and text tokens in the current and preceding segments. (b) Action-aware masking further restricts access to action-specific text tokens. Colors indicate query-to-key modality pairs; gray cells are masked. head denotes the environment description for the current segment, and a0,…,an denote actions within that segment.
Transfer HunyuanVideo 1.5 VAE to a more compact DCAE-V latent space.
2
Segment-causal CPT
Broad video corpus with segment-wise captions, followed by higher-quality data
Establish segment-wise causal generation before introducing finer-grained action control.
Action-aware CPT
Curated videos with balanced sampling and temporally localized action captions
Learn action timing and cross-segment action isolation; expand the context length from 180K to 360K tokens.
Camera-conditioned CPT
Videos with and without camera trajectories, mixed at a 1:1 ratio
Add continuous camera control while retaining text-driven generation.
3
High-quality annealing
High-quality videos from balanced sources; detailed captions with Kimi K3 and camera trajectory annotations
Refine visual quality, text conditioning, and camera control.
4
Segment forcing
A subset of high-quality annealing data
Align training with segment-wise autoregressive inference and its evolving KV-cache state.
Table 2: Multi-stage training of Hy-Nomas. Stage 2 progresses from segment-causal to action-aware and camera-conditioned CPT. Segment-causal CPT includes a continuation on higher-quality data. Downstream LoRA adaptation is performed separately after the listed training stages.
Figure 7: Segment Forcing with streaming backward. Each current segment receives a flow-matching loss; a one-step low-noise estimation supplies cached context for later segments. Streaming backward retains only one segment’s computation graph at a time, accumulating gradients for a single optimizer step after the full sequence. The cache update of the final segment is skipped.
Figure 8: Downstream LoRA adaptation workflow. Train LoRA adapters on the base autoregressive model (before distillation), then merge the learned weights into the distilled model for efficient deployment, without full-model fine-tuning or repeating distillation.
Figure 9: Prompt Enhancer (PE) workflow . PE generates an initial dense caption in the first turn. In subsequent turns, users can choose text prompts or camera control, and PE combines the selected input with the full interaction history to generate a dense caption that AR-DiT uses to produce the next video segment. Each turn’s dialogue and generated video are appended to the history.
Figure 10: Overview of Hy-Nomas-Bench. Top: Model outputs for an example case from Bench-3 (interleaved text–camera control). Bottom: (a) World settings span diverse scenes, visual styles, subjects, and viewpoints. (b) 280 cases cover multi-turn text interaction, camera navigation, interleaved control, and spatial memory. (c) Unified control maps instructions to text, binary key states [T,8] ( W/A/S/D and four arrow keys), or per-frame 4×4 camera-to-world matrices [T,4,4] , where T is the frame count. Generation continues across turns without world resets. (d) Three annotators evaluate prompt following, physical plausibility, camera control, and memory consistency through pairwise comparisons. Results are summarized by win/tie/loss rates and net win rate (win minus loss, in percentage points).
Figure 11: Illustration of the design of Bench-4: Spatial memory. Each uses a static anchor object and four outbound–return pairs. Bench-4 evaluates object permanence under different camera movements, and appearance/layout consistency. Left: Different trajectory types. Center: Scene, camera frustum, and trajectory. Right: Inference results by Hy-Nomas.
System
Multi-turn free-form text
Camera navigation
Interleaved text and camera
Spatial memory
Hy-Nomas
✓
✓
✓
✓
Genie 3
–
✓
–
✓
LingBot-World v2
✓
✓
✓
✓
Happy Oyster
✓
✓
–
✓
Table 3: Models evaluated on each benchmark. Checkmarks indicate the model is evaluated on the benchmark; dashes indicate that the benchmark settings is not applicable to the model’s provided interaction interface. For LingBot-World v2, we adapt their open-source model for Bench-3.
Figure 12: Qualitative comparison of multi-turn text interaction. In the first example, Hy-Nomas clearly shows the apple being grasped, lifted, and brought to the plate. LingBot-World v2 changes the apple’s color and gripper structure, while HappyOyster-Directing shows discontinuous views without a clear grasp-and-lift progression. In the second example, Hy-Nomas visualizes the requested spatula, cutting on the board, and strawberries in the glass bowl. LingBot-World v2 only shows cutting strawberries and a knife without spatula, while HappyOyster-Directing does not move strawberries into the bowl.
Figure 13: Pairwise human evaluation of multi-turn text control. We compare Hy-Nomas with HappyOyster-Directing and LingBot-World v2 on prompt following and physical plausibility. Within each evaluation dimension, the two 100% stacked bars show wins (green), ties (gray), and losses (pink) from the perspective of Hy-Nomas.
Figure 14: Qualitative comparison of camera control in an indoor scene. Rows show Hy-Nomas (Ours), LingBot-World v2, HappyOyster-Wandering, and Genie 3. The comparison illustrates how each model responds to navigation commands while maintaining a third-person view of the character and the surrounding room. The middle frame of LingBot-World v2 shows a duplicated character, illustrating a scene-consistency failure during navigation. Visible keyboard indicators denote active controls in the source recordings.
Figure 15: Pairwise human evaluation of camera control. We compare Hy-Nomas with Genie 3, HappyOyster-Wandering, and LingBot-World v2 on 100 shared trajectories. We measure how accurately the model responds to navigation commands.
Figure 16: Pairwise human evaluation of interleaved text and camera control. We compare Hy-Nomas with the adapted LingBot-World v2 baseline on 60 scenarios with alternating text and camera inputs. We consider prompt following, physical plausibility, and camera control during the evaluation.
Figure 17: Pairwise human evaluation of spatial memory. We compare Hy-Nomas with Genie 3, HappyOyster, and LingBot-World v2 on 60 shared scenarios involving reversible or revisiting camera trajectories. We measure how well the model remembers the anchor object and its surrounding environments.
Figure 18: Qualitative comparison of spatial memory under viewpoint revisitation. Rows show Hy-Nomas (Ours), LingBot-World v2, HappyOyster-Wandering, and Genie 3; columns show successive frames from a camera-only rollout. Hy-Nomas and Genie 3 retain the closed toolbox in revisited views, whereas LingBot-World v2 changes it to an open toolbox. This case illustrates object-state consistency; aggregate results in Figure 17 provide the broader comparison.
Figure 19: Qualitative results of LoRA-SFT. We compare the distilled model with domain-specific LoRA-SFT variants. The upper examples show improved cinematic aesthetics after cinematic LoRA-SFT, while the lower examples show more physically plausible robotic manipulation, with cup deformation more faithfully reflecting the gripper’s applied pressure. LoRA-SFT denotes the distilled model with the trained LoRA weights merged into it.
Figure 20: Pairwise human evaluation of cinematic LoRA-SFT against the distilled model. We evaluate prompt following and physical plausibility on Bench-1, and camera control on Bench-2. The three 100% stacked bars show wins (green), ties (gray), and losses (pink) from the perspective of LoRA-SFT relative to the base model. LoRA-SFT denotes the distilled model with the trained LoRA weights merged into it.
Figure 21: Rank-gap-weighted human evaluation of Hy-Nomas-PE (ours). Bench-1 and bench-3 are reported separately; each pair compares Hy-Nomas-PE with Qwen3.8-27B and Kimi-K3. Bench-1 evaluates prompt following and physical plausibility; bench-3 also evaluates camera control. Category labels are shared within each benchmark. Each 100% stacked bar shows win (green), tie (gray), and loss (pink) votes from the perspective of Hy-Nomas-PE.
DreamX-World 1.0 is a general-purpose interactive text/image-to-video world model for controllable long-horizon generation. It supports camera navigation, revisits to previously observed regions, and promptable events across photorealistic, game-style, and stylized domains. Our data engine combines camera-accurate Unreal Engine rendering, action-rich gameplay recordings, and real-world videos with recovered camera geometry. For camera control, we introduce E-PRoPE, a lightweight variant of projective positional encoding that retains PRoPE's projective camera geometry while applying camera-aware attention to spatially reduced tokens. We convert a bidirectional video generator into a few-step autoregressive world model using causal forcing, DMD-style distillation, and long-rollout training. Training on self-generated long-horizon contexts exposes the model to its own generated history and reduces the style and color drift that accumulates across autoregressive chunks. Memory-Conditioned Scene Persistence retrieves earlier views through camera-geometry-based retrieval, while residual recycling makes the conditioning path less sensitive to imperfect memory latents. Event Instruction Tuning adds composable event control, and reinforcement learning alignment recovers camera control and visual quality after distillation. With mixed-precision DiT execution, residual reuse, 75%-pruned VAE decoding, and asynchronous pipeline parallelism, DreamX-World 1.0 reaches up to 16,FPS on eight RTX,5090 GPUs. On our 5-second basic evaluation, DreamX-World 1.0 achieves a camera-control score of 73.75 and an overall score of 84.76, outperforming HY-WorldPlay 1.5 and LingBot-World in overall score, which achieve 80.79 and 80.45, respectively.
Unlike conventional video game development, which relies on labor-intensive pipelines for asset production, animation, physics, and programming, video world models generate interactive environments from user inputs instantly. It enable us to create customized, explorable, and continuously evolving virtual world from text, an image, or video. Realizing this vision requires four tightly coupled capabilities: interaction, persistent spatiotemporal consistency, stable long-horizon generation, and efficient response. We present AlayaWorld, an interactive long-horizon video world model that generates 24-fps video at 540p and 720p. Built on a 15B video diffusion transformer, AlayaWorld generates short latent chunks autoregressively under camera trajectories and switchable text prompts. Its bounded visual context combines a persistent sink frame, compressed temporal history, geometry-aligned spatial memory, and recent-frame conditioning. To reduce long-term drift, the model is trained with corrupted histories and prediction residuals collected from its own roll-outs. We further introduce a discrete autoregressive distillation formulation that combines distribution-matching distillation, self-forcing++, and consistency distillation, reducing inference from approximately 30 sampling steps to four steps per chunk. On iWorld-Bench, AlayaWorld achieves the best performance over long-horizon generation. Conceived as a full-stack, open-source, and long-term project, AlayaWorld is intended to provide an extensible foundation for future research on interactive video world models.
We present H3-World, an efficient framework that turns the 33B MiniMax-H3 video generator into an interactive world model. Our key finding is that, as large video generators become more capable, language is emerging as a natural interface for control. MiniMax-H3, for example, already supports zero-shot control of character behavior and camera motion through natural-language instructions. Building on this, H3-World turns this coarse language interface into precise, temporally grounded world control, without introducing dedicated action modules. Specifically, we represent each action as a structured combination of character and camera instructions, and align them with the corresponding temporal video latents. To make the control temporally precise, we further introduce temporal attention routing, which restricts each instruction to its intended time interval and reduces control leakage across actions. Importantly, H3-World directly reuses the semantic representations learned during large-scale video pretraining and requires only lightweight adaptation. With only 8,000 gameplay samples, 10,000 LoRA optimization steps, and 0.199% trainable parameters, H3-World achieves effective character and camera control while preserving strong generation quality. It also generalizes to unseen scenarios. These results show that the control capabilities emerging in large video generators can be efficiently transformed into interactive world control.
Danze Chen, Zeqing Wang, Ziyue Lin +2
Tencent · National University of Singapore · The Hong Kong Polytechnic University