cs.CVOct 27, 2026
SaveHy-Nomas: Towards Free-form Interactive World Generation
Organizations: Tencent Hy Visual Generation Team
Abstract
Interactive world generation requires models to follow successive commands while preserving relevant history and action consequences. We present Hy-Nomas, a video world model with an 8B diffusion backbone that supports free-form text instructions and continuous camera control across multiple interaction turns. It generates variable-length segments conditioned on current inputs and cached history. Our central design aligns structured video captions with action-aware causal attention: descriptive context remains accessible across segments, while action instructions are restricted to their intended temporal windows. A vision-language Prompt Enhancer (PE) translates user requests and multimodal interaction histories into structured conditions for the next segment. A multi-stage curriculum progressively introduces causal generation, text interaction, camera control, and training on model-generated histories, followed by diffusion distillation. In human evaluation, Hy-Nomas shows superior performance over existing world models. We release model weights, code and benchmarks, together with a lightweight adaptation workflow, to support reproducible research on interactive world generation.
Figures & tables
Figure 1: Interactive world generation with Hy-Nomas. Free-form text instructions and continuous camera control can be interleaved across successive interaction turns. At each turn, a vision-language Prompt Enhancer converts the user request and observed world history into a structured text prompt and a duration for the next video segment. Conditioned on these inputs and cached context, Hy-Nomas autoregressively generates the next segment, allowing the world to unfold through continued interaction.
Figure 2: Data pipeline. Raw videos are split, transcoded, and filtered by basic video properties and visual quality. Hierarchical clustering supports within-cluster deduplication, and a VLM assigns labels to each video from a predefined taxonomy. Video tagging adds captions and, for selected clips, camera-trajectory annotations. Across each training stage, adaptive sampling adjusts the training distribution in response to weaknesses identified from evaluations of earlier stages.
Figure 3: Structured video captioning. The VLM annotates segments sequentially, maintains subject identities, interleaves scene and camera descriptions chronologically, and records subject actions separately with timestamps. A final text-only turn reviews, translates, and validates the annotations to produce aligned English and Chinese captions. During training, camera-motion sentences and their tags are removed when camera trajectories are available and retained otherwise.
Figure 5: Overview of the model architecture. A structured multi-segment prompt, a video clip, and a camera trajectory are encoded into text, video, and camera tokens, respectively. Text and video tokens are processed by dual-stream MMDiT blocks. Within each block, segment-wise causal attention with 3D RoPE allows tokens to attend to the current and preceding segments, while diffusion timestep embeddings modulate both streams. The two streams pass through separate MLPs, and camera cross-attention further conditions the video stream on camera tokens. During autoregressive generation, keys and values associated with both text and video tokens are stored in the KV cache as context for subsequent segments. The 3D VAE decoder maps denoised video latents to RGB frames. Purple, blue, and green denote the text, video, and camera streams, respectively.
| Component | Configuration |
| Video autoencoder | DCAE-V, spatial and temporal compression |
| Latent representation | 32 channels; patch size |
| Text encoder | Qwen2.5-VL-7B, width 3,584, maximum length 1,280 |
| Transformer backbone | 54 dual-stream MMDiT blocks |
| Hidden width / MLP width | 2,048 / 8,192 |
| Self-attention | 16 heads, 128 dimensions per head |
Table 1: Summary of primary architectural settings.
Figure 6: Comparison of segment-wise and action-aware causal attention. (a) Tokens attend to all video and text tokens in the current and preceding segments. (b) Action-aware masking further restricts access to action-specific text tokens. Colors indicate query-to-key modality pairs; gray cells are masked. head denotes the environment description for the current segment, and denote actions within that segment.
| Stage | Training regime | Primary data or supervision | Main objective or capability |
| 1 | Latent-space adaptation | HunyuanVideo 1.5 initialization; 20K steps updating input/output layers, then 60K full-model updates | Transfer HunyuanVideo 1.5 VAE to a more compact DCAE-V latent space. |
| 2 | Segment-causal CPT | Broad video corpus with segment-wise captions, followed by higher-quality data | Establish segment-wise causal generation before introducing finer-grained action control. |
| Action-aware CPT | Curated videos with balanced sampling and temporally localized action captions | Learn action timing and cross-segment action isolation; expand the context length from to tokens. | |
| Camera-conditioned CPT | Videos with and without camera trajectories, mixed at a 1:1 ratio | Add continuous camera control while retaining text-driven generation. | |
| 3 | High-quality annealing | High-quality videos from balanced sources; detailed captions with Kimi K3 and camera trajectory annotations | Refine visual quality, text conditioning, and camera control. |
| 4 | Segment forcing | A subset of high-quality annealing data | Align training with segment-wise autoregressive inference and its evolving KV-cache state. |
Table 2: Multi-stage training of Hy-Nomas. Stage 2 progresses from segment-causal to action-aware and camera-conditioned CPT. Segment-causal CPT includes a continuation on higher-quality data. Downstream LoRA adaptation is performed separately after the listed training stages.
Figure 7: Segment Forcing with streaming backward. Each current segment receives a flow-matching loss; a one-step low-noise estimation supplies cached context for later segments. Streaming backward retains only one segment’s computation graph at a time, accumulating gradients for a single optimizer step after the full sequence. The cache update of the final segment is skipped.
Figure 8: Downstream LoRA adaptation workflow. Train LoRA adapters on the base autoregressive model (before distillation), then merge the learned weights into the distilled model for efficient deployment, without full-model fine-tuning or repeating distillation.
Figure 9: Prompt Enhancer (PE) workflow . PE generates an initial dense caption in the first turn. In subsequent turns, users can choose text prompts or camera control, and PE combines the selected input with the full interaction history to generate a dense caption that AR-DiT uses to produce the next video segment. Each turn’s dialogue and generated video are appended to the history.
Figure 10: Overview of Hy-Nomas-Bench. Top: Model outputs for an example case from Bench-3 (interleaved text–camera control). Bottom: (a) World settings span diverse scenes, visual styles, subjects, and viewpoints. (b) 280 cases cover multi-turn text interaction, camera navigation, interleaved control, and spatial memory. (c) Unified control maps instructions to text, binary key states ( W/A/S/D and four arrow keys), or per-frame camera-to-world matrices , where is the frame count. Generation continues across turns without world resets. (d) Three annotators evaluate prompt following, physical plausibility, camera control, and memory consistency through pairwise comparisons. Results are summarized by win/tie/loss rates and net win rate (win minus loss, in percentage points).
Figure 11: Illustration of the design of Bench-4: Spatial memory. Each uses a static anchor object and four outbound–return pairs. Bench-4 evaluates object permanence under different camera movements, and appearance/layout consistency. Left: Different trajectory types. Center: Scene, camera frustum, and trajectory. Right: Inference results by Hy-Nomas.
| System | Multi-turn free-form text | Camera navigation | Interleaved text and camera | Spatial memory |
| Hy-Nomas | ||||
| Genie 3 | – | – | ||
| LingBot-World v2 | ||||
| Happy Oyster | – |
Table 3: Models evaluated on each benchmark. Checkmarks indicate the model is evaluated on the benchmark; dashes indicate that the benchmark settings is not applicable to the model’s provided interaction interface. For LingBot-World v2, we adapt their open-source model for Bench-3.
Figure 12: Qualitative comparison of multi-turn text interaction. In the first example, Hy-Nomas clearly shows the apple being grasped, lifted, and brought to the plate. LingBot-World v2 changes the apple’s color and gripper structure, while HappyOyster-Directing shows discontinuous views without a clear grasp-and-lift progression. In the second example, Hy-Nomas visualizes the requested spatula, cutting on the board, and strawberries in the glass bowl. LingBot-World v2 only shows cutting strawberries and a knife without spatula, while HappyOyster-Directing does not move strawberries into the bowl.
Figure 13: Pairwise human evaluation of multi-turn text control. We compare Hy-Nomas with HappyOyster-Directing and LingBot-World v2 on prompt following and physical plausibility. Within each evaluation dimension, the two 100% stacked bars show wins (green), ties (gray), and losses (pink) from the perspective of Hy-Nomas.
Figure 14: Qualitative comparison of camera control in an indoor scene. Rows show Hy-Nomas (Ours), LingBot-World v2, HappyOyster-Wandering, and Genie 3. The comparison illustrates how each model responds to navigation commands while maintaining a third-person view of the character and the surrounding room. The middle frame of LingBot-World v2 shows a duplicated character, illustrating a scene-consistency failure during navigation. Visible keyboard indicators denote active controls in the source recordings.
Figure 15: Pairwise human evaluation of camera control. We compare Hy-Nomas with Genie 3, HappyOyster-Wandering, and LingBot-World v2 on 100 shared trajectories. We measure how accurately the model responds to navigation commands.
Figure 16: Pairwise human evaluation of interleaved text and camera control. We compare Hy-Nomas with the adapted LingBot-World v2 baseline on 60 scenarios with alternating text and camera inputs. We consider prompt following, physical plausibility, and camera control during the evaluation.
Figure 17: Pairwise human evaluation of spatial memory. We compare Hy-Nomas with Genie 3, HappyOyster, and LingBot-World v2 on 60 shared scenarios involving reversible or revisiting camera trajectories. We measure how well the model remembers the anchor object and its surrounding environments.
Figure 18: Qualitative comparison of spatial memory under viewpoint revisitation. Rows show Hy-Nomas (Ours), LingBot-World v2, HappyOyster-Wandering, and Genie 3; columns show successive frames from a camera-only rollout. Hy-Nomas and Genie 3 retain the closed toolbox in revisited views, whereas LingBot-World v2 changes it to an open toolbox. This case illustrates object-state consistency; aggregate results in Figure 17 provide the broader comparison.
Figure 19: Qualitative results of LoRA-SFT. We compare the distilled model with domain-specific LoRA-SFT variants. The upper examples show improved cinematic aesthetics after cinematic LoRA-SFT, while the lower examples show more physically plausible robotic manipulation, with cup deformation more faithfully reflecting the gripper’s applied pressure. LoRA-SFT denotes the distilled model with the trained LoRA weights merged into it.
Figure 20: Pairwise human evaluation of cinematic LoRA-SFT against the distilled model. We evaluate prompt following and physical plausibility on Bench-1, and camera control on Bench-2. The three 100% stacked bars show wins (green), ties (gray), and losses (pink) from the perspective of LoRA-SFT relative to the base model. LoRA-SFT denotes the distilled model with the trained LoRA weights merged into it.
Figure 21: Rank-gap-weighted human evaluation of Hy-Nomas-PE (ours). Bench-1 and bench-3 are reported separately; each pair compares Hy-Nomas-PE with Qwen3.8-27B and Kimi-K3. Bench-1 evaluates prompt following and physical plausibility; bench-3 also evaluates camera control. Category labels are shared within each benchmark. Each 100% stacked bar shows win (green), tie (gray), and loss (pink) votes from the perspective of Hy-Nomas-PE.
More in this topic
Interactive World Models
Interactive world models predict and generate how environments change in response to actions, supporting simulation, planning, and control.
See all 117 papers