Organizations: East China Normal University · Université Sorbonne Paris Nord, LIPN, CNRS · Université Paris Cité, LLF, CNRS · Dalian Jiaotong University
Recent video world models allow users to explore generated environments through camera navigation. Continuous hand motion provides a fine-grained action channel for interacting with the people and objects encountered during exploration. We present TouchéWorld, a framework that integrates gesture-driven interaction into a camera-controlled video world model. Given an initial image and a scene-description prompt, the model generates video conditioned on temporally aligned camera and hand controls. We construct joint camera--hand training data from heterogeneous egocentric videos through annotation completion, consistency-based filtering, and temporal alignment. Building on a pretrained camera-controlled world model, we combine spatial hand-skeleton conditioning with numerical hand-motion features through parameter-efficient adaptation. Causal adaptation and few-step distillation support chunk-level online generation, allowing users to update their controls based on generated visual feedback. We further introduce TouchéBench to evaluate hand-motion following, camera--hand coordination, online interaction, and object response. Together, these components establish a framework for studying continuous exploration and hand-driven interaction within generative worlds.
Figures & tables
Figure 1: From camera-guided exploration to continuous hand–world interaction. Camera control allows users to navigate generated environments, while hand motion provides a fine-grained action channel for interacting with encountered scene contents. In the illustrated example, the user approaches an osmanthus tree and reaches into its canopy, producing hand contact and corresponding changes in the foliage. With scene-only text conditioning, camera and hand inputs guide the unfolding interaction, while causal chunkwise generation incorporates updated controls into subsequent visual output.
Figure 2: Joint camera–hand data construction through annotation completion, consistency-based filtering, and temporal alignment.
Figure 3: Overview of TouchéWorld: joint camera–hand conditioning, causal adaptation, few-step distillation, and online interaction with updated controls.
Figure 4: TouchéBench evaluates hand-motion following (Q1), camera–hand coordination (Q2), online interaction (Q3), and object response (Q4).
Figure 5: Generation across scenes using shared camera and hand controls from an osmanthus-tree interaction. The final case compares methods on the store-shelf scene. Frames are sampled at matched relative progress.
Figure 6: Object-response comparison on HOI4D. Magenta overlays show SAM3-tracked masks; green contours show reference masks. Image-space overlap can differ from action-consistent object motion.
Figure 7: Action-dependent continuations from a shared two-second source context under rotating, lifting/stacking, and sliding controls. Branch labels describe evaluation goals and are not supplied as action-text prompts.
EgoStandard-200
EgoLife-300
HOT3D-45 (OOD)
Method
PSNR ↑
SSIM ↑
LPIPS ↓
DINOv2 ↑
KID ↓
PSNR ↑
SSIM ↑
LPIPS ↓
DINOv2 ↑
KID ↓
PSNR ↑
SSIM ↑
LPIPS ↓
DINOv2 ↑
KID ↓
Bidirectional
LTX-2.5 I2V
15.95
0.615
0.551
0.724
0.00887
13.61
0.473
0.635
0.537
0.00648
14.04
0.517
0.558
0.707
0.01766
SolarWM (camera-only)
16.36
0.643
0.494
0.728
0.01732
14.53
0.527
0.584
0.542
0.01033
14.77
0.543
0.513
0.719
0.01154
JCV
16.63
0.643
0.486
0.757
0.00308
13.47
0.473
0.647
0.486
0.01533
12.64
0.470
0.634
0.726
0.02330
Hand2World
13.81
0.619
0.635
0.402
0.07425
12.08
0.468
0.692
0.354
0.02642
13.63
0.512
0.553
0.600
0.07396
Table 1: Video quality on unseen in-domain scenes and out-of-domain HOT3D. All our variants use hand-weighted loss. Best results are bold; KID is unscaled.
Table 2: TouchéBench results. All our variants use hand-weighted loss. PCK, checklist scores, and IoU are percentages; L2 and target-center offset (TO) are in pixels. Q2 Interaction averages action-execution and final-state pass rates. Best results are bold.
Model
Hand loss
PSNR ↑
SSIM ↑
LPIPS ↓
DINO ↑
KID ↓
EgoStandard-200
Teacher
w/o
19.13
0.705
0.350
0.812
0.00280
Teacher
w/
19.18
0.708
0.345
0.816
0.00265
Causal
w/o
17.61
0.668
0.416
0.780
0.00354
Causal
w/
17.60
0.666
0.410
0.785
0.00367
EgoLife-300
Table 3: Effect of hand-weighted loss on video quality. Better results within each Teacher or Causal pair are bold, including ties.
Table 4: Effect of hand-weighted loss on TouchéBench. Scores are percentages except L2 (pixels). Weighted variants match the main-table models. Better results within each model pair are bold, including ties.
Interactive video world models enable exploration through text and camera controls, yet direct object interaction requires finer control. Hands provide a natural interface, but translating continuous hand movements into coherent visual responses remains challenging because of complex hand-object dynamics and demanding real-time computation. To address these challenges, we introduce LynnReal-Hand, a hand-controlled video world model that unifies precise interaction modeling with efficient streaming generation. We curate 100K interaction clips spanning diverse real and synthetic environments and develop hand-aware supervision to improve fine-grained hand motions and object interactions. Building on this foundation, we learn a causal flow map for few-step generation and introduce region-aware score distillation that combines complementary video and image priors to refine interaction dynamics and visual appearance. To further reduce inference latency, we propose Ultra-Fast-VAE, a compact video decoder that preserves high reconstruction fidelity with substantially lower computational cost. Extensive evaluations on HandBench demonstrate superior hand-following accuracy and visual quality compared with existing approaches. Our full-sequence model achieves an FVD of 181.67 and a MANO Dice score of 93.02, while the three-step causal model maintains a MANO Dice score of 92.49. For real-time interaction, 540p streaming on a single H200 reaches 18.03 FPS with 741ms latency and an optimized peak GPU memory footprint of only 34.89GB. We will release the datasets, model weights, and the full training and inference code to support research on hand-controlled video world models.
Controllable video world models target interactive world exploration, where models must faithfully execute explicit action commands while preserving visual quality and temporal coherence. However, most existing approaches rely on continuous camera trajectories as action conditions, which often lead to unreliable action following, especially under complex motion sequences. In this work, we identify action representation entanglement as a key bottleneck in controllable video generation, and show that continuous camera representations lead to high feature similarity across distinct motion patterns, degrading action controllability. Based on this insight, we propose DisCo, a controllable video world model that conditions generation on a compact set of discrete action primitives to improve action separability. We further introduce DisCoBench, a comprehensive benchmark for evaluating the ability of models in short-term, long-horizon, and highly dynamic exploration scenarios. Extensive experiments demonstrate that DisCo achieves significantly more reliable action following while preserving visual quality.
Interactive world generation requires models to follow successive commands while preserving relevant history and action consequences. We present Hy-Nomas, a video world model with an 8B diffusion backbone that supports free-form text instructions and continuous camera control across multiple interaction turns. It generates variable-length segments conditioned on current inputs and cached history. Our central design aligns structured video captions with action-aware causal attention: descriptive context remains accessible across segments, while action instructions are restricted to their intended temporal windows. A vision-language Prompt Enhancer (PE) translates user requests and multimodal interaction histories into structured conditions for the next segment. A multi-stage curriculum progressively introduces causal generation, text interaction, camera control, and training on model-generated histories, followed by diffusion distillation. In human evaluation, Hy-Nomas shows superior performance over existing world models. We release model weights, code and benchmarks, together with a lightweight adaptation workflow, to support reproducible research on interactive world generation.