Organizations: East China Normal University · Université Sorbonne Paris Nord, LIPN, CNRS · Université Paris Cité, LLF, CNRS · Dalian Jiaotong University
Recent video world models allow users to explore generated environments through camera navigation. Continuous hand motion provides a fine-grained action channel for interacting with the people and objects encountered during exploration. We present TouchéWorld, a framework that integrates gesture-driven interaction into a camera-controlled video world model. Given an initial image and a scene-description prompt, the model generates video conditioned on temporally aligned camera and hand controls. We construct joint camera--hand training data from heterogeneous egocentric videos through annotation completion, consistency-based filtering, and temporal alignment. Building on a pretrained camera-controlled world model, we combine spatial hand-skeleton conditioning with numerical hand-motion features through parameter-efficient adaptation. Causal adaptation and few-step distillation support chunk-level online generation, allowing users to update their controls based on generated visual feedback. We further introduce TouchéBench to evaluate hand-motion following, camera--hand coordination, online interaction, and object response. Together, these components establish a framework for studying continuous exploration and hand-driven interaction within generative worlds.
Figures & tables
Figure 1: From camera-guided exploration to continuous hand–world interaction. Camera control allows users to navigate generated environments, while hand motion provides a fine-grained action channel for interacting with encountered scene contents. In the illustrated example, the user approaches an osmanthus tree and reaches into its canopy, producing hand contact and corresponding changes in the foliage. With scene-only text conditioning, camera and hand inputs guide the unfolding interaction, while causal chunkwise generation incorporates updated controls into subsequent visual output.
Figure 2: Joint camera–hand data construction through annotation completion, consistency-based filtering, and temporal alignment.
Figure 3: Overview of TouchéWorld: joint camera–hand conditioning, causal adaptation, few-step distillation, and online interaction with updated controls.
Figure 4: TouchéBench evaluates hand-motion following (Q1), camera–hand coordination (Q2), online interaction (Q3), and object response (Q4).
Figure 5: Generation across scenes using shared camera and hand controls from an osmanthus-tree interaction. The final case compares methods on the store-shelf scene. Frames are sampled at matched relative progress.
Figure 6: Object-response comparison on HOI4D. Magenta overlays show SAM3-tracked masks; green contours show reference masks. Image-space overlap can differ from action-consistent object motion.
Figure 7: Action-dependent continuations from a shared two-second source context under rotating, lifting/stacking, and sliding controls. Branch labels describe evaluation goals and are not supplied as action-text prompts.
EgoStandard-200
EgoLife-300
HOT3D-45 (OOD)
Method
PSNR ↑
SSIM ↑
LPIPS ↓
DINOv2 ↑
KID ↓
PSNR ↑
SSIM ↑
LPIPS ↓
DINOv2 ↑
KID ↓
PSNR ↑
SSIM ↑
LPIPS ↓
DINOv2 ↑
KID ↓
Bidirectional
LTX-2.5 I2V
15.95
0.615
0.551
0.724
0.00887
13.61
0.473
0.635
0.537
0.00648
14.04
0.517
0.558
0.707
0.01766
SolarWM (camera-only)
16.36
0.643
0.494
0.728
0.01732
14.53
0.527
0.584
0.542
0.01033
14.77
0.543
0.513
0.719
0.01154
JCV
16.63
0.643
0.486
0.757
0.00308
13.47
0.473
0.647
0.486
0.01533
12.64
0.470
0.634
0.726
0.02330
Hand2World
13.81
0.619
0.635
0.402
0.07425
12.08
0.468
0.692
0.354
0.02642
13.63
0.512
0.553
0.600
0.07396
Table 1: Video quality on unseen in-domain scenes and out-of-domain HOT3D. All our variants use hand-weighted loss. Best results are bold; KID is unscaled.
Table 2: TouchéBench results. All our variants use hand-weighted loss. PCK, checklist scores, and IoU are percentages; L2 and target-center offset (TO) are in pixels. Q2 Interaction averages action-execution and final-state pass rates. Best results are bold.
Model
Hand loss
PSNR ↑
SSIM ↑
LPIPS ↓
DINO ↑
KID ↓
EgoStandard-200
Teacher
w/o
19.13
0.705
0.350
0.812
0.00280
Teacher
w/
19.18
0.708
0.345
0.816
0.00265
Causal
w/o
17.61
0.668
0.416
0.780
0.00354
Causal
w/
17.60
0.666
0.410
0.785
0.00367
EgoLife-300
Table 3: Effect of hand-weighted loss on video quality. Better results within each Teacher or Causal pair are bold, including ties.
Table 4: Effect of hand-weighted loss on TouchéBench. Scores are percentages except L2 (pixels). Weighted variants match the main-table models. Better results within each model pair are bold, including ties.