cs.CVAug 24, 2026
SaveEchoWM: Open and Enterable Omnimodal World Models
Organizations: HKUST · Joy Future Academy, JD · PKU · HKU ·
Abstract
We present EchoWM, an omnimodal world model for enterable generative media that responds to continuous navigation while jointly generating 720p video, environmental sound, music and speech. We organize interaction around camera intent: in first-person scenes, it specifies observer motion, while in third-person scenes, camera--character dynamics are learned from data without view-specific controllers. Discrete commands and continuous poses are mapped to a shared metric-scale relative 6-DoF trajectory, with dataset-level calibration preserving motion magnitude across heterogeneous data. To jointly learn audio-visual generation and trajectory control, we construct a complementary data engine and adopt progressive training followed by autoregressive post-training for long-horizon generation. Extensive evaluations show that \model achieves strong trajectory following and high visual quality on public world-model benchmarks, supporting both first- and third-person interaction across varied subjects, and maintaining synchronized environmental sound and speech over long-horizon generation.
Figures & tables
| Output | Participation | Audio feedback | |||||
| Model | High-res. video | FPP + TPP | Continue traj. | Discrete keyboard | Env. sound | Background music | Spoken speech |
| Interactive world models | |||||||
| LingBot-World [ 27 ] | – | – | – | – | |||
| DreamX-World 1.0 [ 4 ] | – | – | – | – | |||
| SANA-WM [ 54 ] | – | – | – | – | |||
| Matrix-Game 3.0 [ 37 ] | – | – | – | – | |||
Table 1: Reported native capabilities of representative generative video and world models. A checkmark denotes explicit native support, whereas “–” means that the capability is not reported. Pose trajectory and discrete navigation denote continuous and discrete viewpoint-related interaction, respectively.
| Audio-visual content | Action signal | Coverage | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Source | Visual | Audio | Speech | Scene | Action | Logs | Pose | Human | Non-game |
| quality | richness | dynamics | clarity | reliability | operation | domain | |||
| Collected gameplay | – | – | |||||||
| Internet gameplay | – | – | |||||||
| UE renders | – | – | – | – | – | ||||
| General Internet video | – | – | |||||||
Table 2: Complementary strengths of the four data sources. Filled versus hollow dots encode relative source strength; checkmarks indicate availability; dashes indicate absence. Scene dynamics refers to subject and environment motion beyond the commanded camera or character trajectory.
Figure 2: Gameplay recordings with native game audio. Representative frames from the internally collected gameplay corpus span six activity categories, with synchronized native-game audio waveforms shown below the visual sequences.
Figure 3: Game-video dataset statistics. The combined training corpora contain predominantly third-person clips, a broad mix of game activities, and a smaller speech-present subset. Percentages are normalized within the reported collection.
Figure 4: World data engine. We combine internally collected game recordings, human-played Internet game recordings, Unreal Engine renders, and general Internet video to complement action logs, natural audio, interactive motion, reliable poses, and cross-domain appearance. A shared pipeline constructs synchronized clips, filters and annotates video and audio, recovers reliable metric poses, and forms AV-rich, control-clean, and balanced high-quality mixtures for AV-CPT, Action-SFT, and Joint-FT, respectively.
Figure 5: Trajectory coverage of the control-clean mixture. (a) Aggregate lateral–forward motion coverage over 28,605 sampled trajectories; the marked P90 range summarizes the robust spatial extent. (b) Representative straight, diagonal, return-loop, and orbit trajectories shown at a common spatial scale. Black and orange markers denote trajectory start and end points, respectively, and dashed circles show fitted orbits.
Figure 6: Overview of EchoWM. Media context, structured text, and user controls share a unified camera-intent interface. Discrete controls or metric poses are converted into globally calibrated relative 6-DoF trajectories and serialized as an event stream. A relative UCPE branch injects this trajectory into video self-attention before audio–visual cross-attention, while the audio stream receives no direct trajectory condition. Four progressive training stages yield synchronized audio–visual generation and support multi-turn inference through synchronized tail-window conditioning.
Figure 7: Camera intent unifies first- and third-person interaction. The same user intent is realized as observer motion in first-person scenes and as coordinated camera–character–world evolution in third-person scenes. The coupling is learned from heterogeneous data rather than specified by an explicit camera rig.
Figure 8: Pose-magnitude distributions for global scale calibration. The plots show the peak-normalized distribution and empirical CDF of the per-clip maximum translation magnitude for the action-clean and AV-rich speech-present subsets. Action-clean clips contain broader motion magnitudes (mode , median ), whereas speech-present AV-rich clips concentrate at smaller motions (mode , median ). The heterogeneous support motivates one robust dataset-level percentile scale rather than independently normalizing each source.
Figure 9: Chunk-level causal attention patterns used for streaming post-training. (a) The audio-visual teacher-forcing pattern used by short-horizon reconstruction: each noisy query attends to the preceding clean context and the noisy tokens of its current chunk, while clean targets and future chunks are excluded. (b) The mask used by Forward 2 of long-horizon SGF, where access to clean history is additionally constrained by the sink-plus-FIFO KV-cache policy.
| Model | Avg. | Quality | Setting | Interaction | Consistency | Physical |
|---|---|---|---|---|---|---|
| EchoWM | 81.7 | 81.5 | 79.4 | 87.2 | 89.8 | 70.6 |
| EchoWM -Flash | 81.0 | 81.1 | 88.3 | 87.9 | 77.5 | 70.1 |
| HiDream-O1-World [ 48 ] | 80.9 | 81.0 | 82.2 | 80.0 | 88.0 | 73.3 |
| Alaya-EVOKE [ 48 ] | 80.8 | 82.8 | 83.8 | 78.6 | 86.9 | 72.1 |
| LingBot-World (fast v2) [ 27 ] | 79.4 | 81.8 | 76.8 | 82.8 | 86.5 | 69.1 |
| Kling 3.0 [ 48 ] | 79.0 | 81.4 | 91.0 | 69.4 | 83.7 | 69.3 |
Table 3: Comparison on the 158-case WBench Navigation split [ 48 ] . Results are sorted by Average.
Figure 10: WBench Navigation comparison.
Figure 11: WBench Navigation comparison.
Figure 12: WBench Navigation comparison.
Figure 13: Qualitative comparison of causal world models on WBench Navigation. EchoWM -Flash shows better action adherence and multi-turn consistency in scene structure, subject identity, and visual style.
| Method | VBench Overall | CMC | ||
|---|---|---|---|---|
| Simple-Trajectory Split | ||||
| Matrix-Game 3 [ 37 ] | 79.41 | 4.474 | 0.406 | 0.453 |
| LingBot-World-v2 [ 27 ] | 81.13 | 3.092 | 0.282 | 0.310 |
| HY-WorldPlay [ 31 ] | 82.05 | 2.256 | 0.284 | 0.298 |
| DreamX-World [ 4 ] | 82.42 | 1.839 | 0.261 | 0.273 |
| SANA-WM [ 54 ] | 81.75 | 0.489 | 0.194 | 0.195 |
Table 4: Short-horizon comparison on SANA-WM-Bench using the front 241 frames of each trajectory. Each split contains 80 scenes. VBench Overall follows the benchmark convention of reporting normalized visual quality on a 0–100 scale. Camera accuracy reports rotation error in degrees, relative translation error , and camera-motion consistency error CMC after similarity alignment.
| Trajectory and visual quality | Revisit consistency | Temporal | ||||||
| Method | VBench | CMC | PSNR | SSIM | LPIPS | IQ | ||
| Simple-Trajectory Split | ||||||||
| Infinite-World [ 40 ] | 79.18 | 16.55 | 1.98 | 2.08 | 12.60 | 0.284 | 0.595 | 6.72 |
| LingBot-World [ 27 ] | 81.82 | 10.47 | 2.01 | 2.05 | 14.59 | 0.366 | 0.394 | 0.04 |
| HY-WorldPlay [ 31 ] | 68.82 | 17.89 | 2.36 | 2.45 | 12.83 | 0.321 | 0.616 | 23.59 |
| Matrix-Game 3 [ 37 ] | 78.53 | 12.96 | 1.83 | 1.92 | 12.29 | 0.326 | 0.553 | 2.41 |
Table 5: Long-horizon comparison on the full SANA-WM-Bench trajectory. Revisit consistency is measured on same-pose pairs. VBench Overall is the normalized Quality component. Temporal degradation is in percentage points; lower is better, and a negative value indicates improvement in the last window. Best and second-best results within each split are shown in bold and underlined, respectively.
| Trajectory and visual quality | Revisit consistency | Temporal | ||||||
| Method | VBench | CMC | PSNR | SSIM | LPIPS | IQ | ||
| Simple-Trajectory Split | ||||||||
| Evoke [ 47 ] | 80.11 | 9.859 | 2.159 | 2.190 | 13.50 | 0.2774 | 0.5010 | -0.49 |
| LingBot-World-v2 [ 27 ] | 79.20 | 22.516 | 2.391 | 2.541 | 11.90 | 0.1715 | 0.5906 | 2.59 |
| SANA-WM + Causal Refiner [ 54 ] | 79.55 | 17.875 | 2.222 | 2.338 | 14.41 | 0.2660 | 0.5748 | -0.75 |
| SANA-WM w/o Refiner [ 54 ] | 78.21 | 15.359 | 2.288 | 2.393 | 9.19 | 0.1661 | 0.6321 | -0.12 |
Table 6: Long-horizon comparison of causal world models on SANA-WM-Bench under the 961-frame protocol. EchoWM -Flash denotes our four-step causal streaming model. Revisit consistency is measured on same-pose pairs, and measures temporal imaging-quality degradation. Best and second-best results within each split are shown in bold and underlined, respectively.
Figure 14: Imaging quality comparison on the Simple- and Hard-Trajectory splits. We report the first-window, mean, minimum, and final-window IQ. Our method achieves consistently higher mean and minimum IQ than Evoke and SANA-WM with Causal Refiner. Although the competing methods exhibit smaller IQ drops, their overall IQ remains lower, indicating that a small drop can also result from consistently low imaging quality rather than stronger long-horizon quality preservation.
Figure 15: Qualitative comparison of long-horizon causal streaming on representative SANA-WM-Bench trajectories under the 961-frame protocol. We compare EchoWM -Flash with LingBot-World-v2 (LBW-V2), SANA-WM with and without Causal Refiner, and Evoke, using the same initial observation and camera-trajectory condition. Frames are shown at 12-second intervals. EchoWM -Flash exhibits less accumulated autoregressive drift while better preserving trajectory adherence, scene identity, visual style, and major spatial structure over the extended rollout.
| Competitor | Dimension | Competitor preferred | Both good | Both bad | EchoWM preferred |
|---|---|---|---|---|---|
| LingBot-World-v2 | Semantic following | 15.14% | 24.93% | 25.93% | 34.00% |
| Initial-world consistency | 7.71% | 62.36% | 15.29% | 14.64% | |
| Motion | 20.00% | 13.71% | 41.00% | 25.29% | |
| Spatial-temporal consistency | 14.71% | 27.36% | 25.29% | 32.64% | |
| Visual aesthetics | 19.93% | 28.00% | 19.57% | 32.50% | |
| Overall preference | 21.57% | 4.21% | 27.71% | 46.50% |
Table 7: Aggregate pairwise user-study judgments over 200 cases. Each cell reports the percentage of judgments. “Both good” and “both bad” preserve the two distinct same-quality outcomes.
| vs. LingBot-World-v2 | vs. HappyOyster | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Scene | Dimension | LBW-v2 | Both good | Both bad | EchoWM | HappyOyster | Both good | Both bad | EchoWM |
| Autonomous driving | Semantic following | 30.48 | 29.52 | 17.14 | 22.86 | 18.33 | 44.17 | 13.33 | 24.17 |
| Initial-world consistency | 3.81 | 66.67 | 15.24 | 14.29 | 10.83 | 50.00 | 0.00 | 39.17 | |
| Motion | 21.90 | 6.67 | 42.86 | 28.57 | 5.83 | 10.83 | 35.83 | 47.50 | |
| Spatial-temporal consistency | 10.48 | 31.43 | 29.52 | 28.57 | 10.00 | 24.17 | 12.50 | 53.33 | |
| Visual aesthetics | 11.43 | 34.29 | 21.90 | 32.38 | 19.17 | 3.33 | 17.50 | 60.00 | |
Table 8: Scene-level user-study results against LingBot-World-v2 and HappyOyster. All values are percentages. The LingBot-World-v2 comparison uses seven annotators, while the HappyOyster comparison uses eight annotators.
Figure 16: First-person world generalization. Temporally ordered rollouts cover game worlds, real-world scenes, fantastic worlds, and stylized worlds. Across varied layouts and rendering styles, EchoWM uses the same camera-intent interface for observer-centric navigation.
Figure 17: Generalization beyond the training domains. Our training mixture contains no 2D or 2.5D game data and no robot data. The top three rows show 2D/2.5D game scenes and the bottom three rows show robot environments; columns are temporally ordered frames under navigation controls. EchoWM preserves coherent viewpoint evolution and scene or subject structure across these unseen domains.
Figure 18: Third-person generalization across visible subjects. Representative rollouts span humans, animals, other foreground subjects, and scenes containing both humans and additional subjects. The shared camera-intent interface supports learned camera–character evolution across these subject configurations without a subject-specific controller.
Figure 19: Qualitative 3D reconstruction from generated trajectories. Each example shows two views of the reconstructed scene together with six temporally ordered frames from the corresponding generated sequence. The recognizable scene-level layouts across viewpoint changes provide qualitative evidence of multi-view spatial coherence.
Figure 20: Native synchronized audio-visual generation. Each example pairs its world and audio description with time-stamped video frames, the generated waveform, and a log-frequency spectrogram. The cases contain environmental sound, background music, and character speech within the same generated audio-visual sequence, rather than adding audio as post-processing.
Figure 21: Visual memory during long-horizon continuation. Each row samples a temporally ordered trajectory that moves through and revisits a scene. Returning to previously observed views tests whether layout, appearance, and scene identity remain coherent over continued generation.
Figure 22: Qualitative scale-control comparison. LingBot-World-v2 [ 27 ] and EchoWM receive matched reference observations and navigation conditions. When the camera approaches a foreground character or scene structure, LingBot-World-v2 exhibits larger changes in apparent subject scale and less predictable displacement response, whereas EchoWM preserves a more gradual scale change across the trajectory. This qualitative comparison complements the metric-scale and speed-response ablations: a fixed global translation scale keeps requested displacement amplitudes comparable across clips and makes approach behavior more controllable.
Figure 23: Qualitative inference-time speed control. Each row shows five temporally ordered frames from a navigation rollout under the same scene and control interface. The changing vehicle position illustrates the realized displacement over the rollout and complements the quantitative response curves described above.
Figure 24: Qualitative viewpoint semantics and third-person tracking. LingBot-World-v2 (LBW2) [ 27 ] and EchoWM receive matched reference observations and navigation conditions in first-person (FPP), third-person human (TPP-Human), and third-person object (TPP-Ball) cases. In the FPP example, LBW2 changes to an exocentric observation as the rollout proceeds, whereas EchoWM retains the requested first-person semantics. In both TPP examples, LBW2 loses the controlled subject from view, while EchoWM keeps the human or ball visible and maintains a more stable camera–subject relationship across the sequence.
More in this topic
Video World Models
Video world models generate and predict action-conditioned visual futures to simulate changing scenes for embodied agents.
See all 256 papers