Thinker-WM: A Unified World Model with Shared Multimodal Representations for Embodied Intelligence

Thinker Team; Peng Gao; Jichao Jiao; Chen Liao; Daqin Luo; Yiming Ma; Baiyu Pan; Hailin Shi

cs.AIOct 13, 2026

Thinker-WM: A Unified World Model with Shared Multimodal Representations for Embodied Intelligence

Authors: Thinker Team, Peng Gao, Jichao Jiao, Chen Liao, Daqin Luo, Yiming Ma, Baiyu Pan, Hailin Shi,

Organizations: Ubtech

Abstract

World models have emerged as a promising paradigm for embodied AI, yet existing systems often remain fragmented in the modalities they model and the functions they support. In particular, visual dynamics, geometric structure, and executable action are commonly coupled through separate interfaces or downstream modules rather than being jointly represented and evolved within a shared dynamics model. In this work, we present Thinker-WM, a unified multimodal world model that jointly models and generates RGB, metric depth, and robot action through a shared intermediate Unified Dynamics Transformer with modality-specific encoders and decoders. The model performs multimodal unification at the representation level: appearance, geometry, and action are transformed into a common dynamic representation and jointly evolved by shared parameters, while their distinct output structures are preserved. The same parameterization supports multiple functions, including joint video-action generation, world prediction, policy inference, action-conditioned generation, video-to-action generation, and inverse dynamics. To examine whether the shared representation captures action-related information, we introduce a lightweight motion probe that shows the frozen visual representation contains recoverable motion information. Thinker-WM is trained on a heterogeneous corpus comprising approximately 20K hours of egocentric video, 10K hours of real-robot data, and 1K hours of simulation data, including approximately 4K hours of real-robot data with aligned depth supervision. Extensive experiments demonstrate strong performance across embodied world-model evaluations. Thinker-WM ranks first on EWMBench, WorldOlympiad and WorldArena, while controlled studies further demonstrate the benefits of action conditioning, geometric interaction, and multi-function training within the shared representation.

Figures & tables

Figure 1: Thinker-WM overview. Left: the pre-training mixture, which combines egocentric, robot, and simulation data. Middle: every modality is lifted into and decoded back from one unified representation by its own encoder and decoder, while a shared Unified Dynamic Transformer (UDT) models the future world within that representation using a single set of weights. Right: open-loop results on real-world tasks, where video, depth, and action are generated jointly by the same network.

Component	Blocks	Weight origin	Role
Visual encoder $\mathcal{E}_{v,d}$	8 blocks	Inherited	Encode RGB and physical-fidelity tokens.
RGB UDT $\mathcal{U}$	14 blocks	Inherited	Model appearance and interaction.
Physical-fidelity UDT $\mathcal{U}$	14 blocks	Inherited	Model geometry and interaction.
Visual decoder $\mathcal{D}_{v,d}$	8 blocks	Inherited	Decode RGB and physical-fidelity outputs.
Action encoder $\mathcal{E}_{a}$	8 blocks	Initialized	Encode action tokens.
Action decoder $\mathcal{D}_{a}$	8 blocks	Initialized	Decode executable actions.

Table 1: Structure and weight origin of the three-stage architecture and auxiliary branches. Pretrained visual blocks are inherited from Wan2.2 TI2V-5B and remain trainable, while the VAE and VLM are frozen.

Panel of Figure 2 — Figure 2: Multimodal encoding and decoding pipelines. RGB video, metric depth, and action are mapped into the unified latent space through modality-specific encoders and processed back into their respective output spaces through modality-specific decoders. The frozen Qwen3-VL-2B converts the structured prompt into the conditioning sequence $h_{\texttt{und}}$ , while robot identity is provided through learnable tokens $r$ . The decoder stage predicts the flow-matching velocity field for each modality rather than directly producing decoded samples.

Figure 3: Attention pattern of the UDT during pretraining. Self-attention provides cross-modal interaction among RGB, depth, and action, while cross-attention provides the dedicated task-conditioning path from the prompt representation. Depth interacts asymmetrically with the other generative streams during pretraining.

Figure 4: Attention pattern of the UDT during post-training. Compared with pretraining, the geometric stream is opened to bidirectional interaction with RGB and action, enabling multimodal refinement under reliable depth supervision.

Mode	Training Detail	Role
WAM	joint denoising of visual streams and action	joint generation
WM	action branch omitted	visual world model
VLA	first video frame kept clean	policy on the current observation
A2V	clean action injected, prompt dropped	action-conditioned visual generation
V2A	clean video injected, prompt dropped	video-conditioned action generation
Uncond	prompt dropped, joint denoising	unconditional branch for guidance

Table 2: Function-oriented training modes. Each training mode corresponds to one function of the shared model.

Figure 5: Function modes of Thinker-WM. Each panel shows the modality configuration of one function, with dashed outlines indicating omitted inputs or outputs.

Figure 6: Training data composition . For each category, two example episodes are shown with five video frames, their aligned depth labels, and the corresponding action curves; task text serves as the language record. The grids show additional episodes sampled from the training mixture. Depth is obtained by stereo matching for egocentric video, predicted by a monocular teacher for real robot video, and rendered exactly in simulation.

Figure 7: Depth data pipeline . We generate depth pseudo-labels for egocentric and robot videos, filter low-quality clips, and align the retained depth with RGB frames for world model training.

Figure 8: Overview of the Data Factory producer–consumer pipeline. Heterogeneous multimodal data are organized into episode-centric catalogs

E_{i}=(V_{i},S_{i},A_{i},L_{i},\Sigma_{i},\Lambda_{i})

and accessed lazily through a SQLite offset index. The model specifies each training sample as

q_{j}=(e_{j},I_{j}^{V},I_{j}^{S},I_{j}^{A},I_{j}^{L})

, including within-source window and modality selection, while source-level sampling controls the mixture of data sources across batches and ranks. The resulting samples are adapted, collated, and asynchronously prefetched into a rank-local ready-batch queue with occupancy

0\leq q_{t}\leq Q

, from which the training process consumes prepared batches. The effective pipeline throughput is bounded by

\lambda_{\mathrm{eff}}\leq\min(\lambda_{p},\lambda_{c})

, where

\lambda_{p}

and

\lambda_{c}

denote the producer and consumer rates.

Figure 9: Throughput matching between data preparation and model computation. Batches are labelled in arrival order; preparation stays ahead of consumption within each VAE reuse window, so the model never waits.

Figure 10: Load balancing across ranks in a synchronous run. Every row is one rank and every column one optimization step, and the function mode of a step is sampled with a generator seeded only by the step number, so a column carries the same mode on every rank and the same choice is reproduced after a restart. The legend names the six modes, which are sampled in the configured proportions.

Stage	Main data	Updated scope	Capability installed
0	internet-scale video	inherited	visual and world prior
1	robot data egocentric human video data from the previous stage	video pathway	robot-view video prediction
2	curated robot data curated egocentric human video data from the previous stage	all parameters	video-action coupling + geometry
3	target-robot data data from the previous stage	all parameters	target-robot specialization

Table 3: The four training stages. Each stage installs one or more capabilities while mixing in a fixed small fraction of the data from the previous stage. Stage 0 is the pretrained Wan2.2 model, used as it stands.

Figure 11: Overview of Motion Probe. (a) Offline visual motion teacher construction: SpatialTrackerV2 extracts 3D point trajectories from ground-truth RGB video, motion-aware filtering and median aggregation produce a robust visual center trajectory, and a per-window Sim(3) transform aligns it with the recorded end-effector motion. The aligned teacher is frozen and cached. (b) Probing the frozen UDT: a lightweight Motion Head predicts end-effector displacement from the shared video representation. The UDT weights remain frozen, and only the Motion Head is trained. Low probe error indicates that the shared representation encodes action-relevant motion.

Type	Model	SceneC $\uparrow$	HSD $\uparrow$	Dyn $\uparrow$	nDTW $\uparrow$	Diversity $\uparrow$	BLEU $\uparrow$	CLIP $\uparrow$	Logics $\uparrow$	Overall $\uparrow$
General video	Veo3	0.8415	0.2130	0.1932	0.1613	0.0221	0.2139	0.8965	0.9474	3.49
	Wan2.6	0.6712	0.2034	0.0900	0.1715	0.0502	0.1616	0.8743	1.0000	3.22
	Kling26	0.8211	0.3272	0.1822	0.3423	0.0173	0.2591	0.9014	1.0000	3.85
	LTX-2	0.7850	0.2076	0.1283	0.2443	0.0120	0.1425	0.8869	0.5000	3.01
	Sora2	0.8526	0.2807	0.3494	0.2754	0.0314	0.2466	0.9100	0.9474	3.89
Embodied world model	Cosmos	0.7963	0.2500	0.2052	0.2533	0.0803	0.1230	0.8458	0.7333	3.29

Table 4: Generation quality on EWMBench. Every dimension is normalized to

[0,1]

, the Overall column is the sum of the eight dimensions, and higher is better. Numbers for the baseline models are taken from Qwen-RobotWorld [ 40 ] , which reports them on the same evaluation set. The last block is our model.

Figure 12: Overall EWMBench score across the four training stages of Section 6 , with the final checkpoint of each stage as a point. The dashed lines are four reference baselines from Table 4 , namely Wan2.6, Sora2 as the strongest general video model, LVP, and the strongest overall, Qwen-RobotWorld.

Model	Physical	3D Cons.	CLIP	Interact.	All
Uranus	0.925	0.511	0.304	0.730	0.722
Cosmos-Predict-2.5	0.937	0.479	0.321	0.721	0.712
LingBot-World	0.949	0.393	0.314	0.710	0.684
Rolling Forcing	0.870	0.389	0.329	0.566	0.608
LongLive	0.857	0.472	0.327	0.470	0.600
Yume-1.5	0.851	0.288	0.312	0.624	0.588

Table 5: WorldOlympiad Evaluation Results on the Robotics Subdomain Leaderboard.

Figure 13: Official screenshot of the WorldOlympiad robotics leaderboard, where our proposed model ranks first.

	Metric	Ours	DexWorldEngine	BWM-Fast	UNIS	MotuBrain	Wan2.6	Cosmos-Predict2.5
Visual Quality	Image Quality	37.26	51.44	51.22	53.94	54.05	67.36	49.48
	Aesthetic Quality	37.65	40.96	40.15	40.79	41.02	44.40	34.14
	JEPA Similarity	93.94	86.71	97.87	90.60	96.88	72.57	90.65
Content Consistency	Subject Consistency	82.58	81.46	81.42	79.05	83.51	73.15	67.24
	Background Consistency	90.19	88.85	90.17	86.44	89.91	84.29	72.72
	Photometric Consistency	46.33	4.42	3.08	2.13	5.10	34.57	45.21

Table 6: WorldArena dimension-level scores under the 12-metric protocol.

Dimension	Joint	Action-conditioned	$\Delta$
Overall	4.66	5.03	$+0.37$
HSD	0.5638	0.7369	$+0.1731$
nDTW	0.7102	0.8736	$+0.1634$
Dyn	0.3630	0.3636	$+0.0006$

Table 7: Generation with the ground-truth action sequence as a condition, compared with joint generation on the same model. Numbers are averaged over the evaluated checkpoints, and the dimensions that compare motion against the ground truth are shown separately.

Method	AbsRel $\downarrow$	RMSE (m) $\downarrow$
RynnWorld-4D	0.1625	0.2075
Thinker-WM	0.1360	0.1790

Table 8: Generated depth accuracy on 48 AgiBot validation sequences. RMSE is measured in meters. Lower is better for both metrics.

Configuration	Interaction	6K	7K	8K	Overall
RGB-only	None	12/20	12/20	12/20	60.0%
RGB + Depth	One-way	12/20	12/20	14/20	63.3%
Thinker-WM	Bidirectional	15/20	15/20	15/20	75.0%

Table 9: Closed-loop success on the RoboTwin turn_switch task. Interaction denotes RGB–depth information flow. Each checkpoint is evaluated over 20 trials; overall success aggregates the three checkpoints.

Figure 14: Gradient geometry of the six training modes. Every mode is forced on its own pass over the same batches with a shared noise realization, and the pairwise cosine of the loss gradients with respect to the shared parameters is averaged over five checkpoints spanning the joint training stage. Left, the cosine matrix on the shared UDT, with the diagonal omitted. Right, the gradient norm per mode on the shared UDT, where the bar is the mean over the five checkpoints and the whisker covers their range. No pair is negative, and every auxiliary mode carries gradients of comparable magnitude in a direction that adds to the joint one.

Figure 15: Action MSE of a motion probe evaluated on frozen UDT video representations from different training checkpoints. Lower is better. Both plotted series exhibit a monotonic decrease in MSE, indicating that action-relevant motion information becomes increasingly recoverable from the learned representation.

Metric	Result
Teacher Quality
Frame-wise directional cosine similarity	0.65
Cumulative displacement cosine similarity	0.83
End-point error (m)	0.041
Trajectory-shape score	0.86
Path correlation	0.99

Table 10: Reliability of the offline visual motion teacher on the in-house test split. The upper block reports agreement between teacher-estimated and ground-truth end-effector motion. The lower block summarizes the pseudo-label filtering statistics. All retention and rejection percentages are computed relative to the full set of candidate windows.

Figure 16: Long-horizon generation on three recorded episodes. Each block samples the frames of the model’s rollout of one episode. Only the first frame of a rollout is a ground-truth observation; every later conditioning frame is produced by the model, so the $115.2$ seconds shown are generated without any test-time supervision.

Figure 17: Trajectory transfer. Top to bottom: the original execution of the scene, the recorded window whose end-effector trajectory is supplied as the condition, and the video generated from that trajectory. Frames are sampled at the times marked below.

Figure 18: The end-effector trajectory supplied as the condition, in the unified action space. The position panels are drawn at

\pm 0.25

and the gripper panels at

\pm 1.2

so that the motion is visible; the rotation panels keep the range of the training monitor,

\pm 1.1

Space	RMSE
Native reader space, 14 dimensions	0.0540
Unified action space, 20 dimensions	0.0293

Table 11: Accuracy of the action sequence recovered from a recorded window by the inverse mode, measured against the recorded trajectory.

Figure 19: Closed loop over a recorded window. Top to bottom: the original execution of the scene, the recorded window from which the action is recovered, and the video generated from the recovered action. The layout follows Figure 17 .

Figure 20: The action recovered from the recording by the inverse mode ( V2A , dashed) against the recorded trajectory ( Reference , solid) over the same window, in the unified action space.

Figure 21: Zero-shot generation from a hand-held frame that is outside the training distribution. The prompt card repeats the instruction that was supplied, of which the model only sees the Task field.

Figure 22: Zero-shot generation from the same conditioning frame as Figure 21 , under the instruction given in the prompt card.

Figure 23: Zero-shot manipulation across diverse scenes. Trained only on RoboTwin clean demonstrations without viewpoint variation, the model follows new instructions to press an alarm clock (top) and grasp a roller (bottom) under different backgrounds and a $65^{\circ}$ camera view, without fine-tuning. Each row shows nine keyframes combining head and wrist camera views.

Method	Clean $\uparrow$	Randomized $\uparrow$	Drop $\downarrow$
Thinker-WM (Ours)	92.6	91.4	1.2

Table 12: Manipulation success rates (%) on RoboTwin 2.0 across 50 tasks.

World Models for Robotics

World models for robotics predict how scenes and robot states change under actions to support planning, control, and learning.

See all 384 papers

Jun 5, 2026

Dreaming when Necessary: Advancing World Action Models with Adaptive Multi-Modal Reasoning

World Action Models (WAMs) offer a promising approach to embodied intelligence, yet existing methods rely heavily on video prediction as action priors and lack adaptive multimodal reasoning, limiting their effectiveness on long-horizon, complex tasks. We observe that WAMs require different multimodal reasoning modes under different execution contexts: textual reasoning is essential during task transitions to guide high-level action prediction, while visual reasoning is critical during fine-grained manipulation for precise control. Motivated by this observation, we propose AdaWAM, a world action model with adaptive multimodal reasoning abilities. AdaWAM integrates a lightweight dynamic router that autonomously triggers textual or visual reasoning as needed during task execution. Experiments on both simulated and real-world embodied tasks show that AdaWAM substantially improves inference efficiency while outperforming state-of-the-art embodied policies. Codes and demos are available at: https://adawam.github.io/.

Yinzhou Tang, Jingbo Xu, Yu Shang +4

Tsinghua University · Manifold AI

Efficient Multimodal Inference World Models for Robotics Multimodal Reasoning World Action Models

Ask PDF

Oct 13, 2026

UniW(A)M: A Unified Mixture-of-Experts Framework for World and World-Action Models

World models (WMs) and world-action models (WAMs) focus on future world prediction and executable action generation, respectively. Although these models learn shared physical dynamics and interaction semantics from embodied trajectories, they are typically trained in isolation, which restricts knowledge reuse between prediction and control. The central challenge is therefore to develop a unified learning framework that shares these dynamics while preserving the distinct modeling capabilities required by each paradigm. We propose UniW(A)M, a mixture-of-experts world-(action) model built upon diffusion Transformers that unifies action-conditioned world modeling and diverse WAM variants. Shared experts capture common dynamics, while atomic capability experts are reused across interfaces so that each interface keeps its own conditional pathway. At deployment, a plug-in adaptive interface router (PAIR) learns from a small subset of target-task chunk scenarios to select a suitable WAM interface for each execution chunk, balancing task success and execution efficiency while keeping the backbone frozen. We evaluate UniW(A)M on LIBERO, RoboTwin 2.0, and three real-world manipulation tasks, where it achieves 88.3% average real-world success versus 76.7% for the strongest baseline. On WorldArena, it leads action-conditioned world modeling on all three tasks. AC-WM co-training contributes to action generation on every action interface and PAIR raises OOD success from 63.3% to 76.7%.

Jingbo Xu, Jianjie Fang, Zhuohang Li +9

Tsinghua University · Zhongguancun Academy · East China Normal University +1

World Models for Robotics World Action Models Embodied World Models Action-Conditioned World Models Mixture-of-Experts Models

Ask PDF

May 12, 2026

World Action Models: The Next Frontier in Embodied AI

Vision-Language-Action (VLA) models have achieved strong semantic generalization for embodied policy learning, yet they learn reactive observation-to-action mappings without explicitly modeling how the physical world evolves under intervention. A growing body of work addresses this limitation by integrating world models, predictive models of environment dynamics, into the action generation pipeline. We term this emerging paradigm World Action Models (WAMs): embodied foundation models that unify predictive state modeling with action generation, targeting a joint distribution over future states and actions rather than actions alone. However, the literature remains fragmented across architectures, learning objectives, and application scenarios, lacking a unified conceptual framework. We formally define WAMs and disambiguate them from related concepts, and trace the foundations and early integration of VLA and world model research that gave rise to this paradigm. We organize existing methods into a structured taxonomy of Cascaded and Joint WAMs, with further subdivision by generation modality, conditioning mechanism, and action decoding strategy. We systematically analyze the data ecosystem fueling WAMs development, spanning robot teleoperation, portable human demonstrations, simulation, and internet-scale egocentric video, and synthesize emerging evaluation protocols organized around visual fidelity, physical commonsense, and action plausibility. Overall, this survey provides the first systematic account of the WAMs landscape, clarifies key architectural paradigms and their trade-offs, and identifies open challenges and future opportunities for this rapidly evolving field.

Siyin Wang, Junhao Shi, Zhaoyang Fu +11

1Fudan University · 2Shanghai Innovation Institute · 3National University of Singapore

Vision-Language-Action Models World Action Models World Model Evaluation Embodied World Models World Models for VLA Models

Ask PDF

Abstract

World Models for Robotics

Explore similar work

Dreaming when Necessary: Advancing World Action Models with Adaptive Multi-Modal Reasoning

UniW(A)M: A Unified Mixture-of-Experts Framework for World and World-Action Models

World Action Models: The Next Frontier in Embodied AI