This paper presents ShareVerse, a video generation framework enabling multi-agent shared world modeling, addressing the gap in existing works that lack support for unified shared world construction with multi-agent interaction. ShareVerse leverages the generation capability of large video models and integrates three key innovations: 1) A dataset for large-scale multi-agent interactive world modeling is built on the CARLA simulation platform, featuring diverse scenes, weather conditions, and interactive trajectories with paired multi-view videos (front/ rear/ left/ right views per agent) and camera data. 2) We propose a spatial concatenation strategy for four-view videos of independent agents to model a broader environment and to ensure internal multi-view geometric consistency. 3) We integrate cross-agent attention blocks into the pretrained video model, which enable interactive transmission of spatial-temporal information across agents, guaranteeing shared world consistency in overlapping regions and reasonable generation in non-overlapping regions. ShareVerse, which supports 49-frame large-scale video generation, accurately perceives the position of dynamic agents and achieves consistent shared world modeling.
Figures & tables
Figure 1. ShareVerse: Collaborative Video Generation for Shared World Modeling. ShareVerse empowers distributed agents to collaboratively synthesize a globally consistent virtual environment. We bridge isolated generative priors through two core mechanisms: (1) implicit cross-agent interaction, which resolves visual conflicts during concurrent exploration (red/blue vehicles); and (2) a global Spatiotemporal Memory Cache, which guarantees long-term environmental permanence during asynchronous revisitation (green vehicle).
Figure 2. Illustration of the dataset construction process. We build the multi-agent synchronized training dataset by rendering in CARLA. We equip each agent with four cameras (front, rear, left, and right) and choose six main trajectory pairs, as shown in (a) – (f). Each pair enables interaction between agents.
Figure 3. Method overview. Given one image from each of two agents, where four views are concatenated to depict a scene, ShareVerse performs a prediction task to generate future videos conditioned on the camera trajectories from users. In the generation process, two agents explore the world, exchange captured visual information, and perceive each other’s positions.
Figure 4. Pipeline overview. By retrieving memory frames from memory cache and concentrating them with frames to be generated to guide generation, our approach ensures long-term consistency in shared world simulation.
Figure 5. Evaluation of cross-agent visual consistency. (a-e) Generated 4-view video frames and zoomed-in local crops are compared across baselines and our method. The red and blue boxes denote the identical architectural structure observed by Agent 1 (left) and Agent 2 (right), respectively. ShareVerse accurately preserves structural integrity across distinct viewports and closely aligns with the Ground Truth. (f) Quantitative metrics further confirm that ShareVerse consistently outperforms baselines across PSNR, SSIM, and LPIPS.
Methods
Mean Error
Mean Completeness
Standard
3.3174
0.3935
Aether
2.6217
0.5404
Syncammaster
2.2783
0.5713
Ours
1.6286
0.6206
Table 1. Quantitative analysis of cross-agent consistency
Method
PSNR ↑
SSIM ↑
LPIPS ↓
Full sequence
Front view only
20.51
0.6762
0.3230
Four views concat
22.28
0.6862
0.2258
w/o cross-agent attn
20.42
0.6304
0.2935
Full sequence (Ours)
21.76
0.6637
0.2570
Autoregressive
Table 2. Ablation Study on reference-based metrics.
Figure 6. The visualization of the memory mechanism’s effect.
Figure 7. Qualitative analysis of spatiotemporal and geometric consistency across diverse interactive trajectories. We evaluate ShareVerse under six representative two-agent scenarios. For each scenario panel, we visualize the agent trajectories (top-left), the generated 4-view video frames (top-right), and the 3D point clouds reconstructed independently from each agent (bottom). Within the generated frame grids, color-coded dashed boxes denote identical physical regions observed from disparate viewing angles. For example, in the opposing straight paths scenario (a), the dashed boxes highlight specific storefront facades, illustrating that both agents render identical street views despite facing opposite directions. Similarly, red and blue dashed boxes in the 3D point clouds highlight the structural consistency of building geometries across viewports. These 3D reconstructions accurately reflect the architectural details found in the generated 2D video frames, while the point clouds from the two agents exhibit an accurate geometric alignment with each other. Notably, even in the highly challenging X-junction crossing (f), where spatial overlap is transient and complex, our model maintains an accurate shared world. This is directly evidenced by the 3D reconstruction of the corner brown building, where both agents independently construct the identical geometric shape of the architecture from completely different approach angles.
Figure 8. Front-view visualization for different training paradigms. We compare the results of agents trained solely on front-view videos and agents trained on 4-view videos. The frames are sampled by the front view of two agents at different timesteps, but the overlap regions (gray shadow), which are enclosed by the red and blue dashed boxes on images, are supposed to be identical. Obviously, agents with 1 view generate visual hallucinations within overlap regions while agents with 4 view achieve consistency.
Figure 9. Panoramic visualization for analyzing single-agent view coherence. For each example, we display the generated 4-view video frames (left) and the corresponding 360-degree panorama stitched from these frames (right). By projecting the four distinct viewports onto a panoramic canvas using the exact camera parameters, it is evident that the architectural facades and distant horizons exhibit remarkable seamlessness across camera boundaries.
Figure 10. Qualitative comparison of multi-agent dynamic object consistency during mutual visibility. Each row displays the generated video frames from the specific camera views where the two agents observe each other at the intersection moment. The Ground Truth results are displayed as a reference (row 5). The comparison focuses on whether interacting agents can faithfully perceive each other. Crucially, ShareVerse (row 4) accurately renders the specific red vehicle (the other agent) with high visual fidelity in the correct spatial location, whereas other baselines fail to synthesize the dynamic object consistently. Note that while the generated frames differ slightly from the Ground Truth reference due to inherent stochasticity in video generation, ShareVerse still maintains superior frame quality and cross-agent consistency.
Figure 11. More visual results compared with baselines.
Autoregressive video world models enable temporally coherent generation for a single observer. Extending them to multiple agents requires consistency across independently controlled views and temporal gaps under causal streaming. We present ConsistWorld, a multi-agent world model that generates camera-controlled video streams of a static scene from one shared image. We formulate consistency as routing evidence from committed multi-agent history and concurrently generated peer views to the tokens being generated. Pose Conditioned Memory Retrieval selects relevant historical observations from all agents, recovering evidence beyond the recent context window. Visibility-Gated Peer Sharing regulates current peer information according to estimated historical coverage and current-view overlap. Together, they determine which historical observations enter the context and where concurrent peer information contributes, supporting long-term recall and coordinated exploration. Both mechanisms use camera geometry and maintain a bounded active context for a fixed agent count and retrieval budget. Experiments on evidence sharing cases and video length and agent number generalizations show that ConsistWorld achieves a strong cross-time and cross-agent consistency while preserving competitive generation quality.
Qianxun Xu, Xianfang Zeng, Xinyao Liao +3
Westlake University · StepFun · University of California, Los Angeles
Video world models are a foundational generative technology for embodied AI and the Metaverse, yet existing approaches are inherently limited to a single agent observing from a single perspective. Extending these models to multi-agent settings introduces two critical challenges: data scarcity (coordinated multi-view recordings are prohibitively expensive to collect for general open-domain scenarios) and world state alignment (independently generated video streams cannot ensure that shared physical environments and events evolve consistently across views). To address these challenges, we propose MetaWorld, a novel framework that scales multi-agent video world models to open-domain environments directly from single-view videos. First, we introduce Monocular World-State Unrolling (MWSU) to explicitly decompose monocular footage into the camera operator's ego-motion and the visible subject's spatial trajectory. This camera-trajectory decomposition naturally extracts synchronized multi-agent motion data within a shared 3D space, completely bypassing the need for multi-camera setups. Second, for precise visual control, we develop the Subject-Aware World Generator to enable appearance-driven simulation conditioned on per-agent identity images. Finally, to ensure both views are grounded in the identical physical reality, we propose World-State Alignment, a per-frame inter-branch cross-attention mechanism inserted at every transformer layer of the video DiT. By jointly synchronizing the denoising process, WSA enforces both static geometric consistency and dynamic motion consistency, encouraging that the shared 3D environment and physical events remain well-aligned across both egocentric views. Extensive experiments demonstrate that MetaWorld achieves superior cross-view consistency and identity fidelity, establishing a highly scalable, physics-driven paradigm for multi-agent video world modeling.
Teng Hu, Mingchun Lu, Yating Wang +6
Shanghai Jiao Tong University · Zhejiang University · Nanyang Technological University
Video world models have achieved remarkable success in simulating environmental dynamics in response to actions by users or agents. They are modeled as action-conditioned video generation models that take historical frames and current actions as input to predict future frames. Yet, most existing approaches are limited to single-agent scenarios and fail to capture the complex interactions inherent in real-world multi-agent systems. We present MultiWorld, a unified framework for multi-agent multi-view world modeling that enables accurate control of multiple agents while maintaining multi-view consistency. We introduce the Multi-Agent Condition Module to achieve precise multi-agent controllability, and the Global State Encoder to ensure coherent observations across different views. MultiWorld supports flexible scaling of agent and view counts, and synthesizes different views in parallel for high efficiency. Experiments on multi-player game environments and multi-robot manipulation tasks demonstrate that MultiWorld outperforms baselines in video fidelity, action-following ability, and multi-view consistency. Project page: https://multi-world.github.io/