cs.CVMar 3, 2026
SaveShareVerse: Multi-Agent Consistent Video Generation for Shared World Modeling
Organizations: Shanghai Jiao Tong University, China · Fudan University, China · StepFun, China
Abstract
This paper presents ShareVerse, a video generation framework enabling multi-agent shared world modeling, addressing the gap in existing works that lack support for unified shared world construction with multi-agent interaction. ShareVerse leverages the generation capability of large video models and integrates three key innovations: 1) A dataset for large-scale multi-agent interactive world modeling is built on the CARLA simulation platform, featuring diverse scenes, weather conditions, and interactive trajectories with paired multi-view videos (front/ rear/ left/ right views per agent) and camera data. 2) We propose a spatial concatenation strategy for four-view videos of independent agents to model a broader environment and to ensure internal multi-view geometric consistency. 3) We integrate cross-agent attention blocks into the pretrained video model, which enable interactive transmission of spatial-temporal information across agents, guaranteeing shared world consistency in overlapping regions and reasonable generation in non-overlapping regions. ShareVerse, which supports 49-frame large-scale video generation, accurately perceives the position of dynamic agents and achieves consistent shared world modeling.
Figures & tables
Figure 1. ShareVerse: Collaborative Video Generation for Shared World Modeling. ShareVerse empowers distributed agents to collaboratively synthesize a globally consistent virtual environment. We bridge isolated generative priors through two core mechanisms: (1) implicit cross-agent interaction, which resolves visual conflicts during concurrent exploration (red/blue vehicles); and (2) a global Spatiotemporal Memory Cache, which guarantees long-term environmental permanence during asynchronous revisitation (green vehicle).
Figure 2. Illustration of the dataset construction process. We build the multi-agent synchronized training dataset by rendering in CARLA. We equip each agent with four cameras (front, rear, left, and right) and choose six main trajectory pairs, as shown in (a) – (f). Each pair enables interaction between agents.
Figure 3. Method overview. Given one image from each of two agents, where four views are concatenated to depict a scene, ShareVerse performs a prediction task to generate future videos conditioned on the camera trajectories from users. In the generation process, two agents explore the world, exchange captured visual information, and perceive each other’s positions.
Figure 4. Pipeline overview. By retrieving memory frames from memory cache and concentrating them with frames to be generated to guide generation, our approach ensures long-term consistency in shared world simulation.
Figure 5. Evaluation of cross-agent visual consistency. (a-e) Generated 4-view video frames and zoomed-in local crops are compared across baselines and our method. The red and blue boxes denote the identical architectural structure observed by Agent 1 (left) and Agent 2 (right), respectively. ShareVerse accurately preserves structural integrity across distinct viewports and closely aligns with the Ground Truth. (f) Quantitative metrics further confirm that ShareVerse consistently outperforms baselines across PSNR, SSIM, and LPIPS.
| Methods | Mean Error | Mean Completeness |
|---|---|---|
| Standard | 3.3174 | 0.3935 |
| Aether | 2.6217 | 0.5404 |
| Syncammaster | 2.2783 | 0.5713 |
| Ours | 1.6286 | 0.6206 |
Table 1. Quantitative analysis of cross-agent consistency
| Method | PSNR | SSIM | LPIPS |
|---|---|---|---|
| Full sequence | |||
| Front view only | 20.51 | 0.6762 | 0.3230 |
| Four views concat | 22.28 | 0.6862 | 0.2258 |
| w/o cross-agent attn | 20.42 | 0.6304 | 0.2935 |
| Full sequence (Ours) | 21.76 | 0.6637 | 0.2570 |
| Autoregressive | |||
Table 2. Ablation Study on reference-based metrics.
Figure 6. The visualization of the memory mechanism’s effect.
Figure 7. Qualitative analysis of spatiotemporal and geometric consistency across diverse interactive trajectories. We evaluate ShareVerse under six representative two-agent scenarios. For each scenario panel, we visualize the agent trajectories (top-left), the generated 4-view video frames (top-right), and the 3D point clouds reconstructed independently from each agent (bottom). Within the generated frame grids, color-coded dashed boxes denote identical physical regions observed from disparate viewing angles. For example, in the opposing straight paths scenario (a), the dashed boxes highlight specific storefront facades, illustrating that both agents render identical street views despite facing opposite directions. Similarly, red and blue dashed boxes in the 3D point clouds highlight the structural consistency of building geometries across viewports. These 3D reconstructions accurately reflect the architectural details found in the generated 2D video frames, while the point clouds from the two agents exhibit an accurate geometric alignment with each other. Notably, even in the highly challenging X-junction crossing (f), where spatial overlap is transient and complex, our model maintains an accurate shared world. This is directly evidenced by the 3D reconstruction of the corner brown building, where both agents independently construct the identical geometric shape of the architecture from completely different approach angles.
Figure 8. Front-view visualization for different training paradigms. We compare the results of agents trained solely on front-view videos and agents trained on 4-view videos. The frames are sampled by the front view of two agents at different timesteps, but the overlap regions (gray shadow), which are enclosed by the red and blue dashed boxes on images, are supposed to be identical. Obviously, agents with 1 view generate visual hallucinations within overlap regions while agents with 4 view achieve consistency.
Figure 9. Panoramic visualization for analyzing single-agent view coherence. For each example, we display the generated 4-view video frames (left) and the corresponding 360-degree panorama stitched from these frames (right). By projecting the four distinct viewports onto a panoramic canvas using the exact camera parameters, it is evident that the architectural facades and distant horizons exhibit remarkable seamlessness across camera boundaries.
Figure 10. Qualitative comparison of multi-agent dynamic object consistency during mutual visibility. Each row displays the generated video frames from the specific camera views where the two agents observe each other at the intersection moment. The Ground Truth results are displayed as a reference (row 5). The comparison focuses on whether interacting agents can faithfully perceive each other. Crucially, ShareVerse (row 4) accurately renders the specific red vehicle (the other agent) with high visual fidelity in the correct spatial location, whereas other baselines fail to synthesize the dynamic object consistently. Note that while the generated frames differ slightly from the Ground Truth reference due to inherent stochasticity in video generation, ShareVerse still maintains superior frame quality and cross-agent consistency.
Figure 11. More visual results compared with baselines.