Generative world models offer a compelling foundation for augmented-reality (AR) applications: by predicting future image sequences that incorporate deliberate visual edits, they enable temporally coherent, augmented future frames that can be computed ahead of time and cached, avoiding per-frame rendering from scratch in real time. In this work, we present SEGAR, a preliminary framework that combines a diffusion-based world model with a selective correction stage to support this vision. The world model generates augmented future frames with region-specific edits while preserving others, and the correction stage subsequently aligns safety-critical regions with real-world observations while preserving intended augmentations elsewhere. We demonstrate this pipeline in driving scenarios as a representative setting where semantic region structure is well defined and real-world feedback is readily available. We view this as an early step toward generative world models as practical AR infrastructure, where future frames can be generated, cached, and selectively corrected on demand.
Figures & tables
Figure 1: SEGAR system pipeline overview. In Stage I, we train a Vista-based generative stylizer to take three condition frames ( t∈[1,3] ) and output future frames with desired augmented edits ( t∈[4,12] ). In Stage II, the generative stylizer finetuned with LoRA takes the augmented future frame and the corresponding real-world observation as inputs, and outputs a corrected frame in which safety-critical regions are aligned with reality while augmented edits are preserved. In this example, a car that appears in frames 10–12 is absent from the initial prediction because it was not visible in the three condition frames. The corrected output faithfully restores the car while retaining the augmented appearance of non-critical regions.
Figure 2: Given an input image sequence, we compute inpainting regions using semantic segmentation. The resulting masks guide VACE’s inpainting process to augment static scene elements into a Tokyo-style appearance.
Figure 3: Example training target for Stage I training. The three frames in the top row are condition frames, whose clean latents are injected into the denoising process via dynamic prior injection [ 4 ] . The remaining nine future frames are generated using VACE with desired visual augmentations.
Figure 4: (Left) Real-world image. (Middle) Semantic segmentation of safety-critical regions. (Right) The Buffer zone is introduced by dilating the safety-critical mask, shown in black. Loss is not computed in the buffer zone.
Figure 5: Visualization of latent representations for the real observation before (top) and after (bottom) masking (masked regions inpainted in gray), showing that spatial structure is preserved in the latent space (right) and validating the use of mask downsampling for region-specific loss computation.
Figure 6: Side-by-side comparison for a sequence of future predictions and their corresponding corrected images.
Critical
Augmented
Real vs Corr.
Real vs Aug.
Aug. vs Corr.
SSIM ↑
0.9431
0.7698
0.8662
LPIPS ↓
0.2854
0.3968
0.1295
Table 1: Similarity metrics across safety-critical (road) and augmentation (building) regions. Real vs. Aug. serves as the baseline drift introduced by Stage I. Stage II substantially improves alignment with reality in safety-critical regions (SSIM 0.770→0.943 , LPIPS 0.397→0.285 ) while preserving the intended augmentation elsewhere (SSIM 0.866 , LPIPS 0.130 ).
Figure 7: Side-by-side comparison shows that our model successfully corrects the semantically important region while preserving edits in various scenarios (pedestrians, road signs, other vehicles, etc.).
Figure 8: More demos showing how the correction stage corrects safety-critical regions of the image, such as cars or pedestrians.
Figure 9: Failure case illustrating object-level inconsistency at region boundaries.
What if a world simulation model could render not an imagined environment but a city that actually exists? Prior generative world models synthesize visually plausible yet artificial environments by imagining all content. We present Seoul World Model (SWM), a city-scale world model grounded in the real city of Seoul. SWM anchors autoregressive video generation through retrieval-augmented conditioning on nearby street-view images. However, this design introduces several challenges, including temporal misalignment between retrieved references and the dynamic target scene, limited trajectory diversity and data sparsity from vehicle-mounted captures at sparse intervals. We address these challenges through cross-temporal pairing, a large-scale synthetic dataset enabling diverse camera trajectories, and a view interpolation pipeline that synthesizes coherent training videos from sparse street-view images. We further introduce a Virtual Lookahead Sink to stabilize long-horizon generation by continuously re-grounding each chunk to a retrieved image at a future location. We evaluate SWM against recent video world models across three cities: Seoul, Busan, and Ann Arbor. SWM outperforms existing methods in generating spatially faithful, temporally consistent, long-horizon videos grounded in actual urban environments over trajectories reaching hundreds of meters, while supporting diverse camera movements and text-prompted scenario variations.
Spatial augmented reality (SAR) directly projects digital content onto physical scenes using projectors, creating immersive experience without head-mounted displays. However, for SAR to support intelligent interaction, such as reasoning about the scene or answering user queries, it must semantically distinguish between the physical scene and the projected content. Standard Vision Language Models (VLMs) struggle with this virtual-physical ambiguity, often confusing the two contexts. To address this issue, we introduce ProCap, a novel framework that explicitly decouples projected content from physical scenes. ProCap employs a two-stage pipeline: first it visually isolates virtual and physical layers via automated segmentation; then it uses region-aware retrieval to avoid ambiguous semantic context due to projection distortion. To support this, we present RGBP (RGB + Projections), the first large-scale SAR semantic benchmark dataset, featuring 65 diverse physical scenes and over 180,000 projections with dense, decoupled annotations. Finally, we establish a dual-captioning evaluation protocol using task-specific tokens to assess physical scene and projection descriptions independently. Our experiments show that ProCap provides a robust semantic foundation for future SAR research. The source code, pre-trained models and the RGBP dataset are available on the project page: https://ZimoCao.github.io/ProCap/.
Game worlds have traditionally been built through labor-intensive production pipelines, making them costly to develop, difficult to customization, and expensive to modify after deployment. Recent advances in video world models offer a fundamentally different paradigm. Rather than explicitly authoring every component of a virtual environment, these models autoregressively synthesize future observations conditioned on the current world state and user interactions, enabling playable worlds to be generated online. Trained on both gameplay recordings and real-world videos, they can capture diverse visual appearances and physical dynamics, opening new opportunities for interactive applications beyond gaming, including embodied intelligence. In this paper, we present AlayaWorld, a full-stack open-source framework for building interactive generative worlds. AlayaWorld enables open-ended real-time interaction, allowing users to freely navigate and perform diverse actions such as combat, spell casting, and monster summoning. The framework unifies the complete development-from data preparation model architecture, model training, inference acceleration, and deployment-within a modular and extensible architecture. Alongside the framework, we release reproducible pipelines, reference implementations, evaluation tools, and comprehensive documentation, establishing a practical foundation for future research and real-time applications of generative world models.