cs.CVAug 16, 2026

WorldRover: A Scalable Synthetic Video Data Engine for World Exploration with Rich Annotations

Authors: Xiaojie Xu, Zhengyuan Lin, Runyi Li, Yihao Liu, Kaipeng Zhang, Yongtao Ge

Organizations: Alaya Lab · The University of Tokyo

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Learning to generate or reconstruct explorable worlds requires video paired with more than RGB: camera motion, scene geometry, temporal correspondence and, for interactive models, control signals. Real capture can provide some of these signals, but dense geometry and long-range correspondence usually rely on estimation or specialised instrumentation. Rendering provides these quantities directly, yet existing synthetic resources rarely combine them on the same frames while also supporting controlled changes of viewpoint and appearance. We introduce WorldRover, a data engine for generating richly annotated, long-range explorations of artist-built environments. At its core, WorldRover-Engine is an Unreal Engine pipeline that executes and offline-renders minute-scale routes while preserving their full trajectories and scene geometry. The same exploration can be replayed from first-person, third-person, and 360 panoramic cameras under different environmental states. Using WorldRover-Engine, we construct WorldRover-10M, whose sequences pair RGB with metric depth, camera trajectories, and trajectory-derived action signals throughout each exploration. Third-person subsets additionally provide dense optical flow, long-range 2D/3D point tracks with visibility, and a character trajectory distinct from the camera trajectory. The engine can render a traversal from first-person, third-person and 360 panoramic viewpoints, under different environmental states or with a neutral white material, while preserving the route and scene geometry. WorldRover therefore turns long-horizon world exploration into a scalable data-generation problem, providing supervision for models that must build, maintain, and revisit coherent representations of an explorable world.

Figures & tables

Figure 1: WorldRover-10M at a glance. Four rows, top to bottom. Scene and routes : one UE environment with three routes drawn as camera frustums, and four frames along each, colour-keyed to the overview. (1) Diverse viewpoints : the same instants observed as first-person, third-person and 360∘360^{\circ} panoramic video; every panel is RGB with metric depth inset, released with the camera that produced it. Third-person adds the character’s own trajectory and the trajectory-derived action stream. (2) Diverse scene styles : the first-person view re-rendered under a different weather state, a different lighting state, and as an untextured white model, with scene, motion and camera held fixed. (3) Diverse motion labels : dense optical flow and long-range point tracks, shown in 2D and in world space; a track stores a 3D position, its 2D projection and visibility per frame. Third-person only, with independent body and camera motion.
Figure 2: WorldRover-Engine overview. Asset processing runs once per scene on a Windows editor host; content generation runs once per sequence, headless, on a multi-GPU Linux host. The two halves meet at a versioned on-disk hand-off, so a campaign is replayable without an editor session.
Figure 3: Trajectory generation. The three strategies for producing walking trajectories, on one scene. (1) NavMesh planner : a route ( orange ) planned over a baked navigation mesh ( green ); the mesh is built once and reused, so planning is fast and coverage broad, at the cost of per-scene tuning. (2) Reactive explorer : a path ( blue ) driven by local obstacle sensing ( red ); it needs no precomputed map, but senses step by step and can walk itself into a corner. (3) Manual recording : a human-driven path ( purple ), which yields a specific trajectory on demand at the cost of manual effort. Together they trade off automation, coverage and control.
Figure 4: Third-person control semantics. Centre: the start state. W/A/S/D (black) translate the character along camera-relative directions — W away from the camera, S toward it, A and D to its left and right — and the character turns to face its heading while the camera follows with a slowly varying orbit azimuth. L/R (blue) yaw the camera about a stationary character, so the character holds its image position while the scene rotates around it.
Figure 5: Panoramic assembly. (a) Six cube faces at 100∘100^{\circ} FOV, giving a 10∘10^{\circ} overlap between neighbours, rendered with tone mapping disabled so all faces share one radiometric scale. (b) Reprojection onto an equirectangular image; dashed outlines, colour-matched to (a), mark each face’s region and the yellow bands its overlap, which exposes the pincushion distortion of the side faces and the polar caps. (c) A cos-weighted feather blend merges the overlaps into a linear-HDR panorama. (d) Exposure is estimated with latitude-aware luminance weighting and smoothed across frames; local base/detail tone mapping compresses the result before sRGB conversion.
Figure 6: Point-track ground truth. Six frames 44 s apart, time encoded magenta →\rightarrow cyan; each curve follows one 3D point on the character. Filled markers and solid curves are visible, hollow rings and dashed curves occluded and drawn at their amodal ground-truth position. In the fifth panel a wall encloses the camera and hides the character completely, yet the tracks retain their amodal positions with zero visibility flags.
Data, when presentContents
RGB videoH.264, hardware-encoded
Depth16-bit log-quantized, FFV1 lossless, ++ decode sidecar
Optical flow16-bit affine-quantized, FFV1 lossless, ++ decode sidecar (third-person only)
Point tracksper-track 3D position, 2D projection and visibility, per frame (third-person only)
Camera trajectoryper-frame pose and intrinsics of the rendered camera
Character trajectoryper-frame pose and speed (third-person only)
Table 1: Released data format. How each item is stored when a subset includes it.
Figure 7: A sample of the WorldRover scene library. Fifteen of 30+ UE environments — waterfront city, transit interior, cathedral, alpine meadow, cyberpunk street, office interior, Paris street, suburban home, jungle ruin, and more — all rendered under dynamic Lumen global illumination.
Figure 8: Diverse appearances. The same first-person frame of med_village , rendered from an identical camera at the same instant under five environmental states. Because lighting is simulated at render time rather than baked, indirect illumination, colour bleeding and shadow softness respond correctly to each state; each state uses its own calibrated fixed exposure, with no automatic adjustment during rendering, so Night stays dark rather than normalised toward mid-grey. Re-rendering an instant under another state therefore changes the image physically, not by recolouring a fixed frame.
Figure 9: White model. Three environments, the ordinary render above and the white-model re-render of the same frame below. With scene surfaces drawn in the engine’s default material, geometry and occlusion remain visible from large structures to fine detail — arcade columns and planters, station ironwork, individual leaves and branches — while texture, colour, decals and emissive light do not: the tiled floor flattens, the screens on the ticket machines go blank, and the burning wreck in the clearing is left as geometry alone.
Figure 10: Character library. Asset-review sheets — the avatars shown in one shared scene at a fixed pose — rather than frames from the released dataset. Third-person sequences are rendered with humanoids, quadrupeds and birds, which differ in body plan, gait and how they turn, so subject articulation is a diversity axis in its own right.
Engine/library capacity
Environments30+ artist-built UE scenes, interior to city scale
Lighting statesday, dawn, evening, sunset, night
Weather statesclear, overcast, fog, snow
Materialstextured render or untextured white model
Characters70+ animated humanoids, animals and creatures
Routesnavmesh planner, reactive explorer, manual recording
Table 2: WorldRover-Engine capacity and the current WorldRover-10M release. The upper block describes the engine and asset library, not the coverage of every released subset. In the lower block, scenes are counted per viewpoint and therefore do not add; 32 is the number of distinct environments.
Figure 11: Distributions over the released sequences ( n=6003n=6003 , from per-sequence metadata). Left: sequence duration, stacked by trajectory strategy in the colours of Figure 3 — reactive routes end sooner than planned ones, which run on to the long tail. Centre: distance travelled, a median of 122122 m. Right: mean speed per sequence, which falls into three clusters because the motion model samples one of three walking speeds; each cluster sits a little below its setting, the cost of turning and rate limiting.
More in this topic

Synthetic Data Generation

Synthetic data generation creates artificial examples, simulations, and datasets to train, test, or protect machine-learning systems.

See all 625 papers

Explore similar work

CardsList
Aug 10, 2026

Sekai2: From World Exploration to Interactive World Modeling

Video world models must capture how scenes evolve over time and across viewpoints. Training them for long-horizon generation and camera control therefore benefits from long videos paired with camera trajectories and temporally grounded semantics. Existing corpora rarely offer the three together: large-scale web video provides broad visual diversity but no trajectories or time-aligned text, while pose-annotated datasets are typically short-range or reconstruction-oriented. We introduce Sekai2, a multi-source real-world video dataset that carries the world-exploration footage of Sekai toward interactive world modeling. The release contains 128,892 clips totaling 2,826 hours from 10,428 source videos across 113 countries or regions, and is deliberately weighted toward sustained observation: under a common 120-second decomposition, 43,594 segments reach the full two minutes and account for 51.4% of all footage. Every clip includes a released camera trajectory and hierarchical annotations disentangling subject motion, environment dynamics, static scene content, and camera behavior, resulting in 649,597 temporally grounded segments. Crucially, we further introduce 982 panoramic sequences captured along non-linear trajectories with loops and revisits. These revisits provide repeated observations of the same locations across time and viewpoints, offering essential supervision for learning persistent scene representations, long-term spatial memory, and geometrically consistent world models. Corpus-scale analyses demonstrate complete pose-and-caption coverage, broad geographic and semantic diversity, varied camera trajectories, and highly non-redundant temporal descriptions. Together, these properties make Sekai2 a scalable resource for long-horizon video generation, camera-controllable synthesis, and interactive world-model pre-training.

Kang He, Wenshuo Peng, Zihui Gao +3

1Alaya Lab · 2Shanghai Innovation Institute · 3Wuhan University +1

Camera-Controlled Video GenerationLong-Horizon Video GenerationInteractive World ModelsVideo World Models
AskPDF
Sep 10, 2026

World in World: Explore the World with World Models

Autoregressive video world models enable interactive, long-horizon exploration, but flexible control remains challenging. Exploring a source video from new viewpoints requires the generated rollout to remain synchronised with the recorded event, place observed content in the requested view, plausibly complete newly exposed regions, and recover previously generated appearance on revisits. Existing methods typically address these requirements through task-specific modules or additional training. We present World in World, a training-free inference-time interface that converts heterogeneous control evidence into camera- and time-labelled clean visual states, which are read through the native self attention of a frozen causal video model. The evidence comprises source-video observations, target-view scene projections, geometry renderings that guide completion of newly exposed subject regions, and retrieved generated states beyond the rolling cache. Each evidence source carries token-level support and its own availability schedule. A correspondence router combines persistent point identities with geometry to establish token correspondences, guiding supported queries towards matching source-video tokens. Evidence-wise attention CFG (EWA) then independently regulates each auxiliary channel's additional contribution using attention responses from the same denoising forward pass. The shared interface supports camera-controlled rerendering, long-horizon revisiting, and human-motion transfer with the same frozen backbone. We evaluate World in World on camera-controlled video rerendering under diverse viewpoint changes, assessing perceptual quality, temporal consistency, and camera-following accuracy.

Chenxi Song, Yanming Yang, Chi Zhang

Westlake AGI Lab · AGI Lab, Westlake University

Camera-Controlled Video GenerationLong-Horizon Video GenerationVideo GenerationTemporal Consistency in Video GenerationVideo World Models
AskPDF
Sep 21, 2026

WorldCrafter: Consistent Video World Model with Implicit 3D-aware Memory

Video world models enable interactive exploration of dynamic environments, yet struggle to respect prior observations over long horizons and across viewpoints. We present WorldCrafter, a video world model that learns a camera-queryable implicit 3D-aware memory for this purpose. The key insight is to let the requested viewpoint shape how multi-view evidence is compressed into the video generator's limited token budget. Trained jointly with the video generator, a memory encoder and pose-conditioned readout module integrate historical observations into a fixed set of target view-specific tokens before denoising, without explicit depth-based correspondences. By combining this memory with recent temporal context and few-step distillation, WorldCrafter enables streaming scene exploration from a single input image or text prompt. Experiments across static and dynamic scenes show substantial gains in long-horizon consistency and camera-control accuracy while preserving visual quality during minute-scale exploration.

Wangbo Yu, Kunhao Liu, Wenbo Hu +8

ARC Lab, Tencent IEG · Peking University

3D-Aware Video GenerationCamera-Conditioned Video GenerationMemory-Augmented Video GenerationTemporal Consistency in Video GenerationVideo World Models
AskPDF