cs.CVAug 16, 2026

WorldRover: A Scalable Synthetic Video Data Engine for World Exploration with Rich Annotations

Authors: Xiaojie Xu, Zhengyuan Lin, Runyi Li, Yihao Liu, Kaipeng Zhang, Yongtao Ge

Organizations: Alaya Lab · The University of Tokyo

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Learning to generate or reconstruct explorable worlds requires video paired with more than RGB: camera motion, scene geometry, temporal correspondence and, for interactive models, control signals. Real capture can provide some of these signals, but dense geometry and long-range correspondence usually rely on estimation or specialised instrumentation. Rendering provides these quantities directly, yet existing synthetic resources rarely combine them on the same frames while also supporting controlled changes of viewpoint and appearance. We introduce WorldRover, a data engine for generating richly annotated, long-range explorations of artist-built environments. At its core, WorldRover-Engine is an Unreal Engine pipeline that executes and offline-renders minute-scale routes while preserving their full trajectories and scene geometry. The same exploration can be replayed from first-person, third-person, and 360 panoramic cameras under different environmental states. Using WorldRover-Engine, we construct WorldRover-10M, whose sequences pair RGB with metric depth, camera trajectories, and trajectory-derived action signals throughout each exploration. Third-person subsets additionally provide dense optical flow, long-range 2D/3D point tracks with visibility, and a character trajectory distinct from the camera trajectory. The engine can render a traversal from first-person, third-person and 360 panoramic viewpoints, under different environmental states or with a neutral white material, while preserving the route and scene geometry. WorldRover therefore turns long-horizon world exploration into a scalable data-generation problem, providing supervision for models that must build, maintain, and revisit coherent representations of an explorable world.

Figures & tables

Figure 1: WorldRover-10M at a glance. Four rows, top to bottom. Scene and routes : one UE environment with three routes drawn as camera frustums, and four frames along each, colour-keyed to the overview. (1) Diverse viewpoints : the same instants observed as first-person, third-person and 360∘360^{\circ} panoramic video; every panel is RGB with metric depth inset, released with the camera that produced it. Third-person adds the character’s own trajectory and the trajectory-derived action stream. (2) Diverse scene styles : the first-person view re-rendered under a different weather state, a different lighting state, and as an untextured white model, with scene, motion and camera held fixed. (3) Diverse motion labels : dense optical flow and long-range point tracks, shown in 2D and in world space; a track stores a 3D position, its 2D projection and visibility per frame. Third-person only, with independent body and camera motion.
Figure 2: WorldRover-Engine overview. Asset processing runs once per scene on a Windows editor host; content generation runs once per sequence, headless, on a multi-GPU Linux host. The two halves meet at a versioned on-disk hand-off, so a campaign is replayable without an editor session.
Figure 3: Trajectory generation. The three strategies for producing walking trajectories, on one scene. (1) NavMesh planner : a route ( orange ) planned over a baked navigation mesh ( green ); the mesh is built once and reused, so planning is fast and coverage broad, at the cost of per-scene tuning. (2) Reactive explorer : a path ( blue ) driven by local obstacle sensing ( red ); it needs no precomputed map, but senses step by step and can walk itself into a corner. (3) Manual recording : a human-driven path ( purple ), which yields a specific trajectory on demand at the cost of manual effort. Together they trade off automation, coverage and control.
Figure 4: Third-person control semantics. Centre: the start state. W/A/S/D (black) translate the character along camera-relative directions — W away from the camera, S toward it, A and D to its left and right — and the character turns to face its heading while the camera follows with a slowly varying orbit azimuth. L/R (blue) yaw the camera about a stationary character, so the character holds its image position while the scene rotates around it.
Figure 5: Panoramic assembly. (a) Six cube faces at 100∘100^{\circ} FOV, giving a 10∘10^{\circ} overlap between neighbours, rendered with tone mapping disabled so all faces share one radiometric scale. (b) Reprojection onto an equirectangular image; dashed outlines, colour-matched to (a), mark each face’s region and the yellow bands its overlap, which exposes the pincushion distortion of the side faces and the polar caps. (c) A cos-weighted feather blend merges the overlaps into a linear-HDR panorama. (d) Exposure is estimated with latitude-aware luminance weighting and smoothed across frames; local base/detail tone mapping compresses the result before sRGB conversion.
Figure 6: Point-track ground truth. Six frames 44 s apart, time encoded magenta →\rightarrow cyan; each curve follows one 3D point on the character. Filled markers and solid curves are visible, hollow rings and dashed curves occluded and drawn at their amodal ground-truth position. In the fifth panel a wall encloses the camera and hides the character completely, yet the tracks retain their amodal positions with zero visibility flags.
Data, when presentContents
RGB videoH.264, hardware-encoded
Depth16-bit log-quantized, FFV1 lossless, ++ decode sidecar
Optical flow16-bit affine-quantized, FFV1 lossless, ++ decode sidecar (third-person only)
Point tracksper-track 3D position, 2D projection and visibility, per frame (third-person only)
Camera trajectoryper-frame pose and intrinsics of the rendered camera
Character trajectoryper-frame pose and speed (third-person only)
Table 1: Released data format. How each item is stored when a subset includes it.
Figure 7: A sample of the WorldRover scene library. Fifteen of 30+ UE environments — waterfront city, transit interior, cathedral, alpine meadow, cyberpunk street, office interior, Paris street, suburban home, jungle ruin, and more — all rendered under dynamic Lumen global illumination.
Figure 8: Diverse appearances. The same first-person frame of med_village , rendered from an identical camera at the same instant under five environmental states. Because lighting is simulated at render time rather than baked, indirect illumination, colour bleeding and shadow softness respond correctly to each state; each state uses its own calibrated fixed exposure, with no automatic adjustment during rendering, so Night stays dark rather than normalised toward mid-grey. Re-rendering an instant under another state therefore changes the image physically, not by recolouring a fixed frame.
Figure 9: White model. Three environments, the ordinary render above and the white-model re-render of the same frame below. With scene surfaces drawn in the engine’s default material, geometry and occlusion remain visible from large structures to fine detail — arcade columns and planters, station ironwork, individual leaves and branches — while texture, colour, decals and emissive light do not: the tiled floor flattens, the screens on the ticket machines go blank, and the burning wreck in the clearing is left as geometry alone.
Figure 10: Character library. Asset-review sheets — the avatars shown in one shared scene at a fixed pose — rather than frames from the released dataset. Third-person sequences are rendered with humanoids, quadrupeds and birds, which differ in body plan, gait and how they turn, so subject articulation is a diversity axis in its own right.
Engine/library capacity
Environments30+ artist-built UE scenes, interior to city scale
Lighting statesday, dawn, evening, sunset, night
Weather statesclear, overcast, fog, snow
Materialstextured render or untextured white model
Characters70+ animated humanoids, animals and creatures
Routesnavmesh planner, reactive explorer, manual recording
Table 2: WorldRover-Engine capacity and the current WorldRover-10M release. The upper block describes the engine and asset library, not the coverage of every released subset. In the lower block, scenes are counted per viewpoint and therefore do not add; 32 is the number of distinct environments.
Figure 11: Distributions over the released sequences ( n=6003n=6003 , from per-sequence metadata). Left: sequence duration, stacked by trajectory strategy in the colours of Figure 3 — reactive routes end sooner than planned ones, which run on to the long tail. Centre: distance travelled, a median of 122122 m. Right: mean speed per sequence, which falls into three clusters because the motion model samples one of three walking speeds; each cluster sits a little below its setting, the cost of turning and rate limiting.
More in this topic

Synthetic Data Generation

Synthetic data generation creates artificial examples, simulations, and datasets to train, test, or protect machine-learning systems.

See all 625 papers

Explore similar work

CardsList

  1. Sekai2: From World Exploration to Interactive World Modeling

    Aug 10, 2026Kang He, Wenshuo Peng, Zihui Gao +3Camera-Controlled Video GenerationLong-Horizon Video Generation

    1Alaya Lab · 2Shanghai Innovation Institute · 3Wuhan University +1

  2. World in World: Explore the World with World Models

    Sep 10, 2026Chenxi Song, Yanming Yang, Chi ZhangCamera-Controlled Video GenerationLong-Horizon Video Generation

    Westlake AGI Lab · AGI Lab, Westlake University

  3. WorldCrafter: Consistent Video World Model with Implicit 3D-aware Memory

    Sep 21, 2026Wangbo Yu, Kunhao Liu, Wenbo Hu +83D-Aware Video GenerationCamera-Conditioned Video Generation

    ARC Lab, Tencent IEG · Peking University