Video2Asset: Online Construction of Persistent 3D Street-Scene Assets

Zihan Zheng; Chunlei Cai; Zhenlong Wu; Linjun Li; Yichen Gong; Xiaoyun Zhang; Qiang Hu

cs.CVNov 6, 2026

Video2Asset: Online Construction of Persistent 3D Street-Scene Assets

Authors: Zihan Zheng, Chunlei Cai, Zhenlong Wu, Linjun Li, Yichen Gong, Xiaoyun Zhang, Qiang Hu

Organizations: Cooperative Medianet Innovation Center, Shanghai Jiao Tong University · Agentic Labs, Acrab AI

Abstract

Online 3D street-scene asset construction turns multi-view video into a continually updated scene of independently editable objects. The central challenge is to accumulate partial, intermittent observations under motion and occlusion without fragmenting identities or mixing evidence across objects. To address this challenge, we propose Video2Asset, an online framework for constructing persistent 3D assets from multi-view video. Using occupancy as an intermediate representation, we construct identity-aware observations with Historical Carrier Guidance for instance decoding and an identity encoder fusing instance-aligned 3D features with multi-view appearance. Accumulated-Evidence Association compares current observations with identity, geometric, and appearance evidence accumulated within asset records to determine identity inheritance and update world-coordinate states. We instantiate meshes from an asset library using these states and combine them with an occupancy-derived static background to support scene visualization, mesh replacement, object deletion, and trajectory editing. Experiments on several datasets demonstrate improvements in temporal identity consistency and asset quality across multiple occupancy models, including a 15.4% reduction in asset fragmentation on nuScenes with SGE.

Figures & tables

Figure 2: Overview of Video2Asset. (a) Multi-view video is converted into identity-aware observations through occupancy modeling. The observations are associated with persistent asset records to update world-coordinate states and construct an editable 3D scene. (b) Historical Carrier Guidance (HCG) aligns previous instance masks to guide query-conditioned aggregation of current occupancy features. (c) Accumulated-Evidence Association (AEA) compares current observations with identity, geometric, and appearance evidence accumulated in candidate asset records for matching and identity-inheritance prediction, followed by one-to-one assignment.

Dataset	Method	Observation Quality				Temporal Metrics				Asset Quality
Dataset	Method	mIoU $\uparrow$	$\mathrm{STQ}_{1}\uparrow$	$\mathrm{AQ}_{1}\uparrow$	IoU $\uparrow$	STQ $\uparrow$	AQ $\uparrow$	HOTA $\uparrow$	IDF1 $\uparrow$	$\mathrm{AP}_{25}\uparrow$	$\mathrm{AP}_{50}\uparrow$	$\mathrm{AF}\downarrow$
	TrackOcc [ 4 ]	33.92	25.91	19.80	56.91	12.89	4.90	11.25	7.24	8.88	6.62	6.35
	Ours (with TrackOcc)	35.73	29.17	23.83	57.08	23.24	15.12	25.99	24.59	19.29	15.45	2.75
	LaGS [ 20 ]	45.23	41.42	37.93	58.78	38.41	32.62	40.23	41.95	43.74	30.03	0.27
	Ours (with LaGS)	45.36	42.13	39.13	58.81	38.63	32.90	40.36	42.19	45.24	31.17	0.25
	SGE [ 21 ]	44.23	40.75	37.54	58.05	37.99	32.63	40.25	41.73	41.91	26.31	0.34

Table 1: Quantitative comparison on CarlaOcc, nuScenes, and Waymo. Shaded rows denote Video2Asset with the indicated occupancy model. Metrics cover observation quality, temporal consistency, and asset quality.

AF $\downarrow$	TrackOcc	LaGS	SGE
Baseline	2.48	0.87	0.95
Ours	1.90	0.69	0.86

Table 2: Asset fragmentation under low visibility. Experiments are conducted on nuScenes dataset.

Figure 3: Qualitative comparison of street-scene assets. Each group compares a baseline with Video2Asset (Ours) built on the corresponding occupancy model: TrackOcc (left), LaGS (middle), and SGE (right). GT denotes a reference scene instantiated from ground-truth annotations.

Variant	HCG	AEA	Observation Quality				Temporal Metrics				Asset Quality
Variant	HCG	AEA	mIoU $\uparrow$	$\mathrm{STQ}_{1}\uparrow$	$\mathrm{AQ}_{1}\uparrow$	IoU $\uparrow$	STQ $\uparrow$	AQ $\uparrow$	HOTA $\uparrow$	IDF1 $\uparrow$	$\mathrm{AP}_{25}\uparrow$	$\mathrm{AP}_{50}\uparrow$	$\mathrm{AF}\downarrow$
Base	$\times$	$\times$	33.92	25.91	19.80	56.91	12.89	4.90	11.25	7.24	8.88	6.62	6.35
+ HCG	$\checkmark$	$\times$	35.73	29.17	23.83	57.08	18.02	9.09	17.68	13.78	19.26	15.37	4.83
+ AEA	$\times$	$\checkmark$	33.92	25.91	19.80	56.91	20.20	12.03	22.81	20.08	8.88	6.62	3.67
Full	$\checkmark$	$\checkmark$	35.73	29.17	23.83	57.08	23.24	15.12	25.99	24.59	19.29	15.45	2.75

Table 3: Ablation experiments. Results on CarlaOcc with TrackOcc as the base model.

Metric $\uparrow$	Base	Local	Shared	HCG
mIoU	33.64	33.25	32.25	33.85
$\mathrm{STQ}_{1}$	20.51	20.48	18.84	22.82
$\mathrm{AQ}_{1}$	12.50	12.61	11.00	15.38
IoU	63.37	63.62	63.54	63.69
STQ	13.14	13.65	12.43	16.34
AQ	5.14	5.60	4.79	7.89

Table 4: HCG ablation experiments. Results on nuScenes with TrackOcc as the base model.

Variant	STQ $\uparrow$	AQ $\uparrow$	HOTA $\uparrow$	IDF1 $\uparrow$
HCG only	16.34	7.89	19.97	18.59
Geometry tracker	16.13	7.69	19.23	18.49
Simple memory association	16.66	8.20	21.35	20.03
w/o decoupling	16.93	8.47	22.09	20.84
Single-frame history	16.36	7.91	19.95	18.64
Uniform history aggregation	16.38	7.93	20.14	18.82

Table 5: AEA ablation experiments. Results on nuScenes with TrackOcc as the base model.

Figure 4: Asset-level editing of exported street scenes.

Component	Ours	Ours	Ours
Component	(with TrackOcc)	(with LaGS)	(with SGE)
Baseline	127.32	303.08	324.99
HCG	3.67	5.89	4.24
Identity encoder	4.30	4.46	3.17
AEA	8.44	8.46	9.02
Total	143.73	321.89	341.42

Table 6: Runtime breakdown. Latency (ms/frame) on nuScenes.

3D Reconstruction

3D reconstruction recovers a scene’s shape, appearance, and camera motion from images, video, depth, or other sensor measurements.

See all 978 papers

Scene-SAM3D: Multi-View Scene Asset Generation Without Fine-Tuning
Jul 18, 2026Yuqi Zhang, Yadan Luo, Xiangyu Sun +33D Asset Generation 3D Scene Generation
The University of Queensland · Shanghai AI Laboratory · East China Normal University
OmniShowcase: Omnidirectional Subject-Consistent Video Generation from Multi-View Images
Oct 26, 2026Junyeong Ahn, Geonwoo Kim, Kinam Kim +4Video Generation Identity-Preserving Video Generation
KAIST AI · Inha University · FLIPTION
Scal3R: Learning Efficient Multi-Relative Pose Query for Scalable Online 3D Reconstruction
Sep 3, 2026Chin-Yang Lin, Yang-Che Sun, Cheng Sun +5Relative Pose Estimation Streaming 3D Reconstruction
Department of Computer Science, National Yang Ming Chiao Tung University · NVIDIA

Jul 18, 2026J/K move · Enter open · S save

Scene-SAM3D: Multi-View Scene Asset Generation Without Fine-Tuning

Yuqi Zhang, Yadan Luo, Xiangyu Sun, Fengyi Zhang+2

The University of Queensland · Shanghai AI Laboratory · East China Normal University

3D Asset Generation 3D Scene Generation 3D Scene Representation 3D Scene Layout Generation 3D Diffusion Models Image-to-3D Scene Generation

Open paper AskPDF arXiv

High-quality 3D scene assets are critical for embodied applications such as robotic manipulation, navigation, and simulation. Despite their strong object priors, recent single-image 3D generation models such as SAM3D remain insufficient for real-world scenes, where severe occlusions, redundant observations, and cross-view inconsistencies make reliable scene generation challenging. We introduce Scene-SAM3D, a training-free framework that extends SAM3D from single-view object generation to calibrated multi-view scene asset generation. Scene-SAM3D selects a compact set of complementary views, reducing observation redundancy while providing additional evidence for regions occluded in individual views. Based on the selected views, it performs step-efficient latent velocity fusion to integrate multi-view evidence and suppress cross-view conflicts in canonical space. Finally, a lightweight rigid-object Gaussian optimization refines the scene layout within 200 iterations while preserving the generated object geometry. Experiments on Replica and ScanNet++ demonstrate consistent improvements at both instance and scene levels, with our method reducing scene-level CD by 43.8% on Replica and 30.9% on ScanNet++, while cutting flow-model sampling FLOPs and wall-time latency by nearly 20% under the same multi-view setting. Code will be released at https://github.com/xibi777/Scene-SAM3D.

Abstract

3D Reconstruction

Explore similar work

Scene-SAM3D: Multi-View Scene Asset Generation Without Fine-Tuning

OmniShowcase: Omnidirectional Subject-Consistent Video Generation from Multi-View Images

Scal3R: Learning Efficient Multi-Relative Pose Query for Scalable Online 3D Reconstruction