RISE: Roadside Infrastructure Sequence Understanding across 3D Tracking and Structured Vision-Language Reasoning

Yanbo Jiang; Haotian Zheng; Jiahao Wang; Hanxiao Ren; Yitao Xu; Yining Xing; Zehong Ke; Hao Cheng

cs.CVAug 17, 2026

RISE: Roadside Infrastructure Sequence Understanding across 3D Tracking and Structured Vision-Language Reasoning

Authors: Yanbo Jiang, Haotian Zheng, Jiahao Wang, Hanxiao Ren, Yitao Xu, Yining Xing, Zehong Ke, Hao Cheng,

Organizations: School of Vehicle and Mobility, Tsinghua University · TsingCloud

Abstract

We present RISE (Roadside Infrastructure Sequence Understanding and Evaluation), a framework spanning metric 3D tracking and structured vision-language reasoning in roadside sequences. For metric tracking, our image-only method combines SAM3 video identities with calibration-guided mask agreement for multi-view identity association, recovering persistent 3D tracks without LiDAR or task-specific 3D training. Its calibration-conditioned geometry allows the procedure to be instantiated at different calibrated multi-camera intersections without layout-specific retraining. On 20 human-reviewed clips from six intersections, the generated tracks achieve 66.9 MOTA within the defined multi-view evaluation scope. For structured vision-language reasoning, a human-reviewed MLLM pipeline mines high-value clips and uses a constrained full-context Oracle to construct bbox-grounded predictive QA without exposing future evidence to evaluated models. The resulting RISE-VQA dataset contains 33,910 QA pairs from 557 clips across 16 intersections and 61 roadside views. Its intersection-held-out RISE-Bench evaluates semantic choices, coordinates, future boxes, and interaction sets with deterministic task-specific metrics. Experiments show consistent benefits from domain adaptation and generally from temporal context, while revealing persistent challenges in spatial grounding, future localization, and interaction reasoning.

Figures & tables

Dataset	Venue	View	Unit	Source	QA Gen.	#QA	Scoring	3D Comp.
NuScenes-QA ( Qian et al. 2024 )	AAAI’24	ego	Multi-view	nuScenes	Template	460k	Accuracy	Inherited
DriveLM ( Sima et al. 2024 )	ECCV’24	ego	Frame	nuScenes	Temp.+Human	443k	SPICE/GPT	Inherited
LingoQA ( Marcu et al. 2024 )	ECCV’24	ego	Video	Self	Human+LLM	420k	Lingo-Judge	–
SUTD-TrafficQA ( Xu et al. 2021 )	CVPR’21	mixed	Video	Mixed	Human	62.5k	Acc.	–
TUMTraf VideoQA ( Zhou et al. 2025 )	ICML’25	roadside	Video	Self	LLM+Ver.	85.0k	Acc.	–
RoadSceneVQA ( Guan et al. 2026 )	AAAI’26	roadside	Image	Rope3D	LLM+Ver.	34.7k	Text/GPT	Inherited

Table 1: Comparison with traffic-scene vision-language datasets. “Inherited” denotes 3D labels from a source dataset; “Img. tracker” denotes the image-derived tracking component proposed in RISE.

Figure 1: Identity-first multi-view 3D tracking in RISE. View-local SAM3 video IDs are grouped into global identity backbones through calibrated voxel signatures and MWIS, followed by metric cuboid fitting and track-level refinement.

Figure 2: Structured VQA production in RISE. The pipeline mines high-value clips, constructs frame-level references, generates Oracle-grounded predictive QA pairs, and applies targeted human review.

Information	Annotation Role	Train/Test Input
Early frames $I_{0}$ – $I_{4}$	Evidence	Yes
Future frames $I_{5}$ – $I_{19}$	Oracle evidence	No
Future boxes	Supervision target	No
Infrastructure priors	Reference	No

Table 2: Observation–supervision boundary in the constrained Oracle protocol. Realized future observations determine predictive targets but remain hidden from train/test inputs.

Figure 3: Examples of bbox-grounded structured VQA. Boxes identify queried agents and future locations, while colored geometry provides infrastructure and interaction context.

Figure 4: Distribution of the 34,545 human-reviewed QA pairs before benchmark filtering. Left: frequent concepts in the question corpus. Right: fine-grained sub-category counts.

Metric	Value	Metric	Value
Intersections	16	Roadside views	61
4-view / 3-view	13 / 3	Event clips	557
Reviewed QA	34,545	Released QA	33,910
Training QA	29,219	Held-out Pool	5,326
RISE-Bench QA	4,691	Reviewed Static	11,846 (34.3%)
		Reviewed Dynamic	22,699 (65.7%)

Table 3: RISE-VQA and RISE-Bench statistics.

Task Family	#QA	Metric
Multiple choice	3,794	Choice score
Coordinate output	745	IoU / F1 / C-ADE
Set prediction	152	Precision / Recall / F1
Total	4,691	–

Table 4: RISE-Bench composition.

Model	Access	Setting	S-MCQ	D-MCQ	MCQ	Det-F1	Line-F1	T-IoU	C-ADE $\downarrow$	Inter-F1
Qwen2.5-VL-7B	Open	ZS-5f ‡	.479	.533	.510	.078	.001	.305	120.4	.066
InternVL3-8B	Open	ZS-5f ‡	.503	.424	.458	.029	.029	.050	306.8	.015
MiniCPM-V-4.5-8B	Open	ZS-5f ‡	.507	.571	.544	.061	.055	.257	131.3	.042
GPT-5.5	Closed	ZS-5f	.558	.593	.578	.392	.798	.248	117.8	.106
GPT-5.5 †	Closed	ZS-5f	.610	.632	.623	.649	.920	.402	60.8	.238
Gemini-3.1-Pro †	Closed	ZS-5f ‡	.502	.491	.495	.187	.719	.323	112.9	.214

Table 5: Main results on RISE-Bench. S-MCQ and D-MCQ denote static and dynamic MCQ scores. ZS-5f and FT-5f denote five-frame zero-shot inference and LoRA adaptation, respectively. ‡ marks bbox evaluation normalized from model-native serialization and coordinate systems; † marks thinking-enabled proprietary zero-shot runs. Other rows use direct-answer inference. Bold indicates the best result within the proprietary-reference or fine-tuned group.

Model	Fr.	S-MCQ	D-MCQ	T-IoU	C-ADE $\downarrow$
Qwen2.5-VL	1f	.784	.730	.312	109.1
	5f	.782	.781	.381	74.8
InternVL3	1f	.698	.738	.324	104.0
	5f	.734	.791	.433	62.2
MiniCPM-V-	1f	.727	.730	.322	115.1
4.5-8B	5f	.722	.764	.344	91.2

Table 6: Temporal-context ablation. Each pair uses the same backbone and LoRA recipe; only the number of input frames differs.

Eval.	Split	S-MCQ	D-MCQ	Det-F1	Line-F1	T-IoU	C-ADE $\downarrow$
All	Held	.782	.781	.643	.860	.381	74.8
All	Clip	.857	.806	.754	.944	.379	62.0
Com.	Held	.772	.775	.574	.846	.399	71.0
Com.	Clip	.841	.796	.654	.896	.420	57.0

Table 7: Split-policy ablation with Qwen2.5-VL-7B. Held and Clip denote intersection-held-out and clip-random train/test splitting, respectively. All denotes their respective test sets (4,691 and 4,359 QA pairs); Com. denotes the same 1,451 QA pairs absent from both training sets.

Figure 5: Human review interface for initialized 3D tracks. Multi-view projections and the semantic-voxel BEV guide manual correction.

Metric	Value	Metric	Value
Generated boxes	3,240	GT boxes	3,317
GT tracks	251	Scope-excl. inst.	1,158
Exclusion rate	25.9%	No frame-wise edit	1,900 (57.3%)

Table 8: Statistics of the human-reviewed subset. Dimensions are adjusted once per track; a frame requires no frame-wise correction when its XY center and heading remain unchanged.

Metric	Value	Metric	Value
MOTA $\uparrow$	66.9	MOTP BEV $\uparrow$	87.0
IDS $\downarrow$	40	F1 $\uparrow$	83.9
Precision / Recall $\uparrow$	84.9 / 82.9	XY err. (m) $\downarrow$	.071

Table 9: 3D tracking quality at a BEV IoU matching threshold of 0.5. MOTP is the mean BEV IoU over valid matches.

Figure 6: Projected cuboids from the multi-view 3D tracker across representative roadside sequences.

Temporal Reasoning in VLMs

See all 23 papers

Inter-3D VQA: A Roadside Multimodal Benchmark for 3D Spatiotemporally Grounded Visual Question Answering
Aug 28, 2026Shaozu Ding, Linan Song, Dajiang Suo3D VQA Spatial Reasoning Benchmarks
The Polytechnic School, Arizona State University
RESOLVE: A Multi-Resolution and Multi-Modal Dataset for Roadside Cooperative Perception
Jun 30, 2026Shaozu Ding, Linan Song, Marco De Vincenzi +1Autonomous Driving Datasets Camera-LiDAR Fusion
The Polytechnic School, Arizona State University, USA · Department of Computer Science and Engineering, New York University, USA
OmniTraffic: A Controllable Generation Pipeline and Benchmark for Spatio-Temporal Traffic Reasoning
Jun 14, 2026Maonan Wang, Zhengyan Huang, Kemou Jiang +13Autonomous Driving Datasets Spatiotemporal Reasoning
The Chinese University of Hong Kong, Shenzhen · Shanghai AI Lab · Beihang University +3

Aug 28, 2026J/K move · Enter open · S save

Inter-3D VQA: A Roadside Multimodal Benchmark for 3D Spatiotemporally Grounded Visual Question Answering

Shaozu Ding, Linan Song, Dajiang Suo

The Polytechnic School, Arizona State University

3D VQA Spatial Reasoning Benchmarks Autonomous Driving Benchmarks VLMs for Autonomous Driving 3D VLMs

Open paper AskPDF arXiv

Recent advances in visual question answering (VQA) and multimodal large language models (MLLMs) have enabled natural-language reasoning over traffic scenes. However, existing benchmarks are largely built from ego-vehicle views or 2D roadside videos, limiting their ability to evaluate 3D-grounded reasoning over real-world distances, trajectories, infrastructure topology, and safety-critical interactions. We introduce Inter-3D VQA, a large-scale roadside multimodal benchmark for 3D spatiotemporally grounded VQA at intersections. Built from synchronized point clouds and multi-view images, Inter-3D VQA contains 407K QA pairs covering lane-level positions, object relationships, motion patterns, and near-miss-oriented interaction reasoning. We further propose Inter-Geo, an MLLM baseline that integrates object- and scene-level aligned LiDAR representations, and Inter-Metrics, a unified evaluation framework for textual consistency, numerical accuracy, and semantic correctness. Experiments show that Inter-Geo outperforms image-based VLMs, especially on grounded spatial and temporal reasoning tasks. Our benchmark and codes are available at https://github.com/ASU-Suo-Lab/Inter-3D-VQA .

Abstract

Temporal Reasoning in VLMs

Explore similar work

Inter-3D VQA: A Roadside Multimodal Benchmark for 3D Spatiotemporally Grounded Visual Question Answering

RESOLVE: A Multi-Resolution and Multi-Modal Dataset for Roadside Cooperative Perception

OmniTraffic: A Controllable Generation Pipeline and Benchmark for Spatio-Temporal Traffic Reasoning