Organizations: Fujian Key Laboratory of Urban Intelligent Sensing and Computing, Xiamen University, Xiamen, China. · Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, Xiamen, China. · The Hong Kong Polytechnic University, Hong Kong SAR, China. · Zhongguancun Academy, China.
3D visual grounding (3DVG) localizes language-referred objects in 3D scenes, with broad applicability in both indoor and outdoor environments. However, existing 3DVG research focuses on static indoor objects, limiting its use in dynamic outdoor autonomous driving. Motion descriptions, which capture the temporal dynamics of moving objects, offer a promising way to address this challenge. We therefore introduce MoT3DVG, a large-scale dataset for dynamic-aware outdoor 3DVG with temporally evolving motion descriptions. It contains 850 scenarios with 31,128 frames from nuScenes dataset, and provides 144,568 language prompts with motion-aware descriptions for dynamic objects across time. We further propose DynaVG, a novel framework that effectively leverages temporal cues for outdoor 3DVG. In addition to the standard modality-specific encoders, DynaVG introduces a Short-Long Integrated Dynamic Encoding (SLIDE) module before the language-point cloud alignment. SLIDE models both local motion cues and global temporal context via short-term window encoding and long-term window shifting. Extensive experiments demonstrate its state-of-the-art performance on MoT3DVG, validating the effectiveness of motion-aware encoding. Our work reveals open challenges and promising directions for future research in outdoor 3DVG.
Figures & tables
Dataset
Sensor
3D
M.V.
Mot.
T. Mot.
Expr.
Frames
Obj.
Scenes
Talk2Car Deruyttere et al. (2019)
C
✗
✗
✓
✗
11.9K
9,217
10.5K
-
Refer-KITTI Wu et al. (2023a)
C
✗
✗
✓
✗
0.8K
6,650
-
18
Mono3DRefer Zhan et al. (2024)
C
✓
✗
✓
✗
41K
2,025
8.2K
-
Talk2Car-3D Liu et al. (2024)
L + C
✓
✗
✓
✗
8.4K
5,534
-
-
TOD 3 Cap Jin et al. (2024)
L + C
✓
✓
✓
✗
2.3M
34,149
-
850
NuPrompt Wu et al. (2025)
L + C
✓
✓
✓
✗
35K
34,149
-
850
Table 1: Comparison of outdoor language-based datasets, where "C" and "L" mean Camera and LiDAR; "M.V.", "Mot.", "T. Mot", "Expr.", and "Obj." are short for multi-view, motion, temporal motion, expressions, and objects, respectively. Our dataset focuses specifically on language descriptions of the continuous dynamic behaviors exhibited by outdoor objects.
Figure 1: Objects in indoor and outdoor scenarios. (a) In most cases, spatial relationships among indoor objects remain static in short time intervals, whereas (b) objects in outdoor scenes exhibit non-fixed spatial relationships due to their dynamic behaviors.
Figure 2: The visualization of MoT3DVG . (a) present the word cloud of the dataset, which encompasses diverse motion states of dynamic objects; (b) present examples of temporal motion description, where orange denotes the language-referred object, while green indicates non-referred objects.
Figure 3: The overview of proposed DynaVG . It takes a temporal sequence of point cloud frames and a language description as input. Leveraging the proposed SLIDE module, DynaVG encodes historical temporal cues into current frame point features, fuses them with the language embedding, and decodes the language-referred object.
Method
Setting
Unique
Multiple
Overall
@1.0
@0.5
mIoU
@1.0
@0.5
mIoU
@1.0
@0.5
mIoU
BUTD-DETR Jain et al. (2022)
Indoor
17.00
8.75
7.30
26.10
15.42
9.25
44.27
28.87
15.38
EDA Wu et al. (2023d)
36.34
15.54
13.08
52.43
36.57
16.95
50.74
35.80
16.49
AugRefer Wang et al. (2025a)
32.83
15.54
12.23
53.75
38.50
17.11
54.59
37.31
17.48
WildRefer Lin et al. (2024)
Outdoor
15.04
8.02
9.79
35.47
18.60
14.24
33.40
17.65
14.07
3EED Li et al. (2025b)
51.38
31.58
16.73
58.45
47.96
18.23
67.22
54.94
21.16
Table 2: Benchmark results of state-of-the-art 3DVG methods on the MoT3DVG dataset. We report accuracy, which is evaluated by distances of 1.0 and 0.5m, and mIoU across the predicted grounding boxes for Unique subset, Multiple subset, and Overall . All metrics are presented in percentage (%).
Method
Range
Unique
Multiple
Overall
@1.0
@0.5
mIoU
@1.0
@0.5
mIoU
@1.0
@0.5
mIoU
3EED Li et al. (2025b)
0-20m
57.08
43.38
17.88
64.64
56.36
18.73
73.23
64.55
21.46
DynaVG (Ours)
50.68
38.81
19.02
75.95
67.43
22.02
74.22
64.87
21.81
Improvement
-6.40
-4.57
+1.14
+11.31
+11.07
+3.29
+0.99
+0.32
+0.35
3EED Li et al. (2025b)
20-40m
43.79
18.30
14.45
48.95
32.38
18.32
57.02
37.24
21.27
DynaVG (Ours)
52.29
21.57
19.35
59.71
43.71
22.25
59.93
40.32
22.51
Table 3: Comparison with state-of-the-art 3DVG method on MoT3DVG across various distance ranges. We partition the dataset into three distance intervals according to the valid range of the dataset to investigate the impact of varying distances on model performance.
Table 7
Method
View
@1.0
@0.5
mIoU
3EED Li et al. (2025b)
Front
61.84
46.11
20.76
DynaVG (Ours)
65.25
51.06
21.63
Improvement
+3.41
+4.95
+0.87
3EED Li et al. (2025b)
S.Left
60.06
48.90
19.46
DynaVG (Ours)
63.33
50.25
20.02
Improvement
+3.27
+1.35
+0.54
Table 6: Comparison with state-of-the-art 3DVG methods on different view. ‘S.Left’, ‘S.Right’ are the views of the side left and side right, respectively.
STWE
LTWS
@1.0
@0.5
mIoU
67.22
54.94
21.16
✓
67.41
55.20
21.41
✓
67.71
55.23
21.71
✓
✓
69.66
57.34
22.22
Table 7: Ablation study on components of the SLIDE module.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Visualization of instance trajectories. Each red box denotes a timestep. ID denotes the current instance token, sceneID represents the index of the current scene, and class name indicates the category of the current object.
Table 11
Figure 6: Dataset statistics of MoT3DVG . (a) Distribution of boxes by number of points. (b) Distribution of caption length. The numbers in the outer ring denote the word lengths contained within the captions, while the numbers in the inner ring indicate the proportion of each length across the entire dataset.
Table 15: Effect of auxiliary motion supervision. Parameter counts are reported in millions and memory usage in MiB.
Figure 7: Qualitative visualization comparison with state-of-the-art outdoor 3DVG methods. The data are displayed with RGB frames and LiDAR point clouds, along with the corresponding language description. The ground truth boxes are given in orange. We use green and blue for the prediction boxes of 3EED and DynaVG, respectively.
Figure 8: Visualization of failure cases. The ground truth boxes are given in red. We use orange and darkcyan for 3EED and DynaVG prediction boxes, respectively.
Total num.
Positive num.
0.5
1.0
3EED Li et al. [2025b]
218
101
128
DynaVG
87
111
Appendix
Table 16: Number count of referent objects in the near-field Unique split.
Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou) · MMLab, CUHK · School of Transportation Science and Engineering, Harbin Institute of Technology +7