Efficient Multi-View 3D Object Detection by Dynamic Token Selection and Fine-Tuning

Danish Nazir; Antoine Hanna-Asaad; Lucas Görnhardt; Jan Piewek; Thorsten Bagdonat; Tim Fingscheidt

cs.CVApr 15, 2026

Efficient Multi-View 3D Object Detection by Dynamic Token Selection and Fine-Tuning

Authors: Danish Nazir, Antoine Hanna-Asaad, Lucas Görnhardt, Jan Piewek, Thorsten Bagdonat, Tim Fingscheidt

Organizations: Group Innovation, Volkswagen AG, 38440, Wolfsburg, Germany · Institute for Communications Technology, Technische Universität Braunschweig, 38106 Braunschweig, Germany

Abstract

Existing multi-view three-dimensional (3D) object detection approaches widely adopt large-scale pre-trained vision transformer (ViT)-based foundation models as backbones, being computationally complex. To address this problem, current state-of-the-art (SOTA) ToC3D for efficient multi-view ViT-based 3D object detection employs ego-motion-based relevant token selection. However, there are two key limitations: (1) The fixed layer-individual token selection ratios limit computational efficiency during both training and inference. (2) Full end-to-end retraining of the ViT backbone is required for the multi-view 3D object detection method. In this work, we propose an image token compensator combined with a token selection for ViT backbones to accelerate multi-view 3D object detection. Unlike ToC3D, our approach enables dynamic layer-wise token selection within the ViT backbone. Furthermore, we introduce a parameter-efficient fine-tuning strategy, which trains only the proposed modules, thereby reducing the number of fine-tuned parameters from more than $300$ million (M) to only $1.6$ M. Experiments on the large-scale NuScenes dataset across three multi-view 3D object detection approaches demonstrate that our proposed method decreases computational complexity (GFLOPs) by $48\%$ ... $55\%$ , inference latency (on an NVIDIA-GV100 GPU) by $9\%$ ... $25\%$ , while still improving mean average precision by $1.0\%$ ... $2.8\%$ absolute and NuScenes detection score by $0.4\%$ ... $1.2\%$ absolute compared to so-far SOTA ToC3D.

Figures & tables

Panel of Figure 1 — Fig. 1: High-level overview of the basic multi-view $(v=1,2,\dots)$ 3D object detection pipeline used across multi-view 3D object detection methods, including StreamPETR [ 1 ] , Sparse4Dv2 [ 20 ] , and RayDN [ 16 ] for training and inference. Here, $\mathbf{E}$ and $\mathbf{D}$ are the image encoder and task decoder, respectively. $\mathbf{FPN}$ represents a feature pyramid network. Details of the image encoder $\mathbf{E}$ are depicted in Fig. 2 .

Panel of Figure 2 — Fig. 2: (a) Layer architecture of the image encoder $\mathbf{E}$ utilized in Fig. 1 for the baseline [ 3 ] . (b) Details the output projection block [ 18 ] . Blocks marked with are updated during fine-tuning, blocks with remain frozen.

Block	FLOPs (G)	$\#$ params (M)
WMHSA	$6.9$	$4.2$
Output Projection	$16.8$	$8.4$
Total	$23.7$	$12.6$

TABLE I: Computational complexity breakdown of a naive transformer module in the EVA-02-L [ 18 ] encoder without any image feature token selection, measured with an input resolution of

320\!\times\!800

\mathcal{D}^{\mathrm{val}}_{\mathrm{NS}}

Panel of Figure 4 — Fig. 3: (a) Proposed layer architecture of the image encoder $\mathbf{E}$ utilized in Fig. 1 . (b) Proposed token compensator $(\mathbf{TC})$ block. Blocks marked with are updated during fine-tuning, blocks with remain frozen.

Panel of Figure 5 — Fig. 4: Our token router block used in Fig. 3 (a), adopted from [ 34 , 23 ] . The output projection block (details in Fig. 2 (b)) remains frozen during the fine-tuning stage.

$d_{h}$	mAP ↑ (%)	NDS ↑ (%)	GFLOPs ↓	Latency (ms) ↓
	mAP ↑ (%)	NDS ↑ (%)	GFLOPs ↓	$\tau^{\mathrm{E}}$	$\tau$
$16$	$50.1$	$58.9$	$\mathbf{5,754}$	$\mathbf{444}$	$\mathbf{485}$
$32$	$\mathbf{50.9}$	$\mathbf{59.6}$	$\underline{5,759}$	$\underline{454}$	$\underline{492}$
$64$	$50.8$	$59.5$	$5,860$	$496$	$537$

TABLE II: Ablation study to select the internal hidden dimension

d_{h}

of the proposed token compensator utilized in Fig. 2 (b) on

\mathcal{D}^{\mathrm{val}}_{\mathrm{NS}}

with an EVA-02-L [ 18 ] encoder. The encoder employs

r=0.5

and is integrated into StreamPETR [ 1 ] . The encoder-only (

\tau^{\mathrm{E}}

) and complete pipeline latencies (

\tau

) are reported. Best results bold, second-best underlined.

Panel of Figure 7 — Fig. 5: Comparison of mAP and NDS in dependence on GFLOPs and latency (ms), measured with an input resolution of $320\times 800$ for $\mathcal{D}^{\mathrm{val}}_{\mathrm{NS}}$ for our proposed approach vs. ToC3D [ 3 ] , both integrated into StreamPETR [ 1 ] . Symbol $r$ represents the average image feature token activation rate in the image encoder layer.

Panel of Figure 8 — Fig. 6: Comparison of $\#$ activated tokens at the output projection block (Fig. 2 (b)) in each image encoder layer, measured with an input resolution of $320\times 800$ on 300 randomly sampled instances from $\mathcal{D}^{\mathrm{val}}_{\mathrm{NS}}$ for our proposed approach vs. ToC3D [ 3 ] , both integrated into StreamPETR [ 1 ] .

Enc.

Methods

Image Resolution

mAP ↑ (%)

NDS ↑ (%)

mATE ↓

mASE ↓

mAOE ↓

mAVE ↓

mAAE ↓

Model Size ↓ (M)

GFLOPs ↓

Latency ↓

Avg. rank ↓

\tau^{\mathrm{E}}

\tau

RN-50

BEVDet [ 8 ]

320\!\times\!800

31.5

39.2

0.703

0.269

0.580

\underline{0.864}

\underline{0.243}

\mathbf{48.1}

\mathbf{1,\!191}

\mathbf{17}

\mathbf{53}

\underline{2.09}

BEVDepth [ 9 ]

320\!\times\!800

\underline{34.6}

\underline{40.2}

\underline{0.639}

\mathbf{0.267}

\underline{0.556}

0.984

0.258

\underline{53.6}

\underline{1,\!349}

\underline{37}

\underline{79}

\underline{2.09}

SOLOFusion [ 10 ]

320\!\times\!800

\mathbf{42.6}

\mathbf{53.6}

\mathbf{0.582}

\underline{0.272}

\mathbf{0.460}

\mathbf{0.261}

\mathbf{0.197}

65.3

1,\!451

40

567

\mathbf{1.81}

SAM-B [ 19 ]

StreamPETR [ 1 ]

320\!\times\!800

\mathbf{46.3}

\mathbf{55.5}

\mathbf{0.615}

\underline{0.266}

0.417

\underline{0.267}

0.204

\mathbf{99.7}

3,\!546

166

205

2.63

+

tgGBC [ 4 ]

320\!\times\!800

43.4

54.2

0.638

0.275

\mathbf{0.388}

\mathbf{0.244}

0.215

\mathbf{99.7}

3,\!530

159

188

2.81

TABLE III: Comparison of mAP, NDS, and various 3D object detection error metrics in dependence on GFLOPs and latency (ms), measured with two input resolutions on

\mathcal{D}^{\mathrm{val}}_{\mathrm{NS}}

for various methods. In each table segment, best results bold, second-best underlined.

Token Compensator	$r$	mAP ↑ (%)	NDS ↑ (%)	GFLOPs ↓	Latency ↓
		mAP ↑ (%)	NDS ↑ (%)	GFLOPs ↓	$\tau^{\mathrm{E}}$	$\tau$
DAR-TR-PEFT [ 23 ]	$0.1$	$43.0$	$53.4$	$\mathbf{2,\!291}$	$\mathbf{121}$	$\mathbf{158}$
Ours	$0.1$	$\mathbf{44.1}$	$\mathbf{53.9}$	$2,\!295$	$123$	$160$
DAR-TR-PEFT [ 23 ]	$0.3$	$43.6$	$53.6$	$\mathbf{2,\!447}$	$\mathbf{128}$	$\mathbf{168}$
Ours	$0.3$	$\mathbf{46.0}$	$\mathbf{55.1}$	$2,\!449$	$129$	$173$
DAR-TR-PEFT [ 23 ]	$0.5$	$46.0$	$55.1$	$\mathbf{2,\!677}$	$\mathbf{140}$	$\mathbf{180}$

TABLE IV: Comparison of mAP, NDS in dependence on GFLOPs and latency (ms), on

\mathcal{D}^{\mathrm{val}}_{\mathrm{NS}}

for our proposed token compensator (

\mathbf{TC}

, Fig. 3 (b)) vs. the token compensator by DAR-TR-PEFT [ 23 ] . Both, our approach and DAR-TR-PEFT [ 23 ] are integrated into StreamPETR [ 1 ] employing the SAM-B [ 19 ] encoder. Best results bold. Latency in ms.

Methods

mAP ↑ (%)

NDS ↑ (%)

mATE ↓

mASE ↓

mAOE ↓

mAVE ↓

mAAE ↓

Model Size ↓ (M)

GFLOPs ↓

Latency ↓

Avg. rank ↓

\tau^{\mathrm{E}}

\tau

RayDN [ 16 ]

53.6

\mathbf{62.3}

0.540

0.257

\mathbf{0.250}

\mathbf{0.201}

0.204

\mathbf{331.9}

9,\!787

615

639

3.09

+

ToC3D [ 3 ]

50.9

60.9

0.523

\mathbf{0.251}

\underline{0.254}

0.238

\mathbf{0.190}

339.6

9,\!359

490

510

3.18

+

Ours (

r=0.5

)

\mathbf{54.0}

\mathbf{62.3}

\mathbf{0.519}

\underline{0.252}

0.275

\underline{0.231}

0.193

\underline{333.5}

6,\!396

495

518

\mathbf{2.27}

+

Ours (

r=0.3

)

\underline{53.8}

\underline{62.1}

0.523

\underline{0.252}

0.274

0.233

\underline{0.192}

\underline{333.5}

\underline{5,\!484}

\underline{480}

\underline{503}

\underline{2.45}

+

Ours (

r=0.1

)

53.7

62.1

\underline{0.522}

\underline{0.252}

0.275

0.234

0.193

\underline{333.5}

\mathbf{4,\!909}

\mathbf{442}

\mathbf{464}

\mathbf{2.27}

TABLE V: Generalization study of our proposed method vs. ToC3D on two further 3D object detection methods , RayDN [ 16 ] and Sparse4Dv2 [ 20 ] . Evaluation metrics mAP, NDS, and various 3D object detection errors in dependence on GFLOPs and latencies (ms) are measured with an input resolution of

320\!\times\!800

\mathcal{D}^{\mathrm{val}}_{\mathrm{NS}}

. In each table segment, best results bold, second-best underlined.

Method	Fine-tuned (M)	Total (M)
StreamPETR [ 1 ]	–	$316.6$
$+$ ToC3D [ 3 ]	$324.4$	$+7.8$
$+$ Ours	$\mathbf{1.6}$	$+\mathbf{1.6}$
RayDN [ 16 ]	–	$331.9$
$+$ ToC3D [ 3 ]	$339.6$	$+7.7$
$+$ Ours	$\mathbf{1.6}$	$+\mathbf{1.6}$

TABLE VI: Comparison of trainable and total model parameters of our proposed approach vs. ToC3D on

\mathcal{D}_{\mathrm{NS}}^{\mathrm{val}}

[ 21 ] with an EVA-02-L [ 18 ] encoder on three multi-view 3D object detection methods. Symbol

+

indicates integration into the respective multi-view 3D object detection method. Best results bold.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 13 — Fig. 7: Window-based multi-head self attention (WMHSA) [ 26 ] from Figs. 2 , and 3 . The internal dimension $d_{a}$ refers to the per-head internal dimension.

Multi-View 3D Object Detection

Multi-View 3D Object Detection combines images from multiple cameras to locate and classify objects in three-dimensional space.

See all 25 papers

Revisiting Token Compression for Accelerating ViT-based Sparse Multi-View 3D Object Detectors
Apr 16, 2026Mingqian Ji, Shanshan Zhang, Jian YangMulti-View 3D Object Detection Efficient ViTs
PCA Lab, School of Computer Science and Engineering, Nanjing University of Science and Technology · PCA Lab, School of Intelligence Science and Technology, Nanjing University
SToRe3D: Sparse Token Relevance in ViTs for Efficient Multi-View 3D Object Detection
May 13, 2026Sandro Papais, Lezhou Feng, Charles Cossette +1Multi-View 3D Object Detection Token Pruning
University of Toronto · Zoox Inc.
PQR3D: Progressive Query Refinement over Reference-Conditioned Temporal Windows for Multi-View 3D Object Detection
Sep 26, 2026Hui Ye, Yudong Liu, Yiran Chen +2Multi-View 3D Object Detection 3D Object Detection
Department of Computer Science, Georgia State University · Department of Electrical and Computer Engineering, Duke University · School of Computing, University of Connecticut

Apr 16, 2026J/K move · Enter open · S save

Revisiting Token Compression for Accelerating ViT-based Sparse Multi-View 3D Object Detectors

Mingqian Ji, Shanshan Zhang, Jian Yang

PCA Lab, School of Computer Science and Engineering, Nanjing University of Science and Technology · PCA Lab, School of Intelligence Science and Technology, Nanjing University

Multi-View 3D Object Detection Efficient ViTs 3D Object Detection Visual Token Compression

Open paper AskPDF arXiv

Vision Transformer (ViT)-based sparse multi-view 3D object detectors have achieved remarkable accuracy but still suffer from high inference latency due to heavy token processing. To accelerate these models, token compression has been widely explored. However, our revisit of existing strategies, such as token pruning, merging, and patch size enlargement, reveals that they often discard informative background cues, disrupt contextual consistency, and lose fine-grained semantics, negatively affecting 3D detection. To overcome these limitations, we propose SEPatch3D, a novel framework that dynamically adjusts patch sizes while preserving critical semantic information within coarse patches. Specifically, we design Spatiotemporal-aware Patch Size Selection (SPSS) that assigns small patches to scenes containing nearby objects to preserve fine details and large patches to background-dominated scenes to reduce computation cost. To further mitigate potential detail loss, Informative Patch Selection (IPS) selects the informative patches for feature refinement, and Cross-Granularity Feature Enhancement (CGFE) injects fine-grained details into selected coarse patches, enriching semantic features. Experiments on the nuScenes and Argoverse 2 validation sets show that SEPatch3D achieves up to 57% faster inference than the StreamPETR baseline and 20% higher efficiency than the state-of-the-art ToC3D-faster, while preserving comparable detection accuracy. Code is available at https://github.com/Mingqj/SEPatch3D.

Abstract

Multi-View 3D Object Detection

Explore similar work

Revisiting Token Compression for Accelerating ViT-based Sparse Multi-View 3D Object Detectors

SToRe3D: Sparse Token Relevance in ViTs for Efficient Multi-View 3D Object Detection

PQR3D: Progressive Query Refinement over Reference-Conditioned Temporal Windows for Multi-View 3D Object Detection