Test-time Correction: An Online 3D Detection System via Visual Prompting

Hanxue Zhang; Zetong Yang; Yanan Sun; Li Chen; Fatma Güney; Hongyang Li

cs.CVDec 10, 2024

Test-time Correction: An Online 3D Detection System via Visual Prompting

Authors: Hanxue Zhang, Zetong Yang, Yanan Sun, Li Chen, Fatma Güney, Hongyang Li

Organizations: Shanghai Jiao Tong University, Shanghai, China · Shanghai AI Lab, Shanghai, China · GAC R&D Center, Guangzhou, China · The University of Hong Kong, Hong Kong, China ·

Abstract

This paper introduces Test-time Correction (TTC), an online 3D detection system designed to rectify test-time errors using various auxiliary feedback, aiming to enhance the safety of deployed autonomous driving systems. Unlike conventional offline 3D detectors that remain fixed during inference, TTC enables immediate online error correction without retraining, allowing autonomous vehicles to adapt to new scenarios and reduce deployment risks. To achieve this, we equip existing 3D detectors with an Online Adapter (OA) module -- a prompt-driven query generator for real-time correction. At the core of OA module are visual prompts: image-based descriptions of objects of interest derived from auxiliary feedback such as mismatches with 2D detections, road descriptions, or user clicks. These visual prompts, collected from risky objects during inference, are maintained in a visual prompt buffer to enable continuous correction in future frames. By leveraging this mechanism, TTC consistently detects risky objects, achieving reliable, adaptive, and versatile driving autonomy. Extensive experiments show that TTC significantly improves instant error rectification over frozen 3D detectors, even under limited labels, zero-shot settings, and adverse conditions. We hope this work inspires future research on post-deployment online rectification systems for autonomous driving.

Figures & tables

Fig. 1 : Comparison of Error Correction between the conventional offline loop (left) and the new proposed online TTC System (right). Offline error correction pipeline improves model capability during the development stage, which typically requires expensive workloads and computational overhead over days or weeks for model updates. While TTC system additionally enables deployed 3D detectors with on-the-fly error rectification ability.

Fig. 2 : Visual prompts could be arbitrary views of objects, across zones, styles, timestamps, etc .

Fig. 3 : Overall Framework. (Left:) The TTC system centers on a TTC-3D Detector which utilizes visual prompts $\mathcal{P}_{v}$ from the visual prompt buffer for test-time error rectification. (Right:) The TTC-3D Detector can be based on any traditional detector (BEV or monocular). It supports 3D detection from any combination of four prompts, i.e . , object $\mathcal{P}_{o}$ , box $\mathcal{P}_{b}$ , point $\mathcal{P}_{p}$ , and novel visual prompts $\mathcal{P}_{v}$ , arbitrary views of target objects across scenarios and timestamps.

Fig. 4 : Concrete mechanism of visual prompt alignment. This figure illustrates monocular input. When multi-view images are employed, this alignment operation flattens the different views and still generates $N$ peak candidate positions.

Method	Backbone	Type	mAP (%) $\uparrow$	EDS (%) $\uparrow$	FP/[email protected] $\downarrow$	FN/[email protected] $\downarrow$
MonoDETR [ 73 ]	R101	Monocular	42.6	50.1	49.9	1.2
TTC-MonoDETR	R101	Monocular	45.0 ( +2.4 )	51.7 ( +1.6 )	49.9	0.9
MV2D [ 63 ]	R50	Multiview	43.0	52.8	32.2	2.0
TTC-MV2D	R50	Multiview	52.7 ( +9.7 )	57.2 ( +4.4 )	32.2	0.8
Sparse4Dv2 [ 34 ]	R50	Multiview + Temporal	44.4	53.2	42.7	1.9
TTC-Sparse4Dv2	R50	Multiview + Temporal	54.2 ( +9.8 )	57.8 ( +4.6 )	42.7	0.7

TABLE 1: Effect of TTC on various 3D detectors. TTC improves test-time performance of offline-trained detectors during online inference without extra training. To compare directly with the baselines’ official scores, we set the confidence threshold to 0.01. For false-alarm analysis, we report FP/frame and FN/frame .

TABLE 2: Experiments on out-of-training-distribution scenarios. TTC system achieves substantial gains with limited or even no labels under different challenging test cases.

Fig. 5 : Qualitative visualization of real-world scenes (collected from YouTube). We visualize the zero-shot 3D detection results in a real-world scenario. In this case, the prompt buffer contains a visual prompt of a deer. Higher responses from the visual prompt alignment are highlighted by brighter colors. As shown, although trained solely on nuScenes , TTC system can still accurately localize “unseen” objects in the input image. Best viewed in color.

Fig. 7 : Experiments on visual prompts from arbitrary temporal frames. TTC-3D Detectors can effectively locate and detect target objects in future frames, using its image patch at Frame #0.

Fig. 8 : Robustness over positional perturbations. We evaluate TTC-MonoDETR under controlled perturbations during test-time inference. Specifically, we introduce translation offsets to the ground-truth 2D centers with maximum ratios of 0%, 10%, 20%, 30%, and 40%.

Feedback Interval $N$	0	2	4	6
TTC-BEVFormer	47.8	47.5	47.3	46.9
TTC-Sparse4Dv2	53.5	53.1	52.4	51.8
TTC-MV2D	51.0	50.7	50.2	49.6

TABLE 4: mAP comparisons under different feedback collection intervals. “N=0” means feedback is collected every frame, while “N=2” means every 2 frames.

Prompt Type	mAP $\uparrow$	EDS $\uparrow$
Vehicle	20.9	45.3
Vehicle + Animal	20.9	45.3

TABLE 5: Robustness to noisy visual prompts. Comparison of fixed-buffer TTC-MonoDETR † with and without off-topic animal prompts (dog & kangaroo). Evaluation follow the protocol of Table 3 .

Method	mAP $\uparrow$	EDS $\uparrow$	FP/frame $\downarrow$	FN/frame $\downarrow$
BEVFormer	40.2	49.6	34.9	1.9
TTC-BEVFormer	45.9 ( +5.7 )	51.9 ( +2.3 )	35.0	1.2

TABLE 6: Robustness to rapidly changing prompts. We choose BEVFormer as the base detector.

Method	mAP (%) $\uparrow$	EDS (%) $\uparrow$
MV2D	42.6	51.9
SOT + MV2D	42.1	51.2
TTC-MV2D	50.0	57.3

TABLE 7: Comparison of error correction performance with SOT + 3D detector head. The results demonstrate the superior performance of TTC in online error correction, highlighting the limitations of the baseline approach where simply combining SOT with a 3D detection head does not effectively address error correction.

Feedback Interval $N$	0	2	4	6
MV2D + Offline fine-tune	43.4	42.7	41.0	39.9
TTC-MV2D	51.0	50.7	50.2	49.6

TABLE 8: mAP comparison between TTC-MV2D and MV2D fine-tuned with 2D feedback annotations. “N=0” means collecting feedback at every frame, while “N=2” means every 2 frames. TTC-MV2D consistently outperforms the offline fine-tuned MV2D.

Method	LL3DA	MonoDETR	TTC-MonoDETR
#Params $\downarrow$	118M	68M	80M
FPS (Hz) $\uparrow$	0.42	11.1	9.1

TABLE 9: Parameters and latency comparisons between LLM based 3D detectors, traditional 3D detectors, and related TTC 3D detectors.

Method	nuScenes	Waymo
MonoDETR	42.6	0.7
TTC-MonoDETR	45.0	1.0
K-sync + MonoDETR	42.8	30.8
K-sync + TTC-MonoDETR	45.8 ( +3.0 )	34.1 ( +3.3 )

TABLE 10: mAP results of cross-dataset validation. The model is trained on nuScenes and evaluated on the nuScenes and Waymo. MonoDETR is used as the base detector.

Method	mAP $\uparrow$	EDS $\uparrow$
BEVFormer	41.1	50.2
TTC-BEVFormer (GDINO)	41.8 ( +0.7 )	50.4 ( +0.2 )
TTC-BEVFormer (YOLOv8)	42.4 ( +1.3 )	50.8 ( +0.6 )

TABLE 11: Automated feedback results. 2D–3D misalignment is computed between 2D boxes (Grounding DINO / YOLOv8-nuScenes) and the image reprojection of 3D predictions; unmatched 2D boxes are converted into visual prompts.

Sim. Loss	Loc. Loss	mAP (%)	EDS (%)
-	-	32.6	44.2
$\surd$	-	39.4	48.5
$\surd$	$\surd$	43.3	52.3

TABLE 12: Effect of Visual Prompt Alignment. We select TTC-MonoDETR for this experiment.

No. of Position Predictions	mAP (%) $\uparrow$	Recall (%) $\uparrow$
1	39.9	62.1
4	43.3	69.1
8	43.0	69.4

TABLE 14: Effect of the number of predicted positions

N

of visual prompt alignment. We select TTC-MonoDETR for this experiment.

Fig. 10 : Size of visual prompt buffer during the video stream. Visual prompt buffer stores the missed objects during online inference to rectify test-time errors of deployed 3D detectors. It can adaptively manage the stored prompts and thus maintain the balance between latency and accuracy.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 11 : Intrinsic mismatch across datasets. Equal-size image patches around cars at comparable 3D distance/shape from nuScenes (left) and Waymo (right). Despite similar geometry, the apparent 2D scale differs markedly due to different focal lengths and camera rigs, inducing depth bias and explaining why naive cross-dataset transfer of vision-only 3D detectors degrades severely. Adapted from [ 76 ] .

Fig. 12 : Outcomes of visual prompts and their FP impact. Final predictions consist of dense-query results and visual-prompt results. Visual prompts produce few above-threshold candidates, and those candidates fall into five cases (i–v). After confidence filtering and NMS, only a small portion contributes to false positives (v), explaining why FP/frame remains near baseline.

Method	FP/[email protected] $\downarrow$	FN/[email protected] $\downarrow$
MonoDETR	1.5	2.5
TTC-MonoDETR	1.5	2.1
MV2D	1.3	2.9
TTC-MV2D	1.3	2.2
Sparse4Dv2	1.1	3.0
TTC-Sparse4Dv2	1.1	2.2

Appendix

TABLE 15: False-alarm metrics at a deployment-oriented confidence threshold (

\tau{=}0.3

Fig. 13 : Visualizations on nuScenes scenarios with in-domain visual prompts of labeled objects. TTC system enables continuous 3D detection and tracking based on visual prompts. The images in the first row indicate the visual prompts in prompt buffer, and images in other rows represent 3D detection results prompted by the corresponding visual prompts. Different identities are indicated with different colors.

Fig. 14 : Visualizations on nuScenes scenarios with in-domain visual prompts of labeled objects. TTC system enables continuous 3D detection and tracking based on visual prompts. The images in the first row indicate the visual prompts in prompt buffer, and images in other rows represent 3D detection results prompted by the corresponding visual prompts. Different identities are indicated with different colors.

Fig. 15 : Visualizations on nuScenes scenarios with in-domain visual prompts of un-labeled objects. TTC system enables 3D detection and tracking of “novel” objects unseen during training. The image in the first row indicates the visual prompt in the prompt buffer, and images in other rows represent 3D detection results prompted by the corresponding visual prompts. Interestingly, with the one-to-N mapping mechanism of the visual prompt alignment, TTC system can detect multiple objects with similar visual descriptions to the visual prompt simultaneously.

Fig. 16 : Visualizations on nuScenes scenarios with in-domain visual prompts of un-labeled objects. TTC system enables 3D detection and tracking of “novel” objects unseen during training. The image in the first row indicates the visual prompt in the prompt buffer, and images in other rows represent 3D detection results prompted by the corresponding visual prompts.

Fig. 17 : Visualizations on nuScenes scenarios with in-domain visual prompts of un-labeled objects. TTC system enables 3D detection and tracking of “novel” objects unseen during training. The image in the first row indicates the visual prompt in the prompt buffer, and images in other rows represent 3D detection results prompted by the corresponding visual prompts. Interestingly, with the one-to-N mapping mechanism of the visual prompt alignment, TTC system can detect multiple objects with similar visual descriptions to the visual prompt simultaneously.

Fig. 18 : Visualizations on nuScenes scenarios with out-domain visual prompts of labeled objects. TTC system can perform 3D detection and tracking via visual prompts with arbitrary styles ( imagery style ). The image in the first row indicates the visual prompt in the prompt buffer, and images in other rows represent 3D detection results prompted by the corresponding visual prompts. With the one-to-N mapping mechanism of the visual prompt alignment, TTC detectors can detect multiple objects with similar visual descriptions to the visual prompt at the same time.

Fig. 19 : Visualizations on nuScenes scenarios with out-domain visual prompts of labeled objects. TTC system can perform 3D detection and tracking via visual prompts with arbitrary styles ( Lego style ). The image in the first row indicates the visual prompt in the prompt buffer, and images in other rows represent 3D detection results prompted by the corresponding visual prompts. With the one-to-N mapping mechanism of the visual prompt alignment, TTC detectors can detect multiple objects with similar visual descriptions to the visual prompt at the same time.

Fig. 20 : Visualizations on real world scenarios with out-of-domain visual prompts. At here, we show the similarity map from visual prompt alignment with visual prompts of arbitrary objects unseen during training. Brighter colors highlight higher responses. As illustrated, our method works well in novel scenarios with visual prompts of arbitrary object-of-interests. Best viewed in color.

Fig. 21 : Visualizations on real world scenarios with out-of-domain visual prompts. At here, we show the similarity map from visual prompt alignment with visual prompts of arbitrary objects unseen during training. Brighter colors highlight higher responses. As illustrated, our method works well in novel scenarios with visual prompts of arbitrary object-of-interests. Best viewed in color.

Fig. 22 : Visualizations on real world scenarios with out-of-domain visual prompts. This figure demonstrates the TTC’s effectiveness in real-world 3D detection, even with visual prompts of unseen objects. demonstrate a case of real-world detection As illustrated, our method works well in real-world scenarios with visual prompts of unseen objects. Different object identities are indicated by distinct colors.

Fig. 23 : Visualizations on real world scenarios with out-of-domain visual prompts. This figure demonstrates a case of real-world 3D object detection. We provide the visual prompt from the same scene as the target object, though it is unseen during training, and our TTC system then detects the target objects in the subsequent video frames.

Fig. 24 : Visualizations on real world scenarios with out-of-domain visual prompts. This figure demonstrates a case of real-world 3D object detection. We provide a visual prompt from a separate image as the pre-defined prompt and visualize it at the beginning of the sequence. As described, our TTC system can successfully detect the "Deer" object using this stylized deer prompt sourced from the internet. This example highlights the TTC’s capability to effectively leverage diverse, user-supplied visual prompts to accurately identify target objects, even when the prompts are not directly from the same scene.

Fig. 25 : Visualizations on real world scenarios with out-of-domain visual prompts. Another case demonstrates the use of an internet-sourced "Deer" visual prompt to detect the deer in a different real-world scenario. As shown, our TTC system effectively detects the deer even when it is partially obscured by snow. This example further illustrates the robust performance of the TTC framework in accurately localizing target objects, even in challenging environmental conditions, by leveraging flexible visual prompts provided by users.

Autonomous Driving Perception

Autonomous driving perception enables vehicles to detect and interpret road users, drivable space, and hazards using onboard and shared sensors.

See all 381 papers

ModalPatch: A Plug-and-Play Module for Robust Multi-Modal 3D Object Detection under Modality Drop
Mar 3, 2026Shuangzhi Li, Lei Ma, Xingyu Li3D Object Detection Missing-Modality Learning
University of Alberta, Canada · The University of Tokyo, Japan
RECO: Region-Aware Compensation for Extrinsic Perturbations in Roadside 3D Detection
Jul 23, 2026Junsheng Du, Zhaocheng He, Yuhuan LuAutonomous Driving Perception 3D Object Detection
School of Intelligent Systems Engineering, Sun Yat-sen University, Shenzhen, China · Faculty of Applied Sciences, Macao Polytechnic University, Macao SAR, China
FogTTA: Online Test-Time Adaptation for Robust Transformer-based Object Detection in Foggy Weather
Oct 21, 2026Ali Abedi, Q. M. Jonathan Wu, Ning Zhang +1Object Detection Continual Test-Time Adaptation
University of Windsor · University of Toronto Mississauga

Mar 3, 2026J/K move · Enter open · S save

ModalPatch: A Plug-and-Play Module for Robust Multi-Modal 3D Object Detection under Modality Drop

Shuangzhi Li, Lei Ma, Xingyu Li

University of Alberta, Canada · The University of Tokyo, Japan

3D Object Detection Missing-Modality Learning Multimodal Sensor Fusion

Open paper AskPDF arXiv

Multi-modal 3D object detection is pivotal for autonomous driving, integrating complementary sensors like LiDAR and cameras. However, its real-world reliability is challenged by transient data interruptions and missing, where modalities can momentarily drop due to hardware glitches, adverse weather, or occlusions. This poses a critical risk, especially during a simultaneous modality drop, where the vehicle is momentarily blind. To address this problem, we introduce ModalPatch, the first plug-and-play module designed to enable robust detection under arbitrary modality-drop scenarios. Without requiring architectural changes or retraining, ModalPatch can be seamlessly integrated into diverse detection frameworks. Technically, ModalPatch leverages the temporal nature of sensor data for perceptual continuity, using a history-based module to predict and compensate for transiently unavailable features. To improve the fidelity of the predicted features, we further introduce an uncertainty-guided cross-modality fusion strategy that dynamically estimates the reliability of compensated features, suppressing biased signals while reinforcing informative ones. Extensive experiments show that ModalPatch consistently enhances both robustness and accuracy of state-of-the-art 3D object detectors under diverse modality-drop conditions.

Abstract

Autonomous Driving Perception

Explore similar work

ModalPatch: A Plug-and-Play Module for Robust Multi-Modal 3D Object Detection under Modality Drop

RECO: Region-Aware Compensation for Extrinsic Perturbations in Roadside 3D Detection

FogTTA: Online Test-Time Adaptation for Robust Transformer-based Object Detection in Foggy Weather