CARE: Camera-Residual Reserves for First Sightings in Adaptive LiDAR Sensing

Jiachen Gong; Yun Li; Ehsan Javanmardi; Wencan Mao; Manabu Tsukada

cs.CVAug 25, 2026

CARE: Camera-Residual Reserves for First Sightings in Adaptive LiDAR Sensing

Authors: Jiachen Gong, Yun Li, Ehsan Javanmardi, Wencan Mao, Manabu Tsukada

Organizations: The University of Tokyo, Tokyo, Japan · National Institute of Informatics, Tokyo, Japan

Abstract

Adaptive LiDAR scanning concentrates a limited sensing budget on regions of interest predicted from past object tracks, lowering data volume in autonomous driving while maintaining detection accuracy. However, existing scanning policies face three challenges. First, history-driven approaches depend on past tracks, so unseen objects are detected late or missed. Second, random or uniform sampling outside the predicted regions has no awareness of where new objects appear. Third, camera-guided alternatives spend budget on all camera detections, resampling objects already covered, costing recall in crowded scenes and range when budgets are scarce. This paper introduces the CAmera-REsidual reserve (CARE), a training-free allocation rule that reserves part of a fixed ray budget for the directions of current camera detections that the track forecasts cannot explain; the rest follows the base history policy, and unused reserve returns to a random floor. The paper makes three contributions. First, a leakage-free ray-budget evaluation on nuScenes (150 scenes, 4,148 events) measuring the first-sighting loss of history-driven scanning, with a strict-causal variant using the preceding keyframe. Second, CARE raises first-sighting recall by 5.2, 5.2, and 4.3 points at 10%, 20%, and 35% budgets over the history policy, with paired intervals excluding zero; the camera cue drives this gain, and the first-sighting versus overall trade-off is a budget-dependent Pareto choice. Third, a safety-bounded forgetting module that releases budget from receding or static tracks beyond a speed-dependent guard distance; at tight budgets, forgetting without the guard significantly harms near-field recall, so the guard is what keeps it safe. The pipeline runs end to end on a real vehicle and, in closed-loop simulation, detects an occluded pedestrian earlier and brakes more reliably than history-driven scanning.

Figures & tables

Figure 1: Overview of CARE at one keyframe. Camera detections $D_{t}$ that the projected track forecasts (angular hulls $H_{i}$ ) cannot explain form the residual layer $U$ (Algorithm 2 ), which fills the reserve of the shared allocator (Algorithm 1 ) ahead of the seeded floor; the SBF stage (Algorithm 3 ) shrinks the hulls of low-criticality tracks. The output mask $S_{t}$ is held over the sweeps, and the 3D detections update the track memory.

Figure 2: First-sighting recall versus overall recall at the (a) 10%, (b) 20%, and (c) 35% budgets on nuScenes validation. The gray staircase traces the Pareto frontier; CARE sits at its knee in every panel, and Table 1 gives the paired contrasts.

Figure 3: First-sighting recall (left) and overall recall (right) versus ray-cell budget for CARE, CARE+SBF, and the policy-level re-implementations: history-driven scanning [ 21 ] , the uncertainty reserve [ 2 ] , beam reduction [ 25 ] , the all-camera reserve, and the learned scorer [ 11 ] .

First-sighting recall (points)
Contrast	10%	20%	35%
vs. History [ 21 ]	+5.2 [4.3,6.2]	+5.2 [4.3,6.2]	+4.3 [3.5,5.1]
vs. Uncertainty reserve [ 2 ]	+3.5 [2.6,4.4]	+2.7 [1.7,3.8]	+2.1 [1.1,3.1]
Overall recall (points)
vs. Beam reduction [ 25 ]	+17.4 [15.9,19.0]	+11.5 [10.4,12.7]	+6.4 [5.4,7.5]
vs. Learned scorer [ 11 ]	+2.4 [1.9,3.0]	+0.7 [0.4,1.0]	+0.6 [0.3,0.9]

Table 1: Paired per-scene bootstrap contrasts, CARE minus the named policy, in points of recall with 95% confidence intervals; positive favors CARE; bold marks intervals excluding zero in favor of CARE. First-sighting contrasts not shown are statistically tied, except beam reduction at 10%, where CARE leads by 2.0 points; the significant overall-recall contrasts against history and the uncertainty reserve appear in the text. Cells use one decimal, or two where an endpoint would round to zero; the identical history cells at 10% and 20% differ before rounding. Absolute levels appear in Figure 3 .

	mAP			NDS
Policy	10%	20%	35%	10%	20%	35%
History	0.076	0.097	0.110	0.131	0.164	0.169
All-camera	0.070	0.090	0.107	0.124	0.130	0.162
CARE+SBF (ours)	0.066	0.088	0.106	0.120	0.128	0.162
CARE (ours)	0.068	0.091	0.111	0.124	0.149	0.164

Table 2: Official nuScenes metrics under the ray-cell budgets. Because these metrics average over every in-range object frame, and most of those frames belong to objects the tracker already follows, a gain that is confined to first sightings carries little weight in the average, and CARE stays close to the history policy on mAP while trailing it on NDS. This dilution is the reason we measure first sightings separately. The detector head emits five of the ten protocol classes, so the absolute values are not comparable with standard ten-class reports.

Figure 4: The vehicle platform: (a) drive-by-wire chassis with LiDAR, forward camera, and GNSS; (b) camera view and (c) LiDAR cloud from a separate drive.

Panel of Figure 5 — Figure 5: The dense CARLA scene. (a) The ego follows a street lined with tracked vehicles; the pedestrian crossing from behind the parked truck (dashed track, magnified inset) is the first sighting the reserve must catch. (b) Median first-detection distance of the crosser, five seeds per point, interquartile bars. The all-camera reserve collapses at 3%, spread over the five tracked vehicles; CARE and CARE+SBF match the full-scan bound throughout, so the three top curves coincide.

Explore similar work

Mar 2, 2026

SCATR: Mitigating New Instance Suppression in LiDAR-based Tracking-by-Attention via Second Chance Assignment and Track Query Dropout

LiDAR-based tracking-by-attention (TBA) frameworks inherently suffer from high false negative errors, leading to a significant performance gap compared to traditional LiDAR-based tracking-by-detection (TBD) methods. This paper introduces SCATR, a novel LiDAR-based TBA model designed to address this fundamental challenge systematically. SCATR leverages recent progress in vision-based tracking and incorporates targeted training strategies specifically adapted for LiDAR. Our work's core innovations are two architecture-agnostic training strategies for TBA methods: Second Chance Assignment and Track Query Dropout. Second Chance Assignment is a novel ground truth assignment that concatenates unassigned track queries to the proposal queries before bipartite matching, giving these track queries a second chance to be assigned to a ground truth object and effectively mitigating the conflict between detection and tracking tasks inherent in tracking-by-attention. Track Query Dropout is a training method that diversifies supervised object query configurations to efficiently train the decoder to handle different track query sets, enhancing robustness to missing or newborn tracks. Experiments on the nuScenes tracking benchmark demonstrate that SCATR achieves state-of-the-art performance among LiDAR-based TBA methods, outperforming previous works by 7.6% AMOTA and successfully bridging the long-standing performance gap between LiDAR-based TBA and TBD methods. Ablation studies further validate the effectiveness and generalization of Second Chance Assignment and Track Query Dropout. Code can be found at the following link: https://github.com/TRAILab/SCATR

Brian Cheong, Letian Wang, Sandro Papais +1

University of Toronto

Multi-Object Tracking 3D MOT

Ask PDF

May 11, 2026

Think as Needed: Geometry-Driven Adaptive Perception for Autonomous Driving

Autonomous driving scenes range from empty highways to dense intersections with dozens of interacting road users, yet current 3D detection models apply a fixed computation budget to every frame, wasting resources on simple scenes while lacking capacity for complex ones. Existing approaches compound this problem: Transformer-based interaction models scale quadratically with the number of detected objects, and frame-by-frame processing causes the system to immediately forget objects the moment they become occluded. We propose Enhanced HOPE, an adaptive perception architecture that measures the geometric complexity of each incoming LiDAR frame using an unsupervised statistical estimator and routes it through a shallow or deep processing path accordingly, requiring no manual scene labels. To keep interaction modeling efficient, we replace quadratic pairwise attention with a linear-time subspace-based network that groups nearby objects into clusters and processes them jointly. The computational savings from these two mechanisms free up resources for a persistent temporal memory module that retains previously detected objects and traffic rules across frames, enabling the system to recall occluded objects seconds after they disappear from view. On the nuScenes and CARLA benchmarks, Enhanced HOPE reduces latency by 38% on simple scenes with no accuracy loss, improves mean Average Precision by 2.7 points on rare long-tail scenarios, and tracks objects through occlusions lasting over 5 seconds, where all tested baselines fail.

Donghyun Kim, Jaehyoung Park

Stony Brook University

Adaptive Inference Lidar-Based 3D Object Detection Autonomous Driving Perception 3D Object Detection Multi-Object Tracking

Ask PDF

Jun 17, 2026

C-ARC: Continuous-Adaptive Range Clustering for Non-Repetitive LiDAR Sensors

Real-time LiDAR clustering identifies structures in point clouds, which is an essential prerequisite for many mobile robotics algorithms. Current methods are mostly developed for repetitive mechanical LiDAR sensors. Recently, the use of non-repetitive LiDAR sensors is strongly increasing due to their small cost and form factor. Such non-repetitive Risley prism-based sensors violate two key assumptions of repetitive mechanical sensors: structured scan lines and well-defined frame boundaries. Their Rhodonea-curve trajectories produce non-uniform point distributions, and the absence of a rotation cycle renders conventional scan line indexing inapplicable. To meet such new requirements, we developed C-ARC, a Continuous-Adaptive Range Clustering framework that maintains a persistent dual-graph over a sliding window, decoupling high-frequency point insertion from on-demand cluster retrieval. This is crucial for key functionalities like SLAM or tracking. An adaptive range grid resolution mechanism calibrates grid dimensions at initialization using an exponential control loop, balancing the sparsity-collision trade-off without prior knowledge of the scanning pattern. Implemented as an open-sourced single-threaded C++17 library, C-ARC produces real-time cluster output at 20 Hz on commodity hardware for the Livox Mid-360. Evaluation on the Livox Avia identifies unbounded cell occupancy as the primary limitation for sensors with strongly concentrated scan patterns. The adaptive resolution mechanism additionally improves clustering quality for existing grid-based methods on non-repetitive data.

Nick B. Schroeder, Jonathan Lichtenfeld, Oskar von Stryk

Technical University of Darmstadt, Darmstadt, Germany · Technical University of Darmstadt, Hochschulstr. 10, 64289 Darmstadt, Germany. · Simulation, Systems Optimization and Robotics Group, Technical University of Darmstadt, Darmstadt, Germany +1

Clustering Point Cloud Understanding

Ask PDF