EvoMem3D: Adaptive-Depth Temporal Representation Memory for Multi-Camera 3D Detection
Organizations: Google Research
Abstract
Modern multi-camera 3D detectors increasingly rely on large Vision Transformer (ViT) image encoders, making repeated full-depth encoding a dominant inference cost even though adjacent driving frames are highly redundant. We present EvoMem3D, an adaptive-depth temporal framework that caches the most recent full-depth detector-interface appearance representation as a read-only anchor and combines it with policy-selected partial features from the current frame. A lightweight pre-encoding controller chooses among semantic encoder exits before image processing, allocating shallow computation to stable frames and deeper computation when the visual state changes. To reuse appearance without carrying stale geometry, we decouple temporal appearance from geometric alignment: current-frame calibration defines the decoder geometry at every timestamp, while a motion-conditioned positional adapter accounts for anchor-to-current ego motion. On nuScenes, EvoMem3D achieves 61.50 NDS and 52.76 mAP while reducing end-to-end latency from 364.4 to 168.7 ms (x2.16) relative to standard dense DETR3D. With StreamPETR, it reaches 61.36 NDS and 52.55 mAP at a x1.92 speedup. Controlled studies isolate the effects of current evidence, temporal reuse, memory correction, and adaptive routing, while our audit confirms that the policy learns to distinguish semantic driving regimes.
Figures & tables
| Method | NDS | mAP | GFLOPs | Latency (ms) | Speedup |
| Spatial Token Compression SOTA (StreamPETR ( Wang et al., 2023 ) ViT-L Baseline) | |||||
| Dense Baseline | 61.18 | 52.10 | 4601 | 350.6 | 1.00 |
| ToC3D-Fast ( Zhang et al., 2024 ) | 60.90 | 51.70 | 3935 | 281.0 | 1.13 |
| ToC3D-Faster ( Zhang et al., 2024 ) | 60.50 | 51.30 | 3131 | 237.2 | 1.34 |
| SEPatch3D-Fast ( Ji et al., 2026 ) | 61.20 | 52.10 | 4199 | 250.2 | 1.27 |
| SEPatch3D-Faster ( Ji et al., 2026 ) | 60.30 | 51.60 | 3023 | 194.3 | 1.63 |
| Method | NDS | mAP | mATE | mASE | mAOE | mAVE | mAAE |
|---|---|---|---|---|---|---|---|
| DETR3D | 60.81 | 51.19 | 0.5420 | 0.2578 | 0.2809 | 0.2440 | 0.1922 |
| SEPatch3D-Fast | 61.20 | 52.10 | 0.557 | 0.253 | 0.270 | 0.236 | 0.199 |
| SEPatch3D-Faster | 60.30 | 51.60 | 0.574 | 0.258 | 0.288 | 0.236 | 0.198 |
| EvoMem3D | 61.50 | 52.76 | 0.5267 | 0.2518 | 0.2726 | 0.2351 | 0.1973 |
| Policy | NDS | mAP | GFLOPs | Lat. (ms) | Speedup |
|---|---|---|---|---|---|
| n/a | 61.90 | 52.5 | 5229 | 373.4 | 1.00 |
| Reuse rep. | 36.58 | 21.48 | 1809.0 | 154.8 | |
| Always Early | 55.75 | 45.59 | 3176.9 | 161.9 | |
| Fixed Early | 58.23 | 48.98 | 3220.7 | 163.5 | |
| Fixed depth cycle | 59.30 | 50.05 | 3562.9 | 206.3 | |
| PolicyNet | 61.50 | 52.76 | 4115 | 168.7 |
| Policy features | MAE | |
|---|---|---|
| Online metadata | ||
| Online + current Early probe | ||
| Offline metadata | ||
| Offline + current Early probe |
| Configuration | Lat. (ms) | Peak Alloc. | Peak Res. | GFLOPs |
|---|---|---|---|---|
| DETR3D | 373.40 | 1781 | 2850 | 5229 |
| EvoMem3D | 168.71 | 1918 | 2716 | 4115 |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Routing | Non-full-frame computation | NDS | mAP | GFLOPs | Latency (ms) | FPS | Speedup |
|---|---|---|---|---|---|---|---|
| Full depth | - | 61.90 | 52.50 | 5229 | 373.4 | 2.68 | |
| Naive reuse | Anchor only | 36.58 | 21.48 | 1809.0 | 104.8 | 9.54 | |
| Partial only | Early | 55.75 | 45.59 | 3176.9 | 121.9 | 8.2 | |
| Partial only | Intermediate | 59.46 | 50.55 | 3715.8 | 197.9 | 5.05 | |
| Partial only | Late | 60.81 | 51.96 | 4254.6 | 236.1 | 4.24 | |
| Memory update | Fixed Early | 58.23 | 48.98 | 3220.7 | 163.5 | 6.18 |
| (a) Controller signal and constraints | |||||
|---|---|---|---|---|---|
| Variant | Test signal | Age bound | NDS | mAP | Lat. (ms) |
| Online | Metadata | ✗ | 55.21 | 44.86 | 226.91 |
| Online + age bound | Metadata | ✓ | 57.94 | 48.57 | 212.28 |
| Current-GT oracle | Current GT | – | 57.17 | 47.24 | – |
| Previous-GT oracle | Previous GT | – | 57.38 | 47.48 | – |
| Prev.-output summary | Previous prediction | ✓ | 60.05 | 50.61 | 193.35 |
| Class | AP | ATE | ASE | AOE | AVE | AAE |
|---|---|---|---|---|---|---|
| Car | 69.30 | 0.367 | 0.145 | 0.056 | 0.225 | 0.189 |
| Truck | 47.73 | 0.575 | 0.191 | 0.067 | 0.195 | 0.205 |
| Construction vehicle | 21.08 | 0.846 | 0.414 | 0.873 | 0.133 | 0.361 |
| Bus | 55.58 | 0.623 | 0.189 | 0.049 | 0.418 | 0.281 |
| Trailer | 28.85 | 0.863 | 0.229 | 0.341 | 0.165 | 0.164 |
| Barrier | 64.54 | 0.323 | 0.279 | 0.141 | – | – |
| Non-full-frame operation | NDS | mAP | FPS |
|---|---|---|---|
| Partial only | 55.75 | 45.59 | 6.12 |
| Gated residual | 58.23 | 48.98 | 6.12 |
| Cross-attention | 54.33 | 43.58 | 5.95 |
| Gated cross-attention | 57.61 | 48.30 | 5.91 |
| Per-camera attention | 55.04 | 44.30 | 5.99 |
| Inverted attention | 52.91 | 42.49 | 5.90 |
Multi-View 3D Object Detection
Multi-View 3D Object Detection combines images from multiple cameras to locate and classify objects in three-dimensional space.
See all 28 papersPQR3D: Progressive Query Refinement over Reference-Conditioned Temporal Windows for Multi-View 3D Object Detection
Efficient Multi-View 3D Object Detection by Dynamic Token Selection and Fine-Tuning
ToC3D for efficient multi-view ViT-based 3D object detection employs ego-motion-based relevant token selection. However, there are two key limitations: (1) The fixed layer-individual token selection ratios limit computational efficiency during both training and inference. (2) Full end-to-end retraining of the ViT backbone is required for the multi-view 3D object detection method. In this work, we propose an image token compensator combined with a token selection for ViT backbones to accelerate multi-view 3D object detection. Unlike ToC3D, our approach enables dynamic layer-wise token selection within the ViT backbone. Furthermore, we introduce a parameter-efficient fine-tuning strategy, which trains only the proposed modules, thereby reducing the number of fine-tuned parameters from more than million (M) to only M. Experiments on the large-scale NuScenes dataset across three multi-view 3D object detection approaches demonstrate that our proposed method decreases computational complexity (GFLOPs) by ... , inference latency (on an NVIDIA-GV100 GPU) by ... , while still improving mean average precision by ... absolute and NuScenes detection score by ... absolute compared to so-far SOTA ToC3D.