Modern multi-camera 3D detectors increasingly rely on large Vision Transformer (ViT) image encoders, making repeated full-depth encoding a dominant inference cost even though adjacent driving frames are highly redundant. We present EvoMem3D, an adaptive-depth temporal framework that caches the most recent full-depth detector-interface appearance representation as a read-only anchor and combines it with policy-selected partial features from the current frame. A lightweight pre-encoding controller chooses among semantic encoder exits before image processing, allocating shallow computation to stable frames and deeper computation when the visual state changes. To reuse appearance without carrying stale geometry, we decouple temporal appearance from geometric alignment: current-frame calibration defines the decoder geometry at every timestamp, while a motion-conditioned positional adapter accounts for anchor-to-current ego motion. On nuScenes, EvoMem3D achieves 61.50 NDS and 52.76 mAP while reducing end-to-end latency from 364.4 to 168.7 ms (x2.16) relative to standard dense DETR3D. With StreamPETR, it reaches 61.36 NDS and 52.55 mAP at a x1.92 speedup. Controlled studies isolate the effects of current evidence, temporal reuse, memory correction, and adaptive routing, while our audit confirms that the policy learns to distinguish semantic driving regimes.
Figures & tables
Figure 1: Adaptive-Depth Temporal Representation Memory. While standard multi-camera 3D detectors execute a full-depth vision transformer for every frame, EvoMem3D treats the dense detector-interface representation as a recurrent memory. A lightweight policy dynamically allocates transformer depth based on ego motion and scene change, updating the persistent high-level memory with shallow current-frame evidence.
Figure 2: Method overview. Before image encoding, the controller selects a semantic exit using deployable temporal signals. A full-depth appearance representation serves as the temporal reference; partial actions combine it with projected current features to produce At . Relative ego motion conditions only the current decoder input.
Figure 3: Accuracy–latency trade-off comparing methods using ViT-L. Triangle is based on StreamPETR, and circle based on DETR3D base model.
Method
NDS
mAP
GFLOPs
Latency (ms)
Speedup
Spatial Token Compression SOTA (StreamPETR ( Wang et al., 2023 ) ViT-L Baseline)
Dense Baseline
61.18
52.10
4601
350.6
1.00 ×
ToC3D-Fast ( Zhang et al., 2024 )
60.90
51.70
3935
281.0
1.13 ×
ToC3D-Faster ( Zhang et al., 2024 )
60.50
51.30
3131
237.2
1.34 ×
SEPatch3D-Fast ( Ji et al., 2026 )
61.20
52.10
4199
250.2
1.27 ×
SEPatch3D-Faster ( Ji et al., 2026 )
60.30
51.60
3023
194.3
1.63 ×
Table 1: Comparison of accuracy, latency, and inference speedup on the nuScenes validation set ( 320×800 ). We compare EvoMem3D against recent state-of-the-art spatial token compression architectures. To ensure a fair evaluation across different hardware setups, we group methods by their testing environment (indicated by their dense baseline latency) and report their relative speedup 1 against their respective full-encoding baseline.
Method
NDS ↑
mAP ↑
mATE ↓
mASE ↓
mAOE ↓
mAVE ↓
mAAE ↓
DETR3D
60.81
51.19
0.5420
0.2578
0.2809
0.2440
0.1922
SEPatch3D-Fast
61.20
52.10
0.557
0.253
0.270
0.236
0.199
SEPatch3D-Faster
60.30
51.60
0.574
0.258
0.288
0.236
0.198
EvoMem3D
61.50
52.76
0.5267
0.2518
0.2726
0.2351
0.1973
Table 2: Detailed nuScenes detection metrics. We report the standard DETR3D baseline, published SEPatch3D operating points, and EvoMem3D . Higher NDS/mAP is better; lower true-positive errors are better. Published SEPatch3D results use a different detector environment and are included for context rather than as a matched controlled comparison. Metric definitions are provided in Appendix C.
Policy
NDS
mAP
GFLOPs
Lat. (ms)
Speedup
n/a
61.90
52.5
5229
373.4
1.00 ×
Reuse rep.
36.58
21.48
1809.0
154.8
2.41×
Always Early
55.75
45.59
3176.9
161.9
2.31×
Fixed Early
58.23
48.98
3220.7
163.5
2.28×
Fixed depth cycle
59.30
50.05
3562.9
206.3
1.81×
PolicyNet
61.50
52.76
4115
168.7
2.21×
Table 3: Policy analysis using DETR3D on nuScenes val accuracy, computational cost, and measured latency at 320×800 .
Figure 4: Learned behavioral regimes of the pre-encoding controller. Each row displays a representative driving sequence from one of the four behavioral clusters. Each timestamp shows the front-camera view, BEV prediction, selected encoder depth, policy confidence, and anchor age.
Policy features
R2
MAE
Online metadata
0.706±0.043
0.0391±0.0020
Online + current Early probe
0.902±0.021
0.0216±0.0010
Offline metadata
0.853±0.037
0.0268±0.0014
Offline + current Early probe
0.952±0.010
0.0149±0.0006
Table 4: Scene-grouped regression of adjacent full-feature cosine similarity. Values are mean ± standard deviation across five folds. Offline metadata uses annotations and is an oracle.
Configuration
Lat. (ms) ↓
Peak Alloc. ↓
Peak Res. ↓
GFLOPs ↓
DETR3D
373.40
1781
2850
5229
EvoMem3D
168.71
1918
2716
4115
Table 5: Hardware footprint and resource profiling on nuScenes val. We report end-to-end per-frame latency, peak CUDA allocated/reserved memory (MB), and computational cost under the same six-camera, 320×800 eval protocol.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Routing
Non-full-frame computation
NDS
mAP
GFLOPs
Latency (ms)
FPS
Speedup
Full depth
-
61.90
52.50
5229
373.4
2.68
1.00×
Naive reuse
Anchor only
36.58
21.48
1809.0
104.8
9.54
3.55×
Partial only
Early
55.75
45.59
3176.9
121.9
8.2
3.06×
Partial only
Intermediate
59.46
50.55
3715.8
197.9
5.05
1.89×
Partial only
Late
60.81
51.96
4254.6
236.1
4.24
1.58×
Memory update
Fixed Early
58.23
48.98
3220.7
163.5
6.18
2.28×
Appendix
Table 6: Controlled depth-routing comparison. All methods use the same detector checkpoint and refresh constraints. “Partial only” uses the current partial representation without the cached anchor, whereas “Anchor update” combines it with the latest full-depth anchor.
(a) Controller signal and constraints
Variant
Test signal
Age bound
NDS
mAP
Lat. (ms)
Online
Metadata
✗
55.21
44.86
226.91
Online + age bound
Metadata
✓
57.94
48.57
212.28
Current-GT oracle
Current GT
–
57.17
47.24
–
Previous-GT oracle
Previous GT
–
57.38
47.48
–
Prev.-output summary
Previous prediction
✓
60.05
50.61
193.35
Appendix
Table 7: Controller ablations. (a) isolates controller information and inference-time constraints. Ground-truth rows are oracle diagnostics and are not deployable. (b) evaluates the training recipe using deployable previous-frame predictions at inference.
Figure 5: Additional policy-signal diagnostics. Left: distributions of the adjacent-frame similarity targets (global cosine and patch CKA) used to derive depth labels. Right: the online metadata cues most correlated with representation change; ego-motion magnitudes dominate. These associations are predictive, not causal.
Class
AP
ATE
ASE
AOE
AVE
AAE
Car
69.30
0.367
0.145
0.056
0.225
0.189
Truck
47.73
0.575
0.191
0.067
0.195
0.205
Construction vehicle
21.08
0.846
0.414
0.873
0.133
0.361
Bus
55.58
0.623
0.189
0.049
0.418
0.281
Trailer
28.85
0.863
0.229
0.341
0.165
0.164
Barrier
64.54
0.323
0.279
0.141
–
–
Appendix
Table 8: Detailed nuScenes validation results by class. AP is averaged over the standard center-distance thresholds and reported as a percentage; true-positive errors are reported in their native fractional units. A dash denotes a metric that nuScenes does not define: AOE, AVE, and AAE for traffic cones, and AVE and AAE for barriers.
Figure 6: Full-feature drift versus matched-depth degradation. Five equal-count drift groups (low to high) show mean L8, L12, and L16 error minus matched L24 error. Bars are scene-bootstrap 95% intervals; ρ denotes frame-level Spearman correlation. Depth matters more than the weak drift trend within a depth.
Figure 7: Frame-level matched-depth counterfactuals. Each point is one validation frame evaluated from a shared pre-frame state. Colored markers show drift-quintile means and black lines show isotonic fits. The wide scatter and small correlations caution against treating full-feature drift as a per-frame detection-regret oracle.
Figure 8: Full-L24 drift and absolute error under naive reuse. Top: matched-prediction error; bottom: box-only error. Columns use one- and two-frame drift offsets (2,000 and 1,900 frames). Blue markers are drift-quintile means with scene-bootstrap 95% intervals; black lines are isotonic fits. Reported ρ intervals all include zero. Absolute error is not reuse-induced degradation.
Non-full-frame operation
NDS
mAP
FPS
Partial only
55.75
45.59
6.12
Gated residual
58.23
48.98
6.12
Cross-attention
54.33
43.58
5.95
Gated cross-attention
57.61
48.30
5.91
Per-camera attention
55.04
44.30
5.99
Inverted attention
52.91
42.49
5.90
Appendix
Table 9: Updater ablation with L8 partial features and a fixed full-encoding schedule. A lightweight gated residual yields the best accuracy without the latency overhead of cross-attention.
Figure 9: Evidence for controller inputs. Left: scene-grouped prediction of full-representation change. Right: the current early representation is informative about final-feature change. A cascade that uses this probe must include its computation in end-to-end latency.
Figure 10: Forward pass latency distribution on the nuScenes validation split using the StreamPETR model. The adaptive routing policy maintains a highly efficient median latency of 149.18 ms, while effectively bounding tail latencies ( ∼230 ms) during full-depth feature refreshes. Peak CUDA memory footprint during these operations requires 3.37 GB allocated and 16.69 GB reserved.
Figure 11: t-SNE visualization and clustering of learned policy logits. We project the pre-encoding controller’s output logits across 700 training clips into two dimensions using t-SNE and partition them into four clusters (color-coded). A manual audit of 10% of the data reveals that the policy organizes driving dynamics into four interpretable regimes without explicit behavioral labels: low-speed cruising , sharp turns , dense scenes , and fast motion (qualitative examples shown in Fig. 4 ).
Department of Computer Science, Georgia State University · Department of Electrical and Computer Engineering, Duke University · School of Computing, University of Connecticut
Group Innovation, Volkswagen AG, 38440, Wolfsburg, Germany · Institute for Communications Technology, Technische Universität Braunschweig, 38106 Braunschweig, Germany