Autonomous vehicles stream high-resolution surround-camera frames into memory before perception runs. This sensor-to-memory path consumes energy when cells store ones and adjacent bytes toggle on the data bus, so its cost follows bit-1 density and switching activity rather than pixel semantics. We present MotiMem-Omega, a semantic-aware memory-interface coder that lowers this cost while preserving perception predictions. Its semantic importance field protects traffic participants, especially vulnerable road users, while assigning lower fidelity to sky and empty background. Cross-dataset bit-sensitivity sweeps determine class weights, with a safety floor for pedestrians, cyclists, and motorcyclists. Each image block then selects a precision tier by minimizing a joint energy-distortion cost. When ego pose is available, a motion-compensated prior carries protected regions between frames. We estimate interface-energy reduction from the two measured proxies using a coefficient-swept memory-energy model. Across 29 detectors on 12 driving datasets, 5 occupancy models, and 5 segmentation networks, MotiMem-Omega retains about 90% of detection mean average precision, 91% of vulnerable-road-user recall, over 98% of occupancy accuracy, and the strongest segmentation retention among energy-reducing methods. It reduces front-camera bit-1 density by 52%, corresponding to a modeled memory-interface energy reduction near 36%, with a lower end of 27% under the literature coefficient sweep. It also gives higher retention than the baseline and energy-matched truncation at the same or lower bit-1 density, whereas image codecs preserve accuracy without reducing memory-interface energy.
Figures & tables
Fig. 1: Detection accuracy versus memory-interface energy across the 12 driving datasets. The vertical axis is mAP retention (each of the 29 detectors measured against its own original-frame detections, then averaged over the 12 datasets); the horizontal axis is normalized bit-1 density ρ1 , our memory-interface energy proxy, on the same 12-dataset front-camera basis as Table II . MotiMem- Ω gives the strongest accuracy–energy trade-off among the energy-reducing methods, retaining about 90% of mAP at ρ1≈0.48 , where energy-matched truncation (uniform 4-bit, global k=4 ) and the baseline give up 7 to 10 points of accuracy at the same or higher energy, while image codecs lie at ρ1≥1 with no memory saving.
Fig. 2: Overview of the MotiMem- Ω encoding pipeline: t−1 detections → semantic importance field → tiered hierarchical coding ( + sky prior) → DRAM/SRAM frame buffer → decode → perception; the field is projected t−1→t by ego-compensated motion.
Symbol
Meaning
It,I~t
original / decoded frame at time t
Φ,Ψ
MotiMem- Ω encoder / decoder
Dt−1
detections on the previous frame
ρ1,α
bit-1 density / switching activity
E(b)
memory-interface energy proxy, Eq. ( 2 )
ρ(⋅),ε
retention / tolerated accuracy loss
TABLE I: Primary notation.
Per-dataset mAP retention [ 77 ]
All-class [ 77 ]
VRU
Energy [ 6 ]
Image qual.
Method
nuSc [ 14 ]
A2D2 [ 21 ]
AV2 [ 16 ]
BDD [ 20 ]
CADC [ 19 ]
CaR [ 22 ]
JODD [ 25 ]
KUAS [ 24 ]
ONCE [ 17 ]
PnSt [ 18 ]
USyd [ 23 ]
Way [ 15 ]
Avg
R
A50
A75
R
A50
A75
mAP
ρ1
α
SSIM [ 78 ]
PSNR
LPIPS [ 79 ]
Original
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.000
∞
.000
Ω
.90
.95
.89
.81
.82
.90
.94
.90
.92
.93
.91
.91
.90
.91
.92
.92
.91
.93
.93
.91
.48
.52
.953
25.0
.168
base [ 4 ]
.83
.91
.87
.66
.81
.79
.88
.80
.87
.88
.82
.83
.83
.84
.87
.86
.79
.84
.83
.79
.63
.35
.940
30.5
.242
global-k4
.77
.89
.84
.62
.79
.79
.85
.74
.84
.84
.78
.80
.80
.81
.86
.84
.74
.82
.80
.74
.58
.28
.931
29.8
.263
uniform-4b
.77
.89
.84
.62
.79
.79
.85
.74
.84
.84
.78
.80
.80
.81
.86
.84
.74
.82
.80
.74
.48
.26
.929
29.2
.264
TABLE II: Comprehensive detection comparison: every encoding method (rows) on every metric, averaged over the 29 detectors. Rows: Ω = MotiMem- Ω , base = conference MotiMem [ 4 ] , then the naive truncations (global-k4, uniform-4b) and the image codecs JPEG [ 36 ] , WebP [ 38 ] , PNG [ 39 ] , JPEG2000 [ 37 ] , AVIF [ 40 ] , HEIF [ 41 ] . Left block: mAP retention per dataset + average; right blocks: aggregate all-class and VRU recall (R), AP@IoU 0.5/0.75 (A50/A75), COCO mAP, energy proxies ρ1,α (both normalized so the original frame is 1.00 ), and SSIM/PSNR/LPIPS. All figures are retention vs. the original-frame predictions ( ↑ ; ρ1,α , LPIPS ↓ ). MotiMem- Ω (bold) gives the highest average mAP among the energy-reducing methods while using low human-visible fidelity (low PSNR), indicating that the bit budget is concentrated on task-relevant content; codecs match retention only at ρ1≈1 .
All classes
VRU
Energy
Dataset
Location
R
A50
A75
mAP
R
A50
A75
mAP
ρ1
α
nuSc [ 14 ]
US, Boston
Ω
.92
.92
.92
.90
.91
.95
.94
.93
.57
.50
base
.84
.88
.86
.83
.76
.82
.81
.77
.63
.38
nv
.79
.84
.82
.77
.70
.80
.79
.73
.58
.29
A2D2 [ 21 ]
Germany
Ω
.96
.96
.96
.95
.93
.95
.95
.94
.46
.51
base
.92
.94
.93
.91
.85
.90
.89
.87
.63
.38
TABLE III: Per-dataset detection retention and energy for the three key methods (12 datasets, each averaged over 29 detectors). base = MotiMem [ 4 ] , nv = naive global-k4; R = recall, A50/A75 = AP@IoU 0.5/0.75 (middle block: VRU); ρ1 / α are the energy proxies, both normalized to the original frame (per-dataset values scaled to the Table VI means). The 12 sets span five continents and day/night/snow/desert. Ω (bold) has the strongest overall retention profile among the energy-reducing methods and the lowest ρ1 , with one near-tie (CADC A75, .85 vs .86 ).
Fig. 3: Cross-dataset operating-point sweep used to choose the fixed configuration ( 12 datasets ×29 detectors). Each panel varies one design variable and plots retention against the energy proxies; stars mark the selected points. (a) Object precision kobj : all usable values k≤7 are near-lossless ( ±1 -bit flip, flat ∼98% retention), and k=7 is both the maximum precision and an odd- k low- ρ1 point. (b) Background precision kbg : retention saturates after k=5 and ρ1 jumps at k=6 , so kbg=5 . (c) Sky precision ksky : sky moves from untruncated k=5 to ksky=2 , returning energy at a small accuracy cost. (d) Block-mode weight λ : the combined energy is minimized at λ=1 . (e) RoI dilation: VRU recall saturates at 20 px.
Fig. 4: Qualitative detection over the 12 datasets (rows) × seven encoding methods (columns), with native YOLO26 boxes on each decoded frame. MotiMem- Ω (red) keeps detections that the conference base MotiMem [ 4 ] (orange) and the energy-matched naive truncations (global-k4, uniform-4b) drop: the base has no motion-compensated closed loop, so an object that moved out of its t−1 RoI is truncated and lost, whereas Ω ’s tiered allocation keeps it. Image codecs alter the bitstream but not the predictions. Snow (CADC), night (BDD), and clear scenes are covered.
Architecture
Ω
base
g-k4
unif
JPEG
ρ1
.57
.63
.58
.48
.96
DeepLabV3 [ 31 ]
.934
.899
.885
.886
.904
FCN [ 32 ]
.889
.861
.827
.827
.862
LRASPP [ 33 ]
.836
.789
.769
.770
.817
SegFormer [ 34 ]
.715
.626
.625
.625
.713
Mask2Former [ 35 ]
.758
.702
.701
.701
.754
TABLE IV: Segmentation mIoU retention (vs. original-frame map) under every encoding method, on five architectures (nuScenes; base = MotiMem [ 4 ] , g-k4 = global- k 4, unif = uniform-4bit; ρ1 row = bit-1 density). Ω (bold) leads on every architecture; JPEG only matches it at ρ1≈1 (no saving). The lead also holds across 12 datasets (DeepLabV3 0.906 vs. 0.858 , +4.8 pp; range +1.2 to +10.7 ).
Fig. 5: Semantic segmentation under the seven encoding methods (columns) across the five architectures of Table IV (rows; DeepLabV3 / FCN / LRASPP are Pascal-VOC, SegFormer / Mask2Former are Cityscapes), on one nuScenes frame. Each cell blends the model’s seg map over the decoded frame; mislabelled pixels (class changed vs. the original-frame map) are highlighted in red , with the error-% (excluding the deliberately-truncated sky/background) printed. The decoded frames are near-identical by design (MotiMem preserves visual quality), so the visible signal is the downstream seg-map error, not the pixels. MotiMem- Ω (the second column) reproduces the original labelling with almost no error across all five models.
Model
Metric
Ω
base
g-k4
unif
JPEG
ρ1
.59
.63
.59
.45
1.04
GaussRender [ 29 ]
mIoU
99.7
94.8
92.8
92.8
97.7
DAOcc [ 28 ]
mIoU
99.9
97.8
97.5
97.5
99.6
OPUS [ 26 ]
mIoU
99.6
96.8
95.7
95.6
99.1
GaussianFormer-2 [ 30 ]
mIoU
98.2
96.9
96.3
96.4
99.2
SparseOcc [ 27 ]
RayIoU
99.8
96.3
95.6
95.5
99.1
TABLE V: 3D occupancy retention (% of each model’s native mIoU/RayIoU vs. original) under every encoding method, on five SOTA models (base = MotiMem [ 4 ] , g-k4 = global- k 4, unif = uniform-4bit; ρ1 row = bit-1 density). Ω (bold) retains the most among energy-saving methods; base and naive lose 2 – 7 pp, while JPEG matches only at ρ1>1 (no saving).
Fig. 6: 3D occupancy on the five SOTA models (rows): BEV (top) + 3D (bottom) under the original input vs. MotiMem- Ω (red), the conference base MotiMem [ 4 ] (orange), the non-semantic truncations, and JPEG; changed voxels are highlighted in light blue. Each column title gives the retention (% of the model’s native mIoU/RayIoU) and the bit-1 density ρ1 . Ω alone is both high-retention ( 98.2 – 99.9% ) and low-energy ( ρ1=0.6 ), whereas JPEG matches retention but at ρ1>1 (no memory saving).
Task
Method
ρ1
α
ΔE (%)
Det/Seg
MotiMem- Ω
0.480
0.516
35.5 [27.7, 43.3]
naive global-k4
0.580
0.277
37.0 [26.1, 46.3]
JPEG q50
0.957
1.493
−10.4 [ −16.0 , −2.3 ]
Occ
MotiMem- Ω
0.593
0.526
30.2 [23.1, 37.0]
naive global-k4
0.589
0.247
37.3 [26.0, 46.9]
JPEG q50
1.041
1.508
−14.6 [ −19.9 , −6.9 ]
TABLE VI: Derived memory-interface energy. From the measured ρ1 and α ratios, the model of Eq. ( 2 ) yields a reduction ΔE with a coefficient-agnostic range.
Variant
mAP
VRU
ρ1
α
MotiMem- Ω (full)
0.900
0.910
0.480
0.516
no Gray
0.904
0.916
0.517
0.522
sky at bg- k
0.932
0.921
0.514
0.636
ego-CV
0.903
0.915
0.482
0.547
2D-vel.
0.903
0.915
0.480
0.544
static
0.903
0.915
0.482
0.547
TABLE VII: Detection design sensitivity. Top rows perturb energy-related design choices; last rows swap the temporal prior. Metrics are normalized to the original frame.
High-resolution sensors are critical for robust autonomous perception but impose a severe memory wall on battery-constrained electric vehicles. In these systems, data movement energy often outweighs computation. Traditional image compression is ill-suited as it is semantically blind and optimizes for storage rather than bus switching activity. We propose MotiMem, a hardware-software co-designed interface. Exploiting temporal coherence,MotiMem uses lightweight 2D Motion Propagation to dynamically identify Regions of Interest (RoI). Complementing this, a Hybrid Sparsity-Aware Coding scheme leverages adaptive inversion and truncation to induce bitlevel sparsity. Extensive experiments across nuScenes, Waymo, and KITTI with 16 detection models demonstrate that MotiMem reduces memory-interface dynamic energy by approximately 43 percent while retaining approximately 93 percent of the object detection accuracy, establishing a new Pareto frontier significantly superior to standard codecs like JPEG and WebP.
Haohua Que, Mingkai Liu, Jiayue Xie +5
University of Georgia · Peking University · Tsinghua University +1
Fixed roadside cameras repeatedly observe a stable scene overlaid by sparse moving traffic. Temporal memory can recover weak observations, but reusing moving evidence at stale locations can corrupt occupancy predictions. Motion compensation addresses displacement, while reliance on the resulting history remains a separate learning problem. We introduce RoadOcc, which learns soft routing among fixed-coordinate history (Persist), velocity-addressed history (Transport), and current evidence (Refresh). Motion state and class-consistent historical support supervise these source choices. Dynamic-aware cross-attention (DCA) updates candidate locations, multi-scale voxel velocity estimation (VVE) constructs transport addresses from multi-scale current--history correspondence, and velocity-guided dynamic sparse fusion (VDSF) combines routed evidence under fixed sparse-token budgets. On InfraOcc, RoadOcc reaches 65.29 mIoU and 32.37 dynamic mIoU, gains of 4.44 and 4.71 over STCOcc. Controlled address experiments show that VVE raises dynamic mIoU by 0.87 over fixed-coordinate reading. Across three seeds, supervised P/T/R adds 1.40 dynamic points over motion-corrected retrieval, while removing Refresh costs 0.32 points. Results from two transfer models, Occ3D-nuScenes, and longer intervals provide additional support. Code will be released.
Xiaokai Bai, Lei Yang, Songkai Wang +3
College of Information Science and Electronic Engineering, Zhejiang University · School of Mechanical and Aerospace Engineering, Nanyang Technological University · School of Automotive Studies, Tongji University
End-to-end autonomous driving models plan future trajectories from raw sensor input. While earlier driving benchmarks often measured deviation from the human trajectory, current benchmarks such as NAVSIM and Bench2Drive evaluate models with richer simulation-based metrics intended to capture safe and compliant driving. A high benchmark score should reflect that a model can understand the scene in front of it and act accordingly. But how much of that score specifically comes from reacting to the dynamic part of that scene? To probe this, we remove a model's camera input and replace it with memories from prior drives at the same location. The retrieved memories can provide persistent scene information, including road layout and location-conditioned regularities, but not the current traffic state. Surprisingly, memory is nearly sufficient on NAVSIM, reaching or even exceeding the performance of leading end-to-end methods without actually observing the evaluated scene. Our results suggest that a high NAVSIM score does not require a planner to react to the current traffic scene and should be treated with caution. This effect is benchmark-dependent: driving from memory causes substantially larger performance drops on Bench2Drive and RealEngine. We provide our code at https://github.com/boschresearch/MemoryDrivoR .
Christian Löwens, Thorben Funke, Alexandru Paul Condurache
Bosch Research · University of Lübeck · Automated Driving, Bosch