Multimodal perception enables robust autonomous driving but incurs unnecessary computational cost when all sensors remain active. This paper presents PRAM-R, a unified Perception-Reasoning-Action-Memory framework with LLM-Guided Modality Routing for adaptive autonomous driving. PRAM-R adopts an asynchronous dual-loop design: a fast reactive loop for perception and control, and a slow deliberative loop for reasoning-driven modality selection and memory updates. An LLM router selects and weights modalities using environmental context and sensor diagnostics, while a hierarchical memory module preserves temporal consistency and supports long-term adaptation. We conduct a two-stage evaluation: (1) synthetic stress tests for stability analysis and (2) real-world validation on the nuScenes dataset. Synthetic stress tests confirm 87.2% reduction in routing oscillations via hysteresis-based stabilization. Real-world validation on nuScenes shows 6.22% modality reduction with 20% memory recall while maintaining comparable trajectory accuracy to full-modality baselines in complex urban scenarios. Our work demonstrates that LLM-augmented architectures with hierarchical memory achieve efficient, adaptive multimodal perception in autonomous driving.
Figures & tables
Fig. 1 : Perception-Reasoning-Action-Memory Framework with modality routing for complex scene interpretation and motion control in autonomous driving: (1) Perception-layer, (2) Language-layer, (3) Action-layer, and (4) Memory-Layer.
Category
Modality
Diagnostic Indicators
Description
Enabled
Contextual Indicators
Interpretation
Reference
Sensor
Camera
Brightness mean
Average gray-level intensity or exposure value within image patches.
Ratio of luminance difference between bright and dark regions.
✓
Visibility quality
Low contrast → Low visibility → Fog, rain, or night scenes.
[ 10 ]
Edge density
Number of edges per image area, estimated by gradient magnitude.
✓
Texture density
Low edge density → Low texture density → Less structural information, degraded image clarity.
[ 20 ]
LiDAR
Point cloud density
Average number of LiDAR points per spatial cell.
✓
Lidar data completeness
Low density → Low data completeness → Sparse data.
[ 7 ] , [ 31 ]
Noise ratio
Fraction of points with abnormal intensity or geometry.
✓
Lidar signal quality
High noise ratio → Low signal quality → fog or heavy rain.
[ 7 ]
Reflectivity ratio
Ratio of measured to nominal reflectivity.
✓
Lidar signal quality
Low reflectivity ratio → Low signal quality → Fog or heavy rain.
[ 2 ]
TABLE I : Diagnostic and Contextual Indicators for Modality Routing
Component
Generation
Reading
Updating
Expiration
P-Routing Context
System start or first valid diagnostics
Before and after each routing decision
When new diagnostics or context arrive
After next round of routing decision
P-Semantic Cache
First deep-perception output in scene
Before each reasoning cycle
New deep-perception result
After each reasoning cycle
R-Scene Record
First completed reasoning cycle
Planning request or logging call
After every reasoning cycle
Scene end
R-Seed State
After first reasoning cycle
At the start of the next reasoning cycle
When significant context change or periodic refresh
Scene/route reset
A-Policy Log
First control command in scene
Validation or diagnostics
Each control cycle
Scene end
Knowledge Repository
Distill from all other memory components with quality threshold met
Scene reference or experience analysis
Continuous consolidation
Policy pruning or dataset management
TABLE II : Trigger Conditions of Memory Components in the PRAM-R Framework
Fig. 2 : Hierarchical and sequential organization of PRAM-R memory components and their temporal scales.
Fig. 3 : Threshold-Fluctuation Stress Test. Comparison of routing behavior under high-frequency perturbations. (a) Weights before and after EMA and hysteresis gating across timestep. (b) Corresponding binary activation states.
Modality
Switches (Thr)
Switches (Hyst)
Reduction (%)
Camera
11
2
81.8%
LiDAR
18
2
88.9%
Radar
18
2
88.9%
Total
47
6
87.2%
TABLE III : Effect of Hysteresis on Routing Stability under Fluctuations (threshold-only vs hysteresis)
Fig. 4 : Combined Memory Recall Rate. Comparison of short-term (rolling) and long-term (cumulative) recall rates for two different sampling frequencies.
Variant
RE(%)
RC
RSI
MRR
ADE(m)
FDE(m)
Static Fusion (PLA)
0.00
1.000
1.000
0.000
1.013
2.026
PRAM-R (w/o Memory)
6.02
0.992
0.662
0.000
N/A
N/A
PRAM-R (w/o Dual-loop)
5.67
0.998
0.641
0.313
N/A
N/A
Full PRAM-R
6.22
0.991
0.699
0.2000
1.124
2.182
TABLE IV : Ablation study on memory and hierarchical routing components.
Fig. 5 : Comprehensive routing statistics across nuScenes. (a) Modality activation rates demonstrate adaptive engagement. (b) Mean weight trajectories illustrate stable yet flexible fusion. (c) Reliability distributions differs from each modality.