cs.CVSep 26, 2026

RCVLA: 4D Radar-Grounded Semantic Reasoning and Trajectory Arbitration for Autonomous Driving

Authors: Lianqing Zheng, Xiaokai Bai, Yixuan Luo, Runwei Guan, Minghao Liu, Zhiqiang Wei, Hui-liang Shen, Xichan Zhu, 

Organizations: Tongji · ZJU · HKUST · UTokyo · 

Abstract

4D radar provides geometric and motion cues that complement visual semantics, but integrating it into vision-language-action (VLA) models requires both radar--language alignment for semantic reasoning and explicit use of radar measurements for trajectory refinement and selection. To support these capabilities, we construct Cap4DR with 86,016 radar-image-text samples for alignment pretraining and OmniHD-QA with 520,161 question-answer pairs for instruction tuning across scene description, key-object reasoning, occupancy understanding, and trajectory planning. Building on these datasets, we propose RCVLA, a radar-camera VLA framework consisting of a radar-grounded semantic reasoning stage (RCVLA-Sem) and a trajectory arbitration stage (RCVLA-Phys). RCVLA-Sem performs gated bidirectional interaction between camera and radar tokens for driving question answering and reference trajectory generation, while auxiliary heads provide object and occupancy queries. RCVLA-Phys refines reference-guided trajectory candidates through truncated diffusion conditioned on these queries and cluster-level radar measurements, then calibrates candidate scores using radar-derived time-to-collision risk. On OmniHD-QA, RCVLA-Sem improves CIDEr by 9.92 points and reduces key-object velocity error by 21.9%21.9\% relative to OmniDrive. RCVLA-Phys further reduces average L2 error from 0.3480.348 to 0.259 m0.259\,\mathrm{m} and average open-loop collision rate from 0.576%0.576\% to 0.175%0.175\% relative to RCVLA-Sem. Ablation studies further show that language-aligned radar tokens improve semantic reasoning, while cluster-level radar measurements and risk calibration improve trajectory arbitration. Code will be released.

Figures & tables

Explore similar work

CardsList
  1. Radar4D-VLM: Proposal-Grounded Temporal 4D Radar Reasoning Across Frozen Language Models

    Aug 4, 2026Jiaju Han, Xuemeng Sun, Qike Zhang +6VLMs for Autonomous DrivingMillimeter-Wave Radar

  2. CAR-VLA: Complexity-Aware and Risk-Adaptive Reasoning for Autonomous Driving

    Sep 28, 2026Xiaolei Chen, Zhuolin He, Yuxuan Liang +13VLMs for Autonomous DrivingAutonomous Driving

  3. 4DRC-OCC: Robust Semantic Occupancy Prediction Through Fusion of 4D Radar and Camera

    Mar 8, 2026David Ninfa, Andras Palffy, Holger Caesar3D Occupancy PredictionAutonomous Driving Perception