RSKD: Grounding Semantic Consistency in Driving VLMs via Distillation

Chuchu Xie; Yang Zhong; Kangan Qian; Hao Ye; Kun Jiang; Diange Yang

cs.CVOct 26, 2026

RSKD: Grounding Semantic Consistency in Driving VLMs via Distillation

Authors: Chuchu Xie, Yang Zhong, Kangan Qian, Hao Ye, Kun Jiang, Diange Yang

Organizations: School of Vehicle and Mobility, Tsinghua University, Beijing, China · Automotive and Robotics, Xiaomi Corporation, Beijing, China

Abstract

Knowledge distillation can transfer large multimodal models to compact models for autonomous driving, but uniform token-level matching can dilute planning-relevant semantics and amplify teacher errors. We propose RSKD, a semantic-consistency-guided distillation framework that routes supervision according to both the task importance of an answer phrase and its semantic consistency with the teacher prediction. Task-critical semantic phrases are localized in the ground-truth answer, and a phrase-level consistency score controls the balance between Jensen--Shannon divergence (JSD) distillation and ground-truth supervision. Teacher information is emphasized only where it is semantically consistent, while semantically inconsistent phrases receive stronger ground-truth supervision. We evaluate RSKD on DrivingVQA, LingoQA, and MapLMv2. RSKD improves over the compared student-based baselines on DrivingVQA and LingoQA and remains competitive on MapLMv2.

Figures & tables

Figure 1: Overview of the proposed RSKD.

Method	DrivingVQA $\uparrow$	LingoQA $\uparrow$	MapLMv2 $\uparrow$
Teacher	49.81	0.642	79.00
Student	3.30	0.584	53.26
SFT	14.06	0.631	81.85
GKD	16.34	0.640	81.08
ToDi [ 7 ]	16.35	0.649	81.80
RSKD	16.60	0.672	81.82

Table 1: Main results on three autonomous-driving vision-language benchmarks. Higher is better. Trained student results are averaged over multiple random seeds; the best student-based result is bold.

Variant	Loc.	Cons.	JSD Rt.	SFT Rt.	Score $\uparrow$
Random Spans	Rand.	✓	✓	✓	0.474
w/o Consistency	✓	–	–	–	0.475
w/o SFT Routing	✓	✓	✓	–	0.468
Full RSKD	✓	✓	✓	✓	0.480

Table 2: Ablation on a fixed 50% LingoQA subset under the same reduced training budget.

VLM Distillation

Vision-language model distillation transfers multimodal capabilities to smaller, faster models for efficient deployment.

See all 126 papers

CritiqueDriveVLM: From Verifier-Guided Reinforcement Learning to Latent Thought Distillation for Autonomous Driving
Jul 5, 2026Zhaohong Liu, Hao Ye, Xianlin Zhang +1VLMs for Autonomous Driving Efficient VLM Inference
State Key Laboratory of Networking and Switching Technology · Beijing University of Posts and Telecommunications, Beijing, China · School of Digital Media & Design Arts
Uncertainty-Aware Knowledge Distillation for Multimodal Large Language Models
Mar 22, 2026Jingchen Sun, Shaobo Han, Deep Patel +3Uncertainty-Aware Knowledge Distillation Knowledge Distillation
NEC Laboratories America, Inc., USA · University at Buffalo, SUNY · Rutgers University
On-Policy Distillation of Language Models for Autonomous Vehicle Motion Planning
Apr 9, 2026Amirhossein Afsharrad, Amirhesam Abedsoltan, Ahmadreza Moradipari +1Language Model Distillation Autonomous Driving Planning
Stanford University · University of California, San Diego · University of California, Santa Barbara

Jul 5, 2026J/K move · Enter open · S save

CritiqueDriveVLM: From Verifier-Guided Reinforcement Learning to Latent Thought Distillation for Autonomous Driving

Zhaohong Liu, Hao Ye, Xianlin Zhang, Mengshi Qi

State Key Laboratory of Networking and Switching Technology · Beijing University of Posts and Telecommunications, Beijing, China · School of Digital Media & Design Arts

VLMs for Autonomous Driving Efficient VLM Inference VLM Distillation VLM Reasoning RL for VLMs

Open paper AskPDF arXiv

End-to-end Vision-Language Models (VLMs) show immense potential in autonomous driving. However, standard Supervised Fine-Tuning (SFT) often suffers from reasoning hallucinations and conservative biases. While traditional tool-augmented frameworks and Chain-of-Thought (CoT) approaches mitigate these issues, they incur exorbitant token consumption and unacceptable latency, rendering real-time deployment impractical. To resolve this reliability-efficiency trade-off, we propose CritiqueDriveVLM, a novel unified three-stage framework internalizing reasoning directly into the VLM. First, we introduce Critique-Driven Multi-Turn Reinforcement Learning (RL) guided by a multi-dimensional verifier. By providing granular scalar feedback and a multi-turn penalty, we force the policy to internalize logical deduction, cultivating a robust System-2 Teacher that achieves high accuracy without fragile external tools. Subsequently, we propose Latent Thought Distillation to overcome the latency bottleneck. By aligning the Student's latent representations with the Teacher's fully converged reasoning states, we compress deep logical capabilities into a fast, CoT-free System-1 Student. Extensive experiments on the widely-used DriveLMM-01 benchmark demonstrate remarkable improvements. Compared to the base model, our tool-free Teacher significantly boosts Multiple Choice Quality (MCQ) from 55.54% to a state-of-the-art 76.54%. Crucially, our distilled Student preserves competitive reasoning depth while drastically minimizing generation length to an average of merely 28 tokens. This slashes inference latency by 88% (from 3482 ms to 416 ms), paving a highly robust pathway for low-latency autonomous driving.Our source code is available at https://github.com/MICLAB-BUPT/CritiqueDriveVLM.

Abstract

VLM Distillation

Explore similar work

CritiqueDriveVLM: From Verifier-Guided Reinforcement Learning to Latent Thought Distillation for Autonomous Driving

Uncertainty-Aware Knowledge Distillation for Multimodal Large Language Models

On-Policy Distillation of Language Models for Autonomous Vehicle Motion Planning