Orion-Lite: Distilling LLM Reasoning into Efficient Vision-Only Driving Models

Jing Gu; Niccolò Cavagnero; Gijs Dubbelman

cs.CVApr 9, 2026

Orion-Lite: Distilling LLM Reasoning into Efficient Vision-Only Driving Models

Authors: Jing Gu, Niccolò Cavagnero, Gijs Dubbelman

Organizations: Eindhoven University of Technology

Abstract

Leveraging the general world knowledge of Large Language Models (LLMs) holds significant promise for improving the ability of autonomous driving systems to handle rare and complex scenarios. While integrating LLMs into Vision-Language-Action (VLA) models has yielded state-of-the-art performance, their massive parameter counts pose severe challenges for latency-sensitive and energy-efficient deployment. Distilling LLM knowledge into a compact driving model offers a compelling solution to retain these reasoning capabilities while maintaining a manageable computational footprint. Although previous works have demonstrated the efficacy of distillation, these efforts have primarily focused on relatively simple scenarios and open-loop evaluations. Therefore, in this work, we investigate LLM distillation in more complex, interactive scenarios under closed-loop evaluation. We demonstrate that through a combination of latent feature distillation and ground-truth trajectory supervision, an efficient vision-only student model Orion-Lite can even surpass the performance of its massive VLA teacher, ORION. Setting a new state-of-the-art on the rigorous Bench2Drive benchmark, with a Driving Score of 80.6. Ultimately, this reveals that vision-only architectures still possess significant, untapped potential for high-performance reactive planning.

Figures & tables

Figure 1: Overview of the proposed distillation framework. A joint distillation and trajectory supervision strategy (top) yields a student model, Orion-Lite , that is $3\times$ faster than its teacher, establishing a new state-of-the-art on the closed-loop Bench2Drive benchmark (bottom).

Figure 2: Latency and Driving Score Comparison. Our distilled framework demonstrates a massive reduction in inference latency compared to the teacher model while improving the overall Driving Score. Latency is measured by the averaged inference step-time on CARLA evaluated on an A6000 GPU.

Method	Reference	Condition	Modality	Closed-loop Metrics				Open-loop Metric	Latency (ms) $\downarrow$
Method	Reference	Condition	Modality	DS $\uparrow$	SR (%) $\uparrow$	Efficiency $\uparrow$	Comfortness $\uparrow$	Avg. L2 $\downarrow$	Latency (ms) $\downarrow$
TCP* [ 36 ]	NeurIPS 22	TP	C	40.7	15.0	54.3	47.8	1.70	83
TCP-ctrl*	NeurIPS 22	TP	C	30.5	7.3	56.0	51.5	-	83
TCP-traj*	NeurIPS 22	TP	C	59.9	30.0	76.5	18.1	1.70	83
TCP-traj w/o distillation	NeurIPS 22	TP	C	49.3	20.5	78.8	23.0	1.96	83
ThinkTwice* [ 17 ]	CVPR 23	TP	C	62.4	31.2	69.3	16.2	0.95	762

Table 1: Closed-loop and Open-loop Results of E2E-AD Methods in Bench2Drive. All models are trained on the standard base set (1K clips), except for SimLingo † , which utilizes extended external training data (e.g., 3.1M samples). C/L refers to camera/LiDAR. Avg. L2 is averaged over predictions in 2 seconds under 2Hz. Latency is measured by the average inference step-time during CARLA evaluation on an A6000 GPU. * denotes expert feature distillation. NC: navigation command, TP: target point, DS: Driving Score, SR: Success Rate. Note: 0.5B and 7B indicate the parameters of the LLM module alone, rather than the full model’s parameter count.

Method	Reference	Condition	Modality	Ability (%) $\uparrow$
Method	Reference	Condition	Modality	Merging	Overtaking	Emergency Brake	Give Way	Traffic Sign	Mean
TCP* [ 36 ]	NeurIPS 22	TP	C	16.2	20.0	20.0	10.0	7.0	14.6
TCP-ctrl*	NeurIPS 22	TP	C	10.3	4.4	10.0	10.0	6.5	8.2
TCP-traj*	NeurIPS 22	TP	C	8.9	24.3	51.7	40.0	46.3	34.2
TCP-traj w/o distillation	NeurIPS 22	TP	C	17.1	6.7	40.0	50.0	28.7	28.5
ThinkTwice* [ 17 ]	CVPR 23	TP	C	27.4	18.4	35.8	50.0	54.2	37.2

Table 2: Multi-Ability Results of E2E-AD Methods under base set. * denote expert feature distillation. C/L refers to camera/LiDAR. NC: navigation command, TP: target point.

Panel of Figure 3 — Figure 3: Qualitative comparison in interactive scenarios. Top rows: rollouts from the Orion teacher model. Bottom rows: rollouts from our student model. Sequences are visualized at 5-frame intervals, with overlaid points indicating predicted future trajectories. Green checks ( $\checkmark$ ) denote successful, intervention-free maneuvers, while red crosses ( $\times$ ) indicate task failures. While Orion frequently hesitates or fails to safely merge or overtake behind obstacles, our student model demonstrates robust spatial awareness, successfully executing smooth and decisive maneuvers.

Mimic Loss	Traj. GT	Closed-loop		Ability (%) $\uparrow$
Mimic Loss	Traj. GT	DS $\uparrow$	SR $\uparrow$	M	O	EB	GW	TS	Mean
-	✓	73.9	50.0	26.3	66.7	71.7	10.0	63.7	47.7
✓	-	76.0	50.7	26.6	66.7	68.3	40.0	64.7	53.3
✓	✓	80.6	55.5	28.8	75.6	78.3	50.0	70.0	60.5

Table 3: Impact of supervision. We evaluate the model’s performance on closed-loop metrics and specific driving abilities. M : Merging, O : Overtaking, EB : Emergency Brake, GW : Give Way, TS : Traffic Sign, DS/SR : Driving Score/Success Rate.

Setting	Epochs	Closed-loop		Ability (%) $\uparrow$
Setting	Epochs	DS $\uparrow$	SR $\uparrow$	M	O	EB	GW	TS	Mean
Orion	18 (default)	77.7	54.6	25.0	71.1	78.3	30.0	69.2	54.7
Orion	24	77.1	50.7	25.3	57.8	70.0	40.0	69.0	52.0
Orion-Lite	20	80.6	55.5	28.8	75.6	78.3	50.0	70.0	60.5

Table 4: Impact of training duration. Comparison between the standard Orion model, Orion with extended training duration and our Orion-Lite.

Loss	L2 (m) $\downarrow$				Collision (%) $\downarrow$
Loss	1s	2s	3s	Avg.	1s	2s	3s	Avg.
$\mathcal{L}_{1}$	0.32	0.76	1.30	0.79	0.27	0.51	0.85	0.54
$\mathcal{L}_{2}$	0.32	0.75	1.30	0.79	0.36	0.55	0.90	0.60
KL	0.32	0.75	1.30	0.79	0.34	0.63	0.91	0.63
Huber	0.31	0.70	1.24	0.75	0.41	0.64	1.04	0.70

Table 5: Impact of distance metrics. We evaluate the impact of different distance metrics for distillation.

Encoder Init.	Frozen	Closed-loop		Ability (%) $\uparrow$
Encoder Init.	Frozen	DS $\uparrow$	SR $\uparrow$	M	O	EB	GW	TS	Mean
EVA-02-L	✓	54.3	21.8	11.3	33.3	28.3	30.0	41.1	28.8
EVA-02-L		72.4	47.3	23.8	53.3	70.0	30.0	67.9	49.0
Orion	✓	77.4	51.8	30.0	66.7	68.3	30.0	65.8	52.2

Table 6: Ablation on vision encoder initialization. We evaluate driving performance using different pre-trained weights.

Figure 4: Impact of Decoder Depth. Driving Score and mean Multi-ability Score across varying numbers of transformer decoder layers.

End-to-End Autonomous Driving

End-to-end autonomous driving learns to turn sensor observations and driving context into vehicle actions or trajectories.

See all 225 papers

CritiqueDriveVLM: From Verifier-Guided Reinforcement Learning to Latent Thought Distillation for Autonomous Driving
Jul 5, 2026Zhaohong Liu, Hao Ye, Xianlin Zhang +1VLMs for Autonomous Driving Efficient VLM Inference
State Key Laboratory of Networking and Switching Technology · Beijing University of Posts and Telecommunications, Beijing, China · School of Digital Media & Design Arts
EvoDriveVLA: Evolving Driving VLA Models via Collaborative Perception-Planning Distillation
Mar 10, 2026Jiajun Cao, Xiaoan Zhang, Xiaobao Wei +10End-to-End Autonomous Driving Autonomous Driving Planning
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · XPeng Motors
RT-VLA: Real-Time Vision-Language-Action Models via Knowledge Distillation
Jun 12, 2026Xiangyu Huang, Zhenlin Hua, Han Zhou +2End-to-End Autonomous Driving Efficient VLM Inference
Carnegie Mellon University

Jul 5, 2026J/K move · Enter open · S save

CritiqueDriveVLM: From Verifier-Guided Reinforcement Learning to Latent Thought Distillation for Autonomous Driving

Zhaohong Liu, Hao Ye, Xianlin Zhang, Mengshi Qi

State Key Laboratory of Networking and Switching Technology · Beijing University of Posts and Telecommunications, Beijing, China · School of Digital Media & Design Arts

VLMs for Autonomous Driving Efficient VLM Inference VLM Distillation VLM Reasoning RL for VLMs

Open paper AskPDF arXiv

End-to-end Vision-Language Models (VLMs) show immense potential in autonomous driving. However, standard Supervised Fine-Tuning (SFT) often suffers from reasoning hallucinations and conservative biases. While traditional tool-augmented frameworks and Chain-of-Thought (CoT) approaches mitigate these issues, they incur exorbitant token consumption and unacceptable latency, rendering real-time deployment impractical. To resolve this reliability-efficiency trade-off, we propose CritiqueDriveVLM, a novel unified three-stage framework internalizing reasoning directly into the VLM. First, we introduce Critique-Driven Multi-Turn Reinforcement Learning (RL) guided by a multi-dimensional verifier. By providing granular scalar feedback and a multi-turn penalty, we force the policy to internalize logical deduction, cultivating a robust System-2 Teacher that achieves high accuracy without fragile external tools. Subsequently, we propose Latent Thought Distillation to overcome the latency bottleneck. By aligning the Student's latent representations with the Teacher's fully converged reasoning states, we compress deep logical capabilities into a fast, CoT-free System-1 Student. Extensive experiments on the widely-used DriveLMM-01 benchmark demonstrate remarkable improvements. Compared to the base model, our tool-free Teacher significantly boosts Multiple Choice Quality (MCQ) from 55.54% to a state-of-the-art 76.54%. Crucially, our distilled Student preserves competitive reasoning depth while drastically minimizing generation length to an average of merely 28 tokens. This slashes inference latency by 88% (from 3482 ms to 416 ms), paving a highly robust pathway for low-latency autonomous driving.Our source code is available at https://github.com/MICLAB-BUPT/CritiqueDriveVLM.

Abstract

End-to-End Autonomous Driving

Explore similar work

CritiqueDriveVLM: From Verifier-Guided Reinforcement Learning to Latent Thought Distillation for Autonomous Driving

EvoDriveVLA: Evolving Driving VLA Models via Collaborative Perception-Planning Distillation

RT-VLA: Real-Time Vision-Language-Action Models via Knowledge Distillation