DriveCode: Domain Specific Numerical Encoding for LLM-Based Autonomous Driving

Zhiye Wang; Yanbo Jiang; Rui Zhou; Bo Zhang; Fang Zhang; Zhenhua Xu; Yaqin Zhang; Jianqiang Wang

cs.CVMar 1, 2026

DriveCode: Domain Specific Numerical Encoding for LLM-Based Autonomous Driving

Authors: Zhiye Wang, Yanbo Jiang, Rui Zhou, Bo Zhang, Fang Zhang, Zhenhua Xu, Yaqin Zhang, Jianqiang Wang

Organizations: School of Information Science and Engineering, Lanzhou University, Lanzhou, China · State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University, Beijing, China · The School of Vehicle and Mobility, Tsinghua University, Beijing, China · The Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China. ·

Abstract

Large language models (LLMs) have shown great promise for autonomous driving. However, discretizing numbers into tokens limits precise numerical reasoning, fails to reflect the positional significance of digits in the training objective, and makes it difficult to achieve both decoding efficiency and numerical precision. These limitations affect both the processing of sensor measurements and the generation of precise control commands, creating a fundamental barrier for deploying LLM-based autonomous driving systems. In this paper, we introduce DriveCode, a novel numerical encoding method that represents numbers as dedicated embeddings rather than discrete text tokens. DriveCode employs a number projector to map numbers into the language model's hidden space, enabling seamless integration with visual and textual features in a unified multimodal sequence. Evaluated on OmniDrive, DriveGPT4, and DriveGPT4-V2 datasets, DriveCode demonstrates superior performance in trajectory prediction and control signal generation, confirming its effectiveness for LLM-based autonomous driving systems.

Figures & tables

Figure 1: A sample procedure of DriveCode. Numbers are first extracted from text prompts and then processed by a number projector to achieve continuous number processing.

Figure 2: DriveCode overview. Our proposed approach consists of three parts: image projection, text tokenization and number projection. The images are first encoded by a vision tower and projected into the language embedding space via an image projector. In parallel, textual descriptions and instructions are tokenized. The third part is the main contribution of our work: continuous numerical signals are vectorized through a dedicated number projector to form aligned numerical tokens. These visual, textual, and numerical tokens are concatenated into a unified sequence and processed by an LLM for further training and inference.

Figure 3: Parallel autoregressive generation of text and numbers. Numbers are projected and fed into the next step without conversion to text embeddings.

Figure 4: Examples of three datasets. All numbers in these datasets are replaced with <number_token>.

Method	Speed (m/s)					Turning angle (degree)
Method	RMSE $\downarrow$	$A_{0.1}\uparrow$	$A_{0.5}\uparrow$	$A_{1.0}\uparrow$	$A_{5.0}\uparrow$	RMSE $\downarrow$	$A_{0.1}\uparrow$	$A_{0.5}\uparrow$	$A_{1.0}\uparrow$	$A_{5.0}\uparrow$
ADAPT	3.02	9.56	24.77	37.07	90.39	11.98	27.93	66.83	75.13	89.45
DriveGPT4	1.30	30.09	60.88	79.92	98.44	8.98	59.23	72.89	79.59	95.32
xVal	1.13	26.58	63.46	82.53	99.10	8.78	56.99	72.89	80.08	93.20
DriveCode	1.08	27.50	64.60	82.99	99.10	7.71	57.18	72.54	80.25	93.71

Table I : Quantitative results of control signals prediction on the whole DriveGPT4 testing dataset. RMSE denotes root mean square error.

A_{\delta}

denotes the percentage of samples with absolute error within threshold

\delta

Method	L2 Error (m) $\downarrow$	CIDEr $\uparrow$	BLEU4 $\uparrow$
Text	3.0797	2.2009	26.6945
DriveCode (Ours)	2.8274	2.3829	27.3690

Table II : Comparison of Text and DriveCode on trajectory L2 and text answering. (OmniDrive)

Method	Theta Error (degree) $\downarrow$	Point Error (L2, m) $\downarrow$	Speed Error (m/s) $\downarrow$
xVal	0.07409	0.01166	0.02162
DriveCode (Ours)	0.07377	0.01137	0.02131

Table III : Comparison of xVal and DriveCode on control signals prediction. (DriveGPT4-V2)

Figure 5: Qualitative comparison of trajectory prediction.

Method	Driving Score $\uparrow$
SimLingo	78.02
DriveCode (Ours)	86.79

Table IV : Closed-loop driving evaluation trained on the DriveGPT4-V2 dataset.

Dataset	Method	Theta Error (deg) $\downarrow$	Point Error (L2, m) $\downarrow$	Speed Error (m/s) $\downarrow$
DriveGPT4 Dataset	Raw value (DriveCode)	7.71	–	1.08
	Per-type z-score	8.90	–	1.14
	Per-type min-max	8.88	–	1.11
DriveGPT4-V2 Dataset	Raw value (DriveCode)	0.07377	0.01137	0.02131
	Per-type z-score	0.07276	0.01131	0.02164
	Per-type min-max	0.07066	0.01261	0.02229

Table A1 : Comparison of normalization strategies on control signal prediction across DriveGPT4 dataset and DriveGPT4-V2 dataset. DriveGPT4 dataset does not include point error evaluation, hence marked as –. Z-score and min-max normalization are applied separately to different numerical categories such as speed, steering angle, and waypoint coordinates.

Setting	Speed RMSE $\downarrow$	Angle RMSE $\downarrow$
Original DriveCode	1.08	7.71
Mixed units	1.34	8.89

Table A2: Unit variation robustness under original and mixed-unit data. speed uses m/s, km/h, mph; angle uses degree and radian (DriveGPT4 dataset)

Task	GT Value Range	DriveCode	Variant	Text-only
Turning Angle	[0, 0.26)	1.1416 (95% CI: 0.7626, 1.4807; N=1404)	1.4035 (95% CI: 0.9880, 1.7955; N=1403)	1.5338 (95% CI: 1.1637, 1.8924; N=1401)
	[0.26, 6.98)	2.1996 (95% CI: 1.9950, 2.4145; N=608)	2.7438 (95% CI: 2.1821, 3.3726; N=608)	2.7683 (95% CI: 2.4180, 3.1255; N=607)
	[6.98, $\infty$ )	34.1333 (95% CI: 23.0185, 44.7093; N=104)	37.6696 (95% CI: 25.9657, 48.5228; N=106)	38.7174 (95% CI: 26.9667, 50.4053; N=107)
Speed	[0, 1.44)	0.6544 (95% CI: 0.4411, 0.8653; N=705)	0.7037 (95% CI: 0.4835, 0.9238; N=705)	0.8359 (95% CI: 0.5740, 1.0823; N=705)
	[1.44, 7.54)	1.1763 (95% CI: 1.0438, 1.3050; N=705)	1.2319 (95% CI: 1.0848, 1.3835; N=706)	1.3251 (95% CI: 1.1534, 1.4963; N=706)
	[7.54, 15.70)	1.3535 (95% CI: 1.1455, 1.5731; N=600)	1.3422 (95% CI: 1.1592, 1.5386; N=600)	1.4156 (95% CI: 1.2430, 1.6002; N=598)

Table A3 : Numerical magnitude robustness analysis on the DriveGPT4 dataset. Samples are grouped according to the magnitude of ground-truth values using the 33.3%, 66.7%, and 95% quantiles.

Task	GT Value Range	DriveCode	Variant	Text-only
theta	$(-\infty,-0.05)$	9.3818 (95% CI: 6.4341, 12.2534; N=421)	9.2308 (95% CI: 6.2741, 12.1288; N=421)	9.8490 (95% CI: 6.6775, 12.6932; N=422)
	$[-0.05,0.05]$	1.1889 (95% CI: 0.7754, 1.5594; N=1251)	1.4857 (95% CI: 1.0493, 1.9053; N=1250)	1.6188 (95% CI: 1.2121, 1.9940; N=1249)
	$(0.05,\infty)$	14.0074 (95% CI: 7.0870, 19.6879; N=444)	16.3430 (95% CI: 9.2816, 22.5797; N=446)	16.7208 (95% CI: 9.4964, 22.9874; N=444)
speed	$(-\infty,-0.05)$	3.5897 (95% CI: 1.6277, 5.1443; N=13)	3.6364 (95% CI: 1.5787, 5.2521; N=13)	3.6722 (95% CI: 1.4158, 5.3855; N=13)
	$[-0.05,0.05]$	0.1735 (95% CI: 0.1107, 0.2385; N=366)	0.2152 (95% CI: 0.1223, 0.3109; N=366)	0.3085 (95% CI: 0.1404, 0.4671; N=366)
	$(0.05,\infty)$	1.1533 (95% CI: 1.0518, 1.2607; N=1737)	1.1882 (95% CI: 1.0862, 1.2877; N=1738)	1.2948 (95% CI: 1.1820, 1.4103; N=1736)

Table A4 : Sign robustness analysis on the DriveGPT4 dataset. Samples are grouped into negative, near-zero, and positive intervals according to the sign of ground-truth values. Lower is better. Best results are in bold .

Method	Theta Error (degree) $\downarrow$	Speed Error (m/s) $\downarrow$
Variant	8.63	1.11
DriveCode (Ours)	7.71	1.08

Table A5 : Comparison of Variant and DriveCode on control signals prediction (DriveGPT4).

Method	Theta Error (degree) $\downarrow$	Point Error (L2, m) $\downarrow$	Speed Error (m/s) $\downarrow$
Text	0.07950	0.01363	0.02231
Variant	0.07078	0.01140	0.02139
Variant2	0.08877	0.01391	0.02264
DriveCode (Ours)	0.07377	0.01137	0.02131

Table A6: Comparison of different numerical encoding variants on control signal prediction (DriveGPT4-V2).

Method	Latency(s) $\downarrow$	Avg_time_per_sample(s) $\downarrow$
xVal	6776.4096	3.1979
Text	7152.3616	3.3769
Variant	6763.7911	3.1920
DriveCode (Ours)	6737.9131	3.1798

Table A7 : Efficiency comparison between the baseline model, model using text, model using only <number_token> in output, and DriveCode. Tested on DriveGPT4 dataset.

Figure A1 : Examples of lane changing with DriveCode: successful lane change, no collision.

Autonomous Driving

Autonomous driving develops vehicles that perceive road scenes, predict interactions, and plan and control safe movement with limited or no human input.

See all 536 papers

SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving
Dec 11, 2025Peizheng Li, Zhenghao Zhang, David Holtz +6Vision-Language Models VLMs for Autonomous Driving
Mercedes-Benz AG · TU Munich · Karlsruhe Institute of Technology +4
LAD-Drive: Bridging Language and Trajectory with Action-Aware Diffusion Transformers
Mar 2, 2026Fabian Schmidt, Karol Fedurko, Markus Enzweiler +1Language-Conditioned Robot Planning Autonomous Driving Planning
Institute for Intelligent Systems, Esslingen University of Applied Sciences, Germany. · Department of Computer Science, University of Freiburg, Germany.
DRIV-EX: Counterfactual Explanations for Driving LLMs
Feb 28, 2026Amaia Cardiel, Eloi Zablocki, Elias Ramzi +1Counterfactual Explanations Counterfactual Reasoning in Language Models
Aptikal, Université Grenoble Alpes, Grenoble, France · Valeo.ai, Paris, France

Dec 11, 2025J/K move · Enter open · S save

SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving

Peizheng Li, Zhenghao Zhang, David Holtz, Hang Yu+5

Mercedes-Benz AG · TU Munich · Karlsruhe Institute of Technology+4

Vision-Language Models VLMs for Autonomous Driving End-to-End Autonomous Driving 3D Spatial Reasoning Positional Encoding Autonomous Driving Planning

Open paper AskPDF arXiv

End-to-end autonomous driving methods built on vision language models (VLMs) have undergone rapid development driven by their universal visual understanding and strong reasoning capabilities obtained from the large-scale pretraining. However, we find that current VLMs struggle to understand fine-grained 3D spatial relationships which is a fundamental requirement for systems interacting with the physical world. To address this issue, we propose SpaceDrive, a spatial-aware VLM-based driving framework that treats spatial information as explicit positional encodings (PEs) instead of textual digit tokens, enabling joint reasoning over semantic and spatial representations. SpaceDrive employs a universal positional encoder to all 3D coordinates derived from multi-view depth estimation, historical ego-states, and text prompts. These 3D PEs are first superimposed to augment the corresponding 2D visual tokens. Meanwhile, they serve as a task-agnostic coordinate representation, replacing the digit-wise numerical tokens as both inputs and outputs for the VLM. This mechanism enables the model to better index specific visual semantics in spatial reasoning and directly regress trajectory coordinates rather than generating digit-by-digit, thereby enhancing planning accuracy. Extensive experiments validate that SpaceDrive achieves state-of-the-art open-loop performance on the nuScenes dataset and the second-best Driving Score of 78.02 on the Bench2Drive closed-loop benchmark over existing VLM-based methods. Code is available at: https://github.com/zhenghao2519/SpaceDrive.

Abstract

Autonomous Driving

Explore similar work

SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving

LAD-Drive: Bridging Language and Trajectory with Action-Aware Diffusion Transformers

DRIV-EX: Counterfactual Explanations for Driving LLMs