Toward Unified Multimodal Representation Learning for Autonomous Driving

Ximeng Tao; Dimitar Filev; Gaurav Pandey

cs.CVMar 9, 2026

Toward Unified Multimodal Representation Learning for Autonomous Driving

Authors: Ximeng Tao, Dimitar Filev, Gaurav Pandey

Organizations: J. Mike Walker ’66 Department of Mechanical Engineering, Texas A&M University, College Station, TX 77843, USA · The Department of Engineering Technology and Industrial Distribution Texas A&M University, College Station, TX 77843, USA

Abstract

Contrastive Language-Image Pre-training (CLIP) has shown impressive performance in aligning visual and textual representations. Recent studies have extended this paradigm to 3D vision to improve scene understanding for autonomous driving. A common strategy is to employ pairwise cosine similarity between modalities to guide the training of a 3D encoder. However, considering the similarity between individual modality pairs rather than all modalities jointly fails to ensure consistent and unified alignment across the entire multimodal space. In this paper, we propose a Contrastive Tensor Pre-training (CTP) framework that simultaneously aligns multiple modalities in a unified embedding space to enhance end-to-end autonomous driving. Compared with pairwise cosine similarity alignment, our method extends the 2D similarity matrix into a multimodal similarity tensor. Furthermore, we introduce a tensor loss to enable joint contrastive learning across all modalities. For experimental validation of our framework, we construct a text-image-point cloud triplet dataset derived from existing autonomous driving datasets. The results show that our proposed unified multimodal alignment framework achieves favorable performance for both scenarios: (i) aligning a 3D encoder with pretrained CLIP encoders, and (ii) pretraining all encoders from scratch.

Figures & tables

Fig. 1: Overview of different contrastive representation learning methods. (a) CLIP: aligns two modalities. (b) Aligns a third modality with two already aligned modalities. (c) Performs pairwise alignment between every modality pair. (d) CTP: aligns all modalities toward one point.

Fig. 2: Overview of the multimodal language model–based end-to-end autonomous driving system. The multimodal encoder is pretrained to align all modalities within a unified embedding space, enabling the LLM to jointly understand cross-modal information and generate reasoning, scene descriptions, and future trajectory predictions. In this work, we primarily focus on training the “Multimodal Encoder” that can be used in an end-to-end driving system as shown here.

Fig. 3: Overview of CTP framework. Triplet dataset: LiDAR point clouds, cropped images, and annotations are extracted from autonomous driving datasets to form triplet samples. The annotation expanded into a detailed caption using a VLM. similarity tensor: Image, text, and point cloud features are arranged along the $x$ , $y$ , and $z$ axes to form a 3D similarity tensor. Each element represents a unique combination of three features, and the similarity measures their relationships. During training, the similarity scores of the matched triplets (small purple cubes) are maximized using a cross-entropy loss.

Fig. 4: Different strategies for flattening the plane loss: (a) nm : direct flattening without masking, (b) mask : masking duplicated entries, CTP adopts (b) as the standard flattening strategy.

Fig. 5: Zero-shot classification. Each image–point cloud pair is compared with all text features, and the class is determined by the highest L2-norm similarity. When computing similarity between point cloud or image features and text features only, cosine similarity is employed.

Data	Method	Avg.	Car	Truck	Bus	Ped.	Bicycle	Trailer	C.V.	Motor.	Barrier	T.C.
Depth	pointCLIP [ 18 ]	17.29	0.14	0.00	0.00	0.27	0.13	5.01	0.00	0.00	98.72	0.00
	pointCLIP V2 [ 41 ]	5.41	0.08	0.00	0.05	2.78	0.63	0.00	77.29	0.58	21.52	0.00
	CLIP2Point [ 44 ]	16.63	0.00	0.18	0.00	8.21	1.89	0.00	1.49	0.81	87.84	0.12
I-P	$\text{LidarCLIP}^{*}$ [ 27 ]	73.35	74.69	43.58	84.61	84.14	78.24	59.05	9.46	73.58	76.02	97.73
$\text{T}_{G}$ -I-P	$\text{CLIP}^{2*}$ [ 28 ]	74.66	70.66	44.13	66.99	86.65	59.12	77.47	2.19	63.27	94.09	96.45
$\text{T}_{G}$ -I-P	CTP (Ours)	80.08	77.84	69.92	60.98	94.79	64.29	60.00	0.00	46.43	91.83	95.24

TABLE I: Zero-shot classification accuracy (%) on the nuScenes triplet dataset

Method	KITTI	Waymo
pointCLIP [ 18 ]	9.63	13.79
pointCLIP V2 [ 41 ]	9.44	14.31
CLIP2Point [ 44 ]	10.35	21.27
$\text{LidarCLIP}^{*}$ [ 27 ]	67.79	75.27
$\text{CLIP}^{2*}$ [ 28 ]	74.55	84.86
CTP (Ours)	82.68	86.07

TABLE II: Zero-shot classification accuracy (%) on the KITTI and WOD-P triplet datasets

Method	Avg.	Car	Truck	Bus	Ped.	Bicycle	Trailer	C.V.	Motor.	Barrier	T.C.
$\text{ULIP}^{*}$ [ 30 ]	52.01	37.79	43.49	64.05	79.48	45.41	33.89	43.92	68.37	67.80	42.76
CTP-nm	62.23	50.99	14.29	56.10	95.26	78.57	56.67	23.53	78.57	86.06	79.37
CTP	65.92	62.16	4.51	78.05	91.94	71.43	60.00	47.06	96.43	88.46	52.38

TABLE III: Zero-shot classification accuracy (%) on the nuScenes triplet dataset

Method	KITTI					Waymo
Method	Avg.	Car	Truck	Van	Ped.	Avg.	Car	Sign	Ped.
$\text{ULIP}^{*}$ [ 30 ]	44.05	38.26	46.10	39.60	95.82	53.18	43.62	87.40	68.92
CTP-nm	75.42	81.56	25.00	31.25	92.86	63.18	61.54	92.86	43.33
CTP	84.92	92.20	25.00	37.50	100.00	64.68	63.64	89.29	46.67

TABLE IV: Zero-shot classification accuracy (%) on the KITTI and WOD-P triplet datasets

Method	Dataset	Average Accuracy (%)
Method	Dataset	T–I	T–P	T–(I, P)
$\text{ULIP}^{*}$ [ 30 ]	nuScenes	49.55	37.65	52.01
CTP w/ Cosine similarity		47.43	38.27	48.85
CTP w/ L2-norm similarity		61.34	57.34	65.92
$\text{ULIP}^{*}$ [ 30 ]	KITTI	53.16	17.91	44.05
CTP w/ Cosine similarity		69.61	52.9	70.95

TABLE V: Comparison of different methods for measuring similarity

Multimodal Pretraining

Multimodal pretraining learns reusable representations by training models on paired or mixed data from multiple modalities, such as text, images, audio, and signals.

See all 182 papers

123D: Unifying Multi-Modal Autonomous Driving Data at Scale
May 8, 2026Daniel Dauner, Valentin Charraut, Bastian Berle +10Autonomous Driving Datasets Autonomous Driving
1KE:SAI · University of Tübingen, Tübingen AI Center · 4Valeo Brain +4
Multi-modal Video Representation Alignment for Robust Self-supervised Driver Distraction Detection
Jun 1, 2026David J. Lerch, Livien Majer, Zeyun Zhong +3Multimodal Robustness Cross-Modal Alignment
Fraunhofer IOSB, Karlsruhe, Germany · Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany
MindDriver: Introducing Progressive Multimodal Reasoning for Autonomous Driving
Feb 25, 2026Lingjun Zhang, Yujian Yuan, Changjie Wu +7Reasoning for VLMs VLMs for Autonomous Driving
Amap, Alibaba Group · The Hong Kong University of Science and Technology · The Chinese University of Hong Kong +1

May 8, 2026J/K move · Enter open · S save

123D: Unifying Multi-Modal Autonomous Driving Data at Scale

Daniel Dauner, Valentin Charraut, Bastian Berle, Tianyu Li+9

1KE:SAI · University of Tübingen, Tübingen AI Center · 4Valeo Brain+4

Autonomous Driving Datasets Autonomous Driving Cross-Dataset Generalization

Open paper AskPDF arXiv

The pursuit of autonomous driving has produced one of the richest sensor data collections in all of robotics. However, its scale and diversity remain largely untapped. Each dataset adopts different 2D and 3D modalities, such as cameras, lidar, ego states, annotations, traffic lights, and HD maps, with different rates and synchronization schemes. They come in fragmented formats requiring complex dependencies that cannot natively coexist in the same development environment. Further, major inconsistencies in annotation conventions prevent training or measuring generalization across multiple datasets. We present 123D, an open-source framework that unifies such multi-modal driving data through a single API. To handle synchronization, we store each modality as an independent timestamped event stream with no prescribed rate, enabling synchronous or asynchronous access across arbitrary datasets. Using 123D, we consolidate eight real-world driving datasets spanning 3,300 hours and 90,000 kilometers, together with a synthetic dataset with configurable collection scripts, and provide tools for data analysis and visualization. We conduct a systematic study comparing annotation statistics and assessing each dataset's pose and calibration accuracy. Further, we showcase two applications 123D enables: cross-dataset 3D object detection transfer and reinforcement learning for planning, and offer recommendations for future directions. Code and documentation are available at https://github.com/kesai-labs/py123d.

Abstract

Multimodal Pretraining

Explore similar work

123D: Unifying Multi-Modal Autonomous Driving Data at Scale

Multi-modal Video Representation Alignment for Robust Self-supervised Driver Distraction Detection

MindDriver: Introducing Progressive Multimodal Reasoning for Autonomous Driving