GT-Space: Enhancing Heterogeneous Collaborative Perception with Ground Truth Feature Space

Wentao Wang; Haoran Xu; Guang Tan

cs.LGMar 13, 2026

GT-Space: Enhancing Heterogeneous Collaborative Perception with Ground Truth Feature Space

Authors: Wentao Wang, Haoran Xu, Guang Tan

Organizations: Sun Yat-sen University, Shenzhen Campus, Shenzhen, China · Peng Cheng Laboratory (PCL), Shenzhen, China

Abstract

In autonomous driving, multi-agent collaborative perception enhances sensing capabilities by enabling agents to share perceptual data. A key challenge lies in handling {\em heterogeneous} features from agents equipped with different sensing modalities or model architectures, which complicates data fusion. Existing approaches often require retraining encoders or designing interpreter modules for pairwise feature alignment, but these solutions are not scalable in practice. To address this, we propose {\em GT-Space}, a flexible and scalable collaborative perception framework for heterogeneous agents. GT-Space constructs a common feature space from ground-truth labels, providing a unified reference for feature alignment. With this shared space, agents only need a single adapter module to project their features, eliminating the need for pairwise interactions with other agents. Furthermore, we design a fusion network trained with contrastive losses across diverse modality combinations. Extensive experiments on simulation datasets (OPV2V and V2XSet) and a real-world dataset (RCooper) demonstrate that GT-Space consistently outperforms baselines in detection accuracy while delivering robust performance. Our code will be released at https://github.com/KingScar/GT-Space.

Figures & tables

Figure 1: Comparison of heterogeneous collaborative strategies. (a) Retraining of the encoders: the fusion network is frozen while the local encoder (denoted as E) is retrained to adapt to data integration; (b) Interpreter-based alignment: local encoder and detection head (denoted as D) are frozen, the interpreter can project the received features into the ego’s feature space; (c) Our method does not require any re-training of encoders or heads, and utilizes a modality-agnostic fusion module to integrate heterogeneous modalities using a ground-truth derived common feature space for alignment.

Figure 2: The framework of GT-Space. Heterogeneous features undergo domain conversion before being input into the fusion network. The ground truth object bounding box labels are leveraged to generate a common feature space. Heterogeneous features from different agents are projected into the common feature space for alignment and fusion.

Figure 3: Feature alignment in GT-Space. (a) Ground truth features generation: The object bounding box labels are encoded and mapped onto the BEV map; (b) Multi-modal transformer: Each modality is mapped into a common feature space by a specific projector; (c) Combinatorial contrastive learning: M1, M2, and M3 denote three different modalities, and computing contrastive loss over all pairs of these modalities enhances the model’s general feature extraction ability; (d) Object-level alignment: Contrastive learning pulls features of the same object closer and pushes those of different objects apart.

Agent	Agent 1	Agent 2	Agent 3	Agent 4
Carrier	Vehicle	Infrastructure / Vehicle	Vehicle	Vehicle
Sensor	LiDAR	LiDAR	Camera	Camera
Model	SECOND ( Yan et al., 2018 )	PointPillar ( Lang et al., 2019 )	EfficientNet ( Tan and Le, 2019 )	ResNet50 ( He et al., 2016 )

Table 1: Settings of heterogeneous agents in the experiments.

Dataset	Method	$AP@50\uparrow$			$AP@70\uparrow$
Dataset	Ego A1, collaborates with	A2	A3	A4	A2	A3	A4
OPV2V	Non-collaboration	0.738	0.738	0.738	0.614	0.614	0.614
	Late fusion	0.821	0.772	0.766	0.680	0.615	0.613
	HM-ViT ( Xiang et al., 2023 )	0.857	0.814	0.807	0.731	0.646	0.652
	PnPDA ( Luo et al., 2024 )	0.878	0.809	0.802	0.792	0.651	0.653
	HEAL ( Lu et al., 2024 )	0.887	0.827	0.823	0.801	0.724	0.728

Table 2: Comparison of fusion performance for different heterogeneous modality pairs using AP@50 and AP@70 metrics. A1, A2, A3 and A4 refer to agent 1, agent 2, agent 3 and agent 4, respectively.

Dataset	Method	$AP@50\uparrow$				$AP@70\uparrow$
Dataset	Method	A1	A2	A3	A4	A1	A2	A3	A4
OPV2V	Non-collaboration	0.738	0.744	0.402	0.396	0.614	0.620	0.354	0.337
	Late fusion	0.818	0.818	0.818	0.818	0.690	0.690	0.690	0.690
	HM-ViT ( Xiang et al., 2023 )	0.852	0.853	0.835	0.837	0.756	0.752	0.722	0.725
	PnPDA ( Luo et al., 2024 )	0.861	0.864	0.828	0.833	0.798	0.794	0.719	0.716
	HEAL ( Lu et al., 2024 )	0.894	0.889	0.842	0.843	0.806	0.801	0.726	0.733

Table 3: Perception performance of different agents in multi-agent scenarios.

Figure 4: Performance of under-performing agents on OPV2V. The left column represents Agent 1, and the right column Agent 3. The horizontal axis represents performance without collaboration, while the vertical axis represents collaboration.

Figure 5: Robust Experiment to pose error and communication latency.

Configurations	OPV2V		V2XSet
Configurations	mAP@50	mAP@70	mAP@50	mAP@70
w/o-GT feature	0.868	0.795	0.830	0.782
w/o-projector	0.791	0.683	0.716	0.604
w/o-contrastive loss	0.845	0.721	0.823	0.709
Full version	0.892	0.814	0.873	0.806

Table 4: Effect of each design component in the model.

Figure 6: Visualization of BEV features before and after fusion. (a) Features before fusion of A2 (PointPillar); (b) Enhanced features after fusion.

Figure 7: Visualization of (a) Ground truth feature, (b) projected camera feature $\Phi_{cam}(F_{cam})$ and (c) projected LiDAR feature $\Phi_{lidar}(F_{lidar})$ .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$F_{\text{a}}$	Feature map from $a$ -th agent.
$F_{GT}$	Ground truth feature map.
$B_{i}$	$i$ -th bounding box in a frame.
$\beta_{i}$	Encoded representation of $B_{i}$
$U_{c}$	The feature of the $c$ -th cell
$U_{n}^{i}$	The feature of the $n$ -th cell of box $B_{i}$

Appendix

Table 5: Summary of Symbols.

Figure 8: (a) End-to-end training; (b) Late fusion; (c) Encoder re-training; (d) Interpreter-based method; (e)Adapter-Reverter method; (f) GT-Space(Ours). Blue indicates frozen modules, while red represents modules that are updated during training.

Method	Encoding (ms)	Fusion (ms)	Detection (ms)
HM-ViT ( Xiang et al., 2023 )	7.08	1.38	0.95
PnPDA ( Luo et al., 2024 )	7.08	1.41	0.98
Hetecooper ( Shao et al., 2024 )	7.08	1.47	0.97
HEAL ( Lu et al., 2024 )	7.08	1.54	0.95
STAMP ( Gao et al., 2025 )	7.08	1.52	0.94
GT-Space	7.08	1.48	0.97

Appendix

Table 6: System delays

Method	A1	A2	A3	A4
No fusion	0.441	0.438	0.231	0.228
HM-ViT ( Xiang et al., 2023 )	0.458	0.467	0.294	0.287
PnPDA ( Luo et al., 2024 )	0.463	0.468	0.321	0.330
Hetecooper ( Shao et al., 2024 )	0.471	0.465	0.319	0.324
HEAL ( Lu et al., 2024 )	0.478	0.473	0.343	0.346
STAMP ( Gao et al., 2025 )	0.473	0.469	0.338	0.342

Appendix

Table 7: Performance on RCooper

Method	AMOTA	AMOTP
No fusion	0.083	0.227
HM-ViT ( Xiang et al., 2023 )	0.224	0.354
PnPDA ( Luo et al., 2024 )	0.228	0.357
Hetecooper ( Shao et al., 2024 )	0.227	0.361
HEAL ( Lu et al., 2024 )	0.232	0.364
STAMP ( Gao et al., 2025 )	0.231	0.359

Appendix

Table 8: Performance of tracking task on RCooper

Method (Based on A1, A2, Add)	Base	+ A3	+ A4
PnPDA ( Luo et al., 2024 )	0.878	0.857	0.864
HEAL ( Lu et al., 2024 )	0.887	0.891	0.895
STAMP ( Gao et al., 2025 )	0.876	0.863	0.862
GT-Space	0.891	0.892	0.897

Appendix

Table 9: Newly adding agents.

Agent	Agent 1	Agent 2	Agent 3	Agent 4
Carrier	Vehicle	Infrastructure / Vehicle	Vehicle	Vehicle
Sensor	LiDAR	LiDAR	Camera	Camera
Model	SECOND ( Yan et al., 2018 )	PointPillar ( Lang et al., 2019 )	EfficientNet ( Tan and Le, 2019 )	ResNet50 ( He et al., 2016 )
Encoder Param. (M)	3.79	0.87	56.85	6.88
Agent	Agent 5	Agent 6	Agent 7	Agent 8
Carrier	Vehicle	Vehicle	Vehicle	Vehicle

Appendix

Table 10: Settings of heterogeneous agents in the experiments.

Figure 9: Training efficiency comparison for the different number of newly adding agents.

Dataset	Method	A1	A2	A3	A4
V2XSet	Late fusion	0.798	0.798	0.798	0.798
	HM-ViT ( Xiang et al., 2023 )	0.809	0.793	0.792	0.788
	PnPDA ( Luo et al., 2024 )	0.819	0.804	0.801	0.790
	HEAL ( Lu et al., 2024 )	0.844	0.842	0.818	0.822
	Hetecooper ( Shao et al., 2024 )	0.835	0.821	0.810	0.803
	STAMP ( Gao et al., 2025 )	0.842	0.829	0.801	0.803

Appendix

Table 11: Performance (AP50) of agent A1-A4 in V2XSet using the fusion model trained on OPV2V.

Dataset	Temperature	A1	A2	A3	A4
OPV2V	0.05	0.845	0.835	0.830	0.826
	0.07	0.850	0.842	0.824	0.830
	0.1	0.858	0.846	0.828	0.833

Appendix

Table 12: Performance (AP50) of models trained with different temperature parameters

Method	A1	A2	A3	A4
GT-Space	0.477	0.475	0.349	0.351
GT-Space (average)	0.481	0.482	0.350	0.354

Appendix

Table 13: Results of scale balancing

Figure 10: Visualization of collaborative perception results. We color the predicted and GT boxes.

Explore similar work

HeteroPROMPT: A Real-time and Privacy-Preserving Heterogeneous Collaborative Perception Framework
Jul 28, 2026Armin Maleki, Hayder RadhaAutonomous Driving Perception Cooperative Perception
Electrical and Computer Engineering, Michigan State University
HyDRA: Hybrid Domain-Aware Robust Architecture for Heterogeneous Collaborative Perception
Mar 25, 2026Minwoo Song, Minhee Kang, Heejin AhnHeterogeneous Collaborative Perception Cooperative Perception
School of Electrical Engineering, Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea
Faster-HEAL: An Efficient and Privacy-Preserving Collaborative Perception Framework for Heterogeneous Autonomous Vehicles
Mar 7, 2026Armin Maleki, Hayder RadhaAutonomous Driving Perception Heterogeneous Collaborative Perception
Electrical and Computer Engineering, Michigan State University, East Lansing, Michigan

Jul 28, 2026J/K move · Enter open · S save

HeteroPROMPT: A Real-time and Privacy-Preserving Heterogeneous Collaborative Perception Framework

Armin Maleki, Hayder Radha

Electrical and Computer Engineering, Michigan State University

Autonomous Driving Perception Cooperative Perception Heterogeneous Collaborative Perception

Open paper AskPDF arXiv

Collaborative Perception (CP) improves autonomous systems' awareness of their surroundings by sharing sensor data, intermediate features, and detection results. In real-world deployments, however, collaborating vehicles often use heterogeneous sensors, perception models, datasets, and training domains, creating feature-space shifts that degrade downstream fusion and detection. Existing approaches typically retrain fusion and detection components or introduce modality-specific feature interpreters. These methods scale poorly to newly joining agents and often require access to proprietary metadata, raising privacy concerns. We propose HeteroPROMPT, a real-time and privacy-preserving framework for heterogeneous collaborative perception. HeteroPROMPT rapidly aligns each heterogeneous agent's features with an ego-centric unified feature space through modular prompts and lightweight learning-based tuning, while keeping agent encoders and the collaborative fusion and detection stacks frozen. Its visual prompt-based training and inference modulate Bird's Eye View (BEV) features across channels and spatial locations with low computational overhead. For metadata-free deployment, an autoencoder learns a compact unified representation and extracts modality cues from shared features, enabling real-time modality classification and routing to the appropriate HeteroPROMPT modules without exposing proprietary agent information. Experiments on the OPV2V-H and V2XSet datasets show that HeteroPROMPT improves Average Precision over state-of-the-art heterogeneous CP methods while using orders of magnitude fewer trainable parameters. This offers a scalable and practical CP solution. The proposed modality classifier also predicts the joining agent's modality from compact features with greater than 99.99 percent accuracy during deployment. Code will be available at https://github.com/arminmaleki007/HeteroPROMPT.

Abstract

Explore similar work

HeteroPROMPT: A Real-time and Privacy-Preserving Heterogeneous Collaborative Perception Framework

HyDRA: Hybrid Domain-Aware Robust Architecture for Heterogeneous Collaborative Perception

Faster-HEAL: An Efficient and Privacy-Preserving Collaborative Perception Framework for Heterogeneous Autonomous Vehicles