Linking Modality Isolation in Heterogeneous Collaborative Perception

Changxing Liu; Zichen Chao; Siheng Chen

cs.CVFeb 28, 2026

Linking Modality Isolation in Heterogeneous Collaborative Perception

Authors: Changxing Liu, Zichen Chao, Siheng Chen

Organizations: Shanghai Jiao Tong University · Nanjing University of Science and Technology

Abstract

Collaborative perception leverages data exchange among multiple agents to enhance overall perception capabilities. However, heterogeneity across agents introduces domain gaps that hinder collaboration, and this is further exacerbated by an underexplored issue: modality isolation. It arises when multiple agents with different modalities never co-occur in any training data frame, enlarging cross-modal domain gaps. Existing alignment methods rely on supervision from spatially overlapping observations, thus fail to handle modality isolation. To address this challenge, we propose CodeAlign, the first efficient, co-occurrence-free alignment framework that smoothly aligns modalities via cross-modal feature-code-feature(FCF) translation. The key idea is to explicitly identify the representation consistency through codebook, and directly learn mappings between modality-specific feature spaces, thereby eliminating the need for spatial correspondence. Codebooks regularize feature spaces into code spaces, providing compact yet expressive representations. With a prepared code space for each modality, CodeAlign learns FCF translations that map features to the corresponding codes of other modalities, which are then decoded back into features in the target code space, enabling effective alignment. Experiments show that, when integrating three modalities, CodeAlign requires only 8% of the training parameters of prior alignment methods, reduces communication load by 1024x, and achieves state-of-the-art perception performance on both OPV2V and DAIR-V2X dataset. Code will be released on https://github.com/cxliu0314/CodeAlign.

Figures & tables

Figure 1: Illusion of modality isolation in heterogeneous collaborative perception.

Figure 2: Overview of the CodeAlign framework. The two training stages are: (1) code space construction for each modality and (2) feature–code–feature (FCF) translation between modalities, both trained using single-modality data. Step (3) illustrates the inference process, in which modality A has never encountered modality B during offline training, yet can still collaborate with it seamlessly.

Modality	m1	m2	m3	m4	m5	m6	m7
Sensor Type	LiDAR(64-beam)	LiDAR(64-beam)	LiDAR(64-beam)	LiDAR(32-beam)	LiDAR(64-beam)	Camera	Camera
Encoder	Point-Pillar (voxel: 0.4m)	Second (voxel: 0.1m)	VoxelNet (voxel: 0.4m)	Point-Pillar (voxel: 0.4m)	Point-Pillar (voxel: 0.3m)	LSS (ResNet-101)	LSS (EfficientNet)

Table 1: Modality settings in the experiment. The network after LSS is its backbone.

Modality Type	AP30	AP50	AP70
Non-Isolated	89.96	88.52	80.88
Isolated	82.36	80.51	65.67

Table 2: When m1 and m6 are isolated in their modalities, Pyramid Fusion [ 10 ] exhibits degraded perception performance.

	m1 + m2				m1 + m7				m1 + m7 + m2
Method	AP30	AP50	AP70	TP/M	AP30	AP50	AP70	TP/M	AP30	AP50	AP70	TP/M	Comm Load
No Collaboration	81.18	79.44	68.26	0	81.18	79.44	68.26	0	81.18	79.44	68.26	0	0
No Alignment	78.33	76.35	61.62	0	77.59	76.42	66.01	0	72.91	71.58	59.66	0	32MB
Late Fusion	92.73	91.46	79.34	0	84.47	80.12	62.40	0	88.24	85.02	68.45	0	0.5KB
Pyramid Fusion [ 10 ]	91.98	91.36	85.92	6.5	82.58	81.13	69.26	20.5	83.95	82.93	68.91	21.4	32MB
HMViT [ 19 ]	70.31	69.74	60.86	33.8	84.22	79.55	54.97	41.3	86.96	83.98	64.39	62.7	32MB

Table 3: Perception performance on OPV2V. The first row explains the tested collaboration scenarios of different isolated modalities, from a total of 2 to 3 cars. TP: training parameters for alignment. At last column, we show the average one-shot communication load.

Table 6

Align Method	AP30	AP50	AP70
Group Code Space Construction	89.04	87.54	79.63
FCF Translation	87.96	86.00	72.92

Table 6: Performance of different align method for modalities that have collaborative data (consider m1 and m6). Results show the benefit of group code space construction.

Alignment Method	AP30	AP50	AP70	TP/M	TP(n modalities)
Backward Alignment [ 10 ]	82.28	81.28	74.66	17.7	$\sum_{n}Enc$
One-to-one Code Translator	90.30	88.61	79.01	1.9	$(0.2M)n^{2}$
Multi-head Code Translator	90.17	88.51	78.33	1.5	$(0.5M)n$
Multi-head Code Translator*	90.25	88.58	78.42	1.5	$(0.5M)n$

Table 7: Ablation on model structures of cross-modal alignment with 4 isolated modalities. * means using data balance strategy.

Figure 3: Robustness under pose error under scene m1+m7.

Figure 4: Visualization of BEV features under scene m1+m7+m2, illustrating the strong alignment ability of FCF translation.

Modality	Encoder Type	#Param	Module	#Param
m1	PointPillar	0.23M	Adapter	0.30M
m2	Second	0.97M	Codebook	0.02M
m3	VoxelNet	0.56M	Translator	0.12M
m4	PointPillar	0.23M	Backend	5.27M
m5	PointPillar	0.23M	- Pyramid Fusion	3.79M
m6	LSS(ResNet-101)	1.77M	- Shrink Net	1.48M

Table 8: Parameter for modality-specific encoders and modality-agnostic modules.

Training	Adapter	m1				m6
Training	Adapter	AP30	AP50	AP70	TP/M	AP30	AP50	AP70	TP/M
E2E	✗	95.07	94.64	91.05	5.50	55.08	45.51	25.19	7.04
Fix E&B	✗	93.69	93.05	85.54	0.02	58.71	49.89	29.51	0.02
Fix E&B	✓	95.2	94.65	89.37	0.30	58.85	50.30	31.10	0.30

Table 9: Performance of different strategy in forming code space. Training methods include e2e training(E2E) and fix encoder and backbone training(Fix E&B). TP is training parameter.

Method		m1			m6
Strategy	Adapter	AP30	AP50	AP70	AP30	AP50	AP70
E2E	✗	95.07	94.64	91.05	55.08	45.51	25.19
Fix Encoder	✗	95.36	94.57	85.34	59.39	50.85	31.34
	4×ResBlock	94.78	94.52	91.17	59.13	51.09	33.04
	1×ConvNeXt block	95.62	95.14	90.4	59.37	51.25	31.48
	2-layer MLP	95.45	94.91	89.67	59.4	51.29	32.23

Table 10: More ablation studies on code space construction of a single modality.

Source	Type	m1 + m2			m1 + m6			Comm Load
Source	Type	AP30	AP50	AP70	AP30	AP50	AP70	Comm Load
Encoded Feature	D2D	95.65	95.06	90.72	88.93	86.96	75.04	32MB
Encoded Feature	D2C	95.44	94.88	89.83	88.13	86.41	73.47	0.03MB
Adapted Feature	D2C	95.49	94.9	90.11	88.28	86.42	73.56	0.03MB
Reconstructed Feature	D2C	93.96	93.23	85.87	76.13	73.42	53.89	0.03MB
Code Map	C2C	94.58	93.87	87.4	76.93	74.26	55.16	0.03MB

Table 11: Ablation on input source and output type of translator.

Modality	E2E Pyramid Fusion			Code Space Construction
Modality	AP30	AP50	AP70	AP30	AP50	AP70
m1	94.69	94.14	90.29	95.2	94.65	89.37
m2	94.93	94.54	91.72	94.74	94.31	90.23
m3	95.86	95.46	91.51	95.00	94.47	88.89
m4	89.26	88.66	83.68	86.79	85.82	77.47
m5	95.02	94.60	89.94	94.71	93.57	87.96

Table 12: Single-modality performance comparison between end-to-end Pyramid Fusion and code space construction.

Scenario	AP30	AP50	AP70
m1 single	81.18	79.44	68.26
m1 + m2	95.49	94.90	90.11
m1 + m3	95.22	94.67	89.48
m1 + m4	94.63	93.93	87.61
m1 + m5	95.06	94.53	89.23
m1 + m6	88.13	86.41	73.47

Table 13: Performance of more modalities aligned to m1 when m1 is ego with feature-code-feature translation.

Data Type	AP30	AP50	AP70
Collaborative Data	95.44	94.88	89.83
Single Data	94.83	94.24	88.09

Table 14: Feature-code-feature translation can be trained under single-agent data. We show an example of scene m1+m2.

Codebook Size	m1			m6			m1 + m6
Codebook Size	AP30	AP50	AP70	AP30	AP50	AP70	AP30	AP50	AP70
4	94.01	93.37	84.80	56.88	47.90	26.84	84.49	82.48	66.30
8	94.85	94.24	87.94	58.07	49.36	30.07	86.86	84.59	69.49
16	95.20	94.65	89.37	58.85	50.30	31.10	88.13	86.41	73.47
32	95.02	94.42	89.09	58.56	49.84	29.20	88.23	86.08	73.82
64	95.18	94.72	89.75	59.02	50.55	31.93	87.75	85.98	74.56

Table 15: Ablation on different codebook sizes.

Strategy	$L_{sim}$	AP30	AP50	AP70
E2E	\	89.3	88.03	80.44
+ Codebook(16)	\	89.54	87.74	77.87
++ Fix Enc ++ Add Adapter	\	88.37	86.83	78.09
	L2	88.82	87.31	78.88
	Instance	88.99	87.01	76.08
	Reconstruction	88.89	87.36	79.06

Table 16: More ablation studies on group code space construction, especially on similarity losses, in group (m1,m6).

	AP30	AP50	AP70	Comm Load
Late Fusion	89.61	86.47	69.97	0.5KB
HEAL	92.50	91.74	85.64	32MB
CodeAlign	92.27	91.48	85.42	0.03MB

Table 17: Alignment performance between group (m1,m6) and group (m2,m7) in order [m1,m7,m2,m6].

	AP30	AP50	AP70
Late Fusion	75.25	69.00	50.20
HEAL(pyramid fusion)	85.73	84.21	76.41
HEAL(backward alignment)	83.76	82.55	73.59

Table 18: HEAL degrades the alignment between the original group after being aligned to other groups. In order [m2,m7]

Method	Dif Enc	Dif Sensor	Extensible	Remarks
V2X-ViT [ 24 ]	✓			E2E attention fusion; needs same-scene features for spatial attention.
HM-ViT [ 19 ]		✓		E2E attention fusion; needs same-scene features for spatial attention.
MPDA [ 21 ]	✓			E2E attention fusion; needs same-scene features for spatial attention.
PnPDA [ 13 ]	✓			Contrastive loss for hetero features requires co-occurring modalities.
PolyInter [ 17 ]	✓		✓	Channel selection and spatial attention module need co-occurring modalities.
STAMP [ 4 ]	✓	✓	✓	Aligns to protocol net; protocol modality must be present in every scene.

Table 19: Comparison of heterogeneous collaborative perception methods in terms of encoder/sensor diversity, extensibility, and adaptability to Modality Isolation. Remarks detail limitations regarding isolation handling and key drawbacks.

Cross-Modal Alignment

Cross-modal alignment maps representations from different data types into compatible spaces so models can compare, transfer, and combine information.

See all 440 papers

One Model to Translate Them All: Universal Any-to-Any Translation for Heterogeneous Collaborative Perception
May 18, 2026Yang Li, Weize Li, Quan Yuan +7Cooperative Perception Heterogeneous Collaborative Perception
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China.
CauseCollab: Causal Unified and Modality-Agnostic Network for Heterogeneous Collaborative Perception
Sep 3, 2026Weize Li, Yang Li, Quan Yuan +3Causal Representation Learning Cross-Modal Alignment
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China.
CodeBind: Decoupled Representation Learning for Multimodal Alignment with Unified Compositional Codebook
May 18, 2026Zeyu Chen, Jie Li, Kai HanCross-Modal Learning Multimodal Disentangled Representation Learning
Visual AI Lab, The University of Hong Kong

May 18, 2026J/K move · Enter open · S save

One Model to Translate Them All: Universal Any-to-Any Translation for Heterogeneous Collaborative Perception

Yang Li, Weize Li, Quan Yuan, Congzhang Shao+6

State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China.

Cooperative Perception Heterogeneous Collaborative Perception

Open paper AskPDF arXiv

By sharing intermediate features, collaborative perception extends each agent's sensing beyond standalone limits, but real-world feature modality heterogeneity remains a key barrier to effective fusion. Most existing methods, including direct adaption and protocol-based transformation, typically rely on training adapters for newly emerging feature modalities and often require additional retraining or fine-tuning. Such repeated training is costly and is often infeasible across manufacturers due to model and data privacy constraints, limiting real-world scalability. To address this issue, we propose UniTrans, a universal any-to-any feature modality translation model that instantiates translators on the fly for arbitrary modalities. UniTrans pretrains a bank of translator expert parameters and learns their combination coefficients as a function of source-to-target modality mapping. The mapping is measured in a modality-intrinsic latent space, where an intrinsic encoder extracts modality-specific yet scene-invariant codes from single-frame intermediate features, enabling UniTrans to instantiate translators in a zero-shot manner. Experiments on OPV2V-H and DAIR-V2X demonstrate that UniTrans consistently outperforms state-of-the-art methods in both simulated and real-world settings, enabling efficient any-to-any translation through a universal model. The code is available at https://github.com/CheeryLeeyy/UniTrans.

Abstract

Cross-Modal Alignment

Explore similar work

One Model to Translate Them All: Universal Any-to-Any Translation for Heterogeneous Collaborative Perception

CauseCollab: Causal Unified and Modality-Agnostic Network for Heterogeneous Collaborative Perception

CodeBind: Decoupled Representation Learning for Multimodal Alignment with Unified Compositional Codebook