Collaborative perception leverages data exchange among multiple agents to enhance overall perception capabilities. However, heterogeneity across agents introduces domain gaps that hinder collaboration, and this is further exacerbated by an underexplored issue: modality isolation. It arises when multiple agents with different modalities never co-occur in any training data frame, enlarging cross-modal domain gaps. Existing alignment methods rely on supervision from spatially overlapping observations, thus fail to handle modality isolation. To address this challenge, we propose CodeAlign, the first efficient, co-occurrence-free alignment framework that smoothly aligns modalities via cross-modal feature-code-feature(FCF) translation. The key idea is to explicitly identify the representation consistency through codebook, and directly learn mappings between modality-specific feature spaces, thereby eliminating the need for spatial correspondence. Codebooks regularize feature spaces into code spaces, providing compact yet expressive representations. With a prepared code space for each modality, CodeAlign learns FCF translations that map features to the corresponding codes of other modalities, which are then decoded back into features in the target code space, enabling effective alignment. Experiments show that, when integrating three modalities, CodeAlign requires only 8% of the training parameters of prior alignment methods, reduces communication load by 1024x, and achieves state-of-the-art perception performance on both OPV2V and DAIR-V2X dataset. Code will be released on https://github.com/cxliu0314/CodeAlign.
Figures & tables
Figure 1: Illusion of modality isolation in heterogeneous collaborative perception.
Figure 2: Overview of the CodeAlign framework. The two training stages are: (1) code space construction for each modality and (2) feature–code–feature (FCF) translation between modalities, both trained using single-modality data. Step (3) illustrates the inference process, in which modality A has never encountered modality B during offline training, yet can still collaborate with it seamlessly.
Modality
m1
m2
m3
m4
m5
m6
m7
Sensor Type
LiDAR(64-beam)
LiDAR(64-beam)
LiDAR(64-beam)
LiDAR(32-beam)
LiDAR(64-beam)
Camera
Camera
Encoder
Point-Pillar (voxel: 0.4m)
Second (voxel: 0.1m)
VoxelNet (voxel: 0.4m)
Point-Pillar (voxel: 0.4m)
Point-Pillar (voxel: 0.3m)
LSS (ResNet-101)
LSS (EfficientNet)
Table 1: Modality settings in the experiment. The network after LSS is its backbone.
Modality Type
AP30
AP50
AP70
Non-Isolated
89.96
88.52
80.88
Isolated
82.36
80.51
65.67
Table 2: When m1 and m6 are isolated in their modalities, Pyramid Fusion [ 10 ] exhibits degraded perception performance.
m1 + m2
m1 + m7
m1 + m7 + m2
Method
AP30
AP50
AP70
TP/M
AP30
AP50
AP70
TP/M
AP30
AP50
AP70
TP/M
Comm Load
No Collaboration
81.18
79.44
68.26
0
81.18
79.44
68.26
0
81.18
79.44
68.26
0
0
No Alignment
78.33
76.35
61.62
0
77.59
76.42
66.01
0
72.91
71.58
59.66
0
32MB
Late Fusion
92.73
91.46
79.34
0
84.47
80.12
62.40
0
88.24
85.02
68.45
0
0.5KB
Pyramid Fusion [ 10 ]
91.98
91.36
85.92
6.5
82.58
81.13
69.26
20.5
83.95
82.93
68.91
21.4
32MB
HMViT [ 19 ]
70.31
69.74
60.86
33.8
84.22
79.55
54.97
41.3
86.96
83.98
64.39
62.7
32MB
Table 3: Perception performance on OPV2V. The first row explains the tested collaboration scenarios of different isolated modalities, from a total of 2 to 3 cars. TP: training parameters for alignment. At last column, we show the average one-shot communication load.
Table 6
Align Method
AP30
AP50
AP70
Group Code Space Construction
89.04
87.54
79.63
FCF Translation
87.96
86.00
72.92
Table 6: Performance of different align method for modalities that have collaborative data (consider m1 and m6). Results show the benefit of group code space construction.
Alignment Method
AP30
AP50
AP70
TP/M
TP(n modalities)
Backward Alignment [ 10 ]
82.28
81.28
74.66
17.7
∑nEnc
One-to-one Code Translator
90.30
88.61
79.01
1.9
(0.2M)n2
Multi-head Code Translator
90.17
88.51
78.33
1.5
(0.5M)n
Multi-head Code Translator*
90.25
88.58
78.42
1.5
(0.5M)n
Table 7: Ablation on model structures of cross-modal alignment with 4 isolated modalities. * means using data balance strategy.
Figure 3: Robustness under pose error under scene m1+m7.
Figure 4: Visualization of BEV features under scene m1+m7+m2, illustrating the strong alignment ability of FCF translation.
Modality
Encoder Type
#Param
Module
#Param
m1
PointPillar
0.23M
Adapter
0.30M
m2
Second
0.97M
Codebook
0.02M
m3
VoxelNet
0.56M
Translator
0.12M
m4
PointPillar
0.23M
Backend
5.27M
m5
PointPillar
0.23M
- Pyramid Fusion
3.79M
m6
LSS(ResNet-101)
1.77M
- Shrink Net
1.48M
Table 8: Parameter for modality-specific encoders and modality-agnostic modules.
Training
Adapter
m1
m6
AP30
AP50
AP70
TP/M
AP30
AP50
AP70
TP/M
E2E
✗
95.07
94.64
91.05
5.50
55.08
45.51
25.19
7.04
Fix E&B
✗
93.69
93.05
85.54
0.02
58.71
49.89
29.51
0.02
Fix E&B
✓
95.2
94.65
89.37
0.30
58.85
50.30
31.10
0.30
Table 9: Performance of different strategy in forming code space. Training methods include e2e training(E2E) and fix encoder and backbone training(Fix E&B). TP is training parameter.
Method
m1
m6
Strategy
Adapter
AP30
AP50
AP70
AP30
AP50
AP70
E2E
✗
95.07
94.64
91.05
55.08
45.51
25.19
Fix Encoder
✗
95.36
94.57
85.34
59.39
50.85
31.34
4×ResBlock
94.78
94.52
91.17
59.13
51.09
33.04
1×ConvNeXt block
95.62
95.14
90.4
59.37
51.25
31.48
2-layer MLP
95.45
94.91
89.67
59.4
51.29
32.23
Table 10: More ablation studies on code space construction of a single modality.
Source
Type
m1 + m2
m1 + m6
Comm Load
AP30
AP50
AP70
AP30
AP50
AP70
Encoded Feature
D2D
95.65
95.06
90.72
88.93
86.96
75.04
32MB
Encoded Feature
D2C
95.44
94.88
89.83
88.13
86.41
73.47
0.03MB
Adapted Feature
D2C
95.49
94.9
90.11
88.28
86.42
73.56
0.03MB
Reconstructed Feature
D2C
93.96
93.23
85.87
76.13
73.42
53.89
0.03MB
Code Map
C2C
94.58
93.87
87.4
76.93
74.26
55.16
0.03MB
Table 11: Ablation on input source and output type of translator.
Modality
E2E Pyramid Fusion
Code Space Construction
AP30
AP50
AP70
AP30
AP50
AP70
m1
94.69
94.14
90.29
95.2
94.65
89.37
m2
94.93
94.54
91.72
94.74
94.31
90.23
m3
95.86
95.46
91.51
95.00
94.47
88.89
m4
89.26
88.66
83.68
86.79
85.82
77.47
m5
95.02
94.60
89.94
94.71
93.57
87.96
Table 12: Single-modality performance comparison between end-to-end Pyramid Fusion and code space construction.
Scenario
AP30
AP50
AP70
m1 single
81.18
79.44
68.26
m1 + m2
95.49
94.90
90.11
m1 + m3
95.22
94.67
89.48
m1 + m4
94.63
93.93
87.61
m1 + m5
95.06
94.53
89.23
m1 + m6
88.13
86.41
73.47
Table 13: Performance of more modalities aligned to m1 when m1 is ego with feature-code-feature translation.
Data Type
AP30
AP50
AP70
Collaborative Data
95.44
94.88
89.83
Single Data
94.83
94.24
88.09
Table 14: Feature-code-feature translation can be trained under single-agent data. We show an example of scene m1+m2.
Codebook Size
m1
m6
m1 + m6
AP30
AP50
AP70
AP30
AP50
AP70
AP30
AP50
AP70
4
94.01
93.37
84.80
56.88
47.90
26.84
84.49
82.48
66.30
8
94.85
94.24
87.94
58.07
49.36
30.07
86.86
84.59
69.49
16
95.20
94.65
89.37
58.85
50.30
31.10
88.13
86.41
73.47
32
95.02
94.42
89.09
58.56
49.84
29.20
88.23
86.08
73.82
64
95.18
94.72
89.75
59.02
50.55
31.93
87.75
85.98
74.56
Table 15: Ablation on different codebook sizes.
Strategy
Lsim
AP30
AP50
AP70
E2E
\
89.3
88.03
80.44
+ Codebook(16)
\
89.54
87.74
77.87
++ Fix Enc ++ Add Adapter
\
88.37
86.83
78.09
L2
88.82
87.31
78.88
Instance
88.99
87.01
76.08
Reconstruction
88.89
87.36
79.06
Table 16: More ablation studies on group code space construction, especially on similarity losses, in group (m1,m6).
AP30
AP50
AP70
Comm Load
Late Fusion
89.61
86.47
69.97
0.5KB
HEAL
92.50
91.74
85.64
32MB
CodeAlign
92.27
91.48
85.42
0.03MB
Table 17: Alignment performance between group (m1,m6) and group (m2,m7) in order [m1,m7,m2,m6].
AP30
AP50
AP70
Late Fusion
75.25
69.00
50.20
HEAL(pyramid fusion)
85.73
84.21
76.41
HEAL(backward alignment)
83.76
82.55
73.59
Table 18: HEAL degrades the alignment between the original group after being aligned to other groups. In order [m2,m7]
Method
Dif Enc
Dif Sensor
Extensible
M.I. Adapt
Remarks
V2X-ViT [ 24 ]
✓
E2E attention fusion; needs same-scene features for spatial attention.
HM-ViT [ 19 ]
✓
E2E attention fusion; needs same-scene features for spatial attention.
MPDA [ 21 ]
✓
E2E attention fusion; needs same-scene features for spatial attention.
PnPDA [ 13 ]
✓
Contrastive loss for hetero features requires co-occurring modalities.
PolyInter [ 17 ]
✓
✓
Channel selection and spatial attention module need co-occurring modalities.
STAMP [ 4 ]
✓
✓
✓
Aligns to protocol net; protocol modality must be present in every scene.
Table 19: Comparison of heterogeneous collaborative perception methods in terms of encoder/sensor diversity, extensibility, and adaptability to Modality Isolation. Remarks detail limitations regarding isolation handling and key drawbacks.
By sharing intermediate features, collaborative perception extends each agent's sensing beyond standalone limits, but real-world feature modality heterogeneity remains a key barrier to effective fusion. Most existing methods, including direct adaption and protocol-based transformation, typically rely on training adapters for newly emerging feature modalities and often require additional retraining or fine-tuning. Such repeated training is costly and is often infeasible across manufacturers due to model and data privacy constraints, limiting real-world scalability. To address this issue, we propose UniTrans, a universal any-to-any feature modality translation model that instantiates translators on the fly for arbitrary modalities. UniTrans pretrains a bank of translator expert parameters and learns their combination coefficients as a function of source-to-target modality mapping. The mapping is measured in a modality-intrinsic latent space, where an intrinsic encoder extracts modality-specific yet scene-invariant codes from single-frame intermediate features, enabling UniTrans to instantiate translators in a zero-shot manner. Experiments on OPV2V-H and DAIR-V2X demonstrate that UniTrans consistently outperforms state-of-the-art methods in both simulated and real-world settings, enabling efficient any-to-any translation through a universal model. The code is available at https://github.com/CheeryLeeyy/UniTrans.
Yang Li, Weize Li, Quan Yuan +7
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China.
Collaborative perception enhances environment understanding through multi-agent information sharing, but its performance in real-world scenarios is constrained by heterogeneous sensor modalities and model architectures. Recent protocol-based two-stage methods alleviate this problem by mapping heterogeneous features into a shared protocol space; however, independently trained modality-specific converters often generate modality-specific pseudo-protocol distributions, leading to semantic inconsistency and error accumulation, which is particularly pronounced in scenarios with large modality discrepancies. To address this issue, we propose CauseCollab, a causal unified and modality-agnostic network. CauseCollab formulates representation learning in the protocol space from a causal perspective, explicitly disentangling semantic factors from modality-specific statistical confounders via causal metric learning. Meanwhile, CauseCollab adopts context-guided Unified Converter for heterogeneous modalities to ensure cross-modal semantic consistency. In addition, integrating new modalities only requires training adapters with minimal parameters. Extensive experiments on the OPV2V and DAIR-V2X datasets demonstrate that CauseCollab achieves state-of-the-art performance, with more significant gains in scenarios involving large modality gaps.
Weize Li, Yang Li, Quan Yuan +3
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China.
Multimodal representation alignment is pivotal for large language models and robotics. Traditional methods are often hindered by cross-modal information discrepancies and data scarcity, leading to suboptimal alignment spaces that overlook modality-unique features. We propose CodeBind, a framework that optimizes multimodal representation spaces through a modality-shared-specific codebook design. By incrementally aligning target and bridging modalities, CodeBind bypasses the need for fully paired data. Unlike traditional hard alignment, CodeBind decomposes features into shared components for semantic consistency and specific components for modality-unique details. This design utilizes a compositional vector quantization scheme, where a shared codebook bridges modality gaps and modality-specific codebooks mitigate representation bias by preventing dominant modalities from overshadowing others. Validated across nine modalities (text, image, video, audio, depth, thermal, tactile, 3D point cloud, EEG), CodeBind achieves state-of-the-art performance in multimodal classification and retrieval tasks.