While the Platonic Representation Hypothesis posits that vision and language models converge toward compatible latent geometries, recovering class-level correspondences between unpaired modalities without cross-modal supervision remains challenging. Existing relational matching methods compare within-modality similarity kernels directly, although the same semantic structure may be represented through different numerical parameterizations across modalities. We propose RankAlign, a rank-based normalization strategy for class-level unpaired vision-language matching. RankAlign transforms each within-modality similarity kernel into an ordinal relational representation before permutation matching, while leaving the correspondence space and matching solver unchanged. The resulting representation is invariant to strictly increasing reparameterizations of similarity values while preserving the ordinal structure used for matching. We provide exact-recovery and stability guarantees that characterize correspondence recovery under both exact and partial ordinal agreement. Experiments across diverse frozen vision and language encoders and multiple datasets demonstrate improved and more stable class-level correspondence recovery, including under label corruption and domain shift.
Figures & tables
Figure 1: Visual-language alignment pipeline. The RankAlign module provides the core ordinal representation of each within-modality similarity kernel. These representations are refined using local neighbor order before permutation matching, without changing the matching space or solver.
Solver
Method
N=50
N=70
N=100
Matching accuracy
FAQ
BlindMatch
0.26 ± 0.10
0.14 ± 0.09
0.04 ± 0.02
RankAlign
0.98 ± 0.02
0.79 ± 0.05
0.66 ± 0.21
OT
BlindMatch
0.04 ± 0.01
0.02 ± 0.02
0.01 ± 0.01
RankAlign
0.98 ± 0.02
0.79 ± 0.05
0.69 ± 0.18
FHG
BlindMatch
0.77 ± 0.02
0.42 ± 0.07
0.15 ± 0.03
Table 1: Runtime and matching accuracy on CIFAR-100 across different problem sizes using CLIP as the vision encoder and Sentence-T as the language encoder. Results are reported as mean ± standard deviation over five random seeds. ∗ indicates that the solver reaches the 300s time limit. Actual wall-clock runtime may be slightly higher due to termination and post-processing overhead.
Figure 2: Matching accuracy as a function of problem size on CIFAR-100 and ImageNet-100. Each color corresponds to a vision backbone. For each backbone, the darker solid line denotes RankAlign , while the lighter dashed line denotes BlindMatch with GW .
Dataset
Metric
0.00
0.10
0.20
0.30
0.40
CIFAR-100
Acc. (%)
78.7 ± 7.3
78.0 ± 5.2
77.6 ± 5.8
73.1 ± 15.9
68.9 ± 6.4
Drop (%)
0.0
0.9
1.4
7.1
12.5
ImageNet-100
Acc. (%)
74.4 ± 5.4
71.0 ± 3.9
71.4 ± 5.4
56.6 ± 12.0
41.5 ± 10.3
Drop (%)
0.0
4.6
4.0
23.9
44.2
Table 2: Robustness under visual-side label corruption on CIFAR-100 and ImageNet-100. We report matching accuracy (%) as mean ± standard deviation over five random seeds and relative performance drop (%) with respect to the clean setting.
Vision model
BlindMatch
RankAlign
DINOv2
0.59 ± 0.29
0.96 ± 0.09
DeiT
0.72 ± 0.05
0.76 ± 0.09
CLIP
0.77 ± 0.13
0.68 ± 0.05
SigLIP
0.54 ± 0.13
0.79 ± 0.15
SigLIP2
0.61 ± 0.09
0.74 ± 0.11
Table 3: Matching accuracy on the LAION pseudo-class dataset. We compare BlindMatch ( GW ) with RankAlign across different visual backbones.
Vision model
BlindMatch
RankAlign
DeiT
0.08 ± 0.01
0.65 ± 0.01
DINOv2
0.06 ± 0.01
0.62 ± 0.30
CLIP
0.08 ± 0.01
0.63 ± 0.26
SigLIP
0.10 ± 0.02
0.45 ± 0.22
SigLIP2
0.08 ± 0.00
0.23 ± 0.04
Table 4: Matching performance on the CheXpert subset. We report matching accuracy across different vision backbones for BlindMatch ( GW ) and RankAlign .
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Transformation
N=30
N=50
N=70
N=100
None
0.03
0.01
0.00
0.00
Z-score
0.03
0.02
0.01
0.00
Row-wise ranking
0.32
0.14
0.09
0.06
Quantile normalization
0.06
0.07
0.04
0.00
RankAlign
1.00
0.98
0.91
0.79
Appendix
Table 5: Kernel transformation ablation on CIFAR-100. All variants use the same prototypes, class subsets, encoders, kernels, and permutation solver, only the kernel transformation differs.
Model
Method
N=30
N=50
N=70
N=100
CLIP
Procrustes
0.01±0.02
0.03±0.01
0.02±0.02
0.00±0.00
Vanilla GW-OT
0.03±0.03
0.03±0.02
0.03±0.03
0.03±0.00
RankAlign
1.00±0.00
0.98±0.02
0.91±0.04
0.79±0.02
DeiT
Procrustes
0.04±0.04
0.02±0.01
0.02±0.01
0.01±0.00
Vanilla GW-OT
0.07±0.04
0.05±0.03
0.07±0.05
0.05±0.00
RankAlign
0.93±0.01
0.95±0.05
0.72±0.05
0.36±0.16
Appendix
Table 6: Comparison with classical structural alignment methods on CIFAR-100. All methods use the same class-level representations and matching instances. We report matching accuracy across problem sizes.
Figure 3: Full larger-scale matching results on CIFAR-100 and ImageNet-100 across all evaluated vision backbones.Each color corresponds to a vision backbone. The darker solid line denotes RankAlign , while the lighter dashed line denotes BlindMatch with the GW .
Korea Advanced Institute of Science and Technology (KAIST), Republic of Korea · Electronics and Telecommunications Research Institute (ETRI), Republic of Korea