What Matters for Cross-Modal Alignment
Organizations: Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · Department of Orthodontics and Orofacial Orthopedics, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany
Abstract
The Platonic Representation Hypothesis posits that models trained on different modalities converge toward a shared representation. Related work aligns frozen image and text encoders on natural images without paired data. We test zero-pair alignment on five datasets across four modalities: natural images, environmental audio, chest radiographs, and 3D computed tomography (CT). It succeeds only on COCO and fails on the other four. We analyze the failures and find that alignment must first locate the subspace where the modalities co-vary and then orient the mapping within it. Consistent with previous studies, medical imaging is the hardest case: its shared subspace carries the smallest fraction of the variance. When this fraction is small, alignment needs image-text pairs, and random-matrix theory predicts how many pairs are needed. These findings motivate LightAlign: a few pairs let a ridge regression between two frozen encoders locate and orient the mapping simultaneously, and a classifier fitted only on report text then predicts findings from the mapped images. On 3D CT, LightAlign surpasses models trained end to end on image-report pairs without finding-level labels: with only 200 pairs, LightAlign outperforms CT-CLIP on CT-RATE (0.754 vs. 0.733 AUROC), and with only 100 CT-RATE pairs, it does so on RAD-ChestCT, an out-of-distribution dataset from a different hospital. On CheXpert, it reaches 0.862 mean AUROC with only about 5% as many pairs as models trained on MIMIC-CXR.
Figures & tables
| Trained (✓) or not ( ✗ ) | CT-RATE | RAD-ChestCT | ||||||
| Method | Image enc. | Text enc. | Labels | Training data | AUROC | F1 | Acc. | AUROC |
| Trained end to end on image–report pairs, no finding labels | ||||||||
| CT-CLIP [ 32 ] | ✓ | ✓ | ✗ | CT-RATE, 20k patients | 0.733 a | 0.707 a | 0.667 a | 0.629 |
| Merlin [ 10 ] | ✓ | ✓ | ✗ f | abdominal CT + EHR codes | 0.662 b | – | – | 0.644 c |
| Jolia [ 43 ] | ✓ | ✗ | ✗ | CT-RATE, INSPECT, Merlin m | 0.699 b | – | – | – |
| fVLM [ 80 ] | ✓ | ✓ | ✗ | CT-RATE + organ masks | 0.778 d | 0.751 d | 0.718 d | 0.680 |
| Unscaled | Mirror | ||||
| Dataset | Share | FOSCTTM | FOSCTTM | Mirrored | Corrected |
| (combinations) | mean/best | mean/best | seeds | w/o paired data | |
| COCO (4) | 0.19–0.45 | 0.43/0.29 | – | – | – |
| Flickr30k (6) | 0.15–0.38 | 0.49/0.46 | 0.23/0.07 | 3/18 | 0/3 |
| Clotho (8) | 0.05–0.28 | 0.48/0.28 | 0.28/0.07 | 12/24 | 1/12 |
| IU X-ray (6) | 0.02–0.035 | 0.49/0.45 | 0.39/0.33 | 3/18 | 3/3 |
| Image–report pairs | |||
| Variant | 100 | 400 | 1,600 |
| Map and readout (test set) | |||
| Ours: ridge regression, report classifier | 0.727 | 0.782 | 0.809 |
| Map: MLP trained with InfoNCE | 0.647 | 0.685 | 0.731 |
| Readout: zero-shot prompt | 0.709 | 0.727 | 0.734 |
| No map: image classifier on the pairs | 0.698 | 0.767 | 0.806 |
Appendix figures & tables46 assets
Supplementary material from the paper’s appendix.
Appendix
| Trained? | ||||
| Method | Image enc. | Text enc. | Pairs | AUROC |
| CheXzero [ 87 ] | ✓ | ✓ | 377k | 0.889 b |
| RadZero [ 66 ] | ✗ a | ✓ | 353k c | 0.903 |
| KAD [ 103 ] | ✓ | ✓ | 377k | 0.905 |
| CARZero [ 46 ] | ✓ | ✓ | – d | 0.923 |
| Ours: frozen RAD-DINO + Qwen3.5-4B, ridge regression, | ||||
| Validation split | |||||||
|---|---|---|---|---|---|---|---|
| Method | scoring method | mean AUROC | stratified AUROC | ||||
| 100 | 400 | 1,600 | 100 | 400 | 1,600 | ||
| Procrustes | prompt | 0.669 0.018 | 0.693 0.007 | 0.687 0.003 | 0.529 0.004 | 0.536 0.005 | 0.536 0.001 |
| Procrustes | classifier | 0.632 0.008 | 0.671 0.006 | 0.698 0.002 | 0.592 0.005 | 0.636 0.004 | 0.669 0.002 |
| CCA | prompt | 0.654 0.019 | 0.673 0.009 | 0.662 0.006 | 0.529 0.004 | 0.532 0.006 | 0.531 0.001 |
| CCA | classifier | 0.552 0.012 | 0.608 0.005 | 0.682 0.004 | 0.538 0.007 | 0.582 0.005 | 0.636 0.002 |
| Model | Evaluation | Macro-F1 | AUROC |
|---|---|---|---|
| Generative VLMs, zero-shot Yes/No answers | |||
| RadFM [ 94 ] | native Yes/No | 0.2153 | – |
| MedGemma 1.5 [ 77 ] | Yes/No per finding | 0.3033 | – |
| M3D-LaMed [ 4 ] | one-token Yes/No | 0.3046 | – |
| ClinFusion-32B [ 99 ] | single Yes/No prompt | 0.3741 | – |
| ClinFusion-8B [ 99 ] | single Yes/No prompt | 0.4419 | – |
| Image encoder | Wasserstein– Procrustes FOSCTTM min–max ( ) | mini-vec2vec FOSCTTM | mini-vec2vec AUROC | Random pairs AUROC | Anatomy anchors AUROC | 100 true pairs AUROC | Anatomy anchors FOSCTTM |
|---|---|---|---|---|---|---|---|
| VoxelFM | 0.453–0.519 (9) | 0.517 0.010 | 0.487 | 0.430 | 0.671 | 0.698 | 0.396 |
| VoxelFM patch | 0.468–0.519 (9) | 0.499 0.018 | 0.480 | 0.437 | 0.666 | 0.693 | 0.400 |
| CT-FM | 0.485–0.515 (9) | 0.497 0.006 | 0.517 | 0.465 | 0.636 | 0.598 | 0.423 |
| V-JEPA 2-L | 0.447–0.474 (6) | 0.446 0.011 | 0.599 | 0.491 | 0.665 | 0.659 | 0.398 |
| V-JEPA 2-H | 0.445–0.495 (2) | 0.522 0.046 | 0.485 | 0.498 | 0.669 | 0.656 | 0.394 |
| V-JEPA 2-g | 0.440–0.503 (2) | 0.427 0.005 | 0.625 | 0.479 | 0.666 | 0.664 | 0.396 |
| CT combinations, prompt AUROC per seed | ||||||
| Image encoder | Text model | seed | unscaled | GW signs | oracle signs (needs pairs) | |
| VoxelFM block 11 | Qwen3.5-9B (middle layer) | 2197670066 | 0.530 | 0.327 | 0.694 | 0.658 |
| VoxelFM block 11 | Qwen3.5-9B (middle layer) | 576165995 | 0.549 | 0.363 | 0.687 | 0.595 |
| VoxelFM block 11 | Qwen3.5-9B (middle layer) | 734796314 | 0.515 | 0.358 | 0.702 | 0.655 |
| VoxelFM patch | Qwen3.5-4B (middle layer) | 2197670066 | 0.475 | 0.285 | 0.700 | 0.695 |
| VoxelFM patch | Qwen3.5-4B (middle layer) | 576165995 | 0.488 | 0.679 | 0.679 | 0.679 |
| Dataset | obs pred | Gaussian | settings |
|---|---|---|---|
| CT-RATE 3D CT | 0.016 | 0.006 | 2,550 |
| COCO | 0.008 | 0.005 | 68 |
| Flickr30k | 0.009 | 0.007 | 102 |
| IU X-ray | 0.030 | 0.008 | 102 |
| Clotho audio | 0.010 | 0.006 | 34 |
| Pilot | seed | image encoder picked | AUROC | gap | |
|---|---|---|---|---|---|
| 200 | 0 | 0.968 | VoxelFM patch | 0.722 | 0.000 |
| 200 | 1 | 0.806 | VoxelFM patch | 0.722 | 0.000 |
| 200 | 2 | 0.724 | V-JEPA 2-g | 0.712 | 0.010 |
| 200 | 3 | 0.786 | VoxelFM | 0.721 | 0.001 |
| 200 | 4 | 0.846 | VoxelFM | 0.721 | 0.001 |
| 400 | 0 | 0.911 | VoxelFM patch | 0.722 | 0.000 |
| Pilot | Pilot | |||
|---|---|---|---|---|
| Score | gap | gap | ||
| 0.703 | 0.0039 | 0.816 | 0.0007 | |
| 0.723 | 0.0025 | 0.909 | 0.0003 | |
| 0.641 | 0.0046 | 0.764 | 0.0008 | |
| m NN | 0.750 | 0.0039 | 0.758 | 0.0013 |
| CKA | 0.789 | 0.0087 | 0.811 | 0.0096 |
| Image–report pairs | 25 | 50 | 100 | 200 | 400 | 800 | 1,600 |
|---|---|---|---|---|---|---|---|
| All 5,994 reports | 0.643 | 0.687 | 0.727 | 0.754 | 0.782 | 0.796 | 0.809 |
| Reports of the pairs only | 0.652 | 0.694 | 0.734 | 0.759 | 0.785 | 0.798 | 0.811 |
| Reports ( ) | 200 | 500 | 1,000 | 2,000 | 4,000 | 5,994 | |
| reports | 0.759 | 0.759 | 0.759 | 0.759 | 0.756 | 0.754 |
| Image encoder | Qwen3.5-9B layer | 25 | 50 | 100 | 200 | 400 | 800 | 1,600 |
|---|---|---|---|---|---|---|---|---|
| VoxelFM block 10 | middle | 0.627 0.005 | 0.680 0.019 | 0.726 0.008 | 0.755 0.004 | 0.778 0.003 | 0.795 0.001 | 0.805 0.001 |
| VoxelFM block 11 | middle | 0.629 0.006 | 0.686 0.015 | 0.729 0.009 | 0.760 0.004 | 0.783 0.003 | 0.801 0.001 | 0.812 0.001 |
| VoxelFM block 12 (last) | middle | 0.638 0.006 | 0.686 0.014 | 0.726 0.008 | 0.756 0.004 | 0.779 0.003 | 0.797 0.001 | 0.805 0.004 |
| VoxelFM block 12 (last) | last | 0.638 0.018 | 0.685 0.013 | 0.722 0.010 | 0.751 0.004 | 0.776 0.003 | 0.789 0.001 | 0.801 0.001 |
| VoxelFM patch | middle | 0.599 0.013 | 0.660 0.013 | 0.708 0.010 | 0.738 0.002 | 0.763 0.003 | 0.782 0.001 | 0.796 0.001 |
| VoxelFM patch | last | 0.598 0.015 | 0.653 0.018 | 0.703 0.010 | 0.730 0.006 | 0.763 0.003 | 0.782 0.001 | 0.796 0.001 |
| mean AUROC, 18 findings | |||||
| scoring method | all (3039) | chest (3002) | study (1564) | patient (1293) | |
| 0 | prompt | 0.500[-0.247, -0.215] | 0.500[-0.250, -0.217] | 0.496[-0.254, -0.217] | 0.495[-0.263, -0.224] |
| 0 | classifier | 0.478[-0.265, -0.241] | 0.478[-0.267, -0.242] | 0.473[-0.270, -0.245] | 0.474[-0.280, -0.252] |
| 25 | prompt | 0.654[-0.084, -0.071] | 0.655[-0.085, -0.071] | 0.658[-0.080, -0.066] | 0.664[-0.083, -0.068] |
| 25 | classifier | 0.641[-0.097, -0.082] | 0.643[-0.098, -0.082] | 0.645[-0.094, -0.077] | 0.649[-0.099, -0.081] |
| 50 | prompt | 0.682[-0.056, -0.042] | 0.684[-0.056, -0.043] | 0.681[-0.057, -0.043] | 0.688[-0.059, -0.044] |
| mean AUROC, 16 findings | |||||
| scoring method | all (3039) | chest (3002) | study (1564) | patient (1293) | |
| 0 | prompt | 0.499[-0.252, -0.218] | 0.499[-0.254, -0.220] | 0.495[-0.257, -0.218] | 0.495[-0.264, -0.226] |
| 0 | classifier | 0.479[-0.266, -0.241] | 0.480[-0.267, -0.243] | 0.475[-0.271, -0.244] | 0.476[-0.278, -0.250] |
| 25 | prompt | 0.655[-0.086, -0.072] | 0.656[-0.087, -0.073] | 0.659[-0.081, -0.067] | 0.663[-0.084, -0.069] |
| 25 | classifier | 0.647[-0.095, -0.079] | 0.648[-0.096, -0.079] | 0.650[-0.091, -0.074] | 0.653[-0.095, -0.077] |
| 50 | prompt | 0.684[-0.057, -0.043] | 0.685[-0.057, -0.043] | 0.682[-0.058, -0.044] | 0.687[-0.060, -0.044] |
| split | prompt | ||||||
| AUROC | Atelectasis | Cardiomegaly | Consolidation | Edema | Pleural Effusion | ||
| test | 25 | 0.609[0.589, 0.630] | 0.603 | 0.589 | 0.606 | 0.638 | 0.610 |
| test | 50 | 0.656[0.633, 0.678] | 0.647 | 0.614 | 0.675 | 0.670 | 0.672 |
| test | 100 | 0.693[0.668, 0.717] | 0.689 | 0.652 | 0.702 | 0.721 | 0.701 |
| test | 200 | 0.725[0.699, 0.752] | 0.724 | 0.684 | 0.731 | 0.757 | 0.731 |
| test | 400 | 0.746[0.719, 0.773] | 0.750 | 0.692 | 0.751 | 0.773 | 0.761 |
| split | classifier | ||||||
| AUROC | Atelectasis | Cardiomegaly | Consolidation | Edema | Pleural Effusion | ||
| test | 25 | 0.591[0.575, 0.607] | 0.527 | 0.587 | 0.610 | 0.565 | 0.667 |
| test | 50 | 0.623[0.606, 0.641] | 0.513 | 0.624 | 0.616 | 0.637 | 0.724 |
| test | 100 | 0.678[0.661, 0.695] | 0.567 | 0.657 | 0.676 | 0.721 | 0.769 |
| test | 200 | 0.717[0.698, 0.735] | 0.592 | 0.702 | 0.716 | 0.766 | 0.807 |
| test | 400 | 0.750[0.731, 0.770] | 0.626 | 0.742 | 0.764 | 0.785 | 0.833 |
| Mean AUROC, prompt | ||||||||
| Image encoder | 0 | 25 | 50 | 100 | 200 | 400 | 800 | 1,600 |
| VoxelFM | 0.537 0.016 | 0.668 0.023 | 0.685 0.002 | 0.707 0.009 | 0.715 0.003 | 0.718 0.003 | 0.720 0.002 | 0.721 0.002 |
| VoxelFM patch | 0.522 0.028 | 0.642 0.023 | 0.663 0.023 | 0.700 0.009 | 0.713 0.003 | 0.716 0.003 | 0.720 0.003 | 0.722 0.000 |
| CT-FM | 0.456 0.021 | 0.556 0.045 | 0.574 0.021 | 0.599 0.028 | 0.632 0.013 | 0.659 0.007 | 0.671 0.002 | 0.690 0.001 |
| V-JEPA 2-L | 0.587 0.018 | 0.604 0.037 | 0.634 0.014 | 0.654 0.019 | 0.678 0.015 | 0.689 0.007 | 0.703 0.003 | 0.708 0.001 |
| V-JEPA 2-H | 0.599 0.027 | 0.639 0.024 | 0.651 0.012 | 0.663 0.018 | 0.680 0.006 | 0.696 0.005 | 0.707 0.002 | 0.713 0.001 |
| Dataset | Image encoder | Text model | unscaled FOSCTTM | FOSCTTM | |
| COCO | DINOv2 | MPNet | 0.277 | 0.463 0.091 | – |
| COCO | DINOv2 | Qwen3.5-4B (middle layer) | 0.195 | 0.475 0.035 | – |
| COCO | DINOv2 mean | MPNet | 0.451 | 0.293 0.029 | – |
| COCO | DINOv2 mean | Qwen3.5-4B (middle layer) | 0.371 | – | – |
| COCO | DINOv3 | MPNet | 0.266 | – | – |
| COCO | DINOv3 | Qwen3.5-4B (middle layer) | 0.188 | 0.475 0.020 | – |
| Dataset | Image encoder | Text model | unscaled FOSCTTM | FOSCTTM | |
| Clotho audio | Dasheng | MPNet | 0.093 0.000 | 0.530 0.014 | 0.237 0.093 |
| Clotho audio | Dasheng | Qwen3.5-4B (middle layer) | 0.091 0.000 | 0.518 0.029 | 0.332 0.013 |
| Clotho audio | EAT | MPNet | 0.054 0.000 | 0.504 0.035 | 0.375 0.021 |
| Clotho audio | EAT | Qwen3.5-4B (middle layer) | 0.049 0.000 | 0.485 0.018 | 0.390 0.029 |
| Clotho audio | EAT mean | MPNet | 0.098 0.000 | 0.512 0.022 | 0.329 0.031 |
| Clotho audio | EAT mean | Qwen3.5-4B (middle layer) | 0.104 0.000 | 0.525 0.025 | 0.329 0.021 |
| Dataset | Image encoder | Text model | unscaled FOSCTTM | FOSCTTM | |
| CT-RATE 3D CT | DINOv3 2.5D | MPNet | 0.047 | 0.503 0.015 | – |
| CT-RATE 3D CT | DINOv3 2.5D | Qwen3.5-4B (middle layer) | 0.062 | 0.497 0.005 | – |
| CT-RATE 3D CT | DINOv3 2D | MPNet | 0.054 | 0.488 0.012 | – |
| CT-RATE 3D CT | DINOv3 2D | Qwen3.5-4B (middle layer) | 0.068 | 0.509 0.022 | – |
| CT-RATE 3D CT | LTX VAE | MPNet | 0.005 | 0.492 0.016 | – |
| CT-RATE 3D CT | LTX VAE | Qwen3.5-4B (middle layer) | 0.007 | 0.510 0.024 | – |
| Dataset | Image encoder | Text model | unscaled FOSCTTM | FOSCTTM | |
| CT-RATE 3D CT | VoxelFM | Gemma 4 31B (middle layer) | – | 0.468 0.007 | – |
| CT-RATE 3D CT | VoxelFM | Granite 4.2 8B (middle layer) | – | 0.518 0.025 | – |
| CT-RATE 3D CT | VoxelFM | MedGemma 1.5 4B (middle layer) | – | 0.501 0.014 | – |
| CT-RATE 3D CT | VoxelFM | Ministral 3 14B (middle layer) | – | 0.503 0.012 | – |
| CT-RATE 3D CT | VoxelFM | OLMo 3.1 32B (middle layer) | – | 0.509 0.004 | – |
| CT-RATE 3D CT | VoxelFM | Qwen3-VL-Emb. 8B (middle layer) | – | 0.453 0.010 | – |
| Image encoder | Text model | seed | FOSCTTM | prompt AUROC | flipped | ||
| before | after | before | after | ||||
| COCO, no scaling | |||||||
| DINOv2 class token | MPNet | 734796314 | 0.478 | 0.478 | 0.453 | 0.453 | 0 |
| DINOv2 class token | MPNet | 576165995 | 0.561 | 0.563 | 0.448 | 0.448 | 0 |
| DINOv2 class token | MPNet | 2197670066 | 0.309 | 0.310 | 0.589 | 0.589 | 0 |
| DINOv2 class token | Qwen3.5-4B | 734796314 | 0.417 | 0.418 | 0.477 | 0.477 | 0 |
| CT-RATE, ROC-corner threshold | |||||||||
| scoring method | all (3039) | chest (3002) | study (1564) | patient (1293) | |||||
| F1 | Accuracy | F1 | Accuracy | F1 | Accuracy | F1 | Accuracy | ||
| 50 | prompt | 0.671 | 0.625 | 0.673 | 0.626 | 0.672 | 0.626 | 0.681 | 0.631 |
| 50 | classifier | 0.685 | 0.641 | 0.685 | 0.642 | 0.682 | 0.637 | 0.690 | 0.643 |
| 100 | prompt | 0.691 | 0.648 | 0.693 | 0.651 | 0.693 | 0.650 | 0.704 | 0.659 |
| 100 | classifier | 0.707 | 0.667 | 0.709 | 0.670 | 0.713 | 0.673 | 0.719 | 0.677 |
| CheXpert, threshold fit on the validation set | ||||
| scoring method | MCC | F1 | Accuracy | |
| 50 | prompt | 0.149[0.123, 0.174] | 0.319[0.293, 0.343] | 0.602[0.588, 0.616] |
| 50 | classifier | 0.150[0.129, 0.170] | 0.265[0.242, 0.284] | 0.695[0.682, 0.710] |
| 100 | prompt | 0.206[0.178, 0.233] | 0.386[0.355, 0.412] | 0.618[0.602, 0.635] |
| 100 | classifier | 0.191[0.169, 0.213] | 0.344[0.319, 0.365] | 0.691[0.680, 0.703] |
| 200 | prompt | 0.243[0.211, 0.277] | 0.406[0.374, 0.434] | 0.636[0.616, 0.657] |
| CheXpert, per-finding MCC | ||||||
| scoring method | Atelectasis | Cardiomegaly | Consolidation | Edema | Pleural Effusion | |
| 50 | prompt | 0.180 | 0.078 | 0.120 | 0.177 | 0.190 |
| 50 | classifier | 0.067 | 0.157 | 0.069 | 0.176 | 0.282 |
| 100 | prompt | 0.251 | 0.185 | 0.141 | 0.221 | 0.232 |
| 100 | classifier | 0.084 | 0.201 | 0.112 | 0.198 | 0.359 |
| 200 | prompt | 0.300 | 0.212 | 0.153 | 0.270 | 0.280 |
| Prompt AUROC | ||||||||
| Block | 0 | 25 | 50 | 100 | 200 | 400 | 800 | 1,600 |
| 2 | 0.481 0.004 | 0.510 0.015 | 0.518 0.011 | 0.528 0.011 | 0.544 0.020 | 0.616 0.012 | 0.638 0.005 | 0.666 0.002 |
| 4 | 0.486 0.009 | 0.517 0.024 | 0.526 0.010 | 0.547 0.040 | 0.633 0.045 | 0.670 0.008 | 0.692 0.003 | 0.697 0.001 |
| 6 | 0.490 0.021 | 0.566 0.039 | 0.579 0.026 | 0.620 0.030 | 0.679 0.009 | 0.687 0.006 | 0.699 0.001 | 0.705 0.001 |
| 8 | 0.501 0.015 | 0.603 0.036 | 0.616 0.026 | 0.675 0.008 | 0.697 0.004 | 0.705 0.005 | 0.710 0.003 | 0.718 0.001 |
| 10 | 0.506 0.020 | 0.647 0.029 | 0.676 0.009 | 0.703 0.007 | 0.715 0.002 | 0.717 0.002 | 0.721 0.003 | 0.723 0.001 |
| CT prompt AUROC at | |||||||
| Image encoder | CXR-BERT | Gemma 4 31B | Granite 4.2 8B | MedGemma 1.5 4B | Ministral 3 14B | OLMo 3.1 32B | Qwen3-VL-Emb. 8B |
| VoxelFM | 0.710 | 0.685 | 0.578 | 0.718 | 0.506 | 0.741 | 0.664 |
| VoxelFM patch | 0.708 | 0.686 | 0.572 | 0.721 | 0.508 | 0.740 | 0.665 |
| CT-FM | 0.674 | 0.645 | 0.570 | 0.687 | 0.526 | 0.697 | 0.634 |
| V-JEPA 2-L | 0.698 | 0.673 | 0.571 | 0.705 | 0.508 | 0.717 | 0.653 |
| V-JEPA 2-H | 0.701 | 0.682 | 0.578 | 0.711 | 0.508 | 0.724 | 0.660 |
| CT detection-threshold error, largest squared canonical correlation | |
| Text model | median obs pred |
| CXR-BERT | 0.010 |
| Gemma 4 31B | 0.010 |
| Granite 4.2 8B | 0.010 |
| MedGemma 1.5 4B | 0.009 |
| Ministral 3 14B | 0.010 |
| Other datasets, FOSCTTM of ridge regression | ||||||||
| Dataset | Image encoder | MPNet | Qwen3.5-0.8B (middle layer) | Qwen3.5-27B (middle layer) | Qwen3.5-2B (middle layer) | Qwen3.5-4B (middle layer) | Qwen3.5-9B (middle layer) | |
| COCO | DINOv2 | 1,600 | 0.011 0.000 | 0.056 0.002 | 0.036 0.002 | 0.055 0.001 | 0.052 0.002 | 0.051 0.002 |
| COCO | DINOv2 mean | 1,600 | 0.010 0.000 | 0.040 0.001 | 0.028 0.000 | 0.040 0.001 | 0.039 0.000 | 0.038 0.000 |
| COCO | DINOv3 | 1,600 | 0.010 0.000 | 0.054 0.004 | 0.037 0.002 | 0.053 0.004 | 0.052 0.004 | 0.051 0.004 |
| Flickr30k | DINOv2 | 1,600 | 0.015 0.001 | 0.094 0.003 | 0.061 0.003 | 0.093 0.005 | 0.081 0.003 | 0.080 0.003 |
| Flickr30k | DINOv2 mean | 1,600 | 0.011 0.001 | 0.083 0.000 | 0.055 0.002 | 0.085 0.000 | 0.074 0.002 | 0.070 0.001 |
| Image encoder | Qwen3.5-4B | Gemma 4 31B | Ministral 3 14B | OLMo 3.1 32B | Granite 4.2 8B | MedGemma 1.5 4B | Qwen3-VL-Emb. 8B | CXR-BERT |
| VoxelFM | 0.081 | 0.109 | 0.295 | 0.058 | 0.222 | 0.080 | 0.143 | 0.084 |
| VoxelFM patch | 0.065 | 0.078 | 0.277 | 0.049 | 0.206 | 0.063 | 0.128 | 0.075 |
| CT-FM | 0.025 | 0.054 | 0.179 | 0.016 | 0.146 | 0.027 | 0.078 | 0.034 |
| V-JEPA 2-L | 0.039 | 0.060 | 0.238 | 0.030 | 0.164 | 0.037 | 0.096 | 0.042 |
| V-JEPA 2-H | 0.043 | 0.060 | 0.244 | 0.033 | 0.171 | 0.044 | 0.098 | 0.049 |
| V-JEPA 2-g | 0.042 | 0.061 | 0.242 | 0.031 | 0.167 | 0.042 | 0.090 | 0.046 |
| FOSCTTM | ||||||
| Directions | unscaled | 2 | 4 | 8 | 16 | 32 |
| measurements | 0.504 0.014 | 0.507 0.020 | 0.511 0.014 | 0.559 0.078 | 0.607 0.016 | 0.512 0.123 |
| anchor findings | 0.504 0.014 | 0.506 0.018 | 0.495 0.029 | 0.533 0.083 | 0.554 0.082 | 0.543 0.078 |
| random directions | 0.504 0.014 | 0.504 0.007 | 0.506 0.006 | 0.501 0.013 | 0.500 0.003 | 0.497 0.017 |
| Prompt AUROC | ||||||
| Directions | unscaled | 2 | 4 | 8 | 16 | 32 |
| Whitening, ICA, and skew-ICA | |||||
| ICA 16 | ICA 64 | skew-ICA 16/128 | whitening 16 | whitening 64 | skew-ICA 10/64 |
| COCO, DINOv2 mean, MPNet | |||||
| 0.318 0.105 | 0.503 0.009 | 0.485 0.015 | 0.370 0.075 | 0.474 0.028 | – |
| CT-RATE 3D CT, DINOv2 2D, Qwen3.5-4B (middle layer) | |||||
| 0.476 0.023 | 0.499 0.009 | 0.499 0.002 | 0.490 0.007 | 0.491 0.009 | 0.489 0.009 |
| CT-RATE 3D CT, VoxelFM block 11, Qwen3.5-9B (middle layer) | |||||
| Image encoder | settings | median obs pred | BIMCV-R within 1 | CT = BIMCV-R | CT within 1 of obs. |
| VoxelFM | 8 | 0.022 | 1/8 | 0/4 | 4/4 |
| VoxelFM patch | 8 | 0.016 | 1/8 | 0/4 | 4/4 |
| CT-FM | 8 | 0.032 | 7/8 | 0/4 | 4/4 |
| V-JEPA 2-L | 8 | 0.032 | 6/8 | 2/4 | 4/4 |
| V-JEPA 2-H | 8 | 0.021 | 5/8 | 2/4 | 4/4 |
| V-JEPA 2-g | 8 | 0.024 | 4/8 | 2/4 | 4/4 |
| Layer or block | settings | median obs pred | max obs pred |
| Qwen3.5 size and layer, 15 encoders | |||
| Qwen3.5-0.8B, middle layer | 255 | 0.015 | 0.146 |
| Qwen3.5-0.8B, last layer | 255 | 0.013 | 0.153 |
| Qwen3.5-2B, middle layer | 255 | 0.015 | 0.140 |
| Qwen3.5-2B, last layer | 255 | 0.015 | 0.160 |
| Qwen3.5-4B, middle layer | 255 | 0.015 | 0.142 |
| Dataset | Image encoder | Text model | m NN | calibrated | CKA | calibrated | |||
| COCO | DINOv2 | MPNet | 2,020 | 0.257 | 0.253 | 0.005 | 0.445 | 0.445 | 0.005 |
| COCO | DINOv2 | Qwen3.5-4B (middle layer) | 2,020 | 0.097 | 0.092 | 0.005 | 0.212 | 0.212 | 0.005 |
| COCO | DINOv2 mean | MPNet | 2,020 | 0.224 | 0.220 | 0.005 | 0.567 | 0.567 | 0.005 |
| COCO | DINOv2 mean | Qwen3.5-4B (middle layer) | 2,020 | 0.094 | 0.090 | 0.005 | 0.323 | 0.323 | 0.005 |
| COCO | DINOv3 | MPNet | 2,020 | 0.258 | 0.254 | 0.005 | 0.434 | 0.434 | 0.005 |
| COCO | DINOv3 | Qwen3.5-4B (middle layer) | 2,020 | 0.096 | 0.092 | 0.005 | 0.211 | 0.211 | 0.005 |
| Dataset | Image encoder | Text model | m NN | calibrated | CKA | calibrated | |||
| CT-RATE 3D CT | DINOv3 2.5D | Qwen3.5-4B (middle layer) | 2,025 | 0.011 | 0.007 | 0.005 | 0.051 | 0.051 | 0.005 |
| CT-RATE 3D CT | DINOv3 2D | MPNet | 2,025 | 0.011 | 0.006 | 0.005 | 0.036 | 0.036 | 0.005 |
| CT-RATE 3D CT | DINOv3 2D | Qwen3.5-4B (middle layer) | 2,025 | 0.011 | 0.006 | 0.005 | 0.065 | 0.065 | 0.005 |
| CT-RATE 3D CT | LTX VAE | MPNet | 2,025 | 0.007 | 0.002 | 0.005 | 0.014 | 0.014 | 0.005 |
| CT-RATE 3D CT | LTX VAE | Qwen3.5-4B (middle layer) | 2,025 | 0.008 | 0.003 | 0.005 | 0.015 | 0.015 | 0.005 |
| CT-RATE 3D CT | VideoMAE-B | MPNet | 2,025 | 0.010 | 0.005 | 0.005 | 0.022 | 0.022 | 0.005 |
| Image encoder, layer | prompt AUROC | FOSCTTM | ||
| 100 | 1,600 | 100 | 1,600 | |
| VoxelFM, layer 0 | 0.665 0.010 | 0.685 0.002 | 0.331 0.012 | 0.243 0.002 |
| VoxelFM, layer 4 | 0.667 0.012 | 0.699 0.002 | 0.300 0.012 | 0.223 0.001 |
| VoxelFM, layer 8 | 0.696 0.009 | 0.712 0.001 | 0.289 0.007 | 0.215 0.003 |
| VoxelFM, layer 12 | 0.698 0.011 | 0.716 0.001 | 0.286 0.005 | 0.219 0.002 |
| VoxelFM, layer 16 | 0.707 0.009 | 0.720 0.001 | 0.297 0.007 | 0.237 0.002 |
| Image encoder, layer | prompt AUROC | FOSCTTM | ||
| 100 | 1,600 | 100 | 1,600 | |
| LTX VAE, layer 0 | 0.523 0.020 | 0.581 0.002 | 0.462 0.006 | 0.375 0.004 |
| LTX VAE, layer 4 | 0.530 0.010 | 0.595 0.005 | 0.454 0.007 | 0.373 0.003 |
| LTX VAE, layer 8 | 0.545 0.005 | 0.600 0.006 | 0.453 0.006 | 0.370 0.003 |
| LTX VAE, layer 12 | 0.545 0.003 | 0.607 0.009 | 0.451 0.004 | 0.369 0.004 |
| LTX VAE, layer 16 | 0.548 0.004 | 0.616 0.009 | 0.454 0.002 | 0.375 0.005 |
| Dataset | Image encoder | Text model | seed | enumerate 10 | greedy 40 | skew 40 | best pattern (needs pairs) |
| Flickr30k | DINOv2 | MPNet | 2197670066 | 0.179 | 0.180 | 0.183 | 0.169 |
| Flickr30k | DINOv2 | MPNet | 576165995 | 0.123 | 0.129 | 0.123 | 0.123 |
| Flickr30k | DINOv2 | MPNet | 734796314 | 0.152 | 0.161 | 0.149 | 0.150 |
| Flickr30k | DINOv2 mean | MPNet | 2197670066 | 0.099 | 0.102 | 0.099 | 0.095 |
| Flickr30k | DINOv2 mean | MPNet | 576165995 | 0.091 | 0.098 | 0.091 | 0.091 |
| Flickr30k | DINOv2 mean | MPNet | 734796314 | 0.028 | 0.032 | 0.028 | 0.028 |
| Dataset | Image encoder | Text model | seed | enumerate 10 | greedy 40 | skew 40 | best pattern (needs pairs) |
| Clotho audio | AST | MPNet | 576165995 | 0.066 | 0.066 | 0.066 | 0.066 |
| Clotho audio | AST | MPNet | 734796314 | 0.066 | 0.066 | 0.066 | 0.066 |
| Clotho audio | AST | Qwen3.5-4B (middle layer) | 2197670066 | 0.306 | 0.301 | 0.303 | 0.292 |
| Clotho audio | AST | Qwen3.5-4B (middle layer) | 576165995 | 0.097 | 0.101 | 0.097 | 0.097 |
| Clotho audio | AST | Qwen3.5-4B (middle layer) | 734796314 | 0.148 | 0.154 | 0.148 | 0.148 |
| Clotho audio | Dasheng | MPNet | 2197670066 | 0.255 | 0.268 | 0.460 | 0.217 |
| Pairs | Untuned image classifier | Tuned image classifier | LightAlign |
|---|---|---|---|
| 100 | 0.689[-0.051, -0.036] | 0.698[-0.043, -0.028] | 0.727[-0.013, 0.002] |
| 200 | – | 0.735[-0.006, 0.008] | 0.754[0.014, 0.028] |
| 400 | 0.713[-0.028, -0.012] | 0.767[0.027, 0.041] | 0.782[0.041, 0.056] |
| 1,600 | 0.712[-0.029, -0.012] | 0.806[0.065, 0.080] | 0.809[0.068, 0.084] |
| 5,994 | – | 0.824[0.083, 0.099] | 0.824[0.083, 0.100] |
| Image encoder | Qwen3.5-9B layer | 100 | 200 | 1,600 | 5,994 |
|---|---|---|---|---|---|
| VoxelFM block 11 | middle | 0.727[-0.013, 0.002] | 0.754[0.014, 0.028] | 0.809[0.068, 0.084] | 0.824[0.083, 0.100] |
| VoxelFM block 12 (last) | middle | 0.728[-0.012, 0.002] | 0.753[0.013, 0.027] | 0.807[0.066, 0.082] | 0.824[0.083, 0.099] |
| VoxelFM block 11 | last | 0.723[-0.017, -0.003] | 0.748[0.008, 0.022] | 0.808[0.067, 0.082] | 0.825[0.084, 0.100] |
| VoxelFM block 12 (last) | last | 0.725[-0.015, -0.001] | 0.748[0.008, 0.021] | 0.806[0.066, 0.081] | 0.824[0.084, 0.099] |
Cross-Modal Alignment
Cross-modal alignment maps representations from different data types into compatible spaces so models can compare, transfer, and combine information.
See all 470 papers