To Align or Not To Align: Check Your COMPASS Before You Train

Fabian Gröger; Petar Damjanović; Maria Brbić

cs.AIOct 13, 2026

To Align or Not To Align: Check Your COMPASS Before You Train

Authors: Fabian Gröger, Petar Damjanović, Maria Brbić

Organizations: EPFL · University of Basel · HSLU

Abstract

Combining pretrained models is a common strategy for building AI systems, whether by aligning a vision encoder with a language model or merging fine-tuned checkpoints. However, finding compatible pairs typically requires costly empirical search, as compatibility is known only after alignment or merging. We propose COMPASS (COMPatibility ASsessment Score), a theoretically motivated diagnostic that estimates whether a model pair will be compatible prior to alignment or merging. Building on the observation that representational convergence is most robust at the level of local neighborhoods, COMPASS decomposes model compatibility into two cheap-to-compute axes: a pair-specific ease, defined as the $k$ -NN overlap between the two encoders, and a paradigm-level ceiling, capturing the maximum overlap their training paradigms allow. Together, these two axes provide a ranking of candidate model pairs and prescribe how to act on each pair: (i) prioritize pairs that already agree and whose paradigms are compatible, (ii) invest additional compute on pairs with compatible paradigms but low initial agreement, (iii) switch the pretraining paradigm when it cannot support sufficient agreement, or (iv) deprioritize pairs that score poorly on both axes. We apply COMPASS to $72$ cross-modal alignment experiments and $392$ task-arithmetic model merges and show that compatibility estimates by COMPASS strongly correlate with downstream success, achieving average Pearson correlation of $0.91$ on cross-modal alignment tasks and $0.70$ on model merging.

Figures & tables

Figure 1: Given a library of candidate encoders, the compatibility of a model pair is typically unknown before running the downstream transfer. Using COMPASS , we estimate the compatibility based on local neighborhood agreement: for each probe sample, we compare its local

k

-NN overlap across the two representation spaces. Aggregating this local overlap across different axes gives two scores computed before alignment or merging. Ease is a pair-specific score that measures how close the selected pair is at the start of the relevant dynamics: before encoder training in cross-modal alignment, and before weight merging in task arithmetic. Ceiling is a paradigm-level score that estimates how much agreement the corresponding training paradigms can support on average. Candidate pairs are ranked by combining ease and ceiling. Each axis is thresholded at its median value: green compatibility scores are above the corresponding median while the red ones are below it. The resulting high/low pattern assigns each pair to one of four regimes: act now (high ease, high ceiling), budget-limited (low ease, high ceiling), ceiling-limited (high ease, low ceiling), or incompatible (low ease, low ceiling).

Figure 2: Theorem 4.1 across four settings, each showing a start-condition score against the corresponding endpoint. Top: unimodal validation. Initial random

k

-NN overlap predicts final pretrained overlap across

n=21

vision family pairs (left;

r=0.69

p<10^{-4}

) and

n=42

text family pairs (right;

r=0.91

p<10^{-17}

). Bottom left: cross-modal alignment. Pretrained

k

-NN overlap predicts Flickr30k I2T R@1 after alignment (

r=0.84

p<10^{-20}

) across

n=72

vision-language pairs. Bottom right: model merging via task arithmetic.

k

-NN overlap between fine-tuned checkpoints predicts merge quality (

r=0.67

p<10^{-52}

) across

n=392

model merges. Color encodes the vision/text family throughout. In the top panels (unimodal) and bottom-left (cross-modal alignment), marker shape encodes the other family of the pair (text family in the cross-modal case). The bottom-right merging panel uses a single shape, since each point has a single base-model family. Initialization structure that underlies the unimodal panels is reported in Section H.1 . The same qualitative pattern holds at

k\in\{20,50,100\}

( Section H.2 ).

Figure 3: Absolute Pearson correlations between each candidate predictor and the downstream outcome; significance against zero is indicated by

{}^{*}\!p<0.05

{}^{**}\!p<0.01

{}^{***}\!p<0.001

. Left/Center: cross-modal alignment evaluated on Flickr30k and COCO I2T R@1 (

n=72

pairs). Ease (pretrained

k

-NN overlap) and ceiling (paradigm-level cross-modal overlap) dominate, with their product ( Ease

\times

Ceiling ) reaching

|r|=0.90

(Flickr30k) and

|r|=0.91

(COCO). Right: task-arithmetic merging against mean retention (

n=392

). The product predictor ( Ease

\times

Ceiling ) reaches

|r|=0.70

, with ease at

|r|=0.67

and ceiling at

|r|=0.64

alone.

Figure 4: Alignability phase diagram ( $n=72$ pairs). $x$ -axis: paradigm-level mean cross-modal $k$ -NN overlap (ceiling). $y$ -axis: pretrained $k$ -NN overlap (ease). Color and size encode retrieval I2T R@1 after alignment. Left: Flickr30k. Right: COCO. The quadrant structure is consistent across both benchmarks

Figure 5: Mergeability phase diagram ( $n=392$ pairs). $x$ -axis: paradigm specialization , defined as $1-$ mean across-paradigm $k$ -NN overlap from the convergence experiments (ceiling). $y$ -axis: pair-specific pre-merge $k$ -NN overlap (ease). Color and size encode merge quality.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Value
Adapter	Normalized canonical correlation analysis (CSA, 25 )
Encoders	Frozen
Canonical dimension	$512$
Equal-weight constraint	No
Regularization	None
Layer-selection probe size	$5{,}000$ paired image-caption embeddings

Appendix

Table 1: Cross-modal alignment hyperparameters.

Component	Value
Pretrained source	Public timm checkpoints [ 31 ]
Fine-tuning method	Full fine-tuning
Optimizer	AdamW [ 48 ]
Weight decay	$0.01$
Backbone learning rate	$10^{-4}$
Classifier-head learning rate	$10^{-3}$

Appendix

Table 2: Task-arithmetic fine-tuning and merging hyperparameters.

Dataset	Used for	License
WIT-1024 [ 26 ]	probe set, $k$ -NN overlap	CC BY-SA 3.0
MS-COCO [ 18 ]	alignment training, retrieval	CC BY 4.0
Flickr30k [ 17 ]	retrieval evaluation	Flickr terms of use
Task-arithmetic merging benchmark [ 11 ]
MNIST	merging fine-tuning	CC BY-SA 3.0
SVHN	merging fine-tuning	non-commercial research use

Appendix

Table 3: Datasets used in this paper with their licenses.

Figure 6: Pearson

r

between calibrated alignment and model scale (

k=10

), for each fixed reference model. Left: for cross-modal, local

k

-NN (green) shows positive scaling (mean

r\approx 0.64

), while global CKA (blue) is weaker after calibration (mean

r\approx 0.37

). Center: within-vision. Right: within-text. The local-global gap motivates measuring local

k

-NN overlap rather than global CKA.

Panel of Figure 7 — Figure 7: Initialization structure underlying the unimodal panels of Figure 2 . Both modalities show structured random geometry within architecture families, lower cross-family random overlap, and a large drop between pretrained and random representations.

Figure 8: Combined proxies figure at

k=20

. Top: unimodal init-to-final validation for vision and text. Bottom left: pretrained

k

-NN overlap predicts Flickr30k I2T R@1 after alignment. Bottom right: pre-merge

k

-NN overlap predicts merge quality. The same qualitative pattern as at

k=10

is preserved.

Figure 9: Combined proxies figure at

k=50

. The start-condition overlap remains predictive at larger neighborhood size across all four settings.

Figure 10: Combined proxies figure at

k=100

. Even at a large neighborhood size, the qualitative structure is unchanged.

Ease probe	$\|r\|$ ease	$\|r\|$ ceiling	$\|r\|$ product
MNIST (2K samples, main text)	$0.67$	$0.64$	$0.70$
8-task pool (256 / task, 2,048 samples)	$0.42$	$0.64$	$0.65$

Appendix

Table 4: Probe-set sensitivity for the merging ease score. Replacing the MNIST probe with an 8-task pool weakens the ease signal (the pool dilutes per-pair specialization across 8 distributions), but the product predictor remains positively correlated with merge quality. The same direction holds: pairs with higher ease and higher ceiling merge better.

Figure 11: Left: pair-specific $k$ -NN overlap by paradigm, with median lines. The paradigm ordering is stable across task pairs ( $n=392$ pairwise merges). Color encodes pairwise merge quality (mean retention across the two original tasks). Marker shape encodes base model family. Right: full multi-task task arithmetic (all 8 tasks summed jointly), reporting mean accuracy across all 8 tasks. The pairwise ordering of Figure 5 is preserved at the multi-task scale: discriminative ( $0.52/0.37$ multi-task accuracy) and self-distillation ( $0.45/0.34$ ) retain the most, contrastive VL collapses ( $0.10/0.09$ ). Note that these multi-task accuracies differ from the pairwise mean retentions reported in Section 5.2 , since they aggregate across all 8 tasks at once rather than over pairs of tasks.

Figure 12: Per-regime distribution of downstream outcomes for the three settings. Left: Flickr30k I2T R@1 on

n=72

vision-language pairs. Center: COCO I2T R@1 on the same pairs. Right: mean retention on

n=392

task-arithmetic merges. Boxes show the interquartile range with the median line; whiskers extend to the non-outlier extremes; individual points are overlaid. Color encodes the four regimes: act now (high ease, high ceiling), budget-limited (low ease, high ceiling), ceiling-limited (high ease, low ceiling), and incompatible (low ease, low ceiling). The Kruskal-Wallis

H

statistic and

p

-value at the top of each panel test the null that the four regimes share the same distribution; the test rejects in all three settings.

Outcome	COMPASS $\rho$	COMPASS $\tau$	Ease $\rho$	Ceiling $\rho$	Baseline $\rho$
Flickr30k I2T R@1	0.91	0.73	0.87	0.77	0.63
Flickr30k T2I R@1	0.91	0.73	0.87	0.77	0.65
COCO I2T R@1	0.93	0.77	0.88	0.81	0.67
COCO T2I R@1	0.93	0.77	0.88	0.81	0.66
Merge retention	0.70	0.50	0.66	0.61	0.06

Appendix

Table 5: Rank correlations with downstream transfer performance. The final column uses global CKA for alignment and task-vector cosine for merging.

Predictor	Top-1	Regret	Top-5	Regret	NDCG@5	Top-10	Regret	NDCG@10	Top-20	Regret	NDCG@20	Budget to best
COMPASS	0.635	0.0%	0.578	5.9%	0.957	0.530	5.7%	0.952	0.493	2.1%	0.972	1/72
Ease	0.428	32.6%	0.521	15.2%	0.819	0.530	5.7%	0.893	0.486	3.6%	0.923	2/72
Ceiling	0.635	0.0%	0.563	8.3%	0.945	0.455	19.0%	0.864	0.439	12.8%	0.889	1/72
Global CKA	0.635	0.0%	0.589	4.2%	0.970	0.515	8.2%	0.940	0.440	12.7%	0.901	1/72
Paradigm distance	0.323	49.1%	0.204	66.8%	0.368	0.226	59.8%	0.406	0.245	51.3%	0.458	72/72
Total parameters	0.418	34.2%	0.352	42.6%	0.614	0.328	41.6%	0.609	0.316	37.3%	0.629	21/72

Appendix

Table 6: Selection performance on Flickr30k I2T retrieval (

n=72

candidate pairs). Regret is relative to the oracle top-

k

set.

Predictor	Top-1	Regret	Top-10	Regret	NDCG@10	Top-40	Regret	NDCG@40	Budget to best
COMPASS	0.947	8.4%	0.927	7.1%	0.920	0.890	9.1%	0.910	5/392
Ease	0.947	8.4%	0.935	6.4%	0.924	0.874	10.7%	0.896	5/392
Ceiling	0.968	6.4%	0.926	7.3%	0.930	0.828	15.5%	0.871	162/392
Task-vector cosine	0.963	6.8%	0.826	17.3%	0.841	0.708	27.7%	0.749	328/392
Random ordering	0.585	43.4%	0.585	41.4%	-	0.585	40.3%	-	196.5 expected

Appendix

Table 7: Selection performance for task-arithmetic merging (

n=392

candidate merges).

Pairs trained	1	2	5	10	20
COMPASS	1.00	1.00	1.00	1.00	1.00
Ease	0.67	1.00	1.00	1.00	1.00
Global CKA	1.00	1.00	1.00	1.00	1.00
Random order	0.53	0.66	0.79	0.88	0.95

Appendix

Table 8: Budget selection for Flickr30k I2T retrieval. Values are the best outcome found within the evaluation budget, normalized by the pool optimum.

Merges performed	1	5	10	20	50
COMPASS	0.92	1.00	1.00	1.00	1.00
Ease	0.92	1.00	1.00	1.00	1.00
Ceiling	0.94	0.94	0.95	0.96	0.97
Task-vector cosine	0.93	0.93	0.94	0.95	0.95
Random order	0.57	0.87	0.93	0.95	0.97

Appendix

Table 9: Budget selection for task-arithmetic merging.

Predictor	Alignment: Flickr I2T	Merging: retention
COMPASS	0.91	0.70
Pretraining paradigm distance	-0.60	-
Architecture self-consistency	0.37	-
Total parameters	0.09	-
Random-init $k$ -NN overlap	0.11	-
Mean solo accuracy of two checkpoints	-	0.43

Appendix

Table 10: Comparison with cheap metadata-based predictors.

Metric	Alignment: Flickr I2T	Merging: retention
Mutual $k$ -NN overlap	0.82	0.42
RSA, rank correlation between RDMs	0.38	0.22
Procrustes similarity	0.45	0.37
Linear CKA	0.34	0.25

Appendix

Table 11: Alternative representation-similarity metrics computed on identical inputs.

Evaluation	Alignment	Merging
COMPASS, nothing fitted	0.91	0.70
Regressor, random 5-fold CV	0.94	0.91
Regressor, leave-one-family/backbone out	0.60	0.81
Regressor, leave-one-paradigm out	0.46	-0.54

Appendix

Table 12: Comparison with learned metadata regressors. Unlike COMPASS, these models require previously observed downstream transfer outcomes as labels.

Fold type	Folds	Spearman on rest	Spearman inside held-out group
Leave-one-vision-family-out	6	0.85-0.93	0.81 median (0.56-0.87)
Leave-one-text-family-out	7	0.89-0.92	0.86 median (0.78-0.94)
Leave-one-vision-paradigm-out	5	0.85-0.93	0.80 median (0.56-0.87)
Leave-one-backbone-out (merging)	14	0.67-0.71	0.28 median (-0.13-0.75)
Leave-one-paradigm-out (merging)	5	0.60-0.72	0.27 median (0.20-0.66)

Appendix

Table 13: Generalization under held-out groups. The pooled references are

\rho=0.91

for Flickr30k alignment and

\rho=0.70

for merging.

Held-out vision family	Rest	Held out	Held-out text family	Rest	Held out
CLIP	0.91	0.78	CLIP text	0.92	0.78
SigLIP	0.89	0.87	SigLIP text	0.89	0.86
CLIP fine-tuned	0.92	0.83	OLMo	0.92	0.86
DINOv2	0.93	0.80	BLOOM	0.91	0.90
MAE	0.85	0.87	RoBERTa	0.90	0.94
Classification	0.92	0.56	LLaMA	0.91	0.80

Appendix

Table 14: Per-fold results for Flickr30k I2T alignment.

	CSA	Linear	MLP	LoRA
CSA	-	0.94	0.82	0.96
Linear	0.87	-	0.86	0.99
MLP	0.76	0.84	-	0.85
LoRA	0.91	0.98	0.85	-

Appendix

Table 15: Pair-ranking agreement across alignment procedures. Values below the diagonal are Spearman correlations for Flickr30k I2T and values above the diagonal are for T2I.

Predictor	CSA	Linear	MLP	LoRA
COMPASS	0.87	0.90	0.73	0.92
Ease	0.78	0.78	0.56	0.78
Ceiling	0.86	0.97	0.76	0.95
Global CKA	0.71	0.79	0.60	0.78

Appendix

Table 16: Spearman correlation with Flickr30k I2T R@1 under different alignment procedures.

Procedure	Scaling	Cost for full $19\times 10$ grid
COMPASS , including ceiling	One probe pass per model	29 passes; $\sim$ 21 min; 12 GB cached features
Exhaustive downstream evaluation	One alignment run per pair	190 runs; $\sim$ 86–102 h; 729 GB cached features

Appendix

Table 17: Cost of COMPASS scoring compared with exhaustive downstream alignment. The wall-clock comparison was benchmarked on a single RTX 2080 Ti.

Ceiling estimate	Alignment: Flickr30k I2T	Merging: retention
Original estimate	0.909	0.696
One shared value per paradigm	0.903	0.699
Gaussian noise at 25% of ceiling spread	0.901	0.695
Gaussian noise at 50% of ceiling spread	0.883	0.695
Gaussian noise at 100% of ceiling spread	0.830	0.691
Ceiling values shuffled across paradigms	0.675	0.654

Appendix

Table 18: Sensitivity to ceiling granularity and estimation noise. Values are Spearman correlations with downstream performance.

Vision family held fixed	Pairs	Ease vs. R@1	COMPASS vs. R@1	Within-family ceiling spread
CLIP only	12	0.67	0.78	0.013
SigLIP only	12	0.85	0.87	0.023
CLIP fine-tuned only	10	0.83	0.83	0.017
MAE only	12	0.87	0.87	0.016
DINOv2 only	13	0.77	0.80	0.024
Classification only	13	0.43	0.56	0.009

Appendix

Table 19: Predictive power within individual vision-model families on Flickr30k I2T retrieval.

Encoder	Ceiling before ft	Ceiling after ft	Mean R@1 before	Mean R@1 after
CLIP ViT-B/16	0.138	0.109	0.396	0.319
CLIP ViT-L/14	0.147	0.117	0.579	0.548
CLIP ViT-H/14	0.151	0.127	0.628	0.610

Appendix

Table 20: Change in cross-modal ceiling and mean Flickr30k I2T R@1 after ImageNet-12K classification fine-tuning.

Vision encoder	Ceiling	10 ep	30 ep	100 ep	300 ep
MAE ViT-B	0.055	0.142	0.183	0.216	0.215
AugReg ViT-B	0.101	0.269	0.356	0.411	0.420
DINOv2 ViT-B	0.106	0.362	0.480	0.477	0.471
CLIP ViT-B ft-IN12k	0.109	0.288	0.370	0.454	0.459
SigLIP ViT-B	0.130	0.359	0.472	0.544	0.559
CLIP ViT-B LAION-2B	0.138	0.424	0.528	0.587	0.607

Appendix

Table 21: Flickr30k I2T R@1 as the alignment-training budget increases, with OLMo-1B held fixed.

Language Model Merging

Language model merging combines capabilities from multiple models or adapters by composing their weights, updates, or outputs.

See all 106 papers

Task Alignment: A Simple Proxy for Practical Model Merging Across Diverse Vision Tasks
Apr 14, 2026Pau de Jorge, César Roberto de Souza, Björn Michele +5Model Merging
NAVER LABS Europe
AlignMerge - Alignment-Preserving Large Language Model Merging via Fisher-Guided Geometric Constraints
Dec 18, 2025Aniruddha Roy, Jyoti Patel, Aman Chadha +2Language Model Merging LLM Alignment
HCL · Evalueserve · Apple (USA) +2
Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Merging
Oct 20, 2025Tiancheng Hu, Benjamin Minixhofer, Nigel CollierLLM Alignment Model Calibration
University of Cambridge

Apr 14, 2026J/K move · Enter open · S save

Task Alignment: A Simple Proxy for Practical Model Merging Across Diverse Vision Tasks

Pau de Jorge, César Roberto de Souza, Björn Michele, Mert Bülent Sarıyıldız+4

NAVER LABS Europe

Model Merging

Open paper AskPDF arXiv

Efficiently merging several models fine-tuned for different tasks, but stemming from the same pretrained base model, is of great practical interest. Despite extensive prior work, most evaluations of model merging in computer vision are restricted to image classification using CLIP, where different classification datasets define different tasks. In this work, our goal is to make model merging more practical and show its relevance on challenging scenarios beyond this specific setting. In most vision scenarios, different tasks rely on trainable and usually heterogeneous decoders. Differently from previous studies with frozen decoders, where merged models can be evaluated right away, the non-trivial cost of decoder training renders hyperparameter selection based on downstream performance impractical. To address this, we introduce the task alignment proxy, and show how it can be used to speed up hyperparameter selection by orders of magnitude while retaining performance. Equipped with the task alignment proxy, we extend the applicability of model merging to multi-task vision models beyond CLIP-based classification. Project page: https://europe.naverlabs.com/task-alignment

Abstract

Language Model Merging

Explore similar work

Task Alignment: A Simple Proxy for Practical Model Merging Across Diverse Vision Tasks

AlignMerge - Alignment-Preserving Large Language Model Merging via Fisher-Guided Geometric Constraints

Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Merging