To Align or Not To Align: Check Your COMPASS Before You Train

Fabian Gröger; Petar Damjanović; Maria Brbić

cs.AIOct 13, 2026

To Align or Not To Align: Check Your COMPASS Before You Train

Authors: Fabian Gröger, Petar Damjanović, Maria Brbić

Organizations: EPFL · University of Basel · HSLU

Abstract

Combining pretrained models is a common strategy for building AI systems, whether by aligning a vision encoder with a language model or merging fine-tuned checkpoints. However, finding compatible pairs typically requires costly empirical search, as compatibility is known only after alignment or merging. We propose COMPASS (COMPatibility ASsessment Score), a theoretically motivated diagnostic that estimates whether a model pair will be compatible prior to alignment or merging. Building on the observation that representational convergence is most robust at the level of local neighborhoods, COMPASS decomposes model compatibility into two cheap-to-compute axes: a pair-specific ease, defined as the $k$ -NN overlap between the two encoders, and a paradigm-level ceiling, capturing the maximum overlap their training paradigms allow. Together, these two axes provide a ranking of candidate model pairs and prescribe how to act on each pair: (i) prioritize pairs that already agree and whose paradigms are compatible, (ii) invest additional compute on pairs with compatible paradigms but low initial agreement, (iii) switch the pretraining paradigm when it cannot support sufficient agreement, or (iv) deprioritize pairs that score poorly on both axes. We apply COMPASS to $72$ cross-modal alignment experiments and $392$ task-arithmetic model merges and show that compatibility estimates by COMPASS strongly correlate with downstream success, achieving average Pearson correlation of $0.91$ on cross-modal alignment tasks and $0.70$ on model merging.

Figures & tables

Figure 1: Given a library of candidate encoders, the compatibility of a model pair is typically unknown before running the downstream transfer. Using COMPASS , we estimate the compatibility based on local neighborhood agreement: for each probe sample, we compare its local

k

-NN overlap across the two representation spaces. Aggregating this local overlap across different axes gives two scores computed before alignment or merging. Ease is a pair-specific score that measures how close the selected pair is at the start of the relevant dynamics: before encoder training in cross-modal alignment, and before weight merging in task arithmetic. Ceiling is a paradigm-level score that estimates how much agreement the corresponding training paradigms can support on average. Candidate pairs are ranked by combining ease and ceiling. Each axis is thresholded at its median value: green compatibility scores are above the corresponding median while the red ones are below it. The resulting high/low pattern assigns each pair to one of four regimes: act now (high ease, high ceiling), budget-limited (low ease, high ceiling), ceiling-limited (high ease, low ceiling), or incompatible (low ease, low ceiling).

Figure 2: Theorem 4.1 across four settings, each showing a start-condition score against the corresponding endpoint. Top: unimodal validation. Initial random

k

-NN overlap predicts final pretrained overlap across

n=21

vision family pairs (left;

r=0.69

p<10^{-4}

) and

n=42

text family pairs (right;

r=0.91

p<10^{-17}

). Bottom left: cross-modal alignment. Pretrained

k

-NN overlap predicts Flickr30k I2T R@1 after alignment (

r=0.84

p<10^{-20}

) across

n=72

vision-language pairs. Bottom right: model merging via task arithmetic.

k

-NN overlap between fine-tuned checkpoints predicts merge quality (

r=0.67

p<10^{-52}

) across

n=392

model merges. Color encodes the vision/text family throughout. In the top panels (unimodal) and bottom-left (cross-modal alignment), marker shape encodes the other family of the pair (text family in the cross-modal case). The bottom-right merging panel uses a single shape, since each point has a single base-model family. Initialization structure that underlies the unimodal panels is reported in Section H.1 . The same qualitative pattern holds at

k\in\{20,50,100\}

( Section H.2 ).

Figure 3: Absolute Pearson correlations between each candidate predictor and the downstream outcome; significance against zero is indicated by

{}^{*}\!p<0.05

{}^{**}\!p<0.01

{}^{***}\!p<0.001

. Left/Center: cross-modal alignment evaluated on Flickr30k and COCO I2T R@1 (

n=72

pairs). Ease (pretrained

k

-NN overlap) and ceiling (paradigm-level cross-modal overlap) dominate, with their product ( Ease

\times

Ceiling ) reaching

|r|=0.90

(Flickr30k) and

|r|=0.91

(COCO). Right: task-arithmetic merging against mean retention (

n=392

). The product predictor ( Ease

\times

Ceiling ) reaches

|r|=0.70

, with ease at

|r|=0.67

and ceiling at

|r|=0.64

alone.

Figure 4: Alignability phase diagram ( $n=72$ pairs). $x$ -axis: paradigm-level mean cross-modal $k$ -NN overlap (ceiling). $y$ -axis: pretrained $k$ -NN overlap (ease). Color and size encode retrieval I2T R@1 after alignment. Left: Flickr30k. Right: COCO. The quadrant structure is consistent across both benchmarks

Figure 5: Mergeability phase diagram ( $n=392$ pairs). $x$ -axis: paradigm specialization , defined as $1-$ mean across-paradigm $k$ -NN overlap from the convergence experiments (ceiling). $y$ -axis: pair-specific pre-merge $k$ -NN overlap (ease). Color and size encode merge quality.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Value
Adapter	Normalized canonical correlation analysis (CSA, 25 )
Encoders	Frozen
Canonical dimension	$512$
Equal-weight constraint	No
Regularization	None
Layer-selection probe size	$5{,}000$ paired image-caption embeddings

Appendix

Table 1: Cross-modal alignment hyperparameters.

Component	Value
Pretrained source	Public timm checkpoints [ 31 ]
Fine-tuning method	Full fine-tuning
Optimizer	AdamW [ 48 ]
Weight decay	$0.01$
Backbone learning rate	$10^{-4}$
Classifier-head learning rate	$10^{-3}$

Appendix

Table 2: Task-arithmetic fine-tuning and merging hyperparameters.

Dataset	Used for	License
WIT-1024 [ 26 ]	probe set, $k$ -NN overlap	CC BY-SA 3.0
MS-COCO [ 18 ]	alignment training, retrieval	CC BY 4.0
Flickr30k [ 17 ]	retrieval evaluation	Flickr terms of use
Task-arithmetic merging benchmark [ 11 ]
MNIST	merging fine-tuning	CC BY-SA 3.0
SVHN	merging fine-tuning	non-commercial research use

Appendix

Table 3: Datasets used in this paper with their licenses.

Figure 6: Pearson

r

between calibrated alignment and model scale (

k=10

), for each fixed reference model. Left: for cross-modal, local

k

-NN (green) shows positive scaling (mean

r\approx 0.64

), while global CKA (blue) is weaker after calibration (mean

r\approx 0.37

). Center: within-vision. Right: within-text. The local-global gap motivates measuring local

k

-NN overlap rather than global CKA.

Panel of Figure 7 — Figure 7: Initialization structure underlying the unimodal panels of Figure 2 . Both modalities show structured random geometry within architecture families, lower cross-family random overlap, and a large drop between pretrained and random representations.

Figure 8: Combined proxies figure at

k=20

. Top: unimodal init-to-final validation for vision and text. Bottom left: pretrained

k

-NN overlap predicts Flickr30k I2T R@1 after alignment. Bottom right: pre-merge

k

-NN overlap predicts merge quality. The same qualitative pattern as at

k=10

is preserved.

Figure 9: Combined proxies figure at

k=50

. The start-condition overlap remains predictive at larger neighborhood size across all four settings.

Figure 10: Combined proxies figure at

k=100

. Even at a large neighborhood size, the qualitative structure is unchanged.

Ease probe	$\|r\|$ ease	$\|r\|$ ceiling	$\|r\|$ product
MNIST (2K samples, main text)	$0.67$	$0.64$	$0.70$
8-task pool (256 / task, 2,048 samples)	$0.42$	$0.64$	$0.65$

Appendix

Table 4: Probe-set sensitivity for the merging ease score. Replacing the MNIST probe with an 8-task pool weakens the ease signal (the pool dilutes per-pair specialization across 8 distributions), but the product predictor remains positively correlated with merge quality. The same direction holds: pairs with higher ease and higher ceiling merge better.

Figure 11: Left: pair-specific $k$ -NN overlap by paradigm, with median lines. The paradigm ordering is stable across task pairs ( $n=392$ pairwise merges). Color encodes pairwise merge quality (mean retention across the two original tasks). Marker shape encodes base model family. Right: full multi-task task arithmetic (all 8 tasks summed jointly), reporting mean accuracy across all 8 tasks. The pairwise ordering of Figure 5 is preserved at the multi-task scale: discriminative ( $0.52/0.37$ multi-task accuracy) and self-distillation ( $0.45/0.34$ ) retain the most, contrastive VL collapses ( $0.10/0.09$ ). Note that these multi-task accuracies differ from the pairwise mean retentions reported in Section 5.2 , since they aggregate across all 8 tasks at once rather than over pairs of tasks.

Figure 12: Per-regime distribution of downstream outcomes for the three settings. Left: Flickr30k I2T R@1 on

n=72

vision-language pairs. Center: COCO I2T R@1 on the same pairs. Right: mean retention on

n=392

task-arithmetic merges. Boxes show the interquartile range with the median line; whiskers extend to the non-outlier extremes; individual points are overlaid. Color encodes the four regimes: act now (high ease, high ceiling), budget-limited (low ease, high ceiling), ceiling-limited (high ease, low ceiling), and incompatible (low ease, low ceiling). The Kruskal-Wallis

H

statistic and

p

-value at the top of each panel test the null that the four regimes share the same distribution; the test rejects in all three settings.

Outcome	COMPASS $\rho$	COMPASS $\tau$	Ease $\rho$	Ceiling $\rho$	Baseline $\rho$
Flickr30k I2T R@1	0.91	0.73	0.87	0.77	0.63
Flickr30k T2I R@1	0.91	0.73	0.87	0.77	0.65
COCO I2T R@1	0.93	0.77	0.88	0.81	0.67
COCO T2I R@1	0.93	0.77	0.88	0.81	0.66
Merge retention	0.70	0.50	0.66	0.61	0.06

Appendix

Table 5: Rank correlations with downstream transfer performance. The final column uses global CKA for alignment and task-vector cosine for merging.

Predictor	Top-1	Regret	Top-5	Regret	NDCG@5	Top-10	Regret	NDCG@10	Top-20	Regret	NDCG@20	Budget to best
COMPASS	0.635	0.0%	0.578	5.9%	0.957	0.530	5.7%	0.952	0.493	2.1%	0.972	1/72
Ease	0.428	32.6%	0.521	15.2%	0.819	0.530	5.7%	0.893	0.486	3.6%	0.923	2/72
Ceiling	0.635	0.0%	0.563	8.3%	0.945	0.455	19.0%	0.864	0.439	12.8%	0.889	1/72
Global CKA	0.635	0.0%	0.589	4.2%	0.970	0.515	8.2%	0.940	0.440	12.7%	0.901	1/72
Paradigm distance	0.323	49.1%	0.204	66.8%	0.368	0.226	59.8%	0.406	0.245	51.3%	0.458	72/72
Total parameters	0.418	34.2%	0.352	42.6%	0.614	0.328	41.6%	0.609	0.316	37.3%	0.629	21/72

Appendix

Table 6: Selection performance on Flickr30k I2T retrieval (

n=72

candidate pairs). Regret is relative to the oracle top-

k

set.

Predictor	Top-1	Regret	Top-10	Regret	NDCG@10	Top-40	Regret	NDCG@40	Budget to best
COMPASS	0.947	8.4%	0.927	7.1%	0.920	0.890	9.1%	0.910	5/392
Ease	0.947	8.4%	0.935	6.4%	0.924	0.874	10.7%	0.896	5/392
Ceiling	0.968	6.4%	0.926	7.3%	0.930	0.828	15.5%	0.871	162/392
Task-vector cosine	0.963	6.8%	0.826	17.3%	0.841	0.708	27.7%	0.749	328/392
Random ordering	0.585	43.4%	0.585	41.4%	-	0.585	40.3%	-	196.5 expected

Appendix

Table 7: Selection performance for task-arithmetic merging (

n=392

candidate merges).

Pairs trained	1	2	5	10	20
COMPASS	1.00	1.00	1.00	1.00	1.00
Ease	0.67	1.00	1.00	1.00	1.00
Global CKA	1.00	1.00	1.00	1.00	1.00
Random order	0.53	0.66	0.79	0.88	0.95

Appendix

Table 8: Budget selection for Flickr30k I2T retrieval. Values are the best outcome found within the evaluation budget, normalized by the pool optimum.

Merges performed	1	5	10	20	50
COMPASS	0.92	1.00	1.00	1.00	1.00
Ease	0.92	1.00	1.00	1.00	1.00
Ceiling	0.94	0.94	0.95	0.96	0.97
Task-vector cosine	0.93	0.93	0.94	0.95	0.95
Random order	0.57	0.87	0.93	0.95	0.97

Appendix

Table 9: Budget selection for task-arithmetic merging.

Predictor	Alignment: Flickr I2T	Merging: retention
COMPASS	0.91	0.70
Pretraining paradigm distance	-0.60	-
Architecture self-consistency	0.37	-
Total parameters	0.09	-
Random-init $k$ -NN overlap	0.11	-
Mean solo accuracy of two checkpoints	-	0.43

Appendix

Table 10: Comparison with cheap metadata-based predictors.

Metric	Alignment: Flickr I2T	Merging: retention
Mutual $k$ -NN overlap	0.82	0.42
RSA, rank correlation between RDMs	0.38	0.22
Procrustes similarity	0.45	0.37
Linear CKA	0.34	0.25

Appendix

Table 11: Alternative representation-similarity metrics computed on identical inputs.

Evaluation	Alignment	Merging
COMPASS, nothing fitted	0.91	0.70
Regressor, random 5-fold CV	0.94	0.91
Regressor, leave-one-family/backbone out	0.60	0.81
Regressor, leave-one-paradigm out	0.46	-0.54

Appendix

Table 12: Comparison with learned metadata regressors. Unlike COMPASS, these models require previously observed downstream transfer outcomes as labels.

Fold type	Folds	Spearman on rest	Spearman inside held-out group
Leave-one-vision-family-out	6	0.85-0.93	0.81 median (0.56-0.87)
Leave-one-text-family-out	7	0.89-0.92	0.86 median (0.78-0.94)
Leave-one-vision-paradigm-out	5	0.85-0.93	0.80 median (0.56-0.87)
Leave-one-backbone-out (merging)	14	0.67-0.71	0.28 median (-0.13-0.75)
Leave-one-paradigm-out (merging)	5	0.60-0.72	0.27 median (0.20-0.66)

Appendix

Table 13: Generalization under held-out groups. The pooled references are

\rho=0.91

for Flickr30k alignment and

\rho=0.70

for merging.

Held-out vision family	Rest	Held out	Held-out text family	Rest	Held out
CLIP	0.91	0.78	CLIP text	0.92	0.78
SigLIP	0.89	0.87	SigLIP text	0.89	0.86
CLIP fine-tuned	0.92	0.83	OLMo	0.92	0.86
DINOv2	0.93	0.80	BLOOM	0.91	0.90
MAE	0.85	0.87	RoBERTa	0.90	0.94
Classification	0.92	0.56	LLaMA	0.91	0.80

Appendix

Table 14: Per-fold results for Flickr30k I2T alignment.

	CSA	Linear	MLP	LoRA
CSA	-	0.94	0.82	0.96
Linear	0.87	-	0.86	0.99
MLP	0.76	0.84	-	0.85
LoRA	0.91	0.98	0.85	-

Appendix

Table 15: Pair-ranking agreement across alignment procedures. Values below the diagonal are Spearman correlations for Flickr30k I2T and values above the diagonal are for T2I.

Predictor	CSA	Linear	MLP	LoRA
COMPASS	0.87	0.90	0.73	0.92
Ease	0.78	0.78	0.56	0.78
Ceiling	0.86	0.97	0.76	0.95
Global CKA	0.71	0.79	0.60	0.78

Appendix

Table 16: Spearman correlation with Flickr30k I2T R@1 under different alignment procedures.

Procedure	Scaling	Cost for full $19\times 10$ grid
COMPASS , including ceiling	One probe pass per model	29 passes; $\sim$ 21 min; 12 GB cached features
Exhaustive downstream evaluation	One alignment run per pair	190 runs; $\sim$ 86–102 h; 729 GB cached features

Appendix

Table 17: Cost of COMPASS scoring compared with exhaustive downstream alignment. The wall-clock comparison was benchmarked on a single RTX 2080 Ti.

Ceiling estimate	Alignment: Flickr30k I2T	Merging: retention
Original estimate	0.909	0.696
One shared value per paradigm	0.903	0.699
Gaussian noise at 25% of ceiling spread	0.901	0.695
Gaussian noise at 50% of ceiling spread	0.883	0.695
Gaussian noise at 100% of ceiling spread	0.830	0.691
Ceiling values shuffled across paradigms	0.675	0.654

Appendix

Table 18: Sensitivity to ceiling granularity and estimation noise. Values are Spearman correlations with downstream performance.

Vision family held fixed	Pairs	Ease vs. R@1	COMPASS vs. R@1	Within-family ceiling spread
CLIP only	12	0.67	0.78	0.013
SigLIP only	12	0.85	0.87	0.023
CLIP fine-tuned only	10	0.83	0.83	0.017
MAE only	12	0.87	0.87	0.016
DINOv2 only	13	0.77	0.80	0.024
Classification only	13	0.43	0.56	0.009

Appendix

Table 19: Predictive power within individual vision-model families on Flickr30k I2T retrieval.

Encoder	Ceiling before ft	Ceiling after ft	Mean R@1 before	Mean R@1 after
CLIP ViT-B/16	0.138	0.109	0.396	0.319
CLIP ViT-L/14	0.147	0.117	0.579	0.548
CLIP ViT-H/14	0.151	0.127	0.628	0.610

Appendix

Table 20: Change in cross-modal ceiling and mean Flickr30k I2T R@1 after ImageNet-12K classification fine-tuning.

Vision encoder	Ceiling	10 ep	30 ep	100 ep	300 ep
MAE ViT-B	0.055	0.142	0.183	0.216	0.215
AugReg ViT-B	0.101	0.269	0.356	0.411	0.420
DINOv2 ViT-B	0.106	0.362	0.480	0.477	0.471
CLIP ViT-B ft-IN12k	0.109	0.288	0.370	0.454	0.459
SigLIP ViT-B	0.130	0.359	0.472	0.544	0.559
CLIP ViT-B LAION-2B	0.138	0.424	0.528	0.587	0.607

Appendix

Table 21: Flickr30k I2T R@1 as the alignment-training budget increases, with OLMo-1B held fixed.

Language Model Merging

Language model merging combines capabilities from multiple models or adapters by composing their weights, updates, or outputs.

See all 106 papers

Apr 14, 2026

Task Alignment: A Simple Proxy for Practical Model Merging Across Diverse Vision Tasks

Efficiently merging several models fine-tuned for different tasks, but stemming from the same pretrained base model, is of great practical interest. Despite extensive prior work, most evaluations of model merging in computer vision are restricted to image classification using CLIP, where different classification datasets define different tasks. In this work, our goal is to make model merging more practical and show its relevance on challenging scenarios beyond this specific setting. In most vision scenarios, different tasks rely on trainable and usually heterogeneous decoders. Differently from previous studies with frozen decoders, where merged models can be evaluated right away, the non-trivial cost of decoder training renders hyperparameter selection based on downstream performance impractical. To address this, we introduce the task alignment proxy, and show how it can be used to speed up hyperparameter selection by orders of magnitude while retaining performance. Equipped with the task alignment proxy, we extend the applicability of model merging to multi-task vision models beyond CLIP-based classification. Project page: https://europe.naverlabs.com/task-alignment

Pau de Jorge, César Roberto de Souza, Björn Michele +5

NAVER LABS Europe

Model Merging

Ask PDF

Dec 18, 2025

AlignMerge - Alignment-Preserving Large Language Model Merging via Fisher-Guided Geometric Constraints

Merging large language models (LLMs) is a practical way to compose capabilities from multiple fine-tuned checkpoints without retraining. Yet standard schemes (linear weight soups, task vectors, and Fisher-weighted averaging) can preserve loss while quietly destroying alignment. We argue that merging is not a numerical trick but a geometry-constrained operation around an already-aligned anchor: fusion must be steered to respect safety geometry, not validated post hoc. We introduce AlignMerge, a geometry-aware merging framework that makes alignment an explicit invariant. In a local Fisher chart around an instruction-tuned base, we estimate an alignment subspace with projector P_A and optimize: L_AlignMerge = L_geo + lambda_align * L_align + lambda_bud * L_bud, where L_geo keeps the merge close to its experts in Fisher-Rao geometry, L_align penalizes motion along alignment-sensitive directions, and L_bud enforces a soft alignment budget. As the alignment functional we use the decoding-invariant Alignment Quality Index (AQI), a latent-space criterion that captures how cleanly aligned and misaligned behaviors separate in representation space. Across five model families (LLaMA-3 8B, Mistral 7B, Qwen 2, Phi-3.5, Gemma 2), merging safety anchors with task experts, AlignMerge improves alignment metrics (AQI, toxicity, LLM-judge alignment) while matching or exceeding the best expert on instruction-following, reasoning, and helpfulness. It also exhibits smaller alignment-subspace drift and fewer budget violations than Fisher soups, TIES, SafeMerge, and MergeAlign. These results make alignment-preserving merging a first-class design goal and suggest a path to geometry-aware composition of future foundation models.

Aniruddha Roy, Jyoti Patel, Aman Chadha +2

HCL · Evalueserve · Apple (USA) +2

Language Model Merging LLM Alignment LLM Safety Alignment Latent Space Alignment

Ask PDF

Oct 20, 2025

Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Merging

The "alignment tax" of post-training is typically framed as a drop in task accuracy. We show it also involves a severe loss of calibration, making models overconfident, less reliable, and model outputs less diverse. We show that this trade-off can be navigated effectively via a simple post-hoc intervention: interpolating between a model's weights before and after alignment. Crucially, this is not a strict trade-off. We find that the process consistently reveals Pareto-optimal interpolations - models that improve accuracy beyond both parents while substantially recovering the calibration lost during alignment. Our work demonstrates that simple model merging provides a computationally efficient method for mitigating the full scope of the alignment tax, yielding models that are more capable and more reliable.

Tiancheng Hu, Benjamin Minixhofer, Nigel Collier

University of Cambridge

LLM Alignment Model Calibration Multi-Objective Language Model Alignment Language Model Merging

Ask PDF

Abstract

Language Model Merging

Explore similar work

Task Alignment: A Simple Proxy for Practical Model Merging Across Diverse Vision Tasks

AlignMerge - Alignment-Preserving Large Language Model Merging via Fisher-Guided Geometric Constraints

Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Merging