Platonic Task Arithmetic

Junghwan Park; Woojin Cho

cs.LGOct 1, 2026

Platonic Task Arithmetic

Authors: Junghwan Park, Woojin Cho

Organizations: TelePIX

Abstract

Models specialized for the same task converge to similar behavior, yet the parameter updates that produce it share no common coordinate system, so weight-space task arithmetic stays confined to a single model and cannot cross architectures without a structural correspondence. Drawing on Plato's allegory of the cave, we hypothesize that these model-specific updates are shadows of one shared, model-agnostic object, which we call the platonic task vector. To make it operational for models that pair an image or audio encoder with a text encoder, we introduce Universal Task Descriptors: matrices whose shape is independent of architecture and embedding dimension, which record a task's functional effect and support addition and negation as matrix operations. Transferring a descriptor into a target means editing the target until it reproduces the descriptor on the task's unlabeled probe images and class-name prompts, requiring no per-image labels. We realize this edit in two ways. First, the descriptor factorizes into a shift field on image embeddings, so a single least-squares solve yields a linear operator that folds into the target's last layer as a weight edit; by linearity, a bank of such operators admits any composition at any strength as a signed sum. Second, a low-rank adapter trained on the same objective reaches every layer and fits compositions jointly, at the cost of one optimization per edit. Heterogeneous models share this object only partially, with a model-specific residual comparable in norm to the shared component, yet cross-model transfer still retains 74-80 percent of the gain of the target's own descriptors. Experiments across six model families, eight classification tasks, and an audio-text setting show that task knowledge transfers and composes across heterogeneous models under both realizations.

Figures & tables

Source and target are the same model ( $112$ cells)
Method	Task A $\uparrow$	Task B $\uparrow$	CIFAR-100	ImageNet
Zero-shot	$58.70$	$57.96$	$76.95$	$71.50$
Weight-space arithmetic	$78.50$	$79.05$	$77.85$	$71.83$
Ours, closed form	$83.07$	$76.18$	$76.23$	$70.13$
Ours, adapter	$86.85$	$85.87$	$72.40$	$71.49$
Different models within one architecture ( $168$ cells)

Table 1 : Addition in percent, each block read against its own zero-shot row. Task A and Task B are the two composed tasks, the closed form acts at

\alpha=0.5

, the adapter at full strength and the weight-space routes at

0.5

on the mean of the two updates. Random-UTD, Oracle and the last two rows cover

168

cells.

Figure 2 : Accuracy against negation strength.

Source and target are the same model ( $32$ cells)
Method	Removed task $\downarrow$	CIFAR-100	ImageNet
Zero-shot	$58.33$	$76.95$	$71.50$
Weight-space negation	$28.23$	$63.51$	$69.95$
Ours, closed form	$22.07$	$74.28$	$67.30$
Ours, adapter	$26.43$	$75.64$	$70.21$
Different models within one architecture ( $48$ cells)

Table 2 : Negation at

\lambda=1

in percent, grouped as in Table 1 . Theseus and GradFix read ImageNet on one image per class, where zero-shot is

70.70

Assignment	Route	RESISC45	UCMerced	AID	Mean
None	Zero-shot	$68.35$	$74.29$	$68.58$	$70.41$
$k_{1}=k_{2}$	Closed form	$71.23$	$84.09$	$72.93$	$76.08$
	Adapter	$71.46$	$85.60$	$72.83$	$76.63$
$k_{1}\neq k_{2}$	Closed form	$71.21$	$84.56$	$73.63$	$76.47$
	Adapter	$71.42$	$85.40$	$73.12$	$76.64$
$k_{1}$ alone	Closed form	$70.69$	$83.33$	$72.51$	$75.51$

Table 3 : Composition evaluated on a dataset that enters no fit, in percent, over three targets and by the models

k_{1}

and

k_{2}

supplying the two summed terms, with the closed form at

\alpha=0.5

and the adapter at full strength. Both terms come from one other family when

k_{1}=k_{2}

, from the two other families when

k_{1}\neq k_{2}

and from the target itself in the within-model rows.

Source and target are the same model ( $8$ cells)
Method	Removed $\downarrow$	Retained	UrbanSound8K
Zero-shot	$71.05$	$71.05$	$76.10$
Ours, closed form	$55.98$	$70.73$	$74.25$
Ours, adapter	$57.69$	$70.76$	$74.10$
Source and target are different models ( $24$ cells)
Ours, closed form	$56.68$	$70.45$	$74.08$

Table 4 : Audio–text negation at

\lambda=1

for both realizations, in percent, where each cell removes one of ESC-50 and GTZAN and retains the other. The adapter’s rows are read against its own runs’ zero-shot,

71.02

71.02

and

75.85

(Appendix P ).

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description
Models, encoders and parameters
$\mathcal{M}_{k}$	The $k$ -th vision–language model, $k=1,\dots,K$ .
$\mathcal{M}_{\mathrm{tgt}}$	Target model receiving the edit.
$\ell$ , $\sigma$	Task-term index and the source assignment $\{1,\dots,J\}\to\{1,\dots,K\}$ .
$f^{(k)},g^{(k)}$	Vision and text encoders of $\mathcal{M}_{k}$ , with $g^{(k)}$ frozen.
$f^{(k)}_{\mathrm{pre}},f^{(k)}_{\mathrm{ft}}$	Pre-trained and fine-tuned vision encoders of $\mathcal{M}_{k}$ .

Appendix

Table 5: Notation for models, tasks, descriptors and the two realizations.

Target	Cosine	Relative difference, double	Relative difference, single
CLIP-L	$1.0000000000$	$2.6\times 10^{-14}$	$1.6\times 10^{-7}$
OpenCLIP-L	$1.0000000000$	$2.6\times 10^{-14}$	$1.9\times 10^{-7}$
MetaCLIP-L	$1.0000000000$	$3.5\times 10^{-14}$	$2.4\times 10^{-7}$
EVA-CLIP-L	$1.0000000000$	$4.8\times 10^{-14}$	$3.7\times 10^{-7}$

Appendix

Table 6: Numerical check of Proposition 1 on the grids the paper solves on. For each target and each of the eight tasks, three source families give descriptors of that task on that task’s own probes and prompts, and the operator of

U_{1}-1.5\,U_{2}+U_{3}

is compared against the same combination of the three terms’ operators. Each row reports the worst of that target’s eight tasks, namely the smallest cosine between the two operators over both precisions and the largest relative difference between them. The last two columns separate the algebra from the arithmetic, giving the relative difference when the solve is carried in double precision and when it is carried at the single precision a stored operator is kept in.

Target	Guaranteed	Changed	Bound	Observed	Single
CLIP-L	48.1	9.5	23.78	1.30	71.83
OpenCLIP-L	49.5	9.0	16.82	-0.29	66.30
MetaCLIP-L	46.5	9.1	21.52	1.21	68.03
EVA-CLIP-L	47.2	11.8	28.60	3.58	75.85
Pooled	47.8	9.9	22.68	1.45	70.50

Appendix

Table 7: Corollary 1 on the cross-model addition cells of the four targets that absorb the operator, at

\alpha=0.5

, in percent of evaluation images, pooled over both tasks of every cell and weighted by split size. Guaranteed is the fraction of images with

m_{\ell}(x)>\varepsilon_{\ell}(x)

, whose prediction the corollary fixes, and changed is the fraction whose prediction the sum actually moved. Bound is the corollary’s

\Pr[0<m_{\ell}(x)\leq\varepsilon_{\ell}(x)]

and observed is the accuracy the single-term model loses to the sum, which the bound dominates in every one of the

4{,}480

records. Single is the single-term model’s accuracy. Probes come from training splits, so no evaluation image is a probe.

Agreement measured on	Mean agreement	Rank correlation	Best source identified
Descriptors	$0.59$	$-0.26$	$10$ of $32$
Operators	$0.39$	$-0.62$	$\phantom{0}9$ of $32$
Shift fields on the probes	$0.43$	$-0.59$	$\phantom{0}9$ of $32$

Appendix

Table 8: Where a cross-model residual is measured decides how well it predicts what transfer costs. Each row compares a source’s quantity with that of the target’s own fine-tune on the same task and grid over

160

combinations, and correlates it with the fraction of the attainable gain that the source’s operator forfeits at

\alpha=0.5

over the

150

combinations on which that fraction is defined. The last column counts, over the

32

groups of five candidate sources sharing a target and a task, how often the source that the agreement ranks first is also the most accurate, where chance gives

6.4

; the three do not separate on that question.

	EuroSAT	DTD	Cars	GTSRB	MNIST	RESISC45	SVHN	SUN397
Descriptor agreement	$0.79$	$0.41$	$0.81$	$0.89$	$0.51$	$0.87$	$0.41$	$0.06$
Operator agreement	$0.71$	$0.25$	$0.07$	$0.48$	$0.56$	$0.37$	$0.63$	$0.07$
Shift-field agreement	$0.73$	$0.37$	$0.13$	$0.47$	$0.57$	$0.42$	$0.63$	$0.11$
Gain forfeited	$0.11$	$0.33$	$0.61$	$0.14$	$0.13$	$0.24$	$0.25$	$0.85$

Appendix

Table 9: Per-task view of the same

160

combinations, each agreement averaged over the four targets and their five sources. Agreement is a cosine and the gain is that of the target’s own operator over zero-shot. The last row gives the fraction of that gain which transport forfeits, averaged over those of the

150

combinations of Table 8 that fall on the task. Cars is the task whose descriptor agreement and transport quality disagree most sharply, and SUN397 is the one on which every measure agrees that transport fails.

Task	Prompts	$2$	$4$	$8$	$16$	$32$	$64$	$128$	$256$	Selected $r$
		Rank cap $r$
EuroSAT	$10$	$0.562$	$0.490$	$0.445$	—	—	—	—	—	$8$
DTD	$47$	$0.694$	$0.657$	$0.637$	$0.629$	$0.629$	—	—	—	$32$
Cars	$196$	$0.397$	$0.375$	$0.355$	$0.341$	$0.336$	$0.334$	$0.334$	—	$128$
GTSRB	$43$	$0.345$	$0.322$	$0.305$	$0.295$	$0.293$	—	—	—	$32$
MNIST	$10$	$0.725$	$0.615$	$0.552$	—	—	—	—	—	$8$

Appendix

Table 10: Held-out descriptor reconstruction error of the closed form against the rank cap

r

, pooled over the four targets that absorb the operator and their five cross-model sources, with the cap each task selects under that label-free criterion. A dash marks a cap the closed form cannot reach on that task, since a descriptor read against a task’s own landmarks constrains the operator on at most as many directions as the task has prompts. Each entry averages over the ridge settings, whereas the criterion selects the cap and the ridge jointly, and on every task it takes the largest cap the sweep offers below the ceiling. On seven of the eight tasks the averaged error falls with the cap; on SVHN it rises instead, which is also the task on which the closed form is least accurate.

Family	Identifier	$d^{(k)}$	Adapted modules
CLIP	openai/clip-vit-large-patch14	$768$	q_proj , v_proj
OpenCLIP	laion/CLIP-ViT-L-14-laion2B-s32B-b82K	$768$	q_proj , v_proj
MetaCLIP	facebook/metaclip-l14-400m	$768$	q_proj , v_proj
EVA-CLIP	EVA02-L-14 (via open_clip )	$768$	attn.{q,v}_proj
SigLIP	google/siglip-large-patch16-256	$1024$	q_proj , v_proj
SigLIP2	google/siglip2-large-patch16-256	$1024$	q_proj , v_proj

Appendix

Table 11: Models used in this work, with

d^{(k)}

the embedding dimension and the adapted modules those fine-tuned to produce a descriptor. All vision sources are at Large scale. The two SigLIP families pool through an attention head whose output is a residual sum, so they receive an edit through that head’s last affine rather than through Equation ( 9 ).

$\alpha$	Method	$\Delta$ Task A	$\Delta$ Task B	$\Delta$ CIFAR-100	$\Delta$ ImageNet
$0.5$	Ours	$+18.11$	$+11.63$	$\phantom{0}-1.81$	$\phantom{0}-3.25$
	Single-A	$+19.58$	$-1.12$	$\phantom{0}-0.77$	$\phantom{0}-1.32$
	Single-B	$-1.53$	$+12.87$	$\phantom{0}-0.94$	$\phantom{0}-1.83$
$1.0$	Ours	$+18.44$	$+10.45$	$\phantom{0}-8.18$	$-12.74$
	Single-A	$+21.79$	$-4.39$	$\phantom{0}-3.64$	$\phantom{0}-5.47$
	Single-B	$-5.46$	$+13.77$	$\phantom{0}-4.55$	$\phantom{0}-7.54$

Appendix

Table 12: Addition against strength, over all

3{,}360

cross-model cells of the six targets, as change from zero-shot in percentage points. Single-source rows are paired to the same cells, so the difference between Ours and Single-A on task A is the per-cell cost of composing.

Target	$\Delta$ Composed tasks	$\Delta$ CIFAR-100	$\Delta$ ImageNet	Zero-shot CIFAR-100
CLIP-L	$+17.07\pm 9.54$	$-1.88$	$-4.07$	$74.2$
OpenCLIP-L	$+18.17\pm 10.86$	$-0.62$	$-1.42$	$74.7$
MetaCLIP-L	$+16.24\pm 9.68$	$-1.83$	$-2.74$	$71.0$
EVA-CLIP-L	$+13.04\pm 8.06$	$-2.46$	$-7.43$	$87.9$
SigLIP-L	$+11.75\pm 8.46$	$-1.62$	$-1.48$	$72.0$
SigLIP2-L	$+12.93\pm 7.35$	$-2.44$	$-2.35$	$75.1$

Appendix

Table 13: Addition at

\alpha=0.5

by target,

560

cells each, as change from zero-shot. The four targets that absorb the operator acquire the composed tasks by

13.04

18.17

points and differ far more in what they give up, EVA-CLIP paying roughly five times what OpenCLIP pays on ImageNet. The two that receive an update to a pooling head instead gain a little less and sit inside the range the others occupy on both controls, so the absorption route is not what decides the trade. What a target gains and what it pays are separate axes.

Task	$\Delta$ Ours	$\Delta$ Oracle	Fraction retained
EuroSAT	$+33.55\pm 4.23$	$+38.76$	$0.87$
DTD	$+6.37\pm 2.47$	$+10.20$	$0.62$
Cars	$-0.30\pm 2.42$	$+1.90$	—
GTSRB	$+30.24\pm 5.03$	$+35.81$	$0.84$
MNIST	$+28.45\pm 13.35$	$+33.75$	$0.84$
RESISC45	$+12.85\pm 2.40$	$+17.36$	$0.74$

Appendix

Table 14: Addition at

\alpha=0.5

by composed task over the

3{,}360

cross-model cells of the six targets, as change from zero-shot in percentage points, with the spread over cells. Oracle averages the

42

cells per task in which the target supplies its own descriptors. The retained fraction is the ratio of the two gains and is omitted where Oracle gains under three points, since the ratio is then dominated by noise.

Figure 3 : Addition at

\alpha=0.5

on the OpenCLIP-L target, one marker per cell, with each axis normalized by the per-task fine-tuning ceiling of Table 22 . The composed operator moves both coordinates at once.

CIFAR-100
Task	CLIP-L	OpenCLIP-L	MetaCLIP-L	EVA-CLIP-L	Mean
EuroSAT	$-0.60$	$-0.00$	$-9.70$	$-4.40$	$-3.68$
DTD	$-0.60$	$-0.10$	$-0.80$	$-2.80$	$-1.08$
Cars	$+0.90$	$+0.30$	$+1.80$	$+0.40$	$+0.85$
GTSRB	$-27.90$	$-29.70$	$-23.80$	$-41.90$	$-30.83$
MNIST	$-2.80$	$-3.80$	$-3.80$	$-4.70$	$-3.78$

Appendix

Table 15: Change in the two control benchmarks when a target is fine-tuned on one task, in percentage points against that target’s own zero-shot accuracy. Each entry applies the target’s own source for the row task to the target itself, which is the single-task adaptation descriptor transfer approximates. The upper block is CIFAR-100 and the lower block ImageNet.

Figure 4 : Mean accuracy over the

J

composed tasks, normalized by the per-task fine-tuning ceiling, as the operators of

J

tasks accumulate on one target at

\alpha=0.5

, each measured on the source family the text assigns to its task. The band spans three orderings, and every prefix is the sum of the same stored operators, one per task.

	Cross-model source			Same-model source
$\lambda$	Removed task $\downarrow$	CIFAR-100 $\uparrow$	ImageNet $\uparrow$	Removed task $\downarrow$	CIFAR-100 $\uparrow$	ImageNet $\uparrow$
$0.25$	$-12.14\pm 10.02$	$\phantom{0}-0.11$	$\phantom{0}-0.50$	$-13.98\pm 10.10$	$\phantom{0}-0.03$	$\phantom{0}-0.28$
$0.5$	$-22.13\pm 15.63$	$\phantom{0}-0.94$	$\phantom{0}-2.23$	$-24.75\pm 14.79$	$\phantom{0}-0.39$	$\phantom{0}-1.08$
$0.75$	$-28.52\pm 17.65$	$\phantom{0}-2.69$	$\phantom{0}-4.99$	$-31.74\pm 16.86$	$\phantom{0}-1.37$	$\phantom{0}-2.37$
$1.0$	$-32.93\pm 18.45$	$\phantom{0}-5.07$	$\phantom{0}-8.41$	$-36.26\pm 17.39$	$\phantom{0}-2.67$	$\phantom{0}-4.21$
$1.25$	$-36.27\pm 18.64$	$\phantom{0}-7.90$	$-12.26$	$-39.60\pm 17.14$	$\phantom{0}-4.48$	$\phantom{0}-6.69$

Appendix

Table 16: Negation strength sweep, as change from zero-shot in percentage points, over the

160

cross-model cells and the

32

same-model cells that share the same eight tasks and four targets. The zero-shot references are

58.33

on the removed task,

76.95

on CIFAR-100 and

71.50

on ImageNet. Beyond

\lambda=1

the controls degrade about as fast as the removed task or faster, which is the operating boundary, and the sweep is continued past the useful range to show what the last points of erasure cost.

Addition at $\alpha=0.5$	EuroSAT	DTD	CIFAR-100	ImageNet
Zero-shot	$50.31$	$56.29$	$73.30$	$70.22$
Two sources, one per task ( $6$ cells)	$85.79\pm 0.23$	$65.02\pm 0.13$	$72.53\pm 0.02$	$69.23\pm 0.01$
the study’s own sources	$85.51$	$64.92$	$72.62$	$69.31$
One source, both tasks ( $6$ cells)	$85.57\pm 0.23$	$64.96\pm 0.31$	$72.53\pm 0.15$	$69.36\pm 0.08$
Target’s own sources ( $3$ cells)	$90.61\pm 0.16$	$69.06\pm 0.26$	$72.32\pm 0.12$	$69.74\pm 0.06$
the study’s own sources	$90.30$	$68.74$	$72.37$	$69.69$

Appendix

Table 17: Repeating the fine-tuning of the EuroSAT and DTD sources at three random seeds moves a composed or negated cell by a few tenths of a point. Entries are percentages, averaged over cells first and then over the three seeds, and the interval is the standard deviation over seeds. Rows marked as the study’s own sources repeat the measurement with the checkpoints every other table of the paper reads, in the same frame, and are not part of that mean. Targets are CLIP-L, OpenCLIP-L and MetaCLIP-L, each composing or negating with sources drawn from the other two families.

Assignment	Method	RESISC45	UCMerced	AID	Mean	CIFAR-100
None	Zero-shot	$68.35$	$74.29$	$68.58$	$70.41$	$73.48$
Within-model	Composition	$72.86$	$86.83$	$74.48$	$78.06$	$72.26$
Within-model	Single $\operatorname{RS}_{1}$	$72.20$	$85.87$	$73.47$	$77.18$	$72.76$
Within-model	Oracle (target’s own source)	$87.43$	$91.59$	$91.60$	$90.21$	$72.93$
Cross-model, CLIP	Composition	$71.73$	$84.76$	$73.93$	$76.81$	$71.20$
Cross-model, MetaCLIP	Composition	$71.19$	$85.36$	$72.28$	$76.27$	$74.45$

Appendix

Table 18: Composition evaluated on a dataset that no source term describes, at

\alpha=0.5

and pooled over the three targets. Each cross-model row draws both terms from the named family, on the two targets for which that family is not the target; the distinct row draws its two terms from the two families other than the target, in both orders. Single

\operatorname{RS}_{1}

transfers the first of the two terms alone, and CIFAR-100 is scored here on its full test split, which is why its zero-shot differs from that of Table 1 . Cross-model assignments track the within-model reference closely, which is the claim the experiment supports.

Weight-space task arithmetic
	Arbitrary halves, photographs and sketches		Semantic split, photographs and paintings
Method	Held-out task $\uparrow$	CIFAR-100	Held-out task $\uparrow$	CIFAR-100
Zero-shot	$89.99$	$76.63$	$87.88$	$76.63$
First term alone	$90.63$	$77.49$	$88.47$	$77.34$
Third term alone	$90.80$	$77.49$	$88.70$	$77.34$
Analogy, $\lambda=1$	$90.90$	$76.45$	$88.61$	$76.45$

Appendix

Table 19: The analogy operation on two products of two domains and two subtasks, in percent, with each of the four tasks held out in turn on each of the six model families, giving 24 cells per column. The first product follows the construction of Ilharco et al. [2022] , crossing an arbitrary halving of one class list with two rendering styles, and the second replaces the halving by a semantic split into natural and manufactured classes and narrows the style gap, both of which should favor the operation. Every route is applied at the strength Section 5.2 fixes. The first and third terms are the two positive terms of the expression taken alone, and

\lambda

scales the subtracted term, so

\lambda=0

is the sum of the two positive terms. CIFAR-100 enters no term and no fit, and its column is identical for the first term, the third term and the oracle because those three rows apply the same four single-task edits and differ only in which held-out task each is paired with.

Method	Structural requirement	Target-side signal	Operations	Different families
Weight-space task arithmetic [ Ilharco et al., 2022 ]	Identical parameter space	None	Addition, negation, analogy	n/a
GradFix [ Rinaldi et al., 2026a ]	Matching parameter coordinates	Labeled examples and their gradients	Addition, negation	Partly
Theseus [ Rinaldi et al., 2026b ]	Corresponding submodules, width mismatch allowed	Calibration activations	Addition	Partly
Ours, closed form	None beyond a cross-modal affinity and an affine last layer writing into the embedding	Unlabeled probe images and prompts	Addition, negation	Yes
Ours, adapter	None beyond a cross-modal affinity	Unlabeled probe images and prompts, one optimization per edit	Addition, negation	Yes
Gradient ascent [ Thudi et al., 2022 ]	None, since it trains the target directly	Labels for the removed task	Negation	n/a

Appendix

Table 20: Requirements and scope of the routes to composing task knowledge. Structural requirement is what must correspond between the source model and the target model, target-side signal is what the method consumes from the target beyond its weights, and operations lists which of addition, negation and analogy the route has been formulated for. The last column marks whether the route stays defined when source and target belong to different families. The abbreviation n/a means the method is undefined for that combination.

Against weight-space negation ( $32$ cells)
Method	Strength	$\Delta$ Removed task $\downarrow$	$\Delta$ CIFAR-100 $\uparrow$	$\Delta$ ImageNet $\uparrow$
Weight-space negation	$\lambda=0.5$	$-22.01\pm 13.04$	$-4.16$	$-0.59$
Weight-space negation	$\lambda=1.0$	$-30.10\pm 12.25$	$-13.44$	$-1.56$
Weight-space negation	$\lambda=1.5$	$-35.69\pm 11.91$	$-22.68$	$-2.88$
Ours, closed form, same model	$\lambda=0.5$	$-24.75\pm 14.79$	$-0.39$	$-1.08$
Ours, closed form, same model	$\lambda=1.0$	$-36.26\pm 17.39$	$-2.67$	$-4.21$

Appendix

Table 21: Negation against every comparison, as change from the target’s zero-shot in percentage points, with each route read against the zero-shot of its own control draw. A baseline is defined only on part of this population, so each block is restricted to the cells its baseline was measured on. In the transport block the cross-model row is restricted to the same source assignments as well; in the other two the baseline has no source axis, and the cross-model row there averages over every source family available for the cell. Weight-space negation and the same-model closed form require source and target to coincide, gradient ascent requires labels for the removed task and runs at a fixed budget rather than a strength, so a dash stands in its strength column, and the transport methods require corresponding parameters, so only the cross-model row is defined between arbitrary families.

Target	EuroSAT	DTD	Cars	GTSRB	MNIST	RESISC45	SVHN	SUN397
CLIP-L	$98.3$	$75.1$	$85.4$	$95.1$	$98.9$	$94.9$	$96.2$	$76.4$
OpenCLIP-L	$98.2$	$77.1$	$93.9$	$96.0$	$99.1$	$94.9$	$96.6$	$77.1$
MetaCLIP-L	$98.7$	$79.4$	$89.5$	$95.6$	$99.0$	$95.3$	$96.6$	$77.0$
EVA-CLIP-L	$98.0$	$78.6$	$92.7$	$95.2$	$99.2$	$95.0$	$97.2$	$79.7$

Appendix

Table 22: Per-task fine-tuning ceilings, meaning the target’s own rank-4 source applied to itself, in percent. These are the denominators for the normalized axes in Figure 4 .

Addition	EuroSAT $\uparrow$	DTD $\uparrow$	CIFAR-100	ImageNet
Zero-shot	$50.31$	$56.29$	$73.30$	$70.22$
Ours, closed form	$85.51$	$64.92$	$72.62$	$69.31$
Ours, adapter	$92.45$	$64.41$	$72.83$	$70.04$
Oracle, closed form	$90.30$	$68.74$	$72.37$	$69.69$
Affinity distillation	$95.96$	$71.10$	$66.55$	$67.58$
Posterior distillation	$96.36$	$77.62$	$68.03$	$67.04$

Appendix

Table 23: Distillation and supervised tuning against both realizations, in percent, on the six EuroSAT and DTD cells of the architecture block for addition and the twelve cells removing one of the two tasks for negation. Every trained row uses the adapter realization’s recipe and differs from it only in the loss. The closed form is applied at

\alpha=0.5

and every trained route at full strength; Oracle uses the target’s own descriptors.

Target	Method	Task A	Task B	CIFAR-100	ImageNet
SigLIP-L	Zero-shot	$64.36$	$67.42$	$72.00$	$76.47$
	Random-UTD	$65.35$	–	$70.76$	$75.75$
	Head edit	$80.79$	$74.49$	$70.38$	$74.99$
	Embedding operator	$81.28$	$75.67$	$70.80$	$75.25$
	Oracle	$84.53$	$77.80$	$71.44$	$76.24$
SigLIP2-L	Zero-shot	$66.70$	$66.62$	$75.10$	$71.52$

Appendix

Table 24: The two residual-head families as targets, in percent, resolving the cells that Table 1 and Table 2 pool with the other four. Addition is averaged over the

560

cross-model cells of each target, in which neither source is the target and the two sources differ, and negation over

40

such cells. The head edit is the deployable route and is the one those tables report; the embedding operator is the same solve applied outside the model, which no weight edit of this architecture realizes and which is given only as the reference the head edit is read against. Random-UTD here is a structureless update for one task scored on that task alone, and is therefore not the pair-level control of Table 1 .

		Addition at $\alpha=0.5$			Negation at $\lambda=0.5$		Negation at $\lambda=1$
Target	$d$	Composed $\uparrow$	CIFAR-100	ImageNet	Removed $\downarrow$	CIFAR-100	Removed $\downarrow$	CIFAR-100
CLIP-B	$512$	$+15.16$	$-6.99$	$-13.80$	$-25.89$	$-3.74$	$-33.93$	$-13.64$
MetaCLIP-B	$512$	$+16.83$	$-4.61$	$-9.38$	$-22.30$	$-2.75$	$-30.21$	$-9.54$
OpenCLIP-H	$1024$	$+15.34$	$-1.27$	$-0.94$	$-20.80$	$-0.63$	$-33.36$	$-1.64$
MetaCLIP-H	$1024$	$+11.38$	$-1.10$	$-2.00$	$-30.19$	$-0.41$	$-43.31$	$-2.45$
Large targets, reference	$768$	$+16.13$	$-1.70$	$-3.92$	$-22.13$	$-0.94$	$-32.93$	$-5.07$

Appendix

Table 25: Targets at another scale, in percentage points of change from the target’s own zero-shot. Addition is at

\alpha=0.5

over the

840

cells each target admits, and negation at

\lambda=0.5

and

1

over

48

. Every operator behind this table is solved on its own task’s grid and stored, as everywhere else in the paper, so the Large reference row is the cross-model cells of Table 1 and Table 2 taken over the four Large targets that absorb the operator, which is the only regime available at these scales. Every cell here is cross-model and cross-scale at once, and no oracle or same-model row is possible, since both would need a source fine-tune on the target itself.

Method	ESC-50 $\uparrow$	GTZAN $\uparrow$	UrbanSound8K $\uparrow$
Zero-shot	$89.73$	$52.90$	$76.05$
Random-UTD	$78.35$	$39.85$	$70.45$
Ours, closed form	$91.39$	$55.23$	$76.54$
Oracle	$92.45$	$64.70$	$77.60$
Zero-shot, adapter runs	$90.35$	$51.70$	$75.85$
Ours, adapter	$92.84$	$52.53$	$76.69$

Appendix

Table 26: Audio–text addition of ESC-50 and GTZAN in percent, pooled over the

24

cells in which the two sources and the target are distinct, with the closed form at

\alpha=0.5

, the strength the vision experiments fix, and the adapter at full strength. The reference rows do not depend on a source assignment and are averaged over the four checkpoints. The adapter is scored in runs of its own, and since one checkpoint is not bit-reproducible across runs, its row is read against the zero-shot of those runs.

Source (ESC-50)	Source (GTZAN)	Target	Ours
unfused	large	large-ms	$93.2$ / $71.0$ / $81.4$
unfused	large-ms	large	$95.2$ / $47.6$ / $82.2$
large	large-ms	unfused	$92.0$ / $57.0$ / $70.8$
large	unfused	large-ms	$92.9$ / $78.2$ / $82.2$
large-ms	unfused	large	$94.8$ / $46.2$ / $83.6$
large-ms	large	unfused	$91.9$ / $53.0$ / $69.6$

Appendix

Table 27: Audio addition, per cell, at

\alpha=0.5

. Entries are ESC-50 / GTZAN / UrbanSound8K accuracy in percent. Every cell composes two source models onto a third. Read against each target’s own zero-shot, listed below the table, ESC-50 rises in every cell and GTZAN in five of the six, while the never-composed control moves by at most

0.8

points.

Explore similar work

Task Alignment: A Simple Proxy for Practical Model Merging Across Diverse Vision Tasks
Date pendingPau de Jorge, César Roberto de Souza, Björn Michele +5Continual Model Merging Multiple Vision Tasks
NAVER LABS Europe
Exploring Heterogeneous Model Merging Approach for Complex Knowledge Transfer
Sep 30, 2026Jiahe Fan, Si Chen, Yinghao Hou +4Cross-Task Knowledge Transfer Continual Model Merging
University of Science and Technology of China
Post-Training Leaves Behavioral Shadows on Unrelated Decisions
Sep 24, 2026Ziyang Zhang, Yubin Jing, Yuanhao Zeng +3Post-Training Agent Behavior Modeling
Peking University · Georgia Institute of Technology · ShanghaiTech University +2

Date pending, cs.CVJ/K move · Enter open · S save

Task Alignment: A Simple Proxy for Practical Model Merging Across Diverse Vision Tasks

Pau de Jorge, César Roberto de Souza, Björn Michele, Mert Bülent Sar\iy\ild\iz+4

NAVER LABS Europe

Continual Model Merging Multiple Vision Tasks Computer Vision

Open paper AskPDF arXiv

Efficiently merging several models fine-tuned for different tasks, but stemming from the same pretrained base model, is of great practical interest. Despite extensive prior work, most evaluations of model merging in computer vision are restricted to image classification using CLIP, where different classification datasets define different tasks. In this work, our goal is to make model merging more practical and show its relevance on challenging scenarios beyond this specific setting. In most vision scenarios, different tasks rely on trainable and usually heterogeneous decoders. Differently from previous studies with frozen decoders, where merged models can be evaluated right away, the non-trivial cost of decoder training renders hyperparameter selection based on downstream performance impractical. To address this, we introduce the task alignment proxy, and show how it can be used to speed up hyperparameter selection by orders of magnitude while retaining performance. Equipped with the task alignment proxy, we extend the applicability of model merging to multi-task vision models beyond CLIP-based classification. Project page: https://europe.naverlabs.com/task-alignment

Abstract

Explore similar work

Task Alignment: A Simple Proxy for Practical Model Merging Across Diverse Vision Tasks

Exploring Heterogeneous Model Merging Approach for Complex Knowledge Transfer

Post-Training Leaves Behavioral Shadows on Unrelated Decisions