Semantic Capability Acquisition and Specialization During Vision-Language Model Fine-Tuning

Suguru Onda; Matthew Bailey; Ryan Farrell

cs.CVOct 5, 2026

Semantic Capability Acquisition and Specialization During Vision-Language Model Fine-Tuning

Authors: Suguru Onda, Matthew Bailey, Ryan Farrell

Organizations: Brigham Young University · Carnegie Mellon University

Abstract

Fine-tuning vision-language models (VLMs) is typically evaluated at a single downstream checkpoint, obscuring whether a semantic capability was never acquired or emerged earlier and later declined during specialization. We ask how semantic capabilities are acquired, when they peak, how well they transfer, and what remains at deployment. We study these dynamics as a semantic capability trajectory, tracking identity- and attribute-based capabilities over training. We formulate a trajectory-based framework that separates capability acquisition, capability-specific optima, and later specialization, and introduce Structured Semantic Routing (SSR) to study how the representation of supervision shapes what is acquired. Across six pretrained backbones spanning DFN, MetaCLIP, and OpenAI CLIP, we show that fine-tuning can acquire semantic capability beyond the pretrained state, including gains observed on held-out evaluations. Unstructured name-and-attribute supervision produces strong name-and-attribute retrieval with comparatively weak name-free attribute-profile retrieval, whereas SSR yields substantially stronger name-free attribute-profile retrieval and is further strengthened by stochastic name-branch dropout. Different capabilities can peak at different stages, so a checkpoint selected by target class-name retrieval need not coincide with a transferable semantic optimum. Continued optimization can therefore preserve strong target class-name retrieval while reducing previously acquired transferable semantic capability. In a representative diagnostic study, this late specialization is consistent with reduced cross-modal semantic accessibility while substantial image-only class structure remains available.

Figures & tables

Figure 1: Supervision representation, training pathway, and capability probes. (a) N uses name-only supervision, U encodes the class name and attributes in one ordinary text stream, and SSR/SSR+ND use structured NAME and ATTRIBUTE segments, with a shared CONCEPT segment for CUB/NAB. (b) N/U training and all evaluations use the standard CLIP causal text pathway; SSR/SSR+ND use routed causal attention and a training-only fusion of NAME, pooled ATTRIBUTE, and EOT summaries, with NAME-summary dropout for SSR+ND. (c) Exact P1–P3 prompts and the CAPP counterfactual comparison. Photograph: CUB-200-2011.

Figure 2: Capability acquisition and trajectory structure. (a) DFN5B H/14-378 CUB $\to$ NAB trajectories; large markers = P1-selected deployment checkpoints. (b) Peak P3 gain over pretrained, six ImageNet-trained backbones $\times$ seven evaluation distributions.

Table 3

Figure 3: Why supervision changes what is acquired. (a) P2 decrease when only the class name or only the attributes are replaced; DFN5B H/14-378, CUB-trained (CIs: Table S20 ). (b) SSR+ND vs. SSR peak target P3 gain, 18 backbone

\times

domain pairs (Table S21 ). (c) Direct name-access ablation on the CUB target (DFN5B/CUB; one run per variant). Restricting ATTRIBUTE-route NAME access increases target P3 and Pairwise CAPP; NO is canonical SSR. Stars mark P1-selected deployment; PARTIAL deploys later, so this is a trajectory rather than matched-deployment comparison. Full trajectories: Figure S4 .

Figure 4: Capability-specific timing and specialization. (a) Representative peak versus deployment P3 gain for DFN5B H/14-378 fine-tuned on CUB-200. (b) Fraction of target (ImageNet val) and cross-ontology evaluations whose P3 optimum precedes deployment, by method across six backbones. (c) Peak-to-deployment P3 decrease for cross-ontology trajectories whose P3 optimum precedes deployment;

n

varies because cells peaking at deployment have no post-peak interval.

Figure 5: Late semantic specialization. (a) Joint backbone movement before the P3 peak versus acquisition, and from peak to deployment versus the post-peak P3 decrease, across held-out N/U/SSR trajectories;

r

= Pearson,

\rho

= Spearman,

n

= trajectory cells. (b) Representative fixed-competitor analysis for SSR+ND, DFN5B H/14-378, CUB

\to

NAB-348. The peak-selected hardest wrong class is held fixed at deployment; the correct-minus-wrong margin decreases by 0.00356 (95% class-bootstrap CI

[0.00194,0.00518]

). Because the competitor is the hardest wrong profile selected at the peak, its mean similarity can already exceed that of the correct profile; the analysis tests the subsequent change in their fixed-competitor margin.

Appendix figures & tables63 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1: Structured token sequences and canonical routed-attention masks. (a) CUB/NAB sequence with the shared concept, showing every start/end boundary token ( <sot> , <name> / <name_end> , <concept> / <concept_end> , <att> / <att_end> , <eot> ); the example attributes are “small” and “black crown”. (b) ImageNet sequence without a concept (class “goldfish”, attributes “orange-red body” and “bifurcated caudal fin”). (c, d) Token-level masks built from the mask definitions used in training: a blue cell means the row token attends to the column token, black is the token itself, light grey is masked. Every token attends <sot> ; attention within a segment is causal. In (c), ATTRIBUTE tokens also read the CONCEPT tokens ( <concept> and its content, but not <concept_end> ) and never the NAME segment or other attribute segments; <eot> reads only <name_end> and the <att_end> tokens. In (d) there is no concept and ATTRIBUTE tokens read only their own segment. “object” is a P3 evaluation-template word only, never a training token.

Figure S2: Token-level masks for the controlled name-access variants (direct name-access ablation; CUB/NAB shared-concept setup; same conventions as Figure S1 ). Orange cells mark access beyond the canonical routing. (a) FULL: ATTRIBUTE tokens also read all NAME tokens ( <name> , content, <name_end> ). (b) PARTIAL: ATTRIBUTE tokens also read only the <name_end> boundary token, not the class-name content. (c) NO: no NAME access; identical to the canonical SSR routing. In all variants <eot> still reads <name_end> , so NO removes direct NAME access from the ATTRIBUTE route only and the model as a whole is not name-free.

Dataset / class	Source	Extracted attributes used	P1 / P3 strings
CUB: Kentucky Warbler	Source: Cornell Lab, All About Birds species account ( Cornell Lab of Ornithology, 2025 )	small; olive-green upperparts; bright yellow underparts; black crown; black sideburns; yellow spectacles; no wing bars	P1: “a photo of a Kentucky Warbler”; P3: “a bird with {attributes}”
NAB: female or juvenile Lesser Goldfinch	Source: Cornell Lab, All About Birds species account ( Cornell Lab of Ornithology, 2025 )	stub bill; long pointed wings; short notched tail; olive back; dull yellow underparts; black wings; whitish wingbars	P3: “a bird with {attributes}”
FineView: Monarch	Source: BAMONA species account ( Lotts and Naberhaus, 2026 )	orange upperside; wide black borders; black veins; blurred black veins; white border spots; white wing-tip spots	P3: “a butterfly with {attributes}”
ImageNet: Goldfish	(LLM-generated; no raw source)	orange-red body; bright yellow body; torpedo-shaped body; large reflective scales; bifurcated caudal fin; stiff dorsal fin; smooth operculum	P3: “a object with {attributes}”

Appendix

Table S1: Verified examples of source resource, extracted attributes, and probe strings. NAB classes are plumage forms (e.g., female or juvenile), split from the per-species source text by the LLM. CUB attribute list shown is the CUB-59 strict-set list.

Long-context TULIP variant of the same backbone
Text input	DFN5B ViT-H/14-378	OpenAI ViT-L/14
Class name only	88.80	57.18
Concept + class name	87.88	55.93
Concept + class name + description ( $\sim$ 77 tokens)	84.77	50.07
Concept + class name + description (200–300 tokens)	36.02	41.50
Concept + class name + description (100–500 tokens)	28.55	39.30

Appendix

Table S2: Zero-shot CUB retrieval accuracy (%) for two pretrained backbones as text is added to the class-name prompt (frozen encoders, evaluation only). The lower block uses the long-context TULIP variant ( Najdenkoska et al., 2024 ) of each backbone, which can process descriptions beyond the 77-token window without truncation.

Backbone	open_clip model	Pretrained tag
DFN5B H/14-378	ViT-H-14-378-quickgelu	dfn5b
DFN2B L/14	ViT-L-14	dfn2b_s39b
MetaCLIP H/14	ViT-H-14-quickgelu	metaclip_fullcc
MetaCLIP L/14	ViT-L-14-quickgelu	metaclip_fullcc
OpenAI L/14	ViT-L-14-quickgelu	openai
OpenAI B/32	ViT-B-32-quickgelu	openai

Appendix

Table S3: Pretrained initializations (open_clip model names and pretrained tags read from the run parameter files). Backbone descriptions and parameter counts are in Table S17 .

Method	LR	LR sl	WD	Eff. batch	Routing
N	5e-06	5e-06	0.1	2000	–
U	5e-06	5e-06	0.1	2000	–
SSR	5e-06	5e-06	0.1	2000	no-concept
SSR+ND	5e-06	5e-06	0.1	2000	no-concept, $p{=}0.5$

Appendix

Table S4: ImageNet-1k training settings used across the six backbones; available run parameter records agree with these settings, with early OpenAI records unavailable as noted below. Training is a chain of 11 resumed steps, each one epoch over the incremental data slice up to cumulative fractions 0.25, 0.5, 1, 2, 5, 10, 20, 40, 60, 80, 100% (4k to 1.3M images; 2 to 650 cumulative effective batches), with a constant learning rate ( --skip-scheduler ; the declared warmup of 10000 is therefore inert). Milestone indices are not epochs and not proportional to gradient updates. For OpenAI L/14 and B/32 N/U/SSR, run parameter files were located only for steps 6–11 and 7–11 respectively; these are consistent with the other backbones, and steps 1–5 are unresolved.

Backbone	Method	LR	LR sl	WD	Eff. batch	Warmup	Deploy ep.	Routing
CUB-200 training
DFN5B H/14-378	N	5e-06	1e-05	0.1	2000	1000	40	–
	U	5e-06	1e-05	0.1	2000	1000	25	–
	SSR	5e-06	5e-06	0.1	2000	1000	20	shared-concept
	SSR+ND	5e-06	5e-06	0.1	2000	1000	60	shared-concept
DFN2B L/14	N	5e-06	1e-05	0.1	2000	1000	20	–

Appendix

Table S5: Training settings read from each canonical run’s saved parameter record (CUB-200 and NAB-555 training, 100 epochs, seed 0, AdamW-style open_clip trainer with multi-positive contrastive loss, bf16, no extra augmentation, cosine schedule). LR is the base learning rate; LR sl is the learning rate of the structured-routing layers (for N/U it is the unused default). Effective batch is 20

\times

gradient-accumulation steps (100 for CUB, 111 for NAB). Settings are parent-specific and not normalized: the OpenAI L/14 and B/32 CUB runs use weight decay 0.2 and warmup 10000, whereas the DFN/MetaCLIP runs use 0.1 and 1000; DFN5B CUB SSR and SSR+ND use LR

{}_{\mathrm{sl}}=5\times 10^{-6}

while all other structured runs use

10^{-5}

. SSR+ND uses name-branch dropout

p=0.5

. Deploy ep. is the run’s own P1-selected checkpoint.

Method	CUB/NAB	ImageNet
N	Ordinary causal text pathway; class-name-only text	same
U	Ordinary causal text pathway; unstructured descriptive text (class name and attributes in one stream)	same
SSR	Routed causal attention over NAME and ATTRIBUTE segments; shared CONCEPT segment (context only, not fused); ATTRIBUTE summary pooled from the <att_end> states; learned-gate Structured Text Fusion of NAME, ATTRIBUTE and EOT summaries (training only); the six added start/end marker embeddings (NAME, ATTRIBUTE, CONCEPT) are randomly initialized and trained	Same routing and fusion; no CONCEPT segment; the four added start/end marker embeddings (NAME, ATTRIBUTE) are randomly initialized and trained
SSR+ND	SSR with NAME-summary dropout $p=0.5$ at training-time fusion	same
Name-access ablation	FULL: the ATTRIBUTE route reads the full NAME segment; PARTIAL: it reads only the NAME_END token; NO: no direct NAME access (identical to canonical SSR)	–

Appendix

Table S6: Implementation summary of the training-time routings (reproducibility mapping; all conditions are evaluated with the ordinary text encoder and EOT embedding).

Dataset	Method	Seeds	i2t R@1	t2i R@1	Status
CUB-200	N	5	91.98 $\pm$ 0.05	94.20 $\pm$ 0.06	converged
	U	5	92.02 $\pm$ 0.10	94.03 $\pm$ 0.05	converged
	earlier SSR routing variant	5	92.01 $\pm$ 0.10	94.30 $\pm$ 0.11	converged
NABirds	N	5	89.00 $\pm$ 0.06	92.78 $\pm$ 0.04	not conv.
	U	5	88.91 $\pm$ 0.07	92.86 $\pm$ 0.24	not conv.
	earlier SSR routing variant	5	88.92 $\pm$ 0.09	93.14 $\pm$ 0.10	not conv.

Appendix

Table S7: Archived multi-seed runs (mean

\pm

sample std; DFN5B H/14-378 only; seeds 10/20/30/40/50; in-domain validation R@1 at the best checkpoint; sample standard deviation). None of these runs uses the current canonical SSR recipe (current: shared-concept routing, LR

5\times 10^{-6}

; archived structured runs: earlier routing variants, LR

10^{-5}

; the NABirds runs used a different earlier variant from the CUB-200 and FineView runs), so they are historical matched robustness checks, never pooled with the trajectory results, and no SSR+ND, ImageNet, or other-backbone multi-seed runs exist. No P1/P2/P3/CAPP was evaluated for them. NABirds runs were walltime-limited and are not converged (best checkpoints fall at epochs

\approx

19–23, with unequal numbers of logged epochs across seeds). On CUB, historical SSR used 30 accumulation steps versus 100 for N/U, so epoch-based comparisons are confounded.

Dataset	Method	Seed 10	Seed 20	Seed 30	Seed 40	Seed 50
CUB-200	N	91.95/94.22	92.02/94.23	91.92/94.25	92.03/94.10	92.00/94.18
	U	91.93/93.95	92.05/94.08	91.93/94.03	92.00/94.07	92.17/94.03
	hist. SSR	92.08/94.30	91.98/94.18	91.93/94.20	92.15/94.38	91.92/94.43
NABirds	N	89.07/92.76	88.94/92.74	89.02/92.79	89.02/92.84	88.93/92.79
	U	88.92/92.66	89.02/92.74	88.88/92.66	88.86/93.18	88.87/93.04
	hist. SSR	88.78/93.14	88.94/93.31	88.96/93.07	88.90/93.11	89.03/93.05

Appendix

Table S8: Per-seed best-checkpoint i2t / t2i R@1 (%) for the archived multi-seed runs summarized in Table S7 .

	CUB-trained			NABirds-trained
$p$	CUB-200 (target)	NAB-348	FineView	NAB-555 (target)	CUB-59	FineView
0	40.42	17.35	9.90	15.29	53.80	10.94
0.25	44.60	17.34	9.48	16.11	54.17	11.13
0.5	46.82	17.54	9.33	16.85	54.98	11.23
0.75	48.95	17.87	9.48	17.41	55.79	11.15
1	50.35	18.07	9.52	17.93	56.10	10.58

Appendix

Table S9: Historical NAME-drop probability calibration (DFN5B H/14-378; one run per

p

): maximum observed P3 (R@1, %) over saved epochs

>0

(the NABirds

p{=}0

run was evaluated only through epoch 40, the others to epoch 100) on the training-domain target and on held-out sets. The

p{=}0.5

CUB-trained run is the earlier weight-decay-0.2 run, not the canonical SSR+ND run. Held-out columns use each run’s own P3 peak over epochs (a peak-anywhere convention, not the P1-selected checkpoint).

$p$	$\to$ NAB-348 (%)	$\to$ FineView (%)
0	23.1	12.4
0.25	19.5	7.9
0.5	18.8	9.5
0.75	20.8	10.8
1	20.9	15.4

Appendix

Table S10: Historical NAME-drop calibration, CUB-trained DFN5B H/14-378: relative held-out P3 decrease,

(\mathrm{P3}_{\mathrm{peak}}-\mathrm{P3}_{\mathrm{P1\text{-}selected}})/\mathrm{P3}_{\mathrm{peak}}

, where the P1-selected checkpoint is each run’s own highest-CUB-P1 epoch. Values are those of the archived retention analysis (they agree with recomputation from the peak table’s source to rounding). One run per

p

; the

p{=}0

point is the baseline SSR run.

Method	P3 PT	P3 peak (milestone)	Gain peak	P3 deploy (milestone)	Gain deploy	Forget post-peak
ImageNet val (target)
N	33.12	33.78 (11)	0.66	33.78 (11)	0.66	0.00
U	33.12	35.38 (2)	2.26	24.80 (11)	-8.32	10.58
SSR	33.12	45.34 (10)	12.22	45.12 (11)	12.00	0.22
SSR+ND	33.12	48.12 (9)	15.00	48.00 (11)	14.88	0.12
DFN5B H/14-378

Appendix

Table S11: All six backbones, fine-tuned on ImageNet-1k and evaluated on ImageNet val (target); PT is the pretrained checkpoint evaluated directly on ImageNet val. Deploy is each run’s own P1-selected checkpoint and peak is the maximum P3 over the evaluated trajectory; best peak and deployment values are bolded independently within each backbone. Structured supervision produces substantially larger P3 acquisition than N/U, while U often shows an early gain that is reduced by deployment. Numbers in parentheses after P3 peak and P3 deploy give the ImageNet training milestone index (1–11), not an optimizer step count or epoch: milestones are 11 checkpoints at geometrically-spaced cumulative ImageNet-1k data fractions (0.25, 0.5, 1, 2, 5, 10, 20, 40, 60, 80, 100%), with PT as milestone 0. Because each later milestone spans a much larger increment of data than each earlier one, the milestone index is not proportional to the number of gradient updates or to wall-clock training progress. Deploy is each run’s own ImageNet-val P1-selected milestone, shared across all evaluation datasets for that run; peak is the milestone of the maximum P3 on that evaluation dataset, which can therefore differ from the deployment milestone and from peak timing on other datasets. When multiple milestones attain the same maximum P3, the earliest is reported.

Method	P3 PT	P3 peak (milestone)	Gain peak	P3 deploy (milestone)	Gain deploy	Forget post-peak
ImageNet-V2 (same-ontology shift)
N	29.29	30.76 (11)	1.47	30.76 (11)	1.47	0.00
U	29.29	30.90 (3)	1.61	22.24 (11)	-7.04	8.66
SSR	29.29	40.68 (10)	11.39	40.48 (11)	11.19	0.20
SSR+ND	29.29	42.83 (10)	13.54	42.68 (11)	13.39	0.15
DFN5B H/14-378

Appendix

Table S12: All six backbones, fine-tuned on ImageNet-1k and evaluated on the same-ontology shifts ImageNet-V2/-R/-Sketch; PT is the pretrained checkpoint evaluated directly on each listed evaluation dataset. Deploy is each run’s own (ImageNet-val) P1-selected checkpoint and peak is the maximum P3 over the evaluated trajectory on that dataset; best values are bolded independently within each backbone. Numbers in parentheses after P3 peak and P3 deploy give the ImageNet training milestone index (1–11), not an optimizer step count or epoch: milestones are 11 checkpoints at geometrically-spaced cumulative ImageNet-1k data fractions (0.25, 0.5, 1, 2, 5, 10, 20, 40, 60, 80, 100%), with PT as milestone 0. Because each later milestone spans a much larger increment of data than each earlier one, the milestone index is not proportional to the number of gradient updates or to wall-clock training progress. Deploy is each run’s own ImageNet-val P1-selected milestone, shared across all evaluation datasets for that run; peak is the milestone of the maximum P3 on that evaluation dataset, which can therefore differ from the deployment milestone and from peak timing on other datasets. When multiple milestones attain the same maximum P3, the earliest is reported.

Method	P3 PT	P3 peak (milestone)	Gain peak	P3 deploy (milestone)	Gain deploy	Forget post-peak
CUB-200 (cross-ontology held-out)
N	34.36	33.64 (1)	-0.72	20.73 (11)	-13.63	12.91
U	34.36	35.48 (1)	1.12	15.26 (11)	-19.11	20.23
SSR	34.36	34.76 (2)	0.40	23.35 (11)	-11.01	11.41
SSR+ND	34.36	34.88 (3)	0.52	24.99 (11)	-9.37	9.89
DFN5B H/14-378

Appendix

Table S13: All six backbones, fine-tuned on ImageNet-1k and evaluated cross-ontology on CUB-200, NAB-348 (strict dataset-class-disjoint held-out), and FineView Butterfly (FVB); PT is the pretrained checkpoint evaluated directly on each held-out dataset. Deploy is each run’s own P1-selected checkpoint (shared with Table S11 ) and peak is the maximum P3 over the trajectory; best peak and deployment values are bolded independently within each backbone (trajectory regimes in Table S16 ). Cross-ontology P3 acquisition is substantially smaller and more backbone-dependent than acquisition on ImageNet and its same-ontology shifts, and its peak often occurs earlier than the shared deployment milestone. Numbers in parentheses after P3 peak and P3 deploy give the ImageNet training milestone index (1–11), not an optimizer step count or epoch: milestones are 11 checkpoints at geometrically-spaced cumulative ImageNet-1k data fractions (0.25, 0.5, 1, 2, 5, 10, 20, 40, 60, 80, 100%), with PT as milestone 0. Because each later milestone spans a much larger increment of data than each earlier one, the milestone index is not proportional to the number of gradient updates or to wall-clock training progress. Deploy is each run’s own ImageNet-val P1-selected milestone, shared across all evaluation datasets for that run; peak is the milestone of the maximum P3 on that evaluation dataset, which can therefore differ from the deployment milestone and from peak timing on other datasets. When multiple milestones attain the same maximum P3, the earliest is reported.

Method	PT	Peak	Peak ep.	Deploy	Deploy ep.	Gain peak	Gain deploy	Forget post-peak
CUB-200 (target)
N	34.35	35.23	4	31.12	40	0.88	-3.23	4.11
U	34.45	38.87	12	35.78	25	4.42	1.33	3.09
SSR	34.35	40.42	20	40.42	20	6.08	6.08	0.00
SSR+ND	34.35	46.55	75	46.31	60	12.20	11.96	0.24
NAB-348 (strict cross-ontology held-out)

Appendix

Table S14: Representative DFN5B H/14-378 fine-tuned on CUB-200, with dense trajectory evaluation on CUB-200 (target) and NAB-348 (strict cross-ontology held-out); PT is the pretrained checkpoint evaluated directly on each dataset. Peak is the maximum P3 over the evaluated trajectory and deploy is each method’s own P1-selected checkpoint; best peak and deployment values are bolded independently within each dataset. All four fine-tuning conditions acquire held-out P3 above the pretrained state at some point, but their peak timing and post-peak retention differ substantially.

Method	Mean Gain peak (pp)	Median Gain peak (pp)	N positive	N $>$ 0.5pp
ImageNet val (target)
N	3.63	3.14	6/6	6/6
U	1.47	1.81	5/6	5/6
SSR	7.51	6.84	6/6	6/6
SSR+ND	13.23	13.70	6/6	6/6
ImageNet-V2 (same-ontology shift)

Appendix

Table S15: Six-backbone descriptive summary of peak P3 gain over pretrained, fine-tuned on ImageNet-1k and evaluated on ImageNet val (target), its same-ontology shifts, and cross-ontology held-out data (CUB-200, NAB-348, FineView Butterfly); PT for each backbone is evaluated directly on the corresponding dataset. Means and medians are descriptive across the six deliberately diverse pretrained backbones, not stochastic-replicate estimates; best values within each distribution are bolded, with ties retained. Peak P3 gains are largest under structured supervision on ImageNet-family evaluations, whereas cross-ontology gains are markedly smaller.

Backbone	Method	CUB-200	NAB-348	FVB
DFN5B H/14-378	N	-0.72, no acquisition / degraded	-0.41, no acquisition / degraded	-0.10, no acquisition / degraded
	U	1.12, acquire $\to$ lose	0.47, acquire $\to$ lose	0.23, acquire $\to$ lose
	SSR	0.40, acquire $\to$ lose	0.87, acquire $\to$ lose	0.33, acquire $\to$ lose
	SSR+ND	0.52, acquire $\to$ lose	1.23, acquire $\to$ lose	-0.14, no acquisition / degraded
DFN2B L/14	N	0.00, no acquisition / degraded	-0.01, no acquisition / degraded	0.17, acquire $\to$ retain
	U	-0.31, no acquisition / degraded	-0.20, no acquisition / degraded	-0.23, no acquisition / degraded

Appendix

Table S16: All six ImageNet-1k-trained backbones, evaluated cross-ontology on CUB-200, NAB-348 (strict held-out), and FineView Butterfly (FVB); PT is the pretrained checkpoint evaluated directly on the relevant held-out dataset. Each cell reports peak P3 gain over pretrained (pp) and a trajectory regime describing whether that gain was retained, partially reduced, or fully lost by deployment, or never clearly acquired. Poor deployed transfer can arise either because a capability was never clearly acquired or because it was acquired earlier and subsequently reduced.

Backbone	Family	Params	Pretraining source	ImageNet-val P1 (%)
DFN5B H/14-378	DFN	986M	DFN5B (5B pool)	83.9
DFN2B L/14	DFN	427M	DFN-2B (2B pool)	81.8
MetaCLIP H/14	MetaCLIP	986M	MetaCLIP (fullcc)	80.3
MetaCLIP L/14	MetaCLIP	427M	MetaCLIP (fullcc)	79.0
OpenAI L/14	OpenAI	427M	OpenAI (WIT-400M)	73.3
OpenAI B/32	OpenAI	151M	OpenAI (WIT-400M)	63.1

Appendix

Table S17: The six pretrained backbones used throughout the ImageNet-1k experiments (Figure 2 b), listed before any downstream fine-tuning. The final column is pretrained P1 (class-name retrieval R@1) evaluated on ImageNet val; no downstream trajectory result is represented in this table.

Method	P1 PT	P1 deploy	P3 PT	P3 peak	Gain peak (P3)
ImageNet val (target)
N	83.90	87.82	33.12	33.78	0.66
U	83.90	87.66	33.12	35.38	2.26
SSR	83.90	87.00	33.12	45.34	12.22
SSR+ND	83.90	86.96	33.12	48.12	15.00
DFN5B H/14-378

Appendix

Table S18: All six backbones, fine-tuned on ImageNet-1k and evaluated on ImageNet val (target) and its same-ontology shifts; PT is the pretrained checkpoint evaluated directly on each listed dataset. P1 deploy is class-name retrieval at each run’s own P1-selected deployment checkpoint, and P3 peak is the maximum P3 over the trajectory; within each backbone, the highest deployed P1 and peak P3 are bolded independently. Large differences in P3 acquisition can occur while deployed P1 remains comparatively similar.

Method	P2 ord.	P2 name-sw.	P2 attr-sw.	$\Delta_{name}$ [95% CI]	$\Delta_{attr}$ [95% CI]	P3
CUB-200 (target)
PT	86.31	0.10	76.54	-86.2 [-88.6, -83.4]	-9.8 [-13.2, -6.6]	34.36
N	90.68	0.07	89.68	-90.6 [-92.1, -88.8]	-1.0 [-1.9, -0.2]	31.10
U	91.35	0.05	90.63	-91.3 [-92.8, -89.7]	-0.7 [-1.5, -0.1]	35.78
SSR	89.44	0.14	85.33	-89.3 [-91.3, -86.9]	-4.1 [-6.6, -2.0]	40.40
SSR+ND	90.04	0.22	86.92	-89.8 [-91.5, -87.8]	-3.1 [-4.7, -1.7]	46.31

Appendix

Table S20: DFN5B H/14-378 fine-tuned on CUB-200, evaluated on CUB-200 (target) and NAB-348 (strict cross-ontology held-out) under five conditions (PT, N, U, SSR, SSR+ND); PT is the pretrained checkpoint evaluated directly on each listed dataset, and all fine-tuned methods are at their own P1-selected deployment checkpoint. Ordinary, name-swapped, and attribute-swapped P2 (R@1 %) are shown with

\Delta_{name}

\Delta_{attr}

, the pp change from ordinary, with class-level bootstrap 95% CIs (1000 replicates). Name-swap nearly eliminates P2 retrieval, whereas attribute-swap preserves most performance, showing that ordinary name+attribute retrieval remains strongly class-name anchored.

		SSR			SSR+ND
Backbone	Train	Gain peak	Gain deploy	Forget	Gain peak	Gain deploy	Forget
DFN5B H/14-378	ImageNet-trained	12.22	12.00	0.22	15.00	14.88	0.12
DFN5B H/14-378	CUB-trained	6.08	6.08	0.00	12.20	11.96	0.24
DFN5B H/14-378	NAB-trained	4.09	-0.23	4.32	5.65	1.02	4.62
DFN2B L/14	ImageNet-trained	5.30	4.96	0.34	8.10	8.10	0.00
DFN2B L/14	CUB-trained	3.78	3.69	0.09	8.99	8.99	0.00

Appendix

Table S21: All six backbones, fine-tuned separately on ImageNet-1k, CUB-200, and NABirds (18 backbone

\times

training-domain pairs), each evaluated on its own target training domain (ImageNet-trained

\to

ImageNet val, CUB-trained

\to

CUB-200, NAB-trained

\to

NAB-555) – this is a target-domain comparison, not a held-out-transfer table. Gain peak is target P3 peak minus target PT P3 (PT evaluated directly on the target dataset), Gain deploy uses each method’s own P1-selected deployment checkpoint, and Forget is their difference (underlies Figure 3 b; all values pp). SSR+ND exceeds SSR in peak target P3 acquisition in all 18 tested backbone

\times

training-domain pairs.

Backbone	Train domain	PT	N	U	SSR	SSR+ND
DFN5B H/14-378	CUB-200	72.99	73.90	74.85	76.30	76.76
DFN5B H/14-378	NAB-555	65.92	67.56	67.52	65.65	65.79
DFN2B L/14	CUB-200	72.70	72.46	70.36	73.38	74.42
DFN2B L/14	NAB-555	64.23	63.35	64.08	63.67	64.67
MetaCLIP H/14	CUB-200	70.57	70.47	69.43	71.68	72.93
MetaCLIP H/14	NAB-555	63.14	60.88	63.48	65.63	66.76

Appendix

Table S22: All six backbones (DFN/MetaCLIP backbones plus the additional OpenAI L/14 and B/32 backbones, marked below) fine-tuned separately on CUB-200 and on NABirds, each evaluated on its own target training ontology (CUB-trained

\to

CUB-200, NAB-trained

\to

NAB-555); PT is the pretrained checkpoint evaluated directly on the target dataset. Pairwise class-slot-macro accuracy at each method’s own P1-selected deployment checkpoint; multi-way top-1 in Table S23 . Structured supervision consistently improves controlled local attribute discrimination on the CUB target ontology, whereas NAB-target effects are more backbone-dependent.

Backbone	Train domain	PT	N	U	SSR	SSR+ND
DFN5B H/14-378	CUB-200	49.13	49.44	50.15	52.97	53.53
DFN5B H/14-378	NAB-555	35.93	37.34	37.77	35.33	35.29
DFN2B L/14	CUB-200	46.32	46.24	43.19	47.37	49.70
DFN2B L/14	NAB-555	32.81	31.31	32.58	32.29	32.97
MetaCLIP H/14	CUB-200	45.26	44.85	41.64	46.19	47.89
MetaCLIP H/14	NAB-555	31.80	27.98	30.64	33.80	35.10

Appendix

Table S23: All six backbones (DFN/MetaCLIP backbones plus the additional OpenAI L/14 and B/32 backbones, marked below) fine-tuned separately on CUB-200 and on NABirds, each evaluated on its own target training ontology (CUB-trained

\to

CUB-200, NAB-trained

\to

NAB-555); PT is the pretrained checkpoint evaluated directly on the target dataset. Multi-way class-slot-macro top-1 accuracy at each method’s own P1-selected deployment checkpoint; pairwise version in Table S22 . Structured supervision consistently improves controlled local attribute discrimination on the CUB target ontology, whereas NAB-target effects are more backbone-dependent.

Backbone	Train $\to$ eval	Population	PT	N	U	SSR	SSR+ND
DFN5B H/14-378	CUB $\to$ NAB	NAB-555 (full)	65.95	67.23	68.05	67.66	66.56
DFN5B H/14-378	CUB $\to$ NAB	NAB-348 (strict)	65.63	67.05	68.22	66.85	65.92
DFN5B H/14-378	NAB $\to$ CUB	CUB-200 (full)	72.99	73.99	72.88	71.10	70.50
DFN5B H/14-378	NAB $\to$ CUB	CUB-59 (strict)	74.73	74.72	73.57	70.00	69.06
DFN2B L/14	CUB $\to$ NAB	NAB-555 (full)	64.24	64.07	63.56	64.01	63.97
DFN2B L/14	CUB $\to$ NAB	NAB-348 (strict)	63.70	63.23	62.86	63.44	63.09

Appendix

Table S24: All six backbones (DFN/MetaCLIP backbones plus the additional OpenAI L/14 and B/32 backbones, marked below), evaluated cross-ontology in both directions: CUB-trained

\to

NAB (full held-out NAB-555, strict dataset-class-disjoint NAB-348) and NAB-trained

\to

CUB (full held-out CUB-200, strict dataset-class-disjoint CUB-59); “full” and “strict” are evaluation populations, not different training datasets, and are never pooled. PT is the pretrained checkpoint evaluated directly on each held-out population, and all methods are at their own P1-selected deployment checkpoint (pairwise class-slot-macro accuracy). Held-out CAPP effects are backbone- and transfer-direction-dependent; structured target-domain gains (Tables S22 – S23 ) do not imply universal cross-ontology gains.

Figure S3: Name-free attribute-profile retrieval (P3) change versus Pairwise CAPP change across intervention conditions, each relative to the pretrained model, at every run’s own P1-selected deployment checkpoint. Left: DFN/MetaCLIP backbones; right: additional OpenAI L/14 and B/32 backbones (never pooled). Colors: N, U, SSR, SSR+ND; circles CUB-trained and triangles NABirds-trained; filled markers are the training-domain target and open markers the strict dataset-class-disjoint held-out set (NAB-348 for CUB-trained, CUB-59 for NABirds-trained; full held-out sets are not plotted). Across many intervention conditions, gains in name-free attribute-profile retrieval and local counterfactual discrimination tend to move in the same direction, although the probes measure distinct capabilities. No correlation coefficient is computed. Delta values use the pretrained result from the same evaluation job as each point and may therefore differ slightly from differences reconstructed from separately tabulated evaluations.

Variant	Eval.	P3 PT	P3 peak (ep.)	P3 deploy (ep.)	Gain peak	Gain deploy	Forget
FULL	CUB-200	34.36	36.16 (4)	34.17 (20)	1.79	-0.19	1.99
	NAB-348	13.81	16.26 (12)	15.91 (20)	2.45	2.10	0.35
PARTIAL	CUB-200	34.45	37.80 (12)	36.97 (75)	3.35	2.52	0.83
	NAB-348	13.87	17.01 (12)	13.38 (75)	3.14	-0.49	3.63
NO	CUB-200	34.35	40.42 (20)	40.42 (20)	6.08	6.08	0.00
	NAB-348	13.85	17.32 (12)	16.90 (20)	3.48	3.06	0.42

Appendix

Table S25: Direct name-access variants (FULL / PARTIAL / NO), DFN5B H/14-378 fine-tuned on CUB-200, one run per variant (seed 0). Only the routing mask differs. FULL: the ATTRIBUTE route sees the full NAME segment; PARTIAL: only the NAME_END boundary signal; NO: no NAME access (identical to the canonical SSR run). EOT remains globally name-accessible in all three. Deployment is the CUB-P1-selected epoch (FULL 20, PARTIAL 75, NO 20), so PARTIAL’s deployment epoch differs from the others.

Variant	Eval.	Ep. 0 (PT)	Ep. 8	Ep. 20	Ep. 100
FULL	CUB-200	72.99	74.28	74.85	73.74
	NAB-348	65.62	65.65	66.72	65.81
PARTIAL	CUB-200	72.97	75.14	75.59	74.32
	NAB-348	65.62	65.94	67.05	66.07
NO	CUB-200	72.99	75.44	76.31	75.06
	NAB-348	65.62	65.94	66.85	65.60

Appendix

Table S26: Pairwise CAPP (class-slot-macro accuracy, %) for the name-access variants at matched epochs. On the CUB target, CAPP follows NO

>

PARTIAL

>

FULL at epochs 8, 20, and 100 (differences of 0.3–1.5 pp); on NAB-348 the variants are nearly indistinguishable and the gain is transient.

Variant	Eval.	Dep. ep.	P2 ord.	P2 name-sw.	P2 attr-sw.	Name-sw. damage	Attr-sw. damage
FULL	CUB-200	20	89.90	0.05	86.52	89.85	3.38
	NAB-348	20	79.66	0.30	69.82	79.35	9.83
PARTIAL	CUB-200	75	90.28	0.07	88.64	90.21	1.64
	NAB-348	75	73.42	0.23	65.60	73.19	7.83
NO	CUB-200	20	89.51	0.12	85.42	89.39	4.09
	NAB-348	20	78.97	0.33	68.39	78.64	10.59

Appendix

Table S27: Ordinary, name-swapped, and attribute-swapped P2 (R@1 %) at each variant’s deployment epoch. Name-swap collapses P2 to

\leq

0.33% in every variant, so restricting the ATTRIBUTE route’s name access does not remove the class-name dominance of ordinary P2.

Variant	Attr. end-mean	Name end-mean	Attr. pool-proj	Name pool-proj
FULL	0.39	0.84	0.40	0.83
PARTIAL	0.40	0.76	0.40	0.76
NO	0.65	0.35	0.68	0.36

Appendix

Table S28: Representational similarity (RSA) of the ATTRIBUTE-route representation at the last layer, epoch 20, to the pretrained name-free attribute-profile geometry and to class-name geometry, under two readouts (att_end_mean and att_pool_proj). Restricting direct NAME access raises similarity to the attribute-profile geometry and lowers similarity to class-name geometry under both readouts (main-text summary values use att_end_mean for the attribute profile and att_pool_proj for class name; both are shown here).

Figure S4: Direct name-access ablation trajectories, DFN5B H/14-378 fine-tuned on CUB-200 (one run per variant, single seed). Rows: P3 (name-free attribute-profile retrieval) and Pairwise CAPP; columns: CUB-200 (target) and strict held-out NAB-348. FULL, PARTIAL and NO direct name access denote how much of the NAME segment the ATTRIBUTE routes may read (Appendix A.5 ); NO is the canonical SSR run. Values are the change from each run’s own pretrained epoch-0 evaluation; stars mark each run’s P1-selected deployment epoch (FULL 20, PARTIAL 75, NO 20). P3 is evaluated at 11 epochs and CAPP at 7 (0, 4, 8, 12, 20, 75, 100), so the CAPP segment between epochs 20 and 75 is a straight line across unmeasured epochs. Restricting direct name access from the ATTRIBUTE route changes P3 and CAPP acquisition and trajectory behavior, supporting a role for representation and routing rather than content alone; NO is not name-free globally, because the EOT summary still reads NAME_END, and PARTIAL deploys later, so deployment-level comparisons are timing-confounded.

Figure S5: ImageNet-trained P1/P3 trajectories, DFN/MetaCLIP backbones (columns: DFN5B H/14-378, DFN2B L/14, MetaCLIP H/14, MetaCLIP L/14). For each evaluation population, the top row is P1 and the bottom row P3: ImageNet val (target), NAB-348 (strict dataset-class-disjoint held-out) and CUB-200 (held-out). The horizontal axis is the cumulative ImageNet training-data fraction at the 11 geometric milestones (not epochs). Deployment markers show the ImageNet-val-P1-selected checkpoint, which is shared by every population of a run; P3 uses the templates in Appendix B.3 (ImageNet: a object with {attributes} ; bird evaluations: a bird with {attributes} ).

Figure S6: ImageNet-trained P1/P3 trajectories, additional OpenAI backbones (OpenAI L/14 and B/32; the two families are never pooled). Layout, populations and definitions as in Figure S5 . The OpenAI deployment checkpoint is the ImageNet-val-P1 argmax over the 11 milestones; for L/14 it is the final milestone.

Figure S7: CUB-trained P1/P3 trajectories, DFN/MetaCLIP backbones. Rows: CUB-200 (target) and NAB-348 (strict dataset-class-disjoint held-out), each with P1 above P3. Horizontal axis: CUB training epoch. Deployment markers show each run’s CUB-P1-selected checkpoint. The DFN5B SSR+ND curve is the canonical SSR+ND run; dense fine-grained trajectories for the DFN/MetaCLIP backbones come from the per-epoch trajectory evaluations rather than a consolidated table, and the full NAB-555 held-out population has no P3 trajectory for the DFN/MetaCLIP backbones (it is available for OpenAI in Figure S9 ).

Figure S8: NABirds-trained P1/P3 trajectories, DFN/MetaCLIP backbones. Rows: NAB-555 (target) and CUB-59 (strict dataset-class-disjoint held-out), each with P1 above P3. Horizontal axis: NABirds training epoch (DFN5B N/U/SSR were evaluated to epoch 40). Deployment markers show each run’s NAB-P1-selected checkpoint. Definitions and single-run caveats as in Figure S7 .

Figure S9: OpenAI fine-grained P1/P3 trajectories (columns: OpenAI L/14 and B/32, CUB-trained and NABirds-trained). Rows: the training-domain target (CUB-200 or NAB-555), the full held-out population (NAB-555 or CUB-200; not dataset-class-disjoint) and the strict held-out population (NAB-348 or CUB-59; dataset-class-disjoint), each with P1 above P3; full and strict populations are never pooled. Deployment markers show each run’s target-domain-P1-selected epoch (argmax over the saved epochs after epoch 0). All four methods are single runs; OpenAI CUB-trained runs use weight decay 0.2 for all four methods.

Figure S10: P3 peak (diamond), Pairwise-CAPP peak (open triangle), and P1-selected deployment epoch (tick) for each method and evaluation set, DFN5B H/14-378 fine-tuned on CUB-200 (single run per method). The three need not coincide.

Method	Eval.	P3 peak (ep.)	P3 deploy (ep.)	CAPP peak (ep.)	CAPP deploy (ep.)
N	CUB-200 (target)	4	40	15	40
N	NAB-348 (held-out)	12	40	20	40
U	CUB-200 (target)	12	25	15	25
U	NAB-348 (held-out)	10	25	15	25
SSR	CUB-200 (target)	20	20	12	20
SSR	NAB-348 (held-out)	12	20	12	20

Appendix

Table S29: Dense P3 and Pairwise-CAPP peak/deployment epochs, DFN5B H/14-378 fine-tuned on CUB-200, evaluated on CUB-200 (target) and NAB-348 (strict held-out); underlies Figure S10 . Deployment epoch is each method’s own P1-selected checkpoint, shared by construction between the P3 and CAPP columns. CAPP is Pairwise class-slot-macro accuracy. Single run per method.

Population	Method	Cells	Before	At
target	N	6	2	4
	U	6	5	1
	SSR	6	3	3
	SSR+ND	6	1	5
	all	24	11	13
same-ontology	N	18	6	12

Appendix

Table S30: Timing of each evaluation distribution’s own P3-peak milestone relative to the run’s P1-selected deployment milestone, by method (six ImageNet-trained backbones). Target = ImageNet val (6 cells per method); same-ontology = ImageNet-V2/-R/-Sketch (18); cross-ontology = CUB-200, NAB-348, FineView Butterfly (18). Milestones are cumulative-data indices and are compared only categorically. Panel b of Figure 4 plots “before” for target and cross-ontology. Six heterogeneous backbones, not replicates.

Population	Method	Valid $n$	Median (pp)	IQR	$>0.5$ pp
target	N	2	0.34	[0.32, 0.36]	0
	U	5	3.32	[1.00, 5.56]	4
	SSR	3	0.34	[0.28, 1.00]	1
	SSR+ND	1	0.12	[0.12, 0.12]	0
	all	11	0.38	[0.31, 2.49]	5
cross-ontology	N	16	1.40	[0.62, 2.01]	14

Appendix

Table S31: Peak-to-deployment P3 decrease,

\mathrm{P3}_{\mathrm{peak}}-\mathrm{P3}_{\mathrm{deploy}}

(absolute pp), over cells whose own P3 peak precedes deployment (

t_{P3}<t_{\mathrm{deploy}}

); cells with peak at or after deployment have no evaluated post-peak interval and are excluded. Only 11 target cells qualify (n = 1–5 per method), too few for stable per-method target summaries, so Figure 4 c shows cross-ontology cells only. The

>0.5

pp count is a descriptive robustness count, not a significance test. The by-dataset rows are cross-ontology cells pooled over methods. OpenAI L/14 target and same-ontology cells all have the peak at the final milestone (Table S33 ), so absence of observed decline there is bounded by the evaluated schedule.

Backbone	T-bef	T-at	S-bef	S-at	X-bef	X-at
DFN5B H/14-378	3	1	10	2	12	0
DFN2B L/14	2	2	6	6	9	3
MetaCLIP H/14	1	3	6	6	11	1
MetaCLIP L/14	3	1	7	5	11	1
OpenAI L/14	0	4	0	12	11	1
OpenAI B/32	2	2	11	1	11	1

Appendix

Table S32: Per-backbone counts of evaluation cells (over the four methods N/U/SSR/SSR+ND and the datasets in each population) whose P3 peak milestone falls before, at, or after the run’s P1-selected deployment milestone. T = ImageNet val target (4 cells per backbone), S = same-ontology shifts ImageNet-V2/-R/-Sketch (12 cells), X = cross-ontology CUB-200/NAB-348/FVB (12 cells). Classification compares peak and deployment milestone indices from Tables S11 – S13 ; milestones are cumulative-data indices, not epochs, and are never pooled as absolute values across schedules. Column sums over backbones give the population totals in Table S30 .

Dataset	Method	Peak ms.	Deploy ms.	Gain peak
ImageNet val	N	11	11	5.24
	U	11	11	2.28
	SSR	11	11	9.66
	SSR+ND	11	11	19.88
ImageNet-V2	N	11	11	4.74
	U	11	11	1.89

Appendix

Table S33: OpenAI L/14 (ImageNet-trained): peak and deployment milestone indices for the ImageNet target and same-ontology shifts. In every one of these 16 cells the P1-selected deployment checkpoint is the final evaluated milestone (11) and coincides with the P3 maximum, so no post-peak interval exists in the evaluated trajectory; this is a statement about the audited schedule, not evidence that specialization cannot occur for this backbone. Cross-ontology cells (Table S13 ) do show peaks before deployment.

Training domain	Cells	$t_{P3}<t_{\mathrm{deploy}}$	$t_{P3}=t_{\mathrm{deploy}}$	$t_{P3}>t_{\mathrm{deploy}}$
CUB-trained	held-out	23	0	1
CUB-trained	target	11	1	0
ImageNet-trained	held-out	33	3	0
ImageNet-trained	target	10	2	0
NABirds-trained	held-out	22	1	1
NABirds-trained	target	8	3	1

Appendix

Table S34: Temporal order of each evaluation distribution’s own P3-peak checkpoint and the P1-selected deployment checkpoint over the 120 N/U/SSR cells (4 backbones

\times

3 training domains

\times

3 methods

\times

evaluation sets). Only

t_{P3}<t_{\mathrm{deploy}}

cells are temporally valid “post-peak” trajectories.

	Pre-peak vs. $\mathrm{Gain}_{\mathrm{peak}}$				Post-peak vs. $\mathrm{Forget}_{\mathrm{postpeak}}$
Subset	$n$	Image	Text	Joint	$n$	Image	Text	Joint
Held-out pooled (primary)	84	$-$ 0.17/+0.33	$-$ 0.14/+0.35	$-$ 0.16/+0.32	78	+0.43/+0.51	+0.54/+0.64	+0.50/+0.61
CUB-trained	24	+0.49/+0.58	+0.63/+0.69	+0.58/+0.63	23	+0.42/+0.49	+0.68/+0.72	+0.58/+0.59
NABirds-trained	24	$-$ 0.72/ $-$ 0.65	$-$ 0.73/ $-$ 0.70	$-$ 0.73/ $-$ 0.72	22	+0.64/+0.64	+0.56/+0.79	+0.64/+0.75
ImageNet-trained	36	+0.33/+0.57	+0.35/+0.57	+0.34/+0.53	33	+0.10/ $-$ 0.02	+0.56/+0.54	+0.39/+0.43
N	28	$-$ 0.25/+0.31	$-$ 0.23/+0.35	$-$ 0.24/+0.32	26	+0.45/+0.54	+0.40/+0.58	+0.44/+0.60

Appendix

Table S35: Encoder-separated and joint movement analysis (Pearson

r

/ Spearman

\rho

), held-out cells of the N/U/SSR grid; post-peak columns use

t_{P3}<t_{\mathrm{deploy}}

cells. Image movement: mean over image-encoder blocks ( visual.transformer.resblocks.* ) of

\|\Delta W\|_{F}/\|W_{\mathrm{PT}}\|_{F}

. Text movement: the same quantity over the shared pretrained text transformer ( transformer.resblocks.* ; 24 blocks for DFN5B/MetaCLIP H/14, 12 for DFN2B/MetaCLIP L/14). Joint = (image + text)/2, the “pretrained-backbone movement” used in Figure 5 a. Excluded because they lack a pretrained analogue or are not comparable: AIP and StructuredTextFusion parameters, the newly appended SSR token-embedding rows (4–6), positional embeddings, and the logit scale. Descriptive associations across heterogeneous backbones; not replicates.

Subset / normalization	$n$	Img– $\mathrm{F}$	Txt– $\mathrm{F}$	Img–Txt	Img $\|$ Txt	Txt $\|$ Img	Img	Txt	$R^{2}$	VIF
		Pearson $r$			Partial $r$		Std. coef.
Pooled (raw)	78	+0.43	+0.54	+0.85	$-$ 0.06	+0.36	$-$ 0.10	+0.62	0.29	3.5
CUB-trained	23	+0.42	+0.68	+0.72	$-$ 0.14	+0.60	$-$ 0.15	+0.79	0.48	2.1
NABirds-trained	22	+0.64	+0.56	+0.76	+0.40	+0.14	+0.51	+0.17	0.42	2.4
ImageNet-trained	33	+0.10	+0.56	+0.45	$-$ 0.22	+0.59	$-$ 0.20	+0.66	0.35	1.3
Domain-standardized	78	+0.34	+0.60	+0.62	$-$ 0.04	+0.52	$-$ 0.04	+0.62	0.36	1.6

Appendix

Table S36: Post-peak image versus text movement, held-out

t_{P3}<t_{\mathrm{deploy}}

cells (n = 78 pooled). F =

\mathrm{Forget}_{\mathrm{postpeak}}

; Img–Txt is the image/text movement correlation; Partial

r

controls the other encoder; Std. coef. are from the descriptive regression

\mathrm{F}\sim z(D_{\mathrm{image}})+z(D_{\mathrm{text}})

. Image and text movement are strongly coupled, and the apparent relative importance of the encoders depends on normalization (text dominates pooled and domain-standardized, image and text are indistinguishable within backbone; VIF up to about 11), so broad encoder attribution is not identifiable. Descriptive only; not a population-level regression or causal claim.

	Pre-peak: $D_{\mathrm{pre}}$ vs. $\mathrm{Gain}_{\mathrm{peak}}$			Post-peak: $D_{\mathrm{post}}$ vs. $\mathrm{Forget}_{\mathrm{postpeak}}$
Subset	$n$	$r$	$\rho$	$n$	$r$	$\rho$
Held-out, all (primary)	84	$-$ 0.17	+0.33	78	+0.43	+0.51
All valid cells	108	$-$ 0.15	+0.28	107	+0.44	+0.52
CUB-trained (held-out)	24	+0.49	+0.58	23	+0.42	+0.49
NABirds-trained (held-out)	24	$-$ 0.72	$-$ 0.65	22	+0.64	+0.64
ImageNet-trained (held-out)	36	+0.33	+0.57	33	+0.10	$-$ 0.02

Appendix

Table S37: Direct image-encoder movement versus P3 change across the N/U/SSR trajectory grid (Pearson

r

, Spearman

\rho

; absolute percentage points).

D_{\mathrm{pre}}

and

D_{\mathrm{post}}

are the layer-averaged Frobenius displacements

\|W_{\mathrm{peak}}-W_{\mathrm{PT}}\|/\|W_{\mathrm{PT}}\|

and

\|W_{\mathrm{deploy}}-W_{\mathrm{peak}}\|/\|W_{\mathrm{PT}}\|

over all image-encoder transformer-block parameters (attention, MLP, LayerNorm; projection, embeddings, text encoder and SSR modules excluded); net checkpoint displacement, not SGD path length. Post-peak rows use held-out cells with

t_{P3}<t_{\mathrm{deploy}}

unless stated; pre-peak rows need a pretrained P3 baseline. Backbones are heterogeneous models, not replicates, and all values are descriptive associations.

Figure S11: Backbone movement versus held-out P1 decrease, N/U/SSR grid, same 78 held-out cells and same joint image–text backbone movement as the right panel of Figure 5 a. P1 is evaluated at each trajectory’s P3-peak checkpoint and at its deployment checkpoint; no separate P1-optimal checkpoint is selected, and P1 decrease

=

P1 (P3-peak checkpoint)

-

P1 (deployment). This is a robustness analysis; held-out P1 is not treated as a name-free semantic-capability measure.

Probe, level	Rank	Wrong sim. peak $\to$ deploy	$\Delta$ corr.	$\Delta$ wrong	Encroachment [95% CI]	% cls. $\downarrow$ margin	Split-half [95% CI]
P3 hardest	1	0.203 $\to$ 0.211	+0.0044	+0.0080	+0.0036 [+0.0019, +0.0052]	59.5	+0.0034 [+0.0014, +0.0053]
P3 hard	35	0.090 $\to$ 0.098	+0.0044	+0.0084	+0.0039 [+0.0023, +0.0055]	62.6	+0.0047 [+0.0024, +0.0070]
P3 medium	174	$-$ 0.007 $\to$ 0.005	+0.0044	+0.0120	+0.0076 [+0.0057, +0.0094]	67.0	+0.0072 [+0.0049, +0.0095]
P3 easy	312	$-$ 0.096 $\to$ $-$ 0.079	+0.0044	+0.0168	+0.0124 [+0.0103, +0.0144]	74.1	+0.0116 [+0.0088, +0.0141]
P1 hardest	1	0.356 $\to$ 0.399	+0.0408	+0.0430	+0.0022 [+0.0006, +0.0038]	56.0	+0.0021 [+0.0004, +0.0039]
P1 hard	35	0.089 $\to$ 0.127	+0.0408	+0.0375	$-$ 0.0033 [ $-$ 0.0054, $-$ 0.0010]	42.0	$-$ 0.0044 [ $-$ 0.0074, $-$ 0.0013]

Appendix

Table S38: Peak-ranked wrong-class levels for the SSR+ND DFN5B H/14-378 CUB

\to

NAB-348 run. For every image the 347 wrong classes are ranked by similarity at the peak (epoch 20) only (rank 1 = most similar); the class at rank 1, 35 (about the 90th similarity percentile), 174 (median) and 312 (about the 10th percentile) is held fixed and reevaluated at deployment (epoch 60). Encroachment

=\Delta s_{\mathrm{wrong}}-\Delta s_{\mathrm{correct}}=-\Delta

margin, with margin = correct

-

wrong cosine similarity; class-macro means over the 348 NAB-348 classes, paired class bootstrap (10,000 resamples). The split-half column ranks wrong classes from half of each class’s images and evaluates on the other half. P1 and P3 are separate capabilities and are not pooled.

State	Image encoder	Text/routing encoder	CUB P1	CUB P3	NAB-348 P1	NAB-348 P3
peak	ep. 20	ep. 20	90.02	45.56	76.10	17.63
image late only	ep. 60	ep. 20	89.97	45.20	71.23	15.15
text late only	ep. 20	ep. 60	89.97	47.07	76.07	17.09
deploy	ep. 60	ep. 60	90.23	46.29	70.81	14.50

Appendix

Table S39: Encoder-state swap, DFN5B H/14-378 SSR+ND fine-tuned on CUB-200 (single run). Image encoder and text/routing encoder are taken from epoch 20 (peak) or epoch 60 (deployment); “image late only” combines the epoch-60 image encoder with the epoch-20 text/routing encoder, “text late only” the reverse. Values are R@1 (%).

Eval.	Probe	Effect	Change (pp) [95% CI]
CUB-200	P3	image-late	$-$ 0.44 [ $-$ 1.84, +0.89]
CUB-200	P3	text/SSR-late	+1.45 [+0.55, +2.37]
CUB-200	P3	total	+0.59 [ $-$ 1.06, +2.18]
CUB-200	P3	interaction	$-$ 0.42 [ $-$ 1.15, +0.29]
CUB-200	P3	image-late $-$ text/SSR-late	$-$ 1.90 [ $-$ 3.36, $-$ 0.44]
NAB-348	P3	image-late	$-$ 2.53 [ $-$ 3.53, $-$ 1.56]

Appendix

Table S40: Paired class-bootstrap decomposition of the encoder-state swap (Table S39 ); representative DFN5B H/14-378 SSR+ND CUB-trained run. “Image-late” is the effect of substituting the deployment image encoder at fixed peak text/routing state, “text/SSR-late” the reverse, “interaction” the residual of the additive decomposition. Class bootstrap over evaluation classes (10,000 resamples).

	Class-macro R@1 (%)				Effect (pp) [95% CI]
Evaluation	P I +P T	D I +P T	P I +D T	D I +D T	Image-late	Text/SSR-late	Image $-$ text	Interaction
MetaCLIP H/14, CUB $\to$ NAB-348 (ep. 12/75)
held-out, NAB-348 P3	12.39	11.66	12.46	11.64	$-$ 0.73 [ $-$ 1.54, +0.08]	+0.06 [ $-$ 0.38, +0.50]	$-$ 0.80 [ $-$ 1.68, +0.10]	$-$ 0.08 [ $-$ 0.52, +0.34]
held-out, NAB-348 P1	72.12	69.67	72.20	70.37	$-$ 2.44 [ $-$ 3.62, $-$ 1.30]	+0.08 [ $-$ 0.58, +0.74]	$-$ 2.52 [ $-$ 3.82, $-$ 1.25]	+0.61 [+0.12, +1.11]
target, CUB-200 P3	38.35	40.35	39.33	42.01	+2.00 [+0.51, +3.53]	+0.98 [ $-$ 0.01, +1.95]	+1.02 [ $-$ 0.32, +2.35]	+0.68 [ $-$ 0.26, +1.69]
target, CUB-200 P1	83.30	85.87	85.16	87.95	+2.56 [+1.40, +3.77]	+1.85 [+0.66, +3.13]	+0.71 [ $-$ 0.90, +2.33]	+0.23 [ $-$ 0.65, +1.14]

Appendix

Table S41: Additional encoder-state swaps on three additional SSR+ND runs, selected by a rule fixed before any swap was run (Appendix G.5 ). States: P = P3-peak checkpoint, D = deployment checkpoint, I = image encoder, T = text/SSR state (checkpoint epochs peak/deploy in the run label); effects use the same definitions as Table S40 (image-late = D I +P T

-

P I +P T ; text/SSR-late = P I +D T

-

P I +P T ). Paired class bootstrap over evaluation classes (10,000 resamples); NABirds-trained runs are evaluated on the dataset-class-disjoint CUB-59 subset (59 classes), target rows on the run’s own training-domain test set. Independent existing runs, not stochastic replicates.

CUB-200 ( $n{=}200$ )
State	P1 correct	P1 wrong	P1 margin	P3 correct	P3 wrong	P3 margin
peak	0.4533	0.3183	0.1349	0.2157	0.2280	-0.0123
image late only	0.5089	0.3453	0.1636	0.2352	0.2485	-0.0134
text late only	0.4567	0.3064	0.1504	0.2117	0.2201	-0.0084
deploy	0.5232	0.3366	0.1866	0.2327	0.2416	-0.0089
NAB-348 ( $n{=}348$ )

Appendix

Table S42: Cross-state embedding recombination, SSR+ND (DFN5B H/14-378, CUB-trained; single run). For each class, “correct” is the image-to-text cosine similarity to the correct class text anchor, “wrong” is the similarity to the hardest competing class anchor, and margin is correct minus wrong; entries are class-macro averages over the stated number of classes, computed for P1 and P3 text anchors. States follow Table S39 . Figure 5 c plots the NAB-348 margins as change relative to the peak state.

Model	Eval.	Anchor	PT corr.	PT wrong	Peak corr.	Peak wrong	Dep. corr.	Dep. wrong
SSR+ND	CUB-200	P3	0.3100	0.3217	0.2157	0.2280	0.2327	0.2416
	CUB-200	P1	0.3659	0.3290	0.4533	0.3183	0.5232	0.3366
	NAB-348	P3	0.2796	0.3082	0.1383	0.2028	0.1427	0.2197
	NAB-348	P1	0.3669	0.3425	0.4077	0.3559	0.4484	0.4038
U (text-frozen)	CUB-200	P3	0.3100	0.3217	0.1993	0.2109	0.2648	0.2462
	CUB-200	P1	0.3659	0.3290	0.4017	0.3157	0.4669	0.3386

Appendix

Table S43: Correct-class versus hardest-competitor text-anchor similarity (class-macro means) at pretrained (PT), peak, and deployment checkpoints for SSR+ND (peak epoch 20, deployment epoch 60) and an independent text-frozen U run (peak epoch 20, deployment epoch 75; a separate run from the main U run, whose deployment epoch is 25). In the text-frozen run the entire text encoder including layer norms is locked, and the text anchors were verified to be bit-identical across PT, peak, and deployment, so all change is image-side.

Figure S12: Supplementary late-specialization diagnostics. (a) Correct-class versus hardest-wrong text-anchor similarity, peak to deployment, where the hardest wrong class is re-selected at each checkpoint (see Table S44 for why this statistic can overstate encroachment). (b) Change in NAB-348 correct-minus-competitor margin from peak under cross-state recombination (I = image encoder, T = text/routing encoder).

Model	Eval.	Probe	Competitor	Encroachment ( $\times 10^{-3}$ ) [95% CI]	Classes $>0$
SSR+ND (joint)	CUB-200	P1	hardest wrong per checkpoint	$-51.66$ [-55.49, -47.88]	2%
SSR+ND (joint)	CUB-200	P1	fixed at peak (per image)	$-63.80$ [-67.61, -60.04]	1%
SSR+ND (joint)	CUB-200	P1	fixed at peak (per class)	$-71.66$ [-75.93, -67.35]	2%
SSR+ND (joint)	CUB-200	P3	hardest wrong per checkpoint	$-3.41$ [-6.03, -0.85]	40%
SSR+ND (joint)	CUB-200	P3	fixed at peak (per image)	$-11.84$ [-14.41, -9.35]	22%
SSR+ND (joint)	CUB-200	P3	fixed at peak (per class)	$-11.34$ [-14.31, -8.40]	28%

Appendix

Table S44: Peak

\to

deployment competitor encroachment,

\Delta s_{\mathrm{wrong}}-\Delta s_{\mathrm{correct}}

(class-macro mean of image–text-anchor cosine changes; positive = the competitor closes the gap relative to the correct anchor). “Hardest wrong per checkpoint” re-selects the maximizing wrong class independently at peak and at deployment and can rise merely because wrong-class similarities spread or the maximizer changes; the fixed-competitor rows freeze each image’s (or class’s) hardest wrong class at the peak checkpoint and re-evaluate it at deployment. Intervals: paired class bootstrap over evaluation classes (10,000 resamples), not training-seed replication. Peak epoch 20; deployment epoch 60 (SSR+ND) or 75 (U, text encoder frozen; text anchors bit-identical across checkpoints). Single runs, DFN5B H/14-378 fine-tuned on CUB-200.

Model	Eval.	Probe	Competitor	Spearman $\rho$ [95% CI]	Mean $\Delta$ acc. (pp)
SSR+ND (joint)	CUB-200	P1	hardest wrong per checkpoint	$-$ 0.09 [ $-$ 0.24, +0.06]	+0.2
SSR+ND (joint)	CUB-200	P1	fixed at peak (per image)	$-$ 0.09 [ $-$ 0.24, +0.06]	+0.2
SSR+ND (joint)	CUB-200	P3	hardest wrong per checkpoint	$-$ 0.60 [ $-$ 0.69, $-$ 0.51]	+0.6
SSR+ND (joint)	CUB-200	P3	fixed at peak (per image)	$-$ 0.57 [ $-$ 0.65, $-$ 0.47]	+0.6
SSR+ND (joint)	NAB-348	P1	hardest wrong per checkpoint	$-$ 0.57 [ $-$ 0.63, $-$ 0.49]	$-$ 5.3
SSR+ND (joint)	NAB-348	P1	fixed at peak (per image)	$-$ 0.58 [ $-$ 0.64, $-$ 0.50]	$-$ 5.3

Appendix

Table S45: Class-level Spearman correlation between encroachment and the peak

\to

deployment change in class top-1 accuracy (class bootstrap over evaluation classes). The relationship is partly mechanical, because encroachment and top-1 retrieval derive from the same similarity scores; it is supportive consistency, not independent mechanistic evidence.

Model	Eval.	PT	Peak	Deploy
SSR+ND	CUB-200	89.2	90.4	90.6
	NAB-348	90.9	90.6	88.4
U (text-frozen)	CUB-200	89.2	90.3	90.6
	NAB-348	90.8	91.2	88.0

Appendix

Table S46: Image-only leave-one-out prototype accuracy (class-macro, %). For each image, the class prototype is the mean image embedding of the other images of the same class; the image is assigned to the nearest prototype. This is an oracle diagnostic that uses held-out-image prototypes; it is not a zero-shot capability. Class structure remains largely preserved (88.0–91.2%), although NAB-348 shows a 2.7–3.1 pp decline from peak to deployment.

Geometry metric	Target	SSR+ND CUB	SSR+ND NAB	U-tf CUB	U-tf NAB
Tangent-residual alignment (P3)	$\Delta$ P3	0.33*	0.25*	0.15*	0.21*
	$\Delta$ P1	0.05	-0.09	-0.04	0.00
Tangent-residual alignment (P2)	$\Delta$ P3	0.05	0.11*	0.14	0.19*
	$\Delta$ P1	0.01	0.05	0.03	0.10
Raw alignment (P3)	$\Delta$ P3	0.23*	0.19*	0.12	0.19*
	$\Delta$ P1	0.02	-0.15*	-0.06	-0.08

Appendix

Table S47: Class-level Spearman correlations between late-minus-peak changes in coarse geometry metrics and changes in class accuracy (n = 200 CUB classes, 348 NAB classes; 10,000 class resamples for the 95% intervals). Asterisks mark intervals excluding 0. Tangent-residual alignment vs.

\Delta

P3 is weak but non-null in all four cells (

\rho\in[0.15,0.33]

); residual-direction coherence, manifold radius, and effective rank show no reliable relationship with

\Delta

P3, although some show non-null relationships with

\Delta

P1. The two models are SSR+ND and an independent text-frozen U run (U-tf). Full intervals are provided in the supporting results.

Method	$\alpha$	CUB P1	CUB P3	NAB-348 P1	NAB-348 P3
N	0	88.83	34.35	82.93	13.85
	0.25	90.87	36.61	83.60	14.60
	0.5	91.46	35.95	82.25	15.06
	0.75	91.39	33.62	78.06	15.00
	1	91.06	31.12	71.55	13.74
U	0	88.75	34.45	83.03	13.87

Appendix

Table S48: WiSE-style weight interpolation

\theta(\alpha)=(1-\alpha)\theta_{\mathrm{PT}}+\alpha\theta_{\mathrm{FT}}

for N (deployment epoch 40) and U (deployment epoch 25), DFN5B H/14-378 fine-tuned on CUB-200;

\alpha=0

is the pretrained model and

\alpha=1

the deployed checkpoint. Interpolation is applied at evaluation time to all trainable parameters; no training is involved. Values are R@1 (%) on CUB-200 (target) and NAB-348 (strict held-out). Single seed, coarse

\alpha

grid, one backbone and training domain.

Figure S13: Hard-negative semantic-discrimination control, DFN5B H/14-378 fine-tuned on CUB-200 (single seed,

\lambda=0.5

). Starting from SSR, a target-side ranking loss separates the correct attribute profile from a controlled hard negative differing in one attribute value. (a) Logit-scaled similarities of the image to the positive (

s_{\mathrm{pos}}

) and hard-negative (

s_{\mathrm{neg}}

) profiles and their margin, per training epoch (training-log quantities on the hard-negative batches, not an evaluation probe):

s_{\mathrm{neg}}

first rises (until about epoch 19) and then declines while

s_{\mathrm{pos}}

keeps increasing, so the relative margin grows. (b) Hard-negative ranking accuracy (fraction with

s_{\mathrm{pos}}>s_{\mathrm{neg}}

) first reaches 99.9% at epoch 41 (dotted line) and stays there, while the hard-negative text loss keeps decreasing. (c) P3 (name-free attribute-profile retrieval) on CUB-200 (target) and strict held-out NAB-348 at the 11 evaluated checkpoints (no interpolation): the hard-negative run exceeds the matched SSR baseline on target CUB P3 at later checkpoints but not on held-out NAB-348 P3 at matched epochs. This representative control shows that stronger target-side semantic discrimination need not translate into stronger transferable P3.

Epoch	CUB P3	NAB-348 P3	CUB P1	CUB align.	NAB align.
8	37.87 / 38.02	15.91 / 16.18	88.28 / 88.44	0.463 / 0.460	0.263 / 0.262
12	40.25 / 39.87	16.76 / 17.32	88.99 / 89.02	0.469 / 0.457	0.287 / 0.283
20	41.32 / 40.42	15.57 / 16.90	90.06 / 90.30	0.454 / 0.412	0.281 / 0.268
40	43.04 / 39.82	13.71 / 14.36	89.75 / 89.94	0.410 / 0.352	0.254 / 0.244
100	41.27 / 40.01	12.38 / 13.30	90.02 / 90.25	0.355 / 0.315	0.231 / 0.228

Appendix

Table S49: Hard-negative control versus the matched SSR baseline, DFN5B H/14-378, CUB-trained (single seed, single

\lambda

). Each cell is hard-negative / SSR. The objective adds

\lambda L_{\mathrm{HN}}

with

\lambda=0.5

to the multi-positive contrastive loss, where

L_{\mathrm{HN}}=\mathrm{mean}\,\mathrm{softplus}(s_{\mathrm{neg}}-s_{\mathrm{pos}})

is computed on the final fused text representation, and each negative differs from the true profile by one controlled attribute edit (pool of 618 candidates covering all 200 classes). Alignment is representational similarity between image and P3 text geometry. At each run’s own P1-selected checkpoint (hard-negative epoch 20, SSR epoch 75) CUB P3 is 41.32 vs. 39.97 and NAB-348 P3 is 15.57 vs. 13.34; at epoch 100, NAB-348 P3 is 12.38 vs. 13.30. Peak NAB-348 P3 is 16.76 (hard-negative) vs. 17.35 (SSR).

Item	CUB: Kentucky Warbler	NAB-348: adult male Lesser Goldfinch
P1	a photo of a Kentucky Warbler	a photo of a adult male Lesser Goldfinch
P3 template	a bird with {attributes}	a bird with {attributes}
CAPP correct profile	a bird with black crown, bright yellow underparts, olive-green upperparts	a bird with bright yellow belly, black cap, glossy cap, pointed wing, white wing patch
CAPP one-slot counterfactual	a bird with black crown, gray underparts , olive-green upperparts (slot: underparts color)	a bird with bright yellow belly, black cap, glossy cap, rounded wing , white wing patch (slot: wing shape)
Item	FineView: Monarch	ImageNet: Goldfish
P3 template	a butterfly with {attributes}	a object with {attributes}

Appendix

Table S50: Verified probe examples. The CAPP pairs are drawn from the reviewed full-class CAPP profiles (donor values for the swapped slot come from other classes). Only the changed slot is highlighted. The Figure 1 c CAPP panel is an illustrative profile, not one of these verified pairs.

Main item	Supplement table(s)	Population	Checkpoint rule	Evidence type
Fig. 2 b	Tables S11 – S15	ImageNet target, same-ontology shifts, cross-ontology; six backbones	trajectory peak	six-backbone descriptive
Fig. 2 a	Table S14	CUB target, NAB-348 held-out; DFN5B	peak, own-P1 deploy	single run per method
Table 4.2	Table S19	ImageNet val; six backbones	own-P1 deploy	six-backbone descriptive
Fig. 3 a	Table S20	CUB target, NAB-348; DFN5B	own-P1 deploy	single run, 1000-replicate class bootstrap
Fig. 3 b	Table S21	3 training domains $\times$ six backbones (18 pairs)	trajectory peak	six-backbone descriptive
Table 4.2	Tables S22 – S24	CUB/NAB target; six backbones	own-P1 deploy	six-backbone descriptive

Appendix

Table S51: Main-paper items mapped to supplement tables, evaluation populations, checkpoint rules, and evidence type. Evidence types are distinct: single training runs, six heterogeneous backbones (descriptive), class-level bootstrap resampling of the evaluation set, and archived historical multi-seed checks (Table S7 ); they must not be conflated.

Explore similar work

Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosis
Sep 1, 2026Minsik Choi, Geewook Kim, Young Geun KimVision-Language Model Adaptation Dynamic Attention
Korea University · NAVER Cloud AI
From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models
May 19, 2026Juncheng Wu, Hardy Chen, Haoqin Tu +6Recent Vision-Language Models Visual Reasoning
Amazon · UC Santa Cruz · University of Waterloo +1
Seeing and Solving Are Not Enough for Vision-Language Models
Sep 27, 2026Ziheng Wang, Mingxuan Xie, Yilin Liu +3Multimodal Query State-Tracking
Sun Yat-sen University · Zhejiang University · The Hong Kong University of Science and Technology +2

Sep 1, 2026, cs.LGJ/K move · Enter open · S save

Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosis

Minsik Choi, Geewook Kim, Young Geun Kim

Korea University · NAVER Cloud AI

Vision-Language Model Adaptation Dynamic Attention

Open paper AskPDF arXiv

Fine-tuning a pretrained LLM into a vision-language model (VLM) can erode the backbone's text capability, with the damage concentrated on tasks that require following exact output rules, such as instruction following, chain-of-thought reasoning graded on a strictly parsed final answer, and similar evaluations with strict graders. We trace this gap to attention-sink corruption: VL fine-tuning perturbs the early sink position that anchors a large fraction of attention probability, and how well the base LLM preserves its sink tracks how much of the affected capability survives adaptation. Building on this view, we introduce Sink Strength, a single scalar computed on the base LLM in a few seconds on a single GPU that predicts post-VL degradation without any VL training. It consistently tracks relative degradation across the six VLM-LLM pairs and multiple format-sensitive tasks. Complementing this diagnostic, we find that post-pretraining QK-RMSNorm injection fails to reproduce the protection of native QK-RMSNorm, while several off-the-shelf weight-merging settings fail to recover the lost capability after VL training. These negative results underscore the value of screening backbones with Sink Strength before VL training and narrow the intervention space toward head-selective training-time protection.

Abstract

Explore similar work

Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosis

From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models

Seeing and Solving Are Not Enough for Vision-Language Models