Fine-tuning vision-language models (VLMs) is typically evaluated at a single downstream checkpoint, obscuring whether a semantic capability was never acquired or emerged earlier and later declined during specialization. We ask how semantic capabilities are acquired, when they peak, how well they transfer, and what remains at deployment. We study these dynamics as a semantic capability trajectory, tracking identity- and attribute-based capabilities over training. We formulate a trajectory-based framework that separates capability acquisition, capability-specific optima, and later specialization, and introduce Structured Semantic Routing (SSR) to study how the representation of supervision shapes what is acquired. Across six pretrained backbones spanning DFN, MetaCLIP, and OpenAI CLIP, we show that fine-tuning can acquire semantic capability beyond the pretrained state, including gains observed on held-out evaluations. Unstructured name-and-attribute supervision produces strong name-and-attribute retrieval with comparatively weak name-free attribute-profile retrieval, whereas SSR yields substantially stronger name-free attribute-profile retrieval and is further strengthened by stochastic name-branch dropout. Different capabilities can peak at different stages, so a checkpoint selected by target class-name retrieval need not coincide with a transferable semantic optimum. Continued optimization can therefore preserve strong target class-name retrieval while reducing previously acquired transferable semantic capability. In a representative diagnostic study, this late specialization is consistent with reduced cross-modal semantic accessibility while substantial image-only class structure remains available.
Figures & tables
Figure 1: Supervision representation, training pathway, and capability probes. (a) N uses name-only supervision, U encodes the class name and attributes in one ordinary text stream, and SSR/SSR+ND use structured NAME and ATTRIBUTE segments, with a shared CONCEPT segment for CUB/NAB. (b) N/U training and all evaluations use the standard CLIP causal text pathway; SSR/SSR+ND use routed causal attention and a training-only fusion of NAME, pooled ATTRIBUTE, and EOT summaries, with NAME-summary dropout for SSR+ND. (c) Exact P1–P3 prompts and the CAPP counterfactual comparison. Photograph: CUB-200-2011.
Figure 2: Capability acquisition and trajectory structure. (a) DFN5B H/14-378 CUB → NAB trajectories; large markers = P1-selected deployment checkpoints. (b) Peak P3 gain over pretrained, six ImageNet-trained backbones × seven evaluation distributions.
Table 3
Figure 3: Why supervision changes what is acquired. (a) P2 decrease when only the class name or only the attributes are replaced; DFN5B H/14-378, CUB-trained (CIs: Table S20 ). (b) SSR+ND vs. SSR peak target P3 gain, 18 backbone × domain pairs (Table S21 ). (c) Direct name-access ablation on the CUB target (DFN5B/CUB; one run per variant). Restricting ATTRIBUTE-route NAME access increases target P3 and Pairwise CAPP; NO is canonical SSR. Stars mark P1-selected deployment; PARTIAL deploys later, so this is a trajectory rather than matched-deployment comparison. Full trajectories: Figure S4 .
Figure 4: Capability-specific timing and specialization. (a) Representative peak versus deployment P3 gain for DFN5B H/14-378 fine-tuned on CUB-200. (b) Fraction of target (ImageNet val) and cross-ontology evaluations whose P3 optimum precedes deployment, by method across six backbones. (c) Peak-to-deployment P3 decrease for cross-ontology trajectories whose P3 optimum precedes deployment; n varies because cells peaking at deployment have no post-peak interval.
Figure 5: Late semantic specialization. (a) Joint backbone movement before the P3 peak versus acquisition, and from peak to deployment versus the post-peak P3 decrease, across held-out N/U/SSR trajectories; r = Pearson, ρ = Spearman, n = trajectory cells. (b) Representative fixed-competitor analysis for SSR+ND, DFN5B H/14-378, CUB → NAB-348. The peak-selected hardest wrong class is held fixed at deployment; the correct-minus-wrong margin decreases by 0.00356 (95% class-bootstrap CI [0.00194,0.00518] ). Because the competitor is the hardest wrong profile selected at the peak, its mean similarity can already exceed that of the correct profile; the analysis tests the subsequent change in their fixed-competitor margin.
Appendix figures & tables63 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1: Structured token sequences and canonical routed-attention masks. (a) CUB/NAB sequence with the shared concept, showing every start/end boundary token ( <sot> , <name> / <name_end> , <concept> / <concept_end> , <att> / <att_end> , <eot> ); the example attributes are “small” and “black crown”. (b) ImageNet sequence without a concept (class “goldfish”, attributes “orange-red body” and “bifurcated caudal fin”). (c, d) Token-level masks built from the mask definitions used in training: a blue cell means the row token attends to the column token, black is the token itself, light grey is masked. Every token attends <sot> ; attention within a segment is causal. In (c), ATTRIBUTE tokens also read the CONCEPT tokens ( <concept> and its content, but not <concept_end> ) and never the NAME segment or other attribute segments; <eot> reads only <name_end> and the <att_end> tokens. In (d) there is no concept and ATTRIBUTE tokens read only their own segment. “object” is a P3 evaluation-template word only, never a training token.
Figure S2: Token-level masks for the controlled name-access variants (direct name-access ablation; CUB/NAB shared-concept setup; same conventions as Figure S1 ). Orange cells mark access beyond the canonical routing. (a) FULL: ATTRIBUTE tokens also read all NAME tokens ( <name> , content, <name_end> ). (b) PARTIAL: ATTRIBUTE tokens also read only the <name_end> boundary token, not the class-name content. (c) NO: no NAME access; identical to the canonical SSR routing. In all variants <eot> still reads <name_end> , so NO removes direct NAME access from the ATTRIBUTE route only and the model as a whole is not name-free.
Dataset / class
Source
Extracted attributes used
P1 / P3 strings
CUB: Kentucky Warbler
Source: Cornell Lab, All About Birds species account ( Cornell Lab of Ornithology, 2025 )
small; olive-green upperparts; bright yellow underparts; black crown; black sideburns; yellow spectacles; no wing bars
P1: “a photo of a Kentucky Warbler”; P3: “a bird with {attributes}”
NAB: female or juvenile Lesser Goldfinch
Source: Cornell Lab, All About Birds species account ( Cornell Lab of Ornithology, 2025 )
stub bill; long pointed wings; short notched tail; olive back; dull yellow underparts; black wings; whitish wingbars
P3: “a bird with {attributes}”
FineView: Monarch
Source: BAMONA species account ( Lotts and Naberhaus, 2026 )
orange upperside; wide black borders; black veins; blurred black veins; white border spots; white wing-tip spots
Table S1: Verified examples of source resource, extracted attributes, and probe strings. NAB classes are plumage forms (e.g., female or juvenile), split from the per-species source text by the LLM. CUB attribute list shown is the CUB-59 strict-set list.
Text input
DFN5B ViT-H/14-378
OpenAI ViT-L/14
Class name only
88.80
57.18
Concept + class name
87.88
55.93
Concept + class name + description ( ∼ 77 tokens)
84.77
50.07
Long-context TULIP variant of the same backbone
Concept + class name + description (200–300 tokens)
36.02
41.50
Concept + class name + description (100–500 tokens)
28.55
39.30
Appendix
Table S2: Zero-shot CUB retrieval accuracy (%) for two pretrained backbones as text is added to the class-name prompt (frozen encoders, evaluation only). The lower block uses the long-context TULIP variant ( Najdenkoska et al., 2024 ) of each backbone, which can process descriptions beyond the 77-token window without truncation.
Backbone
open_clip model
Pretrained tag
DFN5B H/14-378
ViT-H-14-378-quickgelu
dfn5b
DFN2B L/14
ViT-L-14
dfn2b_s39b
MetaCLIP H/14
ViT-H-14-quickgelu
metaclip_fullcc
MetaCLIP L/14
ViT-L-14-quickgelu
metaclip_fullcc
OpenAI L/14
ViT-L-14-quickgelu
openai
OpenAI B/32
ViT-B-32-quickgelu
openai
Appendix
Table S3: Pretrained initializations (open_clip model names and pretrained tags read from the run parameter files). Backbone descriptions and parameter counts are in Table S17 .
Method
LR
LR sl
WD
Eff. batch
Routing
N
5e-06
5e-06
0.1
2000
–
U
5e-06
5e-06
0.1
2000
–
SSR
5e-06
5e-06
0.1
2000
no-concept
SSR+ND
5e-06
5e-06
0.1
2000
no-concept, p=0.5
Appendix
Table S4: ImageNet-1k training settings used across the six backbones; available run parameter records agree with these settings, with early OpenAI records unavailable as noted below. Training is a chain of 11 resumed steps, each one epoch over the incremental data slice up to cumulative fractions 0.25, 0.5, 1, 2, 5, 10, 20, 40, 60, 80, 100% (4k to 1.3M images; 2 to 650 cumulative effective batches), with a constant learning rate ( --skip-scheduler ; the declared warmup of 10000 is therefore inert). Milestone indices are not epochs and not proportional to gradient updates. For OpenAI L/14 and B/32 N/U/SSR, run parameter files were located only for steps 6–11 and 7–11 respectively; these are consistent with the other backbones, and steps 1–5 are unresolved.
Backbone
Method
LR
LR sl
WD
Eff. batch
Warmup
Deploy ep.
Routing
CUB-200 training
DFN5B H/14-378
N
5e-06
1e-05
0.1
2000
1000
40
–
U
5e-06
1e-05
0.1
2000
1000
25
–
SSR
5e-06
5e-06
0.1
2000
1000
20
shared-concept
SSR+ND
5e-06
5e-06
0.1
2000
1000
60
shared-concept
DFN2B L/14
N
5e-06
1e-05
0.1
2000
1000
20
–
Appendix
Table S5: Training settings read from each canonical run’s saved parameter record (CUB-200 and NAB-555 training, 100 epochs, seed 0, AdamW-style open_clip trainer with multi-positive contrastive loss, bf16, no extra augmentation, cosine schedule). LR is the base learning rate; LR sl is the learning rate of the structured-routing layers (for N/U it is the unused default). Effective batch is 20 × gradient-accumulation steps (100 for CUB, 111 for NAB). Settings are parent-specific and not normalized: the OpenAI L/14 and B/32 CUB runs use weight decay 0.2 and warmup 10000, whereas the DFN/MetaCLIP runs use 0.1 and 1000; DFN5B CUB SSR and SSR+ND use LR sl=5×10−6 while all other structured runs use 10−5 . SSR+ND uses name-branch dropout p=0.5 . Deploy ep. is the run’s own P1-selected checkpoint.
Method
CUB/NAB
ImageNet
N
Ordinary causal text pathway; class-name-only text
same
U
Ordinary causal text pathway; unstructured descriptive text (class name and attributes in one stream)
same
SSR
Routed causal attention over NAME and ATTRIBUTE segments; shared CONCEPT segment (context only, not fused); ATTRIBUTE summary pooled from the <att_end> states; learned-gate Structured Text Fusion of NAME, ATTRIBUTE and EOT summaries (training only); the six added start/end marker embeddings (NAME, ATTRIBUTE, CONCEPT) are randomly initialized and trained
Same routing and fusion; no CONCEPT segment; the four added start/end marker embeddings (NAME, ATTRIBUTE) are randomly initialized and trained
SSR+ND
SSR with NAME-summary dropout p=0.5 at training-time fusion
same
Name-access ablation
FULL: the ATTRIBUTE route reads the full NAME segment; PARTIAL: it reads only the NAME_END token; NO: no direct NAME access (identical to canonical SSR)
–
Appendix
Table S6: Implementation summary of the training-time routings (reproducibility mapping; all conditions are evaluated with the ordinary text encoder and EOT embedding).
Dataset
Method
Seeds
i2t R@1
t2i R@1
Status
CUB-200
N
5
91.98 ± 0.05
94.20 ± 0.06
converged
U
5
92.02 ± 0.10
94.03 ± 0.05
converged
earlier SSR routing variant
5
92.01 ± 0.10
94.30 ± 0.11
converged
NABirds
N
5
89.00 ± 0.06
92.78 ± 0.04
not conv.
U
5
88.91 ± 0.07
92.86 ± 0.24
not conv.
earlier SSR routing variant
5
88.92 ± 0.09
93.14 ± 0.10
not conv.
Appendix
Table S7: Archived multi-seed runs (mean ± sample std; DFN5B H/14-378 only; seeds 10/20/30/40/50; in-domain validation R@1 at the best checkpoint; sample standard deviation). None of these runs uses the current canonical SSR recipe (current: shared-concept routing, LR 5×10−6 ; archived structured runs: earlier routing variants, LR 10−5 ; the NABirds runs used a different earlier variant from the CUB-200 and FineView runs), so they are historical matched robustness checks, never pooled with the trajectory results, and no SSR+ND, ImageNet, or other-backbone multi-seed runs exist. No P1/P2/P3/CAPP was evaluated for them. NABirds runs were walltime-limited and are not converged (best checkpoints fall at epochs ≈ 19–23, with unequal numbers of logged epochs across seeds). On CUB, historical SSR used 30 accumulation steps versus 100 for N/U, so epoch-based comparisons are confounded.
Dataset
Method
Seed 10
Seed 20
Seed 30
Seed 40
Seed 50
CUB-200
N
91.95/94.22
92.02/94.23
91.92/94.25
92.03/94.10
92.00/94.18
U
91.93/93.95
92.05/94.08
91.93/94.03
92.00/94.07
92.17/94.03
hist. SSR
92.08/94.30
91.98/94.18
91.93/94.20
92.15/94.38
91.92/94.43
NABirds
N
89.07/92.76
88.94/92.74
89.02/92.79
89.02/92.84
88.93/92.79
U
88.92/92.66
89.02/92.74
88.88/92.66
88.86/93.18
88.87/93.04
hist. SSR
88.78/93.14
88.94/93.31
88.96/93.07
88.90/93.11
89.03/93.05
Appendix
Table S8: Per-seed best-checkpoint i2t / t2i R@1 (%) for the archived multi-seed runs summarized in Table S7 .
CUB-trained
NABirds-trained
p
CUB-200 (target)
NAB-348
FineView
NAB-555 (target)
CUB-59
FineView
0
40.42
17.35
9.90
15.29
53.80
10.94
0.25
44.60
17.34
9.48
16.11
54.17
11.13
0.5
46.82
17.54
9.33
16.85
54.98
11.23
0.75
48.95
17.87
9.48
17.41
55.79
11.15
1
50.35
18.07
9.52
17.93
56.10
10.58
Appendix
Table S9: Historical NAME-drop probability calibration (DFN5B H/14-378; one run per p ): maximum observed P3 (R@1, %) over saved epochs >0 (the NABirds p=0 run was evaluated only through epoch 40, the others to epoch 100) on the training-domain target and on held-out sets. The p=0.5 CUB-trained run is the earlier weight-decay-0.2 run, not the canonical SSR+ND run. Held-out columns use each run’s own P3 peak over epochs (a peak-anywhere convention, not the P1-selected checkpoint).
p
→ NAB-348 (%)
→ FineView (%)
0
23.1
12.4
0.25
19.5
7.9
0.5
18.8
9.5
0.75
20.8
10.8
1
20.9
15.4
Appendix
Table S10: Historical NAME-drop calibration, CUB-trained DFN5B H/14-378: relative held-out P3 decrease, (P3peak−P3P1-selected)/P3peak , where the P1-selected checkpoint is each run’s own highest-CUB-P1 epoch. Values are those of the archived retention analysis (they agree with recomputation from the peak table’s source to rounding). One run per p ; the p=0 point is the baseline SSR run.
Method
P3 PT
P3 peak (milestone)
Gain peak
P3 deploy (milestone)
Gain deploy
Forget post-peak
ImageNet val (target)
N
33.12
33.78 (11)
0.66
33.78 (11)
0.66
0.00
U
33.12
35.38 (2)
2.26
24.80 (11)
-8.32
10.58
SSR
33.12
45.34 (10)
12.22
45.12 (11)
12.00
0.22
SSR+ND
33.12
48.12 (9)
15.00
48.00 (11)
14.88
0.12
DFN5B H/14-378
Appendix
Table S11: All six backbones, fine-tuned on ImageNet-1k and evaluated on ImageNet val (target); PT is the pretrained checkpoint evaluated directly on ImageNet val. Deploy is each run’s own P1-selected checkpoint and peak is the maximum P3 over the evaluated trajectory; best peak and deployment values are bolded independently within each backbone. Structured supervision produces substantially larger P3 acquisition than N/U, while U often shows an early gain that is reduced by deployment. Numbers in parentheses after P3 peak and P3 deploy give the ImageNet training milestone index (1–11), not an optimizer step count or epoch: milestones are 11 checkpoints at geometrically-spaced cumulative ImageNet-1k data fractions (0.25, 0.5, 1, 2, 5, 10, 20, 40, 60, 80, 100%), with PT as milestone 0. Because each later milestone spans a much larger increment of data than each earlier one, the milestone index is not proportional to the number of gradient updates or to wall-clock training progress. Deploy is each run’s own ImageNet-val P1-selected milestone, shared across all evaluation datasets for that run; peak is the milestone of the maximum P3 on that evaluation dataset, which can therefore differ from the deployment milestone and from peak timing on other datasets. When multiple milestones attain the same maximum P3, the earliest is reported.
Method
P3 PT
P3 peak (milestone)
Gain peak
P3 deploy (milestone)
Gain deploy
Forget post-peak
ImageNet-V2 (same-ontology shift)
N
29.29
30.76 (11)
1.47
30.76 (11)
1.47
0.00
U
29.29
30.90 (3)
1.61
22.24 (11)
-7.04
8.66
SSR
29.29
40.68 (10)
11.39
40.48 (11)
11.19
0.20
SSR+ND
29.29
42.83 (10)
13.54
42.68 (11)
13.39
0.15
DFN5B H/14-378
Appendix
Table S12: All six backbones, fine-tuned on ImageNet-1k and evaluated on the same-ontology shifts ImageNet-V2/-R/-Sketch; PT is the pretrained checkpoint evaluated directly on each listed evaluation dataset. Deploy is each run’s own (ImageNet-val) P1-selected checkpoint and peak is the maximum P3 over the evaluated trajectory on that dataset; best values are bolded independently within each backbone. Numbers in parentheses after P3 peak and P3 deploy give the ImageNet training milestone index (1–11), not an optimizer step count or epoch: milestones are 11 checkpoints at geometrically-spaced cumulative ImageNet-1k data fractions (0.25, 0.5, 1, 2, 5, 10, 20, 40, 60, 80, 100%), with PT as milestone 0. Because each later milestone spans a much larger increment of data than each earlier one, the milestone index is not proportional to the number of gradient updates or to wall-clock training progress. Deploy is each run’s own ImageNet-val P1-selected milestone, shared across all evaluation datasets for that run; peak is the milestone of the maximum P3 on that evaluation dataset, which can therefore differ from the deployment milestone and from peak timing on other datasets. When multiple milestones attain the same maximum P3, the earliest is reported.
Method
P3 PT
P3 peak (milestone)
Gain peak
P3 deploy (milestone)
Gain deploy
Forget post-peak
CUB-200 (cross-ontology held-out)
N
34.36
33.64 (1)
-0.72
20.73 (11)
-13.63
12.91
U
34.36
35.48 (1)
1.12
15.26 (11)
-19.11
20.23
SSR
34.36
34.76 (2)
0.40
23.35 (11)
-11.01
11.41
SSR+ND
34.36
34.88 (3)
0.52
24.99 (11)
-9.37
9.89
DFN5B H/14-378
Appendix
Table S13: All six backbones, fine-tuned on ImageNet-1k and evaluated cross-ontology on CUB-200, NAB-348 (strict dataset-class-disjoint held-out), and FineView Butterfly (FVB); PT is the pretrained checkpoint evaluated directly on each held-out dataset. Deploy is each run’s own P1-selected checkpoint (shared with Table S11 ) and peak is the maximum P3 over the trajectory; best peak and deployment values are bolded independently within each backbone (trajectory regimes in Table S16 ). Cross-ontology P3 acquisition is substantially smaller and more backbone-dependent than acquisition on ImageNet and its same-ontology shifts, and its peak often occurs earlier than the shared deployment milestone. Numbers in parentheses after P3 peak and P3 deploy give the ImageNet training milestone index (1–11), not an optimizer step count or epoch: milestones are 11 checkpoints at geometrically-spaced cumulative ImageNet-1k data fractions (0.25, 0.5, 1, 2, 5, 10, 20, 40, 60, 80, 100%), with PT as milestone 0. Because each later milestone spans a much larger increment of data than each earlier one, the milestone index is not proportional to the number of gradient updates or to wall-clock training progress. Deploy is each run’s own ImageNet-val P1-selected milestone, shared across all evaluation datasets for that run; peak is the milestone of the maximum P3 on that evaluation dataset, which can therefore differ from the deployment milestone and from peak timing on other datasets. When multiple milestones attain the same maximum P3, the earliest is reported.
Method
PT
Peak
Peak ep.
Deploy
Deploy ep.
Gain peak
Gain deploy
Forget post-peak
CUB-200 (target)
N
34.35
35.23
4
31.12
40
0.88
-3.23
4.11
U
34.45
38.87
12
35.78
25
4.42
1.33
3.09
SSR
34.35
40.42
20
40.42
20
6.08
6.08
0.00
SSR+ND
34.35
46.55
75
46.31
60
12.20
11.96
0.24
NAB-348 (strict cross-ontology held-out)
Appendix
Table S14: Representative DFN5B H/14-378 fine-tuned on CUB-200, with dense trajectory evaluation on CUB-200 (target) and NAB-348 (strict cross-ontology held-out); PT is the pretrained checkpoint evaluated directly on each dataset. Peak is the maximum P3 over the evaluated trajectory and deploy is each method’s own P1-selected checkpoint; best peak and deployment values are bolded independently within each dataset. All four fine-tuning conditions acquire held-out P3 above the pretrained state at some point, but their peak timing and post-peak retention differ substantially.
Method
Mean Gain peak (pp)
Median Gain peak (pp)
N positive
N > 0.5pp
ImageNet val (target)
N
3.63
3.14
6/6
6/6
U
1.47
1.81
5/6
5/6
SSR
7.51
6.84
6/6
6/6
SSR+ND
13.23
13.70
6/6
6/6
ImageNet-V2 (same-ontology shift)
Appendix
Table S15: Six-backbone descriptive summary of peak P3 gain over pretrained, fine-tuned on ImageNet-1k and evaluated on ImageNet val (target), its same-ontology shifts, and cross-ontology held-out data (CUB-200, NAB-348, FineView Butterfly); PT for each backbone is evaluated directly on the corresponding dataset. Means and medians are descriptive across the six deliberately diverse pretrained backbones, not stochastic-replicate estimates; best values within each distribution are bolded, with ties retained. Peak P3 gains are largest under structured supervision on ImageNet-family evaluations, whereas cross-ontology gains are markedly smaller.
Backbone
Method
CUB-200
NAB-348
FVB
DFN5B H/14-378
N
-0.72, no acquisition / degraded
-0.41, no acquisition / degraded
-0.10, no acquisition / degraded
U
1.12, acquire → lose
0.47, acquire → lose
0.23, acquire → lose
SSR
0.40, acquire → lose
0.87, acquire → lose
0.33, acquire → lose
SSR+ND
0.52, acquire → lose
1.23, acquire → lose
-0.14, no acquisition / degraded
DFN2B L/14
N
0.00, no acquisition / degraded
-0.01, no acquisition / degraded
0.17, acquire → retain
U
-0.31, no acquisition / degraded
-0.20, no acquisition / degraded
-0.23, no acquisition / degraded
Appendix
Table S16: All six ImageNet-1k-trained backbones, evaluated cross-ontology on CUB-200, NAB-348 (strict held-out), and FineView Butterfly (FVB); PT is the pretrained checkpoint evaluated directly on the relevant held-out dataset. Each cell reports peak P3 gain over pretrained (pp) and a trajectory regime describing whether that gain was retained, partially reduced, or fully lost by deployment, or never clearly acquired. Poor deployed transfer can arise either because a capability was never clearly acquired or because it was acquired earlier and subsequently reduced.
Backbone
Family
Params
Pretraining source
ImageNet-val P1 (%)
DFN5B H/14-378
DFN
986M
DFN5B (5B pool)
83.9
DFN2B L/14
DFN
427M
DFN-2B (2B pool)
81.8
MetaCLIP H/14
MetaCLIP
986M
MetaCLIP (fullcc)
80.3
MetaCLIP L/14
MetaCLIP
427M
MetaCLIP (fullcc)
79.0
OpenAI L/14
OpenAI
427M
OpenAI (WIT-400M)
73.3
OpenAI B/32
OpenAI
151M
OpenAI (WIT-400M)
63.1
Appendix
Table S17: The six pretrained backbones used throughout the ImageNet-1k experiments (Figure 2 b), listed before any downstream fine-tuning. The final column is pretrained P1 (class-name retrieval R@1) evaluated on ImageNet val; no downstream trajectory result is represented in this table.
Method
P1 PT
P1 deploy
P3 PT
P3 peak
Gain peak (P3)
ImageNet val (target)
N
83.90
87.82
33.12
33.78
0.66
U
83.90
87.66
33.12
35.38
2.26
SSR
83.90
87.00
33.12
45.34
12.22
SSR+ND
83.90
86.96
33.12
48.12
15.00
DFN5B H/14-378
Appendix
Table S18: All six backbones, fine-tuned on ImageNet-1k and evaluated on ImageNet val (target) and its same-ontology shifts; PT is the pretrained checkpoint evaluated directly on each listed dataset. P1 deploy is class-name retrieval at each run’s own P1-selected deployment checkpoint, and P3 peak is the maximum P3 over the trajectory; within each backbone, the highest deployed P1 and peak P3 are bolded independently. Large differences in P3 acquisition can occur while deployed P1 remains comparatively similar.
Method
P2 ord.
P2 name-sw.
P2 attr-sw.
Δname [95% CI]
Δattr [95% CI]
P3
CUB-200 (target)
PT
86.31
0.10
76.54
-86.2 [-88.6, -83.4]
-9.8 [-13.2, -6.6]
34.36
N
90.68
0.07
89.68
-90.6 [-92.1, -88.8]
-1.0 [-1.9, -0.2]
31.10
U
91.35
0.05
90.63
-91.3 [-92.8, -89.7]
-0.7 [-1.5, -0.1]
35.78
SSR
89.44
0.14
85.33
-89.3 [-91.3, -86.9]
-4.1 [-6.6, -2.0]
40.40
SSR+ND
90.04
0.22
86.92
-89.8 [-91.5, -87.8]
-3.1 [-4.7, -1.7]
46.31
Appendix
Table S20: DFN5B H/14-378 fine-tuned on CUB-200, evaluated on CUB-200 (target) and NAB-348 (strict cross-ontology held-out) under five conditions (PT, N, U, SSR, SSR+ND); PT is the pretrained checkpoint evaluated directly on each listed dataset, and all fine-tuned methods are at their own P1-selected deployment checkpoint. Ordinary, name-swapped, and attribute-swapped P2 (R@1 %) are shown with Δname / Δattr , the pp change from ordinary, with class-level bootstrap 95% CIs (1000 replicates). Name-swap nearly eliminates P2 retrieval, whereas attribute-swap preserves most performance, showing that ordinary name+attribute retrieval remains strongly class-name anchored.
SSR
SSR+ND
Backbone
Train
Gain peak
Gain deploy
Forget
Gain peak
Gain deploy
Forget
DFN5B H/14-378
ImageNet-trained
12.22
12.00
0.22
15.00
14.88
0.12
DFN5B H/14-378
CUB-trained
6.08
6.08
0.00
12.20
11.96
0.24
DFN5B H/14-378
NAB-trained
4.09
-0.23
4.32
5.65
1.02
4.62
DFN2B L/14
ImageNet-trained
5.30
4.96
0.34
8.10
8.10
0.00
DFN2B L/14
CUB-trained
3.78
3.69
0.09
8.99
8.99
0.00
Appendix
Table S21: All six backbones, fine-tuned separately on ImageNet-1k, CUB-200, and NABirds (18 backbone × training-domain pairs), each evaluated on its own target training domain (ImageNet-trained → ImageNet val, CUB-trained → CUB-200, NAB-trained → NAB-555) – this is a target-domain comparison, not a held-out-transfer table. Gain peak is target P3 peak minus target PT P3 (PT evaluated directly on the target dataset), Gain deploy uses each method’s own P1-selected deployment checkpoint, and Forget is their difference (underlies Figure 3 b; all values pp). SSR+ND exceeds SSR in peak target P3 acquisition in all 18 tested backbone × training-domain pairs.
Backbone
Train domain
PT
N
U
SSR
SSR+ND
DFN5B H/14-378
CUB-200
72.99
73.90
74.85
76.30
76.76
DFN5B H/14-378
NAB-555
65.92
67.56
67.52
65.65
65.79
DFN2B L/14
CUB-200
72.70
72.46
70.36
73.38
74.42
DFN2B L/14
NAB-555
64.23
63.35
64.08
63.67
64.67
MetaCLIP H/14
CUB-200
70.57
70.47
69.43
71.68
72.93
MetaCLIP H/14
NAB-555
63.14
60.88
63.48
65.63
66.76
Appendix
Table S22: All six backbones (DFN/MetaCLIP backbones plus the additional OpenAI L/14 and B/32 backbones, marked below) fine-tuned separately on CUB-200 and on NABirds, each evaluated on its own target training ontology (CUB-trained → CUB-200, NAB-trained → NAB-555); PT is the pretrained checkpoint evaluated directly on the target dataset. Pairwise class-slot-macro accuracy at each method’s own P1-selected deployment checkpoint; multi-way top-1 in Table S23 . Structured supervision consistently improves controlled local attribute discrimination on the CUB target ontology, whereas NAB-target effects are more backbone-dependent.
Backbone
Train domain
PT
N
U
SSR
SSR+ND
DFN5B H/14-378
CUB-200
49.13
49.44
50.15
52.97
53.53
DFN5B H/14-378
NAB-555
35.93
37.34
37.77
35.33
35.29
DFN2B L/14
CUB-200
46.32
46.24
43.19
47.37
49.70
DFN2B L/14
NAB-555
32.81
31.31
32.58
32.29
32.97
MetaCLIP H/14
CUB-200
45.26
44.85
41.64
46.19
47.89
MetaCLIP H/14
NAB-555
31.80
27.98
30.64
33.80
35.10
Appendix
Table S23: All six backbones (DFN/MetaCLIP backbones plus the additional OpenAI L/14 and B/32 backbones, marked below) fine-tuned separately on CUB-200 and on NABirds, each evaluated on its own target training ontology (CUB-trained → CUB-200, NAB-trained → NAB-555); PT is the pretrained checkpoint evaluated directly on the target dataset. Multi-way class-slot-macro top-1 accuracy at each method’s own P1-selected deployment checkpoint; pairwise version in Table S22 . Structured supervision consistently improves controlled local attribute discrimination on the CUB target ontology, whereas NAB-target effects are more backbone-dependent.
Backbone
Train → eval
Population
PT
N
U
SSR
SSR+ND
DFN5B H/14-378
CUB → NAB
NAB-555 (full)
65.95
67.23
68.05
67.66
66.56
DFN5B H/14-378
CUB → NAB
NAB-348 (strict)
65.63
67.05
68.22
66.85
65.92
DFN5B H/14-378
NAB → CUB
CUB-200 (full)
72.99
73.99
72.88
71.10
70.50
DFN5B H/14-378
NAB → CUB
CUB-59 (strict)
74.73
74.72
73.57
70.00
69.06
DFN2B L/14
CUB → NAB
NAB-555 (full)
64.24
64.07
63.56
64.01
63.97
DFN2B L/14
CUB → NAB
NAB-348 (strict)
63.70
63.23
62.86
63.44
63.09
Appendix
Table S24: All six backbones (DFN/MetaCLIP backbones plus the additional OpenAI L/14 and B/32 backbones, marked below), evaluated cross-ontology in both directions: CUB-trained → NAB (full held-out NAB-555, strict dataset-class-disjoint NAB-348) and NAB-trained → CUB (full held-out CUB-200, strict dataset-class-disjoint CUB-59); “full” and “strict” are evaluation populations, not different training datasets, and are never pooled. PT is the pretrained checkpoint evaluated directly on each held-out population, and all methods are at their own P1-selected deployment checkpoint (pairwise class-slot-macro accuracy). Held-out CAPP effects are backbone- and transfer-direction-dependent; structured target-domain gains (Tables S22 – S23 ) do not imply universal cross-ontology gains.
Figure S3: Name-free attribute-profile retrieval (P3) change versus Pairwise CAPP change across intervention conditions, each relative to the pretrained model, at every run’s own P1-selected deployment checkpoint. Left: DFN/MetaCLIP backbones; right: additional OpenAI L/14 and B/32 backbones (never pooled). Colors: N, U, SSR, SSR+ND; circles CUB-trained and triangles NABirds-trained; filled markers are the training-domain target and open markers the strict dataset-class-disjoint held-out set (NAB-348 for CUB-trained, CUB-59 for NABirds-trained; full held-out sets are not plotted). Across many intervention conditions, gains in name-free attribute-profile retrieval and local counterfactual discrimination tend to move in the same direction, although the probes measure distinct capabilities. No correlation coefficient is computed. Delta values use the pretrained result from the same evaluation job as each point and may therefore differ slightly from differences reconstructed from separately tabulated evaluations.
Variant
Eval.
P3 PT
P3 peak (ep.)
P3 deploy (ep.)
Gain peak
Gain deploy
Forget
FULL
CUB-200
34.36
36.16 (4)
34.17 (20)
1.79
-0.19
1.99
NAB-348
13.81
16.26 (12)
15.91 (20)
2.45
2.10
0.35
PARTIAL
CUB-200
34.45
37.80 (12)
36.97 (75)
3.35
2.52
0.83
NAB-348
13.87
17.01 (12)
13.38 (75)
3.14
-0.49
3.63
NO
CUB-200
34.35
40.42 (20)
40.42 (20)
6.08
6.08
0.00
NAB-348
13.85
17.32 (12)
16.90 (20)
3.48
3.06
0.42
Appendix
Table S25: Direct name-access variants (FULL / PARTIAL / NO), DFN5B H/14-378 fine-tuned on CUB-200, one run per variant (seed 0). Only the routing mask differs. FULL: the ATTRIBUTE route sees the full NAME segment; PARTIAL: only the NAME_END boundary signal; NO: no NAME access (identical to the canonical SSR run). EOT remains globally name-accessible in all three. Deployment is the CUB-P1-selected epoch (FULL 20, PARTIAL 75, NO 20), so PARTIAL’s deployment epoch differs from the others.
Variant
Eval.
Ep. 0 (PT)
Ep. 8
Ep. 20
Ep. 100
FULL
CUB-200
72.99
74.28
74.85
73.74
NAB-348
65.62
65.65
66.72
65.81
PARTIAL
CUB-200
72.97
75.14
75.59
74.32
NAB-348
65.62
65.94
67.05
66.07
NO
CUB-200
72.99
75.44
76.31
75.06
NAB-348
65.62
65.94
66.85
65.60
Appendix
Table S26: Pairwise CAPP (class-slot-macro accuracy, %) for the name-access variants at matched epochs. On the CUB target, CAPP follows NO > PARTIAL > FULL at epochs 8, 20, and 100 (differences of 0.3–1.5 pp); on NAB-348 the variants are nearly indistinguishable and the gain is transient.
Variant
Eval.
Dep. ep.
P2 ord.
P2 name-sw.
P2 attr-sw.
Name-sw. damage
Attr-sw. damage
FULL
CUB-200
20
89.90
0.05
86.52
89.85
3.38
NAB-348
20
79.66
0.30
69.82
79.35
9.83
PARTIAL
CUB-200
75
90.28
0.07
88.64
90.21
1.64
NAB-348
75
73.42
0.23
65.60
73.19
7.83
NO
CUB-200
20
89.51
0.12
85.42
89.39
4.09
NAB-348
20
78.97
0.33
68.39
78.64
10.59
Appendix
Table S27: Ordinary, name-swapped, and attribute-swapped P2 (R@1 %) at each variant’s deployment epoch. Name-swap collapses P2 to ≤ 0.33% in every variant, so restricting the ATTRIBUTE route’s name access does not remove the class-name dominance of ordinary P2.
Variant
Attr. end-mean
Name end-mean
Attr. pool-proj
Name pool-proj
FULL
0.39
0.84
0.40
0.83
PARTIAL
0.40
0.76
0.40
0.76
NO
0.65
0.35
0.68
0.36
Appendix
Table S28: Representational similarity (RSA) of the ATTRIBUTE-route representation at the last layer, epoch 20, to the pretrained name-free attribute-profile geometry and to class-name geometry, under two readouts (att_end_mean and att_pool_proj). Restricting direct NAME access raises similarity to the attribute-profile geometry and lowers similarity to class-name geometry under both readouts (main-text summary values use att_end_mean for the attribute profile and att_pool_proj for class name; both are shown here).
Figure S4: Direct name-access ablation trajectories, DFN5B H/14-378 fine-tuned on CUB-200 (one run per variant, single seed). Rows: P3 (name-free attribute-profile retrieval) and Pairwise CAPP; columns: CUB-200 (target) and strict held-out NAB-348. FULL, PARTIAL and NO direct name access denote how much of the NAME segment the ATTRIBUTE routes may read (Appendix A.5 ); NO is the canonical SSR run. Values are the change from each run’s own pretrained epoch-0 evaluation; stars mark each run’s P1-selected deployment epoch (FULL 20, PARTIAL 75, NO 20). P3 is evaluated at 11 epochs and CAPP at 7 (0, 4, 8, 12, 20, 75, 100), so the CAPP segment between epochs 20 and 75 is a straight line across unmeasured epochs. Restricting direct name access from the ATTRIBUTE route changes P3 and CAPP acquisition and trajectory behavior, supporting a role for representation and routing rather than content alone; NO is not name-free globally, because the EOT summary still reads NAME_END, and PARTIAL deploys later, so deployment-level comparisons are timing-confounded.
Figure S5: ImageNet-trained P1/P3 trajectories, DFN/MetaCLIP backbones (columns: DFN5B H/14-378, DFN2B L/14, MetaCLIP H/14, MetaCLIP L/14). For each evaluation population, the top row is P1 and the bottom row P3: ImageNet val (target), NAB-348 (strict dataset-class-disjoint held-out) and CUB-200 (held-out). The horizontal axis is the cumulative ImageNet training-data fraction at the 11 geometric milestones (not epochs). Deployment markers show the ImageNet-val-P1-selected checkpoint, which is shared by every population of a run; P3 uses the templates in Appendix B.3 (ImageNet: a object with {attributes} ; bird evaluations: a bird with {attributes} ).
Figure S6: ImageNet-trained P1/P3 trajectories, additional OpenAI backbones (OpenAI L/14 and B/32; the two families are never pooled). Layout, populations and definitions as in Figure S5 . The OpenAI deployment checkpoint is the ImageNet-val-P1 argmax over the 11 milestones; for L/14 it is the final milestone.
Figure S7: CUB-trained P1/P3 trajectories, DFN/MetaCLIP backbones. Rows: CUB-200 (target) and NAB-348 (strict dataset-class-disjoint held-out), each with P1 above P3. Horizontal axis: CUB training epoch. Deployment markers show each run’s CUB-P1-selected checkpoint. The DFN5B SSR+ND curve is the canonical SSR+ND run; dense fine-grained trajectories for the DFN/MetaCLIP backbones come from the per-epoch trajectory evaluations rather than a consolidated table, and the full NAB-555 held-out population has no P3 trajectory for the DFN/MetaCLIP backbones (it is available for OpenAI in Figure S9 ).
Figure S8: NABirds-trained P1/P3 trajectories, DFN/MetaCLIP backbones. Rows: NAB-555 (target) and CUB-59 (strict dataset-class-disjoint held-out), each with P1 above P3. Horizontal axis: NABirds training epoch (DFN5B N/U/SSR were evaluated to epoch 40). Deployment markers show each run’s NAB-P1-selected checkpoint. Definitions and single-run caveats as in Figure S7 .
Figure S9: OpenAI fine-grained P1/P3 trajectories (columns: OpenAI L/14 and B/32, CUB-trained and NABirds-trained). Rows: the training-domain target (CUB-200 or NAB-555), the full held-out population (NAB-555 or CUB-200; not dataset-class-disjoint) and the strict held-out population (NAB-348 or CUB-59; dataset-class-disjoint), each with P1 above P3; full and strict populations are never pooled. Deployment markers show each run’s target-domain-P1-selected epoch (argmax over the saved epochs after epoch 0). All four methods are single runs; OpenAI CUB-trained runs use weight decay 0.2 for all four methods.
Figure S10: P3 peak (diamond), Pairwise-CAPP peak (open triangle), and P1-selected deployment epoch (tick) for each method and evaluation set, DFN5B H/14-378 fine-tuned on CUB-200 (single run per method). The three need not coincide.
Method
Eval.
P3 peak (ep.)
P3 deploy (ep.)
CAPP peak (ep.)
CAPP deploy (ep.)
N
CUB-200 (target)
4
40
15
40
N
NAB-348 (held-out)
12
40
20
40
U
CUB-200 (target)
12
25
15
25
U
NAB-348 (held-out)
10
25
15
25
SSR
CUB-200 (target)
20
20
12
20
SSR
NAB-348 (held-out)
12
20
12
20
Appendix
Table S29: Dense P3 and Pairwise-CAPP peak/deployment epochs, DFN5B H/14-378 fine-tuned on CUB-200, evaluated on CUB-200 (target) and NAB-348 (strict held-out); underlies Figure S10 . Deployment epoch is each method’s own P1-selected checkpoint, shared by construction between the P3 and CAPP columns. CAPP is Pairwise class-slot-macro accuracy. Single run per method.
Population
Method
Cells
Before
At
After
target
N
6
2
4
0
U
6
5
1
0
SSR
6
3
3
0
SSR+ND
6
1
5
0
all
24
11
13
0
same-ontology
N
18
6
12
0
Appendix
Table S30: Timing of each evaluation distribution’s own P3-peak milestone relative to the run’s P1-selected deployment milestone, by method (six ImageNet-trained backbones). Target = ImageNet val (6 cells per method); same-ontology = ImageNet-V2/-R/-Sketch (18); cross-ontology = CUB-200, NAB-348, FineView Butterfly (18). Milestones are cumulative-data indices and are compared only categorically. Panel b of Figure 4 plots “before” for target and cross-ontology. Six heterogeneous backbones, not replicates.
Population
Method
Valid n
Median (pp)
IQR
>0.5 pp
target
N
2
0.34
[0.32, 0.36]
0
U
5
3.32
[1.00, 5.56]
4
SSR
3
0.34
[0.28, 1.00]
1
SSR+ND
1
0.12
[0.12, 0.12]
0
all
11
0.38
[0.31, 2.49]
5
cross-ontology
N
16
1.40
[0.62, 2.01]
14
Appendix
Table S31: Peak-to-deployment P3 decrease, P3peak−P3deploy (absolute pp), over cells whose own P3 peak precedes deployment ( tP3<tdeploy ); cells with peak at or after deployment have no evaluated post-peak interval and are excluded. Only 11 target cells qualify (n = 1–5 per method), too few for stable per-method target summaries, so Figure 4 c shows cross-ontology cells only. The >0.5 pp count is a descriptive robustness count, not a significance test. The by-dataset rows are cross-ontology cells pooled over methods. OpenAI L/14 target and same-ontology cells all have the peak at the final milestone (Table S33 ), so absence of observed decline there is bounded by the evaluated schedule.
Backbone
T-bef
T-at
T-aft
S-bef
S-at
S-aft
X-bef
X-at
X-aft
DFN5B H/14-378
3
1
0
10
2
0
12
0
0
DFN2B L/14
2
2
0
6
6
0
9
3
0
MetaCLIP H/14
1
3
0
6
6
0
11
1
0
MetaCLIP L/14
3
1
0
7
5
0
11
1
0
OpenAI L/14
0
4
0
0
12
0
11
1
0
OpenAI B/32
2
2
0
11
1
0
11
1
0
Appendix
Table S32: Per-backbone counts of evaluation cells (over the four methods N/U/SSR/SSR+ND and the datasets in each population) whose P3 peak milestone falls before, at, or after the run’s P1-selected deployment milestone. T = ImageNet val target (4 cells per backbone), S = same-ontology shifts ImageNet-V2/-R/-Sketch (12 cells), X = cross-ontology CUB-200/NAB-348/FVB (12 cells). Classification compares peak and deployment milestone indices from Tables S11 – S13 ; milestones are cumulative-data indices, not epochs, and are never pooled as absolute values across schedules. Column sums over backbones give the population totals in Table S30 .
Dataset
Method
Peak ms.
Deploy ms.
Gain peak
Forget
ImageNet val
N
11
11
5.24
0.00
U
11
11
2.28
0.00
SSR
11
11
9.66
0.00
SSR+ND
11
11
19.88
0.00
ImageNet-V2
N
11
11
4.74
0.00
U
11
11
1.89
0.00
Appendix
Table S33: OpenAI L/14 (ImageNet-trained): peak and deployment milestone indices for the ImageNet target and same-ontology shifts. In every one of these 16 cells the P1-selected deployment checkpoint is the final evaluated milestone (11) and coincides with the P3 maximum, so no post-peak interval exists in the evaluated trajectory; this is a statement about the audited schedule, not evidence that specialization cannot occur for this backbone. Cross-ontology cells (Table S13 ) do show peaks before deployment.
Training domain
Cells
tP3<tdeploy
tP3=tdeploy
tP3>tdeploy
CUB-trained
held-out
23
0
1
CUB-trained
target
11
1
0
ImageNet-trained
held-out
33
3
0
ImageNet-trained
target
10
2
0
NABirds-trained
held-out
22
1
1
NABirds-trained
target
8
3
1
Appendix
Table S34: Temporal order of each evaluation distribution’s own P3-peak checkpoint and the P1-selected deployment checkpoint over the 120 N/U/SSR cells (4 backbones × 3 training domains × 3 methods × evaluation sets). Only tP3<tdeploy cells are temporally valid “post-peak” trajectories.
Pre-peak vs. Gainpeak
Post-peak vs. Forgetpostpeak
Subset
n
Image
Text
Joint
n
Image
Text
Joint
Held-out pooled (primary)
84
− 0.17/+0.33
− 0.14/+0.35
− 0.16/+0.32
78
+0.43/+0.51
+0.54/+0.64
+0.50/+0.61
CUB-trained
24
+0.49/+0.58
+0.63/+0.69
+0.58/+0.63
23
+0.42/+0.49
+0.68/+0.72
+0.58/+0.59
NABirds-trained
24
− 0.72/ − 0.65
− 0.73/ − 0.70
− 0.73/ − 0.72
22
+0.64/+0.64
+0.56/+0.79
+0.64/+0.75
ImageNet-trained
36
+0.33/+0.57
+0.35/+0.57
+0.34/+0.53
33
+0.10/ − 0.02
+0.56/+0.54
+0.39/+0.43
N
28
− 0.25/+0.31
− 0.23/+0.35
− 0.24/+0.32
26
+0.45/+0.54
+0.40/+0.58
+0.44/+0.60
Appendix
Table S35: Encoder-separated and joint movement analysis (Pearson r / Spearman ρ ), held-out cells of the N/U/SSR grid; post-peak columns use tP3<tdeploy cells. Image movement: mean over image-encoder blocks ( visual.transformer.resblocks.* ) of ∥ΔW∥F/∥WPT∥F . Text movement: the same quantity over the shared pretrained text transformer ( transformer.resblocks.* ; 24 blocks for DFN5B/MetaCLIP H/14, 12 for DFN2B/MetaCLIP L/14). Joint = (image + text)/2, the “pretrained-backbone movement” used in Figure 5 a. Excluded because they lack a pretrained analogue or are not comparable: AIP and StructuredTextFusion parameters, the newly appended SSR token-embedding rows (4–6), positional embeddings, and the logit scale. Descriptive associations across heterogeneous backbones; not replicates.
Pearson r
Partial r
Std. coef.
Subset / normalization
n
Img– F
Txt– F
Img–Txt
Img ∣ Txt
Txt ∣ Img
Img
Txt
R2
VIF
Pooled (raw)
78
+0.43
+0.54
+0.85
− 0.06
+0.36
− 0.10
+0.62
0.29
3.5
CUB-trained
23
+0.42
+0.68
+0.72
− 0.14
+0.60
− 0.15
+0.79
0.48
2.1
NABirds-trained
22
+0.64
+0.56
+0.76
+0.40
+0.14
+0.51
+0.17
0.42
2.4
ImageNet-trained
33
+0.10
+0.56
+0.45
− 0.22
+0.59
− 0.20
+0.66
0.35
1.3
Domain-standardized
78
+0.34
+0.60
+0.62
− 0.04
+0.52
− 0.04
+0.62
0.36
1.6
Appendix
Table S36: Post-peak image versus text movement, held-out tP3<tdeploy cells (n = 78 pooled). F = Forgetpostpeak ; Img–Txt is the image/text movement correlation; Partial r controls the other encoder; Std. coef. are from the descriptive regression F∼z(Dimage)+z(Dtext) . Image and text movement are strongly coupled, and the apparent relative importance of the encoders depends on normalization (text dominates pooled and domain-standardized, image and text are indistinguishable within backbone; VIF up to about 11), so broad encoder attribution is not identifiable. Descriptive only; not a population-level regression or causal claim.
Pre-peak: Dpre vs. Gainpeak
Post-peak: Dpost vs. Forgetpostpeak
Subset
n
r
ρ
n
r
ρ
Held-out, all (primary)
84
− 0.17
+0.33
78
+0.43
+0.51
All valid cells
108
− 0.15
+0.28
107
+0.44
+0.52
CUB-trained (held-out)
24
+0.49
+0.58
23
+0.42
+0.49
NABirds-trained (held-out)
24
− 0.72
− 0.65
22
+0.64
+0.64
ImageNet-trained (held-out)
36
+0.33
+0.57
33
+0.10
− 0.02
Appendix
Table S37: Direct image-encoder movement versus P3 change across the N/U/SSR trajectory grid (Pearson r , Spearman ρ ; absolute percentage points). Dpre and Dpost are the layer-averaged Frobenius displacements ∥Wpeak−WPT∥/∥WPT∥ and ∥Wdeploy−Wpeak∥/∥WPT∥ over all image-encoder transformer-block parameters (attention, MLP, LayerNorm; projection, embeddings, text encoder and SSR modules excluded); net checkpoint displacement, not SGD path length. Post-peak rows use held-out cells with tP3<tdeploy unless stated; pre-peak rows need a pretrained P3 baseline. Backbones are heterogeneous models, not replicates, and all values are descriptive associations.
Figure S11: Backbone movement versus held-out P1 decrease, N/U/SSR grid, same 78 held-out cells and same joint image–text backbone movement as the right panel of Figure 5 a. P1 is evaluated at each trajectory’s P3-peak checkpoint and at its deployment checkpoint; no separate P1-optimal checkpoint is selected, and P1 decrease = P1 (P3-peak checkpoint) − P1 (deployment). This is a robustness analysis; held-out P1 is not treated as a name-free semantic-capability measure.
Probe, level
Rank
Wrong sim. peak → deploy
Δ corr.
Δ wrong
Encroachment [95% CI]
% cls. ↓ margin
Split-half [95% CI]
P3 hardest
1
0.203 → 0.211
+0.0044
+0.0080
+0.0036 [+0.0019, +0.0052]
59.5
+0.0034 [+0.0014, +0.0053]
P3 hard
35
0.090 → 0.098
+0.0044
+0.0084
+0.0039 [+0.0023, +0.0055]
62.6
+0.0047 [+0.0024, +0.0070]
P3 medium
174
− 0.007 → 0.005
+0.0044
+0.0120
+0.0076 [+0.0057, +0.0094]
67.0
+0.0072 [+0.0049, +0.0095]
P3 easy
312
− 0.096 →− 0.079
+0.0044
+0.0168
+0.0124 [+0.0103, +0.0144]
74.1
+0.0116 [+0.0088, +0.0141]
P1 hardest
1
0.356 → 0.399
+0.0408
+0.0430
+0.0022 [+0.0006, +0.0038]
56.0
+0.0021 [+0.0004, +0.0039]
P1 hard
35
0.089 → 0.127
+0.0408
+0.0375
− 0.0033 [ − 0.0054, − 0.0010]
42.0
− 0.0044 [ − 0.0074, − 0.0013]
Appendix
Table S38: Peak-ranked wrong-class levels for the SSR+ND DFN5B H/14-378 CUB → NAB-348 run. For every image the 347 wrong classes are ranked by similarity at the peak (epoch 20) only (rank 1 = most similar); the class at rank 1, 35 (about the 90th similarity percentile), 174 (median) and 312 (about the 10th percentile) is held fixed and reevaluated at deployment (epoch 60). Encroachment =Δswrong−Δscorrect=−Δ margin, with margin = correct − wrong cosine similarity; class-macro means over the 348 NAB-348 classes, paired class bootstrap (10,000 resamples). The split-half column ranks wrong classes from half of each class’s images and evaluates on the other half. P1 and P3 are separate capabilities and are not pooled.
State
Image encoder
Text/routing encoder
CUB P1
CUB P3
NAB-348 P1
NAB-348 P3
peak
ep. 20
ep. 20
90.02
45.56
76.10
17.63
image late only
ep. 60
ep. 20
89.97
45.20
71.23
15.15
text late only
ep. 20
ep. 60
89.97
47.07
76.07
17.09
deploy
ep. 60
ep. 60
90.23
46.29
70.81
14.50
Appendix
Table S39: Encoder-state swap, DFN5B H/14-378 SSR+ND fine-tuned on CUB-200 (single run). Image encoder and text/routing encoder are taken from epoch 20 (peak) or epoch 60 (deployment); “image late only” combines the epoch-60 image encoder with the epoch-20 text/routing encoder, “text late only” the reverse. Values are R@1 (%).
Eval.
Probe
Effect
Change (pp) [95% CI]
CUB-200
P3
image-late
− 0.44 [ − 1.84, +0.89]
CUB-200
P3
text/SSR-late
+1.45 [+0.55, +2.37]
CUB-200
P3
total
+0.59 [ − 1.06, +2.18]
CUB-200
P3
interaction
− 0.42 [ − 1.15, +0.29]
CUB-200
P3
image-late − text/SSR-late
− 1.90 [ − 3.36, − 0.44]
NAB-348
P3
image-late
− 2.53 [ − 3.53, − 1.56]
Appendix
Table S40: Paired class-bootstrap decomposition of the encoder-state swap (Table S39 ); representative DFN5B H/14-378 SSR+ND CUB-trained run. “Image-late” is the effect of substituting the deployment image encoder at fixed peak text/routing state, “text/SSR-late” the reverse, “interaction” the residual of the additive decomposition. Class bootstrap over evaluation classes (10,000 resamples).
Class-macro R@1 (%)
Effect (pp) [95% CI]
Evaluation
P I +P T
D I +P T
P I +D T
D I +D T
Image-late
Text/SSR-late
Image − text
Interaction
MetaCLIP H/14, CUB → NAB-348 (ep. 12/75)
held-out, NAB-348 P3
12.39
11.66
12.46
11.64
− 0.73 [ − 1.54, +0.08]
+0.06 [ − 0.38, +0.50]
− 0.80 [ − 1.68, +0.10]
− 0.08 [ − 0.52, +0.34]
held-out, NAB-348 P1
72.12
69.67
72.20
70.37
− 2.44 [ − 3.62, − 1.30]
+0.08 [ − 0.58, +0.74]
− 2.52 [ − 3.82, − 1.25]
+0.61 [+0.12, +1.11]
target, CUB-200 P3
38.35
40.35
39.33
42.01
+2.00 [+0.51, +3.53]
+0.98 [ − 0.01, +1.95]
+1.02 [ − 0.32, +2.35]
+0.68 [ − 0.26, +1.69]
target, CUB-200 P1
83.30
85.87
85.16
87.95
+2.56 [+1.40, +3.77]
+1.85 [+0.66, +3.13]
+0.71 [ − 0.90, +2.33]
+0.23 [ − 0.65, +1.14]
Appendix
Table S41: Additional encoder-state swaps on three additional SSR+ND runs, selected by a rule fixed before any swap was run (Appendix G.5 ). States: P = P3-peak checkpoint, D = deployment checkpoint, I = image encoder, T = text/SSR state (checkpoint epochs peak/deploy in the run label); effects use the same definitions as Table S40 (image-late = D I +P T − P I +P T ; text/SSR-late = P I +D T − P I +P T ). Paired class bootstrap over evaluation classes (10,000 resamples); NABirds-trained runs are evaluated on the dataset-class-disjoint CUB-59 subset (59 classes), target rows on the run’s own training-domain test set. Independent existing runs, not stochastic replicates.
State
P1 correct
P1 wrong
P1 margin
P3 correct
P3 wrong
P3 margin
CUB-200 ( n=200 )
peak
0.4533
0.3183
0.1349
0.2157
0.2280
-0.0123
image late only
0.5089
0.3453
0.1636
0.2352
0.2485
-0.0134
text late only
0.4567
0.3064
0.1504
0.2117
0.2201
-0.0084
deploy
0.5232
0.3366
0.1866
0.2327
0.2416
-0.0089
NAB-348 ( n=348 )
Appendix
Table S42: Cross-state embedding recombination, SSR+ND (DFN5B H/14-378, CUB-trained; single run). For each class, “correct” is the image-to-text cosine similarity to the correct class text anchor, “wrong” is the similarity to the hardest competing class anchor, and margin is correct minus wrong; entries are class-macro averages over the stated number of classes, computed for P1 and P3 text anchors. States follow Table S39 . Figure 5 c plots the NAB-348 margins as change relative to the peak state.
Model
Eval.
Anchor
PT corr.
PT wrong
Peak corr.
Peak wrong
Dep. corr.
Dep. wrong
SSR+ND
CUB-200
P3
0.3100
0.3217
0.2157
0.2280
0.2327
0.2416
CUB-200
P1
0.3659
0.3290
0.4533
0.3183
0.5232
0.3366
NAB-348
P3
0.2796
0.3082
0.1383
0.2028
0.1427
0.2197
NAB-348
P1
0.3669
0.3425
0.4077
0.3559
0.4484
0.4038
U (text-frozen)
CUB-200
P3
0.3100
0.3217
0.1993
0.2109
0.2648
0.2462
CUB-200
P1
0.3659
0.3290
0.4017
0.3157
0.4669
0.3386
Appendix
Table S43: Correct-class versus hardest-competitor text-anchor similarity (class-macro means) at pretrained (PT), peak, and deployment checkpoints for SSR+ND (peak epoch 20, deployment epoch 60) and an independent text-frozen U run (peak epoch 20, deployment epoch 75; a separate run from the main U run, whose deployment epoch is 25). In the text-frozen run the entire text encoder including layer norms is locked, and the text anchors were verified to be bit-identical across PT, peak, and deployment, so all change is image-side.
Figure S12: Supplementary late-specialization diagnostics. (a) Correct-class versus hardest-wrong text-anchor similarity, peak to deployment, where the hardest wrong class is re-selected at each checkpoint (see Table S44 for why this statistic can overstate encroachment). (b) Change in NAB-348 correct-minus-competitor margin from peak under cross-state recombination (I = image encoder, T = text/routing encoder).
Model
Eval.
Probe
Competitor
Encroachment ( ×10−3 ) [95% CI]
Classes >0
SSR+ND (joint)
CUB-200
P1
hardest wrong per checkpoint
−51.66 [-55.49, -47.88]
2%
SSR+ND (joint)
CUB-200
P1
fixed at peak (per image)
−63.80 [-67.61, -60.04]
1%
SSR+ND (joint)
CUB-200
P1
fixed at peak (per class)
−71.66 [-75.93, -67.35]
2%
SSR+ND (joint)
CUB-200
P3
hardest wrong per checkpoint
−3.41 [-6.03, -0.85]
40%
SSR+ND (joint)
CUB-200
P3
fixed at peak (per image)
−11.84 [-14.41, -9.35]
22%
SSR+ND (joint)
CUB-200
P3
fixed at peak (per class)
−11.34 [-14.31, -8.40]
28%
Appendix
Table S44: Peak → deployment competitor encroachment, Δswrong−Δscorrect (class-macro mean of image–text-anchor cosine changes; positive = the competitor closes the gap relative to the correct anchor). “Hardest wrong per checkpoint” re-selects the maximizing wrong class independently at peak and at deployment and can rise merely because wrong-class similarities spread or the maximizer changes; the fixed-competitor rows freeze each image’s (or class’s) hardest wrong class at the peak checkpoint and re-evaluate it at deployment. Intervals: paired class bootstrap over evaluation classes (10,000 resamples), not training-seed replication. Peak epoch 20; deployment epoch 60 (SSR+ND) or 75 (U, text encoder frozen; text anchors bit-identical across checkpoints). Single runs, DFN5B H/14-378 fine-tuned on CUB-200.
Model
Eval.
Probe
Competitor
Spearman ρ [95% CI]
Mean Δ acc. (pp)
SSR+ND (joint)
CUB-200
P1
hardest wrong per checkpoint
− 0.09 [ − 0.24, +0.06]
+0.2
SSR+ND (joint)
CUB-200
P1
fixed at peak (per image)
− 0.09 [ − 0.24, +0.06]
+0.2
SSR+ND (joint)
CUB-200
P3
hardest wrong per checkpoint
− 0.60 [ − 0.69, − 0.51]
+0.6
SSR+ND (joint)
CUB-200
P3
fixed at peak (per image)
− 0.57 [ − 0.65, − 0.47]
+0.6
SSR+ND (joint)
NAB-348
P1
hardest wrong per checkpoint
− 0.57 [ − 0.63, − 0.49]
− 5.3
SSR+ND (joint)
NAB-348
P1
fixed at peak (per image)
− 0.58 [ − 0.64, − 0.50]
− 5.3
Appendix
Table S45: Class-level Spearman correlation between encroachment and the peak → deployment change in class top-1 accuracy (class bootstrap over evaluation classes). The relationship is partly mechanical, because encroachment and top-1 retrieval derive from the same similarity scores; it is supportive consistency, not independent mechanistic evidence.
Model
Eval.
PT
Peak
Deploy
SSR+ND
CUB-200
89.2
90.4
90.6
NAB-348
90.9
90.6
88.4
U (text-frozen)
CUB-200
89.2
90.3
90.6
NAB-348
90.8
91.2
88.0
Appendix
Table S46: Image-only leave-one-out prototype accuracy (class-macro, %). For each image, the class prototype is the mean image embedding of the other images of the same class; the image is assigned to the nearest prototype. This is an oracle diagnostic that uses held-out-image prototypes; it is not a zero-shot capability. Class structure remains largely preserved (88.0–91.2%), although NAB-348 shows a 2.7–3.1 pp decline from peak to deployment.
Geometry metric
Target
SSR+ND CUB
SSR+ND NAB
U-tf CUB
U-tf NAB
Tangent-residual alignment (P3)
Δ P3
0.33*
0.25*
0.15*
0.21*
Δ P1
0.05
-0.09
-0.04
0.00
Tangent-residual alignment (P2)
Δ P3
0.05
0.11*
0.14
0.19*
Δ P1
0.01
0.05
0.03
0.10
Raw alignment (P3)
Δ P3
0.23*
0.19*
0.12
0.19*
Δ P1
0.02
-0.15*
-0.06
-0.08
Appendix
Table S47: Class-level Spearman correlations between late-minus-peak changes in coarse geometry metrics and changes in class accuracy (n = 200 CUB classes, 348 NAB classes; 10,000 class resamples for the 95% intervals). Asterisks mark intervals excluding 0. Tangent-residual alignment vs. Δ P3 is weak but non-null in all four cells ( ρ∈[0.15,0.33] ); residual-direction coherence, manifold radius, and effective rank show no reliable relationship with Δ P3, although some show non-null relationships with Δ P1. The two models are SSR+ND and an independent text-frozen U run (U-tf). Full intervals are provided in the supporting results.
Method
α
CUB P1
CUB P3
NAB-348 P1
NAB-348 P3
N
0
88.83
34.35
82.93
13.85
0.25
90.87
36.61
83.60
14.60
0.5
91.46
35.95
82.25
15.06
0.75
91.39
33.62
78.06
15.00
1
91.06
31.12
71.55
13.74
U
0
88.75
34.45
83.03
13.87
Appendix
Table S48: WiSE-style weight interpolation θ(α)=(1−α)θPT+αθFT for N (deployment epoch 40) and U (deployment epoch 25), DFN5B H/14-378 fine-tuned on CUB-200; α=0 is the pretrained model and α=1 the deployed checkpoint. Interpolation is applied at evaluation time to all trainable parameters; no training is involved. Values are R@1 (%) on CUB-200 (target) and NAB-348 (strict held-out). Single seed, coarse α grid, one backbone and training domain.
Figure S13: Hard-negative semantic-discrimination control, DFN5B H/14-378 fine-tuned on CUB-200 (single seed, λ=0.5 ). Starting from SSR, a target-side ranking loss separates the correct attribute profile from a controlled hard negative differing in one attribute value. (a) Logit-scaled similarities of the image to the positive ( spos ) and hard-negative ( sneg ) profiles and their margin, per training epoch (training-log quantities on the hard-negative batches, not an evaluation probe): sneg first rises (until about epoch 19) and then declines while spos keeps increasing, so the relative margin grows. (b) Hard-negative ranking accuracy (fraction with spos>sneg ) first reaches 99.9% at epoch 41 (dotted line) and stays there, while the hard-negative text loss keeps decreasing. (c) P3 (name-free attribute-profile retrieval) on CUB-200 (target) and strict held-out NAB-348 at the 11 evaluated checkpoints (no interpolation): the hard-negative run exceeds the matched SSR baseline on target CUB P3 at later checkpoints but not on held-out NAB-348 P3 at matched epochs. This representative control shows that stronger target-side semantic discrimination need not translate into stronger transferable P3.
Epoch
CUB P3
NAB-348 P3
CUB P1
CUB align.
NAB align.
8
37.87 / 38.02
15.91 / 16.18
88.28 / 88.44
0.463 / 0.460
0.263 / 0.262
12
40.25 / 39.87
16.76 / 17.32
88.99 / 89.02
0.469 / 0.457
0.287 / 0.283
20
41.32 / 40.42
15.57 / 16.90
90.06 / 90.30
0.454 / 0.412
0.281 / 0.268
40
43.04 / 39.82
13.71 / 14.36
89.75 / 89.94
0.410 / 0.352
0.254 / 0.244
100
41.27 / 40.01
12.38 / 13.30
90.02 / 90.25
0.355 / 0.315
0.231 / 0.228
Appendix
Table S49: Hard-negative control versus the matched SSR baseline, DFN5B H/14-378, CUB-trained (single seed, single λ ). Each cell is hard-negative / SSR. The objective adds λLHN with λ=0.5 to the multi-positive contrastive loss, where LHN=meansoftplus(sneg−spos) is computed on the final fused text representation, and each negative differs from the true profile by one controlled attribute edit (pool of 618 candidates covering all 200 classes). Alignment is representational similarity between image and P3 text geometry. At each run’s own P1-selected checkpoint (hard-negative epoch 20, SSR epoch 75) CUB P3 is 41.32 vs. 39.97 and NAB-348 P3 is 15.57 vs. 13.34; at epoch 100, NAB-348 P3 is 12.38 vs. 13.30. Peak NAB-348 P3 is 16.76 (hard-negative) vs. 17.35 (SSR).
Item
CUB: Kentucky Warbler
NAB-348: adult male Lesser Goldfinch
P1
a photo of a Kentucky Warbler
a photo of a adult male Lesser Goldfinch
P3 template
a bird with {attributes}
a bird with {attributes}
CAPP correct profile
a bird with black crown, bright yellow underparts, olive-green upperparts
a bird with bright yellow belly, black cap, glossy cap, pointed wing, white wing patch
CAPP one-slot counterfactual
a bird with black crown, gray underparts , olive-green upperparts (slot: underparts color)
a bird with bright yellow belly, black cap, glossy cap, rounded wing , white wing patch (slot: wing shape)
Item
FineView: Monarch
ImageNet: Goldfish
P3 template
a butterfly with {attributes}
a object with {attributes}
Appendix
Table S50: Verified probe examples. The CAPP pairs are drawn from the reviewed full-class CAPP profiles (donor values for the swapped slot come from other classes). Only the changed slot is highlighted. The Figure 1 c CAPP panel is an illustrative profile, not one of these verified pairs.
Main item
Supplement table(s)
Population
Checkpoint rule
Evidence type
Fig. 2 b
Tables S11 – S15
ImageNet target, same-ontology shifts, cross-ontology; six backbones
trajectory peak
six-backbone descriptive
Fig. 2 a
Table S14
CUB target, NAB-348 held-out; DFN5B
peak, own-P1 deploy
single run per method
Table 4.2
Table S19
ImageNet val; six backbones
own-P1 deploy
six-backbone descriptive
Fig. 3 a
Table S20
CUB target, NAB-348; DFN5B
own-P1 deploy
single run, 1000-replicate class bootstrap
Fig. 3 b
Table S21
3 training domains × six backbones (18 pairs)
trajectory peak
six-backbone descriptive
Table 4.2
Tables S22 – S24
CUB/NAB target; six backbones
own-P1 deploy
six-backbone descriptive
Appendix
Table S51: Main-paper items mapped to supplement tables, evaluation populations, checkpoint rules, and evidence type. Evidence types are distinct: single training runs, six heterogeneous backbones (descriptive), class-level bootstrap resampling of the evaluation set, and archived historical multi-seed checks (Table S7 ); they must not be conflated.
Fine-tuning a pretrained LLM into a vision-language model (VLM) can erode the backbone's text capability, with the damage concentrated on tasks that require following exact output rules, such as instruction following, chain-of-thought reasoning graded on a strictly parsed final answer, and similar evaluations with strict graders. We trace this gap to attention-sink corruption: VL fine-tuning perturbs the early sink position that anchors a large fraction of attention probability, and how well the base LLM preserves its sink tracks how much of the affected capability survives adaptation. Building on this view, we introduce Sink Strength, a single scalar computed on the base LLM in a few seconds on a single GPU that predicts post-VL degradation without any VL training. It consistently tracks relative degradation across the six VLM-LLM pairs and multiple format-sensitive tasks. Complementing this diagnostic, we find that post-pretraining QK-RMSNorm injection fails to reproduce the protection of native QK-RMSNorm, while several off-the-shelf weight-merging settings fail to recover the lost capability after VL training. These negative results underscore the value of screening backbones with Sink Strength before VL training and narrow the intervention space toward head-selective training-time protection.
Recent advances in vision-language models (VLMs) emphasize long chain-of-thought reasoning; yet, we find that their performance on visual tasks is primarily limited by a lack of visual perception as opposed to reasoning itself. In this work, we systematically study the interplay between perception and reasoning in VLM post-training by decomposing their capabilities into three separate training stages: visual perception, visual reasoning, and textual reasoning, incorporating specialized training data. We demonstrate that visual perception (a) requires targeted optimization with specialized data; (b) serves as a fundamental scaffold that should be solidified through staged training before refining visual reasoning; and (c) is more effectively learned via RL than caption-based SFT. Our experiments across multiple VLMs demonstrate that staged training consistently improves both visual perception and reasoning performance over merged training. Notably, models trained with our approach achieve 1.5% higher reasoning accuracy with 20.8% shorter reasoning traces, suggesting that superior perception reduces the need for excessive reasoning. Furthermore, we show that this capability-based staging represents a new curriculum dimension orthogonal to traditional difficulty-based curricula, and combining both yields further additive gains. Our staged-training models achieve superior performance among open-weight VLMs, establishing advanced results on several visual math and perception (e.g., +5.2% on WeMath and +3.7% on RealWorldQA) tasks compared with the base counterpart.
Juncheng Wu, Hardy Chen, Haoqin Tu +6
Amazon · UC Santa Cruz · University of Waterloo +1
Vision-language models (VLMs) answer visual questions by combining visual information extraction with downstream problem solving. We investigate a fundamental question: Does an incorrect answer necessarily reflect a failure in visual extraction or problem solving? A model may succeed at both abilities when tested separately yet still fail on the original multimodal question, a distinction that overall answer accuracy cannot reveal. To study this, we perform a question-level empirical analysis across multiple VLMs and visual domains. We define an exactly scorable task state (i.e., the visual information sufficient to solve a question) and use it to test whether the same model can extract the required state, solve the question from the ground-truth state, and answer the original multimodal question. We find that composition failures, where extraction and solving both succeed but direct answering fails, account for 17.7% to 75.6% of direct-answering errors across multiple VLMs and datasets. To address this failure mode, we introduce a simple yet effective method, termed State Realization Tuning (SRT). SRT fine-tunes LoRA adapters attached to the language-model layers while keeping the pretrained VLM weights frozen. It trains the model to output the ground-truth task state before the final answer in a single autoregressive response. SRT improves over standard supervised fine-tuning by 1.7 to 14.1 percentage points and repairs 92.5% to 98.1% of diagnosed composition failures. A single LoRA adapter trained with SRT also improves performance across substantially different task-state structures. Our work shows that having both visual extraction and problem-solving capabilities does not guarantee correct multimodal answering. Requiring the model to first output the visual information needed to solve the question can help bridge this gap.
Ziheng Wang, Mingxuan Xie, Yilin Liu +3
Sun Yat-sen University · Zhejiang University · The Hong Kong University of Science and Technology +2