When does fusing hand-crafted knowledge with learned representations pay? A cost-normalized benchmark of stacking, substitution, and interference

Ahmad AlMughrabi; Albert Clop; Benjamin Busam; Ricardo Marques; Petia Radeva

cs.CVAug 21, 2026

When does fusing hand-crafted knowledge with learned representations pay? A cost-normalized benchmark of stacking, substitution, and interference

Authors: Ahmad AlMughrabi, Albert Clop, Benjamin Busam, Ricardo Marques, Petia Radeva

Organizations: Universitat de Barcelona, Barcelona, Spain · Technical University of Munich, Munich, Germany · Universitat Pompeu Fabra, Barcelona, Spain

Abstract

Fusing prior knowledge with data-driven learning is attractive where data is scarce, yet no controlled account says when it helps, is redundant, or harms. We benchmark one fixed hand-crafted knowledge source, a pinned bank of Gabor targets injected only during training at $\sim$ 2% overhead, against data-driven alternatives (SimCLR, SimSiam, DINO, ImageNet transfer, augmentation, learned teachers) under one frozen recipe with fixed subsets: 13 datasets, 9 backbones, 150 to 1.28M images, 32--224,px, 2.5M--86M parameters ( $\computeCells$ classification configurations over $\computeRuns$ runs, plus segmentation and detection transplants). Across the training-time combinations we measure, three outcomes recur (decision-level fusion differs). Different-currency sources can stack: the prior composes with DeiT augmentation on attention backbones and is worth $+26$ points to ViT-B/16 at $224$ ,px, $+6.7$ at twice that budget. Same-currency sources substitute: against effective self-supervised pretraining, the combination never usefully exceeds the better single source. Fusing at full strength into an already-informed initialization interferes in proportion to what it carries: ImageNet transfer, $-15$ to $-17$ points, removed by a weaker auxiliary weight. Frozen-feature diagnostics measured on each source alone separate these outcomes retrospectively but do not predict them: a rule built on them calls one of nine unseen pairs. At a practitioner's own label budget, the frozen-feature gain predicts the end-to-end gain to within $0.17$ points across 30 cells and seven datasets; the underlying decomposition, $Δ= G + \readout(\mathrm{base})$ , holds in sign on $\auditRate\%$ of testable cells and is called an unseen backbone family's feature gain in advance. The project page is https://amughrabi.github.io/MomentAux.

Figures & tables

Line of work	Rec.	Cst.	Fea.	Cmb.	Prd.
Fixed spectral filters in the forward path [ 7 , 6 , 21 ]	$\circ$	✗	✗	✗	✗
Auxiliary and distillation targets [ 9 , 14 , 15 ]	$\circ$	✗	$\circ$	✗	✗
Self-supervised pre-training [ 2 , 8 , 3 ]	✗	✗	✓	✗	✗
Transfer learning studies [ 1 ]	$\circ$	✗	$\circ$	✗	✗
Small-data ViT recipes [ 4 ]	✗	$\circ$	✗	$\circ$	✗
Backbone and self-supervised benchmarks [ 19 , 20 ]	$\circ$	✗	✓	✗	✗

Table 1: Where the literature stands on the five properties a fusion question needs, and what this work adds. ✓ = present;

\circ

= partial or incidental; ✗ = absent.

Figure 1: The study in one picture. (a) One frozen recipe and fixed subset indices, so every intervention sees byte-identical images; deviating cells are marked diagnostic and never enter headline tables. (b) MomentAux: a pinned bank of oriented-energy targets regressed from a tap at the third of four feature stages, during training only, with $\lambda$ decaying to exactly zero, so the deployed network is byte-identical to the baseline. (c) Each cell yields the end-to-end gain $\Delta$ and the frozen-feature gain $G$ , whose difference is the readout term of Eq. 1 . (d) The comparator ladder, ordered by declared training cost; all are fused into the same recipe, and pairwise combinations are measured too. (e) One measured cell per fusion outcome, each source alone and then both, as gain over the shared baseline: the fused bar clears both singles, sits level, or is dragged back across zero. Interference is shown at full auxiliary strength, the condition it depends on (Sec. 6.3 ); the frozen features move the same way in all three cases (Sec. 6 ).

Figure 2: The knowledge source, in full, and what it computes (an illustration, not evidence). Top: the eight complex Gabor quadrature pairs of Eq. 2 , two scales $\sigma\in\{2,2\sqrt{2}\}$ crossed with four orientations, even and odd components. Below: a photographic (STL-10) and a satellite (EuroSAT) test image, each with its eight magnitude-response maps of Eq. 3 from the pinned bank under the study’s own calibration, one color scale per example; each sample is chosen deterministically as the test image on which every orientation channel dominates somewhere. Trunks and limbs light the STL-10 orientations, roads and field boundaries the EuroSAT ones, the same fixed kernels in both domains. The bank is fixed before any training, identical everywhere, and fingerprinted by a regression test.

Axis	Coverage
Backbones	ResNet-18, -34 and -50 [ 11 ] , MobileNetV3 [ 35 ] , ConvNeXt-T [ 36 ] , ViT-tiny, -S and -B [ 12 ] , Swin-T [ 17 ]
Interventions	MomentAux ( $1.02\times$ ); SimCLR, SimSiam and DINO ( $2\times$ ); ImageNet transfer; DeiT augmentation; FitNets teacher ( $2\times$ ); HOG target; pairwise combinations
Measured	$3{,}077$ cells over $9{,}471$ retained run records including exploration and repeats

Table 2: The benchmark’s backbones and interventions. Every cell trains the frozen recipe of Sec. 3.1 ; costs are declared as multiples of baseline training compute; the datasets, domains, scales, and resolutions these cross are in Table 3 . Backbone coverage is not uniform across comparators: DINO runs only on ViT-tiny, so its cells cross datasets but not architectures.

				training images at fraction (per class)
Domain	Dataset	Cls.	Px	1%	2%	3%	5%	7%	10%	15%	20%	25%	50%	100%
Photographic	CIFAR-10 [ 37 ]	10	32	500 (50)	1,000 (100)	1,500 (150)	2,500 (250)	3,500 (350)	5,000 (500)	7,500 (750)	10,000 (1000)	12,500 (1250)	25,000 (2500)	50,000 (5000)
	CIFAR-100 [ 37 ]	100	32	500 (5)	1,000 (10)	1,500 (15)	2,500 (25)	3,500 (35)	5,000 (50)	7,500 (75)	10,000 (100)	12,500 (125)	25,000 (250)	50,000 (500)
	STL-10 [ 38 ]	10	96	$\times$	$\times$	150 (15)	250 (25)	350 (35)	500 (50)	750 (75)	1,000 (100)	1,250 (125)	2,500 (250)	5,000 (500)
	Tiny-ImageNet [ 39 ]	200	64	1,000 (5)	2,000 (10)	3,000 (15)	5,000 (25)	7,000 (35)	10,000 (50)	15,000 (75)	20,000 (100)	25,000 (125)	50,000 (250)	100,000 (500)
Satellite	EuroSAT [ 40 ]	10	64	216 (22)	432 (43)	648 (65)	1,080 (108)	1,512 (151)	2,160 (216)	3,240 (324)	4,320 (432)	5,400 (540)	10,800 (1080)	21,600 (2160)

Table 3: Datasets and the partitions they were trained on. Counts are read from the fixed subset index files every arm consumes, so they carry the rounding of the per-class stratified draw. Parentheses give images per class, which sets a cell’s regime. Relabeled controls and multi-source views reuse their parent’s indices exactly and share its row, which is why the twenty dataset identities counted in the sign-law audit are not twenty independent populations: five are relabeled controls of their parent’s pixels and four are band subsets of two parent instruments, leaving eleven independent image sources. The two blanks differ:

\times

marks a fraction that cannot be trained under the frozen recipe, yielding fewer than one batch of 128 images; a dash marks one that is runnable but was not run.

Table 6

	CIFAR-100			Tiny-IN
Fraction	$\Delta_{\mathrm{plain}}$ $\uparrow$	$\Delta_{\mathrm{DeiT}}$ $\uparrow$	Ratio $\uparrow$	$\Delta_{\mathrm{plain}}$ $\uparrow$	$\Delta_{\mathrm{DeiT}}$ $\uparrow$	Ratio $\uparrow$
1%	$+1.4$	$+3.2$	2.4	$+1.3$	$+3.0$	2.4
2%	$+3.3$	$+6.0$	1.8	$+3.3$	$+6.6$	2.0
3%	$+6.2$	$+9.5$	1.5	–	–	–
5%	$+9.4$	$+16.3$	1.7	$+6.8$	$+10.2$	1.5
7%	$+11.0$	$+18.7$	1.7	–	–	–

Table 6: ViT-tiny: gain of the prior under the plain recipe and under DeiT-strength augmentation, with the amplification ratio, on CIFAR-100 (selection set, Sec. 3.6 ) and, in the last three columns, Tiny-ImageNet. Augmentation does not substitute for the prior; it amplifies it and shifts its peak toward more data. Blank: not measured. Three seeds per cell, six on the DeiT

5\%

arm. Best in bold, second best underlined, third best in italics, per

\Delta

column; the ratio columns are derived and unstyled. Ratios, and the ranges quoted for them in the text, are computed from unrounded deltas, and should be read to about

\pm 0.3

: their denominators are augmented baselines whose seed standard deviations reach

4.9

points on Tiny-ImageNet.

Figure 3: The law, in full: readout

=\Delta-G

against baseline accuracy for all 958 law-scope cells with

{\geq}3

seeds on both arms. Shaded band: the measured crossing bracket

[31.8,40.3]

, inside which no sign is predicted. Only the first two legend entries test the account: of the 455 cells whose readout is resolvable against its own seed-paired uncertainty, 393 fall on the predicted side of zero. The other two entries keep the audit’s reach visible: 409 are unresolvable and 94 sit inside the bracket, so the rate is

86.4\%

47.5\%

coverage against a

75.6\%

majority-sign baseline. The scatter at fixed baseline is substantial (

R^{2}=0.27

, residual SD

2.0

points): the account predicts the sign of this term, not its value.

Audit variant	Cells	Correct	Rate	95% CI
$\|\textit{readout}\|>1$ SEM	629	509	80.9%	[77.7, 83.8]
$>1.5$ SEM	531	446	84.0%	[80.6, 86.9]
$>2$ SEM (declared)	455	393	86.4%	[82.9, 89.2]
$>3$ SEM	342	304	88.9%	[85.1, 91.8]
One vote per (dataset, backbone, fraction)	235	192	81.7%	[76.3, 86.1]
Held out: bracket fitted without the audited dataset	391	347	88.7%	[85.2, 91.5]

Table 7: Robustness of the audit: the rate is stable against the resolvability threshold, survives clustering on the units that share a baseline arm, and survives re-fitting the crossing bracket without the audited dataset (the out-of-sample check). It is carried by the left flank, where the account predicts strongly, not by the right, where it predicts

{\approx}0

. Intervals are Wilson 95%.

Population	Cells
Cells with paired $\Delta$ and $G$ (all interventions)	1,734
in scope, ${\geq}3$ seed-matched arms	958
Backbone variants in scope (architecture families)	7 (5)
Dataset identities in scope	20
Inside crossing bracket (no prediction)	94
Unresolved ( $\|\textit{readout}\|\leq 2$ SEM)	409

Table 8: Scope-wide sign-law audit, regenerated from the per-run records by the released audit script. A cell is resolvable when

|\textit{readout}|>2\,\mathrm{SEM}

; cells inside the crossing bracket make no sign prediction. Uncertainty is seed-paired (Sec. 3.5 ), not propagated from independent arms. Scope is aux-from-scratch: cells carrying an ImageNet or self-supervised initialization are excluded here and scored separately in Sec. 6 , and

100\%

of cells report only the aux-vs-baseline gap under the scope rule of Sec. 3.4 , which is why the first row is much larger than the second; the rows narrow to the resolvable subset that actually tests the account.

Dataset	Backbone	Variant	Frac.	Base	$\Delta$	$G$	Readout
EuroSAT	Swin	aux	20%	22.7	$+71.27$	$+55.93$	$+15.34$
EuroSAT	Swin	aux	15%	18.6	$+73.72$	$+61.66$	$+12.06$
EuroSAT	Swin	aux	2%	42.3	$+30.54$	$+36.49$	$-5.95$
Food-101	R18	mag3	50%	71.8	$-1.30$	$+4.66$	$-5.96$
Food-101	R18	mag6o	50%	71.8	$-0.91$	$+4.97$	$-5.88$
Food-101	MNet	aux	50%	55.1	$-0.23$	$+3.70$	$-3.93$

Table 9: The ten largest-magnitude resolvable exceptions, of 62 in total, printed from the same audit as Table 8 ; they cluster, not scatter. Top: Swin cells whose baseline arm is unstable (seed SD

10

–

22

points), so the baseline height the account takes as input is not a task-performance reading; the records flag the first two as bistable and the third narrowly not. Second: the “better features, worse accuracy at sufficiency” signature. Third: PathMNIST, whose linear evaluation is a compressed measuring stick. Bottom: a heavily-augmented ViT cell on DTD, with readout positive far below the crossing.

Figure 4: The spectral imprint at the tap, CIFAR-100 at 5%. Each of the four sample blocks shows the input, the fixed moment target, and the tapped-stage energy of the baseline and prior arms. Over the whole test set, the alignment between tapped features and the target is $+0.215$ for the baseline and $+0.525$ for the prior arm. The four samples are selected cases (prior arm correct, baseline wrong), so they illustrate the alignment difference, not sample it, and only the test-set-wide figure should be read as evidence.

Figure 5: Where the class evidence sits, CIFAR-100 at $5\%$ . The four samples are selected cases (prior arm correct, baseline wrong), so they show what the difference looks like where it is present, not how often. The evidential quantity is measured over the whole test set instead: the Gini coefficient of each class-activation map normalized to sum to one, where higher means evidence concentrated at fewer locations, gives $0.253$ for the baseline against $0.273$ for the prior arm over $2{,}000$ images ( $8.6\,\sigma$ ). The prior’s evidence is measurably more concentrated, by a modest amount, not the categorical difference the selected samples suggest.

Family	$n$	mean $G$	alignment gap
Moment prior (tap L3)	10	$+4.23$	$+0.40$
SimCLR init, no target	5	$+7.68$	$-0.16$
Random fixed target	1	$+1.31$	-0.06
Moment prior, tapped at L1/L2	2	+4.67	$-0.18$
SimCLR init $+$ moment target	1	$\mathbf{+8.08}$	$\mathbf{+0.49}$

Table 10: The imprint follows the target , not the feature gain. The alignment gap is the arm minus its own baseline, CIFAR-100/ResNet-18, with one pinned target applied to every model. The lower block is a

2\times 2

that fell out of the design: crossing {self-supervised initialization or not} with {moment target or not}, the imprint tracks the target while

G

tracks the initialization. Best in bold, second best underlined, third best in italics, per numeric column; the two columns’ markings disagree.

Figure 6: Alignment gap against feature gain, one point per cell, CIFAR-100 / ResNet-18. Filled markers had the moment target during training. The two quantities are independent: every cell whose target was imposed at the tapped stage sits at

+0.31

+0.51

regardless of whether its

G

2.7

8.1

, and every cell without a target there sits below zero across the same range of

G

. The two labeled controls separate the accounts: the same target imposed two stages earlier leaves no imprint at layer 3, and the same self-supervised initialization as the negative points, with the target added, gives the largest gap measured.

Figure 7: What the prior gives a small transformer: locality. Mean attention distance per block, ViT-tiny on CIFAR-100 at 10%, three seeds (band:

\pm

1 SD). The dotted line is the distance a uniform attention map would give on this

8\times 8

token grid (

4.14

patch units), computed, not assumed. The baseline sits on that line at every depth, never learning to attend locally from 5,000 images; the prior pulls the middle blocks well below it, most sharply upstream of the tap. A SimCLR initialization at

2\times

the compute does the same, more weakly.

	prior $-$ SimCLR (points)
Recipe	1%	5%	10%	25%	100%
Plain crop+flip	$+0.7$	$+1.3$	$0.0$	$-1.4$	$-3.9$
DeiT augmentation	$\mathbf{+2.5}$	$\mathbf{+7.2}$	$\mathbf{+5.2}$	$\mathbf{+5.0}$	$\mathbf{+2.0}$

Table 11: Prior versus SimCLR initialization (

2\times

compute) on ViT-tiny, CIFAR-100 (the selection set, Sec. 3.6 ), and the effect of recipe. Under plain augmentation, SSL catches the prior at

{\sim}5

k images; under the DeiT recipe, the prior wins at every fraction because the recipe supplies the invariance SSL was selling. Three seeds per cell, six on the DeiT-recipe

5\%

arm. A positive entry is a fraction where the prior wins; no better-direction marker is used, since neither sign of a method-versus-method difference is better a priori. Per fraction the recipe under which the prior stands better is bold; with two rows the second-best and third-best marks do not apply.

Dataset	Frac.	Transfer adv.	$\Delta$ (tax) $\uparrow$	$G$ $\uparrow$	Readout
CIFAR-10	5%	$+9..{+}14$	$-15.2$	$-15.8\pm 1.7$	$+0.6$
CIFAR-100	7%	$+10..{+}18$	$-16.8$	$-16.3\pm 3.1$	$-0.6$
PathMNIST	10%	$+1.8..{+}2.9$	$-0.4$	$+0.3\pm 0.6$	$-0.7$

Table 12: The interference tax of the ImageNet-transfer fusion, and its feature-side origin. Injecting the prior on top of ImageNet weights at full auxiliary strength is destructive in proportion to what the initialization supplied; linear evaluation shows the loss is carried almost entirely by

G

(features), not by the readout . PathMNIST, where ImageNet features barely transfer, is the null control. Ten seeds on the two CIFAR pairs end-to-end, three on PathMNIST, and three for every linear evaluation, so the readout column differs by a ten-seed

\Delta

against a three-seed

G

. The text below reports the same three cells at reduced

\lambda_{0}

, where the tax disappears.

Fusion	Currencies	Call	Measured	Key evidence
Prior + augmentation	structure + invariance	stack	stack	$\Delta$ amplified $1.4$ – $2.4\times$ (1–25%); $G\uparrow$ (ViTs; 1 of 6 conv cells off-selection)
Prior + effective SSL	structure $\approx$ invariance ∗	substitute	substitute	combo near best single: $+1.4$ above (ViT) to $-2.4$ below (conv off-selection; Sec. 6.2 )
Prior + ineffective SSL	structure + ${\sim}$ nothing	stack	stack	full gain recovered on SimSiam
Prior + ImageNet init	structure vs. mature features	interfere	interfere	$-15$ to $-17$ e2e at $\lambda_{0}{=}1.0$ , carried by $G$ ; null on shift; gone at $\lambda_{0}{\leq}0.3$
Augmentation + SSL	invariance + invariance	substitute	substitute	DeiT recipe collapses SimCLR’s margin
∗ Same currency in effect: at matched cells the two sources’ feature gains coincide, which is why neither adds to the other.

Table 13: The retrospective fusion taxonomy the grid supports, with the measurement that establishes each row. The rule’s call uses only single-source measurements, the two sources’ frozen-feature gains and what each improves, and the measured outcome matches it in all five rows. Every call here is retrospective; the one prospective test of the rule, on pairs outside this table, is called one of nine (Sec. 9.3 ), so this is a taxonomy the rule organizes, not a prospective scoreboard. Each row is scoped to the evidence beside it, and the first row is the narrowest: on two convolutional populations outside the selection set, the same pairing stacks at one cell of six and mildly interferes at the rest (Sec. 6.1 ).

Figure 8: Currency tested directly, not inferred from magnitudes. All arms are ViT-tiny on CIFAR-100 at 10% with one shared baseline, so the only thing that varies is the fused source. (a) agreement between two interventions in which test images they fix and which representations they learn, each normalized by that intervention’s own seed-to-seed agreement, so

1.0

means the two sources differ by no more than two seeds of one of them. Four measures are shown because they do not agree on the effect size. (b) the number of images each intervention fixes: the two same-currency sources fix all but one the same number.

Cell	Registered band	$\Delta$ $\uparrow$	$G$ $\uparrow$	$\|\Delta{-}G\|$ $\downarrow$	Verdict
ImageNet64: 1.28M images, 1000 classes
ResNet-18	$\Delta$ : $0.0{\pm}0.5$	$+0.04$ $\pm$ 0.08	$+0.05$	$0.01$	neutral, exactly
MobileNetV3	(none: in bracket)	$+1.95$ $\pm$ 0.40	$+1.81$	$0.14$	consistent
ViT-tiny	$\Delta\geq+3$	$+3.24$ $\pm$ 0.84	$+2.84$	$0.40$	deficit persists
Swin-T	–	baseline collapses at chance			stabilized by prior
ImageNet-100 @ 224 px, DeiT recipe on ViTs

Table 14: The pre-registered scale block. Every falsifier was declared before the first run and none fired, with three qualifications kept in view: ViT-S’s

G

landed

0.05

below its registered band; ViT-B’s

|\Delta-G|

3.12

exceeds the registered

1.5

, on a cell whose evaluation-ceiling limitation was also registered in advance and whose residual is unresolvable; and ResNet-50’s

G

, registered at

-0.7

-0.2

, measured

+0.19

, a

0.39

miss across zero on a quantity that is a measured zero either way (

|\Delta-G|=0.21

G

at ImageNet64 uses the fixed-budget evaluation (250 per class shown); ImageNet-100 the standard full-train evaluation. Three seeds per cell; the ImageNet-100 cells run

100

epochs, and the text repeats both ViT pairs at

200

, where the gains are

+4.52\pm 0.18

and

+6.71\pm 0.90

Figure 9: The envelope at ImageNet scale, and why its shape is a property of the backbone. Dataset, recipe, fixed subsets and label space are held constant; only the backbone varies. (a) Three backbones, three envelope shapes: ResNet-18 falls to zero and stays there, while ViT-tiny and MobileNetV3 climb across the whole range. (b) The same baselines against the crossing bracket. Each gain decays as its own baseline enters the band, and the three arrive at very different fractions: ResNet-18 by

7

–

15\%

, MobileNetV3 only at

100\%

, ViT-tiny passing above it at

100\%

, exactly where its gain falls. Three seeds per cell.

	ImageNet64, 64 px			ImageNet-100, 224 px
Frac.	R18	ViT-Ti	MNetV3	R50	ViT-S	ViT-B
1%	$+1.90$	$\mathbf{+2.14}$	$-0.52$	$+0.54$	$\mathbf{+2.31}$	$+0.69$
	5.5	1.9	4.3	11.5	7.3	5.3
2%	$+1.59$	$\mathbf{+3.80}$	$+0.01$	$+1.21$	$\mathbf{+7.59}$	$+5.11$
	13.2	2.6	7.6	18.2	6.4	4.1
3%	$+1.38$	$\mathbf{+4.76}$	$+0.49$	–	–	–

Table 15: The envelope at ImageNet scale:

\Delta

by fraction, three seeds per cell, both arms sharing the reduced-epoch budget. Baseline accuracy is given beneath each gain. ImageNet64’s three backbones see identical images, recipe, and label space, so the difference in envelope shape between them is attributable to the backbone alone. The ImageNet-100 ViT columns below

100\%

are reported with a caveat, not as headline cells: their baselines sit close to chance and carry seed standard deviations up to

8.2

points, so the gains there are large but poorly determined. Every ImageNet-100 cell here uses the

100

-epoch budget; the ViT-B pair at

200

epochs gives

+6.71

(Sec. 7 ). Best in bold, second best underlined, per fraction within each stage (the remaining entry is third).

	Prior	$2\times$ SSL		$5\times$ SSL		prior $-$ $5\times$
Frac.	$\mathbf{1.02\times}$ $\uparrow$	SSm $\uparrow$	SC $\uparrow$	SSm $\uparrow$	SC $\uparrow$	SSm	SC
1%	$+1.42$	$-0.02$	$+2.27$	$-0.09$	$\mathbf{+4.54}$	$+1.51$	$-3.12$
2%	$+2.50$	$-0.11$	$+4.88$	$+0.19$	$\mathbf{+8.33}$	$+2.31$	$-5.83$
3%	$+3.68$	$+0.11$	$+5.99$	$+0.54$	$\mathbf{+10.82}$	$+3.14$	$-7.14$
5%	$+5.15$	$+0.13$	$+9.05$	$+1.90$	$\mathbf{+14.38}$	$+3.25$	$-9.23$
7%	$+4.87$	$+0.87$	$+9.86$	$+3.39$	$\mathbf{+13.88}$	$+1.48$	$-9.01$

Table 16: Buying the comparators more compute. CIFAR-100 (the selection set, Sec. 3.6 ), ResNet-18, gain over the baseline by data fraction; three seeds per pre-trained cell, ten for the baseline and the prior at 1, 5 and 10%. SSm is SimSiam and SC is SimCLR, at 200 pre-training epochs (

2\times

baseline compute) or 800 (

5\times

); the prior costs

1.02\times

. The last two columns give the prior minus each

5\times

comparator, where positive means the free prior still wins.

	EuroSAT-MS (one instrument)		So2Sat (two satellites)
Data	$\|\Delta_{\mathrm{acc}}\|$ $\downarrow$	Fusion $\uparrow$	$\|\Delta_{\mathrm{acc}}\|$ $\downarrow$	Fusion $\uparrow$
1%	6.68	$+1.40$	26.31	$-0.57$
2%	1.07	$\mathbf{+2.96}$	33.77	$-0.14$
5%	0.58	+1.34	30.61	$-0.73$
10%	0.66	$+1.15$	28.51	$-0.11$
25%	0.11	$+0.45$	25.98	$\pm 0.00$

Table 17: Does a second source pay? Fusion gain is the fused arm minus the better single source, at matched data. Sorted by data fraction. Three seeds per cell, ResNet-18; the reference configuration is used verbatim throughout. Splitting one instrument’s bands pays at every fraction; pairing two satellites does not pay at any. Best in bold, second best underlined, third best in italics, in the EuroSAT-MS fusion column; the SO2Sat fusion column is left unmarked because there is no gain (its best value is zero, which is the finding), and the asymmetry columns are unmarked inputs.

Pair	Gain $\uparrow$	Disagreement
Prior + SimCLR (50 ep)	$\mathbf{+1.20}$	47.2%
SimCLR (50 ep) + prior, wide bank	$+1.13$	47.2%
Prior + prior, wide bank	$-$ 0.04	28.1%

Table 18: Decision-level fusion on CIFAR-100 at 5%, restricted to pairs whose two arms are within one accuracy point of each other so that the information source, rather than an accuracy gap, is what differs. Gain is over the better single arm; disagreement is the fraction of test images on which the two arms predict different classes. Averaging posteriors requires no further training. Best in bold, second best underlined, third best in italics, in the gain column.

Figure 10: Semantic segmentation, six populations, 200 epochs, three seeds per cell. (a)

\Delta

as a percentage of each cell’s own baseline, not absolute mIoU, since the baselines differ eightfold. Read this way, four of the six envelopes are unimodal, Cityscapes is flat within noise, and Pascal-Context is highest at

1\%

and lower everywhere after; all six occupy the same 1–22% band classification does across

1

–

25\%

, and three go neutral or negative at full data. (b) The readout term against the head’s own classification scale (pixel accuracy), not mIoU: the crossing bracket is an accuracy bracket, and scoring the law on mIoU puts every dense cell on the wrong flank. Filled markers are the resolvable cells; all sit above the bracket, so segmentation confirms the law’s positive branch and cannot test its negative one.

Population	Cls.	1%	2%	5%	10%	25%	100%
VOC	21	$+0.39$	$+0.97$	$+1.15$	$+1.61$	$+2.34$	$+0.46$
rel.		6%	12%	9%	9%	8%	1%
Cityscapes	19	$+0.49$	$+0.41$	$+0.20$	$+0.68$	$+0.73$	$+0.48$
FoodSeg103	104	$+0.12$	$+0.20$	$+0.26$	$+0.43$	$+0.56$	$-0.12$
ADE20K	150	$+0.18$	$+0.20$	$+0.38$	$+0.38$	$+0.37$	$-0.29$
Pascal-Ctx	254	$+0.11$	$+0.13$	$+0.16$	$+0.10$	$+0.16$	$+0.17$

Table 19: The dense envelope.

\Delta

in mIoU points, with

\Delta

as a percentage of that cell’s own baseline beneath it. Three seeds per cell, ResNet-18 with an FCN head at output stride 8, 200 epochs at every fraction, the same budget the classification recipe uses. Pascal-Context is VOC’s own pixels relabelled to 254 classes. The last block (

\dagger

) is a Swin-Tiny encoder and is a diagnostic row, not a headline one: it trains under AdamW, not the frozen recipe, for the reason given in Sec. 10.4 , and carries the bistability caveat that attaches to every Swin cell.

Data	$\Delta$ AP50	$\Delta$ fg_acc	$\Delta$ fg_iou	$G$ (fg_acc)	$G$ (fg_iou)
1% †	$-0.04$	$-0.23$	$-0.0003$	$-0.16$	$+0.0017$
2% †	$-0.00$	$-0.23$	$-0.0001$	$-0.37$	$-0.0006$
5%	$-0.08$	$+0.11$	$-0.0048$	$+0.34$	$+0.0130$
10%	$+0.56$	$-0.32$	$+0.0032$	$+0.08$	$+0.0073$
25%	$-0.41$	$+0.01$	$+0.0017$	$-0.47$	$+0.0011$
100%	$-0.14$	$+0.64$	$+0.0031$	–	–

Table 20: Detection is a null.

\Delta

is aux minus baseline on PASCAL VOC. AP50 is the task metric; fg_acc and fg_iou are the head’s 20-way accuracy and its mean box IoU at ground-truth foreground locations , so neither can collapse the way a ranked-precision integral does.

G

is the same quantity under a frozen trunk with a fresh

1\times 1

head fitted on the full split. Cells marked

\dagger

(1% and 2%) fall under the pre-declared floor rule: both arms below 1.0 AP50, where a difference is not a measurement.

	$\Delta$ at fraction $\uparrow$
Aux target	5%	10%	Note
Moment magnitude (ours)	$+3.18$	$+2.71$	phase-invariant energy
Structure tensor	$+1.07$	$+1.68$	mild nonlinearity
Steerable harmonics	$+1.55$	$+0.91$	principled rot.-inv.
Rotation-invariant energy	$+1.43$	$+0.74$
Oriented edges (Gabor)	$+0.56$	$-0.09$	raw edges do not help

Table 21: Target and mechanism controls on CIFAR-100 with ResNet-18 (each row is the reference configuration with one ingredient varied;

\Delta

vs. the shared baseline, three seeds in the varied arm against the released ten-seed baseline). These cells hold

\lambda{=}0.3

, not the reference configuration’s decaying schedule, so they are comparable to each other, as an ablation requires, but not to Table 5 . On this dataset, the gain needs the moment structure specifically; the text quantifies each control. CIFAR-100 is the set on which the configuration was selected, and the text below reports the same ablation transplanted to three populations that are not.

Regime	Recommendation	Cost	Measured basis
Conv backbone, photo-like data, mid band (2–15%), $2\times$ affordable	SimCLR init	$2\times$	$+2..{+}9$ over baseline; beats prior by $+2..{+}5$
Conv, photo-like, $2\times$ not affordable	prior	$1.02\times$	$+1.4..{+}6.7$ ; ${\sim}$ SSL parity at extremes at $2\times$ ; at $5\times$ SimCLR leads
Conv, extreme scarcity ( ${\leq}1$ – $2\%$ )	prior (or fwd stem)	$1.02\times$	SSL margin ${\leq}1$ pt at $2\times$ ; at $5\times$ SimCLR leads by $+3.1$ ; forward stem best ${\leq}2\%$
Satellite-like statistics, low data	prior	$1.02\times$	beats SimCLR at 7 of 11 EuroSAT fractions
ViT backbones evaluated here under DeiT-strength augmentation	prior and augmentation	$1.02\times$	flip: beats SSL at every fraction at $2\times$ ; at $5\times$ only in the higher-data band (both populations); $+14..{+}25$ over recipe alone
From-scratch attention backbones evaluated in this study	prior	$1.02\times$	$+3.2$ at 1.28M imgs; $+13$ and $+26$ at 224 px at 100 ep, $+4.5$ and $+6.7$ at 200; stabilizes Swin

Table 22: What to use when: the grid distilled to recommendations. Costs are training-compute multiples of the baseline; gains are typical measured ranges, not promises. “Prior” is the reference MomentAux configuration verbatim.

Figure 11: Every resolvable law-scope cell, end-to-end gain against frozen-feature gain. On the dashed diagonal, accuracy realizes the feature gain exactly. The shaded quadrant is the patterned failure: features improve measurably while accuracy falls. Only cells whose baseline sits above the crossing are marked, since below it a negative readout is what the account predicts, not a violation of it. Produced by the same audit as Table 8 , so the figure and the exception list cannot disagree.

Representation Learning

Representation learning develops data transformations that expose useful structure for prediction, transfer, generation, and control.

See all 804 papers

May 13, 2026

FeatCal: Feature Calibration for Post-Merging Models

Model merging combines task experts into one model and avoids joint training, retraining, or deploying many expert models, but the merged model often still underperforms task experts. We study this performance gap through feature drift, the difference between features produced by the merged model and by the expert on the same input. Our theory decomposes this drift into upstream propagation and local mismatch, tracks how it propagates and combines through later layers in forward order, and links final feature drift to output drift. This view motivates FeatCal, which uses a small calibration set to calibrate the merged model weights layer by layer in forward order, reducing feature drift while staying close to merged weights and preserving the benefits of model merging. FeatCal uses an efficient closed-form solution to update model weights, with no gradient descent, iterative optimization, or extra modules. On the main CLIP and GLUE benchmarks, FeatCal beats Surgery and ProbSurgery, the closest post-merging calibration baselines: 85.5% vs. 77.0%/78.8% on CLIP-ViT-B/32 Task Arithmetic (TA) and 85.2% vs. 83.7%/82.2% on FLAN-T5-base GLUE. On CLIP-ViT-B/32, 8 examples per task reach 82.9%, and 256 examples per task take 53 seconds, about 4x faster than both baselines, showing better sample efficiency and lower calibration cost.

Yanggan Gu, Shuo Cai, Zihao Wang +7

The Hong Kong Polytechnic University (PolyU) · The Chinese University of Hong Kong · PolyU-Daya Bay Technology and Innovation Research Institute

Distribution Shift Model Calibration Model Merging

Ask PDF

Jul 1, 2026

Fusion: A Framework for Unified Sequential Token AdaptatIon in VisiOn TraNsformers

Vision Transformers achieve strong image classification accuracy but process all image regions with nearly the same computation, even when many regions are redundant or uninformative. Recent adaptive inference methods reduce this cost by selectively compressing tokens or terminating inference early, but combining these mechanisms often causes unstable intermediate representations and accuracy degradation. We introduce Fusion, a unified adaptive inference framework that coordinates token merging, early exiting, and token pruning through a simple staged design: tokens are merged first, confidence is evaluated next, and pruning is applied only to samples that continue inference. This ordering allows the three mechanisms to operate cooperatively rather than competitively. Fusion further includes lightweight routing modules that adapt compression strength to each input and support inference-time adjustment of the accuracy--latency trade-off without retraining. On ImageNet-1k with DeiT-S, Fusion matches or surpasses state-of-the-art adaptive ViT methods at comparable compute budgets while reducing calibration error by up to

4\times

and inference energy by

48\%

. Experiments across ImageNet-100, CIFAR-100, and ImageNette with multiple ViT backbones demonstrate consistent transferability without dataset-specific tuning.

Aravind Pradeep, Samira Nazari, Mahdi Taheri +1

Brandenburg University of Technology Cottbus-Senftenberg, Germany · University of Zanjan, Iran · Humboldt University, Berlin, Germany +1

Token Merging Efficient ViTs Visual Token Merging Visual Token Pruning Visual Token Compression

Ask PDF

May 18, 2026

A Large-Scale Study on the Accuracy vs Cost Trade-offs of Training and Evaluation Settings in Fine-Grained Image Recognition

Prior work on fine-grained image recognition (FGIR) has established the importance of the backbone selection, but has neglected the accuracy-vs-cost trade-offs under different training and evaluation settings. In this work we conduct a large-scale study with over 2000 experiments across 6 training and evaluation settings, 9 pretrained backbones, and 17 datasets. Preliminary observations on the effectiveness of data augmentation for fine-grained training motivate us to extend Counterfactual Attention Learning (CAL), a state-of-the-art method based on data-aware cropping and masking augmentations, with cross-image discriminative region mixing augmentation. We also propose an efficient evaluation-only variant that maintains competitive accuracy while reducing inference costs by forfeiting the forward pass on discriminative crops that is normally used by CAL and similar FGIR methods. Our results show that data-aware augmentations during training only can enable a model to achieve excellent accuracy even without crops, significantly reducing inference costs. To support future research we share our code and checkpoints at: https://github.com/arkel23/FGIR-Backbones

Edwin Arkel Rios, Augusto Christian Surya, Oswin Gosal +5

National Yang Ming Chiao Tung University, Taiwan · National Tsing Hua University, Taiwan

Data Augmentation Cost-Aware Inference Fine-Grained Image Classification Fine-Grained Visual Perception

Ask PDF

Abstract

Representation Learning

Explore similar work

FeatCal: Feature Calibration for Post-Merging Models

Fusion: A Framework for Unified Sequential Token AdaptatIon in VisiOn TraNsformers

A Large-Scale Study on the Accuracy vs Cost Trade-offs of Training and Evaluation Settings in Fine-Grained Image Recognition