Representation Risk in Pretrained Image Encoders

Ardyn Nordstrom; Morgan Nordstrom; Vamuyan Sesay; Matthew D. Webb

econ.GNSep 28, 2026

Representation Risk in Pretrained Image Encoders

Authors: Ardyn Nordstrom, Morgan Nordstrom, Vamuyan Sesay, Matthew D. Webb

Organizations: School of Public Policy and Administration, Carleton University · Independent Researcher · Department of Economics, Carleton University

Abstract

Applied researchers increasingly convert images into features with pretrained encoders, then use those features in a downstream prediction model. The encoder is often treated as an implementation detail. We show that it can instead be a consequential source of model uncertainty. We call this uncertainty representation risk: plausible pretrained encoders map the same images into different feature spaces and can yield sharply different out-of-sample conclusions from predictive performance. We compare ten modern and legacy frozen encoders across applications involving house prices, racehorse performance, breast-cancer histology, chest radiographs, continuous facial age, and rice disease. With common dimension control, heads, and group-safe splits, validation selects SigLIP 2 for houses, raising test $R^2$ from 0.396 for ResNet50 to 0.629, and DINOv2 for horses, raising $R^2$ from 0.029 to 0.105. No encoder is best in every task. Candidate procedures are constructed using training data and compared on a separate validation partition. The selected procedure reaches 0.658 for houses and 0.979 accuracy for pneumonia. Fixed-split gains are small for horses and rice, while repeated partitions reveal instability in horse feature union. Continuous age selects SigLIP 2 at 4.786 years MAE. The principal representation gaps persist with neural heads, similarly sized DINOv2 and ViT models, and limited adaptation. These results support a simple workflow: benchmark plausible representations, select on locked validation data, combine only when separate validation evidence justifies the additional cost, and report paired and split-level uncertainty. We implement this workflow in LOOKAGAIN-ML, the software package used to conduct the analyses in this paper.

Figures & tables

Figure 1: Representation choice is first-order; model combination is second-order

Dataset	$N$	Test $N$	Task	Primary metric
Toronto houses	7,273	1,090	log sale price	$R^{2}$
OBS horses	9,729	1,459	normalized workout speed	$R^{2}$
BreaKHis	7,909	1,129	binary histology	accuracy
Pneumonia	5,856	878	binary chest X-ray	accuracy
Facial age	9,764	1,465	chronological age	MAE (years)
Rice disease	5,932	759	four leaf classes	accuracy

Table 1: Main datasets and locked test designs.

N

is the common-complete sample used for encoder comparisons.

Figure 2: Representation choice and combination in the two motivating applications. Dots report locked-test

R^{2}

; horizontal bars are 95% grouped-bootstrap intervals from fixed fitted predictions. The same observations, dimension control, and ridge-head protocol are used within each panel. The final row is the procedure selected on outer-validation loss: a linear stack for houses and feature union for horses.

Dataset	ResNet	Selected single	Preferred procedure	Gain (95% CI)
Houses ( $R^{2}$ )	0.396	0.629 (SigLIP 2-B/16)	0.658 (linear stack)	0.029 $[0.017,0.042]$
Horses ( $R^{2}$ )	0.029	0.105 (DINOv2-B/14)	0.111 (feature union)	0.006 $[-0.017,0.031]$
BreaKHis (acc.)	0.895	0.878 (VGG16)	0.931 (equal average)	0.053 $[0.016,0.100]$
Pneumonia (acc.)	0.942	0.965 (ConvNeXt-B)	0.979 (feature union)	0.015 $[0.002,0.028]$
Facial age (MAE)	9.912	4.786 (SigLIP 2-B/16)	4.867 (linear stack)	-0.081 $[-0.447,0.139]$
Rice disease (acc.)	0.997	1.000 (DINOv2-B/14)	1.000 (feature union)	0.000 $[0.000,0.000]$

Table 2: Locked-test headline results. The selected single minimizes outer-validation loss over all ten encoders. The preferred procedure minimizes comparable outer-validation loss over that single and the combination menu fixed in the implemented workflow. Gain is oriented so positive values favor the preferred procedure.

Figure 3: Encoder choice and preferred procedure across the six main tasks. Houses and horses report

R^{2}

; BreaKHis, pneumonia, and rice report accuracy. Facial age is shown on its own MAE-in-years scale, where lower is better. Selection uses validation MSE, MAE, or log loss as appropriate, not the displayed test metric.

Dataset	Most selected	Single mean	Preferred mean	Mean gain	Preferred SD
Houses	SigLIP 2-B/16 (11/12)	0.653	0.682	0.030	0.016
Horses	DINOv2-B/14 (12/12)	0.102	-4.009	-4.111	14.204
BreaKHis	VGG16 (4/12)	0.823	0.882	0.060	0.042
Pneumonia	ConvNeXt-B (6/12)	0.952	0.969	0.018	0.003
Facial age	SigLIP 2-B/16 (12/12)	4.762	4.638	0.124	0.094
Rice disease	DINOv2-B/14 (6/12)	0.999622	0.999348	-0.000274	0.001

Table 3: Twelve group-safe repeated splits. Age columns are MAE; other performance columns use

R^{2}

or accuracy. Gains are oriented so positive values favor the preferred procedure: preferred minus selected-single performance for

R^{2}

and accuracy, and selected-single minus preferred MAE for age. “Preferred SD” is the sample standard deviation of preferred-procedure test performance across the twelve partitions, not a standard error. Means and differences are computed before rounding. “Most selected” reports all encoders tied at the maximum frequency.

Figure 4: Flexible heads do not eliminate the ResNet50–DINOv2 gap. Dots report locked-test

R^{2}

for the common linear head and the small neural head on houses and horses. Neural metrics use predictions averaged across five seeds fixed in the implemented workflow. The full encoder comparison and the mean-rank diagnostic, which uses average ranks for ties and lower-is-better ranks, appear in Appendix G .

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	$N$	Test	Groups	Grouping unit
Houses	7,273	1,090	7,262	saved / exact hash
Horses	9,729	1,459	9,461	reconstructed horse identity
BreaKHis	7,909	1,129	82	patient
Pneumonia	5,856	878	3,117	patient / exact hash
Facial age	9,764	1,465	9,658	exact hash
Rice disease	5,932	759	4,794	exact hash

Appendix

Table 4: Final sample and leakage audit.

N

is the common-complete analysis sample.

Encoder	Houses $R^{2}$	Horses $R^{2}$	Age MAE
ResNet50	0.3964	0.0293	9.912
VGG16	0.3868	0.0291	11.201
InceptionV3	0.4200	0.0350	9.797
MobileNetV2	0.4195	0.0405	10.079
COCO semantic	0.0084	0.0016	20.507
ADE20K semantic	-0.1254	0.0033	28.992

Appendix

Table 5: Regression single-encoder results. Houses and horses report

R^{2}

; facial age reports MAE in years, where lower is better.

Encoder	BreaKHis	Pneumonia	Rice
ResNet50	0.8955	0.9419	0.9974
VGG16	0.8778	0.9510	0.9960
InceptionV3	0.8379	0.9385	0.9802
MobileNetV2	0.9123	0.9601	0.9987
COCO semantic	0.7281	0.7437	0.5837
ADE20K semantic	0.5093	0.8075	0.6469

Appendix

Table 6: Classification single-encoder accuracy. Selection uses validation log loss, not the printed test accuracy.

Method	House	Horse	BreaKHis	Pneum.	Age MAE	Rice
Equal Average	0.5230	0.0573	0.9309	0.9681	9.6181	1.0000
Linear Stack	0.6582	0.1111	0.9300	0.9738	4.8675	1.0000
Selected Subset	0.6615	0.1046	0.9247	0.9772	4.7860	1.0000
Feature Union	0.5923	0.1109	0.9123	0.9795	7.6161	1.0000

Appendix

Table 7: Fixed-split combination candidates. Each candidate is fitted without outer-validation outcomes. Age is MAE; other columns are

R^{2}

or accuracy.

Panel of Figure 5 — Figure 5: Prediction and residual correlation diagnostics for houses (top) and horses (bottom). Each matrix uses the corrected fitted single-encoder predictions on the same locked observations.

Dataset	Spearman $\rho$	LogME top	Agrees with validation
Houses	0.952	SigLIP 2-B/16	yes
Horses	0.855	DINOv2-B/14	yes
BreaKHis	0.248	SigLIP 2-B/16	no
Pneumonia	0.806	ConvNeXt-B	yes
Facial age	1.000	SigLIP 2-B/16	yes
Rice disease	0.952	SigLIP 2-B/16	no

Appendix

Table 8: LogME rank association with validation ranking. Winner agreement is a screening diagnostic and never replaces validation.

	Houses		Horses
Encoder	Linear	NN	Linear	NN
ResNet50	0.3964	0.4225	0.0293	0.0349
VGG16	0.3868	0.4143	0.0291	0.0278
InceptionV3	0.4200	0.4288	0.0350	0.0312
MobileNetV2	0.4195	0.4395	0.0405	0.0359
COCO semantic	0.0084	0.0139	0.0016	0.0025

Appendix

Table 9: Linear and neural heads for the two motivating applications. Values are locked-test

R^{2}

; neural metrics use predictions averaged across five seeds fixed in the implemented workflow.

Encoder	Test MAE	Seed SD
ResNet50	8.071	0.074
VGG16	9.119	0.188
InceptionV3	8.163	0.116
MobileNetV2	8.197	0.040
COCO semantic	17.371	0.094
ADE20K semantic	31.845	3.111

Appendix

Table 10: Continuous-age neural-head MAE and across-seed SD, both in years.

Panel of Figure 6 — Figure 6: Neural-head robustness. Left: within-task normalized neural-head improvement for every dataset–encoder cell; positive values favor the neural head. Right: mean linear-head rank against mean neural-head rank across the six tasks. Ties receive average ranks and lower is better.

Encoder	Parameters (M)	Dimension	Median images/s
ResNet50	23.5	2,048	166.2
VGG16	138.4	512	155.9
InceptionV3	21.8	2,048	150.0
MobileNetV2	2.2	1,280	171.8
COCO semantic	11.0	21	180.5
ADE20K semantic	3.8	150	94.1

Appendix

Table 11: Recorded encoder characteristics and descriptive median cache-creation throughput across seven task records. These heterogeneous timings are not a standardized comparison on common hardware and a common extraction protocol.

Quantity	Value
JPEG images in the source archive	3,670
Excluded cross-class duplicate rows	2
Analytical observations	3,668
Duplicate-safe groups	3,666
Train / validation / locked test	2,568 / 550 / 550
Supplementary group-safe splits	12

Appendix

Table 12: Public TensorFlow Flowers demonstration: sample construction and locked design.

Figure 7: Outer-validation log loss for ten frozen representations under training-only tuning. Lower is better; the highlighted representation was selected without using test outcomes.

Representation	Accuracy	Balanced acc.	Macro F1	Log loss
ResNet50	0.8909	0.8907	0.8897	0.3658
VGG16	0.8855	0.8866	0.8860	0.3421
InceptionV3	0.8782	0.8761	0.8768	0.3428
MobileNetV2	0.9091	0.9097	0.9085	0.3197
COCO semantic	0.3127	0.2986	0.2649	1.5285
ADE20K semantic	0.4200	0.4001	0.3915	1.5292

Appendix

Table 13: Locked-test Flowers performance after training-only tuning. Selection uses outer-validation log loss.

Metric	SigLIP 2	Out-of-fold stack	Difference (95% CI)
Accuracy	0.9836	0.9836	0.0000 $[-0.0055,0.0055]$
Log loss	0.0597	0.0777	0.0180 $[0.0003,0.0406]$

Appendix

Table 14: Paired Flowers comparison from fixed fitted test predictions. Differences are stack minus selected single.

Figure 8: All-ten stack minus validation-selected SigLIP 2-B/16 on the locked test. Bars are paired 95% conditional bootstrap intervals from fixed fitted predictions. Zero denotes no difference; negative log-loss differences favor the stack.

Procedure	Mean acc.	SD	Empirical 95% range	Selection frequency
SigLIP 2	0.9852	0.0042	$[0.9782,0.9904]$	12/12 selected
All-ten cross-fitted stack	0.9861	0.0037	$[0.9805,0.9909]$	5/12 preferred

Appendix

Table 15: Flowers split stability under nested out-of-fold fitting and outer-validation selection.

Figure 9: Empirical test-accuracy variation across twelve group-safe splits fixed in the implemented workflow for the selected single encoder and all-ten cross-fitted stack. Bars show empirical 2.5th–97.5th percentiles, not confidence intervals.

Encoder	Seconds	Images/s	Peak GPU GiB
ResNet50	20.5	179.3	0.17
VGG16	26.1	140.6	0.81
InceptionV3	25.7	143.0	0.20
MobileNetV2	22.0	166.7	0.09
COCO DeepLab	73.9	49.6	–
ADE20K SegFormer	71.0	51.6	–

Appendix

Table 16: Fresh encoder extraction on the replication machine (RTX 3060, 3,668 images). Timings exclude model download.

Figure 10: Observed representation extraction time and outer-validation log loss on the recorded analysis environment. Timings describe this hardware and sample and should not be generalized mechanically.

Explore similar work

Beyond Local Linearity: Scale-Resolved Geometry of Learned Image Encoders
Sep 30, 2026Jakub Szymkowiak, Wojtek Pałubicki, Kamil AdamczewskiNeural Representation Geometry Visual Representation Learning
Adam Mickiewicz University in Pozna´n, Poland · IDEAS NCBR, Warsaw, Poland
Auditing Frozen-Encoder Anomaly Detection Across Mechanical Systems: Representation Provenance, Calibration, and Protocol Effects
Jan 16, 2026Jose Sánchez AndreuMahalanobis Distance Industrial Anomaly Detection
Independent Researcher, Murcia, Spain
Quantifying privacy risks in foundation model representations of brain MRI
Oct 14, 2026Imen Bajar, Héber Hwang Arcolezi, Antoine Boutet +2Medical Imaging Foundation Models Privacy-Preserving ML
École de technologie supérieure (ÉTS), Montréal, QC, Canada · Inria, INSA Lyon, CITI, UR3720, 69621 Villeurbanne, France · CREATIS, CNRS UMR5220, INSERM U1294, Université Lyon, INSA-Lyon, Lyon, France +1

Sep 30, 2026J/K move · Enter open · S save

Beyond Local Linearity: Scale-Resolved Geometry of Learned Image Encoders

Jakub Szymkowiak, Wojtek Pałubicki, Kamil Adamczewski

Adam Mickiewicz University in Pozna´n, Poland · IDEAS NCBR, Warsaw, Poland

Neural Representation Geometry Visual Representation Learning

Open paper AskPDF arXiv

Understanding how learned representations respond to finite input changes is important for characterizing their sensitivity, invariances, and robustness. Yet existing geometric analyses are predominantly local and describe only infinitesimal perturbations. We introduce a scale-resolved statistic that compares an encoder's measured feature displacement with its local linear prediction as the perturbation magnitude increases. Across diverse image encoders, we discover a characteristic plateau-rise-peak-decay profile, which we call the bump. The bump is absent at initialization, emerges early during standard training, and does not form under randomized labels or random-noise inputs. Its shape also varies with the training distribution and robustness objective. These results establish departures from local geometry as a signature of how encoder representations are shaped by learning.

Abstract

Explore similar work

Beyond Local Linearity: Scale-Resolved Geometry of Learned Image Encoders

Auditing Frozen-Encoder Anomaly Detection Across Mechanical Systems: Representation Provenance, Calibration, and Protocol Effects

Quantifying privacy risks in foundation model representations of brain MRI