Supervised Dimensionality Reduction Revisited: Why LDA on Frozen CNN Features Deserves a Second Look

Indar Kumar; Girish Karhana; Sai Krishna Jasti; Ankit Hemant Lade

cs.LGApr 5, 2026

Supervised Dimensionality Reduction Revisited: Why LDA on Frozen CNN Features Deserves a Second Look

Authors: Indar Kumar, Girish Karhana, Sai Krishna Jasti, Ankit Hemant Lade

Organizations: independent researchers

Abstract

Frozen pretrained image representations are widely used for transfer learning: a backbone is kept fixed, feature vectors are extracted, and a lightweight classifier is trained on top. This pipeline usually feeds the full feature vector to the classifier, even when the target task has far fewer classes than the pretraining task. We revisit a classical alternative: supervised dimensionality reduction with Linear Discriminant Analysis (LDA) before linear probing. We evaluate ten dimensionality-reduction strategies on frozen features from six backbones -- ResNet-18, ResNet-50, MobileNetV3-Small, EfficientNet-B0, ViT-B/16, and DINOv2-ViT-S/14 -- across CIFAR-100, Tiny ImageNet, and CUB-200-2011. Under a fixed logistic-regression protocol, LDA improves accuracy over full features in 11 of 12 coarse-grained configurations, with gains up to 4.5 percentage points while reducing feature dimensionality by 48-87%. The same projection consistently hurts on fine-grained CUB-200, where full features win across all six backbones. This establishes a practical boundary condition: LDA is useful when class-level structure is coarse enough to be captured by mean-separating directions, but it can discard subtle cues needed for fine-grained recognition. We also compare LDA with PCA, PCA+LDA, regularized LDA, Local Fisher Discriminant Analysis, Neighbourhood Components Analysis, and three lightweight LDA extensions. The results show that plain LDA offers the best accuracy-cost tradeoff for most coarse-grained settings, while more complex supervised reduction methods rarely justify their additional cost. Overall, the study provides concrete guidance for when post-hoc supervised projection should, and should not, be inserted into frozen-feature image classification pipelines.

Figures & tables

		RN-18	RN-50	MobV3	EffNet	ViT-B/16	DINOv2
		(512D)	(2048D)	(576D)	(1280D)	(768D)	(384D)
Control	Full	62.85	72.06	65.69	71.58	78.81	80.72
Control	PCA	65.07	69.28	64.62	69.77	76.66	80.60
Classical	LDA	66.97	72.29	68.51	72.30	78.79	82.37
Classical	PCA+LDA	66.63	70.69	67.96	70.79	78.79	82.41
Academic	R-LDA	66.68	71.83	67.93	71.93	78.85	82.25

TABLE I : Classification accuracy (%) on CIFAR-100 across six backbones. The best result per backbone is bolded ; the second best is underlined . All reduction methods project to

d=99

dimensions (

C-1

) except RDA/RDA+SMD which use 99

+k

		RN-18	RN-50	MobV3	EffNet	ViT-B/16	DINOv2
		(512D)	(2048D)	(576D)	(1280D)	(768D)	(384D)
Control	Full	59.82	74.13	59.46	71.67	81.58	78.25
Control	PCA	64.50	73.45	61.94	71.75	80.68	79.83
Classical	LDA	64.28	74.98	63.34	72.32	81.66	79.67
Classical	PCA+LDA	64.47	73.99	63.33	72.11	81.66	79.33
Academic	R-LDA	64.77	75.39	62.87	72.53	81.54	79.65

TABLE II : Classification accuracy (%) on Tiny ImageNet across six backbones. Formatting follows Table I . All reduction methods project to

d=199

dimensions.

		RN-18	RN-50	MobV3	EffNet	ViT-B/16	DINOv2
		(512D)	(2048D)	(576D)	(1280D)	(768D)	(384D)
Control	Full	62.89	64.46	63.01	78.05	75.72	87.73
Control	PCA	55.35	54.71	58.08	73.33	72.01	85.62
Classical	LDA	57.73	57.32	59.30	75.32	73.61	85.61
Classical	PCA+LDA	57.73	58.92	59.30	75.16	73.46	85.67
Academic	R-LDA	56.52	59.46	58.82	74.27	73.33	85.76

TABLE III : Classification accuracy (%) on CUB-200-2011 (fine-grained) across six backbones. Bold : best per backbone. Unlike Tables I – II , full features consistently achieve the highest accuracy. The “Gap” column shows the accuracy loss of the best reduction method relative to Full.

Fig. 1 : Accuracy improvement of LDA over full features across all 18 backbone–dataset configurations. Positive bars (CIFAR-100, Tiny ImageNet) indicate coarse-grained tasks where LDA helps; negative bars (CUB-200) reveal the fine-grained boundary condition.

Backbone	Dataset	Method	$\Delta$ (%)	$p$ -value
ResNet-18	C100	Full	$-$ 4.12	$\mathbf{<.001}$
		PCA	$-$ 1.91	$\mathbf{<.001}$
		DSB	$+$ 0.30	$\mathbf{<.001}$
	TinyIN	Full	$-$ 4.58	$\mathbf{<.001}$
		PCA	$+$ 0.04	.273
		R-LDA	$+$ 0.32	$\mathbf{<.001}$

TABLE IV : Multi-seed statistical significance (5 seeds). Each row compares a method against LDA using a paired

t

-test.

\Delta

: mean accuracy difference (method

-

LDA). Bold

p

-values indicate statistical significance at

\alpha=0.05

Fig. 2 : Accuracy vs. wall-clock time (averaged across backbones) for CIFAR-100 and Tiny ImageNet. Dashed line: Pareto frontier. Full features are dominated—slower and less accurate than LDA.

Backbone	Method	10%	25%	50%	100%
ResNet-18	Full	57.7	60.3	61.6	62.9
	PCA	51.4	58.4	62.5	65.1
	LDA	54.7	58.8	63.9	67.0
	DSB	55.3	58.4	64.2	67.3
ResNet-50	Full	66.8	70.0	71.2	72.0
	PCA	59.2	63.6	66.9	69.1

TABLE V : Accuracy (%) at different training set fractions on CIFAR-100 (3-seed mean). Bold : best method per fraction.

Fig. 3 : Accuracy vs. training set fraction on CIFAR-100 for ResNet-18 (left) and ResNet-50 (right). Full features dominate at 10%, but LDA overtakes between 25–50% and widens the gap at 100%.

Fig. 4 : Accuracy vs. number of projected dimensions (

d

) on CIFAR-100 for LDA and PCA, with full-feature baselines (dashed). Both improve monotonically;

d=C-1=99

is optimal.

Fig. 5 : LDA accuracy gain over Full features across all 18 backbone–dataset configurations. Green cells: LDA helps (coarse-grained). Red cells: LDA hurts (fine-grained CUB-200). The boundary is sharp and consistent across all six backbones.

Dimensionality Reduction

Dimensionality reduction maps high-dimensional data to compact representations for visualization, prediction, compression, and scientific modeling.

See all 145 papers

May 8, 2026

Closed-Form Linear-Probe Dataset Distillation for Pre-trained Vision Models

Dataset distillation compresses a large training set into a small synthetic set that preserves downstream training utility. While most existing methods target training networks from scratch, modern visual transfer learning often uses frozen pre-trained encoders followed by lightweight linear probing. Existing distillation methods for this setting either unroll iterative linear-probe updates with trajectory-based gradient matching, or rely on closed-form formulations originally designed for from-scratch training with neural-tangent-kernel (NTK) approximations. Neither route exploits the fact that frozen-feature linear probing admits a closed-form solution determined directly by the pre-trained features themselves, with no infinite-width approximation and no inner-loop trajectory. We propose Closed-Form Linear-Probe Dataset Distillation (CLP-DD), a bilevel formulation that computes the linear probe induced by the synthetic set with a sample-space kernel ridge solver. The synthetic images are then updated by evaluating this induced classifier on real features through a temperature-scaled softmax cross-entropy, where the classifier columns act as learned class anchors in feature space. We further show that the choice of outer objective is decisive: pairing the closed-form inner solver with a standard MSE outer loss substantially underperforms trajectory-based methods, while the discriminative outer loss closes most of the gap. On ImageNet-100 with four pre-trained backbones, CLP-DD substantially improves over LGM without DSA and approaches LGM with DSA at a fraction of the computational cost. On ImageNet-1K, CLP-DD matches or surpasses LGM with DSA on three of four backbones while running roughly

14\times

faster and using less than one-eighth of the GPU memory.

Bincheng Peng, Guang Li, Ping Liu +2

Hokkaido University · University of Nevada, Reno

Linear Probing Dataset Distillation Transfer Learning

Ask PDF

May 15, 2026

Pretraining Objective Matters in Extreme Low-Data FGVC: A Backbone-Controlled Study

Extreme low-data fine-grained classification is common in expert domains where labeling is expensive, yet practitioners still need principled guidance for selecting pretrained encoders. We study emerald inclusion grading with a custom dataset of labeled images across three classes and ask: under matched backbone capacity, how does pretraining objective affect downstream representation quality? We compare four frozen ViT-B/16 encoders trained with supervised classification, contrastive learning (SigLIP2), masked reconstruction (MAE), and self-distillation (DINOv3), and evaluate them with leave-one-out cross-validation using linear and nonlinear probes. To control statistical noise in the low-N regime, we use permutation testing (N=1000) on macro one-vs-rest AUC. Supervised and contrastive encoders provide the strongest linear separability (logistic AUC: 0.768 and 0.735; SVM AUC: 0.739 and 0.697), while MAE improves under nonlinear probes (XGBoost AUC: 0.713). We find that DINOv3 underperforms across probe families in this domain. These results support a practical recommendation for extreme low-data FGVC: prioritize margin-enforcing pretraining objectives when data scarcity restricts probing to linear decision rules, and consider reconstruction-style encoders when nonlinear classifiers are feasible given dataset constraints.

Alexander Hackett, Srikanth Thudumu, Ginny Fisher +1

Santa Clara University · IAAIR

Contrastive Learning Masked Autoencoders Fine-Grained Image Classification Few-Shot Image Classification Self-Supervised Visual Representation Learning Image Classification

Ask PDF

Apr 1, 2026

Representation Selection via Cross-Model Agreement using Canonical Correlation Analysis

Modern vision pipelines increasingly rely on pretrained image encoders whose representations are reused across tasks and models, yet these representations are often overcomplete and model-specific. We propose a simple, training-free method to improve the efficiency of image representations via a post-hoc canonical correlation analysis (CCA) operator. By leveraging the shared structure between representations produced by two pre-trained image encoders, our method finds linear projections that serve as a principled form of representation selection and dimensionality reduction, retaining shared semantic content while discarding redundant dimensions. Unlike standard dimensionality reduction techniques such as PCA, which operate on a single embedding space, our approach leverages cross-model agreement to guide representation distillation and refinement. The technique allows representations to be reduced by more than 75% in dimensionality with improved downstream performance, or enhanced at fixed dimensionality via post-hoc representation transfer from larger or fine-tuned models. Empirical results on ImageNet-1k, CIFAR-100, MNIST, and additional benchmarks show consistent improvements over both baseline and PCA-projected representations, with accuracy gains of up to 12.6%.

Dylan B. Lewis, Jens Gregor, Hector Santos-Villalobos

University of Tennessee, Knoxville, USA

Canonical Correlation Analysis Representation Learning Visual Representation Learning

Ask PDF

Abstract

Dimensionality Reduction

Explore similar work

Closed-Form Linear-Probe Dataset Distillation for Pre-trained Vision Models

Pretraining Objective Matters in Extreme Low-Data FGVC: A Backbone-Controlled Study

Representation Selection via Cross-Model Agreement using Canonical Correlation Analysis