COLORA: Efficient Fine-Tuning for Convolutional Models with a Study Case on Optical Coherence Tomography Image Classification

Mariano Rivera; Angello Hoyos

cs.CVMay 23, 2025

COLORA: Efficient Fine-Tuning for Convolutional Models with a Study Case on Optical Coherence Tomography Image Classification

Authors: Mariano Rivera, Angello Hoyos

Organizations: Centro de Investigación en Matemáticas, A.C. (CIMAT) Guanajuato, Guanajuato 36023, México

Abstract

We introduce CoLoRA (Convolutional Low-Rank Adaptation), a parameter-efficient fine-tuning method for convolutional neural networks (CNNs). CoLoRA extends LoRA to convolutional layers by decomposing kernel updates into lightweight depthwise and pointwise components. This design reduces the number of trainable convolutional-update parameters by over 80% compared with full convolutional fine-tuning, while allowing the learned updates to be merged into the pretrained convolutional kernels, thereby preserving the original model size and inference complexity. Experiments on MedMNIST datasets, particularly OCTMNISTv2, demonstrate that CoLoRA applied to VGG16 and ResNet50 achieves competitive classification performance while substantially reducing the number of trainable parameters. Comparisons with transfer learning, adapters, BitFit, and convolutional LoRA variants further characterize the trade-offs among predictive performance, trainable parameters, and training cost. Additional experiments on CIFAR-100 and Cats vs. Dogs provide preliminary evidence that the proposed adaptation strategy also transfers to non-medical image-classification tasks. Peak GPU-memory measurements further show that parameter efficiency does not translate directly into proportional training-memory savings, with memory consumption depending strongly on the placement of the adapted convolutional layers. Overall, CoLoRA provides a parameter-efficient and deployment-efficient alternative to full fine-tuning for convolutional models.

Figures & tables

Figure 1: Schematic of Rebuffi’s CNN adapter. A $1\times 1$ adapter after each convolutional block mixes preceding activations for domain-specific adaptation while freezing backbone filters.

Figure 2: (a) LoRA on dense layers. (b) 2D convolutional kernel. (c) depthwise–pointwise factorization of a 2D convolution.

Figure 3: CoLoRA layer: a trainable depthwise–pointwise residual is added to a frozen convolution; learned updates can be merged during training and are fused into the backbone before inference, preserving the original inference graph.

Figure 4: Inception-style CoLoRA: pointwise mixing precedes depthwise spatial filtering.

Figure 5: Operations of the CoLoRA layer (2D): $1\times 1$ pointwise mixing per location followed by per–channel depthwise spatial convolution.

Convolution	Training	Inference
Original	$N_{O}=h\,w\,C\,T$	$N_{O}$
CNN Adapter	$N_{A}=C\,T+2$	$N_{O}+N_{A}$
CoLoRA	$N_{C}=h\,w\,T+C\,T$	$N_{O}$

Table 1: Trainable parameter counts for a convolutional layer under different strategies.

(h,w)

: kernel size;

C

: input channels;

T

: filters. The CNN adapter includes BN (2 trainable parameters).

Figure 6: VGG16–CoLoRA architecture. Each convolution is augmented with a CoLoRA residual; the backbone is initialized from ImageNet.

Figure 7: Example of the images in the OCTMNISTv2 dataset.

Class	Index	Train	Val.	Test	Bal.	Dst.
ChN	1	33,484	2708	250	7754	7040
DME	2	10,213	2708	250	7754	7040
Drusen	3	7754	2708	250	7754	7040
Normal	4	46,026	2708	250	7754	7040

Table 2: Number of images in the subset OCTMNISTv2 Original dataset, and in the Balanced (Bal) and Distilled training sets; see text.

Panel of Figure 8 — Figure 8: Median (solid line), interquartile range (IQR, dark band), and full range (light band) of the training and validation learning curves across the 10 independent VGG16–CoLoRA runs. Model selection is based on validation accuracy; the held-out test set is reserved for the final evaluation of the selected checkpoint.

Panel of Figure 9 — Figure 9: Confusion matrices on the OCTMNISTv2 test set for VGG16–CoLoRA.

Class	AUC ROC	Recall	Precision	F1
ChN	0.992	0.972	0.927	0.949
DME	0.998	0.988	0.950	0.969
Drusen	0.983	0.932	0.971	0.951
Normal	0.999	0.948	0.995	0.971
Avg.	0.993	0.960	0.961	0.960

Table 3: VGG16–CoLoRA: class-wise metrics on the balanced training regime (test set).

Panel of Figure 10 — Figure 10: One-vs-rest ROC curves for VGG16–CoLoRA on OCTMNISTv2.

Figure 11: Sorted predictive entropy of the balanced training samples. The distilled subset retains 7,040 samples per class after removing samples from both extremes of the entropy distribution.

Figure 14: Predictions for highest-entropy samples in Fig. 12 .

Figure 15: Predictions for low-entropy samples in Fig. 13 .

	Transfer Learning				CoLoRA Fine-Tuning
Class	AUC	Rec.	Prec.	F1	AUC	Rec.	Prec.	F1
ChN	0.980	0.960	0.863	0.909	0.992	0.980	0.925	0.951
DME	0.983	0.932	0.955	0.943	0.999	0.972	0.988	0.980
Drusen	0.975	0.824	0.915	0.867	0.990	0.904	0.983	0.941
Normal	0.993	0.948	0.937	0.942	0.998	0.992	0.958	0.975
Avg.	0.983	0.916	0.918	0.915	0.995	0.962	0.963	0.962

Table 4: VGG16—Transfer vs. VGG16—CoLoRA on the distilled training set (test metrics).

Method	AUC	Acc.	Params.
ResNet-18 (28) [ 25 ]	0.951	0.758
ResNet-50 (224) [ 25 ]	0.951	0.750
Dedicated CNN (28) [ 43 ]		0.760
ResNet-50 (224) [ 43 ]		0.776
MedViTv1-T [ 27 ]	0.961	0.767	15.2M
MedViTv1-S [ 27 ]	0.960	0.782

Table 5: Comparison on OCTMNISTv2. † 14.7M parameters trained by backprop out of 37.7M total. ‡ 2.54M trained by backprop out of 15.6M total.

CoLoRA	Trainable	Avg. epoch	Test Acc.	Max. Test
Placement	Params.	Time (min)	Mean $\pm$ SD	Acc.
1 to 5	2.55M	4.42	$0.956\pm 0.003$	0.962
2 to 5	2.54M	3.00	$\bf{0.959}\pm 0.003$	0.966
3 to 5	2.51M	2.79	$0.957\pm 0.002$	0.961
4 to 5	2.34M	2.15	$0.940\pm 0.003$	0.957
Only 5	1.67M	1.64	$0.941\pm 0.002$	0.949

Table 6: Effect of CoLoRA layer placement on VGG16 performance and training efficiency. Test accuracy is reported as mean

\pm

standard deviation together with the maximum observed accuracy. The pointwise-only CoLoRA variant, Transfer Learning, and Full Fine-Tuning are included as reference baselines.

Figure 16: Mean test accuracy, standard deviation, and maximum test accuracy across the evaluated CoLoRA placement configurations and VGG16 baselines. The labels $1x1$ , TL and FT corresponds to remove the dephwise kernel in the 1-5 configuration, perform an standard transfer learning procedure, and achive a full-tuning strategy; respectively.

RetinaMNIST	AUC	Acc.	Params.
Baseline	0.750	0.531
MedMamba-T [ 30 ]	0.741	0.543	15.2M
MedMamba-S [ 30 ]	0.718	0.545	23.5M
MedMamba-B [ 30 ]	0.715	0.553	48.1M
MedMamba-X [ 30 ]	0.719	0.570
MedViTv2-T [ 28 ]	0.761	0.547

Table 7: Comparison on RetinaMNIST, PathMNIST, and BloodMNIST datasets. ‡ 2.6M trained by backprop out of 15.6M total.

	Method	Avg epoch Time Mins	Test AUC	Test Acc.	Trained Params.
	Baseline		0.958	0.776
Balanced dataset	Transfer Learning	1.32	0.979	0.933	0.87M
	Adapters	1.56	0.995	0.957	1.5M
	BitFit	1.71	0.979	0.896	0.87M
	Conv-LoRA R4 $\alpha 8$	2.10	0.991	0.944	0.90M
	Conv-LoRA R8 $\alpha 16$	2.25	0.990	0.943	0.93M

Table 8: Performance comparison between PEFT methods and CoLoRA in a VGG16 architecture on balanced and distilled OCTMNISTv2.

Method	AUC	Top-1	Top-5	Params.
Transfer Learning	$0.889\pm 0.004$	$0.258\pm 0.002$	$0.535\pm 0.004$	0.09M
Adapters	$0.942\pm 0.004$	$0.449\pm 0.004$	$0.730\pm 0.008$	0.74M
BitFit	$0.914\pm 0.003$	$0.327\pm 0.003$	$0.613\pm 0.003$	0.09M
Conv-LoRA	$0.947\pm 0.002$	$0.424\pm 0.006$	$0.732\pm 0.008$	0.15M
CoLoRA block 2-5	$\bf{0.961}\pm 0.001$	$\bf{0.451}\pm 0.004$	$\bf{0.766}\pm 0.003$	1.77M
CoLoRA block 5	$0.911\pm 0.004$	$0.324\pm 0.003$	$0.621\pm 0.009$	0.89M

Table 9: Performance comparison on CIFAR-100. Values are reported as mean

\pm

standard deviation across the independent runs.

Method	AUC	Top-1	Top-5	Params.
Transfer Learning	$0.737\pm 0.009$	$0.087\pm 0.008$	$0.210\pm 0.009$	0.90M
Conv-LoRA	$0.697\pm 0.016$	$0.071\pm 0.009$	$0.174\pm 0.014$	0.95M
CoLoRA block 2-5	$\bf{0.754}\pm 0.008$	$\bf{0.109}\pm 0.005$	$\bf{0.241}\pm 0.012$	2.57M

Table 10: Performance comparison on the class-balanced ImageNet-R subset using 35 training, 5 validation, and 10 test images per class. Values are reported as mean

\pm

standard deviation across 10 independent runs.

Explore similar work

Jun 25, 2025cs.CV

WaRA: Wavelet Low-Rank Adaptation for Medical Image Classification

Adapting large pretrained vision models to medical image classification is often limited by memory, computation, and task-specific specializations. Parameter-efficient fine-tuning (PEFT) methods like LoRA reduce this cost by learning low-rank updates, but operating directly in feature space can struggle to capture the localized, multi-scale features common in medical imaging. We propose WaRA, a wavelet-structured adaptation module that performs low-rank adaptation in a wavelet domain. WaRA reshapes patch tokens into a spatial grid, applies a fixed discrete wavelet transform, updates subband coefficients using a shared low-rank adapter, and reconstructs the additive update through an inverse wavelet transform. This design provides a compact trainable interface while biasing the update toward both coarse structure and fine detail. For extremely low-resource settings, we introduce Tiny-WaRA, which further reduces trainable parameters by learning only a small set of coefficients in a fixed basis derived from the pretrained weights through a truncated SVD. Experiments on medical image classification across four modalities and datasets demonstrate that WaRA consistently improves performance over strong PEFT baselines, while retaining a favorable efficiency profile. Our code is publicly available at https://github.com/moeinheidari7829/WaRA.

Moein Heidari, Yijin Huang, Yasamin Medghalchi +3

School of Biomedical Engineering, The University of British Columbia, Vancouver, BC, Canada · Department of Electronic and Electrical Engineering, Southern University of Science and Technology, Shenzhen, China · Department of Electrical and Computer Engineering, The University of British Columbia, Vancouver, BC, Canada +2

Medical Image Classification Low-Rank Adaptation Adapters Wavelets

Ask PDF

Oct 6, 2026cs.LG

MemFLoRA: Memory-Floor LoRA for CNN Adaptation at the Edge

On-device learning is necessary when the model encounters user-,sensor-, or environment-specific shifts after deployment. Although parameter-efficient fine-tuning (PEFT) methods, particularly Low-Rank Adaptation (LoRA) variants, enable efficient adaptation at the edge, the limiting resource for Convolutional Neural Network (CNN) adaptation is often not the number of trainable parameters but the activation state that must be retained until the backward pass. This paper introduces Memory-Floor LoRA (MemFLoRA), a low-rank CNN adapter built around a memory-first design principle rather than a direct application of transformer-oriented LoRA. Instead of merely reducing trainable weights, we define an activation-memory-floor criterion: trainable backward computations must not depend on full-width layer inputs. The resulting adapter freezes the down-projection, trains a scale-matched up-projection, and combines eval-mode backbone normalization with activation-minimal backward rules, reducing saved state to the low-rank branch. Evaluated on three Human Activity Recognition (HAR) datasets and two CNN backbones under subject, body-location, and sensor-placement shifts, MemFLoRA reduces saved-activation memory by 98.5-98.7% and peak training-state memory by 94.9-97.3% relative to full fine-tuning, while matching or exceeding CNN PEFT baselines.

Mehmet Emre Akbulut, Johannes Geier, Ulf Schlichtmann

Technical University of Munich Munich, Germany

Model Activations

Ask PDF

Jul 8, 2026cs.CV

LoCA: Spatially-Aware Low-Rank Convolutional Adaptation of Vision Foundation Models

Pre-trained Vision Foundation Models (VFMs) provide strong visual representations for diverse downstream tasks. The key challenge of VFM adaptation stems from the prohibitive costs of full fine-tuning and catastrophic forgetting. To address this, Low-Rank Adaptation (LoRA) has emerged as the prevailing paradigm for Parameter-Efficient Fine-Tuning (PEFT). However, LoRA is typically designed for transformer self-attention layers parameterized by 2D matrices. Since convolutional kernels inherently couple spatial and channel information within a 4D tensor, forcing them into a monolithic 2D matrix disrupts the inherent spatial topology. In this paper, we propose Low-Rank Convolutional Adaptation (LoCA), a convolution-aware PEFT framework that addresses spatial-channel entanglement by decoupling channel and spatial adaptation. LoCA introduces a low-rank channel adaptation for dense cross-channel mixing and refines spatial bases extracted from pre-trained kernels via Singular Value Decomposition (SVD). Experimental results show that LoCA preserves pre-trained spatial priors and achieves competitive or state-of-the-art performance across fine-grained classification, domain-generalized semantic segmentation, and generative benchmarks.

Sojung An, Junha Lee, Sujeong You +2

Korea University, Seoul, Republic of Korea · Korea Institute of Industrial Technology, Ansan, Republic of Korea · Seoul National University, Seoul, Republic of Korea

Parameter-Efficient Fine-Tuning Methods Vision Foundation Models Entanglement Design Structure Matrices

Ask PDF