QINA: Quantum-Inspired Nonlinear Adapters for Pretrained Vision Models

Mostafa Mehdipour Ghazi

cs.CVSep 1, 2026

QINA: Quantum-Inspired Nonlinear Adapters for Pretrained Vision Models

Authors: Mostafa Mehdipour Ghazi

Organizations: Pioneer Centre for AI University of Copenhagen

Abstract

Adapting large pretrained vision models under limited data and frozen-backbone constraints remains a central challenge in transfer learning. While lightweight adapters and parameter-efficient fine-tuning methods are widely adopted, most rely on generic multilayer perceptrons or low-rank linear updates, offering limited control over the spectral and geometric structure of feature transformations. We investigate whether structured nonlinear feature lifting can improve representational alignment in frozen regimes. We introduce Quantum-Inspired Nonlinear Adapters (QINA), compact modules that perform learnable trigonometric feature lifting followed by bounded nonlinear aggregation. The design induces structured oscillatory basis functions with an explicit norm-dependent Lipschitz bound, enabling spectral reshaping of pretrained representations without increasing the receptive field or significantly expanding parameter count. Importantly, the method operates entirely within standard deep learning frameworks and does not require quantum hardware. Through systematic experiments across natural and medical imaging datasets, classification and segmentation tasks, multiple adapters and placements, and varying training budgets, we show that performance in frozen regimes is primarily representation-limited. Nonlinear lifting improves adaptation, and the proposed structured trigonometric formulation consistently outperforms identity baselines, fixed Fourier feature mappings, and parameter-matched baseline adapters. Within the evaluated frozen-backbone settings, structured spectral parameterization provides a more effective inductive bias than generic nonlinear adapters. This work highlights the importance of geometry- and spectrum-aware adaptation mechanisms for large pretrained vision models.

Figures & tables

Figure 1: Structured spectral adaptation via QINA in frozen backbones. Left: Internal architecture of the QINA module, consisting of linear encoding, trigonometric lifting, linear decoding, and bounded nonlinearity. Middle: In residual CNN blocks, QINA operates within the residual branch on intermediate feature maps, producing a bounded correction that is added to the shortcut connection while keeping the backbone parameters frozen. Right: In Transformer encoder layers, QINA is inserted before MHA in the first residual block.

Dataset	Domain	Task	Dimension	# Class	# Train
STL-10	Natural scene	Classification	96 $\times$ 96	10	5,000
BRISC	Brain tumor	Classification	486 $\times$ 475	4	5,000
ISIC	Skin lesion	Classification	905 $\times$ 936	9	8,238
FETAL	Fetal plane	Classification	548 $\times$ 822	6	7,129
BUSI	Breast tumor	Classification	501 $\times$ 616	3	780
BRISC	Brain tumor	Segmentation	506 $\times$ 505	2	3,933

Table 1: Summary of the datasets used in the study. The reported image resolutions correspond to average values due to variations across samples within each dataset.

Panel of Figure 2 — Figure 2: Representative samples from natural and medical datasets. Natural images exhibit diverse color and semantic structure, whereas medical image modalities (MRI, ultrasound, dermoscopy, endoscopy) show distinct intensity statistics, texture patterns, and localized cues.

Figure 3: Validation performance on STL-10 under frozen-backbone training across different classification backbones and training budgets. QINA consistently improves performance across models and budgets.

Panel of Figure 4 — Figure 4: Validation performance on medical classification datasets under frozen-backbone training across different CNN backbones and training budgets. QINA consistently improves performance across models, datasets, and budgets.

Figure 5: BUSI classification performance using a frozen ImageNet-pretrained ResNet versus a frozen modality-adapted ResNet fine-tuned on FETAL ultrasound data. Although the tasks differ, modality alignment significantly improves downstream performance under freezing.

Panel of Figure 6 — Figure 6: Validation performance (foreground Dice) on medical segmentation datasets using DeepLabv3+ with a frozen pretrained encoder. Consistent improvements across budgets suggest that structured nonlinear adaptation benefits dense prediction tasks.

Figure 7: Ablations on STL-10 evaluating QINA placement (left), hidden dimension (middle), and full fine-tuning (right). Higher-layer insertion and larger hidden dimension improve accuracy in the frozen regime, whereas end-to-end fine-tuning reduces adapter-specific gains.

Figure 8: Representation analysis of adapter features using a frozen ResNet-18 backbone on the STL-10 validation set. Adapters are compared using Fisher Discriminant Ratio (FDR), Effective Rank (ER), and Centered Kernel Alignment (CKA). QINA achieves higher FDR, indicating improved class separability, while exhibiting lower ER, suggesting more compact feature utilization. Lower CKA relative to the frozen backbone and alternative adapters indicates that QINA introduces a distinct transformation of the representation geometry.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: QINA component ablations on STL-10 with ResNet-18. Left: sine-only, cosine-only, and combined sine–cosine encodings yield comparable performance, with the combined encoding providing a small advantage. Right: the bounded

\tanh

output performs slightly better than a linear output across the evaluated training budgets.

Adapter	Component	Parameters	Activation	Learnable params
QINA	Encoder	$W:256\times 128,\;b:128$	$\sin+\cos$	65,920
QINA	Decoder	$W:128\times 256,\;b:256$	$\tanh$	65,920
MLP	Encoder	$W:256\times 128,\;b:128$	identity / ReLU / tanh	65,920
MLP	Decoder	$W:128\times 256,\;b:256$	identity	65,920
Fourier	Encoder	$W:256\times 128,\;b:128$ (fixed)	$\sin$	33,024
Fourier	Decoder	$W:128\times 256,\;b:256$	identity	33,024

Appendix

Table 2: Adapter configurations used in the ResNet-18 comparisons. The adapter was applied to Residual Block 5a . Weight dimensions are reported as input

\times

output; bias dimensions indicate the number of output units. Trainable parameter counts exclude the task head. For Fourier features, the randomly initialized encoder parameters are fixed during training.

Adapter	Component	Parameters	Activation	Learnable params
QINA	Encoder	$W:1024\times 128,\;b:128$	$\sin+\cos$	263,296
QINA	Decoder	$W:128\times 1024,\;b:1024$	$\tanh$	263,296
MLP	Encoder	$W:1024\times 128,\;b:128$	identity / ReLU / tanh	263,296
MLP	Decoder	$W:128\times 1024,\;b:1024$	identity	263,296
Fourier	Encoder	$W:1024\times 128,\;b:128$ (fixed)	$\sin$	132,096
Fourier	Decoder	$W:128\times 1024,\;b:1024$	identity	132,096

Appendix

Table 3: Adapter configurations used in the ResNet-50 comparisons. The adapter was applied to Residual Block 5a . Weight dimensions are reported as input

\times

Adapter	Component	Parameters	Activation	Learnable params
QINA	Encoder	$W:728\times 128,\;b:128$	$\sin+\cos$	187,224
QINA	Decoder	$W:128\times 728,\;b:728$	$\tanh$	187,224
MLP	Encoder	$W:728\times 128,\;b:128$	identity / ReLU / tanh	187,224
MLP	Decoder	$W:128\times 728,\;b:728$	identity	187,224
Fourier	Encoder	$W:728\times 128,\;b:128$ (fixed)	$\sin$	93,912
Fourier	Decoder	$W:128\times 728,\;b:728$	identity	93,912

Appendix

Table 4: Adapter configurations used in the Xception comparisons. The adapter was applied to Block 13 . Weight dimensions are reported as input

\times

Adapter	Component	Parameters	Activation	Learnable params
QINA	Encoder	$W:192\times 128,\;b:128$	$\sin+\cos$	49,472
QINA	Decoder	$W:128\times 192,\;b:192$	$\tanh$	49,472
MLP	Encoder	$W:192\times 128,\;b:128$	identity / ReLU / tanh	49,472
MLP	Decoder	$W:128\times 192,\;b:192$	identity	49,472
Fourier	Encoder	$W:192\times 128,\;b:128$ (fixed)	$\sin$	24,768
Fourier	Decoder	$W:128\times 192,\;b:192$	identity	24,768

Appendix

Table 5: Adapter configurations used in the ViT-Tiny comparisons. The adapter was applied to Block 11 . Weight dimensions are reported as input

\times

Training Budget	10	20	50	100	200	500
	ResNet-18
QINA	0.228 $\pm$ 0.054	0.304 $\pm$ 0.048	0.414 $\pm$ 0.039	0.590 $\pm$ 0.025	0.693 $\pm$ 0.040	0.744 $\pm$ 0.022
Identity	0.114 $\pm$ 0.027	0.116 $\pm$ 0.026	0.124 $\pm$ 0.018	0.272 $\pm$ 0.033	0.396 $\pm$ 0.047	0.402 $\pm$ 0.034
Fourier	0.176 $\pm$ 0.037	0.241 $\pm$ 0.039	0.293 $\pm$ 0.043	0.496 $\pm$ 0.044	0.603 $\pm$ 0.037	0.639 $\pm$ 0.024
MLP-Linear	0.165 $\pm$ 0.023	0.210 $\pm$ 0.044	0.221 $\pm$ 0.035	0.446 $\pm$ 0.040	0.604 $\pm$ 0.039	0.619 $\pm$ 0.031
MLP-ReLU	0.159 $\pm$ 0.027	0.190 $\pm$ 0.042	0.199 $\pm$ 0.029	0.417 $\pm$ 0.038	0.586 $\pm$ 0.041	0.607 $\pm$ 0.034

Appendix

Table 6: Validation F1-score (mean

\pm

SD) on STL-10 under frozen-backbone training across different classification backbones and training budgets.

Training Budget	10	20	50	100	200	500
	ResNet-18
QINA	0.248 $\pm$ 0.052	0.328 $\pm$ 0.056	0.431 $\pm$ 0.032	0.604 $\pm$ 0.029	0.703 $\pm$ 0.037	0.753 $\pm$ 0.021
Identity	0.152 $\pm$ 0.044	0.151 $\pm$ 0.049	0.155 $\pm$ 0.035	0.303 $\pm$ 0.034	0.418 $\pm$ 0.039	0.425 $\pm$ 0.031
Fourier	0.196 $\pm$ 0.039	0.258 $\pm$ 0.050	0.309 $\pm$ 0.036	0.509 $\pm$ 0.044	0.614 $\pm$ 0.036	0.650 $\pm$ 0.025
MLP-Linear	0.189 $\pm$ 0.034	0.226 $\pm$ 0.044	0.239 $\pm$ 0.043	0.462 $\pm$ 0.036	0.614 $\pm$ 0.035	0.630 $\pm$ 0.027
MLP-ReLU	0.178 $\pm$ 0.036	0.209 $\pm$ 0.040	0.220 $\pm$ 0.042	0.436 $\pm$ 0.035	0.595 $\pm$ 0.037	0.617 $\pm$ 0.033

Appendix

Table 7: Validation accuracy (mean

\pm

SD) on STL-10 under frozen-backbone training across different classification backbones and training budgets.

Training Budget	10	20	50	100	200	500
	BRISC
QINA	0.502 $\pm$ 0.085	0.619 $\pm$ 0.086	0.697 $\pm$ 0.038	0.873 $\pm$ 0.016	0.933 $\pm$ 0.020	0.949 $\pm$ 0.012
Identity	0.230 $\pm$ 0.052	0.251 $\pm$ 0.065	0.262 $\pm$ 0.046	0.322 $\pm$ 0.089	0.410 $\pm$ 0.076	0.389 $\pm$ 0.070
Fourier	0.412 $\pm$ 0.073	0.465 $\pm$ 0.095	0.508 $\pm$ 0.081	0.728 $\pm$ 0.041	0.811 $\pm$ 0.034	0.826 $\pm$ 0.022
MLP-Linear	0.418 $\pm$ 0.060	0.506 $\pm$ 0.073	0.529 $\pm$ 0.073	0.759 $\pm$ 0.029	0.831 $\pm$ 0.025	0.834 $\pm$ 0.021
MLP-ReLU	0.370 $\pm$ 0.052	0.461 $\pm$ 0.079	0.473 $\pm$ 0.060	0.725 $\pm$ 0.018	0.815 $\pm$ 0.029	0.828 $\pm$ 0.015

Appendix

Table 8: Validation F1-score (mean

\pm

SD) on medical datasets under frozen-ResNet-18 training across different training budgets.

Training Budget	10	20	50	100	200	500
	BRISC
QINA	0.534 $\pm$ 0.072	0.637 $\pm$ 0.077	0.714 $\pm$ 0.033	0.876 $\pm$ 0.012	0.933 $\pm$ 0.020	0.949 $\pm$ 0.011
Identity	0.278 $\pm$ 0.047	0.296 $\pm$ 0.047	0.303 $\pm$ 0.042	0.354 $\pm$ 0.073	0.439 $\pm$ 0.061	0.423 $\pm$ 0.052
Fourier	0.449 $\pm$ 0.051	0.500 $\pm$ 0.081	0.545 $\pm$ 0.064	0.736 $\pm$ 0.039	0.814 $\pm$ 0.036	0.829 $\pm$ 0.023
MLP-Linear	0.450 $\pm$ 0.072	0.524 $\pm$ 0.053	0.548 $\pm$ 0.059	0.763 $\pm$ 0.027	0.833 $\pm$ 0.024	0.837 $\pm$ 0.019
MLP-ReLU	0.409 $\pm$ 0.061	0.488 $\pm$ 0.053	0.507 $\pm$ 0.050	0.730 $\pm$ 0.020	0.818 $\pm$ 0.027	0.829 $\pm$ 0.016

Appendix

Table 9: Validation accuracy (mean

\pm

SD) on medical datasets under frozen-ResNet-18 training across different training budgets.

Training Budget	5	10	20	50	100	200	500
	BRISC
QINA	0.021 $\pm$ 0.015	0.082 $\pm$ 0.054	0.168 $\pm$ 0.079	0.400 $\pm$ 0.043	0.405 $\pm$ 0.069	0.492 $\pm$ 0.105	0.533 $\pm$ 0.072
Identity	0.012 $\pm$ 0.009	0.074 $\pm$ 0.052	0.147 $\pm$ 0.058	0.330 $\pm$ 0.036	0.375 $\pm$ 0.026	0.435 $\pm$ 0.052	0.435 $\pm$ 0.064
Fourier	0.020 $\pm$ 0.013	0.071 $\pm$ 0.050	0.169 $\pm$ 0.063	0.321 $\pm$ 0.031	0.341 $\pm$ 0.063	0.436 $\pm$ 0.104	0.485 $\pm$ 0.049
MLP-Linear	0.025 $\pm$ 0.025	0.090 $\pm$ 0.055	0.202 $\pm$ 0.041	0.275 $\pm$ 0.091	0.348 $\pm$ 0.046	0.467 $\pm$ 0.064	0.434 $\pm$ 0.102
MLP-ReLU	0.019 $\pm$ 0.018	0.088 $\pm$ 0.042	0.182 $\pm$ 0.057	0.254 $\pm$ 0.071	0.362 $\pm$ 0.048	0.446 $\pm$ 0.063	0.413 $\pm$ 0.102

Appendix

Table 10: Validation Dice (mean

\pm

SD) on medical datasets using DeepLabv3+ with a frozen ResNet-18 across different training budgets.

Training Budget	5	10	20	50	100	200	500
	BRISC
QINA	0.013 $\pm$ 0.009	0.055 $\pm$ 0.036	0.120 $\pm$ 0.060	0.304 $\pm$ 0.033	0.309 $\pm$ 0.054	0.395 $\pm$ 0.095	0.423 $\pm$ 0.061
Identity	0.008 $\pm$ 0.006	0.049 $\pm$ 0.034	0.105 $\pm$ 0.044	0.250 $\pm$ 0.029	0.290 $\pm$ 0.023	0.341 $\pm$ 0.047	0.341 $\pm$ 0.054
Fourier	0.012 $\pm$ 0.008	0.047 $\pm$ 0.033	0.119 $\pm$ 0.048	0.239 $\pm$ 0.029	0.259 $\pm$ 0.049	0.346 $\pm$ 0.092	0.381 $\pm$ 0.039
MLP-Linear	0.017 $\pm$ 0.017	0.061 $\pm$ 0.038	0.146 $\pm$ 0.031	0.205 $\pm$ 0.069	0.264 $\pm$ 0.036	0.369 $\pm$ 0.054	0.343 $\pm$ 0.087
MLP-ReLU	0.012 $\pm$ 0.012	0.060 $\pm$ 0.029	0.131 $\pm$ 0.043	0.188 $\pm$ 0.054	0.276 $\pm$ 0.041	0.351 $\pm$ 0.055	0.326 $\pm$ 0.086

Appendix

Table 11: Validation IoU (mean

\pm

SD) on medical datasets using DeepLabv3+ with a frozen ResNet-18 across different training budgets.

Vision Foundation Model Adaptation

Vision foundation model adaptation tailors pretrained image and video models to new tasks and domains while balancing accuracy, efficiency, and generalization.

See all 326 papers

AdapterTune: Zero-Initialized Low-Rank Adapters for Frozen Vision Transformers
Mar 16, 2026Salim KhazemAdapter Tuning Low-Rank Adaptation
Talan Research Center, Paris, France
SQUARE: Structured Quantum Representation Adapters as Compact Quadratic Feature Maps for Frozen Language Models
Sep 29, 2026Emily Jimin Roh, Hyojun Ahn, Hoyeong Lee +4Quantum Feature Maps Representation Learning
Korea University · Sookmyung Women’s University · Ulsan National Institute of Science and Technology +2
Low-Rank Adaptation of Frozen Vision-Language Models for Blind Image Quality Assessment
Jun 1, 2026Bishr Omer Adam, Xu LiVLM Adaptation Low-Rank Adaptation
School of Electronics and Information, Northwestern Polytechnical University, Xi’an, China

Mar 16, 2026J/K move · Enter open · S save

AdapterTune: Zero-Initialized Low-Rank Adapters for Frozen Vision Transformers

Salim Khazem

Talan Research Center, Paris, France

Adapter Tuning Low-Rank Adaptation Parameter-Efficient Fine-Tuning Vision Transformer

Open paper AskPDF arXiv

Frozen-backbone transfer with Vision Transformers faces two under-addressed issues: optimization instability when adapters are naively inserted into a fixed feature extractor, and the absence of principled guidance for setting adapter capacity. We introduce AdapterTune, which augments each transformer block with a residual low-rank bottleneck whose up-projection is zero-initialized, guaranteeing that the adapted network starts exactly at the pretrained function and eliminates early-epoch representation drift. On the analytical side, we formalize adapter rank as a capacity budget for approximating downstream task shifts in feature space. The resulting excess-risk decomposition predicts monotonic but diminishing accuracy gains with increasing rank, an ``elbow'' behavior we confirm through controlled sweeps. We evaluate on 9 datasets and 3 backbone scales with multi-seed reporting throughout. On a core 5 dataset transfer suite, AdapterTune improves top-1 accuracy over head-only transfer by +14.9 points on average while training only 0.92 of the parameters required by full fine-tuning, and outperforms full fine-tuning on 10 of 15 dataset-backbone pairs. Across the full benchmark, AdapterTune improves over head-only transfer on every dataset-backbone pair tested. Ablations on rank, placement, and initialization isolate each design choice. The code is available at: https://github.com/salimkhazem/adaptertune

Abstract

Vision Foundation Model Adaptation

Explore similar work

AdapterTune: Zero-Initialized Low-Rank Adapters for Frozen Vision Transformers

SQUARE: Structured Quantum Representation Adapters as Compact Quadratic Feature Maps for Frozen Language Models

Low-Rank Adaptation of Frozen Vision-Language Models for Blind Image Quality Assessment