Quantum Adaptive Self-Attention for Quantum Transformer Models

Chi-Sheng Chen; En-Jui Kuo

quant-phApr 5, 2025

Quantum Adaptive Self-Attention for Quantum Transformer Models

Authors: Chi-Sheng Chen, En-Jui Kuo

Organizations: Department of Electrophysics, National Yang Ming Chiao Tung University, Hsinchu, Taiwan

Abstract

A recurring weakness in quantum machine learning (QML) is that reported ``quantum advantages'' are seldom tested against a \emph{capacity-matched} classical control, leaving it unclear whether a gain comes from the quantum substrate or from the architectural change that accompanies it. Our primary contribution is methodological: a protocol for attributing such gains honestly -- a capacity-matched classical bottleneck of identical parameter budget, transparent reporting of where quantum does \emph{not} help, and validation on real quantum hardware -- which we develop and apply through a concrete case study. That case study is Quantum Adaptive Self-Attention (QASA), a hybrid Transformer that replaces the value projection of a \emph{single} encoder layer with a 36-parameter parameterized quantum circuit (PQC), keeping all other layers classical. Across nine synthetic benchmarks and the real-world ETTh1 dataset, QASA improves on a full-capacity classical Transformer for chaotic and trend-dominated signals. To ask whether this is a genuinely \emph{quantum} effect, we introduce a control rarely applied in quantum machine learning -- a capacity-matched classical bottleneck with the same parameter budget -- and find that it matches the PQC on the error metrics. The gain is therefore attributable to the low-rank value-projection \emph{bottleneck} (an \emph{architectural parsimony} principle), not to quantumness; adding further quantum layers only degrades performance and trainability. We accordingly position the quantum layer not as a source of accuracy advantage but as a \emph{competitive} instantiation of this principle: its low-rank compression onto the signal's intrinsic dimensionality is matched by a classical bottleneck, so the gain is architectural rather than quantum.

Explore similar work

Aug 7, 2026quant-ph

Investigating Quantum-Embedded Transformers on Classical Datasets for Cross-Modality Classification

We test whether a parameterized quantum circuit (PQC) improves a hybrid quantum-classical model's performance on classical datasets, using an interface-matched classical map as the control while holding all other components fixed. Our architecture, Quantum-Embedded Attention (QEA), uses a learnable projector to compress backbone features into an

n_q

-dimensional angle vector, a shallow PQC to map those angles to one- and two-qubit Pauli expectations, and a classical attention decoder to produce class logits. We hypothesized the PQC would improve accuracy or seed-to-seed stability over a classical map with matched input/output dimensions. We test this with an interface-matched

2\times2

factorial on Breast Cancer Wisconsin at

n_q\in\{4,8\}

, independently swapping the PQC for a classical map and the attention decoder for a linear head, across five paired seeds per cell. Three of four paired quantum-minus-classical

95\%

confidence intervals include zero; the fourth, a

+1.63

percentage-point contrast for the attention decoder at

n_q=4

, reverses sign at

n_q=8

and does not survive correction across the four contrasts. The experiment thus shows no consistent PQC contribution and cannot establish equivalence. A five-dataset cross-modality grid shows comparable accuracy on AG~News, Breast Cancer Wisconsin, and BirdCLEF but a large deficit on CIFAR-10; these cells are not interface-matched and are interpreted descriptively. We report all planned canonical runs, distinguish current Pauli-readout results from legacy probability-readout experiments, and analyze bottleneck, simulation, finite-shot, and noise limitations. The results do not establish a quantum advantage; they demonstrate why controlled component attribution is necessary before crediting a hybrid model's performance to its quantum layer.

Hao-Yuan Chen

Hybrid Quantum Neural Networks Multimodal Classification Cross-Modality Transformer Architectures Readout Mapping

Ask PDF

Apr 27, 2026quant-ph

Do Quantum Transformers Help? A Systematic VQC Architecture Comparison on Tabular Benchmarks

Variational quantum circuits (VQCs) are a leading approach to quantum machine learning on near-term devices, yet it remains unclear which circuit architecture yields the best accuracy-parameter trade-off on classical tabular data. We present a systematic empirical comparison of four VQC families -- multi-layer fully-connected (FC-VQC), residual (ResNet-VQC), hybrid quantum-classical transformer (QT), and fully quantum transformer (FQT) -- across five regression and classification benchmarks. Our key findings are: \textbf{(i)}~FC-VQCs achieve 90-96% of the

R^2

of attention-based VQCs while using 40-50% fewer parameters, and consistently outperform equal-capacity MLPs (mean

R^2{=}0.829

vs.\ MLP

_{720}

0.753

on Boston Housing, 3-seed average); \textbf{(ii)}~~FC-VQC's Type~~4 inter-block connectivity provides partial cross-token mixing that approximates the role of attention -- explicit quantum self-attention yields only marginal gains on most datasets while significantly increasing parameter count; \textbf{(iii)}~~expressibility saturates at circuit depth~~

{\approx}\,3

, explaining why shallow VQCs already cover the Hilbert space effectively; \textbf{(iv)}~LayerNorm on the fully quantum transformer improves classification accuracy, suggesting normalization is important when all operations are quantum; \textbf{(v)}~in our noise study on Boston Housing, FQT degrades gracefully under depolarizing noise while QT collapses. All results are validated across three random seeds. These findings provide practical architectural guidance for deploying VQCs on near-term quantum hardware.

Chi-Sheng Chen, En-Jui Kuo

Variational Quantum Circuits Quantum Machine Learning Variational Autoencoder

Ask PDF

Sep 15, 2026quant-ph

QiT: Quantum-Inspired Transformer for Visual Recognition Task

Quantum machine learning offers a compelling representational perspective: angle-encoded states inhabit Hilbert spaces in which periodic similarities and interactions can be expressed naturally. Realizing this perspective for visual recognition remains difficult, however, because present quantum neural networks are constrained by limited qubit counts, costly circuit simulation and measurement, noise, and unstable optimization on noisy intermediate-scale quantum devices. We investigate whether useful structural ideas from quantum models can instead be realized as scalable classical Transformer operations. We introduce QiT, a Quantum-inspired Transformer for vision tasks with three components: (i) angle-inspired encoding that maps image tokens to learned trigonometric Hilbert-space features analogous to quantum rotation-based state encoding; (ii) self-attention over these periodic features, inducing a classical cosine kernel approximated to quantum fidelity kernels; and (iii) gated multiplicative emulation, a trainable classical surrogate for interaction terms found in variational circuits. All components are differentiable tensor operations, so QiT claims neither quantum computation nor quantum speedup and retains the

\mathcal{O}(N^2D)

attention complexity of a standard Vision Transformer. Across image-classification benchmarks, QiT is competitive with a matched classical Transformer while avoiding the severe runtime cost observed for a small simulated quantum Transformer. QiT-B reaches 78.3% ImageNet-1K top-1 accuracy with 45.7M parameters and 11.5 GFLOPs. These results position QiT as a scalable baseline for isolating and evaluating quantum-motivated inductive biases in visual recognition.

Badri N. Patro, Vijay Agneeswaran

Self-Supervised Vision Transformers Quantum Machine Learning Transformer Architectures

Ask PDF