Can Quantum Layers offer Performance and Security Advantages in Vision-Language Models?

Md Khalid Syfullah; Alvi Ataur Khalil

quant-phOct 20, 2026

Can Quantum Layers offer Performance and Security Advantages in Vision-Language Models?

Authors: Md Khalid Syfullah, Alvi Ataur Khalil

Organizations: Computer Science, Southern Illinois University Carbondale, USA

Abstract

As quantum computing moves from the noisy intermediate-scale quantum (NISQ) era toward early fault-tolerant quantum computing (EFTQC), quantum-enhanced AI is shifting from a conceptual idea to experimental research. Quantum components are expected to extend the capabilities of existing AI systems, yet it remains unclear how such components will help within large multimodal models. In this study, we try to answer that question for vision-language models (VLMs) using classically simulated quantum layers. With LLaVA-OneVision as the backbone, we evaluate 4- and 8-qubit basic-ring and strongly entangling circuits at four insertion and four replacement locations. Insertion adds quantum layers to the classical model, while replacement substitutes them for classical components. Using the same setup, we compare performance and computational cost between classical and quantum-classical hybrid models. On the Visual Spatial Reasoning dataset, inserting a quantum layer after the visual projector reaches 80.7% development accuracy, 2.28 percentage-points above classical full fine-tuning. Replacing the final decoder feed-forward sublayer retains 79.22% accuracy with 13.07 million fewer trainable parameters. Quantum insertion can thus improve accuracy, and selective replacement can reduce parameters. These benefits depend strongly on where the quantum layer is inserted or which classical component it replaces: earlier replacements lower accuracy, and fewer parameters do not reliably shorten training or reduce computation. To assess potential security benefits, we evaluated the robustness of quantum-classical hybrid models under six attack settings. Overall, our study maps where quantum layers help and hurt within VLMs, offering a starting point for hybrid VLM design; whether these gains generalize remains open. The code is available at https://github.com/TITANS-Lab-Artifacts/QuantumVLM.

Figures & tables

Fig. 1: Overview and access level of the attack methods used in this work.

Fig. 2: Proposed framework showing the LLaVA-OneVision-Qwen2-0.5B-SI backbone and quantum-layer insertion and replacement locations.

Symbol	Meaning
$x,t,y$	Image, statement, and label $y\in\{0,1\}$
$x^{\mathrm{adv}},\delta,\epsilon$	Adversarial image, perturbation, and maximum pixel change
$p_{y}$	Probability for class $y$
$h,h^{\prime},d,d_{\mathrm{out}}$	Input and updated features, input and output dimensions
$n_{q},L,j$	Circuit qubit count, depth, and qubit index
$W_{\mathrm{in}},b_{\mathrm{in}}$	Input projection weights and bias

TABLE I: Summary of notation.

Fig. 3: Four quantum circuits used in this study.

Mode	Params(M)	Accuracy (%)	BA (%)	TR (%)	FR (%)
Zero-shot/Full-Frozen	0.000	54.2	53.1	92.4	13.9
Head only	0.002	64.9	65.0	62.6	67.4
Projector	1.837	58.5	58.4	62.8	54.0
LoRA	0.541	79.0	79.0	79.3	78.8
Full fine-tuning	893.676	78.4	78.3	81.6	75.0

TABLE II: Performance of classical baselines.

Fig. 4: Accuracy (%) across quantum insertion training modes.

Fig. 5: Accuracy (%) under full-backbone training. Error bars show 95% image-cluster bootstrap intervals. Dotted lines mark classical full fine-tuning.

Circuit	Insertion				Replacement
Circuit	Pos.	Accuracy(%)	BA(%)	Params(M)	Pos.	Accuracy(%)	BA(%)	Params(M)
BR4	q1	78.8	78.7	893.69	r1	51.3	50.0	891.85
	q2	78.5	78.3	893.68	r2	53.1	52.9	880.61
	q3	79.1	79.1	893.68	r3	76.4	76.0	880.61
	q4	78.8	78.7	893.68	r4	79.2	79.1	880.61
BR8	q1	78.3	78.2	893.70	r1	51.4	50.0	891.86

TABLE III: Performance under full fine-tuning.

Fig. 6: Training time (top) and peak RAM usage (bottom) under full-backbone training. Dashed lines show the classical baseline. Only training is measured.

Circuit	Position	Insertion					Replacement
Circuit	Position	Clean	FGSM	SPSA	HSJ	T-PGD	Clean	FGSM	SPSA	HSJ	T-PGD
BR4	1	72.5	50.0	65.0	72.5	20.0	55.0	50.0	40.0	52.5	50.0
	2	75.0	47.5	60.0	72.5	22.5	37.5	15.0	12.5	32.5	32.5
	3	75.0	47.5	65.0	75.0	22.5	67.5	37.5	50.0	67.5	20.0
	4	72.5	52.5	67.5	72.5	20.0	70.0	50.0	67.5	70.0	25.0
BR8	1	72.5	50.0	67.5	72.5	22.5	55.0	55.0	55.0	55.0	55.0

TABLE IV: Accuracy (%) before and after attack on 40 examples.

Quantum Machine Learning

Quantum machine learning uses quantum circuits and quantum data to build learning algorithms and study their computational and statistical limits.

See all 256 papers

Jun 29, 2026

RiverONE: Generating Knowledge-Intensive VLM by Simulated Quantum Machines

Quantum computing provides a powerful paradigm for representing and transforming high-dimensional information through superposition, entanglement, and measurement-induced nonlinear features. While current quantum hardware is not yet practical for direct large-scale vision-language model (VLM) inference, simulated quantum computation can be used during model construction to generate structured parameters for compact classical AI systems. We build RiverONE, a lightweight vision-language model for quantum calibration plot understanding, using simulated quantum computation. It employs a specialized visual encoder and an InternVL-based language backbone. To compensate for compression-induced information loss, we introduce quantum-generated parameters, which are materialized as classical tensors after training. This allows RiverONE to run entirely on classical GPUs at inference time, with no quantum hardware or runtime quantum simulation. With approximately 1.9 billion parameters, RiverONE achieves at least 95% of the performance of NVIDIA Ising Calibration 1 on quantum calibration plot understanding tasks while using less than 10% of its parameter count. These results suggest that simulated quantum computation can serve as a practical construction-stage mechanism for building lightweight, knowledge-intensive scientific VLMs. Our code is available at https://github.com/THeWakeSystems/RiverOne.

Xindian Ma, Xinyu Long, Yefei Zhang +11

Thewake RiverYtz Lab

Vision-Language Models Quantum-Inspired ML Chart Understanding

Ask PDF

Apr 28, 2026

QCalEval: Benchmarking Vision-Language Models for Quantum Calibration Plot Understanding

Quantum computing calibration depends on interpreting experimental data, and calibration plots provide the most universal human-readable representation for this task, yet no systematic evaluation exists of how well vision-language models (VLMs) interpret them. We introduce QCalEval, the first VLM benchmark for quantum calibration plots: 243 samples across 87 scenario types from 22 experiment families, spanning superconducting qubits and neutral atoms, evaluated on six question types in both zero-shot and in-context learning settings. The best general-purpose zero-shot model reaches a mean score of 72.3, and many open-weight models degrade under multi-image in-context learning, whereas frontier closed models improve substantially. A supervised fine-tuning ablation at the 9-billion-parameter scale shows that SFT improves zero-shot performance but cannot close the multimodal in-context learning gap. As a reference case study, we release NVIDIA Ising Calibration 1, an open-weight model based on Qwen3.5-35B-A3B that reaches 74.7 zero-shot average score.

Shuxiang Cao, Zijian Zhang, Abhishek Agarwal +29

1NVIDIA · University of Toronto · 13Vector Institute for Artificial Intelligence +10

VLM Evaluation In-Context Learning

Ask PDF

May 7, 2026

Quantum-enhanced Large Language Models on Quantum Hardware via Cayley Unitary Adapters

Large language models (LLMs) have transformed artificial intelligence, yet classical architectures impose a fundamental constraint: every trainable parameter demands classical memory that scales unfavourably with model size. Quantum computing offers a qualitatively different pathway, but practical demonstrations on real hardware have remained elusive for models of practical relevance. Here we show that Cayley-parameterised unitary adapters -- quantum circuit blocks inserted into the frozen projection layers of pre-trained LLMs and executed on a 156-qubit IBM Quantum System Two superconducting processor -- improve the perplexity of Llama 3.1 8B, an 8-billion-parameter model in widespread use, by 1.4% with only 6,000 additional parameters and end-to-end inference validated on real Quantum Processing Unit (QPU). A systematic study on SmolLM2 (135M parameters), chosen for its tractability, reveals monotonically improving perplexity with unitary block dimension, 83% recovery of compression-induced degradation, and correct answers to questions that both classical baselines fail -- with a sharp noise-expressivity phase transition identifying the concrete path to quantum utility at larger qubit scales.

Borja Aizpurua, Sukhbinder Singh, Augustine Kshetrimayum +2

Multiverse Computing, Parque Científico y Tecnológico de Gipuzkoa, Paseo May 7 de Miramón 170, 20014, Donostia / San Sebastián, Spain · Department of Basic Sciences, Tecnun – University of Navarra, San Sebastián, Spain · Multiverse Computing, Centre for Social Innovation, 192 Spadina Avenue, ON M5T 2C2, Toronto, Canada +2

LLM Inference LLM Fine-Tuning Quantum Machine Learning Hybrid Quantum-Classical ML Quantum Neural Networks

Ask PDF

Abstract

Quantum Machine Learning

Explore similar work

RiverONE: Generating Knowledge-Intensive VLM by Simulated Quantum Machines

QCalEval: Benchmarking Vision-Language Models for Quantum Calibration Plot Understanding

Quantum-enhanced Large Language Models on Quantum Hardware via Cayley Unitary Adapters