Q-Interference: Memory-Efficient Phase-Aware Quantum-Inspired Attention

Emama Nahid; Tahmid Imtiaz Imu; Huayue Gu; Liran Ma; Zhipeng Cai; Honghui Xu

cs.CLAug 18, 2026

Q-Interference: Memory-Efficient Phase-Aware Quantum-Inspired Attention

Authors: Emama Nahid, Tahmid Imtiaz Imu, Huayue Gu, Liran Ma, Zhipeng Cai, Honghui Xu

Organizations: Kennesaw State University Marietta, GA, USA · Miami University Oxford, OH, USA · Georgia State University Atlanta, GA, USA

Abstract

GPT attention measures token compatibility through dot-product similarity. This mechanism is simple, effective, and memory-efficient. But it does not explicitly model whether strong token features should reinforce or suppress one another. We introduce Q-Interference, a fully classical quantum-inspired attention mechanism for autoregressive language modeling that augments each query and key feature with an amplitude and a learned phase. The resulting attention score is phase-aware which aligned phases contribute constructively while conflicting phases contribute destructively. Although Q-Interference yields a richer interaction rule than similarity alone, a naive implementation of Q-Interference requires a large token-pair-feature interaction tensor, making it memory-intensive and often impractical. To address this limitation, we propose an exact trigonometric factorization that computes the same score using two standard matrix multiplications avoiding materialization of the large intermediate tensor. Q-Interference fits directly into a Transformer block in GPT and leaves the remainder of the model architecture and next-token prediction objective unchanged. Experiments on public benchmark datasets and baseline models show that the proposed reformulation trains stably in a controlled GPT-style setting and provides a consistent memory advantage over naive phase-aware interference attention. These results support the specific contribution of this work: an exact memory-efficient reformulation that makes phase-aware interference attention practical within a standard GPT pipeline.

Figures & tables

Figure 1: Q-Interference methodology flowchart.

Model	Comparison role	Justification
Baseline GPT	Standard internal control	Directly controlled reference for conventional causal dot-product attention under the same GPT-style training setup.
Q-GPT baseline	Quantum-inspired reference	Provides a published quantum-inspired GPT-style architectural comparison in this setting.
OPT-125M and GPT-Neo-125M	External pretrained reference	Gives context against an open autoregressive GPT-style model family.

Table 1: Baseline selection rationale.

Figure 2: Quality-memory trade-off for the main model comparison.

Dataset	Model	Best Val Loss	Test Loss	Test PPL	Peak GPU Mem (MB)
WikiText-103	Baseline GPT	3.2036	3.2049	24.6534	8055.76
WikiText-103	Q-GPT baseline	4.1142	4.1032	60.5367	7199.26
WikiText-103	Q-Interference (ours)	3.1809	3.1852	24.1718	4227.14
TinyStories	Baseline GPT	1.7279	1.7263	5.6196	8055.76
TinyStories	Q-GPT baseline	2.7036	2.7082	15.0016	7199.26
TinyStories	Q-Interference (ours)	1.7344	1.7394	5.6941	4227.14

Table 2: Baselines vs Q-Interference comparison over benchmark dataset

Dataset	Model	Test Loss	Test PPL	Peak GPU Mem (MB)
WikiText-103	GPT-Neo-125M	2.9673	19.4386	4097.91
WikiText-103	OPT-125M	3.0169	20.4278	4213.64
WikiText-103	Q-Interference (ours)	3.1852	24.1718	4227.14
TinyStories	GPT-Neo-125M	1.5198	4.5713	9934.54
TinyStories	OPT-125M	1.4217	4.1443	6219.52
TinyStories	Q-Interference (ours)	1.7394	5.6941	4227.14

Table 3: Comparison to pre-trained reference models.

Dataset	Model	Best Val Loss	Test Loss	Test PPL	Peak GPU Mem (MB)
WikiText-103	Naive interference	OOM	OOM	OOM	OOM
WikiText-103	Q-Interference (ours)	3.1809	3.1852	24.1718	4227.14
TinyStories	Naive interference	2.2972	2.3422	10.4037	12138.34
TinyStories	Q-Interference (ours)	1.7344	1.7394	5.6941	4227.14
pile-10k	Naive interference	4.0223	5.1086	165.4355	12138.34
pile-10k	Q-Interference (ours)	4.0745	4.7616	116.9275	4227.14

Table 4: Naive phase-aware attention versus Q-Interference.

Dataset	Model	Best Val Loss	Test Loss	Test PPL	Peak GPU Mem (MB)
WikiText-103	Q-Interference (ours)	3.1809	3.1852	24.1718	4227.14
WikiText-103	Q-Interference w/o phase	3.1830	3.1879	24.2373	3993.53
TinyStories	Q-Interference (ours)	1.7344	1.7394	5.6941	4227.14
TinyStories	Q-Interference w/o phase	1.7352	1.7410	5.7030	3993.53
pile-10k	Q-Interference (ours)	4.0745	4.7616	116.9275	4227.14
pile-10k	Q-Interference w/o phase	4.0736	4.7634	117.1429	3993.53

Table 5: Phase ablation for the final Q-Interference model.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Work	GPT-style	Quantum-inspired	Phase-aware	Memory-aware	Exact memory-efficient
QSANN	$\times$	✓	$\times$	$\times$	$\times$
QMSAN	$\times$	✓	$\times$	$\times$	$\times$
HyQuT	✓	✓	$\times$	$\times$	$\times$
QISA	✓	✓	$\times$	$\times$	$\times$
QubitCache	✓	✓	$\times$	✓	$\times$
Q-Interference (ours)	✓	✓	✓	✓	✓

Appendix

Table 6: Compact comparison of representative prior work and Q-Interference.

Symbol	Meaning
$T$	Sequence length.
$d_{h}$	Dimension of one attention head.
$a_{i}^{q}\in\mathbb{R}_{+}^{d_{h}}$	Nonnegative query amplitude vector for token $i$ .
$a_{j}^{k}\in\mathbb{R}_{+}^{d_{h}}$	Nonnegative key amplitude vector for token $j$ .
$\phi_{i}^{q}\in\mathbb{R}^{d_{h}}$	Query phase vector for token $i$ .
$\phi_{j}^{k}\in\mathbb{R}^{d_{h}}$	Key phase vector for token $j$ .

Appendix

Table 7: Notation used in the proof appendix.

Term	Definition
Best validation loss	The lowest validation cross entropy observed during training. It is used to track model selection and training progress. Lower is better.
Test loss	The cross entropy loss measured on the held-out test split after training. It measures how well the model assigns probability to the correct next token. Lower is better.
Test perplexity	The exponential form of test loss. It can be read as the model’s effective uncertainty when predicting the next token. Lower is better.
Peak training GPU memory	The maximum GPU memory used during training under the reported batch size, context length, precision, and hardware. It measures practical training cost rather than language quality. Lower is better.
Memory reduction	The percentage decrease in peak GPU memory relative to a chosen baseline. A larger reduction means the model needs less GPU memory under the same reported setting.
Parameter matched comparison	A comparison where models have similar parameter counts. This helps reduce the chance that differences are caused only by model size.

Appendix

Table 8: Definitions of metrics and model terms used in the appendix result analysis.

Figure 3: Controlled comparison on the WikiText-103 dataset. The top ruler reports test perplexity and the bottom ruler reports peak training GPU memory. Each marker represents one internal model, and each label reports the exact value. Q-Interference achieves the best internal test perplexity and the lowest peak GPU memory in this controlled setting.

Figure 4: Peak GPU memory across the custom model family. Q-Interference consistently uses less memory than the standard GPT baseline, naive interference, and the Q-GPT baseline under the reported setting. The naive interference model highlights the cost of materializing the phase-aware interaction tensor.

Figure 5: Pretrained 125M model comparison. GPT-Neo-125M and OPT-125M are pretrained decoder-only models, while Q-Interference is trained from scratch in the controlled setup used in this work. The pretrained references obtain stronger final language-modeling quality, but Q-Interference shows a favorable memory profile and uses lower peak GPU memory on three of the four datasets. The comparison is intended to contextualize Q-Interference rather than to serve as a parameter-matched baseline.

Explore similar work

Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention
Jun 18, 2026Vishesh Tripathi, Abhay KumarGrouped-Query Attention Transformer Attention
FrontiersMind
Compress and Forget: bitsandbytes Quantization Amplifies Proactive Interference in LLMs
Aug 19, 2026Shayan Shahrabi-Farahani, Dara RahmatiLLM Quantization Post-Training Quantization
Shahid Beheshti University · Shahid Beheshti University, Tehran, Iran
Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers
Jun 20, 2026Xin GaoTransformer Inference KV Caching
Department of Mathematics and Statistics York University Toronto, M3J 1P3

Jun 18, 2026J/K move · Enter open · S save

Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention

Vishesh Tripathi, Abhay Kumar

FrontiersMind

Grouped-Query Attention Transformer Attention Efficient Attention Mixture-of-Experts Models Multi-Head Attention

Open paper AskPDF arXiv

Self-attention is central to Transformer performance and is often the most expensive part of the Transformer at long context lengths because its pairwise token interactions scale quadratically with sequence length. Standard dense attention also applies the same set of attention heads to every token regardless of token difficulty or information content. This uniform activation can waste compute, especially as sequences grow longer and attention cost increases rapidly. We propose Grouped Query Experts (GQE), a mixture-of-experts layer on top of grouped-query attention (GQA). Within each GQA group, a router selects k query-head experts per token while all key-value (KV) heads remain dense and unchanged. Thus, GQE keeps the KV cache benefits of GQA and reduces only the active query-head computation. On a fixed 30B token budget at the 250M parameter scale, GQE matches the all-active GQA baseline in downstream accuracy while activating half the query heads per token.

Abstract

Explore similar work

Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention

Compress and Forget: bitsandbytes Quantization Amplifies Proactive Interference in LLMs

Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers