quant-phOct 8, 2026

Trainable Quantum-Native Language Models: Dissipative Memory, Capacity Limits, and Hardware Validation

Authors: Deniz Yunus Göğüş

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We study byte-level autoregressive language models whose hidden state is a small quantum register, whose transitions are learned token-conditioned quantum channels, and whose next-byte probabilities come directly from Born-rule measurements without a learned classical vocabulary projection or softmax. In the fixed-width recurrent setting, unitary dynamics cannot contract or reset stored distinctions; experimentally, unitary recurrence loses to its retrained one-token control in every tested variant. Introducing learned dissipation, then token-conditioned dissipative gating, yields useful memory beyond one token. The gated model improves test negative log-likelihood by 0.20 +/- 0.03 nats/byte over retrained parameter-matched one-token controls at context length 12, and by 0.29-0.33 at longer contexts, but never beats retrained two-token controls. A key-value recall benchmark shows a width-dependent threshold: one stored pair meets the pre-registered accuracy bar at 4-5 memory qubits, two pairs at 6-7, and three pairs remain unsolved. This is consistent with, but does not establish, a capacity explanation for the short effective memory horizon. As a secondary result, one-qubit models reach lower stationary memory entropy than a heuristically optimized classical predictive frontier at matched error in 9/9 seeds. On IBM Heron hardware, computational-basis memory-register distributions show signal against a uniform null in 12/12 circuits across prefixes of up to 12 tokens, next-bit probabilities pass pre-registered accuracy bars at four-token prefixes, and bytes are generated autoregressively by sampling physically measured bit probabilities. The tested models are classically simulable and substantially worse than classical n-gram baselines. We claim no computational quantum advantage.

Figures & tables

Figure 1: The three memory dynamics compared. UMW: unitary conjugation (norm-preserving, cannot contract stored distinctions). UMW-D: unitary write followed by a fixed learned dissipation channel (a work qubit coupled, reset, and discarded). UMW-G: one learned CPTP map per token, writing and forgetting jointly. All rows read out with the same 256-outcome Born-rule POVM, with no softmax; the sampled byte is the next input.
Figure 2: Full recurrence against retrained one-token (h1) and two-token (h2) controls of the same parameterization and parameter count; dots are individual seeds, bars their means. Dotted and dashed lines: the tuned bigram (2.3406) and the classical 4-gram (1.5125) on the same test set.
Figure 3: Key–value recall ladder (exp 295, single seed). Left: answer accuracy against width for K=1,2,3K=1,2,3 stored pairs, with the 0.5 bar and the value-marginal floor. Right: value content and full configuration entropy of the largest KK solved at each width, with the single-copy accessible-information ceiling of nn bits drawn as an information-scale reference only, not as a task-solvability threshold.
K (value bits / configuration bits)n = 4n = 5n = 6n = 7
1 (3.3 / 7.3)0.6371.0001.0001.000
2 (6.6 / 13.6)0.2450.3450.5730.621
3 (10.0 / 19.1)0.2020.1670.3120.309
Table 1: Key–value recall ladder (exp 295; answer accuracy, single seed, 6,000 steps, dense channels). Bold: above the pre-registered 0.5 bar.
Figure 4: Perturbed coin, one memory qubit: stationary memory entropy S(ρˉ)S(\bar{\rho}) against excess prediction error, for trained models without (dark) and with (teal) the annealed entropy penalty; markers denote p=0.3p=0.3 (circle), 0.40.4 (square), 0.450.45 (triangle). Grey curves: the attacked classical frontier per pp ; dashed: Cμ=ln⁡2C_{\mu}=\ln 2 ; dotted: the quantum optima Cq(p)C_{q}(p) .
Figure 5: Hardware validation on ibm_marrakesh . (a) Total variation distance between the measured computational-basis memory-register distribution and the simulator’s predicted diagonal, for line (Job A ′ ) and ring (Job A) circuits, against the distance between the prediction and the uniform null (hollow markers). (b) Per-bit ∣phw(bit=1)−psim∣|p_{\mathrm{hw}}(\mathrm{bit}=1)-p_{\mathrm{sim}}| for Jobs B (L=8), B2 (L=8 with dynamical decoupling and twirling) and B3 (L=4); red bars are medians, dashed line the 0.10 bar. (c) Transpiled two-qubit gate counts for the two entanglers.
Figure 6: Hardware-in-the-loop generation. The model is defined with a single 256-outcome vocabulary POVM; the hardware implements a distilled binary-tree approximation in which each of eight conditional bit probabilities is physically measured and their product defines the token probability. Shaded boxes are physical quantum computation (every state transition and every bit probability); unshaded boxes are classical orchestration (shot counting, bit sampling, circuit selection, the prompt buffer). No learned classical vocabulary projection or softmax is used.
Figure S1: Supplementary. (a) K=2K=2 recall accuracy at n=7n=7 against circuit depth per token for line-circuit channels (exp 310; dots are seeds) and the dense-channel reference (exp 295); the depth-30 point is from the follow-up campaign and is discussed in the footnote to § 9 . Points are not interpolated. (b) Byte NLL of the dense teacher, an operator-level distilled student, and task-aware students at depths 12 and 20, with parameters per channel.
Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Milestone / hypothesisBarOutcome
M18 H: UMW trainable with signal shapingpass recall bar at delays ≤\leq 8, 3/3 seedspass (curriculum and auxiliary loss)
M19 H1: pipeline gatememoryless controls recover unigrampass
M19 H2–H7: unitary memory beats h1 (5 variants, 2 widths)NLL < h1fail , all variants
M19 H8: UMW-D beats channel-matched h1≥\geq +0.05 natspass 4/4 arms (+0.091…+0.106)
M19 H9: UMW-D beats h2NLL < h2fail (h2 wins)
M20 H1: UMW-G beats UMW-D≥\geq +0.05, 3/3 seedspass (+0.085…+0.132)
Appendix
Table 9
ClaimEvidence levelWhere
Unitary channels preserve trace distance and cannot implement many-to-one mapsmathematical (elementary)§4.1
No O(1)-width delay-invariant unitary recall solution in the marked-recall settingmathematical (short argument)§4.2, docs/m18_delay_invariance_obstruction.md
Unitary memory loses to retrained h1 in this architectureexperimental (5 variants, 2 widths, single seeds)§4.2
UMW-D and UMW-G beat parameter-count-matched h1experimental (3 seeds each; mean ±\pm s.d.)§5
Recurrence never beats retrained h2 at 4–8 qubitsexperimental (2–3 seeds per context)§5.3
Solvable recall task shifts from one to two pairs with width; three pairs unsolved at n = 7experimental (single seed per cell)§6.2
Appendix
Table 10
More in this topic

Language Modeling

Language modeling develops systems that learn patterns in text and other sequences to understand, predict, and generate language.

See all 498 papers

Explore similar work

CardsList
Mar 27, 2026

Entanglement as Memory: Mechanistic Interpretability of Quantum Language Models

Quantum language models have shown competitive performance on sequential tasks, yet whether trained quantum circuits exploit genuinely quantum resources -- or merely embed classical computation in quantum hardware -- remains unknown. Prior work has evaluated these models through endpoint metrics alone, without examining the memory strategies they actually learn internally. We introduce the first mechanistic interpretability study of quantum language models, combining causal gate ablation, entanglement tracking, and density-matrix interchange interventions on a controlled long-range dependency task. We find that single-qubit models are exactly classically simulable and converge to the same geometric strategy as matched classical baselines, while two-qubit models with entangling gates learn a representationally distinct strategy that encodes context in inter-qubit entanglement -- confirmed by three independent causal tests (p < 0.0001, d = 0.89). On real quantum hardware, only the classical geometric strategy survives device noise; the entanglement strategy degrades to chance. These findings open mechanistic interpretability as a tool for the science of quantum language models and reveal a noise-expressivity tradeoff governing which learned strategies survive deployment.

Nathan Roll

Department of Linguistics, Stanford University

Long-Range Dependency ModelingCircuits in Language ModelsMechanistic InterpretabilitySequence Modeling
AskPDF
Aug 4, 2026

Separating quantum circuits from classical LLMs

Modern large language models - transformers and diffusion language models - are built around two canonical algorithmic tasks: prediction and generation. We prove unconditional separations between low-depth quantum computation and the corresponding bounded-resource classical language-model architectures in both regimes. Concretely, we exhibit the following: 1. Distributional separation. We give a distribution that is sampleable by QNC0\textsf{QNC}^0 circuits (i.e., a family of constant-depth quantum circuits consisting of bounded fan-in gates) that no constant-round diffusion language model (DLM\textsf{DLM}) with shallow scheduling and denoising can sample within constant distance, even when allowed sublinear chain-of-thought and output-token revision/remasking events, the very features modern DLM\textsf{DLM}s rely on. 2. Functional separation. We exhibit a function computable in ∧∘QNC0[log⁡log⁡n]\land \circ \textsf{QNC}^0[\log\log n] (i.e., a family of O(log⁡log⁡n)(\log\log n)-depth QNC0\textsf{QNC}^0 circuits, where nn is the input length, followed by a single classical AND\mathsf{AND} gate) such that any constant-depth decoder-only transformer computing the function must be large: it would have to have width nΩ(1)n^{Ω(1)}. Together, our work initiates the study of quantum advantage in the era of large language models.

Srinivasan Arunachalam, Arkopal Dutt, Hari Krovi +1

IBM Research

Decoder-Only Language ModelsDiffusion Language Models
AskPDF
Sep 21, 2026

Circuit Hypernetworks for Quantum-Augmented Diffusion Language Models

Language models can be adapted by changing the computations applied to individual tokens. Quantum circuits offer one such approach, but evaluating wider circuits inside a large model can be computationally demanding. Here we introduce HyperQ, which adds token-conditioned quantum residual branches to a frozen masked-diffusion language model. A quantum residual branch is a module in each transformer block that reads a token's hidden state, emits the coordinates of that token's circuit, executes it, and adds the measured values back through a residual connection. The backbone remains frozen, and only the added branches are trained. Within each branch, a lightweight circuit hypernetwork emits token-specific rotation angles, coupling strengths, and measurement axes in a shared sparse circuit structure. The required expectation values have an exact classical expression whose evaluation cost grows linearly with the qubit count, enabling circuits from 16 to 64 qubits to be trained within a 1.1-billion-parameter backbone. Across downstream benchmarks, increasing circuit width raises the average score from 47.65 to 54.30. At 64 qubits, HyperQ exceeds the backbone and its low-rank-adapted counterpart by 4.71 and 3.67 points, respectively. HyperQ is fine-tuned on 20,000 prompt-response pairs, compared with 200,000 for the classical baselines. These findings support token-conditioned circuit emission as a tractable architectural approach to quantum-augmented language modelling.

Xiaoqiang Wang, Mengyang Xiong, Jun Dai +1

D´epartement d´informatique et de recherche op´erationnelle, Universit´e de Montr´eal, Montr´eal, QC, Canada. · Mila, Quebec AI Institute, Montr´eal, QC, Canada. · McGill University, Montr´eal, QC, Canada.

LLM Fine-TuningQuantum Machine LearningCircuits in Language ModelsDiffusion Language Models
AskPDF