Trainable Quantum-Native Language Models: Dissipative Memory, Capacity Limits, and Hardware Validation

Deniz Yunus Göğüş

quant-phOct 8, 2026

Trainable Quantum-Native Language Models: Dissipative Memory, Capacity Limits, and Hardware Validation

Authors: Deniz Yunus Göğüş

Abstract

We study byte-level autoregressive language models whose hidden state is a small quantum register, whose transitions are learned token-conditioned quantum channels, and whose next-byte probabilities come directly from Born-rule measurements without a learned classical vocabulary projection or softmax. In the fixed-width recurrent setting, unitary dynamics cannot contract or reset stored distinctions; experimentally, unitary recurrence loses to its retrained one-token control in every tested variant. Introducing learned dissipation, then token-conditioned dissipative gating, yields useful memory beyond one token. The gated model improves test negative log-likelihood by 0.20 +/- 0.03 nats/byte over retrained parameter-matched one-token controls at context length 12, and by 0.29-0.33 at longer contexts, but never beats retrained two-token controls. A key-value recall benchmark shows a width-dependent threshold: one stored pair meets the pre-registered accuracy bar at 4-5 memory qubits, two pairs at 6-7, and three pairs remain unsolved. This is consistent with, but does not establish, a capacity explanation for the short effective memory horizon. As a secondary result, one-qubit models reach lower stationary memory entropy than a heuristically optimized classical predictive frontier at matched error in 9/9 seeds. On IBM Heron hardware, computational-basis memory-register distributions show signal against a uniform null in 12/12 circuits across prefixes of up to 12 tokens, next-bit probabilities pass pre-registered accuracy bars at four-token prefixes, and bytes are generated autoregressively by sampling physically measured bit probabilities. The tested models are classically simulable and substantially worse than classical n-gram baselines. We claim no computational quantum advantage.

Figures & tables

Figure 1: The three memory dynamics compared. UMW: unitary conjugation (norm-preserving, cannot contract stored distinctions). UMW-D: unitary write followed by a fixed learned dissipation channel (a work qubit coupled, reset, and discarded). UMW-G: one learned CPTP map per token, writing and forgetting jointly. All rows read out with the same 256-outcome Born-rule POVM, with no softmax; the sampled byte is the next input.

Figure 2: Full recurrence against retrained one-token (h1) and two-token (h2) controls of the same parameterization and parameter count; dots are individual seeds, bars their means. Dotted and dashed lines: the tuned bigram (2.3406) and the classical 4-gram (1.5125) on the same test set.

Figure 3: Key–value recall ladder (exp 295, single seed). Left: answer accuracy against width for

K=1,2,3

stored pairs, with the 0.5 bar and the value-marginal floor. Right: value content and full configuration entropy of the largest

K

solved at each width, with the single-copy accessible-information ceiling of

n

bits drawn as an information-scale reference only, not as a task-solvability threshold.

K (value bits / configuration bits)	n = 4	n = 5	n = 6	n = 7
1 (3.3 / 7.3)	0.637	1.000	1.000	1.000
2 (6.6 / 13.6)	0.245	0.345	0.573	0.621
3 (10.0 / 19.1)	0.202	0.167	0.312	0.309

Table 1: Key–value recall ladder (exp 295; answer accuracy, single seed, 6,000 steps, dense channels). Bold: above the pre-registered 0.5 bar.

Figure 4: Perturbed coin, one memory qubit: stationary memory entropy

S(\bar{\rho})

against excess prediction error, for trained models without (dark) and with (teal) the annealed entropy penalty; markers denote

p=0.3

(circle),

0.4

(square),

0.45

(triangle). Grey curves: the attacked classical frontier per

p

; dashed:

C_{\mu}=\ln 2

; dotted: the quantum optima

C_{q}(p)

Figure 5: Hardware validation on ibm_marrakesh . (a) Total variation distance between the measured computational-basis memory-register distribution and the simulator’s predicted diagonal, for line (Job A ′ ) and ring (Job A) circuits, against the distance between the prediction and the uniform null (hollow markers). (b) Per-bit

|p_{\mathrm{hw}}(\mathrm{bit}=1)-p_{\mathrm{sim}}|

for Jobs B (L=8), B2 (L=8 with dynamical decoupling and twirling) and B3 (L=4); red bars are medians, dashed line the 0.10 bar. (c) Transpiled two-qubit gate counts for the two entanglers.

Figure 6: Hardware-in-the-loop generation. The model is defined with a single 256-outcome vocabulary POVM; the hardware implements a distilled binary-tree approximation in which each of eight conditional bit probabilities is physically measured and their product defines the token probability. Shaded boxes are physical quantum computation (every state transition and every bit probability); unshaded boxes are classical orchestration (shot counting, bit sampling, circuit selection, the prompt buffer). No learned classical vocabulary projection or softmax is used.

Figure S1: Supplementary. (a)

K=2

recall accuracy at

n=7

against circuit depth per token for line-circuit channels (exp 310; dots are seeds) and the dense-channel reference (exp 295); the depth-30 point is from the follow-up campaign and is discussed in the footnote to § 9 . Points are not interpolated. (b) Byte NLL of the dense teacher, an operator-level distilled student, and task-aware students at depths 12 and 20, with parameters per channel.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Milestone / hypothesis	Bar	Outcome
M18 H: UMW trainable with signal shaping	pass recall bar at delays $\leq$ 8, 3/3 seeds	pass (curriculum and auxiliary loss)
M19 H1: pipeline gate	memoryless controls recover unigram	pass
M19 H2–H7: unitary memory beats h1 (5 variants, 2 widths)	NLL < h1	fail , all variants
M19 H8: UMW-D beats channel-matched h1	$\geq$ +0.05 nats	pass 4/4 arms (+0.091…+0.106)
M19 H9: UMW-D beats h2	NLL < h2	fail (h2 wins)
M20 H1: UMW-G beats UMW-D	$\geq$ +0.05, 3/3 seeds	pass (+0.085…+0.132)

Appendix

Table 9

Claim	Evidence level	Where
Unitary channels preserve trace distance and cannot implement many-to-one maps	mathematical (elementary)	§4.1
No O(1)-width delay-invariant unitary recall solution in the marked-recall setting	mathematical (short argument)	§4.2, docs/m18_delay_invariance_obstruction.md
Unitary memory loses to retrained h1 in this architecture	experimental (5 variants, 2 widths, single seeds)	§4.2
UMW-D and UMW-G beat parameter-count-matched h1	experimental (3 seeds each; mean $\pm$ s.d.)	§5
Recurrence never beats retrained h2 at 4–8 qubits	experimental (2–3 seeds per context)	§5.3
Solvable recall task shifts from one to two pairs with width; three pairs unsolved at n = 7	experimental (single seed per cell)	§6.2

Appendix

Table 10

Language Modeling

Language modeling develops systems that learn patterns in text and other sequences to understand, predict, and generate language.

See all 504 papers

Entanglement as Memory: Mechanistic Interpretability of Quantum Language Models
Mar 27, 2026Nathan RollLong-Range Dependency Modeling Circuits in Language Models
Department of Linguistics, Stanford University
Separating quantum circuits from classical LLMs
Aug 4, 2026Srinivasan Arunachalam, Arkopal Dutt, Hari Krovi +1Decoder-Only Language Models Diffusion Language Models
IBM Research
Circuit Hypernetworks for Quantum-Augmented Diffusion Language Models
Sep 21, 2026Xiaoqiang Wang, Mengyang Xiong, Jun Dai +1LLM Fine-Tuning Quantum Machine Learning
D´epartement d´informatique et de recherche op´erationnelle, Universit´e de Montr´eal, Montr´eal, QC, Canada. · Mila, Quebec AI Institute, Montr´eal, QC, Canada. · McGill University, Montr´eal, QC, Canada.

Mar 27, 2026J/K move · Enter open · S save

Entanglement as Memory: Mechanistic Interpretability of Quantum Language Models

Nathan Roll

Department of Linguistics, Stanford University

Long-Range Dependency Modeling Circuits in Language Models Mechanistic Interpretability Sequence Modeling

Open paper AskPDF arXiv

Quantum language models have shown competitive performance on sequential tasks, yet whether trained quantum circuits exploit genuinely quantum resources -- or merely embed classical computation in quantum hardware -- remains unknown. Prior work has evaluated these models through endpoint metrics alone, without examining the memory strategies they actually learn internally. We introduce the first mechanistic interpretability study of quantum language models, combining causal gate ablation, entanglement tracking, and density-matrix interchange interventions on a controlled long-range dependency task. We find that single-qubit models are exactly classically simulable and converge to the same geometric strategy as matched classical baselines, while two-qubit models with entangling gates learn a representationally distinct strategy that encodes context in inter-qubit entanglement -- confirmed by three independent causal tests (p < 0.0001, d = 0.89). On real quantum hardware, only the classical geometric strategy survives device noise; the entanglement strategy degrades to chance. These findings open mechanistic interpretability as a tool for the science of quantum language models and reveal a noise-expressivity tradeoff governing which learned strategies survive deployment.

Abstract

Language Modeling

Explore similar work

Entanglement as Memory: Mechanistic Interpretability of Quantum Language Models

Separating quantum circuits from classical LLMs

Circuit Hypernetworks for Quantum-Augmented Diffusion Language Models