Multi-head Quantum Neural Networks: Replacing Transformer Blocks with NISQ Modules inside a 30-Billion-Parameter LLM
Organizations: Multiverse Computing, Toronto, Ontario, Canada · The Edward S. Rogers Sr. Department of Electrical and Computer Engineering - University of Toronto, Toronto, Ontario, Canada · Multiverse Computing, San Sebasti´an, Spain · Department of Basic Sciences, Tecnun - University of Navarra, San Sebasti´an, Spain ·
Abstract
We introduce the multi-head quantum neural network (QNN), a quantum module compatible with noisy intermediate-scale quantum (NISQ) devices that can replace transformer blocks inside pre-trained large language models (LLMs) with little loss in accuracy and with far fewer parameters than the blocks it replaces. Encoding a model's hidden neurons into a qubit register is the obstacle: amplitude encoding scales poorly in circuit depth and angle encoding in qubit count, so neither is practical at production-scale width. We instead partition the hidden space into disjoint heads, angle-encode each into an -qubit register, process the heads in parallel, and couple them through constrained classical linear layers, the mixers. Performing surgery on Qwen3-30B-A3B-Instruct-2507, we show that a single multi-head QNN with trainable parameters (, , , ) can replace a contiguous span of eight transformer blocks, displacing of the model's parameters. Ablations show that, for a single replaced block, the quantum heads (circuits followed by a learned readout), and not the classical mixers alone, recover the displaced function, whereas for fused spans the mixers alone come within - of the full module in perplexity. Whether this benefit is specific to trained, entangling quantum circuits is not established: controls that switch the entangler off (two seeds), or freeze the circuit at its initial or at uniformly random angles (one seed), beat the full module on held-out perplexity, and a parameter-matched classical multilayer perceptron remains a few percent ahead of the full module. We train classically with exact statevector simulation, then execute the quantum heads of our fusion checkpoints on a -qubit IBM Heron processor, showing that NISQ-scale quantum modules can be installed in a production-scale LLM and run on hardware.
Figures & tables
| Component | Expression | Count |
|---|---|---|
| Input mixer | ||
| Encoding scales and biases | ||
| Variational gates | ||
| entanglers | ||
| Learned readout | ||
| Output mixer |
| Condition | WikiText-2 | C4 | webmix-recon | top-1 agreement |
| Baseline (unmodified) | 8.672 | 14.305 | 8.120 | – |
| Identity-skip | 11.941 | 18.706 | 11.079 | 0.800 |
| No-circuit (mixers only) | 0.812 | |||
| Full multi-head QNN | 0.856 | |||
| Matched MLP | 0.853 |
| Condition | LAMBADA | HellaSwag | ARC-e | ARC-c | PIQA | WinoG | BoolQ |
|---|---|---|---|---|---|---|---|
| Baseline | 0.711 | 0.798 | 0.835 | 0.628 | 0.805 | 0.728 | 0.885 |
| Identity-skip | 0.653 | 0.781 | 0.803 | 0.599 | 0.781 | 0.663 | 0.881 |
| No-circuit | |||||||
| Full multi-head QNN | |||||||
| Matched MLP | |||||||
| Validation best PPL, mean and sample SD at seeds | |||||||
| Total ISA depth | 45 | 46 | 46 | 45 | 45 |
| Two-qubit depth | 13 | 13 | 13 | 13 | 13 |
| Two-qubit gates | 392 | 392 | 392 | 392 | 392 |
| Total gates | 3020 | 2960 | 3009 | 3002 | 3008 |
| Physical qubits | 112 | 112 | 112 | 112 | 112 |
| Routing swaps | 0 | 0 | 0 | 0 | 0 |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Property | Value |
|---|---|
| Total parameters | |
| Decoder layers | |
| Parameters per decoder layer | |
| of which MoE experts ( experts) | |
| of which attention ( proj. and norm) | |
| of which MoE router |
| Condition | Forward path | Trainable |
|---|---|---|
| Full multi-head QNN | mixer QNN bank learned readout mixer | |
| Circuit-only | as above, learned readout removed; raw expectations pass through | |
| Frozen-readout | as full, readout held at its initialization | |
| LayerNorm readout | as full, learned readout replaced by per-head LayerNorm | |
| No-circuit (mixers only) | mixer identity mixer; all heads removed | |
| Matched MLP |
| Condition | Angles | Entangler | Encoding | Circuit | Readout | Mixers | Total |
|---|---|---|---|---|---|---|---|
| Full multi-head QNN ( , ) | |||||||
| Circuit-only | |||||||
| Frozen readout | frozen | ||||||
| LayerNorm readout | |||||||
| No-circuit (mixers only) | |||||||
| Matched MLP | – | – | – | – | – | – |
| Parameter | Value |
|---|---|
| Stage 1 | Local reconstruction, per-layer input/output fit |
| Stage 2 | Full-model causal-LM optimization of the replacement block |
| Objective | Stage 1: reconstruction loss; Stage 2: next-token negative log-likelihood |
| Training corpus | webmix_calib , up to sequences |
| Validation | sequences from the disjoint tail, no gradients |
| In-corpus evaluation | webmix-recon , sequences; overlaps the training portion |
| Metric | Value |
|---|---|
| WikiText-2 token PPL | |
| C4 token PPL | |
| webmix-recon PPL | |
| LAMBADA | |
| HellaSwag | |
| ARC-easy |
| Condition | Seed | Val PPL | Gate | WikiText-2 | C4 | webmix-recon | top-1 |
|---|---|---|---|---|---|---|---|
| Baseline | – | – | – | – | |||
| Identity-skip | – | ||||||
| Full multi-head QNN | 42 | ||||||
| Full multi-head QNN | 123 | ||||||
| Full multi-head QNN | 7 | ||||||
| Matched MLP | 42 |
| Condition | Val PPL | WikiText-2 | C4 | webmix-recon |
|---|---|---|---|---|
| Full multi-head QNN | ||||
| Matched MLP | ||||
| No-circuit |
| Condition | Trainable | Val PPL | Gate | WikiText-2 | C4 | webmix-recon |
|---|---|---|---|---|---|---|
| Full multi-head QNN (reference) | ||||||
| LayerNorm readout | ||||||
| Frozen readout | ||||||
| Circuit-only | ||||||
| No-circuit (mixers only) |
| Condition | Seed | Val PPL | WikiText-2 | C4 | top-1 |
|---|---|---|---|---|---|
| Full multi-head QNN | 42 | ||||
| Full multi-head QNN | 123 | ||||
| Matched MLP | 42 | ||||
| No entangler | 42 | ||||
| No entangler | 123 | ||||
| Frozen circuit (initial values) | 42 |
| Condition | WikiText-2 | C4 | webmix-recon | |
|---|---|---|---|---|
| – | Baseline | |||
| Identity-skip | ||||
| Full multi-head QNN | ||||
| Frozen readout | ||||
| Circuit-only | ||||
| No-circuit |
| Condition | LAMBADA | HellaSwag | ARC-e | ARC-c | PIQA | WinoG | BoolQ | |
|---|---|---|---|---|---|---|---|---|
| Full multi-head QNN | ||||||||
| Frozen readout | ||||||||
| Circuit-only | ||||||||
| No-circuit | ||||||||
| Matched MLP | ||||||||
| Full multi-head QNN |
| Full multi-head QNN | Frozen readout | Circuit-only | No-circuit | Matched MLP | |
|---|---|---|---|---|---|
| webmix- | NLL | |||||
|---|---|---|---|---|---|---|
| Seed | WikiText-2 | C4 | recon | top-1 | vs. s42 | |
| 42 | – | |||||
| 123 | ||||||
| 42 | – | |||||
| 123 |
| Layer set | Trainable | Val PPL | WikiText-2 | Identity-skip | C4 | webmix-recon | |
|---|---|---|---|---|---|---|---|
| – – |
| LAMBADA | HellaSwag | ARC-e | ARC-c | PIQA | WinoG | BoolQ | top-1 | |
|---|---|---|---|---|---|---|---|---|
| Condition | Removed | Added | Resulting | Reduction | |
|---|---|---|---|---|---|
| Single-block, full multi-head QNN | |||||
| Single-block, MLP | |||||
| Single-block, no-circuit | |||||
| Identity-skip | |||||
| Fusion | |||||
| Fusion |
| Comparison | Eval set | Ratio | CI | Resolved | |
|---|---|---|---|---|---|
| Fusion vs. baseline | WikiText-2 | yes | |||
| Fusion vs. baseline | C4 | yes | |||
| Fusion vs. baseline | webmix-recon | yes | |||
| Fusion vs. baseline | WikiText-2 | yes | |||
| Fusion vs. baseline | C4 | yes | |||
| Fusion vs. baseline | webmix-recon | yes |
Quantum Neural Networks
Quantum neural networks use quantum circuits and measurements to learn representations, make predictions, and model complex data.
See all 148 papers