Organizations: School of Integrated Innovative Technology, King Mongkut’s Institute of Technology Ladkrabang, Bangkok, Thailand · Faculty of Science and Technology, Suan Dusit University, Bangkok, Thailand
When a language model answers, we can usually only estimate why. We build a transformer in which every step is a Born measurement - attention ∣⟨q∣k⟩∣4 normalised over each row, a measured product in place of the feed-forward nonlinearity, and a Born readout. With its denominators held at their values for the input, each class probability is a polynomial in the word amplitudes, so its integrated-gradient evidence is computed by a finite quadrature with no quadrature error: the words' contributions sum to the class probability above its no-information share, to 10−11 or better. On e-SNLI natural language inference, with about 98k parameters outside the embedding, the model reaches 76.98±0.17% test accuracy, level with a softmax transformer of the same size (76.62±0.36%). Deleting the word its evidence ranks first changes the answer in 45% of test pairs, out of the 53% any single deletion can change, against 32% for the softmax model explained the same way and 40% by its attention x gradient. Attention can look more like the words annotators highlighted, yet accounts for the answer less well: deleting the word it ranks first changes the Born Transformer's answer in 32% of pairs. The same weights compile exactly to optoelectronic optics and to qubits, and a coherent attention layer becomes one circuit equal to the model (852 CNOTs on 23 qubits). On IBM's ibm_kingston, reading only the attention overlaps by Hadamard tests, the device gave the model's answers on two short test pairs and kept or changed them under deletion as the model does; the evidence itself did not yet survive the device, and we work out the shots and calibration that measuring it on hardware would need.
Figures & tables
Figure 1: Overview, drawn from the trained model’s own data for the test pair “men play hockey . / aliens play” (contradiction). (a) On light: the modulator holds one field per word (phase as colour); in each of two blocks Born attention weighs the words by ∣⟨qi∣kj⟩∣4 read by interference, and the measured feed-forward reads Re[(Wh)(Vh)] by homodyne detection; the camera’s three regions give the Born probabilities, and each word’s Born evidence (BEW; red for the answer, blue against it; dots: the annotators’ words) sums to Pg−bg exactly. (b) On IBM’s ibm_kingston, with the same weights: a word’s query, 16 complex numbers computed classically, is loaded as the amplitudes of four qubits, and every overlap is a Hadamard test (2 × 272 circuits); the measured overlaps follow the ideal ones ( r=0.992 ), the answer survives (contradiction, 0.988 against 0.983 exact), and deleting “aliens”, the word the evidence ranks first, turns the device’s answer to entailment.
Figure 2: Three models step by step, bottom to top, in the box colours of the original transformer figure: a transformer encoder [ 7 ] , the Born Transformer drawn as optics (a beam through a modulator, interferometers with balanced detectors, beam splitters, a lens and a camera; the same weights run on qubits) and the coherent attention layer drawn as its circuit, wires running upwards (one circuit, one measurement). Bands hold the same stage; detector symbols mark the measurements.
Figure 3: The task. One premise of the e-SNLI test set with its three hypotheses, one per label; highlighted are the words the first annotator marked, quoted beneath each is their written explanation. Right: test accuracy of simple baselines and of the two transformers of the same size (three seeds); a bag of words or the hypothesis alone reaches about 65 %, so the rest requires relating the sentences. Below: the dataset in numbers.
test
dev
gap
AOPC
flip@1
agreement
model
(%)
(%)
median / largest
(random)
(random)
with annotators
Born Transformer
76.98±0.17
77.30
6×10−13 / 8×10−12
0.618 (0.125)
0.45 (0.06)
0.510±0.005
Softmax
76.62±0.36
76.36
0.04 / 47
0.321 (0.059)
0.31 (0.05)
0.468±0.003
Poly-sim-born
78.98±0.31
78.89
10−14 / 5×10−12
0.501 (0.065)
0.45 (0.06)
0.527±0.011
Table 1: e-SNLI, 20 epochs, three seeds: accuracy and agreement as mean ± sample standard deviation, the other columns as means (dev on seed means). Gap: how far the word evidence is from summing to the explained score, the worst seed’s median and the largest over the 1,000 pairs (the Born models’ gaps are in probability, the softmax model’s in units of its logit, which it explains by integrated gradients with 256 nodes, not exact for it). AOPC and flip@1 are measured on the predicted class, with a random order of deletion in brackets. Agreement: share of the evidence’s top- k words that annotators highlighted (chance 0.354).
model
ranking
flip@1
of ceiling
flip@5, lowest
agreement
Born Transformer
evidence (BEW)
0.45 [0.43, 0.48]
84 %
0.009
0.51 [0.50, 0.52]
attention
0.32
59 %
0.077
0.50
attention × gradient
0.37
70 %
0.064
0.53
gradient × input
0.41
78 %
0.062
0.46
leave-one-out
0.53 [0.51, 0.56]
100 %
0.006
0.49
Softmax
IG, logit, embeddings
0.31
56 %
0.103
0.47
Table 2: Every ranking on the same 1,000 test pairs, pooled over three seeds per pair; 95 % bootstrap intervals over pairs in brackets (the others are as wide, ±0.02 for flip@1 and ±0.01 for agreement; resampling the seeds as well widens them to about ±0.03 and ±0.015 , and of the comparisons in the text only Born evidence against Born attention on agreement, +0.01 ( −0.02 to +0.03 ), is not resolved). flip@1: deleting the top word changes the answer; of ceiling: flip@1 over the share of pairs that some single deletion changes; flip@5, lowest: deleting the five ranked last changes it; agreement: share of the top- k that annotators highlighted (chance 0.354; on correctly answered pairs only, every value rises by up to 0.03 and the comparisons in the text hold). The softmax model’s gated integrated gradients use 4,096 nodes (median gap 5×10−8 in the worst seed, largest 0.045). Attention rollout [ 13 ] gives agreement 0.52, 0.55 and 0.42.
attention
feed-forward
test (%)
gap (median)
flip@1
flip@5, lowest
agreement
(q⋅k)2 / row sum
bilinear
78.98±0.31
10−14
0.45
0.02
0.53±0.01
(q⋅k)2 / row sum
GELU
77.13±1.42
4×10−3
0.27
0.11
0.46±0.02
softmax
bilinear
65.65±0.44
not complete
0.15
0.25
0.37±0.01
softmax
GELU
65.47±0.25
not complete
0.17
0.23
0.39±0.01
Table 3: Poly-sim-born (first row) with its attention, its feed-forward or both replaced; same size, training and test pairs as Table 1 , three seeds. Evidence: exact quadrature for the polynomial model, integrated gradients with 256 nodes otherwise. flip@1: deleting the word ranked first changes the answer (random 0.05–0.06); flip@5, lowest: deleting the five ranked last does (random 0.17–0.20). Agreement: chance 0.354.
Figure 4: Born evidence per word (BEW) for four e-SNLI test pairs: the first right answer of each class in test order and the first wrong one. Each word is shaded by its evidence for the model’s answer (red for, blue against; scaled within the pair); dots mark the words annotators highlighted; bars give the Born probabilities. In the wrong case the evidence shows why: “calf” in both sentences pulls towards entailment, and “ropes”, the word that should decide, pushes against it but too weakly.
Figure 5: One test pair through the compiled optics, stage by stage: the modulator’s fields (phase as hue), each block’s attention and field after attention, the homodyne readings of the measured feed-forward and the field after it (with the share of the encoded light left), and the camera with its three class regions; below, the answer, the exact evidence of each word and the attention each word receives in the last block.
coherent attention layer
params
test (%)
CNOTs
qubits
post-selection
dense maps
82k
73.14
136,727
27
10−9 – 10−7
brickwall readout
18k
72.41
17,452
25
10−8 – 10−6
every map brickwall (3 seeds)
3.7k
67.04±0.12
852
23
10−5 – 10−4
Table 4: The coherent attention layer as one circuit (8 slots, p=3 ; 10 epochs). The circuit equals the trained model to ≤3.3×10−16 (GPU statevector, two test pairs). Post-selection with the copies prepared apart, for the two pairs. The dense model reaches 74.67±0.18% at 20 epochs over three seeds.
Figure 6: The coherent attention layer as one circuit, left to right: three copies of the sentence prepared, the query–key brickwall UM , a Bell projection that leaves cj,h=⟨xj∣mh⟩ as an amplitude, the slots paired and summed, the value–readout brickwall U , and one measurement of seven qubits. 23 qubits, 852 CNOTs; the circuit equals the trained model.
run
word deleted, and why
answer
device
exact
95 % interval
pair 3763
–
contradiction
0.988
0.983
0.978–0.991
“aliens”, evidence first
entailment
0.573
0.497
0.494–0.652
“play”, evidence nearest zero
contradiction
0.985
0.986
0.976–0.987
pair 3765
–
entailment
0.587
0.596
0.501–0.668
“hockey”, attention first
entailment
0.819
0.768
0.741–0.867
“men”, evidence first
entailment
0.573
0.497
0.494–0.652
Table 5: The runs on ibm_kingston: the probability of the answer, on the device and in the exact model, with the 95 % shot-noise interval of the device’s value; in every redraw the device’s answer was the same. Pair 3763 is “men play hockey . / aliens play”, pair 3765 “men play hockey . / men play”. Pair 3765 without “men” has the same words as pair 3763 without “aliens”, so the ledger reused those jobs. Jobs, circuits and quantum seconds: Table 10 .
Figure 7: The real-device run. Left: the test pair, attention read by Hadamard tests on ibm_kingston (2,000 shots), against the exact model, with 95 % intervals from redrawing each circuit’s shots. Top middle: the same pair with “aliens” deleted, the word the exact evidence ranks first; the device’s answer turns to entailment, as the model’s. Top right: “play” deleted instead, the word the evidence ranks nearest zero; the answer stays. Bottom: a second test pair (entailment), without “hockey”, the word attention ranks first (entailment rises), and without the hypothesis’s “men”, the word the evidence ranks first (the same words, and jobs, as top middle). Only the attention overlaps ran on the device.
slots (words + reference)
12–16
17–32
33–54
test pairs
28
224
48
qubits
25
27
29
Ppass , median
1.6×10−5
6.8×10−6
2.8×10−6
runs per accepted shot, 1/Ppass
6.3×104
1.5×105
3.5×105
amplification rounds
200
300
470
CNOTs per accepted shot, amplified
3×105
5×105
8×105
Table 6: The coherent attention layer’s post-selection, Eq. ( 9 ), on 300 test pairs (the brickwall model, 852 CNOTs per pass). Amplification rounds: π/(4Ppass) , two passes each.
shots per circuit
100
300
1,000
2,000
104
105
answer agrees
0.92
0.96
0.977
0.990
0.997
0.997
∣ΔP∣ , median
0.054
0.029
0.018
0.013
0.006
0.002
evidence: correlation
0.992
0.999
0.9999
evidence: top-3
0.87
0.96
0.98
visibility spread per circuit
0
0.02
0.05
0.1
0.2
answer agrees ( 104 shots)
0.997
0.990
0.963
0.923
0.873
Table 7: The Born Transformer on qubits, simulated: attention overlaps from Hadamard tests with shot noise (and, in the lower rows, a visibility that varies by circuit, or a row-sum calibration that is off), everything else exact; 300 test pairs (evidence: 100). Correlation and top-3 agreement are against the exact evidence.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Born Transformer
softmax
Poly-sim-born
coherent attention layer
state
complex, 32 modes
real, d=64
real, d=64
3 copies of 5 feature qubits, a slot register of p=3 qubits each, 2 head qubits
blocks, heads
2, two of 16 modes
2, two
2, two
1, four (a 2-qubit register)
attention
∣⟨q∣k⟩∣4 over its row sum
softmax
(q⋅k)2 over its row sum
Bell projection ⟨xj∣mh⟩ , unnormalised
feed-forward
Re[(W1h)(W2h)] , 170 channels, gain, W3
GELU, width 256
bilinear, width 170
none
normalisation
none
LayerNorm
none
none
positions
additive fields, a phase per position
learned, additive
learned, additive
additive fields, a phase per position, a field per sentence
Appendix
Table 8: The models. Parameters: total and, in brackets, outside the word embedding. The Born Transformer also has a reference slot with a learned gain; its embeddings and position fields start at scale 1.
data
e-SNLI: 549,367 training, 9,842 dev, 9,824 test pairs; premise <sep> hypothesis, at most 64 tokens; vocabulary of 22,319 (words seen at least 3 times)
loss
cross-entropy of Pg (Born models), of the logits (softmax, Poly-sim-born)
optimiser
Adam, learning rate 2×10−3 (chosen on dev for the Born and softmax models, and used untuned for the others), cosine decay, gradient norm clipped at 1
schedule
batch 64; 20 epochs (coherent layer: 10); word dropout 0.1; seeds 0–2; the last epoch is reported
precision
float32 for training, float64 for evidence and compilation
evidence
integrated gradients from the zero baseline with the denominators held; Gauss–Legendre with half the degree in nodes (Born Transformer: degree 648, 324 nodes)
Appendix
Table 9: Training and evidence, the same for every model unless stated.
job
id
circuits
quantum s
pilot: first 30 circuits of block 1, pair 3763
dbkn49hkdqps7392qgpg
30
18
pair 3763, block 1
dbknllpkdqps7392rnu0
272
147
pair 3763, block 2
dbknn91kdqps7392rra0
272
147
pair 3763 without “aliens”, block 1
dbkntu5gg49c73djaps0
196
106
pair 3763 without “aliens”, block 2
dbknvdt6p51s73cn2bj0
196
106
pair 3765, block 1
dbkrnt0nceqs73amh5g0
272
147
Appendix
Table 10: Jobs on ibm_kingston (Heron r2), Qiskit 2.6.1 and qiskit-ibm-runtime 0.51.0: SamplerV2, 2,000 shots, optimisation level 3, the best of 8 transpiler seeds, dynamical decoupling (XY4). Pair 3765 without “men” (the same words as pair 3763 without “aliens”) used those jobs. Circuits, calibrations and raw counts are in the released ledger.
A companion paper showed that a transformer's feed-forward layer can be rebuilt from explicit fuzzy set operations - intersection, set-difference, and a self-forgetting sequence quantifier - so its hidden units read as named logical operators at no cost to language-model quality. That left the other half of the transformer opaque. Here we carry the same idea into attention and join the two into one model. The mechanism is minimal: a head's value is passed through a sigmoid, so each value channel becomes a readable detector of whether a feature holds at a token. This adds no parameters and leaves the standard head otherwise untouched. A Boolean variant goes further, restructuring the value into an explicit within-token intersection and negation-capable set-difference. In both designs the output projection is left free, not tied to the vocabulary, which is the load-bearing decision: bounding what a head detects while leaving what it writes unconstrained yields selective detectors, whereas constraining the write does not. A bounded value is shaped into a readable detector by two selectivity pressures - one for sparse firing, one for decisive firing at the rails - and which a design wants is not universal. Across five specialized-attention designs at 125M parameters, 44 to 62 percent of value channels become crisp, contextually selective detectors, and their legibility rises with depth rather than crystallizing only on punctuation. Language-model quality is at parity with a conventional baseline. Finally, we couple the Boolean attention to the legible feed-forward layer and train an end-to-end legible-by-construction language model at benchmark parity: its feed-forward units are named set and quantifier operations throughout, and we can take a token it generates and read the named units that compose to produce it.
Mark Oskin
School of Computer Science and Engineering University of Washington
Can controlled symbolic inputs reveal attention heads that influence a language model's decisions from natural language evidence? We introduce controlled evidence aggregation (CEA), which varies the counts and order of two symbols in otherwise fixed prompts and measures changes in a frozen Transformer's preference between two answer tokens. The preference is measured by their logit difference; the prompts prescribe no correct answer. Balanced utility causal selection (BUC) identifies compact head sets by jointly evaluating how much ablating their outputs removes the count effect and how closely patching mean outputs between count conditions recovers the effect in both directions. We derive conditional bounds linking head attribution to intervention effects and specifying when the attribution shortlist can be recovered from finite samples. In Pythia-410M, ablating sets of 1-12 heads selected separately for four ordering regimes removes 65.5-91.6% of the count effect. The sets retain their influence on count comparisons not used for selection. To test relevance beyond symbolic inputs, we evaluate a natural language task asking which state is supported by more reports. Ablating eight heads selected only on symbolic yes/no prompts lowers mean accuracy by 8.2 percentage points, versus 0.8 for random sets matched by size and layer. Our results demonstrate that controlled symbolic probes can identify compact attention head sets whose causal influence extends to natural language decisions.
The attention mechanism forms the foundation of many modern AI models such as the Transformer. In one subclass of problems where attention is used, inputs and outputs are bound to the probability simplex so that all outputs sum to one. In this setting, softmax attention admits an exact, component-by-component quantum realization. Attention scores are Hadamard-test statistics on block-encoded projections of amplitude-encoded inputs. The exponential softmax is the interior of a cosine-squared family generated by Born-rule measurement under an exact bijection, whose boundary expresses sparse attention with exact zeros at finite parameter values. The softmax temperature is a repetition count where post-selected measurement rounds realize discretized inverse temperature exactly. Value aggregation is a deterministic column-loading channel that dilates the column-stochastic value matrix. The gated residual is the preparation angle of a single ancilla, with the additive identity at a mixing angle of π/2. Every learnable parameter is a rotation-gate angle. The composed layer is exact in the infinite-shot limit with one measure-and-reload step per attention score; a fully-coherent variant is ε-approximate via quantum singular value transformation in the infinite depth limit. The algebraic core is machine-checked in Lean 4.
Eric A. F. Reinhardt, Adam J. Hauser
Department of Physics and Astronomy University of Alabama, Tuscaloosa, AL 35487, USA