How to Turn an Attention Neural Network into an Interpretable Hard-Logic Predicate Circuit: Truth-Table Predicate Extraction and State-Machine Merging under Ship-of-Theseus Replacement
We convert a trained attention network as a whole, attention included, into a hard-logic predicate circuit that runs in its place. Every unit of the circuit is a 0/1 predicate on tokens, and every map between predicates is a minimised truth table. Short tables, such as the product of a query code and a key code, are minimised exactly, and wide feature-to-feature transitions become implication rules by noise-tolerant minimisation. Blocks are replaced one at a time from the output down, in the manner of the Ship of Theseus, and after each replacement the remaining soft part re-adapts to the frozen circuit. Attention selection takes two forms, a row-wise comparison of integer scores (order selection) and a learned rule (rule selection). Both are fitted on the ordinary training set alone, and only on the examples that the teacher classifies correctly, so that the circuit imitates the teacher where the teacher is right. On TweetEval-emotion, the complete circuit of DistilGPT2 reaches a test gold accuracy of 66.5 ± 1.3% with order selection and 57.4 ± 2.5% with rule selection over three seeds, against 74.7% for the teacher and 64.0% for a bag of word pieces fitted on the same tweets, and that of BERT-base reaches 70.0 ± 0.5% with order selection and 63.7 ± 1.2% with rule selection against 77.8%. The circuit needs no multiplier and no weight traffic, and its exported hardware description reproduces it bit for bit. By a synthesis-based estimate it classifies one tweet in about 1-2 μs (DistilGPT2) and 3-5 μs (BERT) on an FPGA card and in 5-9 μs on a palm-sized board whose module draws 7.5-15 W, two to three orders of magnitude faster than the network on a GPU at batch 1 (measured, 0.9-2.6 ms for DistilGPT2 including an optimised CUDA-graph baseline), at a cost of 8.5 (DistilGPT2) and 8.6 (BERT) points of gold accuracy for the synthesised seed-0 circuits.
Figures & tables
Figure 1: The method. (a) Blocks are hardened from the top down (reverse Ship of Theseus), and after each stage the soft blocks below re-adapt to the frozen circuit, so that every stage is a complete classifier. (b) In one block, tokens carry discrete states (1), and the heads of a query token select keys (2) by integer scores (order selection) or by a learned rule (rule selection). Predicates on the selected keys, merged when equal on all data (3), form a truth table on the rows that the data reach (4, grey: don’t care), and its minimised OR of ANDs (5) gives every token its next state (6). (c) The predicates of the readout token that hold add their integer votes. (d) The largest sum gives the class, which is traced through the rules for the tweet of Section 4.5 (DistilGPT2, order selection, emoji drawn as one token).
Model
Gold
Agreement
Mbit
DistilGPT2 (decoder, 6 blocks)
teacher (FP8)
0.747
1
655.3
order selection
0.665 ± 0.013
0.695 ± 0.012
47.87
rule selection
0.574 ± 0.025
0.597 ± 0.029
67.68
bag of words
0.640
0.653
–
BERT-base (encoder, 12 blocks)
Table 1: Complete circuits, fitted on the training tweets that the teacher classifies correctly. Test gold accuracy, agreement with the teacher on test, and size. Circuits: mean ± s.d. over 3 seeds, sizes of seed 0. The bag of words is fitted on the same tweets with the teacher’s labels.
DistilGPT2
BERT-base
Fit set
Gold
Agr.
Gold
Agr.
order selection
teacher-correct
0.662
0.692
0.694
0.705
all
0.639
0.679
0.681
0.701
rule selection
teacher-correct
0.552
0.579
0.624
0.628
all
0.575
0.566
0.598
0.611
bag of words
teacher-correct
0.640
0.653
0.640
0.650
Table 2: Fitting only where the teacher is right. The same conversion fitted on the training tweets that the teacher classifies correctly (teacher-correct: 3,032 for DistilGPT2, 2,695 for BERT) or on all 3,257 training tweets with the teacher’s labels. Test gold accuracy and agreement with the teacher on test, seed 0.
Model
System
Latency, one tweet
Power
Form
DistilGPT2
teacher on an RTX 3090, eager (M)
2.60 ms
board 136–181 W (M)
card in a desktop
teacher on an RTX 5090, eager (M)
2.21 ms
board 119–160 W (M)
card in a desktop
teacher on an RTX 5090, CUDA graphs (M)
0.89 ms (0.29)
board 161 W (M)
card in a desktop
circuit on an Alveo U55C card (E)
1.06–1.91 μ s
card 115 / 150 W (S)
card in a host
circuit on a Kria KV260 board (E)
5.0–9.0 μ s
module 7.5 / 15 W (S)
palm-sized board
BERT-base
teacher on an RTX 3090, eager (M)
4.15 ms
board 139–213 W (M)
card in a desktop
Table 3: Latency and power for one tweet. Circuits: order selection, seed 0, estimated (E) from synthesis and a stated performance and power model, latency of the logic as the mean over the test tweets at 250–450 MHz. Card and module power: typical / maximum, sourced (S) from the vendor; for the ZCU104, whose board power is not published, the dynamic power of the logic (E). GPUs: the teacher at batch 1, measured (M), mean latency, in eager PyTorch the better of FP16 and FP32 with board power over both, with CUDA graphs FP16 (best tweet in parentheses), board power without the host. Gold accuracy (seed 0): DistilGPT2 circuit 0.662, teacher 0.747; BERT circuit 0.694, teacher 0.780.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Item
Setting
Task
TweetEval-emotion, 4 classes, official splits of 3,257 training, 374 validation and 1,421 test tweets
BERT teacher
BERT-base-uncased, 12 blocks of 12 heads, bidirectional attention, classifier on [CLS] . FP16 mixed precision with FP32 master weights and a dynamic loss scale, learning rate 2⋅10−5 , 5 epochs, batch 128, weight decay 0.01, warm-up 10%, class-weighted cross-entropy, maximum length 64, seed 42, epoch selected on validation
DistilGPT2 teacher
DistilGPT2, 6 blocks of 12 heads, causal attention, an end-of-text token appended as the readout token and a new padding token. The BERT protocol with learning rate 2⋅10−4 and 10 epochs, the learning rate chosen on validation macro-F1 among 5⋅10−5 , 10−4 and 2⋅10−4
FP8 teacher
post-training E4M3 quantisation of every linear layer and of both attention products, weights scaled per output channel, operands of every product scaled dynamically per tensor (per head in attention), FP32 accumulation, softmax, LayerNorm and GELU in FP32
Teacher accuracy
gold accuracy on training / test: 0.931 / 0.747 (DistilGPT2), 0.827 / 0.778 (BERT)
Fit set
training tweets that the FP8 teacher classifies correctly: 3,032 (DistilGPT2) and 2,695 (BERT) of 3,257. Ablation: all 3,257 training tweets with the teacher’s labels
Appendix
Table 4: Data, teachers, fit sets and metrics.
Model
Agr. (val / test)
Gold acc.
FP16 inference
1.000 / 1.000
0.780
FP8 attention products only
0.987 / 0.996
0.781
FP8 teacher (linear + attention)
0.976 / 0.991
0.778
Logic multiplier in attention
0.976 / 0.991
0.778
Appendix
Table 5: BERT: from the FP16 teacher to the FP8 teacher and logic arithmetic. Agreement with the teacher’s full-precision predictions on validation and test, and test gold accuracy.
Fit set
Model
DistilGPT2
BERT-base
teacher-correct (main)
order selection
0.621 ± 0.013
0.591 ± 0.024
rule selection
0.469 ± 0.019
0.486 ± 0.066
bag of words
0.428
0.463
all training tweets (ablation)
order selection
0.627
0.624
rule selection
0.470
0.452
bag of words
0.416
0.466
Appendix
Table 6: Agreement with the teacher on the 2,000 out-of-domain tweets, which enter no conversion. Main rows: mean ± s.d. over 3 seeds; ablation rows: seed 0.
Quantity
Status
Basis
Accuracy and agreement
measured
the software circuit on the test split ( Tables 1 and 2 )
Bit-exact export
measured
integer evaluator of the exported description against the software circuit on every token state and class of the 1,421 test tweets; synthesised netlists simulated on real inputs, 0 mismatches
GPU throughput, latency, board power
measured
teacher in eager PyTorch, FP16 and FP32, tweets grouped by length without padding, throughput at batch 256, latency at batch 1, board power sampled about every 50 ms over windows of at least 20 s, on an RTX 3090, an RTX 5090 and an RTX PRO 6000; on the RTX 5090 also with CUDA graphs in FP16, one captured graph per input shape, predictions equal to the eager ones on all test tweets
B200 and B300 throughput
estimated
anchored on the measured RTX PRO 6000, from launch-bound (same throughput) to compute-bound (the smaller of the dense FP16 and memory-bandwidth ratios)
FPGA area
estimated
Yosys and ABC synthesis to 6-input lookup tables and UltraScale+ resource counts, no place-and-route; a design fits a device if it uses at most 70% of its lookup tables and registers; control logic and host interface assumed
Latency of the logic
estimated
cycle count of the pipelined design at the assumed clocks, mean over the lengths of the test tweets
Appendix
Table 7: Status of every quantity in the hardware comparison.
System (price)
Tweets/s
Energy per tweet
(a) card
RTX 3090 ($1,499), teacher in PyTorch (M)
14,845
22.6 mJ
RTX 5090 ($1,999), teacher in PyTorch (M)
45,226
12.1 mJ
RTX 5090 ($1,999), teacher with CUDA graphs (M)
51,166
–
Alveo U55C ($4,747), circuit logic (E)
3.4–6.1 M
18.9–44.5 μ J
(b) system
x86 desktop with RTX 5090 ($3,260), teacher (M; host power E)
45,226
16.4–18.1 mJ
two tokenising desktops and one Kria KV260 board ($3,084), circuit (E)
355k–434k
1.07–1.56 mJ
Appendix
Table 8: Throughput and energy per tweet for the order selection circuit of DistilGPT2. M: measured, E: estimated. (a) Card only, without tokenisation. GPU throughput and energy in FP16 at batch 256. FPGA ranges span clocks of 250–450 MHz and typical to maximum board power. (b) Equal whole-system cost, end to end with tokenisation, against an x86 desktop with an RTX 5090.
GPU (system cost)
GPU system
Circuit system (cost)
Circuit system
Ratio
tweets/s
mJ/tweet
tweets/s
mJ/tweet
tweets/s
energy
DistilGPT2
RTX 5090 ($3.26k)
45.2k M
16.4–18.1
2 PCs + 1 KV260 ($3.08k)
355k–434k
1.07–1.56
7.9–9.6 ×
10.5–16.9 ×
RTX PRO 6000 ($25.6k)
38.9k M
12.6–14.4
17 PCs + 6 KV260 ($25.5k)
3.02–3.69 M
1.05–1.54
78–95 ×
8.2–13.7 ×
B200 ($63.9k)
38.9k–174k E
10.3–46
42 PCs + 15 KV260 ($63.1k)
7.46–9.12 M
1.05–1.54
43–235 ×
6.7–43.8 ×
B300 ($77.0k)
38.9k–174k E
10.4–48.5
51 PCs + 18 KV260 ($76.6k)
9.06–11.08 M
1.05–1.54
52–285 ×
6.8–46.2 ×
Appendix
Table 9: Equal-cost comparison over four classes of GPU, end to end with tokenisation. For each GPU system, the circuit system is the best of the configurations searched at the same whole-system budget: x86 desktops (PCs) that tokenise and feed shared FPGA boards. Throughput in tweets/s, energy per tweet of the whole system. GPU side: M measured, E estimated; circuit side: estimated. Gold accuracy: teachers 0.747 (DistilGPT2) and 0.778 (BERT), circuits in Table 1 .
Figure 2: Every stage of the conversion is a complete classifier. Test gold accuracy and agreement with the teacher on test of the hybrid after the top k blocks have been replaced by circuits, fitted on the teacher-correct training tweets, seed 0: DistilGPT2 with both forms of selection (top), BERT-base with order selection (bottom). k=0 is the FP8 teacher, and the last point is the complete circuit.
DistilGPT2
BERT-base
Part
Literals
Mbit
Literals
Mbit
Input table Λ
–
47.49 / 47.49
–
28.84 / 28.84
Code maps
117,585 / 117,252
0.12 / 0.12
237,104 / 235,536
0.24 / 0.24
Score tables
110,848 / –
0.17 / –
214,992 / –
0.34 / –
Selection rules
– / 757
– / 20.05
– / 1,639
– / 40.11
Transition rules
12,006 / 6,163
–
24,820 / 7,960
–
Appendix
Table 10: Size of the complete circuits by part (teacher-correct fit set, seed 0), order selection / rule selection. Literals count the two-level logic of the minimised tables and rules, and Mbit counts table storage, with the input table over the whole vocabulary at every position. The vote readout reads 2,475 / 281 predicates for DistilGPT2 and 6,087 / 1,500 for BERT. With rule selection the score tables are not part of the circuit, and the selection rules (757 and 1,639 literals as logic) are stored as compiled one-bit tables. Teachers: every parameter at 8 bits, 81,916,416 parameters for DistilGPT2 (with the added padding token and the four-class head) and 109,485,316 for BERT-base.
Neural network interpretability studies how models represent information and make predictions, and how to explain, test, or control their internal computations.
Finding circuits in language models usually means running many careful interventions. We try something simpler: treat attention as a routing map from one forward pass, keep a small set of routes that point toward the answer, and ask whether those routes actually matter. They often do. On induction and IOI (tasks where the "right" circuit is already known), ablating our extracted edges hurts the model much more than ablating a random set of the same size. We evaluate n=100 prompts per cell on GPT-2 Small, GPT-2 Medium, and Pythia-410M, with paired gap tests and bootstrap confidence intervals. The extract step costs one forward; a head-by-head patch sweep costs about two orders of magnitude more. We are not claiming a complete circuit atlas. We are claiming a cheap sketch that carries real causal signal on known tasks, with clear failure modes when it does not. Code and evaluation artifacts are at https://github.com/Aquinf03/live-circuit-routing.
Circuit extraction identifies model components whose presence preserves a target behavior under ablation, but preserving behavior does not single out one circuit. We study this in a synthetic Lean tactic-prediction benchmark with fixed proof rules and randomized surface form. Across dense and weight-sparse checkpoints of the same transformer, evaluated on atomic and compositional proofs, we vary which extracted object is reported: a compact prediction-preserving circuit, a broader graph retaining surrounding read, write, and routing structure, or a subgraph extracted with a post-ablation loss budget. First, overlap between groups of component-to-component connections is low, sometimes close to a random baseline. Overlap is much higher for selected attention-head sets under deterministic selection, but falls sharply when selection among equally scored heads is randomized. Second, the normalized graph-size ranking of conditions that differ in which supervised checkpoint initializes reinforcement learning (RL) agrees between repeated extractions of the compact circuit and between joint and separate query/key representations of each of the other two graphs. For the loss-budgeted graph, query/key ordering agreement persists through the budget sweep. Finally, across those same conditions, the largest exact-match accuracy gains from RL on compositional proofs come with the largest proportion of components in the compositional circuit that lie outside the corresponding atomic circuits. Ablating these additional components in the sparse RL model reduces compositional accuracy while largely preserving accuracy on the corresponding atomic tasks. A circuit-level claim therefore requires specifying which circuit is reported and how it is extracted and compared; behavior alone does not determine its interpretation. We distill these requirements into a reporting practice for circuit-extraction studies.
Yang Sheng, Jie Fu
Fudan University · Shanghai Innovation Institute · IQuest Research
We present a PyTorch package that compiles neural networks and their weights from Turing machine descriptions, producing models that exactly simulate the specified machine without any training. Given a transition function and a set of terminal states, the package constructs a model whose forward pass corresponds to one step of the Turing machine. Two architectures are implemented, each realizing a different theoretical result: (1) a transformer with self-attention, cross-attention, and feedforward layers based on Wei, Chen, and Ma (2021), and (2) a recurrent network based on Siegelmann and Sontag (1995) that encodes the stack in a Cantor set. We develop the constructions from first principles, showing how ReLU networks implement Boolean circuits (AND, OR, NOT, XOR gates and their composition into DNF formulas and binary adders) and how hard attention implements positional lookup on the tape. The package serves as a concrete, runnable reference for the symbolic-neural bridge, and as a foundation for future work on the stability of constructed solutions under gradient-based optimization. Code is available at https://github.com/jonrbates/turing.