How to Turn an Attention Neural Network into an Interpretable Hard-Logic Predicate Circuit: Truth-Table Predicate Extraction and State-Machine Merging under Ship-of-Theseus Replacement
Organizations: The University of Sydney
Abstract
We convert a trained attention network as a whole, attention included, into a hard-logic predicate circuit that runs in its place. Every unit of the circuit is a 0/1 predicate on tokens, and every map between predicates is a minimised truth table. Short tables, such as the product of a query code and a key code, are minimised exactly, and wide feature-to-feature transitions become implication rules by noise-tolerant minimisation. Blocks are replaced one at a time from the output down, in the manner of the Ship of Theseus, and after each replacement the remaining soft part re-adapts to the frozen circuit. Attention selection takes two forms, a row-wise comparison of integer scores (order selection) and a learned rule (rule selection). Both are fitted on the ordinary training set alone, and only on the examples that the teacher classifies correctly, so that the circuit imitates the teacher where the teacher is right. On TweetEval-emotion, the complete circuit of DistilGPT2 reaches a test gold accuracy of 66.5 1.3% with order selection and 57.4 2.5% with rule selection over three seeds, against 74.7% for the teacher and 64.0% for a bag of word pieces fitted on the same tweets, and that of BERT-base reaches 70.0 0.5% with order selection and 63.7 1.2% with rule selection against 77.8%. The circuit needs no multiplier and no weight traffic, and its exported hardware description reproduces it bit for bit. By a synthesis-based estimate it classifies one tweet in about 1-2 s (DistilGPT2) and 3-5 s (BERT) on an FPGA card and in 5-9 s on a palm-sized board whose module draws 7.5-15 W, two to three orders of magnitude faster than the network on a GPU at batch 1 (measured, 0.9-2.6 ms for DistilGPT2 including an optimised CUDA-graph baseline), at a cost of 8.5 (DistilGPT2) and 8.6 (BERT) points of gold accuracy for the synthesised seed-0 circuits.
Figures & tables
| Model | Gold | Agreement | Mbit |
|---|---|---|---|
| DistilGPT2 (decoder, 6 blocks) | |||
| teacher (FP8) | 0.747 | 1 | 655.3 |
| order selection | 0.665 0.013 | 0.695 0.012 | 47.87 |
| rule selection | 0.574 0.025 | 0.597 0.029 | 67.68 |
| bag of words | 0.640 | 0.653 | – |
| BERT-base (encoder, 12 blocks) | |||
| DistilGPT2 | BERT-base | ||||
|---|---|---|---|---|---|
| Fit set | Gold | Agr. | Gold | Agr. | |
| order selection | teacher-correct | 0.662 | 0.692 | 0.694 | 0.705 |
| all | 0.639 | 0.679 | 0.681 | 0.701 | |
| rule selection | teacher-correct | 0.552 | 0.579 | 0.624 | 0.628 |
| all | 0.575 | 0.566 | 0.598 | 0.611 | |
| bag of words | teacher-correct | 0.640 | 0.653 | 0.640 | 0.650 |
| Model | System | Latency, one tweet | Power | Form |
|---|---|---|---|---|
| DistilGPT2 | teacher on an RTX 3090, eager (M) | 2.60 ms | board 136–181 W (M) | card in a desktop |
| teacher on an RTX 5090, eager (M) | 2.21 ms | board 119–160 W (M) | card in a desktop | |
| teacher on an RTX 5090, CUDA graphs (M) | 0.89 ms (0.29) | board 161 W (M) | card in a desktop | |
| circuit on an Alveo U55C card (E) | 1.06–1.91 s | card 115 / 150 W (S) | card in a host | |
| circuit on a Kria KV260 board (E) | 5.0–9.0 s | module 7.5 / 15 W (S) | palm-sized board | |
| BERT-base | teacher on an RTX 3090, eager (M) | 4.15 ms | board 139–213 W (M) | card in a desktop |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Item | Setting |
|---|---|
| Task | TweetEval-emotion, 4 classes, official splits of 3,257 training, 374 validation and 1,421 test tweets |
| BERT teacher | BERT-base-uncased, 12 blocks of 12 heads, bidirectional attention, classifier on [CLS] . FP16 mixed precision with FP32 master weights and a dynamic loss scale, learning rate , 5 epochs, batch 128, weight decay 0.01, warm-up 10%, class-weighted cross-entropy, maximum length 64, seed 42, epoch selected on validation |
| DistilGPT2 teacher | DistilGPT2, 6 blocks of 12 heads, causal attention, an end-of-text token appended as the readout token and a new padding token. The BERT protocol with learning rate and 10 epochs, the learning rate chosen on validation macro-F1 among , and |
| FP8 teacher | post-training E4M3 quantisation of every linear layer and of both attention products, weights scaled per output channel, operands of every product scaled dynamically per tensor (per head in attention), FP32 accumulation, softmax, LayerNorm and GELU in FP32 |
| Teacher accuracy | gold accuracy on training / test: 0.931 / 0.747 (DistilGPT2), 0.827 / 0.778 (BERT) |
| Fit set | training tweets that the FP8 teacher classifies correctly: 3,032 (DistilGPT2) and 2,695 (BERT) of 3,257. Ablation: all 3,257 training tweets with the teacher’s labels |
| Model | Agr. (val / test) | Gold acc. |
|---|---|---|
| FP16 inference | 1.000 / 1.000 | 0.780 |
| FP8 attention products only | 0.987 / 0.996 | 0.781 |
| FP8 teacher (linear + attention) | 0.976 / 0.991 | 0.778 |
| Logic multiplier in attention | 0.976 / 0.991 | 0.778 |
| Fit set | Model | DistilGPT2 | BERT-base |
|---|---|---|---|
| teacher-correct (main) | order selection | 0.621 0.013 | 0.591 0.024 |
| rule selection | 0.469 0.019 | 0.486 0.066 | |
| bag of words | 0.428 | 0.463 | |
| all training tweets (ablation) | order selection | 0.627 | 0.624 |
| rule selection | 0.470 | 0.452 | |
| bag of words | 0.416 | 0.466 |
| Quantity | Status | Basis |
|---|---|---|
| Accuracy and agreement | measured | the software circuit on the test split ( Tables 1 and 2 ) |
| Bit-exact export | measured | integer evaluator of the exported description against the software circuit on every token state and class of the 1,421 test tweets; synthesised netlists simulated on real inputs, 0 mismatches |
| GPU throughput, latency, board power | measured | teacher in eager PyTorch, FP16 and FP32, tweets grouped by length without padding, throughput at batch 256, latency at batch 1, board power sampled about every 50 ms over windows of at least 20 s, on an RTX 3090, an RTX 5090 and an RTX PRO 6000; on the RTX 5090 also with CUDA graphs in FP16, one captured graph per input shape, predictions equal to the eager ones on all test tweets |
| B200 and B300 throughput | estimated | anchored on the measured RTX PRO 6000, from launch-bound (same throughput) to compute-bound (the smaller of the dense FP16 and memory-bandwidth ratios) |
| FPGA area | estimated | Yosys and ABC synthesis to 6-input lookup tables and UltraScale+ resource counts, no place-and-route; a design fits a device if it uses at most 70% of its lookup tables and registers; control logic and host interface assumed |
| Latency of the logic | estimated | cycle count of the pipelined design at the assumed clocks, mean over the lengths of the test tweets |
| System (price) | Tweets/s | Energy per tweet | |
|---|---|---|---|
| (a) card | RTX 3090 ($1,499), teacher in PyTorch (M) | 14,845 | 22.6 mJ |
| RTX 5090 ($1,999), teacher in PyTorch (M) | 45,226 | 12.1 mJ | |
| RTX 5090 ($1,999), teacher with CUDA graphs (M) | 51,166 | – | |
| Alveo U55C ($4,747), circuit logic (E) | 3.4–6.1 M | 18.9–44.5 J | |
| (b) system | x86 desktop with RTX 5090 ($3,260), teacher (M; host power E) | 45,226 | 16.4–18.1 mJ |
| two tokenising desktops and one Kria KV260 board ($3,084), circuit (E) | 355k–434k | 1.07–1.56 mJ |
| GPU (system cost) | GPU system | Circuit system (cost) | Circuit system | Ratio | |||
|---|---|---|---|---|---|---|---|
| tweets/s | mJ/tweet | tweets/s | mJ/tweet | tweets/s | energy | ||
| DistilGPT2 | |||||||
| RTX 5090 ($3.26k) | 45.2k M | 16.4–18.1 | 2 PCs + 1 KV260 ($3.08k) | 355k–434k | 1.07–1.56 | 7.9–9.6 | 10.5–16.9 |
| RTX PRO 6000 ($25.6k) | 38.9k M | 12.6–14.4 | 17 PCs + 6 KV260 ($25.5k) | 3.02–3.69 M | 1.05–1.54 | 78–95 | 8.2–13.7 |
| B200 ($63.9k) | 38.9k–174k E | 10.3–46 | 42 PCs + 15 KV260 ($63.1k) | 7.46–9.12 M | 1.05–1.54 | 43–235 | 6.7–43.8 |
| B300 ($77.0k) | 38.9k–174k E | 10.4–48.5 | 51 PCs + 18 KV260 ($76.6k) | 9.06–11.08 M | 1.05–1.54 | 52–285 | 6.8–46.2 |
| DistilGPT2 | BERT-base | |||
| Part | Literals | Mbit | Literals | Mbit |
| Input table | – | 47.49 / 47.49 | – | 28.84 / 28.84 |
| Code maps | 117,585 / 117,252 | 0.12 / 0.12 | 237,104 / 235,536 | 0.24 / 0.24 |
| Score tables | 110,848 / – | 0.17 / – | 214,992 / – | 0.34 / – |
| Selection rules | – / 757 | – / 20.05 | – / 1,639 | – / 40.11 |
| Transition rules | 12,006 / 6,163 | – | 24,820 / 7,960 | – |
Neural Network Interpretability
Neural network interpretability studies how models represent information and make predictions, and how to explain, test, or control their internal computations.
See all 367 papers