Automatic Tensor-level Execution of Abstract Interpreters for Neural Networks
Organizations: University of Illinois Urbana-Champaign, USA
Abstract
Developing new abstract interpretation-based DNN certifiers is difficult because their mathematical design is expressed at the neuron level, while implementations must operate at the tensor level. Existing libraries such as auto_LiRPA, IVAN, and ERAN support fixed sets of certifiers; extending them requires expertise in formal methods and tensor programming. Domain-specific languages such as ConstraintFlow let researchers specify certifiers at the neuron level, but cannot run them on real DNNs. We present Neuton, a framework that automatically translates neuron-level certifier specifications into tensor-level executables, enabling new DNN certifiers without manual tensor programming. Neuton makes three contributions. First, a stack-based intermediate representation (IR) and shape analysis infer implicit tensor operations, including broadcasts, dimension expansions, and reductions, required for correct execution. Second, domain-specific rewrite rules optimize common tensor operation sequences into more efficient equivalents. Third, gBCSR is a sparse tensor format using variable-sized, layer-aligned blocks over a non-zero default value, tailored to DNN certifier sparsity. We prove that translation preserves neuron-level semantics. We evaluate Neuton on 8 certifiers and 32 DNN benchmarks. Neuton compiles four existing certifiers and produces the first executable implementations of four others: two previously specified but never run, and two new designs. These expose precision-runtime trade-offs unavailable in existing libraries. For example, SkipPoly matches DeepPoly's precision on all 32 DNNs while running as fast or faster on half. Neuton's runtime on existing certifiers is competitive with hand-optimized libraries. Code: https://github.com/uiuc-focal-lab/constraintflow.git
Figures & tables
| Required Features | Libraries | Specification DSLs | Neuton |
|---|---|---|---|
| (e.g., auto_LiRPA, ERAN, IVAN) | (e.g., ConstraintFlow ) | (This Work) | |
| Execution on real DNNs | ✓ | ✗ | ✓ |
| Supports user-defined certifiers | ✗ | ✓ | ✓ |
| Certifiers specified without tensor programming | ✗ | ✓ | ✓ |
| Automatically derived tensor implementation | ✗ | ✗ | ✓ |
| \frac{\Gamma,\sigma,F\vdash e^{\prime}\leadsto(i_{s_{1}},\,i_{e_{1}},\,m_{1})\hskip 9.24994ptF(f)=(n_{f},\,c_{f},\,e_{f})\hskip 9.24994ptm_{e}^{\mathrm{map}}=\langle\texttt{\footnotesize{\color[rgb]{0,0.5,0}{Neuron}}},\;\mathsf{false},\;[\textsf{polySize}],\;[1]\rangle\hskip 9.24994pt\sigma^{+}=\sigma\bigl[n_{f}\mapsto(\textsf{IrMapNeuron}(i_{e_{1}}),\;m_{e}^{\mathrm{map}}{::}m_{1}^{bc}),\;c_{f}\mapsto(\textsf{IrMapCoeff}(i_{e_{1}}),\;m_{e}^{\mathrm{map}}{::}m_{1})\bigr]\hskip 9.24994pt\Gamma,\;\sigma^{+},\;F\;\vdash\;e_{f}\;\leadsto\;(i_{s_{f}},\;i_{e_{f}},\;m_{e}^{f}{::}m_{\mathit{out}})\hskip 9.24994pti_{e_{pc}}=\textsf{IrReduce}(i_{e_{f}}.C,\;m_{e}^{f})\hskip 9.24994pti_{e_{pz}}=\textsf{IrReduce}(i_{e_{f}}.k,\;m_{e}^{f})\hskip 9.24994pt\textsf{alignShapes}(i_{e_{1}}.k,\;m_{1}[t{\mapsto}\texttt{\footnotesize{\color[rgb]{0,0.5,0}{Real}}}],\;i_{e_{pz}},\;m_{\mathit{out}})=(i_{e_{k}}^{\prime},\;i_{e_{pz}}^{\prime},\;m^{\prime})}{\Gamma,\;\sigma,\;F\;\vdash\;e^{\prime}.\texttt{\footnotesize{\color[rgb]{0.05,0.05,0.9}{map}}}(f)\;\leadsto\;\Bigl(\textsf{IrSeq}(i_{s_{1}},\;i_{s_{f}}),\;\textsf{IrCombineToPoly}\bigl(i_{e_{pc}},\;\textsf{IrBinary}(i_{e_{k}}^{\prime},\;i_{e_{pz}}^{\prime},\;{+})\bigr),\;m^{\prime}\Bigr)}\hskip 9.24994pt\textsc{Left={Map}} |
| DNN | Dataset | Model | Training Method | Architecture | Layers | Perturbation ( ) | Batch Size |
|---|---|---|---|---|---|---|---|
| MNIST | 6x500 | PGD | FCN | 6 | 0.005 | 100 | |
| MNIST | 4x1024 | Standard | FCN | 3 | 0.004 | 100 | |
| MNIST | ConvSmall | DiffAI | Convolutional | 3 | 0.1 | 100 | |
| CIFAR10 | 6x500 | Standard | FCN | 6 | 4e-5 | 100 | |
| CIFAR10 | ConvMed | PGD | Convolutional | 3 | 5e-4 | 10 | |
| CIFAR10 | ConvBig | DiffAI | Convolutional | 6 | 0.002 | 10 |
| Existing Implementations | Prototype Specifications | New Designs | ||||||
| IBP | CROWN-IBP | DeepZ | DeepPoly | ReuseCert | PolyZono | SkipPoly | ZID | |
| ConstraintFlow LOCs | 11 | 27 | 14 | 27 | 25 | 35 | 22 | 67 |
| auto_LiRPA | ✓ | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ |
| IVAN | ✓ | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ |
| ERAN | ✗ | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ |
| Neuton | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Name | Dataset | Model | Training Method | Architecture | Layers | Batch | |
|---|---|---|---|---|---|---|---|
| MNIST | 3x50 | Standard | FCN | 3 | 0.004 | 100 | |
| MNIST | 3x100 | Standard | FCN | 3 | 0.004 | 100 | |
| MNIST | 5x100 | DiffAI | FCN | 6 | 0.004 | 100 | |
| MNIST | 6x100 | Standard | FCN | 6 | 0.004 | 100 | |
| MNIST | 9x100 | Standard | FCN | 9 | 0.004 | 100 | |
| MNIST | 6x200 | Standard | FCN | 6 | 0.004 | 100 |
| Certifier | Baseline | Mean | GeoMean | Median | |
| IBP | auto_LiRPA | 3.0 | 1.6 | 1.0 | 32 |
| IVAN | 2.7 | 1.1 | 0.60 | 31 | |
| DeepZ | IVAN | 1.3 | 0.96 | 1.0 | 31 |
| ERAN | 25 | 17 | 16 | 17 | |
| DeepPoly | auto_LiRPA | 0.73 | 0.58 | 0.63 | 32 |
| IVAN | 7.5 | 3.5 | 3.4 | 24 |
| IBP | DeepZ | DeepPoly | CROWN-IBP | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Network | n | aL | IV | ER | n | aL | IV | ER | n | aL | IV | ER | n | aL | IV | ER |
| 0.04 | 0.04 | 0.03 | – | 0.16 | – | 0.19 | 6.45 | 0.45 | 0.12 | 0.41 | 3.73 | 0.17 | 0.10 | – | – | |
| 0.04 | 0.04 | 0.03 | – | 0.23 | – | 0.23 | 13.3 | 0.60 | 0.18 | 0.65 | 8.05 | 0.17 | 0.10 | – | – | |
| 0.09 | 0.05 | 0.03 | – | 0.60 | – | 0.41 | 30.7 | 1.61 | 0.38 | 1.32 | 27.3 | 0.39 | 0.11 | – | – | |
| 0.08 | 0.05 | 0.03 | – | 0.59 | – | 0.33 | 28.3 | 2.00 | 0.37 | 1.29 | 13.3 | 0.39 | 0.11 | – | – | |
| 0.13 | 0.06 | 0.03 | – | 1.18 | – | 0.47 | 38.7 | 3.58 | 0.91 | 2.07 | 21.3 | 0.68 | 0.13 | – | – | |
| Certifier | Variant | Mean | GeoMean | Median | fail | |
|---|---|---|---|---|---|---|
| IBP | N S | 49 | 24 | 24 | 28 | 4 |
| N R | 51 | 8.0 | 3.3 | 32 | 0 | |
| N R S | 55 | 25 | 26 | 28 | 4 | |
| CROWN-IBP | N S | 10 | 5.7 | 6.2 | 28 | 4 |
| N R | 25 | 6.9 | 4.1 | 29 | 3 | |
| N R S | 20 | 8.7 | 8.8 | 28 | 4 |
| IBP | CROWN-IBP | DeepZ | DeepPoly | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Network | n | S | R | RS | n | S | R | RS | n | S | R | RS | n | S | R | RS |
| 0.03 | 0.09 | 0.04 | 0.08 | 0.19 | 0.26 | 0.21 | 0.23 | 0.07 | 0.12 | 0.10 | 0.12 | 0.33 | 0.44 | 0.42 | 0.44 | |
| 0.03 | 0.10 | 0.04 | 0.10 | 0.19 | 0.27 | 0.24 | 0.27 | 0.07 | 0.13 | 0.10 | 0.14 | 0.32 | 0.46 | 0.80 | 0.79 | |
| 0.05 | 0.19 | 0.08 | 0.17 | 0.47 | 0.50 | 0.66 | 0.60 | 0.25 | 0.32 | 0.31 | 0.34 | 1.27 | 1.55 | 3.52 | 3.29 | |
| 0.06 | 0.16 | 0.08 | 0.17 | 0.58 | 0.60 | 0.65 | 0.60 | 0.24 | 0.31 | 0.33 | 0.34 | 1.17 | 1.54 | 3.52 | 3.28 | |
| 0.09 | 0.27 | 0.15 | 0.27 | 0.84 | 0.83 | 1.49 | 1.16 | 0.58 | 0.75 | 0.79 | 0.76 | 2.97 | 3.18 | 8.21 | 6.95 | |
| ReuseCert | PolyZono | ZID | SkipPoly | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Network | n | S | R | RS | n | S | R | RS | n | S | R | RS | n | S | R | RS |
| 0.17 | 0.25 | 0.36 | 0.38 | 0.50 | 0.46 | 0.52 | 0.49 | 0.21 | 0.26 | 0.23 | 0.24 | 0.27 | 0.33 | 0.34 | 0.34 | |
| 0.18 | 0.26 | 0.65 | 1.01 | 0.50 | 0.59 | 0.91 | 0.86 | 0.20 | 0.28 | 0.27 | 0.29 | 0.28 | 0.36 | 0.67 | 0.75 | |
| 0.45 | 0.58 | 2.35 | 2.35 | 1.82 | 1.86 | 3.76 | 3.42 | 0.50 | 0.58 | 0.67 | 0.60 | 0.93 | 1.07 | 3.00 | 2.87 | |
| 0.45 | 0.57 | 2.38 | 2.36 | 1.93 | 1.43 | 3.84 | 3.49 | 0.49 | 0.53 | 0.70 | 0.61 | 0.79 | 1.02 | 2.95 | 2.88 | |
| 0.93 | 1.43 | 4.10 | 4.05 | 3.68 | 4.33 | 9.12 | 7.79 | 0.85 | 1.03 | 1.49 | 1.15 | 1.87 | 1.84 | 6.60 | 6.06 | |
| Baseline | IBP | CROWN-IBP | DeepZ | DeepPoly |
|---|---|---|---|---|
| auto_LiRPA | – | |||
| IVAN | – | (4 TO) | ||
| ERAN ( ) | – | – |
| Variant | IBP | CROWN-IBP | DeepZ | DeepPoly | ReuseCert | PolyZono | ZID | SkipPoly | Overall |
|---|---|---|---|---|---|---|---|---|---|
| N–S | (1) | (1) | (1) | (1) | (1) | (1) | (1) | (1) | (8) |
| N–R | (3) | (2) | (3) | (2) | (10) | ||||
| N–R–S | (1) | (1) | (1) | (2) | (2) | (2) | (1) | (2) | (12) |
Neural Network Verification
Neural network verification develops methods to prove whether networks satisfy properties such as robustness, safety, and bounded output behavior.
See all 125 papers