cs.LGOct 3, 2026
SaveSpecific Algorithmic Interpretability of Neural Networks: A Case Study on Textures
Organizations: Technion – Israel Institute of Technology, Faculty of Mathematics · University of Minnesota, School of Mathematics
Abstract
We develop a principled framework for constructing neural networks whose specific parameter realizations admit an explicit algorithmic interpretation. Existing algorithm-inspired architectures can explain the computational structure of a network, yet after standard training the learned parameters need not retain a clear relation to the motivating algorithm. We address this gap as follows. First, we model each data point as a sample of a class-dependent stochastic process and assume that statistics of this process can be estimated from a single sample and these statistics are sufficient to distinguish the classes. We then construct a neural network whose initial parameters exactly implement an algorithm for estimating these statistics, making the network fully interpretable. To account for mismatch between the idealized model and real data, we fine-tune this network while controlling its deviation from the algorithmic initialization. The trained network hence roughly retains the interpretation of the initial network. A PAC-Bayesian analysis yields a uniform generalization bound whose complexity term scales with the fine-tuning radius, providing a statistical motivation for our approach. We instantiate the framework for texture classification using the scattering transform to estimate the discriminative statistics.
Figures & tables
Figure 1: Texture synthesis by matching wavelet scattering statistics. Synthesized images show that the selected scattering statistics retain perceptually meaningful texture information.
Figure 2: Percentage of data used during training versus test accuracy. Color denotes method and line style denotes regularization strength. The proposed method gives the best performance when the amount of training data is small.
Figure 3: Fine-tuning distance versus generalization gap. Color denotes method, line style denotes regularization strength, and size denotes amount of training data used. Gray lines between fine-tuned scattering networks trained on the same data size allow comparison across regularization strengths. Smaller fine-tuning distance correlates with lower generalization gap.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Trainable scattering architecture. At initialization, the convolutional operators reproduce independent wavelet filtering and hence the predefined scattering transform. Fine-tuning allows the wavelet filters and permitted cross-channel interactions to adapt from data.
| Dataset | Params | Blocks | GAP | Mix. hor. | BN | LR step | WD |
|---|---|---|---|---|---|---|---|
| DTD | 2.74M | 6 | True | 3 | False | 50 | 0.005 |
| EuroSat | 2.08M | 4 | False | 27 | False | 20 | 0.0001 |
| MinC-2500 | 2.56M | 6 | True | 3 | True | 50 | 0.005 |
| SuroTex | 112K | 4 | True | 3 | True | 20 | 0.005 |
Table 1: Architecture configuration for proposed method: includes hyper-parameters that differ across datasets. Rows sharing a configuration are merged (their architectures listed together). Values that are not listed were help fixed across all datasets (for example, angles (L) = 4, and low-pass last was always used).
| Dataset | Params | Blocks | Nonlin. | P proj. |
|---|---|---|---|---|
| DTD | 26.8M | 6 | relu | 64/128/256/512/512/512 |
| EuroSat | 24.1M | 5 | softshrink | 64/128/256/512/512 |
| MinC-2500 | 9.47M | 4 | relu | 64/128/256/512 |
| SuroTex | 37.2M | 6 | softsrhink | 64/128/256/512/512/512 |
Table 2: Architecture configuration for ScatNet B.A., part 1 of 2: hyper-parameters that differ across the reported models. Rows sharing a configuration are merged (their architectures listed together). Values that are not listed were held fixed across all datasets (for example, batch norm was always used).
| Dataset | Frame w. | LR | LR step | Epochs | Batch |
|---|---|---|---|---|---|
| DTD | 512/1024/1024/2048/2048/2048 | 0.01 | 60 | 200 | 128 |
| EuroSat | 1024/2048/4096/8192/8192 | 0.01 | 70 | 300 | 128 |
| MinC-2500 | 512/1024/1024/2048 | 0.01 | 60 | 200 | 32 |
| SuroTex | 1024/2048/4096/8192/8192/8192 | 0.001 | 70 | 300 | 25 |
Table 3: Architecture configuration for ScatNet B.A., part 2 of 2.
Figure 5: Test accuracy versus generalization gap. Color denotes method, line style denotes regularization strength, and marker size denotes percentage of training data used. Increasing regularization strength decreases generalization error on DTD, MinC-2500, and SuroTex.
Figure 6: Test accuracy versus generalization gap, with weight decay regularization on randomly initialized model. Color denotes method, line weight denotes regularization strength, and size denotes percentage of training data. Weight decay controls the generalization gap at the cost of reducing test accuracy on DTD, EuroSat, and MinC-2500.
Figure 7: Test accuracy versus generalization gap, comparing effect of different parameter anchors on randomly initialized model. Within each subplot, color indicates method and parameter anchor, and marker size indicates percentage of training data used. The randomly initialized model with scattering anchor has higher generalization error than the fine-tuned scattering model, likely because the fine-tuning distance is greater on DTD and MinC-2500.
Figure 8: Normalized within-class variance in classifier subspace versus test accuracy. Within sub-plots, color denotes method, marker size denotes percentage of training data used, and cross-hatches denote standard deviation. Rows denote regularization strength and columns denote datasets.
Figure 9: Percentage of data used during training versus relative magnitude of zero-frequency component. Wavelets must satisfy an admissibility condition, that the zero-frequency component has magnitude zero. Color denotes method, line style denotes regularization strength. Increasing regularization strength decreases the relative magnitude of the zero-frequency component on all datasets.
Figure 10: Fine-tuning distance versus (similarity to best-fit Morlet filter). Color denotes method and marker style denotes regularization strength. Marker size denotes the percentage of data used during training. Smaller fine-tuning distance is correlated with more similarity to a Morlet filter across all four datasets.
Figure 11: Relative change in the fitted Morlet parameters during fine-tuning. For each fine-tuned filter, we compare the parameters of its best-fit Morlet wavelet to those of the matched filter in the initial bank. Regularizing the fine-tuning distance keeps the parameters closer to their initial values across datasets.
| Model | Metric | 10% | 20% | 30% | 50% | 100% |
|---|---|---|---|---|---|---|
| Random Init. (L2=0) | Test acc. | 11.06 (0.67) | 16.13 (0.67) | 20.45 (0.40) | 26.43 (0.31) | 34.10 (0.92) |
| Gen. Gap | 87.10 (2.73) | 83.26 (1.04) | 79.46 (0.39) | 72.98 (1.00) | 65.19 (0.54) | |
| NC1 | 6.42 (0.21) | 5.53 (0.30) | 4.46 (0.29) | 3.12 (0.21) | 2.33 (0.17) | |
| Random Init. (L2=0.1) | Test acc. | 9.94 (0.51) | 13.44 (1.15) | 14.66 (0.69) | 15.87 (1.62) | 17.42 (1.12) |
| Gen. Gap | 67.70 (6.84) | 49.16 (10.15) | 25.21 (6.66) | 10.82 (4.18) | 3.66 (2.84) | |
| NC1 | 5.55 (0.30) | 5.37 (0.87) | 4.78 (0.43) | 4.56 (0.52) | 4.33 (0.33) |
Table 4: Test accuracy, generalization gap and normalized within-class variance in classifier subspace (NC1) across architectures for DTD, by training-set fraction. Each cell is the mean over seeds, with the standard deviation in parentheses, set smaller. Best test accuracy per column in bold.
| Model | Metric | 0.5% | 1% | 5% | 10% | 15% |
|---|---|---|---|---|---|---|
| Random Init. (L2=0) | Test acc. | 45.40 (3.57) | 56.45 (2.16) | 74.98 (1.21) | 80.48 (0.27) | 82.57 (0.83) |
| Gen. Gap | 54.60 (3.57) | 42.77 (2.50) | 24.99 (1.27) | 19.51 (0.28) | 17.43 (0.83) | |
| NC1 | 1.29 (0.22) | 0.80 (0.26) | 0.58 (0.12) | 0.52 (0.04) | 0.51 (0.05) | |
| Random Init. (L2=0.1) | Test acc. | 24.78 (4.44) | 57.07 (3.67) | 73.64 (2.94) | 80.38 (0.71) | 83.65 (0.48) |
| Gen. Gap | 13.37 (2.94) | 41.57 (2.78) | 19.65 (7.42) | 15.22 (1.82) | 10.37 (0.54) | |
| NC1 | 2.56 (1.10) | 0.87 (0.25) | 0.49 (0.04) | 0.49 (0.06) | 0.44 (0.03) |
Table 5: Test accuracy, generalization gap and normalized within-class variance in classifier subspace (NC1) across architectures for EuroSat (Part 1 of 2), by training-set fraction. Each cell is the mean over seeds, with the standard deviation in parentheses, set smaller. Best test accuracy per column in bold.
| Model | Metric | 20% | 30% | 50% | 100% |
|---|---|---|---|---|---|
| Random Init. (L2=0) | Test acc. | 84.44 (1.18) | 86.73 (0.27) | 88.94 (0.74) | 91.98 (0.53) |
| Gen. Gap | 15.56 (1.18) | 13.26 (0.28) | 11.03 (0.70) | 7.47 (0.23) | |
| NC1 | 0.47 (0.04) | 0.44 (0.01) | 0.40 (0.01) | 0.33 (0.03) | |
| Random Init. (L2=0.1) | Test acc. | 85.94 (0.81) | 88.00 (0.83) | 88.13 (3.15) | 85.94 (4.40) |
| Gen. Gap | 8.01 (1.52) | 5.04 (0.24) | 2.19 (3.14) | 0.73 (2.15) | |
| NC1 | 0.41 (0.02) | 0.39 (0.01) | 0.39 (0.03) | 0.37 (0.02) |
Table 6: Test accuracy, generalization gap and normalized within-class variance in classifier subspace (NC1) across architectures for EuroSat (Part 2 of 2), by training-set fraction. Each cell is the mean over seeds, with the standard deviation in parentheses, set smaller. Best test accuracy per column in bold.
| Model | Metric | 0.2% | 0.5% | 1% | 5% | 10% |
|---|---|---|---|---|---|---|
| Random Init. (L2=0) | Test acc. | 11.93 (1.34) | 15.46 (0.72) | 19.50 (0.56) | 28.40 (0.30) | 33.88 (0.60) |
| Gen. Gap | 82.64 (3.56) | 77.15 (4.67) | 62.34 (8.63) | 38.96 (9.96) | 31.43 (7.32) | |
| NC1 | 6.28 (1.96) | 6.38 (0.60) | 5.94 (0.68) | 3.59 (0.42) | 2.90 (0.18) | |
| Random Init. (L2=0.1) | Test acc. | 10.87 (1.32) | 14.28 (1.32) | 19.23 (1.70) | 35.64 (0.85) | 41.57 (0.37) |
| Gen. Gap | 84.34 (2.20) | 75.61 (6.87) | 48.63 (2.49) | 31.65 (9.09) | 17.50 (3.41) | |
| NC1 | 8.38 (3.33) | 7.62 (1.46) | 6.50 (0.58) | 2.77 (0.02) | 2.29 (0.04) |
Table 7: Test accuracy, generalization gap and normalized within-class variance in classifier subspace (NC1) across architectures for MinC-2500 (Part 1 of 2), by training-set fraction. Each cell is the mean over seeds, with the standard deviation in parentheses, set smaller. Best test accuracy per column in bold.
| Model | Metric | 20% | 30% | 50% | 100% |
|---|---|---|---|---|---|
| Random Init. (L2=0) | Test acc. | 40.81 (0.38) | 44.35 (0.73) | 49.86 (0.99) | 55.36 (1.21) |
| Gen. Gap | 28.29 (6.62) | 23.53 (4.60) | 13.94 (7.40) | 8.13 (2.77) | |
| NC1 | 2.70 (0.06) | 2.56 (0.11) | 2.34 (0.01) | 2.09 (0.08) | |
| Random Init. (L2=0.1) | Test acc. | 40.65 (12.06) | 49.21 (0.86) | 50.37 (0.74) | 50.69 (0.20) |
| Gen. Gap | 1.69 (2.01) | -0.03 (0.94) | -2.37 (0.72) | -4.05 (0.33) | |
| NC1 | 2.47 (0.93) | 1.98 (0.11) | 2.19 (0.08) | 2.54 (0.13) |
Table 8: Test accuracy, generalization gap and normalized within-class variance in classifier subspace (NC1) across architectures for MinC-2500 (Part 2 of 2), by training-set fraction. Each cell is the mean over seeds, with the standard deviation in parentheses, set smaller. Best test accuracy per column in bold.
| Model | Metric | 2% | 5% | 10% | 15% |
|---|---|---|---|---|---|
| Random Init. (L2=0) | Test acc. | 49.00 (0.95) | 77.93 (4.59) | 87.17 (2.37) | 92.90 (1.57) |
| Gen. Gap | 51.00 (0.95) | 20.29 (4.34) | 9.72 (2.51) | 5.32 (1.79) | |
| NC1 | 0.48 (0.06) | 0.33 (0.02) | 0.30 (0.03) | 0.32 (0.02) | |
| Random Init. (L2=0.1) | Test acc. | 39.80 (21.30) | 77.00 (2.46) | 87.43 (1.85) | 92.07 (2.33) |
| Gen. Gap | 49.53 (3.31) | 20.78 (1.23) | 10.12 (2.83) | 6.23 (1.86) | |
| NC1 | 0.43 (0.13) | 0.33 (0.03) | 0.32 (0.01) | 0.34 (0.02) |
Table 9: Test accuracy, generalization gap and normalized within-class variance in classifier subspace (NC1) across architectures for SuroTex (Part 1 of 2), by training-set fraction. Each cell is the mean over seeds, with the standard deviation in parentheses, set smaller. Best test accuracy per column in bold.
| Model | Metric | 20% | 30% | 50% | 100% |
|---|---|---|---|---|---|
| Random Init. (L2=0) | Test acc. | 94.20 (0.62) | 96.00 (1.73) | 97.23 (0.96) | 98.77 (0.83) |
| Gen. Gap | 3.74 (0.92) | 1.65 (0.20) | 0.68 (1.71) | -0.54 (0.31) | |
| NC1 | 0.30 (0.02) | 0.28 (0.03) | 0.28 (0.02) | 0.26 (0.02) | |
| Random Init. (L2=0.1) | Test acc. | 94.37 (1.60) | 95.57 (1.64) | 97.20 (1.28) | 98.10 (0.70) |
| Gen. Gap | 3.30 (2.05) | 0.54 (1.33) | 0.49 (1.73) | 0.12 (0.72) | |
| NC1 | 0.32 (0.02) | 0.28 (0.03) | 0.31 (0.04) | 0.27 (0.01) |
Table 10: Test accuracy, generalization gap and normalized within-class variance in classifier subspace (NC1) across architectures for SuroTex (Part 2 of 2), by training-set fraction. Each cell is the mean over seeds, with the standard deviation in parentheses, set smaller. Best test accuracy per column in bold.