REALM: Regime-Switching, Explainable, and Activation-Induced Linear Models
Authors: Xiaoran Cheng, Sen Na, Jia Li
Organizations: Department of Statistics, The Pennsylvania State University, University Park, PA 16802 USA · H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, GA 30332 USA
Deep ReLU networks are piecewise-affine mappings that partition the input space into cells, each characterized by a distinct activation pattern. This structure motivates fitting a local linear model within each cell to preserve predictive accuracy while improving interpretability. The challenge is to identify regimes that are stable, data-adaptive, and easy to explain. We propose REALM, a mixture of linear models whose regimes are induced by neural activation patterns. Because the number of activation cells in a deep neural network (DNN) can grow rapidly with depth, we first distill a deep teacher into a wide, shallow student network (WSSN), then binarize and cluster its hidden-layer activations to define the regimes and fit a linear model within each regime. Since the regimes are discovered from internal structure, the router does not carry the predictive burden. To make regime assignment interpretable, we train a multiclass logistic regression, the explanatory gate, to reproduce the regime assignments. The two-level structure is interpretable at both stages in terms of raw tabular or learned convolutional features: the gate identifies features that determine regime assignments, while the linear models identify features that drive predictions within each regime. We analyze an idealized setting that illustrates a trade-off between partition complexity and stability: as the number of regimes grows, finer partitions can improve approximation but may reduce regime-assignment stability. Experiments on tabular and image datasets show that REALM achieves competitive predictive performance relative to other DNN-guided mixture surrogates and inherently interpretable models while producing stable regime-level explanations.
Figures & tables
Fig. 1: Overview of the REALM pipeline. Stage 1 distills a deep teacher fT into the WSSN fS . Stage 2 clusters the WSSN’s binarized activation patterns into K regimes, giving the routing posterior πk(x) . Stage 3 fits a mixture of local linear experts under πk(x) to produce the prediction f(x) , and trains the explanatory gate q(x) to explain the assignments.
Fig. 2: Predicted regime assignments for MLM, SEE-Net, and REALM. Ground-truth boundaries in red, predicted boundaries as black outlines.
Fig. 3: Routing sharpness and regime size balance on MNIST (a), CIFAR-10 (b), and Flowers (c). Top row: distribution of the maximum routing posterior maxkπk(x) over training samples. Bottom row: distribution of regime sizes, i.e., the number of training samples assigned to each non-empty regime. Bars are means over five independent runs, and error bars are ±1.96 standard errors across runs.
Fig. 4: Single-run explanatory gate coefficients on MNIST (a), CIFAR-10 (b), and Flowers (c). Rows are the 10 largest regimes by training-set size. Columns show feature units of u(x) , labeled g_ j . For MNIST, a feature unit is one spatial location in a convolutional channel; for CIFAR-10 and Flowers, it is a max-pooled channel. Color represents the gate coefficient of the labeled feature unit for that regime.
Fig. 5: Training images grouped by regime. (a)–(e) MNIST digits 1 and 3 in five selected regimes, revealing variations in writing style among same-digit images. (f)–(j) CIFAR-10 dogs and trucks in five selected regimes, revealing variations in color, shape, and style among same-class images. (k)–(o) Dandelions and sunflowers from Flowers in five selected regimes, revealing variations in visual tone, framing, and bloom stage among same-class images. Within each regime, each class is shown as a 3×3 grid of randomly sampled training images.
Fig. 6: CIFAR-10 local expert contribution maps. The first column shows the original image and its index (“idx”). The next four columns show the filter groups with the largest total absolute weight × activation. Each group sums the expert-weighted pre-pooling feature maps across the three RGB input channels. The final two columns sum contributions over these four groups (Top-4) and all channels, respectively. All maps are computed for the ground-truth class using the expert with the highest routing posterior. Larger positive contributions appear darker in the per-group maps and brighter in the aggregate maps.
Fig. 7: Effect of replacing the local linear experts with majority-vote predictions. Blue bars give the performance degradation (percentage increase in RMSE for regression, or percentage drop in AUC for classification). Orange bars show the increase in the cross-run standard deviation of the test metric, expressed as a percentage of the full model’s test metric. Negative orange bars indicate a decrease in standard deviation. Positive values mean the ablation is worse than the full REALM.
Fig. 8: Effect of replacing the router with the explanatory gate and refitting the local linear experts on all training samples under the gate posterior qk(x) . Bars and sign convention as in Fig. 7 .
Fig. 9: Effect of the number of regimes K on test performance. Left: regression RMSE (Bike Sharing on the left axis, California Housing on the right). Right: classification AUC.
Bike Sharing
Cal. Housing
Covertype
MNIST
Flowers
CIFAR-10
Train / validation / test
64/16/20%
64/16/20%
64/16/20%
48k/12k/10k
2,569/550/551
45k/5k/10k
Batch size
256
256
512
300
32
200
Learning rate (teacher)
10−3
10−3
10−3
10−3
10−3
10−3
Learning rate (Stage 1)
10−3
10−3
3×10−4
10−3
10−3
10−3
Learning rate (Stages 2–3)
10−3
10−3
10−3
10−3
10−3
10−3
Representation dimension p
19
8
54
2,304
384
384
Table III: Dataset-specific hyperparameters. Shared hyperparameters are given in the text. The minimum regime size τ is used only for tabular data augmentation.
Fig. 10: Routing sharpness and regime sizes on Covertype (a), Bike Sharing (b), and California Housing (c). Conventions follow Fig. 3 of the main paper.
Fig. 11: Single-run local expert coefficients (top) and explanatory gate coefficients (bottom) on Covertype (a), Bike Sharing (b), and California Housing (c). Covertype experts show class-0 logit coefficients. Conventions follow Fig. 4 of the main paper.
Fig. 12: Local expert contribution maps on MNIST (a) and Flowers (b). For MNIST, ch j denotes convolutional channel j . Other conventions follow Fig. 6 .
Interpretable Mesomorphic Neural Networks (IMNs) offer a promising framework that combines the predictive power of deep neural networks with the interpretability of linear models. However, the original formulation lacks safeguards to ensure that the learned interpretations are in fact reliable. In particular, the network is free to concentrate all explanatory variance into a single weight of the linear output layer, achieving strong predictive performance while producing interpretations that are largely meaningless. Paradoxically, the L1 penalty proposed to encourage sparse solutions exacerbates this problem by further incentivizing such degenerate configurations. To address this vulnerability, we introduce Local Fidelity Regularization (LFR), a novel penalty term that prevents degenerate weight collapse by aligning the linear output weights with local data variations. This structural constraint guarantees faithful explanations and substantially improves the reliability of model interpretations. Furthermore, empirical evaluations across the OpenML benchmark suite demonstrate that LFR does not compromise accuracy for explainability; rather, it achieved improved AUROC over the unregularized IMN. By yielding results highly competitive with state-of-the-art black-box models, LFR provides the dual benefit of reliable interpretability and superior predictive performance. Source code and usage instructions are available at https://github.com/hugohammer/LFR-IMN.git.
Hugo L. Hammer, Vajira Thambawita, Kristoffer Herland Hellton +1
1Oslo Metropolitan University (OsloMet) · 2Simula Metropolitan Center for Digital Engineering (SimulaMet)
Formal explainability provides mathematically grounded justifications for individual predictions. However, abductive explanations often exceed human cognitive limits by involving too many features, while probabilistic relaxations have remained largely limited to categorical classification. We present a unified framework for probabilistic explainability based on sparse, anchored linear models, applicable to both binary classification and continuous regression. By mapping instances to the Boolean hypercube, our linear explanations strictly generalize subset-based approaches: they capture both the magnitude and direction of feature contributions while enforcing a prescribed sparsity budget k. We show that minimizing the relevance error for such explanations is \ClassNPPP-hard when the underlying model is a neural network, and we relate this intractable objective to a tractable surrogate---the fidelity error. For a parameterized family of local distributions, the relevance error of any k-sparse explanation is bounded by its fidelity error up to a multiplicative factor that remains small locally. We address the resulting empirical problem using two complementary approaches: a Mixed Integer Programming (MIP) formulation that yields provably optimal empirical solutions while maintaining polynomial sample complexity, and a polynomial-time Iterative Hard Thresholding (IHT) algorithm with provable approximation guarantees. Empirical evaluations show that, unlike state-of-the-art baselines such as LIME and MAPLE, our explanations satisfy both the anchoring and sparsity constraints by construction, while consistently achieving lower relevance error.
Frederic Koriche, Jean-Marie Lagniez, Chi Tran
Computer science Research Institute of Lens (CRIL), UMR CNRS 8188, University of Artois, Rue Jean Souvraz SP 18, Lens, F-62307, France
Neural networks (NNs) achieve outstanding performance in many domains; however, their decision processes are often opaque and their inference can be computationally expensive in resource-constrained environments. We recently proposed Differentiable Logic Networks (DLNs) to address these issues for tabular classification based on relaxing discrete logic into a differentiable form, thereby enabling gradient-based learning of networks built from binary logic operations. DLNs offer interpretable reasoning and substantially lower inference cost. We extend the DLN framework to supervised tabular regression. We first redesign the final output layer (the SumLayer) to support continuous targets. More critically, we find the original two-phase training procedure used for classification is suboptimal for regression, and thus develop a unified, single-stage optimization procedure. We also demonstrate that temperature annealing of the network's differentiable relaxations is decisive for achieving stable convergence and high accuracy. We evaluate the resulting model on 15 public regression benchmarks, comparing it with modern neural networks and classical regression baselines. Regression DLNs match or exceed baseline accuracy while preserving interpretability and fast inference. Our results show that DLNs are a viable, cost-effective alternative for regression tasks, especially where model transparency and computational efficiency are important.
Chang Yue, Niraj K. Jha
Department of Electrical and Computer Engineering, Princeton University, Princeton, NJ 08544, USA