LiD-GLM: Lipschitz-constrained Deep Generalized Linear Models

Tom Splittgerber; Niklas Koenen; Marvin N. Wright; Werner Brannath

stat.MLAug 17, 2026

LiD-GLM: Lipschitz-constrained Deep Generalized Linear Models

Authors: Tom Splittgerber, Niklas Koenen, Marvin N. Wright, Werner Brannath

Organizations: Competence Center for Clinical Trials Bremen, University of Bremen · Leibniz Institute for Prevention Research and Epidemiology – BIPS & University of Bremen

Abstract

The combination of traditional statistical models and neural network (NN) components into semi-structured hybrid models is an intriguing approach to construct models that, ideally, combine traditional interpretability with the unprecedented flexibility of NNs. In order to preserve interpretability, it is usually necessary to restrict the NN components to prevent them from dominating the model. However, existing methods that enforce structural constraints on their NN components severely limit their models' flexibility; in contrast, methods that only enforce weak, indirect constraints lose meaningful interpretability. The method we propose therefore leverages invertible residual neural networks (i-ResNets) to equip generalized linear models with both nonlinear parameter estimation and a flexible correction of their distributional assumptions while always retaining stochastic monotonicity of the modeled distribution in the (formerly linear) predictor. The i-ResNets correspond to a controlled deviation from identity and by constraining their Lipschitz constant one can rigorously limit and quantify how far the hybrid model deviates from its traditional counterpart. This enables a user-specifiable compromise between flexibility and interpretability without limiting the structure of nonlinear and interaction effects that can be learned. Furthermore, we develop specific inherent interpretation techniques for our model and enforce model identifiability through an adapted post-hoc orthogonalization.

Figures & tables

Figure 2: Heatmaps and contour lines ( $\mathbb{P}=\frac{1}{4},\frac{1}{2},\frac{3}{4}$ ) showing learned densities in synthetic binary data example. Shown are 1) the true distribution $\mathbb{P}[Y=1|x_{1},x_{2}]=\text{sigmoid}(x_{1}+\sin(x_{1})-x_{2})$ , 2) logistic regression, 3+4) LiD-GLM with different bounds for $L(\nu_{p})$ and 5) an unconstrained NN.

$k$	$L_{p}^{b}$	$\|\|.\|\|_{q}$	lr ( $10^{x}$ )	wid. $T_{p}$	dep. $T_{p}$	blk. $T_{p}$	act. fkt. $T_{p}$	$\beta$ frozen	test NLL
$\mathcal{H}$	0.01- 1.0	1, 2	-5, -3	9,18, 36	1, 2, 3	1,2	GroupSort, ReLU	yes, no
0	0.65	2	$-3$	9	3	2	GroupSort	yes	0.310
1	0.7	2	$-3$	36	3	1	GroupSort	no	0.327
2	0.8	1	$-3$	9	1	1	GroupSort	no	0.520
3	0.7	2	$-3$	9	2	2	GroupSort	yes	0.406
4	0.9	1	$-3$	9	3	1	GroupSort	no	0.551

Table 1: Optimal hyperparameters for the “Cars” data found for LiD-GLM without

T_{d}

on each split

k

. Hyperparamters were chosen from the range (for

L^{b}_{p}

) and the values given in row

\mathcal{H}

. Listed are the bound

L_{p}^{b}

L(\nu_{p})

, the used

p

-norm (

||.||_{q}

), the learning rate (lr), the width (wid.) and number (dep.) of hidden layers in

T_{p}

, number of residual blocks in

T_{p}

(blk.), whether or not the linear coefficients

\beta

were frozen during training, activation function used (act. fkt.) and test negative loglikelihood (test NLL).

Figure 6: Results for the nested cross validation on the “Cars” data..

Figure 7: PDPs for a LiD-GLM trained on the cars data set. For numeric covariates, a scatterplot shows the training data.

	bias	cylinders	horsepower	acceleration	model year	origin 1	origin 2
LM	22.51	-2.46	-3.29	-1.41	2.47	2.13	4.24
LiD-GLM	22.39	-3.0	-2.49	-1.31	2.57	2.77	4.82
$R^{2}$	-	0.99	0.97	0.99	0.99	0.96	0.94

Table 2: Coefficient table for LiD-GLM trained on the cars data. Shown are the coefficients of the Linear Model and the LiD-GLM (including PHO) and the coefficients of determination

R^{2}

Figure 9: Results for the nested cross validation on the “Insurance” data..

Figure 10: Distribution correction learned by LiD-GLM on the insurance dataset.

Figure 11: Results for the nested cross validation on the “Stroke” data.

$k$	$L_{p}^{b}$	$\|\|.\|\|_{q}$	lr ( $10^{x}$ )	wid. $T_{p}$	dep. $T_{p}$	blk. $T_{p}$	act. fkt. $T_{p}$	$\beta$ frozen	test NLL
$\mathcal{H}$	0.1- 15.0	1,2	-5, -4, -3	18, 36, 48	3, 4, 5, 6	3, 4, 5	ReLU GroupSort	yes, no
0	15.00	2	-3	36	6	5	GroupSort	no	0.162
1	15.00	2	-3	36	6	4	GroupSort	no	0.127
2	13.94	2	-3	48	5	5	GroupSort	no	0.159
3	12.87	2	-3	48	6	4	GroupSort	no	0.146
4	13.94	2	-3	36	5	5	GroupSort	no	0.129

Table 3: Optimal hyperparameters for LiD-GLM found for each split

k

in the ”Stroke” data example. For a further table description, see Table 1 .

	$\beta_{0}$	$\beta_{1}$	$\beta_{2}$	$\beta_{3}$	$\beta_{4}$	$\beta_{5}$	$\beta_{6}$	$\beta_{7}$	$\beta_{8}$	$\beta_{9}$
LM	-1.0	-0.41	0.09	1.23	1.13	0.87	0.13	0.42	-0.16	0.13
LiD-GLM	-2.06	-0.54	0.16	1.8	1.64	1.6	0.34	0.61	-0.46	0.09
$R^{2}_{i}$	-	0.96	0.99	0.64	0.54	0.79	0.98	0.98	1.0	0.85

Table 4: Coefficient table for LiD-GLM trained on the Stroke data. Shown are the coefficients of the Linear Model and the LiD-GLM (including PHO) and the coefficients of determination

R^{2}

for the variables

\beta_{0}

=Intercept,

\beta_{1}

=Sex,

\beta_{2}

=Age,

\beta_{3}

=Hypertension,

\beta_{4}

=Heart disease,

\beta_{5}

=ever married,

\beta_{6}

=Residence type,

\beta_{7}

= Average glucose level,

\beta_{8}

=BMI,

\beta_{9}

=smoking status.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

learning rate	width	depth	width 2	depth 2	number batches	dropout rate
$10^{-4}$ , $10^{-3}$ $10^{-2}$ , $10^{-1}$	9, 18, 36	2, 3, 4	9, 18, 36	2, 3, 4	1, 10	0, 0.1, 0.2

Appendix

Table 5: Tested hyperparameters for the SDDR model on the “Cars”dataset. Used was a Normal distribution family and a maximum of 3000 epochs.

learning rate	width	depth	width 2	depth 2	number batches	dropout rate
$10^{-4}$ , $10^{-3}$ $10^{-2}$ , $10^{-1}$	9, 18, 36	2, 3, 4	9, 18, 36	2, 3, 4	1, 10	0, 0.1, 0.2

Appendix

Table 6: Tested hyperparameters for the SDDR model on the “Insurance”dataset. Used was a Normal distribution family and a maximum of 3000 epochs.

learning rate	width	depth	number of batches
$10^{-5}$ , $10^{-4}$ $10^{-3}$	8, 16, 32, 64, 100	2, 3, 4 5, 6	1, 10

Appendix

Table 7: Tested hyperparameters for the neural network on the “Stroke”dataset. Used was a maximum of 5000 epochs.

learning rate	width	depth	weight decay	dropout rate
$10^{-4}$ , $10^{-3}$ $10^{-2}$	36, 64, 100	3, 4 5, 6	0, 0.01	0, 0.1

Appendix

Table 8: Tested hyperparameters for the SDDR model on the “Stroke”dataset. Used was a Bernoulli family, a maximum of 3000 epochs with a batch size of 3000 with a patience of 200, an output shape of one and zero degrees of freedom.

$k$	$L_{p}^{b}$	wid. $T_{p}$	dep. $T_{p}$	$L_{d}^{b}$	wid. $T_{d}$	dep. $T_{d}$	blk. $T_{d}$	act. $T_{d}$	Training	test NLL
$\mathcal{H}$	0.01- 1.0	9, 18	2, 3	0.4, 0.6 0.99	4, 8	2, 3	1,2	Swish, ReLU	full, “CW”
0	1.00	9	2	0.99	4	3	1	Swish	full	0.254
1	1.00	9	3	0.99	4	3	2	Swish	full	0.269
2	1.00	9	2	0.99	8	3	2	Swish	full	0.599
3	0.88	9	2	0.40	4	3	2	ReLU	“CW”	0.429
4	1.00	9	2	0.60	4	2	1	Swish	full	0.551

Appendix

Table 9: The best hyperparameters found for LiD-GLM with

T_{d}

on each split

k

in the cars data example. Used were

||.||_{2}

, with one residual block and the ReLU activation used for

T_{p}

and a learning rate of

10^{-3}

for 3000 epochs with early stopping (patience 200) using the negative loglikelihood loss training either the entire model at once (”full”), or first the mean prediction and then

T_{d}

(component-wise, ”CW”).

$k$	$\|\|.\|\|_{q}$	$L_{p}^{b}$	lr ( $10^{x}$ )	wid. $T_{p}$	dep. $T_{p}$	blk. $T_{p}$	act. $T_{p}$	$\beta$ frozen	test NLL
$\mathcal{H}$	1, 2	0.01- 1.0	-5, -3	9, 18, 36	1, 2, 3	1, 2	GroupSort, ReLU	yes, no
0	1	0.85	$-3$	18	1	1	ReLU	no	0.446
1	1	0.85	$-3$	9	1	1	GroupSort	no	0.438
2	2	0.95	$-3$	9	3	1	GroupSort	no	0.495
3	2	0.90	$-3$	9	2	1	ReLU	no	0.482
4	2	0.55	$-3$	9	2	2	GroupSort	no	0.533

Appendix

Table 10: Optimal hyperparameters for the “Insurance” data found for LiD-GLM without

T_{d}

on each split

k

. Used were, 3000 epochs with a patience of 200, 1 batch and the negative loglikelihood loss.

$k$	$L_{p}^{b}$	wid. $T_{p}$	dep. $T_{p}$	$L_{d}^{b}$	wid. $T_{d}$	dep. $T_{d}$	blk. $T_{d}$	act. $T_{d}$	Training	test NLL
$\mathcal{H}$	0.01- 1.0	9, 18	2, 3	0.4, 0.99, 2.99	6, 16	2, 3	1, 2	Swish, ReLU	full, ”CW”
0	0.63	18	2	2.99	6	3	2	Swish	full	-1.184
1	1.0	18	3	2.99	6	3	2	Swish	full	-1.180
2	0.63	9	3	2.99	6	3	2	Swish	full	-0.865
3	0.88	9	3	2.99	16	3	2	Swish	full	-1.224
4	0.63	9	2	2.99	6	3	2	Swish	full	-0.672

Appendix

Table 11: The best hyperparameters found for LiD-GLM with

T_{d}

on each split

k

in the “Insurance” data example. Used were

||.||_{2}

, lr =

10^{-3}

, 3000 epochs with a patience of 200 and 1 batch, 1 residual block in

T_{p}

and the ReLU activation in

T_{p}

. After initialization, either all parameters were given to the optimizer at once (”full”), or first

\beta

and

T_{p}

and afterwards

T_{d}

were trained (component-wise, ”CW”).

Variable	GLM	LiD-GLM	$R^{2}$
bias	-1.23	-1.45	-
age	0.31	0.31	0.97
sex	0.00	0.01	1.00
bmi	0.03	0.03	1.00
children	0.05	0.05	1.00
smoker	1.95	1.82	0.92

Appendix

Table 12: Coefficient table for LiD-GLM trained on the Stroke data. Shown are the coefficients of the Linear Model and the LiD-GLM (including PHO) and the coefficients of determination

R^{2}

for the variables

\beta_{0}

=Intercept,

\beta_{1}

=Sex,

\beta_{2}

=Age,

\beta_{3}

=Hypertension,

\beta_{4}

=Heart disease,

\beta_{5}

=ever married,

\beta_{6}

=Residence type,

\beta_{7}

= Average glucose level,

\beta_{8}

=BMI,

\beta_{9}

=smoking status.

	$\beta_{0}$	$\beta_{1}$	$\beta_{2}$	$\beta_{3}$	$\beta_{4}$	$\beta_{5}$	$\beta_{6}$	$\beta_{7}$	$\beta_{8}$	$\beta_{9}$
LM	-1.0	-0.41	0.09	1.23	1.13	0.87	0.13	0.42	-0.16	0.13
LiD-GLM	-7.83	-1.38	0.25	4.09	3.76	5.33	1.32	1.28	-1.6	-0.19
$R^{2}_{i}$	-	0.96	0.99	0.52	0.09	0.93	0.98	0.99	1.0	0.86

Appendix

Table 13: Coefficient table for LiD-GLM trained on the Stroke data. Shown are the coefficients of the Linear Model and the LiD-GLM (including PHO) and the coefficients of determination

R^{2}

for the variables

\beta_{0}

=Intercept,

\beta_{1}

=Sex,

\beta_{2}

=Age,

\beta_{3}

=Hypertension,

\beta_{4}

=Heart disease,

\beta_{5}

=ever married,

\beta_{6}

=Residence type,

\beta_{7}

= Average glucose level,

\beta_{8}

=BMI,

\beta_{9}

=smoking status.

Figure 18: Fluctuation in LiD-GLM estimation of

\beta

on all splits on the cars dataset with

T_{d}

with and without PHO.

Figure 19: Fluctuation in LiD-GLM estimation of

\beta

on all splits on the cars dataset without

T_{d}

with and without PHO.

Figure 20: Fluctuation in LiD-GLM estimation of

\beta

on all splits on the insurance dataset with

T_{d}

with and without PHO.

Figure 21: Fluctuation in LiD-GLM estimation of

\beta

on all splits on the insurance dataset without

T_{d}

with and without PHO.

Interpretable ML

Interpretable machine learning develops models and explanations that make predictions, evidence, and decision processes understandable to people.

See all 306 papers

A renormalization-group inspired lattice-based framework for piecewise generalized linear models
May 6, 2026Joshua C. ChangInterpretable ML Statistical Learning Theory
NIH Clinical Center, Rehabilitation Medicine · Mederrata Research Inc · Sound Prediction Inc
Improved Predictive Performance and Interpretability for Mesomorphic Neural Networks Using Local Fidelity Regularization
Jun 29, 2026Hugo L. Hammer, Vajira Thambawita, Kristoffer Herland Hellton +1Interpretable ML Neural Network Interpretability
1Oslo Metropolitan University (OsloMet) · 2Simula Metropolitan Center for Digital Engineering (SimulaMet)
Neural Generalized Mixed-Effects Models
Apr 13, 2026Yuli Slavutsky, Sebastian Salazar, David M. BleiMixed-Effects Models Generalized Linear Models
Department of Statistics, Columbia University, New York, USA · Department of Computer Science, Columbia University, New York, USA

May 6, 2026J/K move · Enter open · S save

A renormalization-group inspired lattice-based framework for piecewise generalized linear models

Joshua C. Chang

NIH Clinical Center, Rehabilitation Medicine · Mederrata Research Inc · Sound Prediction Inc

Interpretable ML Statistical Learning Theory Generalized Linear Models

Open paper AskPDF arXiv

We formally introduce a class of models inspired by renormalization group (RG) theory, built on additive hierarchical expansions analogous to those appearing in functional ANOVA and mixed-effects models. Like ReLU convolutional neural networks, they are almost everywhere locally linear; unlike ReLU networks, their partition structure is explicit, interpretable, and easy to modify or constrain. In these models, one defines a multidimensional lattice partition of the input space and uses it to scaffold variations in regression parameters. Each dimension of the lattice corresponds to an attribute by which the statistics of the problem may vary. The parameters are themselves expressed in the form of an expansion, where each term captures variations relative to a lower (coarser) interaction scale. These models admit multiple equivalent interpretations: as piecewise GLMs, as hierarchical mixed-effects regressions, or as regression trees with structured parameter sharing. Since RG motivates the design of these models, we use techniques from statistical physics -- specifically replica analysis -- to study their generalization properties. Specifically, we analyze the behavior of the Watanabe-Akaike Information Criterion (WAIC) as a proxy for generalization loss. This analysis yields two practical results: (i) guidance on the lattice design as a function of dataset size and predictor dimensionality; and (ii) a principled scaling law for the regularization prior when adding higher-order terms to the expansion so that one can increase model complexity without an expected increase in generalization loss. We evaluate the methodology on public datasets and find performance competitive against both blackbox methods and other intrinsically interpretable approaches.

Abstract

Interpretable ML

Explore similar work

A renormalization-group inspired lattice-based framework for piecewise generalized linear models

Improved Predictive Performance and Interpretability for Mesomorphic Neural Networks Using Local Fidelity Regularization

Neural Generalized Mixed-Effects Models