Distributional Regression for Arbitrary Response Variables Using ReLU Neural Networks

Junpeng Ren; Carlos Misael Madrid Padilla; Oscar Hernan Madrid Padilla

stat.MLOct 12, 2026

Distributional Regression for Arbitrary Response Variables Using ReLU Neural Networks

Authors: Junpeng Ren, Carlos Misael Madrid Padilla, Oscar Hernan Madrid Padilla

Organizations: Department of Statistics and Data Science, University of California, Los Angeles, Los Angeles, CA, USA · Department of Statistics and Data Science, Washington University in St Louis, St. Louis, MO, USA

Abstract

This paper develops a unified framework for distributional regression with arbitrary response variables. Rather than relying on scalar thresholds or Euclidean structure, we model conditional event probabilities indexed by measurable sets, which extends distributional regression to general response spaces while retaining a direct connection to classical event-based formulations. Within this framework, we study two complementary estimators: a preliminary estimator based on separate regression of event indicators, and a constrained estimator that enforces inclusion-consistent predictions through a directed acyclic graph (DAG) over a finite set family. We establish general error bounds for both estimators and further specialize the results to dense ReLU networks, yielding explicit convergence rates overcoming the curse of dimensionality. Simulation studies and real-data experiments further demonstrate the practical effectiveness of the proposed framework.

Figures & tables

	S1 (Swiss roll manifold)			S2 (RDPG)
Method	$n{=}1000$	$n{=}2000$	$n{=}4000$	$n{=}1000$	$n{=}2000$	$n{=}4000$
Pre-NN	3.07 (0.15)	1.91 (0.10)	1.04 (0.05)	4.64 (0.10)	3.72 (0.07)	3.09 (0.06)
Constrain-NN	0.96 (0.07)	0.72 (0.04)	0.57 (0.04)	3.48 (0.11)	3.05 (0.07)	2.81 (0.06)
Pre-NN-Clip	2.49 (0.16)	1.56 (0.11)	0.88 (0.05)	4.39 (0.10)	3.55 (0.07)	2.99 (0.06)
DRF	1.28 (0.13)	1.22 (0.12)	1.16 (0.11)	7.13 (1.46)	6.94 (1.43)	6.30 (1.21)
Engression	4.93 (2.55)	4.66 (2.44)	4.36 (2.39)	4.20 (1.35)	3.95 (1.34)	3.83 (1.32)

Table 1: Aggregated MSE (

\times 10^{2}

) for S1 and S2 averaged over 30 trials. Best per column in bold.

Method / Category	Overall	Airplane	Bird	Car	Cat	Deer	Dog	Horse	Monkey	Ship	Truck
Pre-NN	3.63	2.68	3.75	3.70	3.70	4.17	3.72	4.50	3.42	2.83	3.79
Constrain-NN	3.37	2.62	3.52	3.45	3.50	3.99	3.56	4.10	3.18	2.35	3.38
Pre-NN-Clip	3.61	2.66	3.73	3.69	3.69	4.15	3.71	4.49	3.41	2.86	3.76
DRF	4.26	3.26	4.68	4.24	4.41	4.87	4.55	5.37	4.34	2.73	4.14
Engression	7.96	6.09	7.91	8.91	7.47	7.97	7.92	9.23	8.12	7.08	8.92

Table 2: Aggregated MSE (

\times 10^{2}

) for S3 across STL-10 classes, averaged over 30 images per class. The “Overall” reports the average across all classes. See full results with standard deviations in Table 6 .

	$W=3$	$W=6$	$W=9$
$h=3$	3.43 / 2.28 / 3.54 / 2.29 / 3.79	3.41 / 2.24 / 3.51 / 2.33 / 3.94	3.35 / 2.23 / 3.46 / 2.36 / 3.92
$h=6$	3.78 / 2.84 / 3.88 / 2.90 / 4.12	3.71 / 2.81 / 3.79 / 2.91 / 4.24	3.65 / 2.79 / 3.73 / 2.92 / 4.18
$h=9$	3.97 / 3.15 / 4.06 / 3.21 / 4.31	3.87 / 3.10 / 3.95 / 3.20 / 4.40	3.79 / 3.09 / 3.87 / 3.17 / 4.30

Table 3: MSE (

\times 10^{2}

) across 76 sets and 12 stations. Each cell reports results for Pre-NN / Constrain-NN / Pre-NN-Clip / DRF / Engression . The best performance is bolded. Full results with standard deviations are reported in Table 10 .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

	Method	Assumptions / Target	Monotonicity	Convergence Rate
Prior Work	Kohler and Langer [2021] DNN non-parametric regression (not distributional)	$Y\in\mathbb{R}$ $f^{}(x)=\mathbb{E}(Y\|X=x)$ $f^{}(\cdot)\in\mathcal{H}(l,\mathcal{P})$	No	$\phi_{n}$ ( MSE )
	Cevid et al. [2022] Forest-weighted distributional regression	$Y\in\mathbb{R}^{u}$ $F_{A}^{}(x)=\mathbb{P}(Y\in A\mid X=x)$ $F_{A}^{}(\cdot)\in\mathcal{H}(1,\{1,d_{0}\})$	Yes	$o_{p}(1)$
	Shen and Meinshausen [2025] Generative DNN-based sampling	$Y\in\mathbb{R}$ for theory; extends to $Y\in\mathbb{R}^{u}$ methodologically $F_{A}^{}(x)=\mathbb{P}(Y\in A\mid X=x)$ $F_{A}^{}(\cdot)\in\mathcal{H}(1,\{1,d_{0}\})$	Yes	$o_{p}(1)$
	Padilla et al. [2025] DNN-based distributional regression for scalar CDF	$Y\in\mathbb{R}$ $F_{t}^{}(x)=\mathbb{P}(Y\leq t\mid X=x)$ $F_{t}^{}(\cdot)\in\mathcal{H}(l,\mathcal{P})$	No	$\dfrac{\log n}{n}+\phi_{n}\log^{4}n$ ( CRPS [ Gneiting and Raftery, 2007 ] , Worst-case MSE)
This Work	Preliminary estimator	$Y\in\mathcal{Y}$ : general, non-Euclidean $\Lambda$ : totally ordered, potentially uncountable $F_{A}^{}(x)=\mathbb{P}(Y\in A\mid X=x)$ , $A\in\Lambda$ $F_{A}^{}(\cdot)\in\mathcal{H}(l_{A},\mathcal{P}_{A})$	No	$\phi_{n}\log^{4}n$ (Uniform MSE over $\Lambda$ )
This Work	Constrained estimator	$Y\in\mathcal{Y}$ : general, non-Euclidean $\Lambda=\{A_{j}\}_{j=1}^{m}\subset\mathcal{Y}$ with inclusion DAG $F_{A_{j}}^{}(x)=\mathbb{P}(Y\in A_{j}\mid X=x)$ $F_{A}^{}(\cdot)\in\mathcal{H}(l_{A},\mathcal{P}_{A})$	Yes	$\phi_{n}\log^{4}n$ (Average MSE over $\Lambda$ with fixed sets)

Appendix

Table 4: Comparison of recent theoretical results in distributional regression. Here,

o_{p}(1)

denotes convergence to zero in probability.

Figure 1: Example noiseless images from the STL-10 dataset across 10 categories.

Panel of Figure 2 — Figure 2: DAG structure for S5. An arrow from $A_{i}$ to $A_{j}$ indicates $A_{i}\subset A_{j}$ .

Panel of Figure 3 — Figure 3: DAG structure for S1. An arrow from $A_{i}$ to $A_{j}$ indicates $A_{i}\subset A_{j}$ .

Scenario	$n_{\text{train}}$	Constrain-NN	Pre-NN	Pre-NN-Clip	DRF	Engression
S1 (Swiss roll)	1000	0.0096 (0.0007)	0.0307 (0.0015)	0.0249 (0.0016)	0.0128 (0.0013)	0.0493 (0.0255)
	2000	0.0072 (0.0004)	0.0191 (0.0010)	0.0156 (0.0011)	0.0122 (0.0012)	0.0466 (0.0244)
	4000	0.0057 (0.0004)	0.0104 (0.0005)	0.0088 (0.0005)	0.0116 (0.0011)	0.0436 (0.0239)
S2 (RDPG)	1000	0.0348 (0.0011)	0.0464 (0.0010)	0.0439 (0.0010)	0.0713 (0.0146)	0.0420 (0.0135)
	2000	0.0305 (0.0007)	0.0372 (0.0007)	0.0355 (0.0007)	0.0694 (0.0143)	0.0395 (0.0134)
	4000	0.0281 (0.0006)	0.0309 (0.0006)	0.0299 (0.0006)	0.0630 (0.0121)	0.0383 (0.0132)

Appendix

Table 5: Mean Squared Error across all simulation scenarios. Best result in each row is bolded.

Class	Pre-NN	Constrain-NN	Pre-NN-Clip	DRF	Engression
Airplane	2.68 (0.95)	2.62 (0.93)	2.66 (0.95)	3.26 (1.07)	6.09 (2.21)
Bird	3.75 (1.34)	3.52 (1.12)	3.73 (1.32)	4.68 (1.53)	7.91 (2.29)
Car	3.70 (1.12)	3.45 (1.05)	3.69 (1.11)	4.24 (1.22)	8.91 (2.36)
Cat	3.70 (1.60)	3.50 (1.62)	3.69 (1.60)	4.41 (1.75)	7.47 (2.27)
Deer	4.17 (1.18)	3.99 (1.14)	4.15 (1.18)	4.87 (1.21)	7.97 (1.84)
Dog	3.72 (1.43)	3.56 (1.32)	3.71 (1.43)	4.55 (1.58)	7.92 (2.56)

Appendix

Table 6: Test MSE (

\times 10^{2}

) for S3 by object class. Standard deviations are shown in parentheses. Results are averaged over 30 images per class.

	Excellent	Good	Light	Moderate	Heavy	Severe	Hazardous	Beyond
PM2.5 upper	35	75	115	150	250	350	500	$\infty$
PM2.5 %	37.34	23.58	15.48	8.39	10.74	3.21	1.04	0.23
PM10 upper	50	150	250	350	420	500	600	$\infty$
PM10 %	33.87	42.66	16.41	4.88	1.15	0.54	0.29	0.19
SO 2 upper	50	150	475	800	1600	2100	2620	$\infty$
SO 2 %	92.81	6.99	0.20	0.00	0.00	0.00	0.00	0.00

Appendix

Table 7: AQI grade breakpoints we used and empirical prevalence (%) for each pollutant. All concentration units in

\mu

g/m 3 . For each pollutant, the first row (“upper”) gives the upper concentration boundary of each grade, and the second row (“%”) gives the empirical prevalence of that grade in the dataset. The eight columns correspond to the eight pollution grades: Excellent, Good, Light pollution, Moderate pollution, Heavy pollution, Severe pollution, Hazardous, and Beyond Index.

Pollutant	Thresholds ( $\mu$ g/m 3 )
PM2.5	$0,\ 75,\ 115,\ 150,\ 250$
PM10	$0,\ 150,\ 250,\ 350$
SO 2	$0,\ 150$
NO 2	$0,\ 80$
CO	$0,\ 4{,}000$
O 3	$0,\ 200$

Appendix

Table 8: Exceedance thresholds after grade merging. Each value defines an exceedance event

\{\text{pollutant}\geq\text{value}\}

; the value

0

indicates the dimension is unconstrained.

Rank	Exceedance condition	Prevalence (%)
Highest prevalence
1	$\text{PM}_{2.5}\geq 75$	39.43
2	$\text{PM}_{2.5}\geq 115$	23.80
3	$\text{PM}_{10}\geq 150$	23.79
4	$\text{PM}_{2.5}\geq 75\;\wedge\;\text{PM}_{10}\geq 150$	21.71
5	$\text{NO}_{2}\geq 80$	19.53

Appendix

Table 9: Exceedance sets with the ten highest and ten lowest empirical prevalences (after pruning). The full collection contains

76

sets.

Horizon $h$	Method	$W=3$	$W=6$	$W=9$
$h=3$	Pre-NN	3.43 (0.68)	3.41 (0.67)	3.35 (0.66)
	Constrain-NN	2.28 (0.48)	2.24 (0.46)	2.23 (0.46)
	Pre-NN-Clip	3.54 (0.71)	3.51 (0.70)	3.46 (0.69)
	DRF	2.29 (0.48)	2.33 (0.50)	2.36 (0.50)
	Engression	3.79 (0.79)	3.94 (0.87)	3.92 (0.87)
$h=6$	Pre-NN	3.78 (0.74)	3.71 (0.74)	3.65 (0.74)

Appendix

Table 10: Test MSE (

\times 10^{2}

) across 76 sets and aggregated over the 12 monitoring stations. Each entry reports

\text{mean},(\text{std})

over stations. The best performance is bolded.

		Input Window $W$ (hours)
Station	Horizon $h$	$W=3$	$W=6$	$W=9$
Aotizhongxin	$h=3$	3.86 / 2.72 / 3.99 / 2.72 / 4.36	3.90 / 2.73 / 4.02 / 2.83 / 4.69	3.86 / 2.71 / 3.99 / 2.88 / 4.70
	$h=6$	4.29 / 3.30 / 4.38 / 3.44 / 4.78	4.27 / 3.37 / 4.37 / 3.48 / 5.08	4.25 / 3.39 / 4.36 / 3.52 / 5.06
	$h=9$	4.46 / 3.62 / 4.54 / 3.75 / 5.01	4.44 / 3.65 / 4.52 / 3.79 / 5.27	4.38 / 3.75 / 4.47 / 3.78 / 5.16
Changping	$h=3$	3.06 / 2.03 / 3.16 / 2.04 / 3.25	3.00 / 1.98 / 3.10 / 2.04 / 3.22	2.91 / 1.92 / 2.99 / 2.03 / 3.15
	$h=6$	3.38 / 2.48 / 3.48 / 2.50 / 3.51	3.23 / 2.41 / 3.32 / 2.48 / 3.46	3.12 / 2.37 / 3.19 / 2.43 / 3.37

Appendix

Table 11: Test MSE (

\times 10^{2}

) across 76 sets, reported per station and per (horizon

h

, input window

W

) configuration. Each cell reports results for Pre-NN / Constrain-NN / Pre-NN-Clip / DRF / Engression . The best performance in each cell is bolded.

	Pre-NN	Constrain-NN	Pre-NN-Clip	DRF	Engression
MSE	$0.1194\ (0.0046)$	$\mathbf{0.1186\ (0.0044)}$	$0.1248\ (0.0046)$	$0.1200\ (0.0032)$	$0.1408\ (0.0039)$

Appendix

Table 12: Test MSE across

m=82

sets, averaged over

30

random splits. Standard deviations in parentheses.

Distributional Regression

See all 13 papers

Generalized Deep Regression for Repeated Measurements
Sep 22, 2026Kexuan LiNeural Network Generalization ReLU Neural Networks
Bristol Myers Squibb
Scalable and Adaptive Distribution Regression via Random Features
Oct 17, 2026Yinsong Wang, Xiao Liu, Yu DingDistributional Regression Random Feature Methods
School of Mechanical, Industrial and Material Engineering, University of Nevada Reno, Reno 89557, USA. · H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta 30332, USA.
Breaking the Curse with BAND: Nonparametric Distribution Estimation in High Dimensions
Jul 29, 2026Shuo-Chieh Huang, Chien-Ming Chi, Jau-er ChenBayesian Networks Density Estimation
Department of Statistics, Rutgers University, Piscataway, NJ, U.S.A. · Institute of Statistical Science, Academia Sinica, Taipei, Taiwan · School of Political Science and Economics, National Taiwan University, Taipei, Taiwan

Sep 22, 2026J/K move · Enter open · S save

Generalized Deep Regression for Repeated Measurements

Kexuan Li

Bristol Myers Squibb

Neural Network Generalization ReLU Neural Networks Nonparametric Regression

Open paper AskPDF arXiv

In this paper, we study the estimation of a marginal regression function from independent units with repeated binary, count, or continuous responses using ReLU deep neural networks. In the model, we assume that the dependence is generated by an unobserved random mean function within each unit. We then fit a neural network with a convex generalized regression loss. We show an oracle inequality by separating conditional measurement variation from between-unit variation. In addition, we prove that with

n

units and

m

measurements per unit, ReLU networks can attain an integrated mean squared error of order

n^{-1}+(nm)^{-2β/(2β+d)}

, up to logarithmic factors, over

β

-Hölder classes. We also derive a weighted oracle inequality for unequal cluster sizes and a rate for compositionally smooth functions. For pointwise ensemble inference, we give a projection central limit theorem and prove infinitesimal jackknife consistency under an explicit asymptotic linearity condition. Simulations and real data examples are provided to support our theoretical findings and practical implications.

Abstract

Distributional Regression

Explore similar work

Generalized Deep Regression for Repeated Measurements

Scalable and Adaptive Distribution Regression via Random Features

Breaking the Curse with BAND: Nonparametric Distribution Estimation in High Dimensions