Input convex neural networks provide a natural way to enforce convexity globally through architectural constraints. We study how this structure-preserving approximation framework extends to semiconvex, quasiconvex, and convex transformable functions. For C1,1 convex functions on compact convex domains, we establish the sharp W1,p minimax rate m−1/d under max-affine approximation with m affine pieces, and show that every max-affine function admits an exact bounded-width ICNN representation with complexity linear in m. The same W1,p approximation rate extends to semiconvex functions through quadratic lifting and to convex transformable functions through monotone composition. For general Lipschitz quasiconvex functions, we develop a quantitative finite-probe approximation theory based on monotone responses to affine projections and obtain a structure-preserving IqCNN with an L∞ approximation guarantee. We further show that these approximation results extend to ReLU-transferable smooth activations, including Softplus and ELU. Numerical experiments on PDE-based approximation problems compare the proposed architectures with parameter-matched MLPs and validation-tuned shape penalties, demonstrating global structural validity without penalty tuning while maintaining competitive approximation accuracy. Overall, the results identify recoverable convex latent structure as the key mechanism enabling gradient-level structure-preserving approximation beyond convexity.
Figures & tables
Function class
Latent structure
Structured model
Norm
Complexity
Convex
Convex
Max-affine / ICNN
W1,p
O(ε−d)
Semiconvex
Quadratic lift to convex
IsCNN
W1,p
O(ε−d)
Convex transformable
Monotone transform of convex
IctNN
W1,p
O(ε−d)
General quasiconvex
Family of affine probes
IqCNN
L∞
O(ε−(d+1))
Table 1: Structure-preserving approximation regimes considered in this paper. Each neural-network result holds for ReLU and for ReLU-transferable activations, including Softplus and ELU, with the same approximation order. Here ε denotes the target approximation error.
d
model
L2
grad. L2
W1,2
λmin
2
ICNN
.176 (.004)
.319 (.010)
.364 (.010)
+.306 (.011)
MLP
.400 (.008)
.921 (.014)
1.004 (.016)
−.696 (.045)
MLP+convex-reg
.221 (.051)
.745 (.154)
.777 (.162)
−.121 (.004)
3
ICNN
.195 (.014)
.334 (.021)
.386 (.025)
+.297 (.023)
MLP
.676 (.161)
1.672 (.296)
1.804 (.335)
−1.016 (.130)
MLP+convex-reg
.428 (.148)
1.226 (.164)
1.300 (.204)
−.251 (.048)
Table 2: Monge–Ampère test errors and sampled minimum Hessian eigenvalue, reported as mean (standard deviation) over three runs. Training uses only PDE and boundary residuals.
Figure 1: Monge–Ampère accuracy (left) and the 1% quantile of the samplewise minimum Hessian eigenvalue (right). ICNN remains on the elliptic, convex branch by construction; a Hessian-regularized MLP reduces but does not eliminate violations.
d
model
L2
grad. L2
W1,2
violation
2
IsCNN
.0072 (.0009)
.0298 (.0042)
.0307 (.0042)
0
MLP
.0025 (.0008)
.0123 (.0033)
.0126 (.0034)
8.01 (1.74)%
MLP+Hessian-reg
.0080 (.0014)
.0321 (.0048)
.0330 (.0050)
.33 (.30)%
3
IsCNN
.0117 (.0012)
.0476 (.0041)
.0490 (.0042)
0
MLP
.0089 (.0030)
.0352 (.0100)
.0363 (.0104)
20.18 (10.16)%
MLP+Hessian-reg
.0175 (.0080)
.0592 (.0213)
.0618 (.0226)
.46 (.63)%
Table 3: Hamilton–Jacobi snapshot errors and fraction of test Hessians that violate D2F+ρtI⪰0 , as mean (standard deviation).
Figure 2: HJ Sobolev error (left) and the fraction of sampled lifted Hessians that violate semiconvexity (right). IsCNN lies exactly at zero violation; the regularized MLP approaches zero only on the sampled diagnostic.
d
model
L2
grad. L2
W1,2
random fail.
Δqc
2
IqCNN
.0196 (.0099)
.0974 (.0284)
.0994 (.0299)
0
0
MLP
.0244 (.0012)
.1168 (.0052)
.1193 (.0053)
.025 (.030)%
.0376 (.0374)
MLP+levelset-reg
.0199 (.0022)
.1011 (.0069)
.1030 (.0071)
0
0
3
IqCNN
.0187 (.0092)
.0914 (.0277)
.0933 (.0290)
0
0
MLP
.0205 (.0020)
.1017 (.0076)
.1038 (.0078)
.0028 (.0048)%
.0609 (.0160)
MLP+levelset-reg
.0195 (.0009)
.0939 (.0024)
.0959 (.0025)
0
0
Table 4: Brascamp–Lieb profile evaluation on the larger cube [−1.5,1.5]d . Entries are mean (standard deviation); random failure rates and Δqc measure quasiconvex/sublevel-set violation.
Figure 3: Accuracy and adversarial sublevel-set violation on the Brascamp–Lieb profiles outside the training cube. The right panel uses an ordinary linear axis, so exact zeros for IqCNN are displayed at zero.
d
model
params.
WR=11,2
WR=21,2
negative D2eF
8
IctNN
1069
.0458 (.0017)
.165 (.017)
latent cert.
MLP
1041
.312 (.131)
.320 (.148)
68.10 (14.42)%
MLP+CT-reg
1041
.177 (.114)
.193 (.071)
0
16
IctNN
1717
.0672 (.0017)
.341 (.002)
latent cert.
MLP
1751
.301 (.040)
.375 (.055)
84.11 (7.05)%
MLP+CT-reg
1751
.237 (.047)
.266 (.065)
.52 (.90)%
Table 5: Convex-transformable target: parameter-matched W1,2 errors on the training cube and the larger cube, with the fraction of negative eigenvalues of D2exp(F) at R=2 . Entries are mean (standard deviation) over three independent trainings. “latent cert.” denotes IctNN’s global learned certificate; exp need not be its learned transform.
Figure 4: Convex-transformable target in dimensions 16 and 32 (mean and one standard deviation over three independent trainings). The dashed line is the training boundary. IctNN is most accurate in-domain and retains a clear advantage through radius 2 in d=32 .
Figure 5: Validation selection of the supervised shape penalties in d=8 . Each quantity is divided by its value for the unregularized MLP ( λ=0 ), so lower is better and one means no change. Curves and error bars are the mean and one standard deviation from three independently trained validation runs. Orange circles show W1,2 error, green squares show the remaining structural violation, and blue rings mark the frozen choices: λ=100 in B–C and λ=1000 in D.
Classical ReLU-based Input Convex Neural Networks (ICNNs) are equivalent to the optimal value functions of Linear Programming (LP). This intrinsic structural equivalence restricts their representational capacity to piecewise-linear polyhedral functions. To overcome this representational bottleneck, we propose the SOC-ICNN, an architecture that generalizes the underlying optimization class from LP to Second-Order Cone Programming (SOCP). By explicitly injecting positive semi-definite curvature and Euclidean norm-based conic primitives, our formulation introduces native smooth curvature into the representation while preserving a rigorous optimization-theoretic interpretation. We formally prove that SOC-ICNNs strictly expand the representational space of ReLU-ICNNs without increasing the asymptotic order of forward-pass complexity. Extensive experiments demonstrate that SOC-ICNN substantially improves function approximation, while delivering competitive downstream decision quality. The code is available at https://anonymous.4open.science/r/SOC-ICNN-4B18/.
Kang Liu, Jianchen Hu, Wei Peng
School of Future Technology Xi’an Jiaotong University Xi’an, China · School of Future Technology School of Automation Science and Engineering Xi’an Jiaotong University Xi’an, China · School of Automation Science and Engineering Xi’an Jiaotong University Xi’an, China
We introduce Hyper Input Convex Neural Networks (HyCNNs), a novel neural network architecture designed for learning convex functions. HyCNNs combine the principles of Maxout networks with input convex neural networks (ICNNs) to create a neural network that is always convex in the input, theoretically capable of leveraging depth, and performs reliably when trained at scale compared to ICNNs. Concretely, we prove that HyCNNs require exponentially fewer parameters than ICNNs to approximate quadratic functions up to a given precision. Throughout a series of synthetic experiments, we demonstrate that HyCNNs outperform existing ICNNs and MLPs in terms of predictive performance for convex regression and interpolation tasks. We further apply HyCNNs to learn high-dimensional optimal transport maps for synthetic examples and for single-cell data, where they oftentimes outperform ICNN-based neural optimal transport methods and other baselines across a wide range of setting.
Shayan Hundrieser, Insung Kong, Johannes Schmidt-Hieber
Department of Applied Mathematics, University of Twente, Enschede, The Netherlands
We study two models of ReLU neural networks: monotone networks (ReLU+) and input convex neural networks (ICNN). Our focus is on expressivity, mostly in terms of depth, and our motivation is to gain a better understanding of the depth required to exactly represent functions using neural networks with ReLU activations, a subject that has received a lot of attention lately. We prove several lower bounds on the depth required to represent functions by monotone ReLU networks and ICNN. For the maximum function MAXn computing the maximum of n real numbers, we show that ReLU+ networks cannot compute MAXn, or even approximate it. We prove a sharp lower bound of n−1 on the ICNN depth complexity of MAXn. We also prove depth separations between ReLU networks and ICNNs; for every k, there is a depth-2ReLU network of size O(k2) that cannot be simulated by a depth-kICNN. The proofs are based on deep connections between neural networks and polyhedral geometry, and also use isoperimetric properties of triangulations.
Egor Bakaev, Florestan Brunck, Christoph Hertrich +2