Two $\operatorname{ReLU}$ Layers is All You Need

Florestan Brunck; Christoph Hertrich; Kilian Ruess; Jack Stade; Amir Yehudayoff

cs.CCOct 20, 2026

Two $\operatorname{ReLU}$ Layers is All You Need

Authors: Florestan Brunck, Christoph Hertrich, Kilian Ruess, Jack Stade, Amir Yehudayoff

Abstract

We prove that every $\operatorname{ReLU}$ network can be exactly simulated by a $\operatorname{ReLU}$ network with two hidden layers. We present two proofs: one that is based on fillings in homological algebra and a direct combinatorial proof.

Figures & tables

Figure 1. The depth hierarchy of

\relu

networks collapses to its first 3 levels.

Figure 2. A paramid given as the pyramid over a two-dimensional parallelepiped

B

with apex

q

Figure 3. This triangle cannot be a boundary because it does not have the augmentation property.

Figure 4. The hexagon has the augmentation property, so it can be filled.

Figure 5. The doubling map

S

scales a parallelogram in

C_{k}

by a factor of

2

, then subdivides the parallelepiped face into

2^{k}

pieces, which are each a translated copy of the original parallelepiped face. In

P_{\bullet}

, this corresponds to scaling a paramid by

2

and then subdividing into

2^{k}

pieces. But subdivision does not change a class in

P_{\bullet}

, so

\rho\circ S

is equal to

2\rho

Figure 6. On left, a partition of the simplex

\Delta

to simplices with

R=\{3\}

. On middle and right, the two pyramids

\operatorname{pyr}_{e_{v}}(Q_{v,R})

for

v=1

and

v=2

Panel of Figure 7 — Figure 7. At the top, the parallelepipeds $B_{v,\pi,M}$ and the simplices $Q_{v,R}$ together with their respective coefficients. At the bottom, the total sum is the same for the parallelepipeds and for the simplices.

Figure 8. Top: the dilation of a parallelogram by

2

can be subdivided into

4

copies of the original parallelogram. Middle: the dilation of a triangle by

2

can be subdivided into two copies of the original triangle and one parallelogram. So the difference of a triangle and a rectangle of the same area is in

\Pi_{\mathbb{Q}^{2}}^{1}

(bottom).

Figure 9. Top: Using inclusion-exclusion, we can write the union

P

of the standard

3

-simplex

\Delta_{3}

– with vertices at

(0,0,0),(1,0,0),(0,1,0)

and

(0,0,1)

– and the paramid

\Pi

– with vertices

(0,0,0),(0,1,0),(-1,1,0),(-1,0,1)

and

(0,0,1)

– as the sum of the simplex and the paramid minus their intersection – the triangular face

F

with vertices

(0,0,0)

(0,1,0)

and

(0,0,1)

. Bottom:

P

is also the Minkowski sum of the triangular face

T

with vertices at

(0,0,0),(-1,1,0)

and

(-1,0,1)

and the segment

E

with vertices at

(0,0,0)

and

(1,0,0)

Panel of Figure 10 — Figure 10. A thermostat walk obtained by merging the ordered positive and negative coordinate lists shown above it. Take the next positive step below level $1$ and the next negative step above level $1$ .

Panel of Figure 11 — Figure 11. Two subdivisions of the tetrahedron $\Delta$ into the simplices $T_{v}=\operatorname{pyr}_{q_{C}}(\Delta_{-v})$ for $v\in C$ .

ReLU Neural Networks

Rectified linear unit (ReLU) neural networks use piecewise-linear activations to study approximation, learning dynamics, expressivity, and verification.

See all 113 papers

A Hand-Checkable Proof That Two Hidden ReLU Layers Compute the Maximum of Six Numbers
Oct 3, 2026Dimitrios MyrisiotisReLU Neural Networks Neural Network Expressivity
Great Bay University China
Shallower ReLU Network Representations via Exact Linear Algebra
Jul 22, 2026Kilian Rueß, Gennadiy Averkov, Florestan Brunck +7Shallow Neural Networks ReLU Neural Networks
University of Technology Nuremberg · Brandenburg University of Technology Cottbus–Senftenberg · University of Copenhagen +3
Two hidden layer suffice: the caterpillar net
Oct 19, 2026Jakob Paul ZimmermannReLU Neural Networks Shallow Neural Networks
FU Berlin · TU Berlin

Oct 3, 2026J/K move · Enter open · S save

A Hand-Checkable Proof That Two Hidden ReLU Layers Compute the Maximum of Six Numbers

Dimitrios Myrisiotis

Great Bay University China

ReLU Neural Networks Neural Network Expressivity

Open paper AskPDF arXiv

Exactly computing the maximum function is a standard test case for studying depth in ReLU networks. Two hidden layers are known to suffice for up to twelve inputs through computer-assisted constructions. For six real inputs, we give an explicit hexagon identity whose local structure yields a self-contained analytical proof of this depth bound. The identity was found by computer-assisted search; we prove it through explicit cancellations that can be checked entirely by hand, without executing a verification program. The identity also yields an explicit network with hidden widths

17

and

41

, zero biases, and rational weights.

Abstract

ReLU Neural Networks

Explore similar work

A Hand-Checkable Proof That Two Hidden ReLU Layers Compute the Maximum of Six Numbers

Shallower ReLU Network Representations via Exact Linear Algebra

Two hidden layer suffice: the caterpillar net