Feedforward Neural Networks

Follow this topic Ask about this topic

Momentum

2 papers in the last four weeks, down 33% on the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 41

Sep 28, 2026cs.LG

Arbitrary-Accuracy Neural Approximation with Optimal Neuron Count and Near-Optimal Bit Complexity

We study the minimum number of hidden neurons required for arbitrary-accuracy approximation of multivariate Hölder-continuous functions on

[0,1]^d

and the associated encoding complexity. For

d\geq 2

, we construct a fixed, explicitly defined activation function for which a closed-form network with two hidden layers of widths

d

and

1

achieves arbitrary accuracy in the uniform norm. We prove that

d+1

is the exact minimum total number of hidden neurons among standard feedforward networks with locally integrable activations and affine outputs. We further give a simpler construction using a single elementary activation that combines the floor and exponential functions. This construction requires three hidden layers of widths

d

,

1

, and

2

, only two neurons above the minimum. If a skip connection is allowed, widths

d

,

1

, and

1

suffice. These constructions use explicit grid addressing and integer encoding of quantized function values. For a bounded

α

-Hölder class, they require

O(\varepsilon^{-d/α}\log(1/\varepsilon))

bits, matching the metric-entropy lower bound up to a logarithmic factor.

Zilan Cheng, Li-Lian Wang, Zhongjian Wang

Division of Mathematical Sciences, School of Physical and Mathematical Sciences, Nanyang Technological University, 637371, Singapore.

Neural Network Approximation Theory Feedforward Neural Networks Neural Network Expressivity

Sep 11, 2026cs.LG

Polyhedral Geometry of Time-to-First-Spike Neural Networks

We study the expressivity of spiking neural networks, which provide a natural framework for asynchronous, event-driven computation complementary to conventional feedforward neural networks. We consider the time-to-first-spike model in a setting for which the input-output map is continuous and piecewise linear, with affine pieces governed by causal feasibility constraints that determine which presynaptic spikes occur before a neuron fires. We first show that each neuron's firing time admits a maxout-like representation with exponentially many, highly constrained affine pieces. We then formalize causal regions as polyhedral regions with fixed causal sets and derive upper and lower bounds on the maximal number of causal regions in both shallow and multilayer feedforward spiking networks. Our theoretical and experimental results show that spiking networks can generate richer partitions of the input space than conventional feedforward ReLU networks.

Manjot Singh, Guido Montúfar, Gitta Kutyniok

Department of Mathematics, Ludwig-Maximilians-Universität München · Munich Center for Machine Learning · Departments of Mathematics and Statistics & Data Science, University of California, Los Angeles +2

Spiking Neural Networks Feedforward Neural Networks Neural Network Expressivity

Sep 1, 2026cs.AI

Wave Function Backpropagation with Explicit Temporal-Interval Dynamics

Conventional neural networks learn predominantly through affine transformations followed by nonlinear activations, while elapsed time is often treated as an auxiliary feature or assumed to be uniformly sampled. This paper introduces Wave Function Backpropagation (WFB), a wave-parameterized learning formulation in which neural responses are represented by learnable amplitude, wavenumber, angular frequency, and phase. The formulation associates an observed state with its temporal interval Delta t through the phase of a differentiable spatiotemporal wave. We derive standard WFB gradients and a spatial-curvature correction based on the Laplacian of the wave response. WFB is instantiated in a deliberately feed-forward trajectory predictor to provide a controlled proof of concept; sequence learning is outside the scope of the present evaluation. With motion features, STD-WFB using real intervals reduces average displacement error (ADE) by 20.4% relative to the original FFN baseline. In a new position-only evaluation that removes temporal leakage through precomputed velocity and acceleration, real-interval WFB reduces ADE by 10.4% relative to the original FFN and remains competitive with parameter-matched ReLU controls, obtaining 2.1% lower mean ADE than the matched FFN with explicit Delta t. Shuffled-interval WFB attains the lowest mean ADE, indicating that the present evidence supports the effectiveness of the wave representation but does not attribute the gain to interval alignment. These results establish WFB as a viable structured feed-forward learning formulation and define a clear basis for subsequent architectural studies.

Byunggu Yu, Justin Kim

Dept. of Computer Science and Information Technology University of the District of Columbia Washington, DC USA

Backpropagation Feedforward Neural Networks

Aug 15, 2026cs.AI

A concentration result for multilayer feedforward neural networks

We consider for an arbitrary fixed

ρ

and for each positive integer

n

a multilayer feedforward artificial neural network with

ρ

layers,

n

neurons in the first layer (the input layer) and only one neuron, the output neuron, in the last layer. Very roughly formulated, the main result is that if the distribution of weights of connections from a layer to the next are, for all large

n

, approximated well by a fixed continuous (but otherwise arbitrary) curve which does not depend on

n

, and if the values of the

n

input neurons are independently and identically distributed with a continuous probability density function, then there is a number

ψ

such that for all

\varepsilon > 0

the probability that the value of the output neuron is in

[ψ- \varepsilon, ψ+ \varepsilon]

tends to 1 as

n

tends to infinity.

Vera Koponen

Multilayer Perceptrons Feedforward Neural Networks

Aug 11, 2026cs.LG

Convergence Guarantees of Gradient Descent for Neural Networks via Generalized Lipschitz Smoothness

We establish convergence guarantees of gradient descent for general feedforward neural networks of arbitrary width or depth, with no special requirements on the initialization or dataset. We only assume that the activation functions are Lipschitz smooth, Lipschitz continuous, and linearly bounded--- properties that hold for linear, tanh, softplus, and sigmoid activation functions. For the loss function, we require that it is Lipschitz smooth in the model outputs, which is true for mean-squared error. The key theoretical insight is that the Lipschitz properties of the activation functions are partially preserved even through repeated compositions, leading to a novel generalized Lipschitz smoothness condition where the change in gradient is upper bounded by the change in the parameter space, multiplied by polynomial terms of the parameter norms at both endpoints. This type of condition holds for both the model function and the loss function, enabling a descent lemma where the loss decreases as long as the learning rate is small enough with respect to the parameter norms. By ensuring that the parameter norms do not grow too quickly to infinity, we prove that the minimum squared gradient norm converges to zero in

T

iterations at rate

O(1/T^{1/L})

for an

L

-layer neural network.

Siqiao Mu, Diego Klabjan

Gradient Descent Neural Network Optimization Feedforward Neural Networks

Aug 7, 2026cs.LG

Machine Learning-Based Inter-Crystal Scatter Recovery for Ultra-High Resolution PET Imaging

Inter-crystal scatter (ICS) events pose a significant challenge in ultrahigh- resolution positron emission tomography (UHR-PET), especially as detector crystals become smaller and their readouts increasingly segmented. Current approaches either reject these events, reducing sensitivity, or accept them with suboptimal positioning algorithms, degrading image resolution. We present a feed forward neural network to optimize ICS event recovery by inferring the line-of-response belonging to the first Compton interaction. Our approach was validated using both Monte Carlo simulations and experimental data from the fully pixelated LabPET-IIbased preclinical and brain UHR-PET scanners.Results demonstrate a 70% to 106% increase in sensitivity while preserving sub-millimeter spatial resolvability (down to 1.6 mm) compared to conventional methods. This ICS recovery approach is an effective solution that compensates for the lower detection efficiency of small, pixelated detectors in UHR-PET, enabling reduced scan times and lower radiation doses while largely preserving image quality.

Alexandre Bernier, Roger Lecomte, Jean-Baptiste Michaud

Department of Electrical & Computer Engineering, Universit´e de Sherbrooke, Sherbrooke, QC, Canada · Interdisciplinary Institute for Technological Innovation, Sherbrooke, QC, Canada · Sherbrooke Molecular Imaging Center of CRCHUS, Sherbrooke, QC, Canada +2

Medical Imaging Image Reconstruction Feedforward Neural Networks

Aug 6, 2026cs.LG

Threshold-Based Early Stopping of Accumulations in Neural Networks with Binary Activation

Binary neural networks are very attractive for constrained deployment, enabling small footprint and low-power inference. For binary activations, the dot products become sign-controlled additions or subtractions, but the number of operations is unchanged. Indeed, every neuron or output channel still accumulates all of its input, even though only the sign will be retained, which is often wasteful. As the accumulation progresses, the running partial sum frequently drifts so far from zero that its final sign becomes highly predictable long before the last term is reached; every contribution evaluated after that point changes the value of the sum but not the final output activation. This paper turns this observation into a post-training early-stopping mechanism. We characterize the behavior of the running accumulations on the training dataset and use this information to predict the final sign as soon as possible. No model parameter is retrained. We count the number of operations under an idealized ordering of weights. On VGG11 applied to the CIFAR-10 dataset, the method removes

86.6\%

of the accumulation terms of the deepest convolution for a

0.37

-point accuracy drop, and

25\%

of the full-network arithmetic when used on the three deepest convolutions simultaneously, for a

1.36

-point drop.

Quentin Luquet de Saint-Germain, Massil Ait Abdeslam, Jean Pierre David

Polytechnique Montréal, Department of Electrical Engineering Montréal, Canada

Efficient Neural Network Inference Binary Neural Networks Feedforward Neural Networks

Aug 4, 2026cs.CV

Non-Destructive Quantification of Urea Adulteration in Bovine Milk Using Transmittance Multispectral Imaging

Adulteration of bovine milk using urea remains a major food quality and health concern, motivating the development of rapid and quantitative screening tools. Conventional approaches, including laboratory-based analytical methods and spectroscopic techniques, have been used for urea detection; however, many remain less suitable for rapid, low-cost routine screening due to requirements such as specialized instrumentation, sample preparation, chemical reagents, or laboratory operation. This study introduces a pragmatic, cost-effective, accurate, and laboratory-validated MSI-based method for quantitative urea estimation under controlled density conditions using a multispectral-imaging-based regression framework. An in-house-built multispectral imaging system operating in twelve discrete spectral bands (365--940~nm) was used to acquire multispectral images of milk samples prepared with controlled urea addition and water for density balancing. Fresh milk was obtained on the day of image acquisition, and the specific gravity of the milk was verified to be 1.032 at 20°C using a hydrometer. Multiple linear regression provided an initial mapping with a high validation

R^2

of 0.9599, while a feed-forward neural network further improved predictive performance with a validation

R^2

of 0.9773. These results demonstrate the feasibility of transmittance multispectral imaging for accurate, non-destructive urea quantification under controlled density-balanced conditions, supporting its potential as a rapid screening approach for milk-quality assessment.

Sharukshan Niranjan, Iresha Ranaweera, Tharindu Chandrarathne +5

Dept. of Electrical and Electronic Eng. University of Peradeniya Peradeniya, Sri Lanka · Dept. of Food Science and Technology University of Peradeniya Peradeniya, Sri Lanka · Dept. of Animal Science University of Peradeniya Peradeniya, Sri Lanka

Feedforward Neural Networks

Jul 28, 2026cs.LG

Algorithmic Separation between Constant-Depth and Logarithmic-Depth Neural Networks

Despite the empirical advantages of deep networks over shallow ones, theoretical depth separations largely concern approximation power, while algorithmic results are mostly limited to comparisons between two- and three-layer networks. In this work, we prove the first algorithmic separation between constant-depth and logarithmic-depth networks. Specifically, we identify a class of Boolean functions with hierarchically structured Fourier spectra that logarithmic-depth networks can learn efficiently using layerwise coordinate descent by reconstructing the spectra hierarchically and adaptively. We also exhibit a subclass for which every constant-depth, polynomial-width network with sufficiently regular activations and controlled spectral norms must incur constant

L^2

approximation error under the uniform distribution over the hypercube.

Yunwei Ren, Zihao Wang, Jason D. Lee

Princeton University · Stanford University · University of California, Berkeley

Boolean Function Learning Neural Network Approximation Theory Feedforward Neural Networks

Jul 23, 2026cs.LG

Searching the Space of Feed-Forward Neural-Network Weight-Update Rules with Fixed Depth Symbolic Regression

We investigate whether symbolic regression can discover explicit neural network weight-update rules that outperform standard hand-designed optimizers on small symbolic regression benchmarks. Candidate update rules are represented as fixed-depth symbolic expressions over operands derived from common optimizers, including gradient, momentum, adaptive-gradient, and moment-estimate quantities. Across 30 benchmark/neural network combinations, the symbolic regression procedure found an update rule outperforming the best hyperparameter-tuned established optimizer in 25 cases, with an aggregate MSE reduction of 44.47% over the improved cases. The discovered rules do not all share a single common symbolic form, but many combine adaptive normalization, momentum-like quantities, nonlinear transformations, and rational expressions. These results suggest that symbolic regression can serve as a lightweight mechanism for discovering compact optimizer variants, while also highlighting the need for larger-scale validation.

Charles Brum, Edward Finkelstein

San Diego, California, USA · UCI, USA

Neural Network Optimization Learning to Optimize Feedforward Neural Networks Symbolic Regression

Jul 17, 2026cs.LG

Kolmogorov--Arnold Networks for Small Language Models

Kolmogorov--Arnold Networks (KANs) replace fixed node activations with learned one-dimensional edge functions, offering an explicit interface for interpretation and a possible alternative to transformer feed-forward networks. We test these claims separately. In a six-layer, 10M-parameter B-spline KAN, we reconstruct all 884,736 feed-forward edges: 87.8% exceed (NLS>0.1) and 0.4% are inactive. Pruning the lowest-activity 20--25% causes negligible loss increase, although structured MLP neuron pruning tolerates comparable sparsity. The audit replicates on BabyLM, but grid-size sweeps show that near-total fPCA compression and high closed-form-fit coverage are properties of the low-capacity grid-2 basis, not universal KAN behavior. For replacement, we evaluate MLP, SwiGLU, grouped Chebyshev, and rational GR-KAN networks on BabyLM. The KAN-family and gated variants improve validation loss over the GELU MLP, but this ordering does not transfer to standardized benchmarks: across ten seeds and 59,875 BLiMP pairs, accuracies span 62.4--63.1%, EWoK remains at chance, and a (+0.7)-point GR-KAN effect on BLiMP reverses on the supplement. Larger tests are also cautionary: parameter-matched MLPEdge underperforms the MLP on Wikitext-103, and 286M-parameter GR-KAN remains below a SwiGLU ClimbMix baseline after stabilization. Thus, small-basis KANs provide a practical, corpus-transferable interface for auditing learned scalar transformations, but the tested replacements show no consistent benchmark, quality, or latency advantage over strong MLP baselines.

Felippe Alves, Renato Vicente

TELUS Digital Research Hub – CIAAM Institute of Mathematics, Statistics and Computer Science University of São Paulo

Small Language Models Neural Network Interpretability Kolmogorov-Arnold Networks Feedforward Neural Networks

Jul 15, 2026cs.LG

Algebraic Representability as the Limiting Regime of Grokking: An Exactly Solvable Model with Holomorphic Activations

Neural networks trained on modular arithmetic exhibit grokking, a delayed transition from memorisation to generalisation known to depend on model capacity: too little and the network memorises slowly or not at all, too much and it generalises almost immediately. What happens at the extreme of this spectrum, when the architecture's expressible function class collapses to a finite-dimensional algebraic variety? We study two-layer networks with a holomorphic monomial activation sigma(z)=z^k, trained on modular tasks encoded via roots of unity. Here the network output, regardless of hidden width, is confined to a (k+1)-dimensional subspace of characters of (Z_p)^2, an O(k/p^2) slice of the full function space. We give a complete algebraic characterisation of this subspace: a task is representable if and only if its discrete Fourier support lies on the diagonal u+v = k (mod p), which for linear-phase targets reduces to the arithmetic criterion m+n=k. This is not merely a constraint on eventual generalisation but on memorisation itself: because the outputs are algebraically confined, a non-representable target cannot be fit even on the training set, and we prove a positive lower bound on the training loss, independent of width. Across 585 runs the algebraic prediction matches the observed outcome with 99.8% accuracy, with no memorisation regime and no grokking; outcomes split cleanly into instant success and outright failure. This binary behaviour is the limiting case of the capacity-grokking relationship: when the expressible class shrinks to a fixed algebraic object, the question of when a network will grok dissolves into whether it can represent the target at all. A bottleneck ablation connects this extreme to standard networks, tracing a continuous path from representational failure, through memorisation without generalisation, to grokking with a shrinking gap as capacity grows.

Chon-Fai Kam, Xavier Cadet, Miloud Bessafi +1

University Paris City & University of Reunion, Paris, France · Dipartimento di Fisica e Chimica Emilio Segrè, Università degli Studi di Palermo, Palermo, Italy · Thayer School of Engineering, Dartmouth College, Hanover, NH 03755, USA +2

Feedforward Neural Networks Neural Network Expressivity Modular Arithmetic Grokking

Jul 13, 2026cs.NE

Backpropagation as a Nilpotent Linear System

Backpropagation is the computational engine of deep learning, yet its mathematical structure is typically treated as a procedural traversal of computational graphs. We present a global operator theory of the \emph{F-adjoint} framework, which reformulates the layerwise backward recursion of an

L

-depth feedforward network into a single linear system

(I-\cB)\Xs=\bG

, where

\bG

is a source vector. We prove that the global backward operator

\cB

is strictly block upper-triangular and nilpotent of index at most

L

. This nilpotency guarantees the exact termination of the Neumann series solution after at most

L

terms, revealing classical backpropagation to be mathematically equivalent to block back-substitution on an upper bidiagonal system. We formalise \emph{F-symmetry} -- the condition in which the backward pass perfectly mirrors the forward pass -- identifying orthogonal weight matrices as canonical examples. Through worked numerical examples, we demonstrate how this operator perspective exposes the single-path collapse of strictly feedforward networks and its breakdown in residual architectures. Finally, we leverage this compositional structure to rigorously derive the mechanics of residual networks (gradient highways) and transfer learning (gradient truncation). This framework elevates backpropagation from an algorithmic recipe to a global nilpotent-operator formulation.

Ahmed Boughammoura

Higher Institute of Informatics and Mathematics of Monastir, University of Monastir, 5000 Monastir, Tunisia.

Backpropagation Neural Network Training Dynamics Feedforward Neural Networks

Jul 10, 2026cs.LG

Statistically Undetectable Backdoors in Deep Neural Networks

We show how an adversarial model trainer can plant backdoors in a large class of deep, feedforward neural networks. These backdoors are statistically undetectable in the white-box setting, meaning that the backdoored and honestly trained models are close in total variation distance, even given the full descriptions of the models (e.g., all of the weights). The backdoor provides access to invariance-based adversarial examples for every input, mapping distant inputs to unusually close outputs. However, without the backdoor, it is provably impossible (under standard cryptographic assumptions) to generate any such adversarial examples in polynomial time. Our theoretical and preliminary empirical findings demonstrate a fundamental power asymmetry between model trainers and model users.

Andrej Bogdanov, Alon Rosen, Neekon Vafa

University of Ottawa. · Bocconi University. · Massachusetts Institute of Technology.

Backdoor Attacks Adversarial Examples Feedforward Neural Networks

Jul 8, 2026cs.LG

On the Principles of Deep Feedforward ReLU Networks

The architecture of deep feedforward neural networks is ubiquitous in deep learning, either as a whole system or as a subnetwork of other architectures, and thus its mechanism is a key ingredient of the black box of neural networks. On the basis of the simplest two-layer ReLU network, this paper systematically studies the mechanism of deep feedforward ReLU networks with multiple hidden layers and successfully explains the training solution obtained by the back-propagation algorithm. The concept of a path, especially in terms of the relationships between paths, plays a central role in uncovering the mystery of the black box. It is shown that a unit of a deep ReLU network can form a piecewise linear manifold to divide the input space, instead of a hyperplane of the two-layer case. How to efficiently use the hidden-layer units to produce both linear functions and partitions of the input space is also a central problem. The principles of a two-layer ReLU network can be generalized to the deeper case to a large extent, such as multiple strict partial orders and continuity restriction. The combination of the basic and simple principles proposed can yield complicated instantiations including the training solutions, and in this sense the black box of deep feedforward ReLU networks is revealed.

Changcun Huang

Shuitu Institute of Applied Mathematics, Chongqing 400700, P.R.C

ReLU Neural Networks Neural Network Interpretability Neural Network Training Dynamics Feedforward Neural Networks Neural Network Expressivity

Jul 6, 2026cs.LG

Compressed Computation under $L^4$ Loss is likely Computation in Superposition

Neural networks are thought to represent concepts as directions in their activation space, and superposition lets them encode more concepts than they have dimensions. It is natural to ask whether they can also compute more functions than they have neurons, i.e., perform computation in superposition. In this regime many functions of sparse inputs are evaluated by a layer with fewer neurons than there are functions to compute. Representation in superposition is by now fairly well understood, but computation in superposition is not, and there are few toy models of it arising through training rather than being hand designed. As a toy model of computation in superposition we study the compressed-computation setup: a single-hidden-layer ReLU network with 50 neurons that must compute the ReLU of each of 100 sparse input features. We show that training it under an

L^4

loss (the mean fourth power of the error), rather than the usual

L^2

, elicits a solution that appears to compute all features in superposition. We then reverse-engineer this solution. We find that the network assigns each feature a sparse binary codeword over neurons and decodes it with a pseudoinverse of the encoder. Given these codewords, a description with only three scalars recovers most of the network's performance, and we validate it by building equivalent networks from hand-designed codes.

Francisco Ferreira da Silva, Stefan Heimersheim

ReLU Neural Networks Feedforward Neural Networks Neural Network Expressivity

Jun 30, 2026cs.LG

Radial Suppression Accelerates Algorithmic Generalization: A Geometric Analysis of Delayed Generalization

Why do neural networks memorize algorithmic training data long before they generalize? We present a geometric case study demonstrating that, on tasks where generalization requires discovering structured low-dimensional circuits, the memorization-generalization delay is driven by radial inflation of hidden representations under cross-entropy optimization. We formalize a radial-angular decomposition of activation-space dynamics and derive three testable propositions: (i) that penalizing radial inflation induces anisotropic, data-dependent weight regularization; (ii) that it suppresses radial gradient energy below the isotropic random baseline, forcing predominantly angular updates; and (iii) that it biases convergence toward flatter minima. To empirically validate these propositions, we study a single-hyperparameter norm penalty that softly constrains activations to a sqrt(d)-radius hypersphere. On modular arithmetic, this penalty accelerates grokking up to 6x across MLPs and Transformers, and halves training steps for a 10M-parameter nanoGPT on 3-digit addition.

Srijan Tiwari, Aditya Chauhan, Manjot Singh

Indian Institute of Technology Roorkee

Neural Network Generalization Neural Representation Geometry Neural Network Memorization Neural Network Training Dynamics Feedforward Neural Networks Grokking

Jun 30, 2026cs.LG

Low-dimensional topology of deep neural networks

We study layered models, including feedforward networks, ResNets, and transformers, by limiting each layer to a width of

d = 3

, i.e.,

\mathbb{R}^3

as representation space. This allows us to track how a neural network changes low-dimensional topological invariants through its layers. Just about any topological structure may be simplified or even trivialized by simply increasing dimension; e.g., any knot is equivalent to an unknot in

\mathbb{R}^4

. By restricting to

\mathbb{R}^3

, we not only isolate the effects of activation and depth from that of width, we work in a space that lends itself to easy visualization. We focus on linking number here, deferring other invariants like link groups, Milnor's

\barμ

-invariants, knot types, ambient cobordisms, to a sequel. We provide full proofs and empirical experiments to justify the following insights: When measured by their power to effect changes in linking numbers, the layer-skipping feature in ResNets is as powerful as the attention mechanism in transformers; both ResNets and transformers are strictly more powerful than feedforward neural networks with monotonic activations, which are in turn more powerful than invertible and flow-based models; but replacing monotonic activation with a nonmonotonic one elevates a feedforward network into the same expressivity class as ResNets and transformers. These results suggest that low-dimensional topology can be a useful tool to guide designs of AI architectures. We also generalize our results from

d = 3

to arbitrary

d > 3

.

Junyu Ren, Lek-Heng Lim

University of Chicago, Chicago, IL, USA.

Transformer Feedforward Neural Networks Neural Network Expressivity

Jun 30, 2026cs.CL

Explicit Fuzzy Logic in the Feed-Forward Layer: Self-Forgetting Quantifiers Discover Legible Grammatical-Licensing Detectors

A transformer's feed-forward (FFN) sublayer materializes the distinctions attention gathers, yet gives no account of what it computes. In a parameter-neutral replacement, each hidden unit is an explicit fuzzy set operation on sigmoid-bounded [0,1] memberships: intersection AB and set-difference A(1-B), the latter a bounded positive negation ("A but not B") that gated/bilinear units lack -- a negation-capable FFN (NC-FFN). On N-bit parity they are the most parameter-efficient reasoning basis at shallow depth; at scale (125M, OpenWebText) NC-FFN ties the GELU baseline's perplexity, every unit carrying explicit logical form. Two limits share one cause: two-operand logic localizes to layer 0 and erodes under training, and the one robust grammatical deficit concentrates in licensing and quantifiers, beyond within-token operators. We resolve both with a small block of sequence quantifiers: a soft existential and a soft proportion, each with a per-unit learned forgetting rate from a sticky init. This recovers the deficit at epoch one (halving the wider epoch-two gap), modestly leads on LAMBADA, and makes the FFN legible: the structure now holds and migrates into depth; the decay un-learns its stickiness (median half-life ~1.5 tokens; zero latch units); and at the semantic layers the units read, without dictionary learning, as grammatical licensing detectors: each fires on a licensor (a comparative, a passive participle, a negative-polarity item) and carries its memory forward to predict the licensed word (than, by, nor). This legibility is localized and free only up to a partition (a fully Boolean FFN diverges in training), but the result is a parameter-neutral, language-model-quality transformer with a readable, interpretable-by-construction grammatical mechanism -- an account not just of what a feed-forward layer represents but how it licenses.

Mark Oskin

School of Computer Science and Engineering University of Washington

Transformer Interpretability Transformer FFNs Transformer Fuzzy Logic Language Modeling Feedforward Neural Networks

Jun 25, 2026cs.LG

Algorithmic Foundations of Deep Learning: Complexity-Theoretic Rates and a Characterization of Universal Approximation

Feedforward neural network (NN) expressivity is typically studied by emulating optimal basis-expansion schemes. While powerful, this perspective is incomplete: it primarily captures complexity through regularity, and therefore does not distinguish intuitively simple and complicated objects with comparable regularity, such as the square-root function and a typical Brownian path. The guiding message is that neural networks should be viewed not only as flexible basis functions, but also as models of computation. If a function is computable by a real-valued circuit over a prescribed elementary gate language, then it can be computed to comparable accuracy by an NN with explicit depth, width, and non-zero-parameter bounds controlled by the depth, width, gate count, and gate structure. Thus, neural-network complexity is not governed by regularity alone, but also by algorithmic complexity. We then show that any definable NN model satisfying a natural parallelization condition, allowing possibly multivariate non-linearities such as attention or layer normalization, is a universal approximator if and only if it contains a non-affine nonlinearity. The scope of our theory is illustrated by deducing universal approximation guarantees for continuous functions, minimax-optimal approximation guarantees for Besov classes, logarithmic-error complexity for holomorphic functions, and by showing that NNs can emulate numerical algorithms such as Newton-Raphson root finding and power iteration without architecture-specific arguments. Its precision is illustrated by shortest-path computation on

k

-vertex graphs: compiling the tropical dynamic-programming circuit yields NNs with O(log(1/ε)) non-zero parameters, exponentially improving in 1/ε over the generic

O(ε^{-c k^2})

Lipschitz-approximation scale, for a constant c>0.

Anastasis Kratsios, Simone Brugiapaglia, Bum Jun Kim +2

McMaster University & Vector Institute, Toronto, ON, Canada · Concordia University, Montréal, QC, Canada · University of Tokyo, Tokyo, Japan +2

Universal Approximation Neural Network Approximation Theory Feedforward Neural Networks Neural Network Expressivity

Jun 12, 2026cs.LG

How Linear Is a Transformer Feed-Forward Block? Per-Block Linear Recoverability Is Learned, Not Architectural

Transformer feed-forward networks (FFNs) are often treated as nonlinear stores of computation, yet how nonlinear a trained FFN block actually is has rarely been measured. We treat each FFN as a position-wise input-to-output map and split it into the exact least-squares linear approximation plus a residual. The held-out variance the closed-form linear map explains defines a block's linear recoverability (R^2_lin), an optimiser-free measure of its linearity. Across all twelve blocks of GPT-2, Pythia-160m, and llama-160m, R^2_lin is highly heterogeneous and non-monotone with depth, ranging from near-linear (>0.99) to strongly nonlinear (<0.3) between adjacent blocks, and is not set by the activation function: same-width GELU models GPT-2 and Pythia-160m have sharply different profiles, so recoverability is a learned property of individual trained blocks, not an architectural one. A low-rank bilinear probe of the residual recovers only a few points of R^2, with gain uncorrelated with residual nonlinearity: the unrecovered computation is not a single position-wise product but higher-order or distributed structure. The measurement also serves as a targeted compression signal: recoverable blocks admit large single-layer replacements (GPT-2's early FFN at 8x fewer parameters for +0.77 perplexity), while low-recoverability blocks flag where this is unsafe. It further exposes a methodological pitfall: trained linear baselines can badly under-converge on ill-conditioned transformer activations, so we report the exact closed-form least-squares ceiling throughout.

Stuart Whipp

Independent Research

Transformer FFNs Transformer Expressivity Transformer Feedforward Neural Networks

Jun 3, 2026cs.LG

HalfNet: Randomized Neural Networks with Learned Subspace Geometry

Many researchers investigated neural networks with some of their weights fixed to values randomly drawn from a given distribution, e.g.,

N(0, I)

. Our proposed HalfNet draws random weights from

N(0, Σ)

, where

Σ

, which defines the geometry of the distribution, has a low-rank factorization that we learn from data. Experiments on MNIST and CIFAR-10 demonstrate that HalfNet can match the performance of fully trained multilayer perceptrons while using substantially fewer parameters. Spectral analysis indicates that much of the predictive power of neural networks lies in the geometry of their weight space rather than in the precise values of individual parameters, and we observe that accuracy scales smoothly with rank. HalfNet is not a neural architecture trick for low-rank structure; it implements a data-dependent random embedding that can also be interpreted through supervised metric learning, or random-feature and kernel perspectives.

Ethem Alpaydin

Independent Researcher Istanbul, Turkey

Neural Representation Geometry Random Feature Methods Feedforward Neural Networks Neural Network Compression

Jun 3, 2026stat.ML

Flatness and Generalization: Learning Multi-Index Models with Homogeneous Neural Networks

A common heuristic used to explain the generalization of first-order gradient methods on non-convex neural networks is that "flat interpolators generalize well" (Hochreiter and Schmidhuber, 1994; Keskar et al., 2017), where flatness can be measured by the trace of the Hessian of the empirical loss. However, Dinh et al. 2017) showed that, using symmetry of the network that can change flatness while keeping the population and empirical losses unchanged, any interpolator can be made sharper or flatter. This result makes the earlier heuristic statement vacuous. In this paper, we show that for learning an unknown multi-index model with

2

-layer non-convex homogeneous neural networks, there is a connection between flatness and generalization, despite the existence of symmetries. This connection pertains to the "flattest" interpolators, i.e., the interpolators that have orderwise minimum flatness among all interpolators. First, we show that there exists a natural class of non-generalizing interpolators whose flatness cannot be made closer to the flattest possible, even using symmetries. Second, we show that for data generated by a sum of single-index models, if the approximation error and label noise are low, any flattest interpolator achieves small population loss, i.e., the flattest interpolators always generalize. This establishes a direct link between flatness and generalization which applies to a large class of activations and realistic data distributions.

Harsh Vardhan, Hossein Taheri, Arya Mazumdar

Neural Network Generalization Flat Minima Statistical Learning Theory Feedforward Neural Networks Multi-Index Models

Jun 2, 2026cs.LG

FFR: Forward-Forward Learning for Regression

The Forward-Forward (FF) algorithm offers a computationally efficient and biologically plausible alternative to backpropagation (BP) by training neural networks through purely local, layer-wise optimization. However, FF is inherently designed for classification via contrastive positive-negative sample pairs, and extending it to regression poses fundamental challenges: continuous target space lacks natural "opposites" for contrastive learning, and the standard goodness function carries no information about target magnitude or ordering. We propose FFR (Forward-Forward for Regression), to our knowledge, the first framework to extend FF to real-world regression and demonstrate competitive performance across diverse realworld datasets. FFR introduces three key innovations: (1) an ordinal competitive goodness function that replaces contrastive pairs with competitive learning between partitioned neuron groups under distance-aware ordinal supervision; (2) a stratified ladder architecture where shallow layers learn coarse ordinal discrimination and deeper layers refine into fine-grained regression, with multi-scale feature aggregation for inter-layer collaboration; and (3) hierarchical prediction with uncertainty estimation, where multi-scale predictors jointly provide robust predictions and a single-pass uncertainty score. Extensive experimental results show FFR recovers on average 98.5% of BP's accuracy across six real-world regression benchmarks while reducing peak training memory to only 27% of BP's at depth 8 and 8% at depth 32, with per-iteration time around 72% of BP's, and substantially outperforms all BP-free competitors.

Xinyang Liu, Xuanyu Liang, Shiqi Ding +4

University of Bristol · University College London · University of Cambridge

Neural Network Optimization Feedforward Neural Networks Local Learning Forward-Forward Learning

Jun 2, 2026hep-ph

Predicting the Neutrino Mass Ordering Using Neural Networks

Determining the neutrino mass ordering remains a central open problem in particle physics. While next-generation long-baseline experiments are expected to resolve this question, current data provide limited sensitivity because the spectral differences between normal and inverted ordering are subtle and entangled with parameter degeneracies. We investigate a machine-learning strategy for mass-ordering determination using a feed-forward neural-network classifier trained on synthetic long-baseline datasets generated with three-flavour oscillation probabilities, matter effects, and statistical fluctuations. We evaluate the classifier against standard

χ^2

and

\log\mathcal{L}

approaches using common discrimination metrics, including receiver-operating-characteristic curves, to quantify sensitivity and to illustrate how operating points can be selected to prioritise purity or efficiency. We find that the neural network achieves performance comparable to conventional fits for the scenarios studied, providing a flexible, independent cross-check of established analyses. The framework can be extended to incorporate systematic uncertainties and to explore joint inference of oscillation parameters, and it may also serve as a pedagogical tool for introducing machine-learning methods in neutrino physics.

T. J. C. Bezerra, L. Asquith, E. Bannister +1

Department of Physics and Astronomy, University of Sussex, Brighton, BN1 9QH, United Kingdom · †Now at Universal Quantum Ltd., Haywards Heath, RH16 1XQ, United Kingdom

High-Energy Physics Feedforward Neural Networks

May 27, 2026cs.LG

Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Bias

Randomly initialized neural networks induce a prior over functions, but the predictor used in practice is produced only after training. We ask how much of this initial bias survives the training pipeline. To make the question measurable, we introduce initialization memory: the dependence of the validation-selected predictor on the scale of the random initialization. We perform controlled CIFAR-10 experiments on ResNets where initialization memory already sharply separates training regimes. Low-learning-rate SGD can interpolate while still remembering its initialization: on ResNet-9 with batch size

b=128

, test accuracy varies by

26.5

percentage points across initialization scales despite

\ge99.5\%

training accuracy. This is not undertraining: extending the same low-learning-rate regime to

5{,}000

epochs leaves the spread essentially unchanged. In contrast, Adam-family methods largely erase the dependence. SGD can also be made to forget when larger learning rates are paired with explicit

L_2

norm control. We interpret these findings in terms of the time scale of forgetting: gradient-flow-like dynamics can preserve initialization memory, whereas stochastic finite-step effects, explicit norm decay, and adaptive preconditioning erase it on scales governed by the size of explicit or implicit regularization. The practical inductive bias of a trained network is therefore not the architectural prior alone, but the architectural prior after being filtered by the forgetting dynamics of the training pipeline; and the same regularizers that improve generalization are precisely those that erase memory of initialization.

Mohua Das, Pierfrancesco Beneventano, Shibshankar Dey +2

MIT · Northwestern University

Neural Network Generalization Neural Network Initialization Neural Network Training Dynamics Feedforward Neural Networks Stochastic Gradient Descent

May 27, 2026cs.LG

The Hamilton-Jacobi Theory of Deep Learning

In this paper, training a neural network is identified, exactly, as a search through Hamilton--Jacobi initial-value problems: each gradient step selects the initial data of a viscous Hamilton--Jacobi equation whose Hopf--Cole propagator best fits the observations; at inference, the input is the spatial point at which that solution is evaluated and the initial condition is already encoded in the weights. The correspondence is exact for log-sum-exp layers and structural for broader architectures: residual networks, transformers, and recurrent architectures (RNNs, LSTMs, SSMs) each discretize the same class of Hamilton--Jacobi equations, with architecture-dependent Hamiltonian and viscosity. A single deformation parameter

\varepsilon

unifies all four perspectives (network, tropical algebra, viscous PDE, convex optimization) in a commutative diagram closed under Lipschitz conditions. Quantitative consequences include: the minimax optimal generalization rate

O(n^{-1/(d+2)})

for fixed

t

; adversarial robustness controlled by

\varepsilon

; backpropagation as the co-state equation of the Hamiltonian system for residual networks (Pontryagin Maximum Principle); scaling exponents consistent with data intrinsic dimension via PDE quadrature; and a closed-form

O(N)

influence function (softmax attribution weights

π_j

) whose entropy landscape undergoes fold bifurcations as

\varepsilon

increases, each merging attribution basins.

Jose Marie Antonio Miñoza, Erika Fille T. Legara, Christopher P. Monterola

Center for AI Research PH · Asian Institute of Management

Neural Network Generalization Neural Network Training Dynamics Feedforward Neural Networks

May 26, 2026cs.LG

More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activations

Feedforward network (FFN) layers account for a large fraction of parameters and nonlinear expressivity in Transformer-based large language models (LLMs). Despite the evolution from ReLU and GELU to gated variants such as SwiGLU, most FFN designs still use a single fixed activation function, applying the same nonlinear transformation to all tokens. In this work, we propose Mixture of Activations (MoA), a token-adaptive FFN design that mixes a dictionary of activation functions using lightweight input-dependent gates while sharing the same linear projections. As an input-independent counterpart, we also introduce learnable activations (LA), which form linear combinations of activation functions for both ReLU-type and SwiGLU-type FFNs. Theoretically, we establish strict finite-width expressive separations among fixed-activation FFNs, LA, and MoA: LA strictly contains fixed-activation FFNs, while MoA strictly contains LA, with the additional expressivity arising from input-dependent nonlinear hybridization. Empirically, we evaluate MoA through extensive pre-training experiments on dense and MoE language models ranging from 0.12B to 2B parameters under different token budgets, optimizers, and learning rate schedules. MoA consistently achieves lower terminal loss and exhibits more favorable scaling behavior than well-tuned baselines, with minimal parameter and computational overhead. These results suggest that token-adaptive activation mixing is a simple and effective mechanism for improving FFN expressivity in LLMs.

Mingze Wang, Jinbo Wang, Yikuan Xia +2

ByteDance Seed · Peking University

Transformer FFNs Transformer Expressivity Transformer Neural Network Activation Functions Feedforward Neural Networks Neural Network Expressivity

May 23, 2026cs.LG

CAffNet: Hard Constraint-Affine Neural Networks

We present a novel framework for embedding hard constraint satisfaction into neural network (NN) architectures, specifically feedforward neural networks and transformers, with input-dependent affine constraints of arbitrary cardinality. Traditional constraint enforcement approaches either rely on penalty-based soft constraints, which offer no guarantee of satisfaction, or on post-processing methods that enforce constraints after the NN is trained, which may lead to suboptimality. We introduce a trainable constraint-affine (CAffine) layer into NNs, yielding CAffNet, which goes beyond enforcing affine constraints via fixed orthogonal or parallel projections and enables joint optimization with network parameters. Moreover, we impose no restrictions on the constraint space dimensions and establish that our construction preserves the universal approximation properties of NNs, while providing provable guarantees on constraint adherence for all inputs. Experimental validation demonstrates robust performance across diverse domains requiring guaranteed constraint satisfaction. The code is available at https://github.com/ice-t-lab/CAffNet.git.

Yang Zhao, Jungeun Lee, Jeong hwan Jeon +1

Department of Mechanical and Industrial Engineering, Northeastern University, Boston, MA 02115 USA · Department of Electrical Engineering, Ulsan National Institute of Science and Technology, Ulsan 44919, Republic of Korea

Feedforward Neural Networks

May 23, 2026cs.LG

ChainzRule: Sample-Efficient, Robust Deep Learning Across Tabular, NLP, and Vision Tasks

Production deep learning systems across enterprise domains operate under constraints that academic benchmarks routinely obscure: labeled data is expensive, inference budgets are tight, and models that cannot explain their behavior are difficult to trust and maintain. We present ChainzRule (CR), a neural architecture replacing typical activations with learnable polynomial layers governed by Differential Regularization (DREG), a layer-wise Jacobian penalty computed analytically during the forward pass at standard inference cost. The core claim is that bounding intermediate derivatives forces the network toward low-frequency, structurally stable representations, simultaneously reducing dependence on labeled data volume, improving robustness to distribution shift, and providing a measurable, gradient-based handle on model behavior. Evaluated across five domains, CR achieves

85.71\% \pm 2.01\%

on Pima Diabetes (statistically superior to SVM and XGBoost),

46.20\% \pm 0.37\%

on SST-5 sentiment classification with a frozen encoder (superior to RNTN using approximately 5% of its training data),

55.79\%

on SST-5 with a fine-tuned BERT backbone (versus BERT-base linear head at

54.9\%

),

70.17\%

on Yelp Full ordinal regression with 3.2M parameters versus a 10-model average of

66.35\%

, and

+2.32\%

mean corruption accuracy on CIFAR-10-C. All results with reported

p

-values fall below the

α= 0.05

threshold after Bonferroni correction. CR maintains a gradient tail ratio

τ

(p99/mean) of

1.01

--

1.02

against

1.07

--

1.09

for all typical activation function baselines across every data fraction, a structural invariant we propose as the mechanistic driver of sample efficiency and a deployment-time proxy for model reliability.

Rowan Martnishn

Sentivity AI

Polynomial Neural Networks Neural Network Robustness OOD Generalization Jacobian Regularization Neural Network Activation Functions Neural Network Interpretability+3

May 22, 2026cs.RO

How Many Training Samples Are Needed for the Inverse Kinematics Solutions by Artificial Neural Networks

Inverse Kinematics (IK) plays a critical role in robotic motion planning and control. The IK solutions of a robot manipulator could be done by conventional ways such as geometric, algebraic, or Jacobian methods, which have drawbacks. The Artificial Neural Networks (ANNs) have become a promising alternative for approximating IK solutions due to their generalization ability and computational efficiency. This approach basically trains only a few samples of the end effector that are recorded for the solution of the IK problem. However, a fundamental question remains: how many training samples are sufficient to achieve reliable and accurate IK predictions? This study investigates the mathematical framework of relating the size of training datasets and the accuracy of ANN-based IK solvers. Using an articulated robotic manipulator, we generate varying amounts of joint-position pairs to train feedforward neural networks and assess their accuracy, convergence, and generalization capability. The results reveal more training samples than 125 did not contribute to the improvement of the model efficiency that the comparable measure dealing with the approximation accuracy over the sampling size, offering valuable insight into data efficiency. This work provides practical guidance for optimizing the data sizing of ANN solutions, balancing computational cost and model accuracy for real-world robotic applications.

Dong-Won Lim

The University of Suwon, Hwaseong 18323, Republic of Korea

Sample Complexity Inverse Kinematics Feedforward Neural Networks

May 21, 2026stat.ML

Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networks

We consider one-hidden layer neural networks trained in the feature-learning regime using gradient descent, and relate the output of the finite-width network

f_{\hatρ_t^m}

to its infinite-width counterpart

f_{ρ_t^{MF}}

, which evolves in the mean-field dynamics. While constant-time horizon bounds for

\|f_{ρ_t^{MF}} - f_{\hatρ_t^m}\|

may be obtained via standard Grönwall estimates, the long-time behavior of the fluctuation is a more delicate matter. Uniform-in-time bounds often rely on (local) strong convexity in the landscape or Logarithmic Sobolev inequalities present in noisy gradient dynamics. In this work, we establish non-asymptotic weak propagation-of-chaos that holds uniformly in time, obtained by exploiting instead the convergence rate of the mean-field deterministic Wasserstein-gradient-flow dynamics. Specifically, denoting by

L_t

the mean-field excess MSE loss at time

t

and

m

the number of neurons, under standard regularity assumptions and the condition

\int_0^\infty L_t^{1/2} dt =O(\log d)

, we obtain the uniform in time bound

\|f_{ρ_t^{MF}}- f_{\hatρ_t^m}\|^2 \lesssim \text{poly}(d) m^{-\min(1,c/6)}

whenever

L_t \lesssim t^{-c}

. Our result holds in a noiseless setting and does not make any assumptions on the geometry of the landscape near the optimum, and extends seamlessly to other forms of discretization, including finite number of samples and time discretization. A key takeaway of our result is that whenever the convergence rate of the mean-field, population-loss dynamics is faster than

t^{-2}

, we can attain a loss of

ε

with only

\text{poly}(d/ε)

neurons, training samples, and GD steps.

Margalit Glasgow, Joan Bruna

Massachusetts Institute of Technology · Courant Institute School of Mathematics, Computing and Data Science, New York University

Shallow Neural Networks Mean-Field Theory Neural Network Training Dynamics Feedforward Neural Networks Gradient Descent Dynamics

May 20, 2026cs.LG

Approximation Theory for Neural Networks: Old and New

Universal approximation theorems provide a mathematical explanation for the expressive power of neural networks. They assert that, under mild conditions on the activation function, feedforward neural networks are dense in broad function classes, such as continuous functions on compact subsets of

\mathbb{R}^d

,

L^p

spaces, or Sobolev spaces. Over the past four decades, these qualitative universality results have evolved into a rich quantitative theory addressing approximation rates, parameter efficiency, and the role of architectural features such as depth and width. This survey presents several glimpses into this theory. We review classical density results for single-hidden-layer networks, as well as quantitative bounds that relate approximation error to network size and smoothness assumptions on target functions. Particular emphasis is placed on depth--width trade-offs and on results demonstrating that deeper architectures can achieve superior parameter efficiency for structured function classes. In addition to standard feedforward neural networks, we also review recent developments on Kolmogorov--Arnold Networks (KANs), which offer an alternative architectural paradigm and whose approximation-theoretic properties have begun to attract significant theoretical attention.

Soumendu Sundar Mukherjee, Himasish Talukdar

Statistics and Mathematics Unit, Indian Statistical Institute, Kolkata

Universal Approximation Neural Network Approximation Theory Feedforward Neural Networks Neural Network Expressivity

May 8, 2026stat.ML

Every Feedforward Neural Network Definable in an o-Minimal Structure Has Finite Sample Complexity

We show that, in a precise sense, a broad class of feedforward neural networks learn (have finite sample complexity) in the PAC model: every fixed finite feedforward architecture whose layers are definable in an o-minimal structure has finite sample complexity in the agnostic PAC setting, even with unbounded parameters. This covers standard fixed-size MLPs, CNNs, GNNs, and transformers with fixed sequence length, together with the operations and layers typically used in such architectures, including linear projections, residual connections, attention mechanisms, pooling layers, normalization layers, and admissible positional encodings. Hence, distribution-free learnability for modern non-recurrent architectures is not an exceptional property of particular activations or architecture-specific VC arguments, but a consequence of tame feedforward computation. Our results reposition finite-sample PAC learnability as a baseline rather than a differentiator: they shift the focus of architectural comparison toward inductive biases, symmetries and geometric priors, scalability, and optimization behaviour.

Anastasis Kratsios, Gregory Cousins, Haitz Sáez de Ocáriz Borde +2

Department of Mathematics & Statistics, McMaster University, Hamilton, ON, Canada · University of Cambridge, United Kingdom · Graduate School of Engineering, The University of Tokyo, Tokyo, Japan +1

Neural Network Generalization PAC Learning Feedforward Neural Networks

May 5, 2026cs.LG

Probabilistic Classification and Uncertainty Quantification of Sahara Desert Climate Using Feedforward Neural Networks

Climate classification plays a vital role in agricultural planning, hydrological studies, and climate science. One of the most widely used systems for classifying global climate zones is the Köppen-Trewartha (KT) classification. However, the KT classification is fundamentally deterministic, offering discrete labels to spatial locations without accounting for uncertainties in classification. In this paper, we provide a framework for probabilistic modeling of climatic zones. We implement a feedforward artificial neural network (ANN) for classification, allowing for efficient, uncertainty-aware categorization of climatic regions, thereby offering a more nuanced understanding of transitional climate zones compared to traditional deterministic methods. We apply this method to the Sahara Desert region over the 30-year period of 1960 - 1989, using data at more than 400,000 space-time locations from the first 11 years to train our model. We assess the model's short- and long-term classification capabilities to evaluate its stability and accuracy over time. We also compare the probabilistic classification from our model with the traditional KT classification. In addition, we use fluctuation analysis methods to highlight the temporal evolution of climatic zones across the Sahara region and identify areas undergoing significant flux of probabilities of their climate classes, providing insights into broader trends in desertification.

Stephen Tivenan, Indranil Sahoo, Yanjun Qian

Department of Statistical Sciences and Operations Research, Virginia Commonwealth University

Uncertainty Quantification Feedforward Neural Networks

May 4, 2026cs.LG

Communication Dynamics Neural Networks: FFT-Diagonalized Layers for Improved Hessian Conditioning at Reduced Parameter Count

Communication Dynamics Neural Networks (CDNNs) apply the circulant-spectral machinery of the Communication Dynamics framework to neural-network layer design. We introduce CDLinear, a block-circulant linear layer with block size B = 2l + 1 that uses 1/B the parameters of a dense layer with the same input and output dimensions. The construction gives an explicit Fourier-domain diagnostic for optimization: for mean-squared loss, the weight Hessian is diagonalized by the discrete Fourier transform, with eigenvalues determined directly by the Fourier spectrum of the input blocks. Under input pre-whitening, the population Hessian condition number is exactly 1, and the empirical condition number is bounded by 1 + O(sqrt(B/N)) for N samples. We implement CDLinear in pure NumPy with hand-derived backward passes and verify gradients by finite differences. On the 8x8 MNIST digits benchmark, across three random seeds, a CDLinear MLP with B = 4 reaches 97.50% +/- 0.23% test accuracy using 2,380 parameters, compared with 98.15% +/- 0.47% for a dense baseline using 8,970 parameters. This gives a 3.8x parameter reduction at a 0.65% accuracy cost. The CD-MLP's mean Hessian condition number is 1.9e4, about 310x smaller than the dense baseline's 5.9e6. We position CDLinear as a special case of structured matrix neural-network layers, with the main contributions being a closed-form Hessian-spectrum diagnostic, a principled discrete sequence of block multiplicities, and an explicit conditioning analysis. We also release a reference PyTorch implementation integrating CDLinear into a DeepSeek-V3-style mixture-of-experts transformer for future large-scale benchmarks.

Lurong Pan

Neural Network Optimization Feedforward Neural Networks Neural Network Compression

May 1, 2026cs.LG

Diffusion Operator Geometry of Feedforward Representations

Neural networks transform data through learned representations whose geometry affects separation, contraction, and generalization. Recent work studies this geometry using discrete curvature on neighborhood graphs, suggesting Ricci-flow-like behavior across layers. We develop a smooth operator-theoretic alternative for feedforward representation snapshots. Each feature cloud induces a Gaussian-kernel diffusion Markov operator, and transport, spectral, label-boundary, and local-scale observables are derived from this single object via Bakry-Emery

Γ

-calculus. In a balanced Gaussian class-conditional snapshot model with shared covariance, the population operator has closed-form class affinities, leakage, and coarse spectra, all controlled by pairwise regularized Mahalanobis separations

c_\varepsilon^{(a,b)}

. We also prove that the resulting operator observables vary smoothly under feature perturbations, while hard neighborhood-graph diagnostics can change discontinuously. Synthetic experiments validate the closed-form Gaussian bridge, while learned MNIST experiments show that the same operator observables track training, width, and perturbation stability. Together, these results give a stable operator-geometric framework for analyzing feedforward representation geometry.

Kanishka Reddy

Department of Applied Mathematics University of Washington Seattle, WA

Neural Representation Geometry Feedforward Neural Networks

Apr 29, 2026cs.LG

Random Cloud: Finding Minimal Neural Architectures Without Training

I propose the \emph{Random Cloud} method, a training-free approach to neural architecture search that discovers minimal feedforward network topologies through stochastic exploration and progressive structural reduction. Unlike post-training pruning methods that require a full train-prune-retrain cycle, this method evaluates randomly initialized networks without backpropagation, progressively reduces their topology, and only trains the best minimal candidate at the end. I evaluate on 7 classification benchmarks against magnitude pruning and random pruning baselines. The Random Cloud matches or outperforms both baselines in 6 of 7 datasets, achieving statistically significant improvements on Sonar (

+4.9

pp accuracy,

p{=}0.017

vs magnitude pruning) with 87% parameter reduction. Crucially, the method is faster than both pruning baselines in 4 of 5 datasets (0.67--0.94

\times

the cost of full training), since it avoids training the full-size network entirely.

Javier Gil Blázquez

Neural Architecture Search Feedforward Neural Networks Neural Network Compression

Apr 28, 2026cs.LO

Verification of Neural Networks (Lecture Notes)

These lecture notes provide an introduction to the verification of neural networks from a theoretical perspective. We discuss feed-forward neural networks, recurrent neural networks, attention mechanisms, and transformers, together with specification languages and algorithmic verification techniques.

Benedikt Bollig

Universit´e Paris-Saclay, CNRS, ENS Paris-Saclay, LMF

Neural Network Verification Feedforward Neural Networks

Jan 30, 2026cs.AI

Complete Identification of Deep ReLU Networks through Łukasiewicz Logic

Two deep ReLU networks can have entirely different architectures and parameters, yet realize the same function. We provide a complete characterization of this nonuniqueness. This is effected by building a symbolic calculus for deep ReLU networks, equivalence and simplification of networks becoming derivation of formulae, in close parallel to Shannon's analysis of switching circuits through Boolean logic. Inspired by Shannon, who turned circuit synthesis into the manipulation of Boolean formulae by the axioms of Boolean algebra, we turn ReLU network identification into the derivation of Łukasiewicz formulae by the axioms of many-valued (MV) logic. Two non-degenerate ReLU networks realize the same function on the unit cube if and only if one is obtained from the other by finitely many applications of the MV axioms for integer weights and biases, the divisible MV axioms for rational ones, and the Riesz MV axioms for real ones. The MV logic axioms characterize all symmetries of ReLU networks, the single-layer ones, which for tanh networks are the only kind, and the deep ones, spanning three or more layers. Our framework consists of three steps, an extraction algorithm turning a network into a substitution graph, whose represented formula has the network's input-output map as its truth function, a completeness theorem, by which functionally equivalent formulae are interderivable, and a construction algorithm returning from graphs to networks. The substitution graph is layered, carrying at each node a formula in the variables of the layer feeding it, encodes the network uniquely, and induces a new normal form for MV logic, compositional rather than flat as in the literature, hence retaining the algebraic structure of the network, with three local operations--node rewrite, layer collapse, layer expansion--realizing every derivation.

Yani Zhang, Helmut Bölcskei

Chair for Mathematical Information Science, ETH Zürich

Parameter Identifiability ReLU Neural Networks Feedforward Neural Networks

May 30, 2024cs.LG

A Computational Tropical Geometry Framework for Neural Networks

We propose a computational tropical geometry framework for the symbolic analysis of neural networks with tropical activations. The number of linear regions of a neural network has been actively studied as a measure of the expressivity of a given architecture. To study these, we work in the setting of tropical geometry---a combinatorial and polyhedral variant of algebraic geometry---where there are known connections between tropical rational maps and feedforward neural networks. We expand this connection by developing concrete computational tools for studying the linear regions of neural networks. We present an algorithm, together with a proof of correctness, which computes the linear regions of a neural network as explicit unions of polyhedra. We further relate the computation of the number of linear regions of a tropical expression to the number of monomials that appear in it, and show how tropical expressions can often be pruned to remove redundant monomials. We introduce the Hoffman constant of a neural network's tropical expression, a geometric quantity that controls the distance from any point in the input space to the farthest linear region. We provide the open source Julia library TropicalNN.jl, which is built on top of the OSCAR computer algebra system and implements the algorithms mentioned above to analyze neural networks symbolically using their tropical representations. We present a set of proof-of-concept computational examples to demonstrate how our tropical geometric theory can be applied to reveal insights on the expressivity of a network architecture.

Paul Lezeau, Thomas Walker, Yueqi Cao +2

Department of Mathematics, Imperial College London, UK · London School of Geometry and Number Theory, UK · Department of Electrical and Computer Engineering, Rice University, USA +2

Feedforward Neural Networks Neural Network Expressivity