cs.LGSep 18, 2026

The Ups and Downs of Backprop Weights

Authors: Giuseppe Chindemi, Benjamin F. Grewe

Organizations: Institute of Neuroinformatics, UZH / ETH Zurich ETH AI Center Zurich, Switzerland

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Backpropagation (BP) has driven the remarkable success of modern deep learning by enabling large hierarchical networks to learn complex functions end-to-end. Yet it does not by itself determine how parameters should be organized so that functional components can be reused and adapted selectively. For example, object recognition and motion prediction may depend on overlapping parameter sets, making them difficult to isolate or modify independently. We call this condition weight entanglement. Modern architectures dynamically select which parts of a network process each sample: nonlinearities gate units, attention selects interactions, and Mixture-of-Experts architectures route inputs to modules. Yet such selection does not ensure that the same functional component remains linked to an identifiable parameter set across samples. We propose weight operators: parameterized modules that implement reusable functional components and can be composed at inference to form the function required by each sample. Learning proceeds in two stages: the model first infers the required operator composition, then updates only the selected operators' parameter sets. Vector Networks (VNs) provide one implementation. They couple operator selection to local error-driven updates within each layer and show that learned operators can be reused in combinations absent from training while updates remain restricted to the selected parameter sets. This provides a basis for testing functional parameter identifiability: whether an operator remains linked to the same functional component during learning. We argue that functional parameter identifiability may provide an organizing principle for models that systematically reuse and recombine learned functions while adapting only the components that need to change.

Figures & tables

Figure 1: From weight entanglement to functionally identifiable operators. (a) Perturbing individual weights in a conventional network can produce broad, entangled output changes (left; adapted from Kumar et al., 2025 ), whereas varying operator coefficients changes distinct features more selectively (right; schematic). (b) An entangled parameterization encodes functions in overlapping parameter directions (left); an operator-based representation composes a function from reusable operators, allowing new tasks to recombine them (right; schematic). (c) A rank-1 Vector Network (VN) forms an input-specific weight operator from a sparse composition of operator atoms (schematic based on Pokel and Grewe, 2026 ), W⋆(x)=∑kgk⋆(x)Ak\bm{W}^{\star}(\bm{x})=\sum_{k}g_{k}^{\star}(\bm{x})\bm{A}_{k} , where Ak=uksk⊤\bm{A}_{k}=\bm{u}_{k}\bm{s}_{k}^{\top} and the inferred coefficients gk⋆g_{k}^{\star} determine which atoms participate and are eligible for learning. (d) VNs improve compositional out-of-distribution (OOD) generalization over dense and structured baselines on the nn -body benchmark (adapted from Pokel and Grewe, 2026 ).
More in this topic

Modular Neural Networks

Modular neural networks organize computation into specialized, reusable components that can be routed, composed, or updated independently.

See all 74 papers

Explore similar work

CardsList
May 5, 2026

Most ReLU Networks Admit Identifiable Parameters

We study the realization map of deep ReLU networks, focusing on when a function determines its parameters up to scaling and permutation. To analyze hidden redundancies beyond these standard symmetries, we introduce a framework based on weighted polyhedral complexes. Our main result shows that for every architecture whose input and hidden layers have width at least two, there exists an open set of identifiable parameters. This implies that the functional dimension of every such architecture is exactly the number of parameters minus the number of hidden neurons. We further show that minimal functional representations can still have non-trivial parameter redundancies. Finally, we establish a generic depth hierarchy, whereby for an open set of parameters the realized function cannot be represented generically by any shallower network.

Moritz Grillo, Guido Montúfar

Max Planck Institute for Mathematics in the Sciences, Leipzig · Departments of Mathematics and Statistics & Data Science, UCLA

Representation IdentifiabilityParameter IdentifiabilityReLU Neural NetworksNeural Network Expressivity
AskPDF
May 27, 2026

Learning Compositional Latent Structure with Vector Networks

Deep networks are powerful function approximators, but they typically store many different computations in shared weight matrices, making it difficult to selectively reuse or adapt parts of them when a familiar structure appears in novel combinations. We introduce the Vector Network (VN), a hierarchical recurrent architecture in which each layer replaces a fixed weight matrix with a library of reusable rank-1 weight atoms. For each input, VN minimizes a layer-local energy to infer a sparse set of active weight atoms and their coefficients, jointly constrained by bottom-up input reconstruction and top-down feedback consistency. These weight atom coefficients then compose an input-specific low-rank weight matrix for that sample. After convergence, slow learning updates only the selected weight atoms through local residual signals scaled by the inferred coefficients. We evaluate VN on four compositional benchmarks spanning 1D signals, 2D spatial decoding, N-body dynamics, and compositional MNIST. VN matches strong baselines in distribution while often achieving out-of-distribution error about an order of magnitude lower when familiar factors must be recombined in novel ways. Vector networks thus make compositional generalization a structural property of the architecture and inference process rather than a brittle byproduct of fitting many behaviors into one shared dense parameter substrate.

Niclas Pokel, Benjamin F. Grewe

Institute of Neuroinformatics, UZH / ETH Zurich ETH AI Center Zurich, Switzerland

Representation LearningRecurrent Neural NetworksSparse CodingCompositional Representation LearningCompositional GeneralizationModular Neural Networks
AskPDF
Jul 13, 2026

Backpropagation as a Nilpotent Linear System

Backpropagation is the computational engine of deep learning, yet its mathematical structure is typically treated as a procedural traversal of computational graphs. We present a global operator theory of the F-adjoint framework, which reformulates the layerwise backward recursion of an LL-depth feedforward network into a single linear system (I−\cB)\Xs=\bG(I-\cB)\Xs=\bG, where \bG\bG is a source vector. We prove that the global backward operator \cB\cB is strictly block upper-triangular and nilpotent of index at most LL. This nilpotency guarantees the exact termination of the Neumann series solution after at most LL terms, revealing classical backpropagation to be mathematically equivalent to block back-substitution on an upper bidiagonal system. We formalise F-symmetry -- the condition in which the backward pass perfectly mirrors the forward pass -- identifying orthogonal weight matrices as canonical examples. Through worked numerical examples, we demonstrate how this operator perspective exposes the single-path collapse of strictly feedforward networks and its breakdown in residual architectures. Finally, we leverage this compositional structure to rigorously derive the mechanics of residual networks (gradient highways) and transfer learning (gradient truncation). This framework elevates backpropagation from an algorithmic recipe to a global nilpotent-operator formulation.

Ahmed Boughammoura

Higher Institute of Informatics and Mathematics of Monastir, University of Monastir, 5000 Monastir, Tunisia.

BackpropagationNeural Network Training DynamicsFeedforward Neural Networks
AskPDF