Neural Network Expressivity

Overview

It asks what mappings a model can represent exactly or approximate, and how the needed width, depth, parameters, precision, and activation choices scale with the task. Common approaches include universal-approximation results and lower bounds, counting linear regions or polynomial features, circuit-complexity and logic characterizations, and geometric analyses of MLPs, recurrent models, transformers, and graph networks. These tools reveal when depth, recurrence, multiplicative gates, or higher-order interactions add capabilities, often using tasks such as parity, memorization, graph distinguishing, and smooth-function approximation. A central tension is that theoretical capacity does not ensure training will find or use it, and results under idealized assumptions about precision or weights may not transfer directly to practical models.

Latest papers

CardsList

123 papers5 papers in the last four weeks, down 50% on the four weeks before. 0.0% of all new papers.

  1. Overcoming a Learning Bottleneck in Impartial Games: A Circuit Complexity Approach

    Nov 10, 2024Søren RiisCircuit ComplexityNeural Network Expressivity

  2. A Computational Tropical Geometry Framework for Neural Networks

    May 30, 2024Paul Lezeau, Thomas Walker, Yueqi Cao +2Feedforward Neural NetworksNeural Network Expressivity

  3. Deep Network Approximation: Beyond ReLU to Diverse Activation Functions

    Jul 13, 2023Shijun Zhang, Jianfeng Lu, Hongkai ZhaoReLU Neural NetworksNeural Network Activation Functions