Limitations of Learning Tanh Neural Networks with Finite Precision

Philipp Grohs; Matěj Trödler

cs.LGJun 9, 2026

Limitations of Learning Tanh Neural Networks with Finite Precision

Authors: Philipp Grohs, Matěj Trödler

Organizations: Faculty of Mathematics, University of Vienna · RICAM, Austrian Academy of Sciences

Abstract

We investigate limitations of learning $\tanh$ neural networks from point evaluations under finite-precision computations and $L^p$ accuracy guarantees, building on Berner, Grohs, and Voigtländer (2023). Our approach is based on a novel construction of sharply localized bump functions via iterated $\tanh$ activations. Using this mechanism, we show that, in a finite-precision setting, no adaptive randomized algorithm based on $m$ samples can achieve a convergence rate higher than the Monte Carlo rate $O(m^{-1/p})$ in the $L^p$ norm, unless the sampling budget grows exponentially with the size of the network parameters and architecture. The results reveal fundamental limitations imposed by finite precision on the learnability of classes containing localized bump functions, extending previous results for ReLU networks to the $\tanh$ setting.

Explore similar work

Benign Loss Landscapes Can Coexist with Worst-Case Hardness
Sep 14, 2026Zach Furman, Stephan Wäldchen, Yangda Bei +1Loss Landscape Tensor Networks
Expressive Power of Floating-Point Neural Networks with Arbitrary Reduction Orders and Inexact Activation Implementations
May 27, 2026Yeachan Park, Geonho Hwang, Wonyeol Lee +1Floating-Point Activation Functions
Floating-Point Networks with Automatic Differentiation Can Represent Almost All Floating-Point Functions and Their Gradients
May 3, 2026Sejun Park, Yeachan Park, Geonho HwangFloating-Point Activation Functions

Sep 14, 2026 · cs.LGJ/K move · Enter open · S save

Benign Loss Landscapes Can Coexist with Worst-Case Hardness

Zach Furman, Stephan Wäldchen, Yangda Bei, Liam Hodgkinson

Deep neural networks are expressive enough to contain worst-case targets that can be evaluated in polynomial time but cannot be learned in polynomial time by gradient descent. For practical tasks they nonetheless learn well, raising the question of what non-generic structure of real-world targets enables this. Existing surrogate models cannot pose this question because they either lack hard-to-learn targets entirely (deep linear networks) or cannot evaluate such targets efficiently (kernel methods, infinite-width limits). We study tree tensor networks (TTNs), a model class that generalizes deep linear networks and Tucker decompositions. We show they embed arbitrary read-once Boolean formulas, and thus contain polynomial-size targets that cannot be learned by gradient descent in polynomial time under the same mechanism as neural networks. Despite this, we prove that their loss landscapes are conditionally benign for every realizable target: every local minimum that is minimum-norm is global. Thus, surprisingly, bad local minima are not what distinguishes between typical and worst-case problems in TTNs. Instead, learning difficulty in TTNs can arise from high-order degenerate saddle points, which we show are caused by rank-deficiency. This is explored through a case study of the parity function, illustrating the potential for TTNs to relate landscape geometry to computational hardness.

Loss Landscape Tensor Networks Np-Hard Gradient Descent

Open paper Ask about it PDF

Abstract

Explore similar work

Benign Loss Landscapes Can Coexist with Worst-Case Hardness

Expressive Power of Floating-Point Neural Networks with Arbitrary Reduction Orders and Inexact Activation Implementations

Floating-Point Networks with Automatic Differentiation Can Represent Almost All Floating-Point Functions and Their Gradients