cs.LGAug 12, 2026

TESLA: Taylor Expansion of Sinusoidal Learnable Activations

Authors: Daehwa Ko, Jaehyeon Kim, Seunghyun Ham, Jay Hoon Jung

Organizations: Korea Aerospace University, Goyang, Republic of Korea

Abstract

The parity problem--deciding whether the number of ones in a binary vector is odd or even--remains challenging for standard neural networks due to linear inseparability and the need for global interactions. We propose TESLA, an activation defined as a learnable combination of sine and cosine terms, enabling explicit control over polynomial degree and selective amplification of high-order components. Theoretically, we show that constraining TESLA's coefficients yields Lipschitz/Rademacher complexity bounds and shapes the training dynamics to emphasize higher-frequency structure. Empirically, on parity with input length n = 32, TESLA attains strong generalization with 100K training samples (approximately 0.002% of the 2^32 input space) and remains robust under heavy corruption, retaining high accuracy with up to 30% label noise. We also compare against periodic and frequency-based baselines (SIREN, SNAKE, and Fourier feature embeddings) on parity and Forrelation. Beyond synthetic structure, TESLA delivers comparable performance on ImageNet-100, indicating that activation-level degree control transfers to more general vision workloads. Code: https://github.com/KAU-QuantumAILab/TESLA

Explore similar work

CardsList
Feb 12, 2026cs.LG

Rational Neural Networks have Expressivity Advantages

We study neural networks with trainable low-degree rational activation functions and show that they are more expressive and parameter-efficient than modern piecewise-linear and smooth activations such as ELU, LeakyReLU, LogSigmoid, PReLU, ReLU, SELU, CELU, Sigmoid, SiLU, Mish, Softplus, Tanh, Softmin, Softmax, and LogSoftmax. For an error target of ε>0\varepsilon>0, we establish approximation-theoretic separations: Any network built from standard fixed activations can be uniformly approximated on compact domains by a rational-activation network with only poly(log⁡log⁡(1/ε))\mathrm{poly}(\log\log(1/\varepsilon)) overhead in size, while the converse provably requires Ω(log⁡(1/ε))Ω(\log(1/\varepsilon)) parameters in the worst case. This exponential gap persists at the level of full networks and extends to gated activations and transformer-style nonlinearities. In practice, rational activations integrate seamlessly into standard architectures and training pipelines, allowing rationals to match or outperform fixed activations under identical architectures and optimizers.
Maosen Tang, Alex Townsend
Activation FunctionsNeural NetworkExpressivityApproximation Algorithms
AskPDF
May 27, 2026cs.LG

Learning High-Dimensional Parity Functions with Product Networks using Gradient Descent

Parity functions are fundamental Boolean operations with critical applications across machine learning, cryptography, and error correction. Yet, learning high-dimensional parity functions poses significant challenges: in a general setting, standard neural network architectures typically require exponential sample complexity, making gradient-based optimization intractable for large number of inputs NN. We demonstrate that compact product-based neural architectures combined with stochastic data sparsity (Bernoulli inputs with pe≤1/Np_e \leq 1/N) and appropriate hyperparameter choice enable efficient parity learning, with theoretical guarantees of convergence. Experiments validate our theory across dimensions up to N=100,000N = 100{,}000, with empirical evidence showing optimal hyperparameter choices for pep_e and learning rate αα, as well as polynomial complexity scaling laws. This work establishes fundamental connections between architectural inductive bias and data sparsity, opening new possibilities for neural arithmetic, structured reasoning, binary neural networks, and machine learning applied to automated protocol discovery.
Guillaume Larue, Louis-Adrien Dufrène, Quentin Lampin +2
Two-Layer Neural NetworksNeural NetworkOptimal Sample ComplexitySparsityGradient DescentPrivacy-Preserving Framework+2
AskPDF
May 20, 2026cs.CV

Activation-Free Backbones for Image Recognition: Polynomial Alternatives within MetaFormer-Style Vision Models

Modern vision backbones treat pointwise activations (e.g., ReLU, GELU) and exponential softmax as essential sources of nonlinearity, but we demonstrate they are not required within MetaFormer-style vision backbones. We design activation-free polynomial alternatives for three core primitives (MLPs, convolutions, and attention), where Hadamard products replace standard nonlinearities to yield polynomial functions of the input. These modules integrate seamlessly into existing architectures: instantiated within MetaFormer, a modular framework for vision backbones, our PolyNeXt models match or exceed activation-based counterparts across model scales on ImageNet classification, ADE20K semantic segmentation, and out-of-distribution robustness. We also substantially outperform prior polynomial networks at reduced computational cost, showing that polynomial variants of standard modules beat complex custom architectures.
Jeffrey Wang, Jonathan Gregory, Grigorios G. Chrysos
Activation FunctionsImagenetMixformerPolynomials
AskPDF