cs.LGOct 10, 2026

Rethinking Activation Functions: Reducing Bias Can Improve Learning

Authors: Zirui Peng, Heyi Zheng, Ziming Liu

Organizations: Department of Physics, Peking University · Department of Mathematics, Peking University · College of AI, Tsinghua University · MetaCircle

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Many widely used activation functions, including ReLU, GELU, Swish, and Sigmoid, produce predominantly positive outputs, resulting in a nonzero post-activation mean that we term activation bias. Despite its ubiquity, the effect of activation bias on learning dynamics remains poorly understood. Using controlled models, we show that activation bias accelerates initial loss reduction but is accompanied by overshooting, non-smooth trajectories, and a steeper loss landscape. This early advantage also coincides with the model's use of task-relevant information encoded in the activation bias component. In contrast, reducing activation bias slows initial learning but ultimately yields a lower final loss. Experiments on larger vision and language models exhibit a similar pattern: smaller activation bias is associated with better late-stage performance. Thus, this seemingly minor property can substantially redirect learning and warrants explicit consideration in future activation function design.

Figures & tables

Figure 1: Overview of activation bias and its effects on learning. Blue denotes the unshifted activation, and red denotes the shifted activation. Reducing activation bias changes the post-activation distribution, model performance, hidden-state trajectory, and loss landscape.
Figure 2: Activation bias accelerates early learning, but this advantage coincides with a pronounced hidden-state overshoot and reverses later in training. Top: The shifted model learns more slowly initially but ultimately reaches a lower loss across activation functions. Bottom: The unshifted model’s hidden state overshoots and follows a non-smooth trajectory.
Activation functionsReLUGELUSwishSigmoid
bb0.2300.2300.1150.1150.0770.0770.5000.500
Table 1: Activation shifts bb used for each activation function.
Figure 3: Activation bias redirects parameter updates, producing a distinct trajectory with a sharp, non-smooth turn. Left: the unshifted student on the initial-gradient plane. Center: the shifted student on the same plane. Right: the shifted student on the complementary plane. Colors closer to red indicate higher loss, while colors closer to blue indicate lower loss.
Figure 4: Reducing activation bias substantially lowers Hessian eigenvalues throughout training, yielding a flatter local loss landscape.
Figure 5: The unshifted model recovers recognizable image content from the activation bias component early in training but relies less on it later; the shifted model recovers little content from this component.
Figure 6: For the ViT image-classification task, the shifted model retains a higher training loss but ultimately achieves lower test loss and higher test accuracy. The loss residuals are Lshifted−LunshiftedL_{\mathrm{shifted}}-L_{\mathrm{unshifted}} , and the accuracy residual is Accshifted−Accunshifted\mathrm{Acc}_{\mathrm{shifted}}-\mathrm{Acc}_{\mathrm{unshifted}} .
Figure 7: (a) Reducing activation bias suppresses hidden-state overshoot in ViT. (b) and (c): The shifted model performs better in both DiT image generation and GPT-2 language pretraining. The loss residual is Lshifted−LunshiftedL_{\mathrm{shifted}}-L_{\mathrm{unshifted}} .
Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: The post-activation means of the shifted models remain near zero throughout training.
Figure 9: Across all five activations, our theory’s predicted shifts achieve near-optimal final loss. Thin curves show individual seeds, and thick curves show their mean.
Figure 10: Empirical evaluation of Proposition B.2 . Left: The theoretical lower and upper bounds closely track the measured leading Gauss–Newton eigenvalue λG(b)\lambda_{G}(b) . Right: The value at the average-mean shift, λG(μˉ)\lambda_{G}(\bar{\mu}) , remains close to the empirical minimum, λG(b∗)\lambda_{G}(b^{*}) , throughout training.
Figure 11: After 100 updates, the largest eigenvalues of the exact Hessian and its Gauss–Newton approximation remain close at b=μˉb=\bar{\mu} .
\toprule Step kkTheory angleFull-model angle
\midrule 177.45∘77.45^{\circ}76.25∘76.25^{\circ}
577.44∘77.44^{\circ}75.01∘75.01^{\circ}
2077.40∘77.40^{\circ}69.22∘69.22^{\circ}
10077.22∘77.22^{\circ}46.58∘46.58^{\circ}
\bottomrule  
Appendix
Table 2: Angles between the unshifted and shifted parameter gradients. The theoretical and measured angles are close early in training.
Figure 12: Shifting tends to yield greater improvements as model width and depth increase. For ReLU, colors show the relative improvement (Lunshifted−Lshifted)/Lunshifted(\mathcal{L}_{\mathrm{unshifted}}-\mathcal{L}_{\mathrm{shifted}})/\mathcal{L}_{\mathrm{unshifted}} , with red indicating improvement and blue indicating deterioration.
Figure 13: As model size increases, the empirically optimal shifts across seeds become more concentrated near the theory-predicted shift.
Figure 14: The shifted model retains its loss advantage in more general settings.
Figure 15: Reducing activation bias suppresses hidden-state overshoot and yields smoother hidden-state mean trajectories.
Figure 16: Architecture of the autoencoder. Numbers on arrows denote channel counts. The shifted model replaces each ReLU with ReLU⁡(z)−b\operatorname{ReLU}(z)-b .
Figure 17: The shifted model initially has a higher reconstruction loss but achieves a lower loss later in training. The plotted residual is Lshifted−LunshiftedL_{\mathrm{shifted}}-L_{\mathrm{unshifted}} .
Figure 18: Across different images, reconstructions from the unshifted model’s activation bias component first approach and then diverge from the full reconstructions, whereas those from the shifted model contain little recognizable structure.
Figure 19: The performance improvements from shifting extend to additional scale-up models and tasks. Residuals are computed as ΔM=Mshifted−Munshifted\Delta M=M_{\mathrm{shifted}}-M_{\mathrm{unshifted}} , where MM denotes loss, FID, or accuracy.
Figure 20: Hidden-state mean trajectories exhibit overshoot across different scale-up models.
Figure 21: In OLMo-2-1B, the shifted model maintains more stable post-activation magnitude distributions over training, with less accumulation of near-zero values. Moreover, its post-activation and residual-stream magnitude distributions closely resemble each other, whereas those of the unshifted model differ substantially.
More in this topic

Loss Landscape

See all 22 papers

Explore similar work

CardsList

  1. dynActivation: A Trainable Activation Family for Adaptive Nonlinearity

    Mar 23, 2026Alois BachmannNeural Network Activation FunctionsNeural Network Training Dynamics

    Ruprecht-Karls-Universität Heidelberg

  2. Bug or Feature2^2: Weight Drift, Activation Sparsity and Spikes

    May 17, 2026Egor Shvetsov, Aleksandr Serkov, Shokorov Viacheslav +3ReLU Neural NetworksActivation Sparsity

  3. Mathematical analysis of one-layer neural network with fixed biases, a new activation function and other observations

    Apr 9, 2026Fabricio Macià, Shu NakamuraShallow Neural NetworksNeural Network Training Dynamics

    M2ASAI, Universidad Politécnica de Madrid. ETSI Navales. Avda. de la Memoria, 4. 28040 Madrid, Spain. · Department of Mathematics, Faculty of Sciences, Gakushuin University, 1-5-1, Mejiro, Toshima, Tokyo, 171-8588 Japan