Spectral Bias

Spectral bias in neural networks refers to the tendency of these models, during training, to prioritize learning low-frequency components of a function before higher-frequency ones. Current research focuses on understanding this bias across various architectures, including multi-layer perceptrons, convolutional networks, and transformers, and exploring methods to mitigate it, such as novel initialization schemes, normalization techniques, and modified activation functions. Addressing spectral bias is crucial for improving the accuracy and efficiency of neural networks in applications requiring high-frequency detail, such as image processing, scientific computing, and solving partial differential equations.

Papers

April 22, 2023

The Devil is in the Upsampling: Architectural Decisions Made Simpler for Denoising with Deep Image Prior
Yilin Liu, Jiang Li, Yunkui Pang, Dong Nie, Pew-thian Yap
Image Denoising Trading Devil Deep Image Prior Spectral Bias Upsampling Operation Level Unlearning

January 14, 2023

Understanding the Spectral Bias of Coordinate Based MLPs Via Training Dynamics
John Lazzari, Xiuwen Liu
Neural Tangent Kernel Training Dynamic Coordinate Based Spectral Bias Frequency Representation

January 7, 2023

Investigations on convergence behaviour of Physics Informed Neural Networks across spectral ranges and derivative orders
Mayank Deshpande, Siddharth Agarwal, Vukka Snigdha, Arya Kumar Bhattacharya
Loss Function Physic Informed Neural Network Neural Tangent Kernel Comprehensive Investigation Spectral Bias Convergence Behavior Derivative Process Spectral Band

October 19, 2022

Mitigating spectral bias for the multiscale operator learning
Xinliang Liu, Bo Xu, Shuhao Cao, Lei Zhang
Neural Operator Spectral Bias Multiscale Problem

September 16, 2022

Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Study
Yongtao Wu, Zhenyu Zhu, Fanghui Liu, Grigorios G Chrysos, Volkan Cevher
Neural Network Neural Tangent Kernel Neural Network Architecture Extrapolation Framework Spectral Bias Standard Deep Polynomial Neural Network Hadamard Layer Polynomial Network

August 9, 2022

On the Activation Function Dependence of the Spectral Bias of Neural Networks
Qingguo Hong, Jonathan W. Siegel, Qinyang Tan, Jinchao Xu
Neural Network Gradient Descent Generalization Performance Spectral Bias

June 29, 2022

Momentum Diminishes the Effect of Spectral Bias in Physics-Informed Neural Networks
Ghazal Farhani, Alexander Kazachek, Boyu Wang
Gradient Descent Physic Informed Neural Network Mixed Effect Neural Tangent Kernel Adam Optimizer Residual Momentum Spectral Bias

June 9, 2022

Overcoming the Spectral Bias of Neural Value Approximation
Ge Yang, Anurag Ajay, Pulkit Agrawal
Neural Tangent Kernel Policy Reinforcement Learning Kernel Regression Spectral Bias Neural Network Approximation Policy Deep Reinforcement Learning

June 6, 2022

Spectral Bias Outside the Training Set for Deep Networks in the Kernel Regime
Benjamin Bowman, Guido Montufar
Deep Network Neural Tangent Kernel Fisher Information Spectral Bias Training Set Kernel Regime Width Network Quantitative Bound

March 17, 2022

On the Spectral Bias of Convolutional Neural Tangent and Gaussian Process Kernels
Amnon Geifman, Meirav Galun, David Jacobs, Ronen Basri
Neural Network Neural Tangent Kernel Spectral Bias Hierarchical Feature Neural Tangent

February 27, 2022

The Spectral Bias of Polynomial Neural Networks
Moulik Choraria, Leello Tadesse Dadi, Grigorios Chrysos, Julien Mairal, Volkan Cevher
Neural Network Neural Tangent Kernel Spectral Bias Polynomial Neural Network Frequency Function

February 7, 2022

Failure and success of the spectral bias prediction for Kernel Ridge Regression: the case of low-dimensional data
Umberto M. Tomasini, Antonio Sclocchi, Matthieu Wyart
Case Relevance Generalization Error Kernel Ridge Regression Spectral Bias Software Failure Laplacian Kernel Low Dimensional Data

January 19, 2022

Overview frequency principle/spectral bias in deep learning
Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo
Deep Learning Spectral Bias Frequency Perspective Frequency Bias Frequency Function

November 6, 2021

Understanding Layer-wise Contributions in Deep Neural Networks through Spectral Analysis
Yatin Dandi, Arthur Jacot
Deep Neural Network Neural Tangent Kernel Layer Wise Spectral Bias Spectral Analysis Frequency Function

Spectral Bias

Papers

The Devil is in the Upsampling: Architectural Decisions Made Simpler for Denoising with Deep Image Prior

Understanding the Spectral Bias of Coordinate Based MLPs Via Training Dynamics

Investigations on convergence behaviour of Physics Informed Neural Networks across spectral ranges and derivative orders

Mitigating spectral bias for the multiscale operator learning

Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Study

On the Activation Function Dependence of the Spectral Bias of Neural Networks

Momentum Diminishes the Effect of Spectral Bias in Physics-Informed Neural Networks

Overcoming the Spectral Bias of Neural Value Approximation

Spectral Bias Outside the Training Set for Deep Networks in the Kernel Regime

On the Spectral Bias of Convolutional Neural Tangent and Gaussian Process Kernels

The Spectral Bias of Polynomial Neural Networks

Failure and success of the spectral bias prediction for Kernel Ridge Regression: the case of low-dimensional data

Overview frequency principle/spectral bias in deep learning

Understanding Layer-wise Contributions in Deep Neural Networks through Spectral Analysis