Functional Regularization

Functional regularization is a technique used to improve the generalization and stability of machine learning models by directly controlling the properties of the learned function, rather than solely relying on data fitting. Current research focuses on developing novel regularization schemes, including adversarial methods and those based on Fourier transforms or learned feature embeddings, often applied within generative models like Non-negative Matrix Factorization (NMF) or deep neural networks. These advancements address issues like spectral bias in neural networks and catastrophic forgetting in continual learning, leading to improved performance in various applications such as source separation, object counting, and reinforcement learning. The impact lies in enhancing the robustness and efficiency of machine learning algorithms across diverse domains.

Papers

March 26, 2024

Maximum Discrepancy Generative Regularization and Non-Negative Matrix Factorization for Single Channel Source Separation
Martin Ludvigsen, Markus Grasmair
Generative Model Adversarial Learning Source Separation Non Negative Matrix Factorization Channel Source Separation Functional Regularization

May 16, 2023

A Scalable Walsh-Hadamard Regularizer to Overcome the Low-degree Spectral Bias of Neural Networks
Ali Gorji, Andisheh Amrollahi, Andreas Krause
Neural Network Strong Generalization Gradient Descent Spectral Bias Standard Deep Functional Regularization

April 24, 2023

Adversarial Generative NMF for Single Channel Source Separation
Martin Ludvigsen, Markus Grasmair
Adversarial Learning Source Separation Non Negative Matrix Factorization Negative Matrix Factorization Channel Source Separation Sound Separation Functional Regularization

April 11, 2023

Density Map Distillation for Incremental Object Counting
Chenshen Wu, Joost van de Weijer
Incremental Learning Object Counting Map Distillation Functional Regularization

March 20, 2023

How (Implicit) Regularization of ReLU Neural Networks Characterizes the Learned Function -- Part II: the Multi-D Case of Two Layers with Random First Layer
Jakob Heiss, Josef Teichmann, Hanna Wutte
Neural Network Model Additive Model Generalized Additive Model Learned Function ReLU Neural Network Functional Regularization

October 21, 2022

Bridging the Gap Between Target Networks and Functional Regularization
Alexandre Piche, Valentin Thomas, Joseph Marino, Rafael Pardinas, Gian Maria Marconi, Christopher Pal, Mohammad Emtiyaz Khan
Deep Reinforcement Learning Implicit Regularization Convex Regularization Goal Node Functional Regularization

June 29, 2022

From Kernel Methods to Neural Networks: A Unifying Variational Formulation
Michael Unser
Neural Network Gradient Norm Kernel Method Variational Method Gradient Regularization Functional Regularization

March 28, 2022

A Framework of Meta Functional Learning for Regularising Knowledge Transfer
Pan Li, Yanwei Fu, Shaogang Gong
New Framework Knowledge Transfer Logic Regularization Functional Learning Functional Regularization Discriminant Function

December 30, 2021

Continually Learning Self-Supervised Representations with Projected Functional Regularization
Alex Gomez-Villa, Bartlomiej Twardowski, Lu Yu, Andrew D. Bagdanov, Joost van de Weijer
Self Supervised Self Supervised Method Continual Learning Performance Self Supervised Continual Learning Functional Regularization

December 6, 2021

Functional Regularization for Reinforcement Learning via Learned Fourier Features
Alexander C. Li, Deepak Pathak
Reinforcement Learning Deep Reinforcement Learning Reinforcement Learning Benchmark Fourier Feature Functional Regularization

Functional Regularization

Papers

Maximum Discrepancy Generative Regularization and Non-Negative Matrix Factorization for Single Channel Source Separation

A Scalable Walsh-Hadamard Regularizer to Overcome the Low-degree Spectral Bias of Neural Networks

Adversarial Generative NMF for Single Channel Source Separation

Density Map Distillation for Incremental Object Counting

How (Implicit) Regularization of ReLU Neural Networks Characterizes the Learned Function -- Part II: the Multi-D Case of Two Layers with Random First Layer

Bridging the Gap Between Target Networks and Functional Regularization

From Kernel Methods to Neural Networks: A Unifying Variational Formulation

A Framework of Meta Functional Learning for Regularising Knowledge Transfer

Continually Learning Self-Supervised Representations with Projected Functional Regularization

Functional Regularization for Reinforcement Learning via Learned Fourier Features