Deep Equilibrium Model

Deep Equilibrium Models (DEQs) are a class of implicit neural networks that represent the output as the fixed point of a single, potentially infinitely deep, layer, offering memory efficiency compared to traditional deep networks. Current research focuses on improving DEQ stability, convergence guarantees, and efficiency through various architectures and algorithms, including those incorporating physics-informed learning, and adapting DEQs for specific applications like inverse problems and differential equation solving. This approach holds significant promise for accelerating computationally intensive tasks in scientific computing and various engineering domains, particularly where memory limitations pose challenges for traditional deep learning methods.

Papers

November 1, 2024

Certified Robustness for Deep Equilibrium Models via Serialized Random Smoothing
Weizhi Gao, Zhichao Hou, Han Xu, Xiaorui Liu
Native Robustness Randomized Smoothing Implicit Model Deep Equilibrium Model Certified Robustness Smoothing Factor Lipschitz Bound

October 31, 2024

Quantum Deep Equilibrium Models
Philipp Schleich, Marta Skreta, Lasse B. Kristensen, Rodrigo A. Vargas-Hernández, Alán Aspuru-Guzik
Quantum Circuit Variational Quantum Algorithm Quantum Machine Learning Model Near Term Quantum Deep Equilibrium Model

October 30, 2024

Understanding Representation of Deep Equilibrium Models from Neural Collapse Perspective
Haixiang Sun, Ye Shi
Neural Topic Neural Collapse Balancing Strategy Implicit Neural Network Deep Equilibrium Model Meaning Representation

July 29, 2024

Constructing artificial life and materials scientists with accelerated AI using Deep AndersoNN
Saleem Abdul Fattah Ahmed Al Dajani, David Keyes
AI Model Material Science Artificial Life Deep Equilibrium Model Deep Equilibrium Cartesian Product Extrapolation

June 6, 2024

Infusing Self-Consistency into Density Functional Theory Hamiltonian Prediction via Deep Equilibrium Models
Zun Wang, Chang Liu, Nianlong Zou, He Zhang, Xinran Wei, Lin Huang, Lijun Wu, Bin Shao
Self Consistency Electronic Structure Hamiltonian Learning Deep Equilibrium Model Deep Equilibrium Density Functional Theory

June 5, 2024

Solving Differential Equations using Physics-Informed Deep Equilibrium Models
Bruno Machado Pacheco, Eduardo Camponogara
Physic Informed Neural Network Differential Equation Physic Based Deep Equilibrium Model Deep Equilibrium Value Problem

March 7, 2024

Solving Inverse Problems with Model Mismatch using Untrained Neural Networks within Model-based Architectures
Peimeng Guan, Naveed Iqbal, Mark A. Davenport, Mudassir Masood
Inverse Problem Model Architecture Unlearned Model Deep Equilibrium Model Model Based Deep Learning Untrained Neural Network Nonlinear Inverse Problem Model Mismatch

March 1, 2024

Subhomogeneous Deep Equilibrium Models
Pietro Sittoni, Francesco Tudisco
Deep Equilibrium Model Implicit Depth Homogeneous Diffusive Network

February 6, 2024

Positive concave deep equilibrium models
Mateusz Gabor, Tomasz Piotrowski, Renato L. G. Cavalcante
Deep Equilibrium Model Positive Emotional Language Deep Equilibrium Convex Analysis

February 5, 2024

Deep Equilibrium Models are Almost Equivalent to Not-so-deep Explicit Models for High-dimensional Gaussian Mixtures
Zenan Ling, Longbo Li, Zhanbo Feng, Yixuan Zhang, Feng Zhou, Robert C. Qiu, Zhenyu Liao
Random Matrix Shallow Network Implicit Neural Network Deep Equilibrium Model Explicit Modelling Dimensional Gaussian Mixture

January 5, 2024

MsDC-DEQ-Net: Deep Equilibrium Model (DEQ) with Multi-scale Dilated Convolution for Image Compressive Sensing (CS)
Youhao Yu, Richard M. Dansereau
Compressed Sensing Deep Equilibrium Model Sparse Signal Image Compressed Sensing Compressive Sensing Shrinkage Thresholding Algorithm

December 12, 2023

One-Step Diffusion Distillation via Deep Equilibrium Models
Zhengyang Geng, Ashwini Pokle, J. Zico Kolter
Diffusion Model Deep Equilibrium Model Deep Equilibrium Generative Transformer Diffusion Distillation

November 27, 2023

Accelerating Hierarchical Associative Memory: A Deep Equilibrium Approach
Cédric Goemaere, Johannes Deleu, Thomas Demeester
Deep Equilibrium Model Energy Minimization Deep Equilibrium Memory Retrieval

October 28, 2023

TorchDEQ: A Library for Deep Equilibrium Models
Zhengyang Geng, J. Zico Kolter
Easy to Use Library Implicit Model Deep Equilibrium Model Deep Learning Community

October 16, 2023

Wide Neural Networks as Gaussian Processes: Lessons from Deep Equilibrium Models
Tianxiang Gao, Xiaokai Huo, Hailiang Liu, Hongyang Gao
Neural Network Gaussian Process Critical Lesson Wide Neural Network Deep Equilibrium Model

October 14, 2023

Two Sides of The Same Coin: Bridging Deep Equilibrium Models and Neural ODEs via Homotopy Continuation
Shutong Ding, Tianyu Cui, Jingya Wang, Ye Shi
Neural ODE Neural Ordinary Differential Equation Deep Equilibrium Model Homotopy Class

July 16, 2023

Revisiting Implicit Models: Sparsity Trade-offs Capability in Weight-tied Model for Vision Tasks
Haobo Song, Soumajit Majumder, Tao Lin
Vision Task Implicit Model Deep Equilibrium Model Sparsity Tradeoff Weight Tied Model

June 10, 2023

An Optimization-based Deep Equilibrium Model for Hyperspectral Image Deconvolution with Convergence Guarantees
Alexandros Gkillas, Dimitris Ampeliotis, Kostas Berberidis
Neural Network Convergence Guarantee Splitting Algorithm Deep Equilibrium Model Spectral Correlation

June 2, 2023

A Closer Look at the Adversarial Robustness of Deep Equilibrium Models
Zonghan Yang, Tianyu Pang, Yang Liu
Native Robustness Adversarial Robustness Glance Annotation Adversarial Vulnerability Deep Equilibrium Model Effective Adversarial

May 29, 2023

Deep Equilibrium Models Meet Federated Learning
Alexandros Gkillas, Dimitris Ampeliotis, Kostas Berberidis
Fusion Weight Deep Equilibrium Model Conventional Deep

Deep Equilibrium Model

Papers

Certified Robustness for Deep Equilibrium Models via Serialized Random Smoothing

Quantum Deep Equilibrium Models

Understanding Representation of Deep Equilibrium Models from Neural Collapse Perspective

Constructing artificial life and materials scientists with accelerated AI using Deep AndersoNN

Infusing Self-Consistency into Density Functional Theory Hamiltonian Prediction via Deep Equilibrium Models

Solving Differential Equations using Physics-Informed Deep Equilibrium Models

Solving Inverse Problems with Model Mismatch using Untrained Neural Networks within Model-based Architectures

Subhomogeneous Deep Equilibrium Models

Positive concave deep equilibrium models

Deep Equilibrium Models are Almost Equivalent to Not-so-deep Explicit Models for High-dimensional Gaussian Mixtures

MsDC-DEQ-Net: Deep Equilibrium Model (DEQ) with Multi-scale Dilated Convolution for Image Compressive Sensing (CS)

One-Step Diffusion Distillation via Deep Equilibrium Models

Accelerating Hierarchical Associative Memory: A Deep Equilibrium Approach

TorchDEQ: A Library for Deep Equilibrium Models

Wide Neural Networks as Gaussian Processes: Lessons from Deep Equilibrium Models

Two Sides of The Same Coin: Bridging Deep Equilibrium Models and Neural ODEs via Homotopy Continuation

Revisiting Implicit Models: Sparsity Trade-offs Capability in Weight-tied Model for Vision Tasks

An Optimization-based Deep Equilibrium Model for Hyperspectral Image Deconvolution with Convergence Guarantees

A Closer Look at the Adversarial Robustness of Deep Equilibrium Models

Deep Equilibrium Models Meet Federated Learning