Posterior Collapse

Posterior collapse, a phenomenon where the learned latent representations in variational autoencoders (VAEs) and related generative models become independent of the input data, is a significant challenge hindering effective representation learning. Current research focuses on understanding the causes of posterior collapse in various architectures, including VAEs, conditional VAEs, hierarchical VAEs, and latent diffusion models, and developing methods to mitigate it, such as contrastive regularization, KL annealing, and inverse Lipschitz constraints on decoder networks. Addressing posterior collapse is crucial for improving the performance and reliability of generative models and representation learning techniques across diverse applications, from image generation and time-series analysis to multi-agent interaction modeling.

Papers

August 20, 2024

A Grey-box Attack against Latent Diffusion Model-based Image Editing by Posterior Collapse
Zhongliang Guo, Lei Fang, Jingyu Lin, Yifei Qian, Shuai Zhao, Zeyu Wang, Junhao Dong, Cunjian Chen, Ognjen Arandjelović, Chun Pong Lau
Adversarial Attack Generative AI Latent Diffusion Model Generative Approach Gray Box Diffusion Based Image Editing Posterior Collapse

May 22, 2024

A Study of Posterior Stability for Time-Series Latent Diffusion
Yangming Li, Yixin Cheng, Mihaela van der Schaar
Diffusion Model Study Feature Latent Variable Latent Diffusion Posterior Collapse

October 24, 2023

Learning Dynamics in Linear VAE: Posterior Collapse Threshold, Superfluous Latent Space Pitfalls, and Speedup with KL Annealing
Yuma Ichikawa, Koji Hukushima
Latent Space Variational Autoencoders Learning Dynamic Variational Posterior Annealing Machine Posterior Collapse Entangled Representation

September 19, 2023

Posterior Contraction Rates for Mat\'ern Gaussian Processes on Riemannian Manifolds
Paul Rosa, Viacheslav Borovitskiy, Alexander Terenin, Judith Rousseau
Gaussian Process Riemannian Manifold Posterior Collapse Intrinsic Method Mat\'ern Correlation

September 14, 2023

Dataset Size Dependence of Rate-Distortion Curve and Threshold of Posterior Collapse in Linear VAE
Yuma Ichikawa, Koji Hukushima
Variational Autoencoder Rate Distortion Canonical Threshold Variational Posterior Dataset Size Posterior Collapse Hyperparameter Importance

September 6, 2023

CR-VAE: Contrastive Regularization on Variational Autoencoders for Preventing Posterior Collapse
Fotios Lygerakis, Elmar Rueckert
Variational Autoencoder Latent Representation Variational Autoencoders Contrastive Regularization Posterior Collapse

June 8, 2023

Beyond Vanilla Variational Autoencoders: Detecting Posterior Collapse in Conditional and Hierarchical Variational Autoencoders
Hien Dang, Tho Tran, Tan Nguyen, Nhat Ho
Variational Autoencoder Conditional Reasoning Hierarchical Variational Variational Posterior Posterior Collapse

April 25, 2023

Controlling Posterior Collapse by an Inverse Lipschitz Constraint on the Decoder Network
Yuri Kinoshita, Kenta Oono, Kenji Fukumizu, Yuichi Yoshida, Shin-ichi Maeda
Variational Autoencoders VAE Model Lipschitz Operator Posterior Collapse Decoder Network Lipschitz Constraint

February 20, 2023

Discouraging posterior collapse in hierarchical Variational Autoencoders using context
Anna Kuzina, Jakub M. Tomczak
Context Information Hierarchical Variational Posterior Collapse Likelihood Based Generative Hierarchical VAE

January 2, 2023

Posterior Collapse and Latent Variable Non-identifiability
Yixin Wang, David M. Blei, John P. Cunningham
Variational Autoencoder Variational Autoencoders Variational AutoEncoder Posterior Collapse

November 1, 2022

Improving Variational Autoencoders with Density Gap-based Regularization
Jianfei Zhang, Jun Bai, Chenghua Lin, Yanmeng Wang, Wenge Rong
Latent Representation Variational Autoencoders Posterior Collapse

July 19, 2022

Forget-me-not! Contrastive Critics for Mitigating Posterior Collapse
Sachit Menon, David Blei, Carl Vondrick
Latent Representation Variational Autoencoders Anti Forgetting Contrastive Example Posterior Collapse Self Supervised Contrastive Representation

June 23, 2022

Epidemic Control Modeling using Parsimonious Models and Markov Decision Processes
Edilson F. Arruda, Tarun Sharma, Rodrigo e A. Alexandre, Sinnu Susan Thomas
Markov Decision Process Optimal Policy Uncertain Reasoning Epidemic Model Parsimonious Machine Learning Posterior Collapse

May 9, 2022

Posterior Collapse of a Linear Latent Variable Model
Zihao Wang, Liu Ziyin
Variational Autoencoders Neural Collapse Bayesian Deep Learning Dimensional Collapse Posterior Collapse

February 9, 2022

Covariate-informed Representation Learning to Prevent Posterior Collapse of iVAE
Young-geun Kim, Ying Liu, Xuexin Wei
Latent Representation Variational AutoEncoder Posterior Collapse Posterior Network Covariate Representation

December 1, 2021

Exploring Social Posterior Collapse in Variational Autoencoder for Interaction Modeling
Chen Tang, Wei Zhan, Masayoshi Tomizuka
Multi Agent Variational Autoencoder Interaction Modeling Multi Agent Trajectory Prediction Posterior Collapse Multi Agent Behavior

November 23, 2021

Depth induces scale-averaging in overparameterized linear Bayesian neural networks
Jacob A. Zavatone-Veth, Cengiz Pehlevan
Bayesian Neural Network Large Depth Deep Linear Posterior Collapse

Posterior Collapse

Papers

A Grey-box Attack against Latent Diffusion Model-based Image Editing by Posterior Collapse

A Study of Posterior Stability for Time-Series Latent Diffusion

Learning Dynamics in Linear VAE: Posterior Collapse Threshold, Superfluous Latent Space Pitfalls, and Speedup with KL Annealing

Posterior Contraction Rates for Mat\'ern Gaussian Processes on Riemannian Manifolds

Dataset Size Dependence of Rate-Distortion Curve and Threshold of Posterior Collapse in Linear VAE

CR-VAE: Contrastive Regularization on Variational Autoencoders for Preventing Posterior Collapse

Beyond Vanilla Variational Autoencoders: Detecting Posterior Collapse in Conditional and Hierarchical Variational Autoencoders

Controlling Posterior Collapse by an Inverse Lipschitz Constraint on the Decoder Network

Discouraging posterior collapse in hierarchical Variational Autoencoders using context

Posterior Collapse and Latent Variable Non-identifiability

Improving Variational Autoencoders with Density Gap-based Regularization

Forget-me-not! Contrastive Critics for Mitigating Posterior Collapse

Epidemic Control Modeling using Parsimonious Models and Markov Decision Processes

Posterior Collapse of a Linear Latent Variable Model

Covariate-informed Representation Learning to Prevent Posterior Collapse of iVAE

Exploring Social Posterior Collapse in Variational Autoencoder for Interaction Modeling

Depth induces scale-averaging in overparameterized linear Bayesian neural networks