Heavy Tailed

Heavy-tailed distributions, characterized by infrequent but extreme values, pose significant challenges for many machine learning and statistical methods designed for lighter-tailed data. Current research focuses on developing robust algorithms and models, such as modified UCB algorithms, Bayesian CART models, and differentially private stochastic gradient descent variants, that can effectively handle heavy-tailed data in various settings, including multi-armed bandits, regression, and reinforcement learning. This work is crucial because heavy-tailed data frequently arises in real-world applications (e.g., finance, insurance, and network traffic), and ignoring this characteristic can lead to inaccurate or unreliable results. Improved understanding and handling of heavy tails will enhance the robustness and applicability of machine learning across diverse fields.

Papers

October 31, 2024

QuACK: A Multipurpose Queuing Algorithm for Cooperative $k$-Armed Bandits
Benjamin Howson, Sarah Filippi, Ciara Pike-Burke
Long Term Cooperation Single Agent Heavy Tailed Decentralized Stochastic

October 29, 2024

Robust training of implicit generative models for multivariate and heavy-tailed distributions with an invariant statistical loss
José Manuel de Frutos, Manuel A. Vázquez, Pablo Olmos, Joaquín Míguez
Generative Adversarial Network Heavy Tailed Multivariate Approach Implicit Generative Invariant Loss

October 18, 2024

Heavy-Tailed Diffusion Models
Kushagra Pandey, Jaideep Pathak, Yilun Xu, Stephan Mandt, Michael Pritchard, Arash Vahdat, Morteza Mardani
Heavy Tailed

October 17, 2024

From Gradient Clipping to Normalization for Heavy Tailed SGD
Florian Hübler, Ilyas Fatkhullin, Niao He
Normalization Dictionary Heavy Tailed Heavy Tailed Noise Gradient Clipping Almost Sure Convergence

October 16, 2024

Model Balancing Helps Low-data Training and Fine-tuning
Zihang Liu, Yuanzhe Hu, Tianyu Pang, Yefan Zhou, Pu Ren, Yaoqing Yang
Training Data Fine Tuning Pre Trained Model Layer Wise Balancing Strategy Heavy Tailed

October 14, 2024

AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models
Haiquan Lu, Yefan Zhou, Shiwei Liu, Zhangyang Wang, Michael W. Mahoney, Yaoqing Yang
Large Language Model Large Relevance Improvement Heavy Tailed Layer Pruning Layerwise Sparsity

October 4, 2024

uniINF: Best-of-Both-Worlds Algorithm for Parameter-Free Heavy-Tailed MABs
Yu Chen, Jiatai Huang, Yan Dai, Longbo Huang
Optimal Regret Heavy Tailed Long Tailed Data Adversarial Environment Logarithmic Regret Best of Both World Algorithm

September 3, 2024

Bayesian CART models for aggregate claim modeling
Yaojun Zhang, Lanpeng Ji, Georgios Aivaliotis, Charles C. Taylor
Heavy Tailed Joint Modeling Bayesian Model Frequency Severity Modeling

September 1, 2024

Benign Overfitting under Learning Rate Conditions for $α$ Sub-exponential Input
Kota Okudo, Kei Kobayashi
Heavy Tailed Benign Overfitting Margin Classifier Gaussian Input

August 19, 2024

Differential Private Stochastic Optimization with Heavy-tailed Data: Towards Optimal Rates
Puning Zhao, Jiafei Wu, Zhe Liu, Chong Wang, Rongfei Fan, Qingming Li
Differential Privacy Convex Optimization Heavy Tailed Stochastic Convex Optimization Optimal Rate Tail Risk

July 19, 2024

Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification
Thomas Kwa, Drake Thomas, Adrià Garriga-Alonso
Reinforcement Learning Optimal Policy Reward Model Reinforcement Learning From Human Feedback Heavy Tailed KL Divergence Heavy Tailed Reward Extinction Risk

June 22, 2024

Flexible Tails for Normalizing Flows
Tennessee Hickling, Dennis Prangle
Normalizing Flow Heavy Tailed Flexible Tail

June 9, 2024

Data-Driven Upper Confidence Bounds with Near-Optimal Regret for Heavy-Tailed Bandits
Ambrus Tamás, Szabolcs Szentpéteri, Balázs Csanád Csáji
Optimal Regret Heavy Tailed Confidence Bound Stochastic Multi Armed Bandit

June 5, 2024

Posterior and variational inference for deep neural networks with heavy-tailed weights
Ismaël Castillo, Paul Egels
Deep Neural Network Variational Inference Heavy Tailed Prior Distribution Posterior Mode Bayesian Deep

June 4, 2024

Private Stochastic Convex Optimization with Heavy Tails: Near-Optimality from Simple Reductions
Hilal Asi, Daogao Liu, Kevin Tian
Near Optimality Constructive Reduction Private Algorithm Heavy Tailed Private Gradient Heavy Tail Approximate Differential Privacy Lipschitz Continuity Assumption

May 27, 2024

Clip Body and Tail Separately: High Probability Guarantees for DPSGD with Heavy Tails
Haichao Sha, Yang Cao, Yong Liu, Yuncheng Wu, Ruixuan Liu, Hong Chen
Single CLIP Gradient Norm Heavy Tailed Private Stochastic Gradient Descent Thin Tail Clipped Stochastic Gradient Descent Heavy Tail High Probability Guarantee

May 22, 2024

Learning heavy-tailed distributions with Wasserstein-proximal-regularized $\alpha$-divergences
Ziyu Chen, Hyemin Gu, Markos A. Katsoulakis, Luc Rey-Bellet, Wei Zhu
Heavy Tailed Stable Learning Beta Divergence Wasserstein 2

February 2, 2024

December 2, 2023

$t^3$-Variational Autoencoder: Learning Heavy-tailed Data with Student's t and Power Divergence
Juno Kim, Jaehyuk Kwon, Mincheol Cho, Hyunjong Lee, Joong-Ho Won
Variational Autoencoder Student Friendly Knowledge Lung VAE 3D Convolutional Heavy Tailed Inverse Divergence

Heavy Tailed

Papers

QuACK: A Multipurpose Queuing Algorithm for Cooperative $k$-Armed Bandits

Robust training of implicit generative models for multivariate and heavy-tailed distributions with an invariant statistical loss

Heavy-Tailed Diffusion Models

From Gradient Clipping to Normalization for Heavy Tailed SGD

Model Balancing Helps Low-data Training and Fine-tuning

AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models

uniINF: Best-of-Both-Worlds Algorithm for Parameter-Free Heavy-Tailed MABs

Bayesian CART models for aggregate claim modeling

Benign Overfitting under Learning Rate Conditions for $α$ Sub-exponential Input

Differential Private Stochastic Optimization with Heavy-tailed Data: Towards Optimal Rates

Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification

Flexible Tails for Normalizing Flows

Data-Driven Upper Confidence Bounds with Near-Optimal Regret for Heavy-Tailed Bandits

Posterior and variational inference for deep neural networks with heavy-tailed weights

Private Stochastic Convex Optimization with Heavy Tails: Near-Optimality from Simple Reductions

Clip Body and Tail Separately: High Probability Guarantees for DPSGD with Heavy Tails

Learning heavy-tailed distributions with Wasserstein-proximal-regularized $\alpha$-divergences

Emergence of heavy tails in homogenized stochastic gradient descent

Improved Quantization Strategies for Managing Heavy-tailed Gradients in Distributed Learning

$t^3$-Variational Autoencoder: Learning Heavy-tailed Data with Student's t and Power Divergence