Fundamental Limit

Fundamental limits research explores the inherent boundaries of performance in various machine learning and signal processing tasks. Current investigations focus on identifying these limits in diverse areas, including deep learning model training and generalization, prompt compression for large language models, and the robustness of biometric systems and AI-generated image detectors, often employing techniques like rate-distortion analysis and approximate message passing. Understanding these fundamental limits is crucial for developing more efficient and reliable algorithms, improving the trustworthiness of AI systems, and guiding the design of future technologies. This research directly impacts the development of more efficient and robust algorithms across numerous applications.

Papers

October 24, 2024

No Free Lunch: Fundamental Limits of Learning Non-Hallucinating Generative Models
Changlong Wu, Ananth Grama, Wojciech Szpankowski
Generative Model Inductive Bias Free Lunch Fundamental Limit

October 23, 2024

Multi-Draft Speculative Sampling: Canonical Architectures and Theoretical Limits
Ashish Khisti, M.Reza Ebrahimi, Hassan Dbouk, Arash Behboodi, Roland Memisevic, Christos Louizos
Fundamental Limit Draft Model Speculative Sampling Canonical Network

October 10, 2024

Theoretical limits of descending $\ell_0$ sparse-regression ML algorithms
Mihailo Stojnic
Maximum Likelihood Optimization Algorithm Deterministic Algorithm Sparse Linear Regression Fundamental Limit Sparse Optimization Random Duality Theory

August 12, 2024

Computability of Classification and Deep Learning: From Theoretical Limits to Practical Feasibility through Quantization
Holger Boche, Vit Fojtik, Adalbert Fono, Gitta Kutyniok
Deep Learning Classification Code Quantization Operator Deep Learning Application Fundamental Limit Banach Mazur Computable Computational Limit

July 22, 2024

Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models
Adway Girish, Alliot Nagle, Marco Bondaschi, Michael Gastpar, Ashok Vardhan Makkuva, Hyeji Kim
Large Language Model Fundamental Limit Black Box Language Model Prompt Compression

May 31, 2024

Generative Adversarial Networks in Ultrasound Imaging: Extending Field of View Beyond Conventional Limits
Matej Gazda, Samuel Kadoury, Jakub Gazda, Peter Drotar
Generative Adversarial Network Conditional Generative Adversarial Network Limited Field Best View Motion Tracking Transesophageal Echocardiography Fundamental Limit Ultrasound Imaging

May 24, 2024

Fundamental computational limits of weak learnability in high-dimensional multi-index models
Emanuele Troiani, Yatin Dandi, Leonardo Defilippis, Lenka Zdeborová, Bruno Loureiro, Florent Krzakala
Learning Rate Fundamental Limit Low Dimensional Structure High Dimensional Regime Weak Learning Multi Index Model

May 22, 2024

Matrix Denoising with Doubly Heteroscedastic Noise: Fundamental Limits and Optimal Spectral Methods
Yihan Zhang, Marco Mondelli
Random Matrix Spectral Method Fundamental Limit Spectral Estimator Heteroscedastic Noise

April 18, 2024

The graph alignment problem: fundamental limits and efficient algorithms
Luca Ganassali
Graph Drawing Practical Algorithm Alignment Problem Efficient Algorithm Fundamental Limit Graph Alignment

March 7, 2024

Fundamental limits of Non-Linear Low-Rank Matrix Estimation
Pierre Mergny, Justin Ko, Florent Krzakala, Lenka Zdeborová
Fisher Information Non Linear Gaussian Model Fundamental Limit Mean Squared Error Matrix Estimation

February 19, 2024

The Fundamental Limits of Least-Privilege Learning
Theresa Stadler, Bogdan Kulynych, Michael C. Gastpar, Nicolas Papernot, Carmela Troncoso
Machine Learning Scientific Inference Feature Representation Privileged Information Fundamental Limit

February 6, 2024

Adaptive Inference: Theoretical Limits and Unexplored Opportunities
Soheil Hor, Ying Qian, Mert Pilanci, Amin Arbabian
High Efficiency Fundamental Limit Adaptive Inference State Inference

November 6, 2023

The theoretical limits of biometry
Gaëlle Candel
Identity Verification Biometric System Fundamental Limit Biometric Signal Biometric Trait

October 20, 2023

Fundamental Limits of Membership Inference Attacks on Machine Learning Models
Eric Aubinais, Elisabeth Gassiat, Pablo Piantanida
Machine Learning Model Membership Inference Attack Sensitive Data Fundamental Limit Nonlinear Regression

October 11, 2023

Towards the Fundamental Limits of Knowledge Transfer over Finite Domains
Qingyue Zhao, Banghua Zhu
Knowledge Transfer Maximum Likelihood Kullback Leibler Divergence Fundamental Limit Probabilistic Classifier Continuous Domain Naive Adaptation

September 29, 2023

Robustness of AI-Image Detectors: Fundamental Limits and Practical Attacks
Mehrdad Saberi, Vinu Sankar Sadasivan, Keivan Rezaei, Aounon Kumar, Atoosa Chegini, Wenxiao Wang, Soheil Feizi
Native Robustness Watermarking Method Agnostic Watermarking Fundamental Limit Diffusion Purification

September 13, 2023

Fundamental Limits of Deep Learning-Based Binary Classifiers Trained with Hinge Loss
Tilahun M. Getu, Georges Kaddoum
Neural Network Deep Learning Binary Classifier Fundamental Limit Hinge Loss Performance Limit

September 1, 2023

Fast and Regret Optimal Best Arm Identification: Fundamental Limits and Low-Complexity Algorithms
Qining Zhang, Lei Ying
Multi Armed Bandit Reward Maximization Fundamental Limit

July 11, 2023

Fundamental limits of overparametrized shallow neural networks for supervised learning
Francesco Camilli, Daria Tieplova, Jean Barbier
Neural Network Supervised Learning Neural Network Model Information Theoretic Two Layer Neural Network Shallow Neural Network Fundamental Limit Optimal Generalization

June 9, 2023

How Sparse Can We Prune A Deep Network: A Fundamental Limit Viewpoint
Qiaozhe Zhang, Ruijie Zhang, Jun Sun, Yingzhuang Liu
Deep Network Many Sparse Network Pruning Fundamental Limit Pruning Ratio

Fundamental Limit

Papers

No Free Lunch: Fundamental Limits of Learning Non-Hallucinating Generative Models

Multi-Draft Speculative Sampling: Canonical Architectures and Theoretical Limits

Theoretical limits of descending $\ell_0$ sparse-regression ML algorithms

Computability of Classification and Deep Learning: From Theoretical Limits to Practical Feasibility through Quantization

Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models

Generative Adversarial Networks in Ultrasound Imaging: Extending Field of View Beyond Conventional Limits

Fundamental computational limits of weak learnability in high-dimensional multi-index models

Matrix Denoising with Doubly Heteroscedastic Noise: Fundamental Limits and Optimal Spectral Methods

The graph alignment problem: fundamental limits and efficient algorithms

Fundamental limits of Non-Linear Low-Rank Matrix Estimation

The Fundamental Limits of Least-Privilege Learning

Adaptive Inference: Theoretical Limits and Unexplored Opportunities

The theoretical limits of biometry

Fundamental Limits of Membership Inference Attacks on Machine Learning Models

Towards the Fundamental Limits of Knowledge Transfer over Finite Domains

Robustness of AI-Image Detectors: Fundamental Limits and Practical Attacks

Fundamental Limits of Deep Learning-Based Binary Classifiers Trained with Hinge Loss

Fast and Regret Optimal Best Arm Identification: Fundamental Limits and Low-Complexity Algorithms

Fundamental limits of overparametrized shallow neural networks for supervised learning

How Sparse Can We Prune A Deep Network: A Fundamental Limit Viewpoint