Mutual Information Estimation

Also known as MI

Follow this topic Ask about this topic

Momentum

3 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 17

All topics

Controlling Dependence in Implicit Generative Models via Spread Mutual Information
Oct 7, 2026Jiahao Yu, Song Liu, José Miguel Hernández-Lobato +1Generative Modeling Mutual Information Estimation
University of Oxford · Work completed at the University of Cambridge. · University of Bristol +1
A Statistical Inference Framework for PMI Estimation and SGNS Word Embeddings
Oct 4, 2026Zhongqi FanWord Embeddings Uncertainty Quantification
Beijing Normal-Hong Kong Baptist University
ALICE: In-context, Zero-shot, Mutual Information Estimation
Sep 28, 2026Giulio Franzese, Simone Rossi, Pietro MichiardiZero-Shot Learning In-Context Learning
EURECOM, Sophia Antipolis, France
When Does Text Inform? Benchmarking Information-Theoretic Metrics for Multimodal Time-Series Forecasting
Sep 12, 2026Emma Andrews, Gianmarco MengaldoForecasting Benchmarks Mutual Information Estimation
National University of Singapore
NMINE: Normalized Mutual Information Neural Estimation
Jul 30, 2026Petra Eerikinharju, Marko Tuononen, Ville HautamäkiMutual Information Estimation
University of Eastern Finland
Towards Diverse and Comprehensive Benchmarks for Mutual Information Estimation
Jul 3, 2026Alberto Foresti, Ivan Butakov, Alexander Tolmachev +3Copula Models Benchmark Design
Department of Data Science EURECOM 450 Route des Chappes, 06410 Biot, France · Applied AI Institute Presnenskaya naberezhnaya, 12, 121205 Moscow, Russia · Moscow Independent Research Institute of Artificial Intelligence Bolshaya Cheremushkinskaya Street, 20, Building 4, Premises 3/1, 117218 Moscow, Russia
DIPHINE: Diffusion-based $Φ$ -ID Neural Estimator
Jun 17, 2026Simon Pedro Galeano Munoz, Mustapha Bounoua, Giulio Franzese +2Mutual Information Estimation
KAUST, Saudi Arabia · EURECOM, France
Estimating Mutual Information between Time Series and Temporal Event Sequences Across Diverse Analysis Tasks
Jun 1, 2026Haoji Hu, Huaqing Mao, Yijun Lin +4Mutual Information Estimation
University of Minnesota - Twin Cities Minneapolis, MN, USA · University of Pittsburgh Pittsburgh, PA, USA · Inha University Incheon, Incheon, Republic of Korea
InfoAtlas: A Foundation Model for Zero-Shot Statistical Dependence Estimate
May 29, 2026Zhengyang Hu, Yanzhi Chen, Hanxiang Ren +5Amortized Inference Mutual Information Estimation
The University of Hong Kong · University of Cambridge · Microsoft +2
PromptNCE: Conditional Probabilities and PMI Using Only LLMs and Contrastive Estimation Prompts
May 20, 2026Juliette Woodrow, Chris PiechLLM Prompting Conditional Distribution Estimation
Department of Computer Science Stanford University
A Mutual Information Lower Bound for Multimodal Regression Active Learning
May 14, 2026Leonardo Ferreira Guilhoto, Akshat Kaushal, Paris PerdikarisActive Learning Mutual Information Estimation
Graduate Group on Applied Mathematics & Computational Science · Department of Computer and Information Science University of Pennsylvania Philadelphia, PA 19104 · Department of Mechanical Engineering and Applied Mechanics University of Pennsylvania Philadelphia, PA 19104
Expert Routing for Communication-Efficient MoE via Finite Expert Banks
May 6, 2026Mohammad Reza Deylam Salehi, Ali KhalesiMixture of Experts Information-Theoretic Generalization Bounds
IEEE Graduate Member · Nice, France · Institut Polytechnique des Sciences Avancées (IPSA) +1
Information Plane Analysis of Binary Neural Networks
May 5, 2026Maximilian Nothnagel, Bernhard C. GeigerNeural Network Generalization Binary Neural Networks
Graz University of Technology, Graz, Austria · Know Center Research GmbH, Graz, Austria · Signal Processing and Speech Communication Laboratory, Graz University of Technology, Graz, Austria +1
Non-Stationarity Breaks Permutation Surrogates in Multi-Agent Reinforcement Learning: Diagnosis and Remedies
Apr 26, 2026Nikolaos Al. Papadopoulos, Konstantinos E. PsannisMutual Information Estimation Multi-Agent Reinforcement Learning
Department of Applied Informatics, University of Macedonia, Egnatia 156, 54636 Thessaloniki, Greece
Not Just How Much, But Where: Decomposing Epistemic Uncertainty into Per-Class Contributions
Feb 24, 2026Mame Diarra Toure, David A. StephensBayesian Neural Networks Classification
Department of Mathematics and Statistics , McGill University
Schur-MI: Fast Mutual Information for Robotic Information Gathering
Feb 12, 2026Kalvik Jakkala, Jason O'Kane, Srinivas AkellaRobotic Data Collection Informative Path Planning
Texas A&M University · University of North Carolina at Charlotte
Let's Measure Information Step-by-Step: AI-Based Evaluation Beyond Vibes
Aug 7, 2025Zachary Robertson, Sanmi KoyejoAdversarial Robustness Reference-Free Evaluation
Department of Computer Science Stanford University

Oct 7, 2026, stat.MLJ/K move · Enter open · S save

Controlling Dependence in Implicit Generative Models via Spread Mutual Information

Jiahao Yu, Song Liu, José Miguel Hernández-Lobato, RuiKang OuYang

University of Oxford · Work completed at the University of Cambridge. · University of Bristol+1

Generative Modeling Mutual Information Estimation Mutual Information Maximization

Open paper AskPDF arXiv

Mutual information (MI) provides an objective for suppressing or encouraging statistical dependence in implicit generative models. However, direct MI evaluation is challenging in implicit models due to typically intractable densities. A remedy is estimating the generator gradient from the difference between conditional and marginal scores. This score difference can, in turn, be estimated by differentiating a log density ratio learned through classification. This construction nevertheless faces two difficulties: (i) singular distributions need not admit the required score functions, and (ii) poor overlap can hinder density-ratio estimation. We therefore introduce Spread Mutual Information (SMI), a weighted integral of MI across noise levels obtained by applying a common spreading kernel to the generated variable. Gaussian spreading yields smooth, strictly positive conditional and marginal densities, extending the gradient construction to distributions that may originally be singular. Across a variaty of experiments, SMI consistently achieves effective dependence control among MI-based methods and remains competitive with established task-specific approaches.

Momentum

Related topics

Latest papers 17