Mutual Information

Also known as MI

Mutual information quantifies how much observing one variable reduces uncertainty about another, making it a common language for dependence, prediction, and information carried by learned representations. Work ranges from Shannon and conditional mutual information bounds to information bottleneck theory, contrastive and neural estimators, and Rényi or Sibson variants for robust and task-specific information gain. These tools help shape self-supervised representations, select features or experiments, assess multimodal access and communication, and bound generalization in statistical learning. A central tension is that finite-sample estimates can saturate or depend strongly on modeling assumptions, while high mutual information or apparent compression does not by itself guarantee useful representations or better generalization.

Latest papers

26 papers4 papers in the last four weeks, up 33% on the four weeks before. 0.0% of all new papers.

Correlation-Aware Exploration for Long-Trace Reasoning
Oct 26, 2026Junxing Hu, Zicheng Zhang, Zhiqian Zhang +4RL for Language Model Reasoning RL for Language Models
Microsoft
An Accuracy-Information Tradeoff for Loss-Difference Conditional Mutual Information
Oct 6, 2026Hazar YuekselStatistical Learning Theory Information-Theoretic Generalization Bounds
The optimal information complexity of VC learning
Oct 6, 2026Steve Hanneke, Juexiao WangStatistical Learning Theory Information-Theoretic Generalization Bounds
Department of Computer Science, Purdue University
Conformal Prediction Sets Quantify Information Gain: A Theoretical Perspective
Oct 6, 2026Kevin Zhang, Stephen BatesSet Prediction Conformal Prediction
EECS, MIT, Cambridge MA, USA.
On the Cardinality of Optimal Representations in the Binary-Source Information Bottleneck
Oct 5, 2026Dier Tang, Jun ChenInformation Bottleneck Mutual Information
Department of Mathematics, The University of Hong Kong · Electrical and Computer Engineering, McMaster University
2D GauSS-MI: Efficient Active Scene Reconstruction with Balanced Visual and Geometric Quality
Sep 18, 2026Yuhan Xie, Jia Pan3D Reconstruction Active View Selection
the School of Computing and Data Science, the University of Hong Kong, Hong Kong SAR, China
A Mathematical Theory of Pragmatic Information
Sep 10, 2026Kai Niu, Ping ZhangRate-Distortion Theory Value of Information
Key Laboratory of Universal Wireless Communications, Ministry of Education · State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, 100876, China
A Common Measure of Communication for Speech Brain-Computer Interfaces
Sep 2, 2026Dulhan Jayalath, Benjamin Ballyk, Oiwi Parker JonesBrain-Computer Interfaces Automated Evaluation
Neural Processing Lab (PNPL ), University of Oxford
Mutual information and sensitivity analysis for feature selection in customer targeting: a comparative study
Aug 20, 2026Nestor Barraza, Sergio Moro, Marcelo Ferreyra +1Feature Selection Mutual Information
Universidad Nacional de Tres de Febrero, Caseros, Argentina · Instituto Universitário de Lisboa (ISCTE-IUL), ISTAR-IUL, Lisboa, Portugal · ALGORITMI Research Centre, University of Minho, Guimarães, Portugal +2
Sparse Mutual Information Graph Averaging for Improving Random Indexing Embeddings
Aug 6, 2026Sriram Loganathan, Gokul Anand, Aung Bo Bo +2Word Embeddings Distributional Semantics
Department of Computer Science San Jose State University San Jose, CA, USA
Scoring Rules! Statistical and Strategic Alignment for Text Evaluation Metrics
Aug 2, 2026Shengwei Xu, Yuxuan Lu, Yifan Wu +2Language Model Generation Evaluation Automated Evaluation
University of Michigan · Peking University · Microsoft Research +1
One-step lowest-variance selection in a Gaussian random-field model motivated by masked diffusion: Total correlation and a square root collision threshold
Jul 20, 2026Linjun LiMasked Diffusion Models Parallel Decoding
Department of Mathematics, University of Pennsylvania, Philadelphia, PA.
Capacity and Redundancy Trade-offs in Multi-Task Learning
Jul 17, 2026Asif KhanMulti-Task Learning Multi-Task Optimization
1Harvard Medical School, Boston, USA
Information Limits and Attractor Dynamics in Economies of Frontier LLM Agents: A Pre-Registered Test
Jul 7, 2026Cheng QianMulti-Agent LLM Systems Value of Information
Independent Researcher
Geometric and Information Compression of Representations in Deep Learning
Jun 19, 2026Linara Adilova, Henning Petzka, Asja Fischer +1Neural Representation Geometry Embedding Compression
Research Center Trustworthy Data Science and Security, Dortmund · Faculty of Computer Science, Ruhr University Bochum · Signal Processing and Speech Communication Laboratory, Graz University of Technology +2
From Generic Correlation to Input-Specific Credit in On-Policy Self Distillation
May 12, 2026Guobin Shen, Lei Huang, Xiang Cheng +4Reinforcement Learning On-Policy Self-Distillation
Xiaohongshu Inc. · Institute of Automation, Chinese Academy of Sciences
Neural Information Causality: A Random-Access Capacity Law for Representation Learning
May 10, 2026Jeongho Bang, Marcin PawłowskiRepresentation Learning Actual Causality
Institute for Convergence Research and Education in Advanced Technology, Yonsei University, Seoul 03722, Republic of Korea · Department of Quantum Information, Yonsei University, Incheon 21983, Republic of Korea · International Centre for Theory of Quantum Technologies, University of Gdańsk, 80-308 Gdańsk, Poland +1
Information Plane Analysis of Binary Neural Networks
May 5, 2026Maximilian Nothnagel, Bernhard C. GeigerNeural Network Generalization Binary Neural Networks
Graz University of Technology, Graz, Austria · Know Center Research GmbH, Graz, Austria · Signal Processing and Speech Communication Laboratory, Graz University of Technology, Graz, Austria +1
Self-Supervised Representation Learning via Hyperspherical Density Shaping
Apr 27, 2026Esteban Rodríguez-Betancourt, Edgar Casasola-MurilloMulti-View Learning Hyperspherical Representation Learning
Posgrado en Computación e Informática · Universidad de Costa Rica · Escuela de Ciencias de la Computación
Efficient Process Reward Modeling via Contrastive Mutual Information
Apr 12, 2026Nakyung Lee, Sangwoo Hong, Jungwoo LeeProcess Reward Models LLM Reasoning
Department of Electrical and Computer Engineering, Seoul National University · Department of Computer Science and Engineering, Konkuk University
Forecastability as an Information-Theoretic Limit on Prediction
Mar 28, 2026Peter Maurice CattTime Series Forecasting Information-Theoretic Lower Bounds
Independent Researcher, Auckland, New Zealand
Maximin Robust Bayesian Experimental Design
Mar 14, 2026Hany Abdulsamad, Sahel Iqbal, Christian A. Naesseth +2Bayesian Experimental Design Mutual Information
UvA–Bosch Delta Lab University of Amsterdam · Department of Statistics University of Oxford · School of Mathematics University of Edinburgh +1
Modality Collapse as Mismatched Decoding: Information-Theoretic Limits of Multimodal LLMs
Feb 26, 2026Jayadev BillaMultimodal Large Language Models Mutual Information
Calculating Mutual Information between a Reward Maximizer and its Environment
Feb 13, 2026Alfred Harwood, Jose Faustino, Alex AltairMarkov Decision Processes Reinforcement Learning
1Dovetail Research · University of Sao Paulo
Theoretical Refinement of CLIP by Utilizing Linear Structure of Optimal Similarity
Oct 17, 2025Naoki Yoshida, Satoshi Hayakawa, Yuhta Takida +3Cross-Modal Learning Kernel Mean Embeddings
The University of Tokyo · Sony Group Corporation · Sony AI
Mutual Information Surprise: Rethinking Unexpectedness in Autonomous Systems
Aug 24, 2025Yinsong Wang, Quan Zeng, Xiao Liu +1Adaptive Sampling Adaptive Learning
H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta & 30332, USA.

Oct 26, 2026J/K move · Enter open · S save

Correlation-Aware Exploration for Long-Trace Reasoning

Junxing Hu, Zicheng Zhang, Zhiqian Zhang, Ai Han+3

Microsoft

RL for Language Model Reasoning RL for Language Models Group Relative Policy Optimization Mutual Information

Open paper AskPDF arXiv

Reasoning models trained with Group Relative Policy Optimization (GRPO) now produce chains of thought thousands of tokens long, and recent studies reveal that such training is prone to policy collapse. Entropy regularization is the standard remedy, but it works by pushing probability mass away from the few tokens that carry it and toward the vast low-probability tail of the vocabulary, and it applies this force once for every token of the reasoning trace. This puts the bonus in direct conflict with the entropy decay that convergence requires, and makes its coefficient difficult to set in the long-trace regime. To resolve this tension, we propose Info-GRPO, an information-theoretic framework that cultivates correlation between the policy and a latent prior. Rather than raising entropy everywhere, Info-GRPO builds on the mutual information between a latent variable and the trajectory it induces: augmenting prompts with latent seeds during training lets the model explore diverse policies correlated with the prior, while the seed-conditioned entropy is guided toward convergence. Diversity is therefore carried across whole reasoning traces rather than by per-token randomness, and exploration no longer fights entropy reduction. Experiments show that Info-GRPO outperforms vanilla GRPO and entropy-regularized GRPO across diverse reasoning benchmarks. On AIME24, it improves

Avg@8

over GRPO by 3.75%, 1.66%, 4.16%, and 4.17% on four backbones, and it reaches the same majority-vote accuracy with

8\times

fewer samples. Code and models will be publicly released.