Information Theory

Information theory provides a mathematical framework for quantifying, storing, and communicating information, with applications spanning diverse fields. Current research focuses on leveraging information-theoretic principles—like mutual information and entropy—to improve machine learning algorithms, particularly in areas such as self-supervised learning, disentangled representation learning, and the design of robust and interpretable models (e.g., transformers). This involves developing new methods for estimating information-theoretic quantities in high-dimensional data and using these estimates to guide model design and analysis. The resulting advancements promise to enhance the efficiency, reliability, and explainability of AI systems across various applications, from healthcare diagnostics to material science.