Variational Information Bottleneck

The Variational Information Bottleneck studies how to encode inputs into compact stochastic representations that discard irrelevant detail while retaining information needed for a target, often by optimizing a task loss plus a KL penalty to a prior. Common approaches use variational encoders to estimate this compression objective, sometimes paired with contrastive supervision, adversarial nuisance removal, or geometric constraints. Applications include robustness to noise and distribution shifts, novelty detection, privacy, disentanglement, and communication under bandwidth limits, with evaluations spanning image, speech, and control tasks. The central tension is choosing how much information to remove: stronger compression can improve generalization or invariance but may erase task cues, and the best regularization depends on the data and deployment setting.