Rethinking the Information Bottleneck: Structured Decomposition under Label-Induced Partitions

Jingyao Zhang; Yuxuan Li; Lu Han; Ali Anaissi; Nguyen H. Tran

cs.LGOct 1, 2026

Rethinking the Information Bottleneck: Structured Decomposition under Label-Induced Partitions

Authors: Jingyao Zhang, Yuxuan Li, Lu Han, Ali Anaissi, Nguyen H. Tran

Organizations: School of Computer Science, The University of Sydney Sydney, NSW 2006, Australia

Abstract

Standard information bottleneck (IB) regularization constrains representations via a single scalar I(Z;X), implicitlytreating all information as homogeneous. However, a single global compression control couples label-relevant structurewith residual within-condition variation, rather than regulating their allocation independently, allowing nuisanceinformation to persist in learned representations. For example, in medical imaging applications, residual variation oftenstems from acquisition conditions, background factors, or subject-specific appearance. This issue becomes particularlypronounced in data-limited settings, where models tend to overfit such variation, hindering generalization. While existingregularization methods can stabilize training, control capacity, or shape representation geometry, they do not explicitlyseparate nuisance-like variation from task-supporting structure. To address this limitation, we revisit IB from a structuredperspective based on a label-induced partition, where condition-level structure and within-condition information playdistinct roles. This leads to a dual-bottleneck formulation: a standard KL term controls global information capacity, while aconditional KL term targets within-condition information. We show that the conditional KL admits an exact decompositioninto a within-condition information term and a prior-mismatch term, explaining its alignment with the design objective.With a simplex-structured conditional prior, the method provides controllable latent geometry and integrates seamlesslyinto existing pipelines. Experiments on classification and segmentation show the clearest gains in low-data classificationand consistent improvements across dense prediction benchmarks.

Figures & tables

Method	Global	Cond.	Geometry / prior	Main distinction
VIB / StdKL	$J_{\mathrm{std}}$	–	global prior	only global bottleneck
CondKL (cond-only)	–	$J_{\mathrm{cond}}$	conditional prior	no label-induced dual control
Center / prototype	–	implicit	empirical centers	feature-space geometry
SupCon	–	implicit	contrastive pairs	pairwise geometry
Generic regularizers	generic	–	none	orthogonal heuristics
DualKL (ours)	$J_{\mathrm{std}}$	$J_{\mathrm{cond}}$	simplex conditional prior	joint two-axis control

Table 1: Objective-level comparison.

J_{\mathrm{std}}

and

J_{\mathrm{cond}}

denote global and label-conditioned KL controls.

Figure 1: BloodMNIST feature heterogeneity. Points denote low-level statistics with sample separation on the x-axis and label separation on the y-axis. Lower-right points correspond to high variation but weak label relevance.

Panel of Figure 2 — Figure 2: Information-plane view of representation control. Panel (a) illustrates the decomposition $I(Z;X)=I(Z;C)+I(Z;X\mid C)$ . Panels (b)–(d) schematically compare global-only, conditional-only, and dual control directions.

Panel of Figure 3 — Figure 3: Visual illustration of complementary control in latent space. Global control mainly contracts overall spread, conditional control tightens within-condition structure, and dual control combines both effects in this illustrative example.

Figure 4: Engineering instantiation of the dual-bottleneck objective. The encoder produces a stochastic latent representation

Z

, the task head optimizes the supervised loss, and two KL surrogates implement global and condition-aware information control.

Panel of Figure 5 — Figure 5: BloodMNIST, 1% training data. No Reg (left two panels) versus DualKL (right two panels), shown in PCA and latent-statistics views. Full four-way comparisons are in Appendix B.3.2 .

Panel of Figure 6 — Figure 6: CHASEDB1. No Reg versus DualKL in PCA and latent-statistics views; full four-way comparisons are in Appendix B.3.2 .

Configuration	BAcc	Macro-F1
DualKL + simplex + shared fixed $\tau$	72.11 $\pm$ 1.42	69.97 $\pm$ 1.71
DualKL + random + shared fixed $\tau$	71.42 $\pm$ 1.53	69.11 $\pm$ 1.37
DualKL + empirical + shared fixed $\tau$	71.63 $\pm$ 1.72	69.94 $\pm$ 2.08
DualKL + simplex + classwise learned $\tau$	71.88 $\pm$ 2.62	69.65 $\pm$ 2.54

Table 2: BloodMNIST 1% training-data ablation results reported as mean

\pm

standard deviation.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Metric	No Reg	CondKL	StdKL	DualKL
Standard	BAcc	92.90 $\pm$ 0.34	94.36 $\pm$ 0.53	94.35 $\pm$ 0.17	94.52 $\pm$ 0.22
Standard	Macro-F1	93.14 $\pm$ 0.26	94.51 $\pm$ 0.50	94.53 $\pm$ 0.18	94.72 $\pm$ 0.19
1% Training Data	BAcc	63.49 $\pm$ 1.07	71.84 $\pm$ 1.78	68.11 $\pm$ 2.65	72.22 $\pm$ 1.38
1% Training Data	Macro-F1	61.43 $\pm$ 0.74	69.97 $\pm$ 1.83	66.25 $\pm$ 3.01	69.98 $\pm$ 1.85
4-shot	BAcc	45.31 $\pm$ 1.77	46.50 $\pm$ 3.09	47.10 $\pm$ 2.97	48.72 $\pm$ 2.59
4-shot	Macro-F1	43.19 $\pm$ 1.97	44.47 $\pm$ 2.90	45.02 $\pm$ 2.44	46.57 $\pm$ 2.58

Appendix

Table 3: BloodMNIST classification results reported as mean

\pm

standard deviation.

Setting	Metric	Strong WD	Head Dropout	Label Smoothing	Center Loss	DualKL
Standard	BAcc	93.02 $\pm$ 0.35	93.08 $\pm$ 0.63	93.05 $\pm$ 0.35	93.81 $\pm$ 0.46	94.70 $\pm$ 0.30
Standard	Macro-F1	93.11 $\pm$ 0.39	93.23 $\pm$ 0.64	93.33 $\pm$ 0.32	93.95 $\pm$ 0.43	94.88 $\pm$ 0.23
1% Training Data	BAcc	62.21 $\pm$ 1.26	63.32 $\pm$ 1.45	65.05 $\pm$ 0.97	68.85 $\pm$ 1.56	71.29 $\pm$ 1.57
1% Training Data	Macro-F1	60.40 $\pm$ 1.53	61.79 $\pm$ 1.78	62.85 $\pm$ 1.07	66.99 $\pm$ 1.23	68.98 $\pm$ 1.36
4-shot	BAcc	43.88 $\pm$ 1.36	45.22 $\pm$ 2.74	45.74 $\pm$ 3.46	46.80 $\pm$ 1.95	48.01 $\pm$ 2.95
4-shot	Macro-F1	42.08 $\pm$ 1.10	42.51 $\pm$ 2.60	43.75 $\pm$ 3.22	44.72 $\pm$ 2.14	45.91 $\pm$ 2.98

Appendix

Table 4: External baseline comparison on BloodMNIST. Results are reported as mean

\pm

standard deviation.

Setting	Metric	No Reg	CondKL	StdKL	DualKL
4% dataset	BAcc	56.01 $\pm$ 0.54	57.79 $\pm$ 0.31	57.95 $\pm$ 0.36	58.04 $\pm$ 0.19
4% dataset	Macro-F1	55.87 $\pm$ 0.52	57.43 $\pm$ 0.32	57.62 $\pm$ 0.38	57.76 $\pm$ 0.19
4-shot	BAcc	28.86 $\pm$ 0.92	30.04 $\pm$ 0.90	30.12 $\pm$ 1.02	30.18 $\pm$ 1.27
4-shot	Macro-F1	28.79 $\pm$ 0.80	29.45 $\pm$ 0.85	29.52 $\pm$ 0.96	29.63 $\pm$ 1.24

Appendix

Table 5: CIFAR-100 low-data classification results. We report mean

\pm

standard deviation over seeds.

Setting	Metric	Strong WD	Head Dropout	Label Smoothing	Center Loss	DualKL
4% dataset	BAcc	55.54 $\pm$ 0.45	55.50 $\pm$ 0.27	57.14 $\pm$ 0.12	57.98 $\pm$ 0.37	58.14 $\pm$ 0.14
4% dataset	Macro-F1	55.44 $\pm$ 0.51	55.35 $\pm$ 0.25	56.73 $\pm$ 0.19	57.80 $\pm$ 0.35	57.86 $\pm$ 0.17
4-shot	BAcc	28.94 $\pm$ 1.03	29.11 $\pm$ 1.07	29.62 $\pm$ 1.47	30.50 $\pm$ 1.19	30.51 $\pm$ 1.37
4-shot	Macro-F1	28.81 $\pm$ 0.89	29.02 $\pm$ 1.01	28.91 $\pm$ 1.46	29.96 $\pm$ 1.13	29.94 $\pm$ 1.35

Appendix

Table 6: External baseline comparison on CIFAR-100 low-data classification. Results are reported as mean

\pm

standard deviation over evaluated seeds.

Configuration	BAcc	Macro-F1
DualKL + simplex + shared fixed $\tau$	48.66 $\pm$ 1.99	46.59 $\pm$ 2.25
DualKL + random + shared fixed $\tau$	46.91 $\pm$ 2.40	44.51 $\pm$ 2.82
DualKL + empirical + shared fixed $\tau$	46.97 $\pm$ 2.83	45.19 $\pm$ 0.70
DualKL + simplex + classwise learned $\tau$	46.43 $\pm$ 1.39	44.55 $\pm$ 0.38

Appendix

Table 7: 4-shot BloodMNIST ablation results reported as mean

\pm

standard deviation.

Configuration	BAcc	Macro-F1
DualKL + simplex + shared fixed $\tau$	94.74 $\pm$ 0.28	94.86 $\pm$ 0.12
DualKL + random + shared fixed $\tau$	94.56 $\pm$ 0.24	94.63 $\pm$ 0.11
DualKL + empirical + shared fixed $\tau$	94.52 $\pm$ 0.16	94.62 $\pm$ 0.16
DualKL + simplex + classwise learned $\tau$	94.68 $\pm$ 0.35	94.79 $\pm$ 0.32

Appendix

Table 8: Full-training BloodMNIST ablation results reported as mean

\pm

standard deviation.

Dataset	Metric	No Reg	StdKL	CondKL	DualKL
DRIVE	IoU	0.5558 $\pm$ 0.0061	0.5823 $\pm$ 0.0037	0.5820 $\pm$ 0.0044	0.5826 $\pm$ 0.0053
DRIVE	Dice	0.7143 $\pm$ 0.0051	0.7359 $\pm$ 0.0030	0.7356 $\pm$ 0.0035	0.7362 $\pm$ 0.0042
STARE	IoU	0.5049 $\pm$ 0.0311	0.5302 $\pm$ 0.0121	0.5312 $\pm$ 0.0156	0.5318 $\pm$ 0.0145
STARE	Dice	0.6701 $\pm$ 0.0278	0.6926 $\pm$ 0.0104	0.6934 $\pm$ 0.0134	0.6939 $\pm$ 0.0125
CHASEDB1	IoU	0.5525 $\pm$ 0.0066	0.5724 $\pm$ 0.0043	0.5720 $\pm$ 0.0034	0.5739 $\pm$ 0.0044
CHASEDB1	Dice	0.7116 $\pm$ 0.0054	0.7280 $\pm$ 0.0035	0.7276 $\pm$ 0.0028	0.7291 $\pm$ 0.0036

Appendix

Table 9: Repeated-seed retinal vessel segmentation results reported as mean

\pm

standard deviation.

Setting	Metric	No Reg	StdKL	CondKL	DualKL
Full	IoU	0.8131 $\pm$ 0.0008	0.8149 $\pm$ 0.0040	0.8118 $\pm$ 0.0051	0.8158 $\pm$ 0.0017
Full	Dice	0.8957 $\pm$ 0.0005	0.8970 $\pm$ 0.0032	0.8949 $\pm$ 0.0031	0.8976 $\pm$ 0.0011
8-shot	IoU	0.6009 $\pm$ 0.0145	0.6039 $\pm$ 0.0141	0.5439 $\pm$ 0.0471	0.6106 $\pm$ 0.0111
8-shot	Dice	0.7489 $\pm$ 0.0116	0.7506 $\pm$ 0.0117	0.7015 $\pm$ 0.0394	0.7562 $\pm$ 0.0087
4-shot	IoU	0.5368 $\pm$ 0.0248	0.5460 $\pm$ 0.0221	0.5440 $\pm$ 0.0202	0.5478 $\pm$ 0.0220
4-shot	Dice	0.6958 $\pm$ 0.0218	0.7036 $\pm$ 0.0188	0.7019 $\pm$ 0.0175	0.7049 $\pm$ 0.0190

Appendix

Table 10: GLAS segmentation results in full-data and few-shot regimes, reported as mean

\pm

standard deviation.

Panel of Figure 7 — Figure 7: BloodMNIST (standard), PCA latent geometry. Representative panel numbers: No Reg (BAcc $0.926$ , Within $709.3$ , Ratio $0.4369$ ), DualKL (BAcc $0.952$ , Within $3.4$ , Ratio $0.0488$ ).

Figure 8: BloodMNIST (standard), latent-statistics geometry. Representative panel medians: No Reg (

0.236

2.271

0.722

6.1

) vs DualKL (

0.021

8.860

0.977

320.6

) for (inter-sample, label-sep, feat-label, sKEI).

Panel of Figure 9 — Figure 9: BloodMNIST (1% training data), PCA latent geometry. Representative panel numbers: No Reg (BAcc $0.618$ , Within $632.0$ , Ratio $1.9702$ ), DualKL (BAcc $0.737$ , Within $10.3$ , Ratio $0.3538$ ).

Figure 10: BloodMNIST (1% training data), latent-statistics geometry. Representative panel medians: No Reg (

0.390

0.487

0.450

0.3

) vs DualKL (

0.202

2.461

0.765

4.7

) for (inter-sample, label-sep, feat-label, sKEI).

Panel of Figure 11 — Figure 11: CHASEDB1, PCA latent geometry. Representative panel numbers: No Reg (IoU $0.557$ , Within $19.75$ , Ratio $0.4314$ ), DualKL (IoU $0.574$ , Within $18.38$ , Ratio $0.3082$ ).

Panel of Figure 12 — Figure 12: CHASEDB1, latent-statistics geometry. Panel medians show No Reg ( $0.343$ , $1.909$ , $0.623$ , $2.6$ ) vs KL ranges: inter-sample $0.195$ – $0.211$ , label-sep $2.892$ – $3.125$ , feat-label $0.783$ – $0.794$ , sKEI $7.3$ – $8.9$ .

Panel of Figure 13 — Figure 13: STARE, PCA latent geometry. Representative panel numbers: No Reg (IoU $0.506$ , Ratio $0.5400$ ), DualKL (IoU $0.538$ , Ratio $0.3840$ ).

Panel of Figure 14 — Figure 14: STARE, latent-statistics geometry. Panel medians show the No Reg to KL-regularized transition: inter-sample $0.329\rightarrow 0.231$ – $0.237$ , label-sep $0.904\rightarrow 2.748$ – $2.857$ , feat-label $0.562\rightarrow 0.723$ – $0.731$ , sKEI $1.3\rightarrow 4.4$ – $5.2$ .

Partition	Definition	$K$	Interpretation
Label	$C_{u}=Y_{u}$	2	Direct foreground/background pixel label.
Count	$C_{u}=\sum_{v\in\mathcal{N}(u)}Y_{v}$	10	Coarse local occupancy and boundary-proximity proxy.
Pattern	$C_{u}=\mathrm{bin}(Y_{\mathcal{N}(u)})$	512	Exact local shape, boundary, and thin-structure pattern.

Appendix

Table 11: Segmentation condition constructions. For each pixel

u

, the condition

C_{u}

is induced from the ground-truth mask in a

3\times 3

neighborhood

\mathcal{N}(u)

Dataset	Label $K=2$		Count $K=10$
	Top-1	$H_{\rm norm}$	Top-1	$H_{\rm norm}$
CHASEDB1	92.9	0.37	81.3	0.37
DRIVE	91.4	0.42	75.8	0.44
GLAS	50.3	1.00	46.5	0.45
STARE	92.7	0.38	80.6	0.38

Appendix

Table 12: Train-split concentration statistics for coarse segmentation conditions. Top-1 mass is reported in percentage;

H_{\mathrm{norm}}

denotes normalized entropy.

Dataset	Obs. $K$	Nontriv. $K$	BG mass	FG mass	Nontriv. mass
CHASEDB1	489	487	81.3	0.3	18.4
DRIVE	508	506	75.8	0.3	23.9
GLAS	269	267	46.5	46.4	7.2
STARE	480	478	80.6	0.4	19.0

Appendix

Table 13: Train-split statistics for the exact

3\times 3

pattern partition (

K=512

). Nontrivial patterns exclude pure-background and pure-foreground neighborhoods. Mass values are percentages.

Dataset	Label $K=2$	Count $K=10$	Pattern $K=512$
CHASEDB1	56.75 / 72.38	57.58 / 73.06	57.16 / 72.73
DRIVE	58.27 / 73.63	58.29 / 73.64	58.13 / 73.51
GLAS	81.83 / 89.89	81.93 / 89.93	82.25 / 90.18
STARE	52.77 / 69.06	52.85 / 69.13	53.47 / 69.67
Average	62.40 / 76.24	62.66 / 76.44	62.75 / 76.52

Appendix

Table 14: Partition-choice ablation for segmentation. We compare the direct pixel label (

K=2

3\times 3

foreground-count condition (

K=10

), and exact binary local-mask pattern (

K=512

). Results are validation IoU/Dice (%).

Panel of Figure 15 — Figure 15: Sensitivity to $(\beta_{\rm std},\beta_{\rm cond})$ on full-data BloodMNIST. Each cell reports the change relative to the grid mean.

Related method	How our method differs
Deep VIB [ 3 ]	Deep VIB provides a variational KL surrogate for a single global information bottleneck, mainly controlling $I(Z;X)$ . Our method keeps this global control through $J_{\rm std}$ , but adds a label-induced conditional control axis $J_{\rm cond}$ motivated by $I(Z;X)=I(Z;C)+I(Z;X\mid C)$ . Thus, our objective separates global capacity control from within-condition information control.
Multi-view conditional VIB-style objectives [ 7 ]	These objectives use conditional information terms to calibrate shared representations across multiple observed views and to handle view imbalance or missing views. Our conditioning variable is not a view but a supervised label-induced partition $C=\pi(Y)$ , and our goal is not multi-view calibration but supervised regularization of within-condition residual information.
Self-supervised compressive visual representations [ 8 ]	This line of work uses conditional information bottleneck-style objectives to compress self-supervised representations, typically between augmented views. Our method uses a supervised condition variable $C=\pi(Y)$ and combines conditional information control with an explicit global KL term. Hence, the method is a supervised dual-bottleneck regularizer rather than a self-supervised compression objective.
Conditional VAE / deep conditional generative models [ 11 ]	Conditional VAEs use condition-dependent latent variables or priors inside a generative likelihood or ELBO for structured output modelling. Our conditional prior is not used for generation; it is a variational information-control prior inside a supervised regularizer, where the conditional KL decomposes into $I(Z;X\mid C)$ plus condition-wise prior mismatch.
Semi-supervised deep generative models [ 12 ]	Semi-supervised generative models use class variables and latent variables to model data density and exploit unlabeled data. Our method does not model $p(x)$ or require unlabeled data; it regularizes a supervised task representation by jointly controlling global capacity and within-condition information.

Appendix

Table 15: Comparison with related bottleneck and conditional-prior formulations.

Explore similar work

Nov 4, 2025cs.LG

Geometry as a Missing Axis of Representation Quality: The Variational Geometric Information Bottleneck under Data Scarcity

We study latent geometry as an explicit component of representation quality in data-scarce learning. For an encoder (φ), we define (Q_{β,γ}(φ)=I(φ(X);Y)-β\mathcal C(φ)-γd_{\mathrm{int}}(φ)), combining task-relevant information with penalties for curvature and intrinsic latent dimension. Thus geometry becomes part of the bottleneck criterion, not only a post hoc diagnostic. Under smooth-manifold, loss-transfer, and estimator-concentration assumptions, we derive non-asymptotic low-label generalization bounds where intrinsic dimension and covering complexity enter explicitly. We characterize the information--geometry frontier and prove empirical-surrogate consistency. The analysis links encoder geometry to learning through latent covering numbers, loss-class entropy, and uniform deviation. We instantiate the theory as \texttt{V-GIB}, adding curvature and dimension penalties to variational bottleneck training. Real low-label benchmarks compare \texttt{V-GIB} with ERM, VIB, and ablations across (1%)--(20%) label fractions. Results show improved performance and reduced geometric complexity in several regimes, especially FashionMNIST and CIFAR-10, while confirming that no fixed regularizer is universally dominant.

Ronald Katende

Department of Mathematics, Kabale University, Katuna Road, Kikungiri Hill, Kabale, 317, Uganda

Discriminative Congruence Transform Information Bottleneck Generalization Bounds Representation Quality Missing Data Frontiers

Ask PDF

Jul 1, 2026cs.CV

Condensing Large-Scale Datasets Directly with Minimal Information Loss

Recent advancements in scaling dataset distillation rely heavily on decoupled information extraction pipelines, comprising SQUEEZE, RECOVER, and RELABEL stages. Despite their scalability to large-scale datasets, these methods suffer from prohibitive computational overhead and poor cross-architecture generalization. In this paper, we reveal the root cause of these bottlenecks: the implicit dual-compression process, from data to model and back to images, inherently induces severe information loss. Crucially, we empirically and theoretically demonstrate that this loss creates a distribution shift that fundamentally compromises the widely adopted RELABEL strategy, transforming the pre-trained model into an unreliable labeler that yields sub-optimal labels. To overcome these critical flaws, we propose CIM, a novel, metric-driven framework that abandons the flawed dual-compression paradigm. Instead, CIM explicitly quantifies and minimizes the information gap between the original and synthetic datasets. By directly aligning the data distributions, our approach ensures high-fidelity information condensation and inherently satisfies the prerequisites for effective relabeling. Extensive experiments demonstrate that CIM establishes a new state-of-the-art. Notably, it distills ImageNet-1K at an IPC=10 in merely 80 minutes on a single RTX-4090 GPU, achieving an unprecedented 48.7% Top-1 accuracy on ResNet-18 and significantly outperforming previous SOTA approaches, such as NRR-DD and DELT, by 2.6% and 2.9%, respectively. Our code is available at https://github.com/LINs-lab/CIM.

Xinyi Shang, Peng Sun, Bei Shi +2

University College London, United Kingdom · Zhejiang University, China · Westlake University, China +1

Diffusion-Based Dataset Distillation Dataset Distillation Imagenet High-Fidelity

Ask PDF

Jun 19, 2026cs.LG

Geometric and Information Compression of Representations in Deep Learning

Deep neural networks transform input data into latent representations that support a wide range of downstream tasks. These representations can be characterized along information-theoretic and geometric dimensions, but their relationship remains poorly understood. A central open question is whether low mutual information (MI) between inputs and representations necessarily implies geometrically compressed latent spaces and vice versa. We investigate this question using class-wise clustering as a measure of geometric compression and theoretically sound MI estimation in conditional entropy bottleneck (CEB) networks and continuous dropout networks. We evaluate the interplay between MI, geometric compression, and generalization on classification tasks under controlled noise injection schemes. Our findings show that low MI does not reliably correspond to geometric compression, and that the connection between the two is more nuanced than often assumed. Indeed, our experiments reveal a negative and nonlinear relationship that can reverse when varying training setup. Our results put forward a hypothesis that generalization acts as a potential confounder in this connection rather than being their direct consequence.

Linara Adilova, Henning Petzka, Asja Fischer +1

Research Center Trustworthy Data Science and Security, Dortmund · Faculty of Computer Science, Ruhr University Bochum · Signal Processing and Speech Communication Laboratory, Graz University of Technology +2

Information Bottleneck Mutual Information Neural Network Representations Discriminative Congruence Transform Deep Learning Dropout

Ask PDF