What Should an Information Bottleneck Encoder Learn? Separating Representation Learning from Rate Control

Hassan Tavakoli; Bella Bose

cs.LGOct 26, 2026

What Should an Information Bottleneck Encoder Learn? Separating Representation Learning from Rate Control

Authors: Hassan Tavakoli, Bella Bose

Organizations: School of EECS, Oregon State University, Corvallis, OR 97331, USA

Abstract

The information bottleneck (IB) trades compression of an input $X$ against prediction of a target $Y$ through a multiplier $β=1/λ$ , yet each operating point is usually served by training, or fully conditioning, a new encoder. For jointly Gaussian data, we show that the IB solution factorizes as $A(β)=D(β)W$ : the information-bearing directions $W$ do not depend on $β$ , and $β$ acts only through scalar gains. We ask whether this separation of representation from rate control survives beyond Gaussian models, treating the budget as a given operating parameter rather than a quantity to be optimized. For a budget-free representation $F(X)$ , the separation gap is controlled by the sufficiency deficit $ε_F=I(X;Y\mid F(X))$ : We show there is at least $ε_F$ in the worst case for finite inputs, and it is bounded uniformly in $λ$ by a function of $ε_F$ for finite label alphabets. The same certificate extends to a new target distribution and to pooled multi-domain data, where a conditional-alignment term appears. On MNIST, SVHN, CIFAR-10 and STL-10 (10 seeds, eight prespecified budgets), a frozen CNN representation with one compressor--decoder pair per budget reduces estimated training multiply--accumulate operations (MACs) by $7.7\times$ and wall-clock time by about $2\times$ relative to end-to-end VIBs, and attains higher budget-averaged accuracy ( $65.6\%$ versus $57.3\%$ per domain; $67.1\%$ versus $57.1\%$ pooled), with the advantage concentrated at strong compression, where the VIBs collapse. At weak compression ( $λ\le0.036$ ), VIB has higher cross-entropy and ECE; its surrogate Lagrangian is higher per domain, indistinguishable from the pooled VIB in the macro average, and lower on the pooled test set.

Figures & tables

Panel of Figure 1 — Figure 1: Representation–rate separation. $F$ is trained once without reference to the budget and characterized by the scalar $\epsilon_{F}\triangleq I(X;Y|F(X))$ ; $C$ and $G$ depend on the budget, $\beta$ .

Method	Role of $\beta$	What depends on $\beta$
VIB, nonlinear IB [ 4 , 8 ]	given	A whole Encoder/Decoder networks (one per $\beta$ )
Rate-condition [ 5 , 6 , 7 ]	given	A whole network
Flexible VIB (FVIB) [ 9 ]	given	closed-form rescaling (Taylor approx.)
Opportunistic IB (OIB) [ 10 ]	given	Gaussian-IB linear loadings
$\beta$ -free IB [ 11 ]	–	$\beta$ -free
Ours	given	New Separation method: only $(C,G)$ per $\beta$ and $F$ is $\beta$ -free

Table 1: Positioning. We treat

\beta

as a given operating parameter and ask what must depend on it.

Panel of Figure 2 — Figure 2: Stage 1: $F_{\theta}$ and an affine head $\ell_{\vartheta}$ are trained with cross-entropy only (no rate term, no $\lambda$ ). Both are then frozen, and the head is kept . Stage 2: the ten-dimensional softmax output $Z$ ( $=U$ ) is cached for each training image. For each budget $\lambda_{j}$ , an independent compressor–decoder pair $(C_{j},G_{j})$ is trained on $\mathrm{CE}+\lambda_{j}R_{\rm agg}$ with $\lambda_{j}$ fixed and never fed to the networks. A new budget therefore costs one additional $(C,G)$ fit. Orange boxes denote trainable components, blue dashed boxes denote frozen/cached components, and dashed red arrows indicate gradient flow.

Experiments	Architecture	Network structure and rate dependence	Total training MACs	Training time (s)
E1 / E2	Separated FCG	E1 trains one CNN $F$ per dataset; E2 trains one CNN on the pooled data. Each trained representation is reused across all eight budgets. For each $\lambda$ , the compressor $C$ : $10\!\to\!128\!\to\!128\!\to\!64$ (3 FC layers), and decoder $G$ : $32\!\to\!128\!\to\!10$ (2 FC layers) are trained separately. Each CNN has 3 Conv layers, 2 FC layers, and a 10-class linear head.	$3.039\!\times\!10^{13}$ / $3.039\!\times\!10^{13}$	141.3 / 138.2
E3 / E4	Independent VIB	Per model: CNN encoder (3 Conv + 2 FC, outputting $\mu$ and variance) and decoder ( $32\!\to\!128\!\to\!10$ , 2 FC). A separate end-to-end model is trained for each $\lambda$ .	$2.347\!\times\!10^{14}$ / $2.347\!\times\!10^{14}$	292.9 / 294.1
E5 / E6	FiLM-conditioned VIB	One rate-conditioned network: 3 Conv, 2 encoder FC, 4 FiLM FC, 2 rate-embedding FC, and 2 decoder FC. The network is conditioned on $\lambda$ throughout.	$2.355\!\times\!10^{14}$ / $2.355\!\times\!10^{14}$	487.3 / 489.3
E7 / E8	OIB	DFT input (3264 real coordinates); pretrained MLP with 3 FC layers ( $3264\!\to\!128\!\to\!128\!\to\!10$ ); closed-form Gaussian-IB compressor; one 2-FC decoder per budget ( $n_{z}\!\to\!128\!\to\!10$ ). The DFT itself is excluded from the MAC count.	$1.830\!\times\!10^{12}$ / $1.745\!\times\!10^{12}$	76.4 / 73.9

Table 2: Architecture and computational cost over the full grid of eight prespecified

\lambda

values. MACs and training times are averaged over 10 seeds from the reported run. FC stands for fully connected (linear) layer.

Experiment	Aggregation	Acc (%)	CE (nats)	$R_{\rm agg}$ (nats)	$\overline{\mathcal{L}}$	ECE
E1 separated, per domain	macro	$65.59\pm 0.49$	$1.1709\pm 0.0078$	$6.18\pm 0.03$	$1.4874\pm 0.0064$	$0.0862\pm 0.0034$
E3 VIB, per domain	macro	$57.30\pm 0.23$	$1.1806\pm 0.0057$	$6.96\pm 0.03$	$1.4139\pm 0.0043$	$0.0511\pm 0.0015$
E6 FiLM-conditioned VIB, per domain	macro	$60.86\pm 0.73$	$1.1227\pm 0.0157$	$4.81\pm 0.18$	$1.4126\pm 0.0121$	$0.0697\pm 0.0022$
E7 OIB, per domain	macro	$42.72\pm 0.39$	$1.7545\pm 0.0057$	$127.78\pm 0.49$	$6.6694\pm 0.0391$	$0.0876\pm 0.0045$
E2 separated, pooled	macro	$66.64\pm 0.75$	$1.1461\pm 0.0135$	$5.37\pm 0.08$	$1.4646\pm 0.0175$	$0.1003\pm 0.0054$
E4 VIB, pooled	macro	$56.87\pm 0.91$	$1.2472\pm 0.0164$	$7.22\pm 0.12$	$1.4790\pm 0.0065$	$0.0569\pm 0.0026$

Table 3: Test metrics averaged over the eight prespecified budgets (best-validation-epoch weights); mean

\pm

95% CI half-width over 10 seeds, all rows from the same run. Here

\overline{\mathcal{L}}:=\frac{1}{8}\sum_{j=1}^{8}\big(\mathrm{CE}_{j}+\lambda_{j}R_{{\rm agg},j}\big)

, for 8 given

\beta

, is the budget-averaged surrogate Lagrangian over the eight budgets, with

\mathrm{CE}_{j},R_{{\rm agg},j}

the test values at

\lambda_{j}

. “macro”: unweighted mean over the four datasets; “pooled test”: size-weighted pooled test set (pooled experiments only: E2, E4, E5, E8). E5/E6: one FiLM rate-conditioned VIB (step-matched to E3); E7/E8: OIB, bank-of-NN.

Panel of Figure 3 — Figure 3: Four-dataset macro-average over 10 seeds (unweighted over MNIST, SVHN, CIFAR-10, STL-10) test metrics versus the prespecified budget grid ( $\lambda$ is given). E1/E2: separated, per-domain/pooled; E3/E4: independent VIBs; E5/E6: one FiLM rate-conditioned VIB; E7/E8: OIB (bank-of-NN, own re-implementation). (a) Accuracy, (b) cross-entropy, (c) surrogate rate $R_{\rm agg}$ ; OIB has $R_{\rm agg}\approx 128$ – $149$ nats (closed-form compressor) and is omitted from (c) for scale.

MNIST
	E1 (sep., per dom.)			E2 (sep., pooled)			E3 (VIB, per dom.)			E4 (VIB, pooled)
$\lambda$	Acc	CE	$R$	Acc	CE	$R$	Acc	CE	$R$	Acc	CE	$R$
0.0050	96.17	0.273	11.34	96.04	0.197	11.05	95.86	0.137	14.61	95.17	0.156	18.55
0.0097	95.93	0.295	8.85	96.02	0.205	7.88	95.64	0.147	10.81	95.01	0.166	11.65
0.0186	95.58	0.310	7.45	95.98	0.217	6.09	95.06	0.164	8.12	94.25	0.197	8.50
0.0360	94.94	0.342	5.99	95.83	0.232	4.92	94.51	0.183	6.16	93.28	0.233	6.60

Table 4: Per-dataset results at every

\lambda

(MNIST, SVHN). Mean over 10 seeds of test accuracy (Acc, %), test cross-entropy (CE, nats) and surrogate rate (

R=R_{\rm agg}

, nats) at each prespecified

\lambda

CIFAR-10
	E1 (sep., per dom.)			E2 (sep., pooled)			E3 (VIB, per dom.)			E4 (VIB, pooled)
$\lambda$	Acc	CE	$R$	Acc	CE	$R$	Acc	CE	$R$	Acc	CE	$R$
0.0050	58.66	1.358	15.32	58.74	1.401	11.95	57.98	1.222	20.76	56.07	1.287	19.23
0.0097	58.32	1.384	11.26	58.67	1.421	8.80	56.20	1.246	13.67	57.29	1.299	13.41
0.0186	57.90	1.408	8.25	58.44	1.438	6.67	56.10	1.259	9.54	56.19	1.306	9.41
0.0360	57.11	1.445	6.09	58.07	1.466	5.15	54.19	1.306	6.45	55.17	1.335	6.66

Table 5: Per-dataset results at every

\lambda

(CIFAR-10, STL-10, pooled test set). Mean over 10 seeds of test accuracy (Acc, %), test cross-entropy (CE, nats) and surrogate rate (

R=R_{\rm agg}

, nats) at each prespecified

\lambda

Panel of Figure 4 — Figure 4: Macro-average over 10 seeds (E1 blue, E2 orange, E3 green dashed, E4 red dashed). (a) Test CE versus surrogate rate $R_{\rm agg}$ , from $\lambda=0.005$ (largest $R_{\rm agg}$ ) to $0.5$ . (b) ECE versus $\lambda$ (bars: 95% CI half-widths). The low E4 ECE at $\lambda=0.5$ reflects a collapsed predictor.

	Per-domain $F$ (E1)			Pooled $F$ (E2)
Dataset	$\mathrm{CE}_{H}$	$\hat{\epsilon}_{H}$	$\Phi(\hat{\epsilon}_{H})$	$\mathrm{CE}_{H}$	$\hat{\epsilon}_{H}$	$\Phi(\hat{\epsilon}_{H})$
MNIST	0.116	0.009	0.39	0.146	0.039	0.71
SVHN	0.917	0.025	0.59	0.909	0.018	0.52
CIFAR10	1.189	$-0.001$	0.00	1.238	0.049	0.78
STL10	1.223	0.011	0.43	1.296	0.084	0.96

Table 6: Empirical proxy for the sufficiency deficit

\epsilon_{U}=I(X;Y\mid U)

of the frozen interface

U=\mathrm{softmax}(\ell_{\vartheta}(F_{\theta}(X)))

, on the full test sets (mean over 10 seeds).

\mathrm{CE}_{H}

is the test cross-entropy of the frozen

F

with its stored linear head, an upper bound on

H(Y\mid U)

;

\mathrm{CE}_{X}

is that of a wider end-to-end CNN trained on

X

, an upper bound on

H(Y\mid X)

\hat{\epsilon}_{H}=\mathrm{CE}_{H}-\mathrm{CE}_{X}

is a difference of two upper bounds, and small negative values (CIFAR-10) are estimation error, i.e. indistinguishable from zero.

\Phi(\hat{\epsilon})

is the uniform stability bound of Theorem 3 evaluated at

\hat{\epsilon}

(negative values clipped to

0

Information Bottleneck

Information Bottleneck studies how to compress representations while retaining information needed for prediction, control, or communication.

See all 57 papers

Sep 29, 2026

Mutual Information Constrained Chernoff Bottleneck

The classical information bottleneck (IB) measures the relevance of a representation

U

X

to a target

Y

I(U;Y)

, which does not directly characterize the error of downstream decisions. For a binary hypothesis

Y

inferred from many separately encoded observations, the optimal error exponent is the Chernoff information between the two conditional distributions of

U

given

Y

. We study the mutual information constrained Chernoff bottleneck, which seeks an encoder that maximizes this Chernoff information subject to a rate constraint

I(U;X) \leq R

. We show that its optimal value

C(R)

increases strictly up to

R = H(V)

, where

V

merges the symbols of

X

with equal likelihood ratio, remains at the uncompressed exponent beyond, and, unlike the IB curve, need not be concave. We further show that

k+1

outputs suffice to attain

C(R)

, where

k

is the cardinality of

V

. We propose an alternating algorithm that updates the encoder via a generalized Blahut--Arimoto algorithm and the Chernoff parameter

s

via a nonlinear equation, and prove that its iterates remain feasible, with nondecreasing and convergent Chernoff information. Comparing with the classical IB for

n

samples, we show that, whenever the two objectives disagree, the IB encoder has the smaller equivocation

H(Y|U^{n})

n = 1

, whereas the Chernoff bottleneck has the smaller error probability and equivocation for all large

n

. Numerical experiments confirm the theory, and on real topic-detection data from the 20 Newsgroups corpus, compressing each word to only

17\%

of its entropy retains

90\%

of the error exponent and nearly the accuracy of the uncompressed classifier.

Dier Tang, Guangyue Han

Department of Mathematics, The University of Hong Kong

Information Bottleneck

Ask PDF

Oct 21, 2026

Optimal Multitask Representations Under a Rate Constraint: An Information Bottleneck Analysis

Which pretraining objective yields the best representation for downstream tasks under an information budget? We formalize this problem as a multitask information bottleneck, using a mutual information constraint to model incomplete information retention. For Gaussian inputs and tasks, we characterize optimal representations and show that a weakly task-aligned specialist can outperform a reconstruction generalist at low rates but lose at high rates. We also construct task families for which every fixed Gaussian prediction objective incurs arbitrarily large relative regret, driven by reversals in optimal allocation priorities. Conversely, the average task information matrix defines an objective with a uniform relative-regret bound controlled by maximum task SNR. CelebA and dSprites experiments illustrate that compression can change which pretraining objective performs best, and even task-aware training can reverse which downstream task is better preserved as rate increases.

Nicholas Stiles, Gregory W. Wornell

Massachusetts Institute of Technology

Representation Learning Information Bottleneck Multi-Task Learning Multi-Task Optimization

Ask PDF

Oct 1, 2026

Rethinking the Information Bottleneck: Structured Decomposition under Label-Induced Partitions

Standard information bottleneck (IB) regularization constrains representations via a single scalar I(Z;X), implicitlytreating all information as homogeneous. However, a single global compression control couples label-relevant structurewith residual within-condition variation, rather than regulating their allocation independently, allowing nuisanceinformation to persist in learned representations. For example, in medical imaging applications, residual variation oftenstems from acquisition conditions, background factors, or subject-specific appearance. This issue becomes particularlypronounced in data-limited settings, where models tend to overfit such variation, hindering generalization. While existingregularization methods can stabilize training, control capacity, or shape representation geometry, they do not explicitlyseparate nuisance-like variation from task-supporting structure. To address this limitation, we revisit IB from a structuredperspective based on a label-induced partition, where condition-level structure and within-condition information playdistinct roles. This leads to a dual-bottleneck formulation: a standard KL term controls global information capacity, while aconditional KL term targets within-condition information. We show that the conditional KL admits an exact decompositioninto a within-condition information term and a prior-mismatch term, explaining its alignment with the design objective.With a simplex-structured conditional prior, the method provides controllable latent geometry and integrates seamlesslyinto existing pipelines. Experiments on classification and segmentation show the clearest gains in low-data classificationand consistent improvements across dense prediction benchmarks.

Jingyao Zhang, Yuxuan Li, Lu Han +2

School of Computer Science, The University of Sydney Sydney, NSW 2006, Australia

Information Bottleneck Invariant Representation Learning

Ask PDF

Abstract

Information Bottleneck

Explore similar work

Mutual Information Constrained Chernoff Bottleneck

Optimal Multitask Representations Under a Rate Constraint: An Information Bottleneck Analysis

Rethinking the Information Bottleneck: Structured Decomposition under Label-Induced Partitions