What Should an Information Bottleneck Encoder Learn? Separating Representation Learning from Rate Control

Hassan Tavakoli; Bella Bose

cs.LGOct 26, 2026

What Should an Information Bottleneck Encoder Learn? Separating Representation Learning from Rate Control

Authors: Hassan Tavakoli, Bella Bose

Organizations: School of EECS, Oregon State University, Corvallis, OR 97331, USA

Abstract

The information bottleneck (IB) trades compression of an input $X$ against prediction of a target $Y$ through a multiplier $β=1/λ$ , yet each operating point is usually served by training, or fully conditioning, a new encoder. For jointly Gaussian data, we show that the IB solution factorizes as $A(β)=D(β)W$ : the information-bearing directions $W$ do not depend on $β$ , and $β$ acts only through scalar gains. We ask whether this separation of representation from rate control survives beyond Gaussian models, treating the budget as a given operating parameter rather than a quantity to be optimized. For a budget-free representation $F(X)$ , the separation gap is controlled by the sufficiency deficit $ε_F=I(X;Y\mid F(X))$ : We show there is at least $ε_F$ in the worst case for finite inputs, and it is bounded uniformly in $λ$ by a function of $ε_F$ for finite label alphabets. The same certificate extends to a new target distribution and to pooled multi-domain data, where a conditional-alignment term appears. On MNIST, SVHN, CIFAR-10 and STL-10 (10 seeds, eight prespecified budgets), a frozen CNN representation with one compressor--decoder pair per budget reduces estimated training multiply--accumulate operations (MACs) by $7.7\times$ and wall-clock time by about $2\times$ relative to end-to-end VIBs, and attains higher budget-averaged accuracy ( $65.6\%$ versus $57.3\%$ per domain; $67.1\%$ versus $57.1\%$ pooled), with the advantage concentrated at strong compression, where the VIBs collapse. At weak compression ( $λ\le0.036$ ), VIB has higher cross-entropy and ECE; its surrogate Lagrangian is higher per domain, indistinguishable from the pooled VIB in the macro average, and lower on the pooled test set.

Figures & tables

Panel of Figure 1 — Figure 1: Representation–rate separation. $F$ is trained once without reference to the budget and characterized by the scalar $\epsilon_{F}\triangleq I(X;Y|F(X))$ ; $C$ and $G$ depend on the budget, $\beta$ .

Method	Role of $\beta$	What depends on $\beta$
VIB, nonlinear IB [ 4 , 8 ]	given	A whole Encoder/Decoder networks (one per $\beta$ )
Rate-condition [ 5 , 6 , 7 ]	given	A whole network
Flexible VIB (FVIB) [ 9 ]	given	closed-form rescaling (Taylor approx.)
Opportunistic IB (OIB) [ 10 ]	given	Gaussian-IB linear loadings
$\beta$ -free IB [ 11 ]	–	$\beta$ -free
Ours	given	New Separation method: only $(C,G)$ per $\beta$ and $F$ is $\beta$ -free

Table 1: Positioning. We treat

\beta

as a given operating parameter and ask what must depend on it.

Panel of Figure 2 — Figure 2: Stage 1: $F_{\theta}$ and an affine head $\ell_{\vartheta}$ are trained with cross-entropy only (no rate term, no $\lambda$ ). Both are then frozen, and the head is kept . Stage 2: the ten-dimensional softmax output $Z$ ( $=U$ ) is cached for each training image. For each budget $\lambda_{j}$ , an independent compressor–decoder pair $(C_{j},G_{j})$ is trained on $\mathrm{CE}+\lambda_{j}R_{\rm agg}$ with $\lambda_{j}$ fixed and never fed to the networks. A new budget therefore costs one additional $(C,G)$ fit. Orange boxes denote trainable components, blue dashed boxes denote frozen/cached components, and dashed red arrows indicate gradient flow.

Experiments	Architecture	Network structure and rate dependence	Total training MACs	Training time (s)
E1 / E2	Separated FCG	E1 trains one CNN $F$ per dataset; E2 trains one CNN on the pooled data. Each trained representation is reused across all eight budgets. For each $\lambda$ , the compressor $C$ : $10\!\to\!128\!\to\!128\!\to\!64$ (3 FC layers), and decoder $G$ : $32\!\to\!128\!\to\!10$ (2 FC layers) are trained separately. Each CNN has 3 Conv layers, 2 FC layers, and a 10-class linear head.	$3.039\!\times\!10^{13}$ / $3.039\!\times\!10^{13}$	141.3 / 138.2
E3 / E4	Independent VIB	Per model: CNN encoder (3 Conv + 2 FC, outputting $\mu$ and variance) and decoder ( $32\!\to\!128\!\to\!10$ , 2 FC). A separate end-to-end model is trained for each $\lambda$ .	$2.347\!\times\!10^{14}$ / $2.347\!\times\!10^{14}$	292.9 / 294.1
E5 / E6	FiLM-conditioned VIB	One rate-conditioned network: 3 Conv, 2 encoder FC, 4 FiLM FC, 2 rate-embedding FC, and 2 decoder FC. The network is conditioned on $\lambda$ throughout.	$2.355\!\times\!10^{14}$ / $2.355\!\times\!10^{14}$	487.3 / 489.3
E7 / E8	OIB	DFT input (3264 real coordinates); pretrained MLP with 3 FC layers ( $3264\!\to\!128\!\to\!128\!\to\!10$ ); closed-form Gaussian-IB compressor; one 2-FC decoder per budget ( $n_{z}\!\to\!128\!\to\!10$ ). The DFT itself is excluded from the MAC count.	$1.830\!\times\!10^{12}$ / $1.745\!\times\!10^{12}$	76.4 / 73.9

Table 2: Architecture and computational cost over the full grid of eight prespecified

\lambda

values. MACs and training times are averaged over 10 seeds from the reported run. FC stands for fully connected (linear) layer.

Experiment	Aggregation	Acc (%)	CE (nats)	$R_{\rm agg}$ (nats)	$\overline{\mathcal{L}}$	ECE
E1 separated, per domain	macro	$65.59\pm 0.49$	$1.1709\pm 0.0078$	$6.18\pm 0.03$	$1.4874\pm 0.0064$	$0.0862\pm 0.0034$
E3 VIB, per domain	macro	$57.30\pm 0.23$	$1.1806\pm 0.0057$	$6.96\pm 0.03$	$1.4139\pm 0.0043$	$0.0511\pm 0.0015$
E6 FiLM-conditioned VIB, per domain	macro	$60.86\pm 0.73$	$1.1227\pm 0.0157$	$4.81\pm 0.18$	$1.4126\pm 0.0121$	$0.0697\pm 0.0022$
E7 OIB, per domain	macro	$42.72\pm 0.39$	$1.7545\pm 0.0057$	$127.78\pm 0.49$	$6.6694\pm 0.0391$	$0.0876\pm 0.0045$
E2 separated, pooled	macro	$66.64\pm 0.75$	$1.1461\pm 0.0135$	$5.37\pm 0.08$	$1.4646\pm 0.0175$	$0.1003\pm 0.0054$
E4 VIB, pooled	macro	$56.87\pm 0.91$	$1.2472\pm 0.0164$	$7.22\pm 0.12$	$1.4790\pm 0.0065$	$0.0569\pm 0.0026$

Table 3: Test metrics averaged over the eight prespecified budgets (best-validation-epoch weights); mean

\pm

95% CI half-width over 10 seeds, all rows from the same run. Here

\overline{\mathcal{L}}:=\frac{1}{8}\sum_{j=1}^{8}\big(\mathrm{CE}_{j}+\lambda_{j}R_{{\rm agg},j}\big)

, for 8 given

\beta

, is the budget-averaged surrogate Lagrangian over the eight budgets, with

\mathrm{CE}_{j},R_{{\rm agg},j}

the test values at

\lambda_{j}

. “macro”: unweighted mean over the four datasets; “pooled test”: size-weighted pooled test set (pooled experiments only: E2, E4, E5, E8). E5/E6: one FiLM rate-conditioned VIB (step-matched to E3); E7/E8: OIB, bank-of-NN.

Panel of Figure 3 — Figure 3: Four-dataset macro-average over 10 seeds (unweighted over MNIST, SVHN, CIFAR-10, STL-10) test metrics versus the prespecified budget grid ( $\lambda$ is given). E1/E2: separated, per-domain/pooled; E3/E4: independent VIBs; E5/E6: one FiLM rate-conditioned VIB; E7/E8: OIB (bank-of-NN, own re-implementation). (a) Accuracy, (b) cross-entropy, (c) surrogate rate $R_{\rm agg}$ ; OIB has $R_{\rm agg}\approx 128$ – $149$ nats (closed-form compressor) and is omitted from (c) for scale.

MNIST
	E1 (sep., per dom.)			E2 (sep., pooled)			E3 (VIB, per dom.)			E4 (VIB, pooled)
$\lambda$	Acc	CE	$R$	Acc	CE	$R$	Acc	CE	$R$	Acc	CE	$R$
0.0050	96.17	0.273	11.34	96.04	0.197	11.05	95.86	0.137	14.61	95.17	0.156	18.55
0.0097	95.93	0.295	8.85	96.02	0.205	7.88	95.64	0.147	10.81	95.01	0.166	11.65
0.0186	95.58	0.310	7.45	95.98	0.217	6.09	95.06	0.164	8.12	94.25	0.197	8.50
0.0360	94.94	0.342	5.99	95.83	0.232	4.92	94.51	0.183	6.16	93.28	0.233	6.60

Table 4: Per-dataset results at every

\lambda

(MNIST, SVHN). Mean over 10 seeds of test accuracy (Acc, %), test cross-entropy (CE, nats) and surrogate rate (

R=R_{\rm agg}

, nats) at each prespecified

\lambda

CIFAR-10
	E1 (sep., per dom.)			E2 (sep., pooled)			E3 (VIB, per dom.)			E4 (VIB, pooled)
$\lambda$	Acc	CE	$R$	Acc	CE	$R$	Acc	CE	$R$	Acc	CE	$R$
0.0050	58.66	1.358	15.32	58.74	1.401	11.95	57.98	1.222	20.76	56.07	1.287	19.23
0.0097	58.32	1.384	11.26	58.67	1.421	8.80	56.20	1.246	13.67	57.29	1.299	13.41
0.0186	57.90	1.408	8.25	58.44	1.438	6.67	56.10	1.259	9.54	56.19	1.306	9.41
0.0360	57.11	1.445	6.09	58.07	1.466	5.15	54.19	1.306	6.45	55.17	1.335	6.66

Table 5: Per-dataset results at every

\lambda

(CIFAR-10, STL-10, pooled test set). Mean over 10 seeds of test accuracy (Acc, %), test cross-entropy (CE, nats) and surrogate rate (

R=R_{\rm agg}

, nats) at each prespecified

\lambda

Panel of Figure 4 — Figure 4: Macro-average over 10 seeds (E1 blue, E2 orange, E3 green dashed, E4 red dashed). (a) Test CE versus surrogate rate $R_{\rm agg}$ , from $\lambda=0.005$ (largest $R_{\rm agg}$ ) to $0.5$ . (b) ECE versus $\lambda$ (bars: 95% CI half-widths). The low E4 ECE at $\lambda=0.5$ reflects a collapsed predictor.

	Per-domain $F$ (E1)			Pooled $F$ (E2)
Dataset	$\mathrm{CE}_{H}$	$\hat{\epsilon}_{H}$	$\Phi(\hat{\epsilon}_{H})$	$\mathrm{CE}_{H}$	$\hat{\epsilon}_{H}$	$\Phi(\hat{\epsilon}_{H})$
MNIST	0.116	0.009	0.39	0.146	0.039	0.71
SVHN	0.917	0.025	0.59	0.909	0.018	0.52
CIFAR10	1.189	$-0.001$	0.00	1.238	0.049	0.78
STL10	1.223	0.011	0.43	1.296	0.084	0.96

Table 6: Empirical proxy for the sufficiency deficit

\epsilon_{U}=I(X;Y\mid U)

of the frozen interface

U=\mathrm{softmax}(\ell_{\vartheta}(F_{\theta}(X)))

, on the full test sets (mean over 10 seeds).

\mathrm{CE}_{H}

is the test cross-entropy of the frozen

F

with its stored linear head, an upper bound on

H(Y\mid U)

;

\mathrm{CE}_{X}

is that of a wider end-to-end CNN trained on

X

, an upper bound on

H(Y\mid X)

\hat{\epsilon}_{H}=\mathrm{CE}_{H}-\mathrm{CE}_{X}

is a difference of two upper bounds, and small negative values (CIFAR-10) are estimation error, i.e. indistinguishable from zero.

\Phi(\hat{\epsilon})

is the uniform stability bound of Theorem 3 evaluated at

\hat{\epsilon}

(negative values clipped to

0

Information Bottleneck

Information Bottleneck studies how to compress representations while retaining information needed for prediction, control, or communication.

See all 57 papers

Mutual Information Constrained Chernoff Bottleneck
Sep 29, 2026Dier Tang, Guangyue HanInformation Bottleneck
Department of Mathematics, The University of Hong Kong
Optimal Multitask Representations Under a Rate Constraint: An Information Bottleneck Analysis
Oct 21, 2026Nicholas Stiles, Gregory W. WornellRepresentation Learning Information Bottleneck
Massachusetts Institute of Technology
Rethinking the Information Bottleneck: Structured Decomposition under Label-Induced Partitions
Oct 1, 2026Jingyao Zhang, Yuxuan Li, Lu Han +2Information Bottleneck Invariant Representation Learning
School of Computer Science, The University of Sydney Sydney, NSW 2006, Australia

Sep 29, 2026J/K move · Enter open · S save

Mutual Information Constrained Chernoff Bottleneck

Dier Tang, Guangyue Han

Department of Mathematics, The University of Hong Kong

Information Bottleneck

Open paper AskPDF arXiv

The classical information bottleneck (IB) measures the relevance of a representation

U

X

to a target

Y

I(U;Y)

, which does not directly characterize the error of downstream decisions. For a binary hypothesis

Y

inferred from many separately encoded observations, the optimal error exponent is the Chernoff information between the two conditional distributions of

U

given

Y

. We study the mutual information constrained Chernoff bottleneck, which seeks an encoder that maximizes this Chernoff information subject to a rate constraint

I(U;X) \leq R

. We show that its optimal value

C(R)

increases strictly up to

R = H(V)

, where

V

merges the symbols of

X

with equal likelihood ratio, remains at the uncompressed exponent beyond, and, unlike the IB curve, need not be concave. We further show that

k+1

outputs suffice to attain

C(R)

, where

k

is the cardinality of

V

. We propose an alternating algorithm that updates the encoder via a generalized Blahut--Arimoto algorithm and the Chernoff parameter

s

via a nonlinear equation, and prove that its iterates remain feasible, with nondecreasing and convergent Chernoff information. Comparing with the classical IB for

n

samples, we show that, whenever the two objectives disagree, the IB encoder has the smaller equivocation

H(Y|U^{n})

n = 1

, whereas the Chernoff bottleneck has the smaller error probability and equivocation for all large

n

. Numerical experiments confirm the theory, and on real topic-detection data from the 20 Newsgroups corpus, compressing each word to only

17\%

of its entropy retains

90\%

of the error exponent and nearly the accuracy of the uncompressed classifier.

Abstract

Information Bottleneck

Explore similar work

Mutual Information Constrained Chernoff Bottleneck

Optimal Multitask Representations Under a Rate Constraint: An Information Bottleneck Analysis

Rethinking the Information Bottleneck: Structured Decomposition under Label-Induced Partitions