On the Limits of Maximal Coding Rate Reduction for Out-of-Distribution Generalisation

Menghui Zhou; Gaoshan Bi; Vitaveska Lanfranchi; Po Yang

cs.LGSep 17, 2026

On the Limits of Maximal Coding Rate Reduction for Out-of-Distribution Generalisation

Authors: Menghui Zhou, Gaoshan Bi, Vitaveska Lanfranchi, Po Yang

Organizations: School of Computer Science University of Sheffield

Abstract

Substantial efforts have been devoted to making deep learning objectives, representations, and architectures interpretable, with the goal of improving the safety, robustness, and generalisation of learning systems in diverse real-world applications. The recently proposed maximal coding rate reduction ( $\mathrm{MCR}^{2}$ ) offers a promising information-theoretic framework for learning structured, discriminative representations of class-wise submanifolds and has inspired interpretable white-box architectures. However, we observe that $\mathrm{MCR}^{2}$ can completely fail under distribution shift, motivating our study of its out-of-distribution (OOD) generalisation limits. We establish two limitations of $\mathrm{MCR}^{2}$ for OOD generalisation. First, the $\mathrm{MCR}^{2}$ objective alone can admit complete prediction failure: a representation based entirely on unstable environmental features can achieve the global coding optimum yet fail completely after correlation reversal, despite an available perfectly stable feature. This exact-optimum example includes test inputs that cannot occur during training. Even when every possible test input can also occur during training, coding quality can be arbitrarily close to optimal while prediction error is arbitrarily close to 100%. Second, directly incorporating the invariance principle underlying widely successful invariant risk minimisation (IRM) and risk extrapolation (REx) does not eliminate this failure. The failing representation admits the same optimal coding operator across training environments, showing that shared coding optimality does not ensure stable prediction. Reliable OOD guarantees for $\mathrm{MCR}^{2}$ therefore require additional new assumptions or learning principles that establish stable predictive relationships across environments.

Figures & tables

Figure 1: Class-wise submanifolds mapped to orthogonal subspaces by $\mathrm{MCR}^{2}$ . Adapted from Figure 1 of Yu et al. (2020) .

Figure 2: Object and background structures in a camel image.

Overall	$(y=0,p=1)$	$(y=1,p=0)$
46.84	57.07	10.90

Table 1: Accuracy (%) on the two background-mismatched Waterbirds test groups.

Panel of Figure 3 — Figure 3: $\mathrm{MCR}^{2}$ saliency maps on Waterbirds.

Panel of Figure 4 — Figure 4: Optimal coding geometry survives correlation reversal; prediction fails.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Correlation reversal preserves coding quality while prediction fails. Lines show theory; points show empirical means with one-standard-deviation error bars.

Figure 6: Near-optimal coding and almost maximal target error coexist with identical input support.

Figure 7: A common coding maximiser coexists with severe target prediction failure.

Figure 8: Larger samples improve estimation while target failure persists. RMSE is measured against the population predictions. Dotted lines in (a,b) are

n^{-1/2}

references, not fitted rates. Panel (c) shows means and one standard deviation; panel (d) reports the fraction of runs observing every input in all four environments.

Setting	Input support	Coding guarantee	Target error
Noiseless reversal	Changes	Exactly optimal	$100\%$
Positive noise	Identical	Near-optimal	$99.900\%$

Appendix

Table 2: The two failure regimes. Coding guarantees refer to the population; errors are empirical means over

50

runs of

200{,}000

samples. The positive-noise row uses

\delta_{\max}=0.001

and

a=2

Explore similar work

May 8, 2026cs.LG

Does Your Neural Network Extrapolate? Feature Engineering as Identifiability Bias for OOD Generalization

Successful deep neural networks discover salient features of data. We show when and why they fail to learn out-of-distribution (OOD)-relevant representations from an in-distribution (ID) training window. This requires decoupling feature learning from data-generating-process (DGP) identifiability. From a single training window, OOD extrapolation is non-identifiable: infinitely many DGPs are

\varepsilon

-observationally equivalent on the training data but diverge arbitrarily outside it, and no in-distribution criterion alone reliably breaks the tie. A structural commitment, the feature map, label map, and model class

(\varphi, ψ, \mathcal{M})

, dictates the assumed DGP and governs OOD generalization while leaving ID performance essentially unchanged. When architecture, pretraining, augmentation, input formats, or domain knowledge implicitly inject the missing commitment, the model succeeds. When it cannot infer OOD-relevant structure from ID evidence, it fails. Changing only the representation can make the same architecture, at the same in-distribution loss, differ by

{\sim}520\times

out of distribution. When the commitment is correct and identifiable, OOD error vanishes. For example, Fourier coordinates turn periodic extrapolation into interpolation on

\mathbb{S}^1

. The same mechanism predicts outcomes in three natural-science settings (mass-action chemistry; Kepler's-third-law exoplanet prediction,

n=2{,}362

; and cross-species coding-DNA detection) and in a 264-run positional-encoding study across Transformer, Mamba, and S4D. Finally, a controlled study shows: correct features are necessary but not sufficient. The model class must express the target, and the transformed training data must cover the relevant representation space.

Leonel Aguilar, Jan Nagler, Christoph Hoelscher +1

Chair of Cognitive Science, ETH Zürich · Centre for Human and Machine Intelligence, Frankfurt School · Aisot Technologies AG +1

OOD Generalization Inductive Bias

Ask PDF

Sep 29, 2026math.NA

Into the danger zone: stable extrapolation in high-dimensional function and operator learning

Out-of-distribution (OOD) generalization is a central challenge in scientific machine learning. We study regression problems in which the test distribution differs from the training distribution and ask: under what assumptions on the target function or operator is stable extrapolation possible, and how far beyond the training domain can one extrapolate? Existing theory controls the test error through additive penalties measuring the discrepancy between the training and test distributions. Such guarantees show robustness to small distribution shifts, but can be very pessimistic in comparison to OOD performance observed empirically. We identify classes of holomorphic functions and operators for which the OOD generalization error converges at algebraic rates even in the presence of large distribution shifts. This phenomenon stems from the increasing smoothness of higher-index coordinates, leading to what we term a `blessing of high dimensionality'. For learning with either polynomials, deep neural networks or deep neural operators, we derive explicit rates for arbitrary test measures supported on suitable domains and quantify how the admissible domain depends on the underlying regularity of the function or operator. Our extrapolation guarantees are independent of the test distribution, depending only on its support. We also present a series of numerical experiments across a range of functions and operators that support the main theoretical findings.

Ben Adcock, Simone Brugiapaglia, Xuemeng Wang

Simon Fraser University · Concordia University

OOD Generalization Generalization Bounds Operator Learning

Ask PDF

Sep 25, 2026cs.LG

Can Circuit Alignment Predict OOD Generalization?

Can out-of-distribution (OOD) generalization be predicted from a trained model's weights alone, without any target-domain data? Existing representational similarity metrics (CKA, SVCCA, RSA) compare activations rather than forecast generalization. We show they are provably insensitive to structural rerouting in the computational graph, the very change distribution shift induces. We close this gap with the Circuit Alignment Score (CAS), which compares class-specific circuits across domains via graph kernels, decomposed into same-class coherence and cross-class confusion. Casting CAS as a Lebesgue integral over the domain distribution, we prove its Monte Carlo estimate recovers the ground-truth ranking of learners by OOD accuracy, with pairwise inversion error vanishing at rate

O(1/M)

, where

M

is the number of sampled domains. Across

48

learners on PACS, CAS attains

0.88

rank correlation with OOD accuracy, versus

0.58

(CKA),

0.23

(SVCCA), and

0.14

(RSA), with similar trends on other benchmarks and even against data-dependent methods, making it the first provably consistent predictor of distributional robustness requiring neither target-domain data nor labels. The code is available at: https://github.com/ayanban011/ACE

Ayan Banerjee, Abhra Chaudhuri, Josep Llados +2

Computer Vision Center, Universitat Autònoma de Barcelona · Fujitsu Research of Europe · Indian Statistical Institute, Kolkata +1

Distribution Shift Robustness OOD Generalization Neural Network Generalization Representation Alignment

Ask PDF