A Fiber Criterion for Representation Identifiability in Supervised Learning

Representation learning is often described as preserving the information in an input that is relevant for prediction. This work asks what relevance means for a fixed supervised decision problem. A representation is defined to be Bayes-sufficient for a joint distribution and loss if some prediction head can use it to implement a Bayes-optimal action rule. This makes the target information loss-dependent. In the almost-surely unique Bayes-action case, the relevant object is a Bayes quotient, which identifies inputs that require the same Bayes-optimal action. A representation is sufficient when it refines this quotient, and Bayes-minimal when it is informationally equivalent to it. The framework connects naturally to property elicitation: zero-one loss requires the Bayes class, squared loss the conditional mean, Brier loss the conditional probability in binary prediction, and log loss or strictly proper scoring rules the predictive distribution. Controlled finite experiments, learned neural bottleneck experiments, and a real-data iNaturalist taxonomic refinement experiment illustrate the distinction between sufficiency, minimality, and retained non-required information. For a fixed supervised problem, the distribution and the loss determine the Bayes action, the Bayes action determines the quotient, and the quotient determines the minimal information required for Bayes-optimal prediction.

Given a generalist model, learning a task-relevant specialist representation is fundamental for downstream applications. Identifiability, the asymptotic guarantee of recovering the ground-truth representation, is critical because it sets the ultimate limit of any model, even with infinite data and computation. We study this problem in a completely nonparametric setting, without relying on interventions, parametric forms, or structural constraints. We first prove that the structure between time steps and tasks is identifiable in a fully unsupervised manner, even when sequences lack strict temporal dependence and may exhibit disconnections, and task assignments can follow arbitrarily complex and interleaving structures. We then prove that, within each time step, the task-relevant latent representation can be disentangled from the irrelevant part under a simple sparsity regularization, without any additional information or parametric constraints. Together, these results establish a hierarchical foundation: task structure is identifiable across time steps, and task-relevant latent representations are identifiable within each step. To our knowledge, each result provides a first general nonparametric identifiability guarantee, and together they mark a step toward provably moving from generalist to specialist models.

Learned representations are central to modern machine learning and are commonly evaluated through predictive performance, robustness, uncertainty estimation, and generalization. However, a representation may remain operationally successful while failing to organize persistent residual structures that conventional metrics do not fully capture. This article introduces VER, the Vigilant Evaluator of Representations, a conceptual framework for monitoring representational adequacy. VER does not propose a new learning algorithm, loss function, or model architecture. It defines a diagnostic process for identifying residual structures and assessing whether they may indicate explanatory insufficiency rather than ordinary error, uncertainty, noise, data limitation, or distribution shift. The framework comprises five operations: representation identification, explanatory-domain delimitation, residual-structure detection, explanatory-resistance evaluation, and vigilance signaling. VER distinguishes stable adequacy, a vigilance condition, and a representational alert. It is intended to complement performance evaluation, uncertainty estimation, out-of-distribution detection, robustness analysis, and explainable AI by making representational adequacy an explicit object of inquiry. The article also outlines a path toward empirical evaluation through benchmarks designed to detect representational inadequacy when predictive performance remains satisfactory. VER is conceptual and methodological; it does not prove that a representation is inadequate, select a replacement representation, or provide an operational implementation.