Why Finite Intelligence Works in an Infinite World: A Mathematical Theory
Organizations: University of Illinois Chicago
Abstract
An information system often has to learn before it knows what it will be asked. A finite representation can succeed without recording an infinite environment in full when those asks share collective structure, rather than demand unrelated facts scattered arbitrarily across it. We formalize this phenomenon through future-task information, the minimum retained information needed to preserve decision value relative to the available evidence. A joint decision-regret state records posterior comparisons among actions for every permitted task. Under compactness assumptions, every retained representation admits a calibrated replacement based on this state, preserving all task regrets without increasing information; the resulting information-regret curve equals an operational coding rate. Two exact boundaries govern finite preservation: weak-star closure for a fixed action count, and the singular mass of the source-evidence law when all finite action menus are admitted. Collective structure determines information requirements beyond individual-question difficulty. Families with identical source laws, pairwise task metrics, and individual-task rates can have different high-resolution exponents. Covariance-identical coordinate and parity families have different exact rate-distortion curves, with optimal parity retention characterized by a sharp Fourier-entropy inequality. For balanced nested task families, refinement sizes and weighted reach determine the rate within an additive constant, attained by prefix coding under explicit decay conditions. Nonlinear learning and computational separations distinguish preservation from efficient discovery and use. The proposed mathematical theory identifies which shared decision requirements allow finite information to remain useful for infinitely many possible purposes.
Figures & tables
| Setting | Prepared in advance | Specified later |
|---|---|---|
| Recommendation Netflix / MovieLens [ 30 , 31 ] | User–item interactions, observed preferences, and learned relations among users and items | The requested recommendation, relevant context, ranking objective, and cost of an incorrect prediction |
| Large language models [ 32 , 33 ] | Parameters and reusable representations learned from broad training data | The question or prompt, desired response, constraints, and criterion for an acceptable answer |
| AI model construction Weight-space programming [ 34 ] | Previously learned weights, parameter relationships, model structure, and reusable computational results | The model to be built, its architecture and objective, training constraints, and required quality or guarantees |
| Question | Mathematical criterion | What is charged |
|---|---|---|
| Preserving future decision value | and the calibrated decision state | Mutual information; fixed-length rate after iid amortization under compactness |
| Possibility with finite information | Weak-star closure; singular mass for all finite menus | Any finite message from some finite observation prefix |
| Discovering a reusable representation | in a declared observation model | Raw samples and a hard -bit message; worst-parameter risk |
| Finding active future tasks | Primal–dual gap in ( 52 ) | Task-search and convex-projection calls |
| Using information efficiently | Declared computational model | Encoder and decoder work, including accessible side information |
Information Bottleneck
Information Bottleneck studies how to compress representations while retaining information needed for prediction, control, or communication.
See all 62 papers