A Theoretical Framework for Understanding and Building "Any-task inference" Capable Foundation Models
Organizations: Columbia University Department of Biomedical Informatics · Harvard Medical School · Clalit Health Services and Ben Gurion University
Abstract
In this work, we provide a formal framework and mathematical definition for the subset of foundation models focused on enabling the property of "any-task inference" (ATI) (similar to zero-shot inference), rather than those designed for pre-training fine-tuning. To achieve this mathematical definition, we formalize the notion of a "space of tasks of interest," which we include in the problem domain, then provide an interface-based formal definition for ATI foundation models as those that map samples of data and tasks to predictions over the space of labels directly. While simple, this definition allows us to show that all ATI foundation models can be decomposed through a "universal task reduction" lens into encoder, model, and decoder aspects. We can leverage this decomposition to also decompose foundation model error into terms corresponding to different aspects of this reduction, stratified into a collection of "design terms," which are theoretical properties that can inform one of whether a particular proposed ATI foundation model can achieve optimality or not due to the structure of the proposed task reduction, and "realized model excess" terms which capture interactions between learned and programmatic components of foundation model systems. Lastly, we can use this perspective to offer two "prototypical" recipes for realizing ATI foundation models in diverse domains, which we term generative and discriminative variants of ATI foundation models. This exercise is grounded throughout by 5 real-world examples of foundation models in different settings, varying greatly in how these capabilities are realized, showing the applicability of this definition to real-world settings.
Figures & tables
| Space | Elements / random variable | Interpretation |
|---|---|---|
| Input space | Original data/input space. | |
| Task space | Space of downstream tasks of interest. | |
| Label space | Downstream label space. | |
| Encoder representation space | Space in which encoder-side representations of the task live. | |
| Decoder representation space | Space in which decoder-side representations of the task live. | |
| Global representation space | Joint task-representation space combining the information used by the encoder and decoder. |
| Role | Object | Definition / type | Induced object(s) | Interpretation |
| Base problem and reduction | ||||
| Base | Task kernel | True task-specific conditional distribution of labels. | ||
| Reduction | Encoder task representation | Represents the task for the purpose of constructing the universal-task input. | ||
| Reduction | Decoder task representation | Represents the task for the purpose of interpreting universal-task outputs in the original label space. | ||
| Reduction | Global representation | Joint representation containing the encoder- and decoder-side task representations. | ||
| Reduction | Encoder | Maps the original input and encoder-side task representation into the input space of the universal task. | ||
| Clinical LLM | EHR AR | PFN | EQ | Multitask | |
| Encoder task repr. | Natural-language prompt / instruction | Context set | Structured program / query | Categorical task index | |
| Encoder | Pair EHR input with prompt / instruction | Forward EHR input | Pair query with context set | Pair EHR input with structured query | Pair EHR input with task index |
| / Universal-task input | |||||
| Training data Typical pretraining data | Clinical text / EHR notes | Longitudinal structured EHR event sequences | Task episodes sampled from a prior: context set plus query | EHR examples paired with structured tasks and labels | EHR examples with labels for a fixed finite task set |
| Model | Autoregressive clinical language model | Autoregressive patient-trajectory model | PFN predictor | Task-conditioned EHR predictor | Shared multitask predictor |
| / Universal-task output | Generated clinical text / answer sequence | Generated EHR trajectory | Prediction in | Prediction in | Prediction in |
Zero-Shot Learning
Zero-shot learning lets models recognize classes or solve tasks absent from task-specific training examples by using semantic descriptions or pretrained knowledge.
See all 194 papers