A phase diagram for training and inference in language models
Organizations: School of Physics, Peking University, Beijing 100871, China · LightRay AI Lab, Beijing, China · Center for High Energy Physics, Peking University, Beijing 100871, China · School of Physics, Zhengzhou University, Zhengzhou 450001, China ·
Abstract
A language model is a complex system whose behavior is determined jointly by the Transformer and the corpus, yet how the representation geometry in the Transformer relates to the actual behavior of language models remains open. We analyze how the representation geometry changes along the forward propagation of actual language models and characterize the representation geometry of a document at a given layer by two collective coordinates, document distinguishability and document dimension. On the phase diagram spanned by these collective coordinates, inference is represented as a flow along depth. We find that, for untrained models, depending on their initialization scale, most of the 156 inference flows move toward the collapse and noise regions of the phase diagram. By contrast, the inference flows of 8 trained models from 5 architecture families all enter and reside in the central region of the phase diagram. Describing the evolution of the flows with a discrete Langevin equation, we find that the difference between the inference flows of untrained runs and those of trained models reduces to whether the effective force on the flows in this region is of saddle or stable-node type. On Pythia-1.4B, we interpret the training process as a phase transition. These results give a unified gateway to the relation between representation geometry and language-model behavior.
Figures & tables
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Quantity or phenomenon on the phase diagram | Description in the literature | Sources |
|---|---|---|
| Foundations | ||
| Inference flow: a discrete dynamical system along depth | Residual networks are discretizations of ordinary differential equations | E [ 20 ] ; Haber and Ruthotto [ 40 ] ; Chen et al. [ 14 ] ; Lu et al. [ 58 ] |
| : participation ratio of the within-document covariance | Participation ratio; effective rank and intrinsic dimension measure dimensionality in other ways | Roy and Vetterli [ 78 ] ; Facco et al. [ 27 ] ; Ansuini et al. [ 2 ] |
| : ratio of the between-document offset to the within-document variation | Ratio of between-class to within-class scatter | Papyan et al. [ 68 ] |
| : approximately the average cosine similarity between the representations of two tokens from the same document; document identification accuracy rises with it (Appendix C ) | Accuracy of linear probes | Alain and Bengio [ 1 ] ; Belinkov [ 5 ] |
| : offset of the document mean relative to the reference | Difference-of-means directions: linear directions of concepts such as truth and gender | Park et al. [ 69 ] ; Marks and Tegmark [ 59 ] |
| Model | Identifier | Revision | ||
|---|---|---|---|---|
| Qwen2.5-1.5B | Qwen/Qwen2.5-1.5B | 28 | 1536 | 8faed761d45a263340a0528343f099c05c9a4323 |
| Qwen2.5-7B | Qwen/Qwen2.5-7B | 28 | 3584 | d149729398750b98c0af14eb82c78cfe92750796 |
| Qwen2.5-14B | Qwen/Qwen2.5-14B | 48 | 5120 | 97e1e76335b7017d8f67c08a19d103c0504298c9 |
| Mistral-7B-v0.3 | mistralai/Mistral-7B-v0.3 | 32 | 4096 | caa1feb0e54d415e2df31207e5f4e273e33509b1 |
| Falcon3-7B | tiiuae/Falcon3-7B-Base | 28 | 3072 | bf3d7ed586cb22a921520e2d681a9d3d7642cde8 |
| Granite-3.1-8B | ibm-granite/granite-3.1-8b-base | 40 | 4096 | 39975ba909950a1bad8fa1cb6981d4d048b75553 |
| Family | Runs ( values runs per ) | |
|---|---|---|
| F | 0.02 | |
| F | 0.005, 0.01, 0.014, 0.028, 0.04, 0.06, 0.1, 0.2 | |
| G | 0.02, 1 | |
| G | 0.035, 0.06, 0.1, 0.17, 0.3, 0.5 | |
| Total | 156 |
| Column in the figure | Setting changed |
|---|---|
| main text | unchanged |
| lin. point: last third | late half replaced by the last third ( ), linearization point moved to |
| cell width: 0.07 | cell width changed from 0.1 dex to 0.07 dex |
| cell width: 0.14 | changed to 0.14 dex |
| cell width: 0.2 | changed to 0.2 dex |
| fit range: shrunk 0.1 | each edge of the fit range moved inward by 0.1 dex |
Representation Geometry in Language Models
Representation geometry in language models studies how embeddings and hidden states organize concepts, computation, and behavior.
See all 446 papers