What's in a Tree ? Hierarchical -gram models as controlled approximators of language
Organizations: Joint Center for Quantum Information and Computer Science, UMD & NIST. · Simplex. · Data Science and AI Institute, and Department of Physics and Astronomy, JHU. · Department of Physics, UMD. ·
Abstract
Hierarchical models have been proposed as synthetic settings for studying language. However, their relevance to realistic language corpora remains underexplored. In this work, we introduce tree-grams, autoregressive generative -gram models that build the conditional probability tensor through a hierarchical composition of order-2 and order-3 tensors. controls the maximal context window, whereas a rank hyperparameter controls model capacity, yielding an interpretable and systematically scalable family of approximators of language. We show how tree-grams outperform existing -gram baselines, while remaining competitive with deep transformer-based language models. We demonstrate the utility of such controlled approximations by using tree-grams as generative models. In particular, we show that hyperparameter transfer under P can be made more robust when narrower models are tuned on simpler data, generated by tree-grams with moderate and . In addition, we show that entropy and effective range of correlations control performance gap after fine-tuning. This is because tree-grams allow us to explicitly control horizon and capacity. When allowed to route across candidate topologies, we observe emergent sparsity in tree-grams. Such concentrated routing permits pruning upto 95% of topologies without significant change in performance.
Figures & tables
| Topology | Full | No LayerNorm | No residual | No MLP |
|---|---|---|---|---|
| aligned | 1.847 | 2.567 ( ) † | 1.946 ( ) | 1.970 ( ) |
| anti-aligned | 1.805 | 2.025 ( ) | 3.142 ( ) | 1.935 ( ) |
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| WT (130 MT) | TS (529 MT) | FWE (1.05 BT) | ||||||||
| Cores | Model | median | mean | max | median | mean | max | median | mean | max |
| 1 | Tree-gram | 58.3 | 91.5 | 320 | 232 | 362 | 1,253 | 956 | 1,097 | 2,985 |
| Transformer | 40.4 | 50.4 | 125 | 163 | 203 | 499 | 779 | 778 | 1,543 | |
| 16 | Tree-gram | 3.9 | 6.2 | 21.2 | 15.7 | 24.3 | 83.6 | 70.2 | 80.2 | 205 |
| Transformer | 2.8 | 3.4 | 8.3 | 12.0 | 14.6 | 35.1 | 58.2 | 58.0 | 104 | |
| 32 | Tree-gram | 2.3 | 3.7 | 12.3 | 9.4 | 14.0 | 46.4 | 41.2 | 48.9 | 124 |
| tree-gram | Tree params | Transformer width by depth | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| 3 | 1024 | 36,709,376 | 1440 | 1072 | 896 | 800 | 720 | 656 | 512 | 480 |
| 4 | 1024 | 56,637,440 | 1856 | 1376 | 1152 | 1008 | 912 | 832 | 656 | 592 |
| 8 | 1024 | 78,665,728 | 2240 | 1648 | 1376 | 1200 | 1072 | 992 | 784 | 704 |
| 16 | 1024 | 104,892,416 | 2640 | 1920 | 1600 | 1392 | 1248 | 1152 | 896 | 832 |
| 32 | 64 | 1,039,744 | 112 | 96 | 96 | 80 | 80 | 80 | 64 | 64 |
| log-RMSE | AICc | KS | KL | |||||
| Model | s0 | s1 | s0 | s1 | s0 | s1 | s0 | s1 |
| Exponential | 2.965 | 2.972 | 191424.8 | 175489.6 | 0.839 | 0.809 | 3.597 | 3.513 |
| Generalized exponential | 0.433 | 0.393 | 50186.3 | 26910.6 | 0.373 | 0.340 | 0.580 | 0.590 |
| Power law / Pareto | 1.622 | 1.940 | 147131.1 | 144180.5 | 0.431 | 0.459 | 0.682 | 0.880 |
| PL with exponential cutoff | 0.351 | 0.371 | 34644.7 | 22679.1 | 0.266 | 0.279 | 0.200 | 0.259 |
Language Modeling
Language modeling develops systems that learn patterns in text and other sequences to understand, predict, and generate language.
See all 547 papers