Telescopic Language Models

Zhilin Guo; Boqiao Zhang; Hakan Aktas; Kyle Fogarty; Nursena Koprucu Aslan; Wenzhao Li; Canberk Baykal; Albert Miao

cs.CLSep 28, 2026

Telescopic Language Models

Authors: Zhilin Guo, Boqiao Zhang, Hakan Aktas, Kyle Fogarty, Nursena Koprucu Aslan, Wenzhao Li, Canberk Baykal, Albert Miao, +9 more

Organizations: University of Cambridge · University of British Columbia · Google

Abstract

One deployed language model must often serve many compute budgets, yet serving each budget still means a separate training or compression run per point. We train a Telescopic Language Model (TLM) to be that continuum: a nested-capacity Transformer supervised by stochastic prefix supervision with a full anchor. At every step, one randomly truncated prefix of the capacity axis is trained against the full next-token target, alongside one full-capacity pass, so the trained artifact is a valid language model at every depth. Two forward-backward passes per step, no architectural change, nothing extra at inference. Fixed-exit suites such as Matryoshka Language Model Suites (MLMS) occupy one point in this design space, and the point has a cost: supervising only a few fixed exits leaves the nested model at chance level everywhere else (perplexity 10^2-10^5 in our baselines). On a 200M proxy suite (20B FineWeb-Edu tokens, identical data stream for all methods), a single TLM run is a valid language model at every one of its twenty layer prefixes, in perplexity and on perplexity-sensitive downstream tasks, reducing the area under the quality-budget curve by 43-44% relative to the fixed-exit suites while matching them at full capacity, at ~12% lower GPU cost per run. The prefix sampling density is a dial: concentrating it on a few depths recovers fixed-exit quality there at the price of the continuum, so the operating points become a training-time choice rather than an architectural one. These results indicate that the training objective, not the nesting itself, is what makes a model elastic.

Figures & tables

Method	200M PPL $\downarrow$	AULB $\downarrow$	$\mathrm{LODA}_{\mathrm{acc}}$ $\uparrow$	$\mathrm{LODA}_{\mathrm{ppl}}$ $\uparrow$
MLMS (best suite)	14.98	5.90	21.8	27.8
TLM ( $\pi$ uniform)	14.99	3.28	38.9	47.3
TLM ( $\pi$ log-unif.)	14.99	3.45	34.9	48.5
TLM ( $\pi$ exit-grid)	14.65	6.21	–	25.7
vanilla twin (ref.)	13.98	–	–	–

Table 1: Headline results (200M proxy suite, 20B tokens/run, identical data stream). Full-size (200M) validation PPL; AULB over the prefix grid

k{=}1..20

;

\mathrm{LODA}_{\mathrm{acc/ppl}}

, the area under the quality–throughput frontier (§ 3.3 ; higher is better). Bold marks the best single-run cascade per column; the vanilla twin is the per-size reference. ‘–’: not evaluated.

Method	GPU-h $\downarrow$	operating points $\uparrow$	GPU-h / point $\downarrow$
vanilla suite (3 models)	216	3	72
MLMS suite, no distill (1 run)	149	3	50
MLMS suite, +distill (1 run)	182–185	3	61–62
TLM, uniform $\pi$ (1 run)	131	20	6.5

Table 2: One run, twenty operating points: TLM delivers each at 6.5 GPU-h versus 50–72 for the suites. Training cost (billed GPU-hours, one A100 per run, 20B tokens/run); TLM and MLMS rows are single runs, the like-for-like row holds distillation off on both sides, and TLM’s twenty points include off-frontier prefixes (see text).

Explore similar work

Tapered Language Models
Jun 22, 2026Reza Bayat, Ali Behrouz, Aaron CourvilleLanguage Modeling Layer-Wise
1Mila · 2Cornell University · 3Université de Montréal +1
Matryoshka Language Model Suites
Aug 10, 2026Nathan Godey, Yoav ArtziLanguage Modeling Matryoshka Representation Learning
Cornell University
Star Elastic: Many-in-One Reasoning LLMs with Efficient Budget Control
May 8, 2026Ali Taghibakhshi, Ruisi Cai, Saurav Muralidharan +17LLM Reasoning Strategies Nvidia
NVIDIA

Jun 22, 2026, cs.LGJ/K move · Enter open · S save

Tapered Language Models

Reza Bayat, Ali Behrouz, Aaron Courville

1Mila · 2Cornell University · 3Université de Montréal+1

Language Modeling Layer-Wise Transformer Architectures

Open paper AskPDF arXiv

Modern language models, including transformer, recurrent, and memory-based variants, share a common chassis: a stack of identical layers in which parameters are allocated uniformly across depth. This is a default inherited from the original transformer and largely unchanged since, yet a growing body of evidence suggests that layers contribute non-uniformly to the final output, with later layers refining the residual stream rather than transforming it. We ask whether parameter capacity should reflect this asymmetry. Our controlled experiment shows that, under a fixed budget, allocating more capacity to earlier layers and less to later layers improves perplexity over a uniform-width baseline, while the reverse allocation hurts. Building on this result, we introduce Tapered Language Models (TLMs), an architectural principle in which a parameter-bearing component is monotonically tapered across depth under a fixed total budget. MLPs are the natural site for this instantiation: they dominate parameter count across all modern LM families and expose width as a single, clean axis of variation. Across three model scales and four architectures (Transformer, Gated Attention, Hope-attention, and Titans), tapering MLP width via a smooth cosine schedule consistently improves perplexity and downstream benchmark performance over uniform baselines, at no additional parameter or compute cost. These findings establish depth-aware capacity allocation as a simple, architecture-agnostic axis of language model design, a free lever hidden in plain sight.

Abstract

Explore similar work

Tapered Language Models

Matryoshka Language Model Suites

Star Elastic: Many-in-One Reasoning LLMs with Efficient Budget Control