Power-Law Correlations Without Apparent Temperature-Tuned Criticality in Fixed-Tree Hierarchical Languages

Marcel Kühn; Bernd Rosenow

cs.LGOct 17, 2026

Power-Law Correlations Without Apparent Temperature-Tuned Criticality in Fixed-Tree Hierarchical Languages

Authors: Marcel Kühn, Bernd Rosenow

Organizations: Institute for Theoretical Physics, Leipzig University · Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI), Leipzig University

Abstract

Sampling temperature can substantially reshape autoregressive generation, but it is unclear when temperature changes in long-range correlations reflect a genuine phase transition rather than a smooth change in statistics. We study this question in a controlled analogue of temperature sampling: a finite-depth probabilistic context-free grammar on a fixed hierarchical tree, where exact next-token conditionals are computed by belief propagation and then rescaled by a softmax temperature before autoregressive sampling. We measure two-point mutual information between tokens as a function of separation. Across the tested temperatures and grammar ensembles, mutual information remains well described by a power-law envelope. Temperature changes the correlation magnitude and fitted exponent, but within the accessible system sizes we find no long-distance plateau characteristic of an ordered phase and no clear crossover to short-range exponential decay. An approximate mapping to an Ising model on a tree explains how exponential decay in tree distance becomes a power law in token distance, allowing power laws to persist without tuning to a special decoding temperature. This controlled baseline clarifies which temperature effects can arise from hierarchical structure alone, before adding training, approximation error, or neural architecture.

Figures & tables

Panel of Figure 1 — Figure 1: Left: Schematic plot of different mutual-information behaviors. In the 2D Ising model, an ordered phase would show a plateau in the mutual information, whereas a disordered phase would show exponential decay. At criticality, correlations decay as a power law. Natural language likewise exhibits an approximate power-law decay of correlations. We ask whether such decay requires tuning to a special sampling temperature. Center: Structure of the synthetic language for $s=2$ and $L=3$ . Right: Protocol for autoregressively sampling text from this language at a modified temperature.

Panel of Figure 2 — Figure 2: Temperature dependence of the two-point mutual information in a grammar drawn from the full-support rule ensemble with heterogeneous probabilities, with $v=16$ , $L=10$ , and $s=2$ , corresponding to sequence length $d=1024$ . More details are given in Appendix A . Left: Empirical mutual information $I(r)$ as a function of token distance $r$ for several decoding temperatures. Points show estimates at individual distances, crosses indicate averages over distances that share the same lowest common ancestor with the last token, and dashed lines show power-law fits. Right: Fitted effective exponent $\alpha$ in $I(r)\sim r^{-\alpha}$ as a function of temperature. Temperature changes both the magnitude and the exponent of the correlations, but over the tested range the decay remains well described by a power law rather than becoming either a distance-independent plateau or a short-range exponential decay. Error bars indicate $2\sigma$ errors of the fit.

Panel of Figure 3 — Figure 3: Temperature dependence of the two-point mutual information in grammars drawn from sparse rule ensembles with branching factor $s=3$ , $L=5$ , $v=16$ , and $m=4$ allowed transition rules per parent. Top row: Strictly sparse grammar with $\varepsilon=0$ . Bottom row: Softened sparse grammar with $\varepsilon=0.1$ . In the strict case, forbidden transitions remain forbidden at all finite temperatures and therefore continue to sustain correlations even at high temperature. In the softened case, high temperature weakens the preferred sparse structure, but the observed decay remains well described by a power law over the sampled range. Additional results for sparse transition rules, including results for $s=2$ and $m=3$ , are shown in Appendix A .

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Additional temperature sweeps for the full-support heterogeneous rule ensemble with $v=16$ , $s=2$ , $L=10$ , $d=1024$ , and $\sigma=5$ . Rows correspond to three independent rule draws. Left column: empirical two-point mutual information $I(r)$ as a function of token distance $r$ . Points show individual distance estimates, crosses show averages over distances with the same common-ancestor level, and dashed lines show power-law fits to the averaged points. Right column: fitted effective exponent $\alpha$ in $I(r)\sim r^{-\alpha}$ as a function of decoding temperature. Each curve is estimated from $2.0\times 10^{7}$ autoregressively sampled strings. Error bars denote two standard errors from the fit; missing error bars indicate fits with only two averaged points.

Panel of Figure 5 — Figure 5: Temperature dependence of two-point mutual information in sparse rule ensembles with branching factor $s=2$ and $m=3$ allowed transition rules per parent. Top row: strictly sparse grammar with $\varepsilon=0$ . Bottom row: softened sparse grammar with $\varepsilon=0.1$ . Left column: empirical mutual information $I(r)$ as a function of token distance $r$ . Points show individual distance estimates, crosses show common-ancestor-level averages, and dashed lines show power-law fits. Right column: fitted effective exponent $\alpha$ in $I(r)\sim r^{-\alpha}$ as a function of decoding temperature. Each curve is estimated from $2.0\times 10^{8}$ autoregressively sampled strings. Error bars denote two standard errors from the fit; missing error bars indicate fits with only two averaged points.

Panel of Figure 6 — Figure 6: Diagnostic comparison between power-law and exponential fits for the full-support heterogeneous-rule ensemble of Fig. 2 . Top left: mutual information shown in semi-logarithmic coordinates over the full distance range. Top right: the same plot zoomed to short and intermediate distances. In these coordinates, an exponential decay would appear as a straight line, but the level-averaged points, especially at medium and high temperatures, visibly deviate from the fitted exponential lines. Bottom: ratio of logarithmic-space fit errors, $\mathrm{RMSE}_{\mathrm{pow}}/\mathrm{RMSE}_{\exp}$ . The ratio is below one for all tested temperatures, indicating that the power-law fit is preferred.

Autoregressive Generation

Autoregressive generation creates structured outputs step by step, conditioning each prediction on what has already been generated.

See all 109 papers

A Hierarchical Language Model with Predictable Scaling Laws and Provable Benefits of Reasoning
May 13, 2026Jason Gaitonde, Frederic Koehler, Elchanan Mossel +2Long-Context Language Modeling Language Model Scaling Laws
Duke University · University of Chicago · Massachusetts Institute of Technology +1
Generative Criticality in Large Language Model Temperature Scaling
Jun 4, 2026Huajian Ruan, Jinyang Li, Xingyu Guo +1LLM Inference Phase Transitions
State Key Laboratory of Nuclear Physics and Technology, Institute of Quantum Matter, South China Normal University, Guangzhou 510006, China · Key Laboratory of Atomic and Subatomic Structure and Quantum Control (MOE), Guangdong-Hong Kong Joint Laboratory of Quantum Matter, Guangzhou 510006, China · Guangdong Basic Research Center of Excellence for Structure and Fundamental Interactions of Matter, Guangdong Provincial Key Laboratory of Nuclear Science, Guangzhou 510006, China +4
Scaling limit of the Random Language Model
Jun 26, 2026Eric De GiuliLanguage Model Scaling Laws Language Modeling
Department of Physics, Toronto Metropolitan University, M5B 2K3, Toronto, Canada

May 13, 2026J/K move · Enter open · S save

A Hierarchical Language Model with Predictable Scaling Laws and Provable Benefits of Reasoning

Jason Gaitonde, Frederic Koehler, Elchanan Mossel, Joonhyung Shin+1

Duke University · University of Chicago · Massachusetts Institute of Technology+1

Long-Context Language Modeling Language Model Scaling Laws Autoregressive Language Modeling Language Modeling

Open paper AskPDF arXiv

We introduce a family of synthetic languages with hierarchical structure -- generated by a broadcast process on trees -- for which the role of context length and reasoning in autoregressive generation can be analyzed precisely. At the heart of our analytic approach is an \emph{exact

k

-gram ansatz} in place of transformers with context length

k

, a substitution we then validate empirically. Using this ansatz we derive explicit asymptotic predictions for distributional statistics of the sequences produced by a trained model, instantiated in two settings. For the Ising broadcast process (a soft-constrained language), we prove that the variance of the generated sum scales log-linearly in the context depth and its kurtosis converges to that of a Gaussian -- both deviating from the true language for any sublinear context. For the coloring broadcast process (a hard-constrained language) in the freezing regime, bounded-context autoregression produces sequences that, with high probability, are inconsistent with any valid coloring of the underlying tree. Together these results imply an

Ω(n)

lower bound on the context length required to faithfully sample length-

n

sequences. In contrast, we prove that an autoregressive reasoning model with only

Θ(\log n)

working memory can sample exactly from the true language -- an exponential improvement. We confirm both the lower-bound predictions and the reasoning-based upper bound empirically with transformers trained on the synthetic language; the trained models track our asymptotic predictions quantitatively across a wide range of context sizes.

Abstract

Autoregressive Generation

Explore similar work

A Hierarchical Language Model with Predictable Scaling Laws and Provable Benefits of Reasoning

Generative Criticality in Large Language Model Temperature Scaling

Scaling limit of the Random Language Model