Rethinking the Relationship between the Power Law and Hierarchical Structures

Kai Nakaishi; Ryo Yoshida; Kohei Kajikawa; Koji Hukushima; Yohei Oseki

cs.CLMay 8, 2025

Rethinking the Relationship between the Power Law and Hierarchical Structures

Authors: Kai Nakaishi, Ryo Yoshida, Kohei Kajikawa, Koji Hukushima, Yohei Oseki

Organizations: RIKEN, Japan · National Institute for Japanese Language and Linguistics, Japan · The University of Tokyo, Japan · Georgetown University, USA · National Institute of Informatics, Japan

Abstract

Statistical analysis of corpora provides an approach to quantitatively investigate natural languages. This approach has revealed that several power laws consistently emerge across different corpora and languages, suggesting universal mechanisms underlying languages. In particular, the power-law decay of correlations has been interpreted as evidence of underlying hierarchical structures in syntax, semantics, and discourse. This perspective has also been extended beyond corpora produced by human adults, including child speech, birdsong, and chimpanzee action sequences. However, the argument supporting this interpretation has not been empirically tested in natural languages. To address this gap, the present study examines the validity of the argument for syntactic structures. Specifically, we test whether the statistical properties of parse trees align with the assumptions in the argument. Using English and Japanese corpora, we analyze the mutual information, deviations from probabilistic context-free grammars (PCFGs), and other properties in natural language parse trees, as well as in the PCFG that approximates these parse trees. Our results indicate that the assumptions do not hold for syntactic structures and that it is difficult to apply the proposed argument not only to sentences by human adults but also to other domains, highlighting the need to reconsider the relationship between the power law and hierarchical structures.

Figures & tables

Figure 1: Sequential and structural distances. (a) If trees are balanced, the sequential distance grows exponentially with the structural distance,

r_{\text{seq}}\sim\exp( \mu r_{\text{str}})

. (b) If trees are strongly biased, the growth is slower. For example, the relation is linear,

r_{\text{seq}}\sim r_{\text{str}}

Figure 2: Example of a parse tree from BLLIP, preprocessed as described in Section 3.1 . This tree has a strong right-branching bias.

Figure 3: CFIB

J

is the MI between the children of two nodes whose categories are fixed.

Figure 4: Estimation and fitting of the MI for (a) the exponential model with

\lambda=0.1

and (b) the power-law model with

\alpha=2

. Fitting was performed for

N_{\text{data}}=8\times 10^{6}

Figure 5: MI

I_{\text{POS}}

between POS tags as a function of the sequential distance

r_{\text{seq}}

. The fitted exponential and power-law decays for

N_{\text{data}}=2.56\times 10^{6}

are also presented.

Figure 6: MI

I_{\text{tag}}

between POS or phrasal tags as a function of the structural distance

r_{\text{str}}

. The fitted exponential and power-law decays for

N_{\text{data}}=2.56\times 10^{6}

are also presented.

Figure 7: Number of POS tag pairs such that the structural and sequential distances are $r_{\text{str}}$ and $r_{\text{seq}}$ , respectively. Black circles represent the average sequential distance for each structural distance.

Figure 8: CFIB

J

for

(x_{0},x_{1})=(\text{NP},\text{NP})

as a function of the structural distance

r_{\text{str}}

. The fitted exponential and power-law decays for

N_{\text{data}}=2.56\times 10^{6}

are also presented.

Figure 9: Statistical properties of syntactic structures in WikiText and NPCMJ. (a1–4) Results for WikiText: (a1) $I_{\text{POS}}$ as a function of $r_{\text{seq}}$ ; (a2) $I_{\text{tag}}$ as a function of $r_{\text{str}}$ ; (a3) relationship between $r_{\text{seq}}$ and $r_{\text{str}}$ ; and (a4) CFIB $J$ for $(x_{0},x_{1})=(\text{NP},\text{NP})$ as a function of $r_{\text{str}}$ . Fitting was performed with $N_{\text{data}}=2.56\times 10^{6}$ . (b1–b4) Corresponding results for NPCMJ. Fitting was performed with $N_{\text{data}}=6.4\times 10^{5}$ for $I_{\text{POS}}$ and $I_{\text{tag}}$ , and with $N_{\text{data}}=4.0\times 10^{4}$ for $J$ .

Figure 10: Statistical properties of trees generated by the PCFG obtained from the maximum likelihood estimation of syntactic structures. (a) MI $I_{\text{POS}}$ between POS tags. (b) MI $I_{\text{tag}}$ between POS or phrasal tags. (c) Distribution and average of sequential distances. (d) CFIB $J$ for $(x_{0},x_{1})=(\text{NP},\text{NP})$ . Fitting was performed for $N_{\text{data}}=8\times 10^{6}$ in (a) and (b), and for $r_{\text{str}}\leq 10$ in (a).

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Corpus	Depth	Branching	Prop. Unary
BLLIP	$11.598(\pm 4.354)$	$1.527(\pm 0.904)$	$0.222$
NPCMJ	$6.642(\pm 3.336)$	$1.547(\pm 1.282)$	$0.271$

Appendix

Table 1: Tree-shape statistics computed on the original parse trees in BLLIP and NPCMJ. The depth and the branching factor are reported as mean

\pm

standard deviation across sentences.

Tags	Part-of-Speech Tags and Phrase Labels
NP	NP NN NNP NNS CD PRP TMP POS QP NNPS EX NX FW
VP	VP VBD VB VBN VBG VBZ VBP TO
S	S S1 SBAR SINV FRAG SQ SBARQ RRC
PP	IN PP RP PRT
DT	DT PRP$ PDT
JJ	JJ ADJP JJR JJS NAC ADJ

Appendix

Table 2: Eleven tags used for BLLIP and WikiText and the corresponding part-of-speech tags and phrase labels.

Tags	Part-of-Speech Tags and Phrase Labels
NP	NP NUMCLP PRN NML PNLP NUMCLPSYM N NPR NUM CL ADJN PRO ADJI D Q FN WPRO PNL PRN WNUM WD FW
IP	IP VB AX AXD VB0 VB2 PASS PASS2 MD
PP	PP P
ADVP	ADVP ADV NEG WADV
CONJP	CONJP CONJ
PP	PP P

Appendix

Table 3: Seven tags used for NPCMJ and the corresponding part-of-speech tags and phrase labels.

Figure 11: (a1, 2) Results for BLLIP under the unbinarized setting: (a1) MI $I_{\text{tag}}$ between POS or phrasal tags, where fitting was performed for $N_{\text{data}}=2.56\times 10^{6}$ ; (a2) distribution and average of sequential distances. (b1, 2) Results for BLLIP under the phrasal-tag-only setting: (b1) MI $I_{\text{tag}}$ between phrasal tags, where fitting was performed for $N_{\text{data}}=2.56\times 10^{6}$ ; (a2) CFIB $J$ , where fitting was pwerformed for $N_{\text{data}}=6.4\times 10^{5}$ . (c1, 2) Results for NPCMJ under the unbinarized setting: (a1) MI $I_{\text{tag}}$ between POS or phrasal tags, where fitting was performed for $N_{\text{data}}=2.56\times 10^{6}$ ; (a2) distribution and average of sequential distances.

Figure 12: Results for BLLIP obtained with different estimators. (a) MI

I_{\text{POS}}

between POS tags. (b) MI

I_{\text{tag}}

between POS or phrasal tags. (c) CFIB

J

for

(x_{0},x_{1})=(\text{NP},\text{NP})

. In all cases,

N_{\text{data}}=2.56\times 10^{6}

Figure 13: Results for the PCFG obtained with different estimators. (a) MI

I_{\text{POS}}

between POS tags. (b) MI

I_{\text{tag}}

between POS or phrasal tags. In both (a) and (b),

N_{\text{data}}=8\times 10^{6}

. (c-e) CFIB

J

for

(x_{0},x_{1})=(\text{NP},\text{NP})

obtained with (c) MM, and (d) NSB, and (e) ZA.

Figure 14: CFIB

J

for BLLIP with (a)

(x_{0},x_{1})=(\text{NP},\text{VP})

and (b)

(x_{0},x_{1})=(\text{VP},\text{VP})

. Fitting was performed for

N_{\text{data}}=2.56\times 10^{6}

in (a) and

N_{\text{data}}=6.4\times 10^{5}

in (b).

Explore similar work

Universal Topological Regularity of Syntactic Structures
Jan 31, 2023Fermín Moscoso del Prado MartínSyntactic Structure Phylogenetic Inference
Department of Language and Communication & Center for Language Studies Radboud University Erasmuslaan 1, 6525 NL, Nijmegen, The Netherlands
A Data-free Universal Prior over Syntactic Structures
Sep 15, 2026Fermín Moscoso del Prado MartínSyntactic Structure Probability
Department of Computer Science and Technology, University of Cambridge, United Kingdom
Deep networks learn to parse uniform-depth context-free languages from local statistics
Jan 31, 2026Jack T. Parley, Francesco Cagnetta, Matthieu WyartContext-Free Grammars Deep Network
Institute of Physics, École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland · Theoretical and Scientific Data Science, SISSA, Trieste, Italy · Department of Physics and Astronomy, Johns Hopkins University, Baltimore, Maryland.

Jan 31, 2023, cs.CLJ/K move · Enter open · S save

Universal Topological Regularity of Syntactic Structures

Fermín Moscoso del Prado Martín

Department of Language and Communication & Center for Language Studies Radboud University Erasmuslaan 1, 6525 NL, Nijmegen, The Netherlands

Syntactic Structure Phylogenetic Inference Regularity Topology Dependency Graph Distributional Information+1

Open paper AskPDF arXiv

Despite their widespread use, the principles governing the organisation of syntactic dependency trees remain poorly understood. I analyse dependency trees from 124 typologically, genetically, and geographically diverse languages. Their topology departs systematically from randomness. Relative to uniformly sampled random trees, dependency trees exhibit greater structural robustness and lower branching heterogeneity. I propose that these universal regularities emerge naturally from incremental grammatical encoding. I model this process using sublinear preferential attachment. The model accurately reproduces the observed topology. More generally, the results demonstrate how a universal statistical property of syntax can emerge from a simple, cognitively motivated generative process. They further illustrate a broader principle of efficiency by construction: communicatively efficient syntactic structures can emerge without direct optimisation for communication.

Abstract

Explore similar work

Universal Topological Regularity of Syntactic Structures

A Data-free Universal Prior over Syntactic Structures

Deep networks learn to parse uniform-depth context-free languages from local statistics