Contributions to the hierarchy of probabilistic languages

Lothar Sebastian Krapp; Remo Nitschke

cs.FLSep 20, 2026

Contributions to the hierarchy of probabilistic languages

Authors: Lothar Sebastian Krapp, Remo Nitschke

Abstract

We reconsider the theory of probabilistic formal languages generated by n-gram models and by probabilistic context-free grammars (PCFGs). The expected hierarchy of probabilistic grammars is established by proving that every probabilistic language generated by an n-gram model is also generated by some PCFG, while some probabilistic languages generated by PCFGs cannot be generated by any $n$ -gram model. We introduce the notion of fully connected PCFGs, namely PCFGs in Chomsky normal form where every production rule only involving non-terminals has non-zero probability. Our main result shows that any probabilistic language generated by an $n$ -gram model differs from any probabilistic language generated by a fully connected PCFG. Therefore, the class of probabilistic languages generated by $n$ -gram models is not a subset of the class generated by fully connected PCFGs.

Explore similar work

The Expressivity Boundary of Probabilistic Circuits: A Comparison with Large Language Models
May 13, 2026Zhiyu Zhao, Xuejie Liu, Muhan Zhang +1Expressivity Autoregressive Language Models
Implicit Representations of Grammaticality in Language Models
May 6, 2026Yingshan Susan Wang, Linlu Qiu, Zhaofeng Wu +2Grammaticality Text Corpora
Linear representations of grammaticality in neural language models
Jul 16, 2026Jane Li, Najoung KimGrammaticality Syntactic Structure

May 13, 2026 · cs.LGJ/K move · Enter open · S save

The Expressivity Boundary of Probabilistic Circuits: A Comparison with Large Language Models

Zhiyu Zhao, Xuejie Liu, Muhan Zhang, Anji Liu

School of Computing, National University of Singapore · School of Intelligence Science and Technology, Peking University · Institute for Artificial Intelligence, Peking University

Probabilistic Circuits (PCs) are deep generative models that support exact and efficient probabilistic inference. Yet in autoregressive language modeling, PCs still lag behind Transformer-based large language models (LLMs), suggesting an important expressivity gap. In this work, we compare PCs and LLMs under a unified autoregressive formulation. First, an output bottleneck: PCs parameterize predictions as convex combinations in probability space, which struggles to represent the sharp distributions typical of language; adopting a logit-space parameterization substantially narrows this gap. Second, a context-encoding bottleneck: we prove that structured-decomposable PCs can match Transformer separation rank on vtree-aligned partitions, but show, both theoretically and empirically, that this capacity is limited to partitions aligned with the fixed routing structure, leading to severe degradation when the data exhibits heterogeneous dependency topologies. We further prove that decomposable PCs are strictly more expressive than structured-decomposable ones, though effectively optimizing them remains an open challenge.

Expressivity Autoregressive Language Models Generative Models Encode

Open paper Ask about it PDF

Abstract

Explore similar work

The Expressivity Boundary of Probabilistic Circuits: A Comparison with Large Language Models

Implicit Representations of Grammaticality in Language Models

Linear representations of grammaticality in neural language models