Markovian Generation Chains in Large Language Models

Mingmeng Geng; Amr Mohamed; Guokan Shang; Michalis Vazirgiannis; Thierry Poibeau

cs.CLMar 11, 2026

Markovian Generation Chains in Large Language Models

Authors: Mingmeng Geng, Amr Mohamed, Guokan Shang, Michalis Vazirgiannis, Thierry Poibeau

Organizations: ENS-PSL & CNRS-Lattice · MBZUAI & Ecole Polytechnique · MBZUAI

Abstract

The widespread use of large language models (LLMs) raises an important question: how do texts evolve when they are repeatedly processed by LLMs? In this paper, we define this iterative inference process as Markovian generation chains, where each step takes a specific prompt template and the previous output as input, without including any prior memory. In iterative rephrasing and round-trip translation experiments, the output either converges to a small recurrent set or continues to produce novel sentences over a finite horizon. Through sentence-level Markov chain modeling and analysis of simulated data, we show that iterative process can either increase or reduce sentence diversity depending on factors such as the temperature parameter and the initial input sentence. These results offer valuable insights into the dynamics of iterative LLM inference and their implications for multi-agent LLM systems.

Figures & tables

Figure 1: Illustration of iterative LLM reprocessing ( Markovian generation chains ). Each node denotes the output after iteration

i

. Under greedy decoding , chains typically enter fixed points or short cycles, limiting sentence-level diversity. Under sampling-based decoding , stochasticity may yield longer transients and more distinct outputs

Round	Qwen2.5-7B-Instruct	Llama-3.1-8B-Instruct
0	We begin with a prologue.	We begin with a prologue.
1	We start with a prologue.	The story commences with a prologue.
2	We begin with a prologue.	The narrative begins with a prologue.
3	We start with a prologue.	The story commences with a prologue.
4	We begin with a prologue.	The narrative begins with a prologue.
5	We start with a prologue.	The story commences with a prologue.

Table 1: Illustrative greedy-decoding trajectories under iterative rephrasing. Qwen2.5-7B enters a 2-cycle after one step, while Llama-3.1-8B alternates between two near-paraphrases with different lexical realizations.

Figure 2: Average number of unique paraphrases generated over 50 iterative rephrasings across three datasets, comparing four instruction-tuned LLMs: GPT-4o-mini, Llama-3.1-8B, Mistral-7B, and Qwen-2.5-7B. Results are shown for greedy decoding (orange) and sampling-based decoding (purple). Error bars represent one standard deviation.

Figure 3: Evolution of text similarity metrics across 50 rephrasing iterations for the BookSum dataset using greedy decoding. Each iteration compares the current rephrased text against the previous iteration’s text as reference.

Model (Decoding)	BookSum	ScriptBase	News2024
Llama (greedy)	0.17	-0.17	0.19
Llama (sampling)	0.17	0.03	0.11
Mistral (greedy)	0.32	0.07	0.11
Mistral (sampling)	0.44	-0.02	0.20
Qwen (greedy)	0.35	0.09	0.20
Qwen (sampling)	0.49	0.16	0.37

Figure 4: Pearson correlation

r

between seed length (words) and the number of distinct outputs over

T=50

iterations.

Figure 5: Number of distinct sentences produced over 50 iterative rephrasings with GPT-4o-mini under different settings. P1 and P2 correspond to Listings 1 and 2 in the Appendix. Boxes denote the interquartile range, and the center line indicates the median.

Figure 6: Prompt heterogeneity across iterations ( GPT-4o-mini , sampling-based decoding). We compare a single fixed prompt against alternating prompts, and a mixed variant constructed from prompt halves.

Sentence	Frequency
Hooray for best friends!	50
I don’t think so.	50
You’re the ones being banished!"	50
Sue now has two children and is expecting a third.	46
Suddenly, the ghost appears.	41
Jo continues to write short romantic stories for the newspaper while secretly working on a novel.	34

Table 2: Highest recurrence frequency observed sentences within the 50-iteration paragraph-level runs ( GPT-4o-mini , sampling-based decoding, prompt P1) on 150 BookSum paragraphs.

Figure 7: Distinct-sentence counts under iterated round-trip translation for GPT-4o-mini (sampling-based decoding) and Google Translate (v3) .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Round	Llama-3.1-8B-Instruct	Qwen2.5-7B-Instruct
0	We begin with a prologue.	We begin with a prologue.
1	The narrative commences with a prologue.	We start with a prologue.
2	The story begins with a prologue that sets the scene.	We begin with a prologue.
3	The narrative commences with a prologue that establishes the backdrop for the story.	We start with a prologue.
5	The story starts with a prologue, which establishes the setting and lays the groundwork for the events that unfold in the narrative.	We start with a prologue.
10	The narrative begins with a prologue, which sets the scene and lays the groundwork for the events that unfold in the story that follows.	We begin with a prologue.

Appendix

Table 3: Examples of repeated rephrasing by different LLMs (temperature=0.7, top-p=0.9).

Round	French	English
0		We begin with a prologue.
1	Nous commençons par un prologue.	We begin with a prologue.
2	Nous commençons par un prologue.	We begin with a prologue.
3	Nous commençons par un prologue.	We begin with a prologue.
4	Nous commençons par un prologue.	We begin with a prologue.

Appendix

Table 4: Examples of iterative translation via GPT-4o-mini (temperature=0.7, top-p=0.9).

Round	French	English
0		Elizabeth reads through Darcy’s letter with a mixture of emotions.
1	Elizabeth lit la lettre de Darcy avec un mélange d’émotions.	Elizabeth read Darcy’s letter with a mix of emotions.
2	Elizabeth a lu la lettre de Darcy avec un mélange d’émotions.	Elizabeth read Darcy’s letter with a mix of emotions.
⋮	⋮	⋮
11	Elizabeth lut la lettre de Darcy avec un mélange d’émotions.	Elizabeth read Darcy’s letter with a mixture of emotions.
12	Elizabeth a lu la lettre de Darcy avec un mélange d’émotions.	Elizabeth read Darcy’s letter with a mixture of emotions.

Appendix

Table 5: Examples of iterative translation via GPT-4o-mini (temperature=0.7, top-p=0.9). The omitted lines are all identical to the content in the second round (the italicized sentences). The similar scenario also occurs after the 19th round.

Figure 8: Evolution of text similarity metrics across 50 rephrasing iterations for the News2024 dataset using greedy decoding. Each iteration compares the current rephrased text against the previous iteration’s text as reference.

Figure 9: Evolution of text similarity metrics across 50 rephrasing iterations for the ScriptBase dataset using greedy decoding. Each iteration compares the current rephrased text against the previous iteration’s text as reference.

Figure 10: Evolution of text similarity metrics across 50 rephrasing iterations for the BookSum dataset using sampling-based decoding. Each iteration compares the current rephrased text against the previous iteration’s text as reference.

Figure 11: Evolution of text similarity metrics across 50 rephrasing iterations for the News2024 dataset using sampling-based decoding. Each iteration compares the current rephrased text against the previous iteration’s text as reference.

Figure 12: Evolution of text similarity metrics across 50 rephrasing iterations for the ScriptBase dataset using sampling-based decoding. Each iteration compares the current rephrased text against the previous iteration’s text as reference.

Dataset	Model	r	p	R²	Slope
booksum	Llama greedy	0.171	3.611e-02	0.029	0.107
	Llama sample	0.166	4.272e-02	0.027	0.191
	Mistral greedy	0.324	5.135e-05	0.105	0.350
	Mistral sample	0.438	2.046e-08	0.192	0.594
	Qwen greedy	0.353	9.427e-06	0.125	0.125
	Qwen sample	0.494	1.330e-10	0.244	0.616

Appendix

Table 6: Results of the Correlation Analysis.

r

represents the Pearson correlation coefficient, and

R^{2}

represents the coefficient of determination.

LLM Inference

Large language model inference studies how models generate outputs efficiently, reliably, and securely across hardware and workloads.

See all 270 papers

The Probabilistic Structure of Large Language Models
Sep 20, 2026Adnan AboulalaâLanguage Modeling Autoregressive Language Modeling
Do LLMs Understand Sequential Structure? A Controlled Study of Inference and Generation
Oct 4, 2026Jerry Wang, Zhengxiang Wang, Ting Yu Liu +3Language Model Generation Evaluation Large Language Model-Based Social Simulation
University of Illinois Urbana-Champaign · Stony Brook University · National Chengchi University
Monitoring latent reasoning in looped language models
Oct 13, 2026Pawel Olszowiec, Paulina Tomaszewska, Maciej Żelaszczyk +2Implicit Reasoning in Language Models Recursive Language Models
Samsung AI Center, Warsaw, Poland · IFTR, Polish Academy of Sciences, Warsaw, Poland · University of Warsaw, Poland

Sep 20, 2026J/K move · Enter open · S save

The Probabilistic Structure of Large Language Models

Adnan Aboulalaâ

Language Modeling Autoregressive Language Modeling LLM Training

Open paper AskPDF arXiv

This paper presents a probabilistic perspective on large language models (LLMs), developed with the aim of bringing together, in a single self-contained account, tools that are usually treated separately across the literature. LLMs are described through probability measures on the set of sequences of tokens, specified via their autoregressive conditional distributions. Training is formulated as a maximum-likelihood estimation problem, addressed by stochastic gradient methods, while text generation is viewed as the sequential simulation of the resulting stochastic process. The role of the asymmetry of the Kullback--Leibler divergence in text generation is examined in relation with characteristic phenomena such as hallucination and the distinction between statistical plausibility and truth. As a complementary illustration of the same viewpoint, we also discuss diffusion models, built around the score function, which cast generation not as sequential token prediction but as the simulation of a reverse-time stochastic process transforming noise into data both in discrete and continuous time.

Abstract

LLM Inference

Explore similar work

The Probabilistic Structure of Large Language Models

Do LLMs Understand Sequential Structure? A Controlled Study of Inference and Generation

Monitoring latent reasoning in looped language models