Ensembling Language Models with Sequential Monte Carlo

Robin Shing Moon Chan; Tianyu Liu; Samuel Kiegeland; Clemente Pasti; Jacob Hoover Vigly; Timothy J. O'Donnell; Ryan Cotterell; Tim Vieira

cs.CLMar 5, 2026

Ensembling Language Models with Sequential Monte Carlo

Authors: Robin Shing Moon Chan, Tianyu Liu, Samuel Kiegeland, Clemente Pasti, Jacob Hoover Vigly, Timothy J. O'Donnell, Ryan Cotterell, Tim Vieira

Organizations: ETH Zürich · CHI-FRO · McGill University · Mila ·

Abstract

Practitioners have access to an abundance of language models and prompting strategies for solving many language modeling tasks; yet prior work shows that modeling performance is highly sensitive to both choices. Classical machine learning ensembling techniques offer a principled approach: aggregate predictions from multiple sources to achieve better performance than any single one. However, applying ensembling to language models during decoding is challenging: naively aggregating next-token probabilities yields samples from a locally normalized, biased approximation of the generally intractable ensemble distribution over strings. In this work, we introduce a unified framework for composing $K$ language models into $f$ -ensemble distributions for a wide range of functions $f\colon\mathbb{R}_{\geq 0}^{K}\to\mathbb{R}_{\geq 0}$ . To sample from these distributions, we propose a byte-level sequential Monte Carlo (SMC) algorithm that operates in a shared character space, enabling ensembles of models with mismatching vocabularies and consistent sampling in the limit. We evaluate a family of $f$ -ensembles across prompt and model combinations for various structured text generation tasks, highlighting the benefits of alternative aggregation strategies over traditional probability averaging, and showing that better posterior approximations can yield better ensemble performance.

Figures & tables

Figure 1: Prompt intersection on GPT-2. We generate 200 strings from the normalized token-level product of “ My favorite physicist is ” and “ My favorite author is ” prompts, then score each under the local ensemble $p_{\mathrm{ens,local}}$ (string probabilities emerging from locally normalized token-level product ensembling) and global ensemble $p_{\mathrm{ens,global}}$ (normalized product of string probabilities). Left: scatter plot of log probabilities with color indicating the generated strings’ global probability relative to their local ones. We also plot completions from the author prompt ( ) and the physicist prompt ( ) that don’t overlap with the ensemble’s completions ( ). Right: Correlation with the explicitly formulated intersection constraint probability $p(\cdot\mid\text{``{My favorite physicist {and} author is}''})$ .

	$\tau$	Name	${\color[rgb]{0,0,0}f}$ -Ensemble	Minimized $D_{\alpha}$
	$-\infty$	Minimum	$\frac{1}{Z}\min_{k=1}^{K}{\color[rgb]{0,0,0}p}_{k}({{\color[rgb]{0,0,0}\bm{x}}})$	$D_{\infty}({\color[rgb]{0,0,0}\Phi{}}\mid\mid{\color[rgb]{0,0,0}p}_{k})$
consensus	$-1$	Harmonic	$\frac{1}{Z}\left(\sum_{k=1}^{K}\frac{w_{k}}{{\color[rgb]{0,0,0}p}_{k}({{\color[rgb]{0,0,0}\bm{x}}})}\right)^{-1}$	$D_{\chi^{2}}({\color[rgb]{0,0,0}\Phi{}}\mid\mid{\color[rgb]{0,0,0}p}_{k})$
consensus	$\phantom{-}0$	Product	$\frac{1}{Z}\prod_{k=1}^{K}{\color[rgb]{0,0,0}p}_{k}({{\color[rgb]{0,0,0}\bm{x}}})^{w_{k}}$	$D_{\mathrm{KL}}({\color[rgb]{0,0,0}\Phi{}}\mid\mid{\color[rgb]{0,0,0}p}_{k})$
[1pt/2pt]	$\phantom{-}1$	Sum	$\frac{1}{Z}\sum_{k=1}^{K}w_{k}{\color[rgb]{0,0,0}p}_{k}({{\color[rgb]{0,0,0}\bm{x}}})$	$D_{\mathrm{KL}}({\color[rgb]{0,0,0}p}_{k}\mid\mid{\color[rgb]{0,0,0}\Phi{}})$
coverage	$\phantom{-}2$	Quadratic	$\frac{1}{Z}\sqrt{\sum_{k=1}^{K}w_{k}{\color[rgb]{0,0,0}p}_{k}({{\color[rgb]{0,0,0}\bm{x}}})^{2}}$	$D_{\chi^{2}}({\color[rgb]{0,0,0}p}_{k}\mid\mid{\color[rgb]{0,0,0}\Phi{}})$
	$+\infty$	Maximum	$\frac{1}{Z}\max_{k=1}^{K}{\color[rgb]{0,0,0}p}_{k}({{\color[rgb]{0,0,0}\bm{x}}})$	$D_{\infty}({\color[rgb]{0,0,0}p}_{k}\mid\mid{\color[rgb]{0,0,0}\Phi{}})$

Table 1: The generalized means family of

{\color[rgb]{0,0,0}f}

-ensembles. The ensemble

{\color[rgb]{0,0,0}\Phi{}}

minimizes

\sum_{k=1}^{K}w_{k}D_{\alpha}({\color[rgb]{0,0,0}\Phi{}}\mid\mid{\color[rgb]{0,0,0}p}_{k})

. The divergences are defined as: Pearson

\chi^{2}

D_{\chi^{2}}({\color[rgb]{0,0,0}p}\mid\mid{\color[rgb]{0,0,0}q})\mathrel{\stackrel{{\scriptstyle\textnormal{\tiny def}}}{{=}}}\sum_{{{\color[rgb]{0,0,0}\bm{x}}}\in{{\color[rgb]{0,0,0}{{\color[rgb]{0,0,0}{{\color[rgb]{0,0,0}\Sigma}}^{\ast}}}}}}\frac{({\color[rgb]{0,0,0}p}({{\color[rgb]{0,0,0}\bm{x}}})-{\color[rgb]{0,0,0}q}({{\color[rgb]{0,0,0}\bm{x}}}))^{2}}{{\color[rgb]{0,0,0}q}({{\color[rgb]{0,0,0}\bm{x}}})}

; KL:

D_{\mathrm{KL}}({\color[rgb]{0,0,0}p}\mid\mid{\color[rgb]{0,0,0}q})\mathrel{\stackrel{{\scriptstyle\textnormal{\tiny def}}}{{=}}}\sum_{{{\color[rgb]{0,0,0}\bm{x}}}\in{{\color[rgb]{0,0,0}{{\color[rgb]{0,0,0}{{\color[rgb]{0,0,0}\Sigma}}^{\ast}}}}}}{\color[rgb]{0,0,0}p}({{\color[rgb]{0,0,0}\bm{x}}})\log\frac{{\color[rgb]{0,0,0}p}({{\color[rgb]{0,0,0}\bm{x}}})}{{\color[rgb]{0,0,0}q}({{\color[rgb]{0,0,0}\bm{x}}})}

; and Rényi-

\infty

D_{\infty}({\color[rgb]{0,0,0}p}\mid\mid{\color[rgb]{0,0,0}q})\mathrel{\stackrel{{\scriptstyle\textnormal{\tiny def}}}{{=}}}\log\max_{{{\color[rgb]{0,0,0}\bm{x}}}\in{{\color[rgb]{0,0,0}{{\color[rgb]{0,0,0}{{\color[rgb]{0,0,0}\Sigma}}^{\ast}}}}}}\frac{{\color[rgb]{0,0,0}p}({{\color[rgb]{0,0,0}\bm{x}}})}{{\color[rgb]{0,0,0}q}({{\color[rgb]{0,0,0}\bm{x}}})}

	JSON Schema			BBH (Word Sorting)			SPIDER (Text-to-SQL)
Method	Q wen	P hi	L lama	Q wen	P hi	L lama	Q wen	P hi	L lama
Prompt 1	$95.6_{\pm 0.8}$	$83.5_{\pm 1.9}$	$86.3_{\pm 1.6}$	$13.9_{\pm 1.0}$	$27.7_{\pm 1.2}$	$38.6_{\pm 1.5}$	$40.0_{\pm 1.5}$	$24.2_{\pm 1.5}$	$38.4_{\pm 1.8}$
Prompt 2	$93.8_{\pm 0.9}$	$79.5_{\pm 5.6}$	$84.3_{\pm 2.5}$	$5.8_{\pm 0.9}$	$24.3_{\pm 1.5}$	$36.5_{\pm 1.9}$	$53.6_{\pm 1.0}$ †	$34.5_{\pm 1.7}$	$36.3_{\pm 1.7}$
Local Prob. Avg.	$95.2_{\pm 3.2}$	$82.0_{\pm 4.7}$	$83.4_{\pm 3.5}$	$9.6_{\pm 3.0}$	$25.8_{\pm 5.8}$	$41.2_{\pm 6.5}$	$45.6_{\pm 3.7}$	$28.4_{\pm 4.8}$	$39.4_{\pm 6.3}$
Best Local ${\color[rgb]{0,0,0}f}$ -Ens.	$96.4_{\pm 0.7}$ †	$87.2_{\pm 4.4}$	$88.6_{\pm 4.4}$	$15.0_{\pm 2.3}$	$27.2_{\pm 5.1}$	$45.0_{\pm 2.9}$ †	$52.6_{\pm 1.4}$	$36.0_{\pm 3.4}$ †	$43.6_{\pm 3.4}$ †
[1pt/2pt] Best Global ${\color[rgb]{0,0,0}f}$ -Ens. (Token SMC, $M=10$ )	$96.0_{\pm 0.2}$	$87.4_{\pm 0.3}$	$88.9_{\pm 1.1}$	$15.3_{\pm 2.0}$	$28.2_{\pm 1.1}$	$42.3_{\pm 0.9}$	$51.2_{\pm 1.8}$	$34.2_{\pm 0.7}$	$41.9_{\pm 1.4}$

Table 2: Summary of model ensembling results. Bold : non-overlapping 95% CIs with best baseline. † : highest average expected accuracy by column (within-model), or overall (cross-model). Shaded : strongest single-prompt baseline.

Figure 2: ${\color[rgb]{0,0,0}f}$ -ensembles for token-level SMC. Values show change in accuracy (%) relative to the best single-prompt baseline. † : best in column. Bold : significant improvement over the best base prompt (non-overlapping 95% CIs). Q=Qwen, P=Phi, L=Llama.

Figure 3: Representative relationship between expected accuracy and the estimated log marginal likelihood,

\log\widehat{Z}

, on BBH for Llama. Each datapoint represents (sample, particle) configurations aggregated across seeds. Both metrics are

z

-scored per example to normalize for variations in problem difficulty and scale. Points are colored by the number of particles used for estimation. Marginal plots display the kernel density estimates for the respective axes. The grey line and shaded band indicate a linear regression fit with a 95% confidence interval. Pearson correlation coefficients

\rho

are annotated in the top-left (

{}^{***}p<0.001

{}^{*}p<0.05

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

			Token SMC			Byte SMC
Model	Base Model	Local Ensemble	$M{=}10$	$M{=}25$	$M{=}75$	$M{=}10$
Llama-3.1-8B	0.76	2.34	15.24	32.88	104.30	63.45
Phi-4	1.42	3.84	23.27	59.90	167.59	166.76
Qwen2.5-7B	0.79	2.96	18.89	45.64	124.50	47.52

Appendix

Table 3: Average per-instance runtime (seconds) on JSON. Token SMC and Byte SMC use

M

particles.

	JSON Schema			BBH (Word Sorting)			SPIDER (Text-to-SQL)
Method	Qwen	Phi	Llama	Qwen	Phi	Llama	Qwen	Phi	Llama
Prompt 1	$95.6_{\pm 0.8}$	$83.5_{\pm 1.9}$	$86.3_{\pm 1.6}$	$13.9_{\pm 1.0}$	$27.7_{\pm 1.2}$	$38.6_{\pm 1.5}$	$40.0_{\pm 1.5}$	$24.2_{\pm 1.5}$	$38.4_{\pm 1.8}$
Prompt 2	$93.8_{\pm 0.9}$	$79.5_{\pm 5.6}$	$84.3_{\pm 2.5}$	$5.8_{\pm 0.9}$	$24.3_{\pm 1.5}$	$36.5_{\pm 1.9}$	$53.6_{\pm 1.0}$	$34.5_{\pm 1.7}$	$36.3_{\pm 1.7}$
$\min$	$95.7_{\pm 0.4}$	$87.4_{\pm 0.3}$ †	$88.9_{\pm 1.1}$	$15.3_{\pm 2.0}$	$28.2_{\pm 1.1}$	$42.3_{\pm 0.9}$ †	$51.2_{\pm 1.8}$	$34.2_{\pm 0.7}$	$41.9_{\pm 1.4}$ †
Product	$96.0_{\pm 0.2}$	$86.4_{\pm 1.0}$	$88.5_{\pm 1.0}$	$15.2_{\pm 1.4}$	$27.6_{\pm 1.2}$	$40.9_{\pm 0.7}$ †	$50.9_{\pm 1.1}$	$30.8_{\pm 0.1}$	$41.7_{\pm 1.7}$ †
Mixture	$95.3_{\pm 0.8}$	$81.3_{\pm 0.5}$	$85.1_{\pm 1.3}$	$9.8_{\pm 1.0}$	$26.5_{\pm 0.0}$	$37.4_{\pm 0.7}$	$45.9_{\pm 0.6}$	$29.4_{\pm 0.1}$	$38.2_{\pm 1.6}$

Appendix

Table 4: Comparison of

{\color[rgb]{0,0,0}f}

-ensemble functions (token-level SMC). Bold : highest average expected accuracy in column. † : non-overlapping 95% CIs with best baseline. Shaded : strongest single-prompt baseline.

Panel of Table 5 — Table 5: Table reporting expected ensemble and base model accuracy at equal model weights, 10 particles with 95% confidence intervals evaluated across 5 seeds. For cross-model ensembling, we choose the prompt which has the best overall average base performance across models.

Figure 4: Scatter plots comparing expected accuracy of two prompt templates for all datasets and models. Color indicates product ensemble (token-level SMC) expected accuracy. Points with higher alpha indicate cases where the ensemble exceeds the best single template. Marker size scales with the improvement margin.

Figure 5: Particle study for JSON. We evaluate the approximation quality estimate

\mathbb{E}[\log\widehat{Z}]

as a function of the number of particles being used for SMC.

Figure 6: Particle study for BBH. We evaluate the approximation quality estimate

\mathbb{E}[\log\widehat{Z}]

as a function of the number of particles being used for SMC.

Figure 7: Particle study for SPIDER. We evaluate the approximation quality estimate

\mathbb{E}[\log\widehat{Z}]

as a function of the number of particles being used for SMC.

	BBH (Word Sorting)			SPIDER (Text-to-SQL)			JSON Schema
Method	Qwen	Phi	Llama	Qwen	Phi	Llama	Qwen	Phi	Llama
$\min$	$0.01_{\pm 0.07}$	$0.15_{\pm 0.04}$ ∗∗∗	$0.22_{\pm 0.04}$ ∗∗∗	$0.04_{\pm 0.03}$	$-0.03_{\pm 0.03}$	$0.09_{\pm 0.03}$ ∗∗	$0.08_{\pm 0.03}$ ∗	$0.12_{\pm 0.03}$ ∗∗∗	$0.12_{\pm 0.02}$ ∗∗∗
Product	$0.03_{\pm 0.06}$	$0.18_{\pm 0.04}$ ∗∗∗	$0.28_{\pm 0.04}$ ∗∗∗	$0.08_{\pm 0.05}$	$0.09_{\pm 0.05}$	$0.16_{\pm 0.04}$ ∗∗∗	$0.16_{\pm 0.03}$ ∗∗∗	$0.21_{\pm 0.02}$ ∗∗∗	$0.31_{\pm 0.03}$ ∗∗∗
Sum	$0.03_{\pm 0.03}$	$0.00_{\pm 0.03}$	$-0.05_{\pm 0.02}$ ∗	$-0.03_{\pm 0.03}$	$-0.03_{\pm 0.02}$	$-0.02_{\pm 0.02}$	$0.03_{\pm 0.03}$	$-0.01_{\pm 0.02}$	$-0.02_{\pm 0.02}$
$\max$	$-0.07_{\pm 0.06}$	$-0.01_{\pm 0.04}$	$-0.14_{\pm 0.04}$ ∗∗∗	$-0.05_{\pm 0.03}$	$-0.09_{\pm 0.03}$ ∗∗	$0.04_{\pm 0.02}$	$-0.05_{\pm 0.04}$	$-0.05_{\pm 0.02}$ ∗	$-0.01_{\pm 0.02}$

Appendix

Table 6: Average per-sample Spearman correlation (

r

) between

\log\widehat{Z}

and expected accuracy. Values are reported as the mean of sample-level correlations

\pm

standard error. Significance levels (

{}^{***}p<0.001

{}^{**}p<0.01

{}^{*}p<0.05

) are determined via a one-sample

t

-test against zero. Note that these values deviate from those in Figure 3 , Figure 8 , Figure 10 , Figure 9 because they are computed per sample and then averaged.

Figure 8: Relationship between expected accuracy and the estimated log marginal likelihood, $\log\widehat{Z}$ , on JSON for all models. Each datapoint represents (sample, particle) configurations aggregated across seeds. Both metrics are $z$ -scored per example to normalize for variations in problem difficulty and scale. Points are colored by the number of particles used for estimation. Marginal plots display the kernel density estimates for the respective axes. The grey line and shaded band indicate a linear regression fit with a 95% confidence interval. Pearson correlation coefficients $\rho$ are annotated in the top-left ( ${}^{***}p<0.001$ , ${}^{*}p<0.05$ ).

Figure 9: Relationship between expected accuracy and the estimated log marginal likelihood, $\log\widehat{Z}$ , on BBH for all models. Each datapoint represents (sample, particle) configurations aggregated across seeds. Both metrics are $z$ -scored per example to normalize for variations in problem difficulty and scale. Points are colored by the number of particles used for estimation. Marginal plots display the kernel density estimates for the respective axes. The grey line and shaded band indicate a linear regression fit with a 95% confidence interval. Pearson correlation coefficients $\rho$ are annotated in the top-left ( ${}^{***}p<0.001$ , ${}^{*}p<0.05$ ).

Figure 10: Relationship between expected accuracy and the estimated log marginal likelihood, $\log\widehat{Z}$ , on SPIDER for all models. Each datapoint represents (sample, particle) configurations aggregated across seeds. Both metrics are $z$ -scored per example to normalize for variations in problem difficulty and scale. Points are colored by the number of particles used for estimation. Marginal plots display the kernel density estimates for the respective axes. The grey line and shaded band indicate a linear regression fit with a 95% confidence interval. Pearson correlation coefficients $\rho$ are annotated in the top-left ( ${}^{***}p<0.001$ , ${}^{*}p<0.05$ ).

Language Model Ensembles

Language model ensembles combine predictions or reasoning from multiple models to improve accuracy, reliability, coverage, or efficiency.

See all 134 papers

Rethinking LLM Ensembling from the Perspective of Mixture Models
May 1, 2026Jiale Fu, Yuchu Jiang, Peijun Wu +3LLM Inference Acceleration LLM Routing
Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education · Southeast University · Qiyuan Tech +2
Faster LLM Inference via Sequential Monte Carlo
Apr 17, 2026Yahya Emara, Mauricio Barba da Costa, Chi-Chih Chang +4Speculative Decoding LLM Inference Acceleration
Cornell University · Makora · MIT +1
Keep the Lineages Alive: Diversity-Preserving SMC for Diffusion LLM Decoding
Oct 19, 2026Minoo Ahmadi, Seyedarmin Azizi, Mehdi Kamal +1Language Model Decoding Diffusion Language Model Decoding
University of Southern California, Los Angeles, USA

May 1, 2026J/K move · Enter open · S save

Rethinking LLM Ensembling from the Perspective of Mixture Models

Jiale Fu, Yuchu Jiang, Peijun Wu, Chonghan Liu+2

Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education · Southeast University · Qiyuan Tech+2

LLM Inference Acceleration LLM Routing Language Model Ensembles

Open paper AskPDF arXiv

Model ensembling is a well-established technique for improving the performance of machine learning models. Conventionally, this involves averaging the output distributions of multiple models and selecting the most probable label. This idea has been naturally extended to large language models (LLMs), yielding improved performance but incurring substantial computational cost. This inefficiency stems from directly applying conventional ensemble implementation to LLMs, which require a separate forward pass for each model to explicitly compute the ensemble distribution. In this paper, we propose the Mixture-model-like Ensemble (ME). By reinterpreting the ensemble as a mixture model, ME stochastically selects a single model at each step to generate the next token, thereby avoiding the need to explicitly compute the full ensemble distribution. ME is mathematically equivalent to sampling from the ensemble distribution, but requires invoking only one model, making it 1.78x-2.68x faster than conventional ensembling. Furthermore, this perspective connects LLM ensembling and token-level routing methods, suggesting that LLM ensembling is a special case of routing methods. Our findings open new avenues for efficient LLM ensembling and motivate further exploration of token-level routing strategies for LLMs. Our code is available at https://github.com/Kamichanw/Mixture-model-like-Ensemble.

Abstract

Language Model Ensembles

Explore similar work

Rethinking LLM Ensembling from the Perspective of Mixture Models

Faster LLM Inference via Sequential Monte Carlo

Keep the Lineages Alive: Diversity-Preserving SMC for Diffusion LLM Decoding