Ensembling Language Models with Sequential Monte Carlo

Robin Shing Moon Chan; Tianyu Liu; Samuel Kiegeland; Clemente Pasti; Jacob Hoover Vigly; Timothy J. O'Donnell; Ryan Cotterell; Tim Vieira

cs.CLMar 5, 2026

Ensembling Language Models with Sequential Monte Carlo

Authors: Robin Shing Moon Chan, Tianyu Liu, Samuel Kiegeland, Clemente Pasti, Jacob Hoover Vigly, Timothy J. O'Donnell, Ryan Cotterell, Tim Vieira

Organizations: ETH Zürich · CHI-FRO · McGill University · Mila ·

Abstract

Practitioners have access to an abundance of language models and prompting strategies for solving many language modeling tasks; yet prior work shows that modeling performance is highly sensitive to both choices. Classical machine learning ensembling techniques offer a principled approach: aggregate predictions from multiple sources to achieve better performance than any single one. However, applying ensembling to language models during decoding is challenging: naively aggregating next-token probabilities yields samples from a locally normalized, biased approximation of the generally intractable ensemble distribution over strings. In this work, we introduce a unified framework for composing $K$ language models into $f$ -ensemble distributions for a wide range of functions $f\colon\mathbb{R}_{\geq 0}^{K}\to\mathbb{R}_{\geq 0}$ . To sample from these distributions, we propose a byte-level sequential Monte Carlo (SMC) algorithm that operates in a shared character space, enabling ensembles of models with mismatching vocabularies and consistent sampling in the limit. We evaluate a family of $f$ -ensembles across prompt and model combinations for various structured text generation tasks, highlighting the benefits of alternative aggregation strategies over traditional probability averaging, and showing that better posterior approximations can yield better ensemble performance.

Figures & tables

Figure 1: Prompt intersection on GPT-2. We generate 200 strings from the normalized token-level product of “ My favorite physicist is ” and “ My favorite author is ” prompts, then score each under the local ensemble $p_{\mathrm{ens,local}}$ (string probabilities emerging from locally normalized token-level product ensembling) and global ensemble $p_{\mathrm{ens,global}}$ (normalized product of string probabilities). Left: scatter plot of log probabilities with color indicating the generated strings’ global probability relative to their local ones. We also plot completions from the author prompt ( ) and the physicist prompt ( ) that don’t overlap with the ensemble’s completions ( ). Right: Correlation with the explicitly formulated intersection constraint probability $p(\cdot\mid\text{``{My favorite physicist {and} author is}''})$ .

	$\tau$	Name	${\color[rgb]{0,0,0}f}$ -Ensemble	Minimized $D_{\alpha}$
	$-\infty$	Minimum	$\frac{1}{Z}\min_{k=1}^{K}{\color[rgb]{0,0,0}p}_{k}({{\color[rgb]{0,0,0}\bm{x}}})$	$D_{\infty}({\color[rgb]{0,0,0}\Phi{}}\mid\mid{\color[rgb]{0,0,0}p}_{k})$
consensus	$-1$	Harmonic	$\frac{1}{Z}\left(\sum_{k=1}^{K}\frac{w_{k}}{{\color[rgb]{0,0,0}p}_{k}({{\color[rgb]{0,0,0}\bm{x}}})}\right)^{-1}$	$D_{\chi^{2}}({\color[rgb]{0,0,0}\Phi{}}\mid\mid{\color[rgb]{0,0,0}p}_{k})$
consensus	$\phantom{-}0$	Product	$\frac{1}{Z}\prod_{k=1}^{K}{\color[rgb]{0,0,0}p}_{k}({{\color[rgb]{0,0,0}\bm{x}}})^{w_{k}}$	$D_{\mathrm{KL}}({\color[rgb]{0,0,0}\Phi{}}\mid\mid{\color[rgb]{0,0,0}p}_{k})$
[1pt/2pt]	$\phantom{-}1$	Sum	$\frac{1}{Z}\sum_{k=1}^{K}w_{k}{\color[rgb]{0,0,0}p}_{k}({{\color[rgb]{0,0,0}\bm{x}}})$	$D_{\mathrm{KL}}({\color[rgb]{0,0,0}p}_{k}\mid\mid{\color[rgb]{0,0,0}\Phi{}})$
coverage	$\phantom{-}2$	Quadratic	$\frac{1}{Z}\sqrt{\sum_{k=1}^{K}w_{k}{\color[rgb]{0,0,0}p}_{k}({{\color[rgb]{0,0,0}\bm{x}}})^{2}}$	$D_{\chi^{2}}({\color[rgb]{0,0,0}p}_{k}\mid\mid{\color[rgb]{0,0,0}\Phi{}})$
	$+\infty$	Maximum	$\frac{1}{Z}\max_{k=1}^{K}{\color[rgb]{0,0,0}p}_{k}({{\color[rgb]{0,0,0}\bm{x}}})$	$D_{\infty}({\color[rgb]{0,0,0}p}_{k}\mid\mid{\color[rgb]{0,0,0}\Phi{}})$

Table 1: The generalized means family of

{\color[rgb]{0,0,0}f}

-ensembles. The ensemble

{\color[rgb]{0,0,0}\Phi{}}

minimizes

\sum_{k=1}^{K}w_{k}D_{\alpha}({\color[rgb]{0,0,0}\Phi{}}\mid\mid{\color[rgb]{0,0,0}p}_{k})

. The divergences are defined as: Pearson

\chi^{2}

D_{\chi^{2}}({\color[rgb]{0,0,0}p}\mid\mid{\color[rgb]{0,0,0}q})\mathrel{\stackrel{{\scriptstyle\textnormal{\tiny def}}}{{=}}}\sum_{{{\color[rgb]{0,0,0}\bm{x}}}\in{{\color[rgb]{0,0,0}{{\color[rgb]{0,0,0}{{\color[rgb]{0,0,0}\Sigma}}^{\ast}}}}}}\frac{({\color[rgb]{0,0,0}p}({{\color[rgb]{0,0,0}\bm{x}}})-{\color[rgb]{0,0,0}q}({{\color[rgb]{0,0,0}\bm{x}}}))^{2}}{{\color[rgb]{0,0,0}q}({{\color[rgb]{0,0,0}\bm{x}}})}

; KL:

D_{\mathrm{KL}}({\color[rgb]{0,0,0}p}\mid\mid{\color[rgb]{0,0,0}q})\mathrel{\stackrel{{\scriptstyle\textnormal{\tiny def}}}{{=}}}\sum_{{{\color[rgb]{0,0,0}\bm{x}}}\in{{\color[rgb]{0,0,0}{{\color[rgb]{0,0,0}{{\color[rgb]{0,0,0}\Sigma}}^{\ast}}}}}}{\color[rgb]{0,0,0}p}({{\color[rgb]{0,0,0}\bm{x}}})\log\frac{{\color[rgb]{0,0,0}p}({{\color[rgb]{0,0,0}\bm{x}}})}{{\color[rgb]{0,0,0}q}({{\color[rgb]{0,0,0}\bm{x}}})}

; and Rényi-

\infty

D_{\infty}({\color[rgb]{0,0,0}p}\mid\mid{\color[rgb]{0,0,0}q})\mathrel{\stackrel{{\scriptstyle\textnormal{\tiny def}}}{{=}}}\log\max_{{{\color[rgb]{0,0,0}\bm{x}}}\in{{\color[rgb]{0,0,0}{{\color[rgb]{0,0,0}{{\color[rgb]{0,0,0}\Sigma}}^{\ast}}}}}}\frac{{\color[rgb]{0,0,0}p}({{\color[rgb]{0,0,0}\bm{x}}})}{{\color[rgb]{0,0,0}q}({{\color[rgb]{0,0,0}\bm{x}}})}

	JSON Schema			BBH (Word Sorting)			SPIDER (Text-to-SQL)
Method	Q wen	P hi	L lama	Q wen	P hi	L lama	Q wen	P hi	L lama
Prompt 1	$95.6_{\pm 0.8}$	$83.5_{\pm 1.9}$	$86.3_{\pm 1.6}$	$13.9_{\pm 1.0}$	$27.7_{\pm 1.2}$	$38.6_{\pm 1.5}$	$40.0_{\pm 1.5}$	$24.2_{\pm 1.5}$	$38.4_{\pm 1.8}$
Prompt 2	$93.8_{\pm 0.9}$	$79.5_{\pm 5.6}$	$84.3_{\pm 2.5}$	$5.8_{\pm 0.9}$	$24.3_{\pm 1.5}$	$36.5_{\pm 1.9}$	$53.6_{\pm 1.0}$ †	$34.5_{\pm 1.7}$	$36.3_{\pm 1.7}$
Local Prob. Avg.	$95.2_{\pm 3.2}$	$82.0_{\pm 4.7}$	$83.4_{\pm 3.5}$	$9.6_{\pm 3.0}$	$25.8_{\pm 5.8}$	$41.2_{\pm 6.5}$	$45.6_{\pm 3.7}$	$28.4_{\pm 4.8}$	$39.4_{\pm 6.3}$
Best Local ${\color[rgb]{0,0,0}f}$ -Ens.	$96.4_{\pm 0.7}$ †	$87.2_{\pm 4.4}$	$88.6_{\pm 4.4}$	$15.0_{\pm 2.3}$	$27.2_{\pm 5.1}$	$45.0_{\pm 2.9}$ †	$52.6_{\pm 1.4}$	$36.0_{\pm 3.4}$ †	$43.6_{\pm 3.4}$ †
[1pt/2pt] Best Global ${\color[rgb]{0,0,0}f}$ -Ens. (Token SMC, $M=10$ )	$96.0_{\pm 0.2}$	$87.4_{\pm 0.3}$	$88.9_{\pm 1.1}$	$15.3_{\pm 2.0}$	$28.2_{\pm 1.1}$	$42.3_{\pm 0.9}$	$51.2_{\pm 1.8}$	$34.2_{\pm 0.7}$	$41.9_{\pm 1.4}$

Table 2: Summary of model ensembling results. Bold : non-overlapping 95% CIs with best baseline. † : highest average expected accuracy by column (within-model), or overall (cross-model). Shaded : strongest single-prompt baseline.

Figure 2: ${\color[rgb]{0,0,0}f}$ -ensembles for token-level SMC. Values show change in accuracy (%) relative to the best single-prompt baseline. † : best in column. Bold : significant improvement over the best base prompt (non-overlapping 95% CIs). Q=Qwen, P=Phi, L=Llama.

Figure 3: Representative relationship between expected accuracy and the estimated log marginal likelihood,

\log\widehat{Z}

, on BBH for Llama. Each datapoint represents (sample, particle) configurations aggregated across seeds. Both metrics are

z

-scored per example to normalize for variations in problem difficulty and scale. Points are colored by the number of particles used for estimation. Marginal plots display the kernel density estimates for the respective axes. The grey line and shaded band indicate a linear regression fit with a 95% confidence interval. Pearson correlation coefficients

\rho

are annotated in the top-left (

{}^{***}p<0.001

{}^{*}p<0.05

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

			Token SMC			Byte SMC
Model	Base Model	Local Ensemble	$M{=}10$	$M{=}25$	$M{=}75$	$M{=}10$
Llama-3.1-8B	0.76	2.34	15.24	32.88	104.30	63.45
Phi-4	1.42	3.84	23.27	59.90	167.59	166.76
Qwen2.5-7B	0.79	2.96	18.89	45.64	124.50	47.52

Appendix

Table 3: Average per-instance runtime (seconds) on JSON. Token SMC and Byte SMC use

M

particles.

	JSON Schema			BBH (Word Sorting)			SPIDER (Text-to-SQL)
Method	Qwen	Phi	Llama	Qwen	Phi	Llama	Qwen	Phi	Llama
Prompt 1	$95.6_{\pm 0.8}$	$83.5_{\pm 1.9}$	$86.3_{\pm 1.6}$	$13.9_{\pm 1.0}$	$27.7_{\pm 1.2}$	$38.6_{\pm 1.5}$	$40.0_{\pm 1.5}$	$24.2_{\pm 1.5}$	$38.4_{\pm 1.8}$
Prompt 2	$93.8_{\pm 0.9}$	$79.5_{\pm 5.6}$	$84.3_{\pm 2.5}$	$5.8_{\pm 0.9}$	$24.3_{\pm 1.5}$	$36.5_{\pm 1.9}$	$53.6_{\pm 1.0}$	$34.5_{\pm 1.7}$	$36.3_{\pm 1.7}$
$\min$	$95.7_{\pm 0.4}$	$87.4_{\pm 0.3}$ †	$88.9_{\pm 1.1}$	$15.3_{\pm 2.0}$	$28.2_{\pm 1.1}$	$42.3_{\pm 0.9}$ †	$51.2_{\pm 1.8}$	$34.2_{\pm 0.7}$	$41.9_{\pm 1.4}$ †
Product	$96.0_{\pm 0.2}$	$86.4_{\pm 1.0}$	$88.5_{\pm 1.0}$	$15.2_{\pm 1.4}$	$27.6_{\pm 1.2}$	$40.9_{\pm 0.7}$ †	$50.9_{\pm 1.1}$	$30.8_{\pm 0.1}$	$41.7_{\pm 1.7}$ †
Mixture	$95.3_{\pm 0.8}$	$81.3_{\pm 0.5}$	$85.1_{\pm 1.3}$	$9.8_{\pm 1.0}$	$26.5_{\pm 0.0}$	$37.4_{\pm 0.7}$	$45.9_{\pm 0.6}$	$29.4_{\pm 0.1}$	$38.2_{\pm 1.6}$

Appendix

Table 4: Comparison of

{\color[rgb]{0,0,0}f}

-ensemble functions (token-level SMC). Bold : highest average expected accuracy in column. † : non-overlapping 95% CIs with best baseline. Shaded : strongest single-prompt baseline.

Panel of Table 5 — Table 5: Table reporting expected ensemble and base model accuracy at equal model weights, 10 particles with 95% confidence intervals evaluated across 5 seeds. For cross-model ensembling, we choose the prompt which has the best overall average base performance across models.

Figure 4: Scatter plots comparing expected accuracy of two prompt templates for all datasets and models. Color indicates product ensemble (token-level SMC) expected accuracy. Points with higher alpha indicate cases where the ensemble exceeds the best single template. Marker size scales with the improvement margin.

Figure 5: Particle study for JSON. We evaluate the approximation quality estimate

\mathbb{E}[\log\widehat{Z}]

as a function of the number of particles being used for SMC.

Figure 6: Particle study for BBH. We evaluate the approximation quality estimate

\mathbb{E}[\log\widehat{Z}]

as a function of the number of particles being used for SMC.

Figure 7: Particle study for SPIDER. We evaluate the approximation quality estimate

\mathbb{E}[\log\widehat{Z}]

as a function of the number of particles being used for SMC.

	BBH (Word Sorting)			SPIDER (Text-to-SQL)			JSON Schema
Method	Qwen	Phi	Llama	Qwen	Phi	Llama	Qwen	Phi	Llama
$\min$	$0.01_{\pm 0.07}$	$0.15_{\pm 0.04}$ ∗∗∗	$0.22_{\pm 0.04}$ ∗∗∗	$0.04_{\pm 0.03}$	$-0.03_{\pm 0.03}$	$0.09_{\pm 0.03}$ ∗∗	$0.08_{\pm 0.03}$ ∗	$0.12_{\pm 0.03}$ ∗∗∗	$0.12_{\pm 0.02}$ ∗∗∗
Product	$0.03_{\pm 0.06}$	$0.18_{\pm 0.04}$ ∗∗∗	$0.28_{\pm 0.04}$ ∗∗∗	$0.08_{\pm 0.05}$	$0.09_{\pm 0.05}$	$0.16_{\pm 0.04}$ ∗∗∗	$0.16_{\pm 0.03}$ ∗∗∗	$0.21_{\pm 0.02}$ ∗∗∗	$0.31_{\pm 0.03}$ ∗∗∗
Sum	$0.03_{\pm 0.03}$	$0.00_{\pm 0.03}$	$-0.05_{\pm 0.02}$ ∗	$-0.03_{\pm 0.03}$	$-0.03_{\pm 0.02}$	$-0.02_{\pm 0.02}$	$0.03_{\pm 0.03}$	$-0.01_{\pm 0.02}$	$-0.02_{\pm 0.02}$
$\max$	$-0.07_{\pm 0.06}$	$-0.01_{\pm 0.04}$	$-0.14_{\pm 0.04}$ ∗∗∗	$-0.05_{\pm 0.03}$	$-0.09_{\pm 0.03}$ ∗∗	$0.04_{\pm 0.02}$	$-0.05_{\pm 0.04}$	$-0.05_{\pm 0.02}$ ∗	$-0.01_{\pm 0.02}$

Appendix

Table 6: Average per-sample Spearman correlation (

r

) between

\log\widehat{Z}

and expected accuracy. Values are reported as the mean of sample-level correlations

\pm

standard error. Significance levels (

{}^{***}p<0.001

{}^{**}p<0.01

{}^{*}p<0.05

) are determined via a one-sample

t

-test against zero. Note that these values deviate from those in Figure 3 , Figure 8 , Figure 10 , Figure 9 because they are computed per sample and then averaged.

Figure 8: Relationship between expected accuracy and the estimated log marginal likelihood, $\log\widehat{Z}$ , on JSON for all models. Each datapoint represents (sample, particle) configurations aggregated across seeds. Both metrics are $z$ -scored per example to normalize for variations in problem difficulty and scale. Points are colored by the number of particles used for estimation. Marginal plots display the kernel density estimates for the respective axes. The grey line and shaded band indicate a linear regression fit with a 95% confidence interval. Pearson correlation coefficients $\rho$ are annotated in the top-left ( ${}^{***}p<0.001$ , ${}^{*}p<0.05$ ).

Figure 9: Relationship between expected accuracy and the estimated log marginal likelihood, $\log\widehat{Z}$ , on BBH for all models. Each datapoint represents (sample, particle) configurations aggregated across seeds. Both metrics are $z$ -scored per example to normalize for variations in problem difficulty and scale. Points are colored by the number of particles used for estimation. Marginal plots display the kernel density estimates for the respective axes. The grey line and shaded band indicate a linear regression fit with a 95% confidence interval. Pearson correlation coefficients $\rho$ are annotated in the top-left ( ${}^{***}p<0.001$ , ${}^{*}p<0.05$ ).

Figure 10: Relationship between expected accuracy and the estimated log marginal likelihood, $\log\widehat{Z}$ , on SPIDER for all models. Each datapoint represents (sample, particle) configurations aggregated across seeds. Both metrics are $z$ -scored per example to normalize for variations in problem difficulty and scale. Points are colored by the number of particles used for estimation. Marginal plots display the kernel density estimates for the respective axes. The grey line and shaded band indicate a linear regression fit with a 95% confidence interval. Pearson correlation coefficients $\rho$ are annotated in the top-left ( ${}^{***}p<0.001$ , ${}^{*}p<0.05$ ).

Language Model Ensembles

Language model ensembles combine predictions or reasoning from multiple models to improve accuracy, reliability, coverage, or efficiency.

See all 134 papers

May 1, 2026

Rethinking LLM Ensembling from the Perspective of Mixture Models

Model ensembling is a well-established technique for improving the performance of machine learning models. Conventionally, this involves averaging the output distributions of multiple models and selecting the most probable label. This idea has been naturally extended to large language models (LLMs), yielding improved performance but incurring substantial computational cost. This inefficiency stems from directly applying conventional ensemble implementation to LLMs, which require a separate forward pass for each model to explicitly compute the ensemble distribution. In this paper, we propose the Mixture-model-like Ensemble (ME). By reinterpreting the ensemble as a mixture model, ME stochastically selects a single model at each step to generate the next token, thereby avoiding the need to explicitly compute the full ensemble distribution. ME is mathematically equivalent to sampling from the ensemble distribution, but requires invoking only one model, making it 1.78x-2.68x faster than conventional ensembling. Furthermore, this perspective connects LLM ensembling and token-level routing methods, suggesting that LLM ensembling is a special case of routing methods. Our findings open new avenues for efficient LLM ensembling and motivate further exploration of token-level routing strategies for LLMs. Our code is available at https://github.com/Kamichanw/Mixture-model-like-Ensemble.

Jiale Fu, Yuchu Jiang, Peijun Wu +3

Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education · Southeast University · Qiyuan Tech +2

LLM Inference Acceleration LLM Routing Language Model Ensembles

Ask PDF

Apr 17, 2026

Faster LLM Inference via Sequential Monte Carlo

Speculative decoding (SD) accelerates language model inference by drafting tokens from a cheap proposal model and verifying them against an expensive target model via rejection sampling. Because rejection truncates the draft block at the first error, throughput degrades when draft and target diverge. Rather than rejecting draft tokens outright, we propose to reweight them. To this end, we introduce sequential Monte Carlo speculative decoding (SMC-SD), which replaces token-level rejection with importance-weighted resampling over a population of draft particles. SMC-SD is a principled approximate inference scheme that trades exactness for additional speed, while preserving theoretical bounds on its per-step approximation error. Because LLM inference is memory bandwidth-bound, the arithmetic needed to draft particles and to score them in parallel comes nearly for free -- SMC-SD uses idle compute to turn verification into a vectorized, fixed-size operation with no rollback. Empirically, SMC-SD achieves 2.36x speed-up over speculative decoding and a 5.2x speed-up over autoregressive decoding, while remaining within 3% of the target model's accuracy on reasoning, instruction-following, and coding benchmarks.

Yahya Emara, Mauricio Barba da Costa, Chi-Chih Chang +4

Cornell University · Makora · MIT +1

Speculative Decoding LLM Inference Acceleration

Ask PDF

Oct 19, 2026

Keep the Lineages Alive: Diversity-Preserving SMC for Diffusion LLM Decoding

Inference-time sequential Monte Carlo (SMC) decoding can improve the reasoning accuracy of masked diffusion language models (diffusion LLMs) without post-training by resampling a population of partial generations according to a reward. Its effectiveness therefore depends on the resampling step, which must concentrate computation on promising candidates while retaining enough diversity to explore alternative reasoning paths. The reward can be internal, the decoder's own confidence on the committed prefix, or external, a process reward model scoring that prefix. Under either reward, multinomial resampling can disrupt this balance by duplicating high-weight particles and discarding lineages that may still reach a correct solution. The effective sample size (ESS) does not register this loss: multinomial resampling resets it to its maximum even as distinct lineages disappear. To retain such lineages, we introduce lineage-preserving SMC (LP-SMC), which combines Chopthin resampling with an input weight floor. Instead of equalizing weights, Chopthin bounds the ratio between the largest and smallest weights it returns and carries the unequal weights forward. The floor limits how far below the maximum a weight may fall before resampling so that lineages with low early weight keep a chance to recover. Across mathematical-reasoning and code-generation benchmarks on two base models, LP-SMC raises oracle coverage, the fraction of problems with at least one correct particle, in every setting, by 1.8 to 25.6 points, and raises end-to-end accuracy under a common reward-model selector in eleven of twelve settings, by up to 7.9 points on LLaDA-1.5 and 18.3 on Dream-7B. The coverage gain persists under the external reward model, which suggests that diversity-preserving resampling and the guidance signal contribute separately to inference-time search.

Minoo Ahmadi, Seyedarmin Azizi, Mehdi Kamal +1

University of Southern California, Los Angeles, USA

Language Model Decoding Diffusion Language Model Decoding Diffusion Language Model Inference LLM Reasoning

Ask PDF

Abstract

Language Model Ensembles

Explore similar work

Rethinking LLM Ensembling from the Perspective of Mixture Models

Faster LLM Inference via Sequential Monte Carlo

Keep the Lineages Alive: Diversity-Preserving SMC for Diffusion LLM Decoding