cs.LGNov 2, 2026

Stacking-Based Weighting for Large Language Bayes in M-Open Settings

Authors: Anant Bhide, Akarsh Gupta, Edmond Cunningham, Justin Domke

Organizations: Manning College of Information and Computer Sciences, University of Massachusetts Amherst, Amherst, MA, USA.

Abstract

Large Language Bayes (LLB) transforms a natural language problem description into a collection of candidate probabilistic programs and performs inference within each model, combining their predictions using Bayesian model averaging. While this provides a flexible framework for probabilistic inference without manual model specification, the resulting candidate model set is often misspecified in practice, and likelihood-based aggregation can concentrate on a single model, under-representing uncertainty across the sampled models. We propose a stacking-based aggregation method for LLB that selects model weights by optimizing predictive performance rather than marginal likelihood, while preserving the original language-model-driven model generation and within-model inference pipeline. Because leave-one-out predictive quantities are intractable in latent-variable models, we derive a variational lower bound that yields a tractable ELBO-based stacking objective. Across real-world probabilistic inference tasks, our method achieves comparable predictive performance to Bayesian model averaging while yielding more dispersed model weights, reducing concentration on a single candidate model under misspecification.

Figures & tables

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Structured Inference with Large Language Gibbs

    Jun 17, 2026Sanghyeok Choi, Henry Gouk, Esmeralda S. WhitammerMarkov Chain Monte CarloBayesian Network Structure Learning

  2. ANCHOR: Abductive Network Construction with Hierarchical Orchestration for Reliable Probability Inference in Large Language Models

    May 11, 2026Wentao Qiu, Guanran Luo, Zhongquan Jian +3Abductive ReasoningBayesian Networks