Stacking-Based Weighting for Large Language Bayes in M-Open Settings
Organizations: Manning College of Information and Computer Sciences, University of Massachusetts Amherst, Amherst, MA, USA.
Abstract
Large Language Bayes (LLB) transforms a natural language problem description into a collection of candidate probabilistic programs and performs inference within each model, combining their predictions using Bayesian model averaging. While this provides a flexible framework for probabilistic inference without manual model specification, the resulting candidate model set is often misspecified in practice, and likelihood-based aggregation can concentrate on a single model, under-representing uncertainty across the sampled models. We propose a stacking-based aggregation method for LLB that selects model weights by optimizing predictive performance rather than marginal likelihood, while preserving the original language-model-driven model generation and within-model inference pipeline. Because leave-one-out predictive quantities are intractable in latent-variable models, we derive a variational lower bound that yields a tractable ELBO-based stacking objective. Across real-world probabilistic inference tasks, our method achieves comparable predictive performance to Bayesian model averaging while yielding more dispersed model weights, reducing concentration on a single candidate model under misspecification.
Figures & tables
| Dataset | BMA top weights | Stacking top weights |
|---|---|---|
| Hurricane | ||
| Inland Flood | ||
| Wildfire |
| Dataset | LLB (BMA) | Stacking |
|---|---|---|
| Hurricane | -5.983 | -5.999 |
| Inland Flood | -370.340 | -370.340 |
| Wildfire | -177.922 | -177.897 |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Structured Inference with Large Language Gibbs
ANCHOR: Abductive Network Construction with Hierarchical Orchestration for Reliable Probability Inference in Large Language Models
unknown'' predictions, while expanding factors increases noise and spurious correlations, weakening conditional independence and degrading reliability. To address these limitations, we propose Anchor, an aggregated Bayesian inference framework over a hierarchical factor space. It constructs dense factor hierarchies through iterative generation and clustering, maps contexts via hierarchical retrieval and refinement, and augments Naïve Bayes with a Causal Bayesian Network to model latent factor dependencies. Experiments show that Anchor markedly reduces unknown'' predictions and produces more reliable probability estimates than direct LLM baselines, achieving state-of-the-art performance while significantly reducing time and token overhead.