Language Model Ensembles

Follow this topic Ask about this topic

Language model ensembles study how to combine multiple models so their complementary strengths can improve answers, predictions, or evaluations. Common approaches include majority and weighted voting, learned stacking and calibration, query-level routing, logit fusion, and deliberation or verification across agents. These methods are tested on tasks such as mathematical reasoning, question answering, forecasting, text classification, and evaluation by LLM judges, with attention to accuracy, uncertainty, and inference cost. A central tension is that ensembles help most when models make different errors, yet correlated failures are common and adding models or computation does not reliably outperform the strongest individual model.

Latest papers

137 papers17 papers in the last four weeks, level with the four weeks before. 0.1% of all new papers.

Disparity Laundering: Consensus Without Fairness in Clinical Multi-Agent Systems
Oct 28, 2026Xiaoyang Wang, Tianrui Wang, Christopher C. YangDemographic Bias in Language Models Multi-Agent LLMs
Drexel University · University of Michigan
Improving Data-Constrained Scaling with Hyperparameter Transfer and Ensembling
Oct 27, 2026Ethan Baron, Shmuel Berman, Anirudh Satheesh +3Language Model Scaling Laws Hyperparameter Transfer
New York University · Princeton University · Q Labs
Multiple Importance Sampling for Guided Speculative Inference
Oct 23, 2026Joseph Rowan, Ashish J. KhistiLLM Inference Acceleration Inference-Time Guidance
Department of Electrical and Computer Engineering, University of Toronto
Circuits of LLM Classifiers
Oct 21, 2026Hadi Khalaf, Sajani Vithana, Kunal Talreja +3Large Language Model-Based Text Classification Language Model Ensembles
Harvard School of Engineering and Applied Science · Kempner Institute for the Study of Natural & Artificial Intelligence · School of Electrical and Computer Engineering, Georgia Institute of Technology +2
Best of Both Worlds: Fine-Tuned Small Models Complement LLMs for Cost and Performance Benefits
Oct 16, 2026Shivam Mathur, Sri Harish Popuri, Priya Krishnan Sundararajan +2Text Classification LLM Inference Efficiency
Walmart Inc.
Two Sample Sizes in Test-Time LLM Ensembling: Buffered ERM and Minimax Theory
Oct 15, 2026Zhi Zhang, Arash A. AminiLanguage Model Ensembles Test-Time Scaling
Department of Statistics and Data Science, UCLA
Judging the Judges: Confidence and Disagreement in Rubrics as Rewards
Oct 15, 2026Michael Tian-Yue Liu, Anne Harrington, Trevor Darrell +1Rubric-Based RL Reward Design for RL
UC Berkeley
A Jury, Not an Electorate: Label-Free Reliability Estimation for Ensembles of Language Models
Oct 15, 2026Alessandro Rozza, Davide Mambelli, Steven Motta +1Language Model Ensembles Ensemble Learning
distributed cognition
Where Do LLM Ensemble Gains Come From? Decomposing Candidate Value, Diversity, and Aggregation
Oct 13, 2026Donghwan KimLanguage Model Ensembles LLM Evaluation
Independent Researcher
JudgeMoE: Distributional Aggregation for LLM-as-a-Judge
Oct 5, 2026Yiqi Liu, Joseph James, Yang Wang +3LLM Evaluation LLM-as-a-Judge
The University of Manchester · The University of Sheffield · University of Pittsburgh +1
Collective Bias Mitigation via Model Routing and Collaboration
Oct 2, 2026Mingzhe Du, Luu Anh Tuan, Xiaobao Wu +7Social Bias in Language Models Demographic Bias in Language Models
Nanyang Technological University · National University of Singapore · Harbin Institute of Technology +1
Enhancing Biomedical Named Entity Recognition via Multiple Programming Languages Instruction Tuning and Ensemble Method
Oct 2, 2026Songtao Li, Yijia Zhang, Jianyuan Yuan +3Named Entity Recognition LLM Fine-Tuning
School of Information Science and Technology, Dalian Maritime University, Dalian, China · School of Automation, Beijing Institute of Technology, Beijing, China · School of Computer Science and Engineering, Northeastern University, Shenyang, China +1
From Discovery to Decision: Finite-Budget Recoverability in LLM Voting
Oct 1, 2026Shaoang Li, Jian LiLLM Inference Efficiency LLM Inference
Stony Brook University
Adapter Thickets: Splitting an RLVR Budget Beats Concentrating It
Oct 1, 2026Jonathan Williams, Esin Tureci, Karthik R. NarasimhanRL for Language Model Reasoning Low-Rank Adaptation
Princeton University
Beyond Leaderboards: Tokenomics of Agentic Small Language Model Ensembles
Oct 1, 2026Alexei N. Skurikhin, Emily M. Taylor, Nathan A. DeBardelebenLLM Inference Efficiency Language Model Generation Evaluation
Los Alamos National Laboratory Los Alamos, New Mexico, USA
RAIM: Robust Aggregation of Inexpensive Models for Hallucination Detection
Sep 30, 2026Elia Onofri, Roberto Di PietroLLM Evaluation LLM-as-a-Judge
Computer, Electrical and Mathematical Sciences and Engineering (CEMSE) Division, King Abdullah University of Science and Technology (KAUST) Thuwal 23955, Saudi Arabia
Reach Into The CHOIR: Free-List Elicitation Uncovers Distinct Model Voices in LLM Ensembles
Sep 29, 2026Ben Wigler, Maria TsfasmanLanguage Model Ensembles Language Model Probing
LoveMind AI, New York, USA
Which Models Work Well Together? Measuring Heterogeneity for LLM Team Selection
Sep 29, 2026Liangyu Teng, Hengsong Liu, Juncen Guo +4Language Model Ensembles Ensemble Diversity
Fudan Institute on Networking Systems of AI
MERGE: Multi-LLM Ensemble for Retrieval via Generative Enrichment
Sep 29, 2026Tzu-I Ho, Yung-Yu Shih, Shang-Yu Su +2LLM Prompting Automatic Prompt Optimization
University of Waterloo · National Taiwan University · Rakuten Group, Inc. +1
Two Heads Are Better Than One: Aggregating Weaker LLMs for Better Forecasts
Sep 27, 2026Cheng Peng, Ruixi Luo, Zhi Chen +1Language Model Ensembles Probabilistic Forecasting
Chinese University of Hong Kong
Beyond Symmetric Agents: Cognitive Diversity and Multi-Agent Debate in Small Language Models
Sep 26, 2026Leonardo Ferreira, Gardenia Liu, Kaden ZhengMulti-Agent Debate Multi-Agent Systems
Boston Children’s Hospital, Harvard Medical School · Harvard School of Engineering and Applied Sciences, Harvard College
Multi-agent Scaling Across Disjunctive and Compensatory Tasks
Sep 25, 2026Carolina Fortuna, Blaz BertalanicMulti-Agent LLM Systems Multi-Agent Systems
Modular Norm RandOpt: Population-Efficient Ensembling through Architecture-Aware Perturbations
Sep 22, 2026Kirato Yoshihara, Hiroaki HamadeLanguage Model Ensembles
The University of Osaka
Agreement Overstates Evidence: Error Dependence in LLM Judge Consensus
Sep 18, 2026Elias Hossain, Niloofar Yousefi, Ser-Nam LimLLM-as-a-Judge LLM Reliability
University of Central Florida
The Wisdom of Artificial Deliberative Crowds
Sep 18, 2026Federico Barrera-Lemarchand, Mariano Sigman, Joaquin NavajasMulti-Agent Deliberation Multi-Agent Consensus
Laboratorio de Neurociencia, Escuela de Negocios, Universidad Torcuato Di Tella, Buenos Aires, Argentina · Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET), Buenos Aires, Argentina
How Many Humans Are 32 LLM Judges Worth?
Sep 18, 2026Chao Li, Yingying Yu, Yunfeng LiAnnotator Disagreement LLM-as-a-Judge
Tsinghua University · University College London · Hebei University of Economics and Business
Mo' Models, Mo' Problems: How to best select model pools when designing Multi-Agent Systems
Sep 15, 2026Sara Vera Marjanović, Jiacheng Xu, Aleksandr Laptev +3Model Selection Multi-Agent LLM Systems
University of Copenhagen · Work done during internship at NVIDIA · NVIDIA
The Universe of Universes: Benefit Yield Functions, Implosion Thresholds, and Infrastructure-Aware Optimization in Multi-LLM Systems
Sep 14, 2026Danielle Franklin, Vasu Raj JainLanguage Model Ensembles Multi-LLM Collaboration
Humanity + AI, Inc.
OrchSLM: Probing the Dynamics of Small Language Model Orchestration
Sep 11, 2026Chengxi Zhang, Yu YaoLLM Agent Orchestration Small Language Models
MIT
LLM-Enhanced Dual-Branch Learning for Large-Scale Multi-Label Text Classification
Sep 11, 2026Hui Ye, Jing Zhang, Xiulong Yang +1Late Fusion Multi-Label Classification
Department of Computer Science, Georgia State University, Atlanta, GA 30303, USA · Amazon, San Diego, CA, USA · School of Computer Science, Central China Normal University, Wuhan 430079, China

Oct 28, 2026J/K move · Enter open · S save

Disparity Laundering: Consensus Without Fairness in Clinical Multi-Agent Systems

Xiaoyang Wang, Tianrui Wang, Christopher C. Yang

Drexel University · University of Michigan

Demographic Bias in Language Models Multi-Agent LLMs Language Model Bias Evaluation Language Model Ensembles Counterfactual Fairness

Open paper AskPDF arXiv

Clinical large language models are increasingly fielded as multi-agent teams, on the assumption that deliberation averages out individual error. It does not average out bias. Across a full factorial of five model families and four aggregation topologies on 120 real MIMIC-IV emergency cases under counterfactual demographic swaps, no configuration neutralizes the disparity: all 20 cells commit the direction of their agents' independent answers, giving Black patients more acute triage. Every model has a topology that amplifies its baseline gap, but which one varies by model, so no design is reliably safe or reliably amplifying: the disparity is relocated, never removed. Late-stage fixes fail: fairness instructions and demographic masking leave the gap untouched, and per-agent self-debiasing widens it, because the bias is already encoded in the agents' ordinary clinical numbers. We call this disparity laundering; the audits now applied to clinical AI, per model, per output, per version, can certify the parts but never the committed decision. Interventions close the gap exactly when they make counterfactual versions visible inside the system and pool them, through deliberation, into the committed decision. A diverse-perspective ensemble does both while keeping the team, with no retraining or architectural change: the disparity falls by 67 to 94% where all five perspectives pool at every round, and the one topology that never pools them is the one it cannot reliably repair. Medical multi-agent systems do not filter demographic bias; they launder it into plausible decisions no single-version audit can catch, and the only defense is to make the bias visible.