Reliable Fusion of Conflicting Experts

Pranuthi Tenali; Sahil Sidheekh; Saurabh Mathur; Vijayalakshmi Saravanan; Erik Blasch; Kristian Kersting; Sriraam Natarajan

cs.LGAug 21, 2026

Reliable Fusion of Conflicting Experts

Authors: Pranuthi Tenali, Sahil Sidheekh, Saurabh Mathur, Vijayalakshmi Saravanan, Erik Blasch, Kristian Kersting, Sriraam Natarajan

Organizations: The University of Texas at Dallas · TU Darmstadt · The University of Texas at Tyler · Air Force Research Lab

Abstract

We study the problem of aggregating opinions from multiple black-box experts in noisy, conflict-prone settings where expert reliability varies across inputs. Static aggregation methods, such as majority voting, fail to capture this variability and often yield unreliable outcomes under disagreement. We propose a tractable, probabilistic-circuit-based fusion framework that dynamically combines expert responses using context-specific credibility estimates, enabling principled and reliable reasoning. The framework is agnostic to the underlying experts and does not require access to their internal representations or any retraining. We empirically validate our approach on multiple-choice question answering tasks using multiple LLMs as experts, comparing against individual models and static ensemble baselines. Our method consistently improves predictive performance and produces more reliable decisions under conflict, highlighting the effectiveness of context-aware credibility modeling for robust multi-expert fusion.

Figures & tables

Figure 1: Our framework dynamically aggregates opinions from an ensemble of black-box experts with distinct, varying specializations. Given an input query $x$ , the context is broadcast to an ensemble of $N$ black-box experts with distinct specializations, each producing a response $y_{i}$ . A probabilistic circuit receives all expert responses alongside the input context and computes context-specific credibility weights $(\pi_{1},\dots,\pi_{N}).$ The weighted combination yields a fused decision $\hat{Y}_{\text{final}}.$

Experiment	Accuracy	AUROC	Precision	Recall	F1 Score
Medical	$55.40\scriptstyle\pm 7.20$	$76.11\scriptstyle\pm 3.69$	$57.22\scriptstyle\pm 7.02$	$55.40\scriptstyle\pm 7.20$	$54.87\scriptstyle\pm 8.42$
CS	$62.13\scriptstyle\pm 5.62$	$78.35\scriptstyle\pm 3.40$	$62.47\scriptstyle\pm 6.28$	$62.13\scriptstyle\pm 5.62$	$61.87\scriptstyle\pm 5.79$
Physics	$60.29\scriptstyle\pm 7.38$	$75.74\scriptstyle\pm 5.68$	$57.22\scriptstyle\pm 7.02$	$60.29\scriptstyle\pm 7.38$	$59.83\scriptstyle\pm 7.55$
SOE	$67.16\scriptstyle\pm 2.04$	$80.19\scriptstyle\pm 0.94$	$68.05\scriptstyle\pm 2.15$	$67.16\scriptstyle\pm 2.04$	$66.59\scriptstyle\pm 2.45$
RE	$62.62\scriptstyle\pm 2.89$	$79.16\scriptstyle\pm 1.72$	$63.18\scriptstyle\pm 3.11$	$62.62\scriptstyle\pm 2.89$	$62.48\scriptstyle\pm 3.18$
NoisyOR	$65.93\scriptstyle\pm 2.99$	$84.30\scriptstyle\pm 1.82$	$67.63\scriptstyle\pm 2.45$	$67.06\scriptstyle\pm 2.87$	$65.35\scriptstyle\pm 3.16$

Table 1: Mean Test Performance on MMLU across individual subject experts (Medical, CS, Physics), the Single-Oracle Expert (SOE) skyline, the Random Expert Selection (RE) baseline, and fusion methods, averaged over

3

trials. Fusion methods include non-context-aware approaches (WM, Noisy-OR, DPC, CWM), and context-aware approaches (C 2 DPC and C 2 WM). The context-aware methods are competitive across all the metrics, with C 2 WM achieving the highest Precision and Recall, and C 2 DPC the highest AUROC while additionally yielding interpretable per-expert context-specific credibility scores.

Panel of Figure 2 — Figure 2: Expected Calibration Error (ECE) on the test set across individual subject experts (Physics, CS, Medical) and different fusion methods averaged over $3$ trials. Fusion methods include non-context-aware approaches (WM, Noisy-OR, DPC, CWM), and context-aware approaches (C 2 DPC, C 2 WM). The context-aware C 2 DPC method is the best calibrated, achieving the lowest ECE.

Language Model Ensembles

Language model ensembles combine predictions or reasoning from multiple models to improve accuracy, reliability, coverage, or efficiency.

See all 137 papers

DLLG: Dynamic Logit-Level Gating of LLM Experts
Jun 3, 2026Bingnan Li, Zhaoyang Zhang, Xiaoze Liu +6Ensemble Learning Language Model Ensembles
AWS Agentic AI, Seattle, US · University of California San Diego, La Jolla, US · Purdue University, West Lafayette, US
A Jury, Not an Electorate: Label-Free Reliability Estimation for Ensembles of Language Models
Oct 15, 2026Alessandro Rozza, Davide Mambelli, Steven Motta +1Language Model Ensembles Ensemble Learning
distributed cognition
Context-specific Credibility-aware Multimodal Fusion with Conditional Probabilistic Circuits
Mar 27, 2026Pranuthi Tenali, Sahil Sidheekh, Saurabh Mathur +3Multimodal Robustness Uncertainty-Aware Multimodal Fusion
†The University of Texas at Dallas · §Department of Computer Science , TU Darmstadt · ‡Air Force Research Lab

Jun 3, 2026J/K move · Enter open · S save

DLLG: Dynamic Logit-Level Gating of LLM Experts

Bingnan Li, Zhaoyang Zhang, Xiaoze Liu, Yantao Shen+5

AWS Agentic AI, Seattle, US · University of California San Diego, La Jolla, US · Purdue University, West Lafayette, US

Ensemble Learning Language Model Ensembles LLM Reasoning

Open paper AskPDF arXiv

Leveraging multiple specialized LLMs can combine complementary strengths, but existing approaches trade adaptability for stability: routing commits prematurely, heuristic ensembling depends on fragile proxies, and parameter merging introduces interference. We propose DLLG (Dynamic Logit-Level Gating), a dynamic logit-level ensembling framework that learns token-level expert fusion from sparse response-level supervision. A lightweight gating module predicts step-wise fusion weights, linking trajectory-level correctness to generation without token-level labels or expert retraining. Across diverse reasoning and code benchmarks, DLLG consistently outperforms strong routing, heuristic ensembling, and parameter-merging baselines across model scales, highlighting learned logit-level fusion as a robust and scalable paradigm for integrating specialized experts.

Abstract

Language Model Ensembles

Explore similar work

DLLG: Dynamic Logit-Level Gating of LLM Experts

A Jury, Not an Electorate: Label-Free Reliability Estimation for Ensembles of Language Models

Context-specific Credibility-aware Multimodal Fusion with Conditional Probabilistic Circuits