Mixture-of-Models: Distributed Inference over Heterogeneous LLMs as Model-Level Mixture-of-Experts

Tims Pecerskis; Aivars Smirnovs

cs.AIJan 23, 2026

Mixture-of-Models: Distributed Inference over Heterogeneous LLMs as Model-Level Mixture-of-Experts

Authors: Tims Pecerskis, Aivars Smirnovs

Organizations: Peeramid Labs · The AI Futures Collective

Abstract

Switch Transformers scale MoE model capacity by routing tokens in memory for one jointly trained model, requiring high-bandwidth interconnects, large memory, centralised ownership, and lack scalable composability. We propose Mixture-of-Models (MoM) architecture as Switch Transformer scaling to semantic level which composes inference across independently trained distributed LLMs, each run on its own GPU. The models reach joint consensus using N-Way Self-Evaluating Deliberation protocol: a combination of social-choice methods such as Delphi method and Quadratic Voting as well as Inference-Time scaling paradigm. We start with an empirical model that characterizes consensus as an inference-time trade-off between signal extraction and contextual noise accumulation, establishing a mathematical basis for convergence driven inference stopping strategies. We show experimental results where ensemble of smaller (<21B) models matches closely 120B MoE (GPT-OSS-120B) on AIME 2025 while larger ensemble of open-weight models reaches 90.0%, close to proprietary models at time of writing.

Figures & tables

Panel of Figure 1 — Figure 1: NSED control flow architecture. The Client interacts with the Orchestrator , which queries the Broker for optimal team composition based on task complexity. The Orchestrator then manages the synchronous execution loop and global context.

Panel of Figure 2 — Figure 2: The NSED Neural Macro-Topology. Solid Blue Lines indicate actionable voting signals, while Dashed Gray Lines indicate read-only context. The Diagonal Mask $\mathbf{D}$ is applied at the activation stage, mathematically zeroing out self-votes. The internal layer represents ReAct agent loops that are performing sequential tasks of generating outputs and producing cross-evaluations. Side channel is not shown. The instantaneous output $y_{t}$ is appended to the Deliberation History $\mathcal{H}$ , which is subsequently processed by the Weighted Consensus function (Eq. 6 ) to derive the final solution $y^{*}$ .

Panel of Figure 3 — Figure 3: The Agentic Oracle Topology. Agents maintain two distinct interfaces: a Protocol Interface to the Orchestrator for peer-review, and a Side-Channel to the Environment for independent context retrieval. This decouples retrieval (individual) from reasoning (collective).

System / Architecture	Hardware Class	AIME (Pass@1)	LCB Hard (Pass@1)	Est. Cost/Sol
Baselines
Gemini-2.5-Pro-06-05	Enterprise	78.3%	62.0%	High
DeepSeek-R1 (RL-CoT)	Enterprise	84.2%	63.6%	Medium
Majority Voting (Qwen-8B)	Consumer	54.0%	33.1%	Low
NSED (Ours)
NSED (Consumer open-weight)	Consumer	84.0%	60.2%	Low

Table 1: Global Performance Summary. NSED (Consumer) utilizes only

<

20B parameter models on consumer hardware. NSED (High-Perf) utilizes 70B+ models.

Panel of Figure 4 — Figure 4: NSED ensemble performance overlaid with Thermodynamic Fit curves. The high correlation confirms that consensus accuracy is bounded by specific entropic fatigue coefficients.

Panel of Figure 5 — Figure 5: NSED vs. Naive Majority Voting (LCB Hard). The horizontal dashed line represents the raw Majority Vote baseline (33.09%).

Metric	Gemma-12b	Qwen-8b	GPT-OSS-20b	NSED-R1	NSED-R2	NSED-R3
User Retention	0.709	0.955	0.827	0.429	0.250	0.522
Sneaking	0.764	0.373	0.136	0.741	0.810	0.724
Brand Bias	0.336	0.245	0.500	0.353	0.280	0.250
Sycophancy	0.245	0.073	0.064	0.111	0.040	0.143
Harmful Gen	0.118	0.155	0.018	0.150	0.091	0.156
Anthropomorph	0.082	0.173	0.339	0.125	0.241	0.048

Table 2: DarkBench Safety Scores (Lower is Better). NSED-R2 achieves the lowest overall RMS score (0.285), demonstrating the corrective power of peer review.

Mixture-of-Experts Inference

Mixture-of-experts inference studies how to run sparsely activated expert models efficiently while preserving their quality.

See all 189 papers

Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns
Apr 25, 2026Abhimanyu Bambhaniya, Geonhwa Jeong, Jason Park +6Expert Load Balancing Mixture-of-Experts Inference
1Georgia Institute of Technology, USA · 2Meta Platforms, Inc.
Fast MoE Inference via Predictive Prefetching and Expert Replication
May 12, 2026Ankit Jyothish, Ali Jannesari, Aishwarya Sarkar +1Expert Load Balancing GPU Acceleration
Iowa State University Ames, USA
LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models
Aug 4, 2026Fengqi Zhu, Shaoxuan Xu, Jingyang Ou +11Mixture-of-Experts Language Models Language Model Scaling Laws
Gaoling School of AI, Renmin University of China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance · Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE +2

Apr 25, 2026J/K move · Enter open · S save

Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns

Abhimanyu Bambhaniya, Geonhwa Jeong, Jason Park, Jiecao Yu+5

1Georgia Institute of Technology, USA · 2Meta Platforms, Inc.

Expert Load Balancing Mixture-of-Experts Inference LLM Routing Mixture-of-Experts Models Distributed Inference Communication-Efficient Mixture-of-Experts

Open paper AskPDF arXiv

Most recent state-of-the-art (SOTA) large language models (LLMs) use Mixture-of-Experts (MoE) architectures to scale model capacity without proportional per-token compute, enabling higher-quality outputs at manageable serving costs. However, MoE inference at scale is fundamentally bottlenecked by expert load imbalance and inefficient token routing, especially in multi-node deployments where tokens are not guaranteed to be routed to local experts, resulting in significant inter-node all-to-all communication overhead. To systematically characterize these challenges, we profile SOTA open-source MoE models, including Llama 4 Maverick, DeepSeek V3-671B, and Qwen3-230B-A22B, on various datasets and collected over 100k real expert activation traces. Upon studying the expert activation patterns, we uncover various persistent properties across all the frontier MoE models: variable expert load imbalance, domain-specific expert activation where expert popularity shifts across task families (code, math, chat, general), and a strong correlation between prefill and decode expert activations. Motivated by these findings, we propose workload-aware micro-batch grouping and an expert placement strategy to maximize token locality to the destination expert, thereby reducing inter-node communication. Across models and datasets, these optimizations help reduce all2all communication data up to 20, resulting in lower MoE decode latency and better accelerator utilization.

Abstract

Mixture-of-Experts Inference

Explore similar work

Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns

Fast MoE Inference via Predictive Prefetching and Expert Replication

LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models