HI-MoE: Hierarchical Instance-Conditioned Mixture-of-Experts for Object Detection

Vadim Vashkelis; Natalia Trukhina

cs.LGApr 6, 2026

HI-MoE: Hierarchical Instance-Conditioned Mixture-of-Experts for Object Detection

Authors: Vadim Vashkelis, Natalia Trukhina

Abstract

Mixture-of-Experts (MoE) architectures enable conditional computation by activating only a subset of model parameters for each input. Although sparse routing has been highly effective in language models and has also shown promise in vision, most vision MoE methods operate at the image or patch level. This granularity is poorly aligned with object detection, where the fundamental unit of reasoning is an object query corresponding to a candidate instance. We propose Hierarchical Instance-Conditioned Mixture-of-Experts (HI-MoE), a DETR-style detection architecture that performs routing in two stages: a lightweight scene router first selects a scene-consistent expert subset, and an instance router then assigns each object query to a small number of experts within that subset. This design aims to preserve sparse computation while better matching the heterogeneous, instance-centric structure of detection. In the current draft, experiments are concentrated on COCO with preliminary specialization analysis on LVIS. Under these settings, HI-MoE improves over a dense DINO baseline and over simpler token-level or instance-only routing variants, with especially strong gains on small objects. We also provide an initial visualization of expert specialization patterns. We present the method, ablations, and current limitations in a form intended to support further experimental validation.

Figures & tables

Panel of Figure 1 — Figure 1: HI-MoE overview. A scene router first selects a scene-consistent expert subset; an instance router then performs per-query top- $K$ routing inside that subset. Sparse experts replace selected transformer FFNs.

Symbol	Meaning
$N_{e}$	Number of instance-level experts in each MoE block
$K$	Number of active instance experts per query
$N_{s}$	Number of scene-level routing logits
$K_{s}$	Number of active scene routes selected by the scene router
$q_{i}$	$i$ -th object query in the decoder
$x_{\mathrm{global}}$	Global scene descriptor produced from encoder features

Table 1: Notation summary for the hierarchical routing module.

Method	AP	AP s	Params
DETR	42.0	20.5	41M
Deformable DETR	48.7	29.6	40M
DINO	51.3	32.1	50M
HI-MoE	53.0	35.4	52M

Table 3: Current COCO validation results reported in this draft. Under the matched setup used here, HI-MoE improves over the dense DINO baseline while adding approximately 2M parameters.

Variant	Routing	AP	AP s	Params (M)	GFLOPs
Dense DINO	Dense FFN	51.3	32.1	50	280
Token-MoE	Flat token-level	52.1	33.5	52	285
Instance-only	Query router only	52.6	34.2	52	282
Scene-only	Scene router + shared experts	52.4	33.8	52	283
HI-MoE (full)	Scene + query routing	53.0	35.4	52	282

Table 4: Ablation on COCO val. The hierarchical scene+query routing variant performs best among the tested sparse alternatives.

top- $K$	AP	GFLOPs	Latency (ms, V100)
1	52.4	270	28
2	53.0	282	32
4	53.5	310	38

Table 5: Compute–accuracy trade-off when varying the number of active experts per query.

Experts	AP	Params
4	52.1	51M
8	52.6	52M
16	53.0	52M

Table 6: Ablation over the number of experts in the current setup.

Placement	AP	Params
Encoder only	52.3	52M
Decoder only	52.8	52M
Encoder + Decoder	53.0	52M

Table 7: Placement ablation for HI-MoE blocks. Decoder-side routing is strongest among single-stage placements, while the combined placement performs best overall.

Expert ID	Small (AP s )	Occluded (AP occ )	Tail (AP tail )	Dominant scene route (within-expert share)
E1	28.5	22.1	15.3	Crowd (45% of E1 assignments)
E3	36.2	29.4	12.1	Indoor (32% of E3 assignments)
E6	24.1	31.2	28.7	Outdoor (51% of E6 assignments)
Avg	31.2	27.3	18.4	–

Table 8: Preliminary expert-level statistics on LVIS. For each displayed expert, the Dominant scene route column reports the scene-route category most frequently associated with that expert, together with the within-expert proportion of routed assignments falling into that category. Because these percentages are normalized separately for each expert, they are not expected to sum to 100% across experts. These results are suggestive of specialization, but they should be complemented by larger-scale routing diagnostics in a future version.

Figure 2: Visualization derived from the expert-level routing statistics in Table 8 . Left: per-expert subset AP for representative experts and the average row. Right: for each displayed expert, the proportion of that expert’s routed assignments associated with its dominant scene-route category (Crowd for E1, Indoor for E3, Outdoor for E6). These percentages are normalized independently per expert and therefore are not additive and are not expected to sum to 100% across the three bars. This figure is intended as a first-step illustration of specialization rather than a complete utilization analysis across all experts and layers.

Object Detection

Object detection locates and classifies objects in images or video, balancing localization accuracy, robustness, and inference speed.

See all 624 papers

Domain-Specialized Object Detection via Model-Level Mixtures of Experts
Apr 20, 2026Svetlana Pavlitska, Malte Stüven, Beyza Keskin +1Object Detection Mixture-of-Experts Models
FZI Research Center for Information Technology · Karlsruhe Institute of Technology (KIT)
When Does Sparse MoE Help in Vision? The Role of Backbone Compute Leverage in Sparse Routing
May 15, 2026Libo Sun, Po-wei Harn, Peixiong He +1Sparse Mixture-of-Experts Expert Routing
Department of Computer Science and Software Engineering Auburn University, Auburn, AL 36849, USA · Department of Information Management National Central University, Taoyuan 320317, Taiwan · Department of Computer Science and Software Engineering Auburn University
Distilled Large Language Model-Driven Dynamic Sparse Expert Activation Mechanism
Mar 21, 2026Qinghui Chen, Zekai Zhang, Zaigui Zhang +5Industrial Anomaly Detection Industrial Inspection
School of Control Science and Engineering, Shandong University, Jinan, China · Laoshan Laboratory, Qingdao, China · Jinan Inspur Data Technology Co., Ltd, Jinan, China +2

Apr 20, 2026J/K move · Enter open · S save

Domain-Specialized Object Detection via Model-Level Mixtures of Experts

Svetlana Pavlitska, Malte Stüven, Beyza Keskin, J. Marius Zöllner

FZI Research Center for Information Technology · Karlsruhe Institute of Technology (KIT)

Object Detection Mixture-of-Experts Models

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) models provide a structured approach to combining specialized neural networks and offer greater interpretability than conventional ensembles. While MoEs have been successfully applied to image classification and semantic segmentation, their use in object detection remains limited due to challenges in merging dense and structured predictions. In this work, we investigate model-level mixtures of object detectors and analyze their suitability for improving performance and interpretability in object detection. We propose an MoE architecture that combines YOLO-based detectors trained on semantically disjoint data subsets, with a learned gating network that dynamically weights expert contributions. We study different strategies for fusing detection outputs and for training the gating mechanism, including balancing losses to prevent expert collapse. Experiments on the BDD100K dataset demonstrate that the proposed MoE consistently outperforms standard ensemble approaches and provides insights into expert specialization across domains, highlighting model-level MoEs as a viable alternative to traditional ensembling for object detection. Our code is available at https://github.com/KASTEL-MobilityLab/mixtures-of-experts/.

Abstract

Object Detection

Explore similar work

Domain-Specialized Object Detection via Model-Level Mixtures of Experts

When Does Sparse MoE Help in Vision? The Role of Backbone Compute Leverage in Sparse Routing

Distilled Large Language Model-Driven Dynamic Sparse Expert Activation Mechanism