cs.LGApr 6, 2026

HI-MoE: Hierarchical Instance-Conditioned Mixture-of-Experts for Object Detection

Authors: Vadim Vashkelis, Natalia Trukhina

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Mixture-of-Experts (MoE) architectures enable conditional computation by activating only a subset of model parameters for each input. Although sparse routing has been highly effective in language models and has also shown promise in vision, most vision MoE methods operate at the image or patch level. This granularity is poorly aligned with object detection, where the fundamental unit of reasoning is an object query corresponding to a candidate instance. We propose Hierarchical Instance-Conditioned Mixture-of-Experts (HI-MoE), a DETR-style detection architecture that performs routing in two stages: a lightweight scene router first selects a scene-consistent expert subset, and an instance router then assigns each object query to a small number of experts within that subset. This design aims to preserve sparse computation while better matching the heterogeneous, instance-centric structure of detection. In the current draft, experiments are concentrated on COCO with preliminary specialization analysis on LVIS. Under these settings, HI-MoE improves over a dense DINO baseline and over simpler token-level or instance-only routing variants, with especially strong gains on small objects. We also provide an initial visualization of expert specialization patterns. We present the method, ablations, and current limitations in a form intended to support further experimental validation.

Figures & tables

Figure 1: HI-MoE overview. A scene router first selects a scene-consistent expert subset; an instance router then performs per-query top- KK routing inside that subset. Sparse experts replace selected transformer FFNs.
SymbolMeaning
NeN_{e}Number of instance-level experts in each MoE block
KKNumber of active instance experts per query
NsN_{s}Number of scene-level routing logits
KsK_{s}Number of active scene routes selected by the scene router
qiq_{i}ii -th object query in the decoder
xglobalx_{\mathrm{global}}Global scene descriptor produced from encoder features
Table 1: Notation summary for the hierarchical routing module.
MethodAPAP sParams
DETR42.020.541M
Deformable DETR48.729.640M
DINO51.332.150M
HI-MoE53.035.452M
Table 3: Current COCO validation results reported in this draft. Under the matched setup used here, HI-MoE improves over the dense DINO baseline while adding approximately 2M parameters.
VariantRoutingAPAP sParams (M)GFLOPs
Dense DINODense FFN51.332.150280
Token-MoEFlat token-level52.133.552285
Instance-onlyQuery router only52.634.252282
Scene-onlyScene router + shared experts52.433.852283
HI-MoE (full)Scene + query routing53.035.452282
Table 4: Ablation on COCO val. The hierarchical scene+query routing variant performs best among the tested sparse alternatives.
top- KKAPGFLOPsLatency (ms, V100)
152.427028
253.028232
453.531038
Table 5: Compute–accuracy trade-off when varying the number of active experts per query.
ExpertsAPParams
452.151M
852.652M
1653.052M
Table 6: Ablation over the number of experts in the current setup.
PlacementAPParams
Encoder only52.352M
Decoder only52.852M
Encoder + Decoder53.052M
Table 7: Placement ablation for HI-MoE blocks. Decoder-side routing is strongest among single-stage placements, while the combined placement performs best overall.
Expert IDSmall (AP s )Occluded (AP occ )Tail (AP tail )Dominant scene route (within-expert share)
E128.522.115.3Crowd (45% of E1 assignments)
E336.229.412.1Indoor (32% of E3 assignments)
E624.131.228.7Outdoor (51% of E6 assignments)
Avg31.227.318.4–
Table 8: Preliminary expert-level statistics on LVIS. For each displayed expert, the Dominant scene route column reports the scene-route category most frequently associated with that expert, together with the within-expert proportion of routed assignments falling into that category. Because these percentages are normalized separately for each expert, they are not expected to sum to 100% across experts. These results are suggestive of specialization, but they should be complemented by larger-scale routing diagnostics in a future version.
Figure 2: Visualization derived from the expert-level routing statistics in Table 8 . Left: per-expert subset AP for representative experts and the average row. Right: for each displayed expert, the proportion of that expert’s routed assignments associated with its dominant scene-route category (Crowd for E1, Indoor for E3, Outdoor for E6). These percentages are normalized independently per expert and therefore are not additive and are not expected to sum to 100% across the three bars. This figure is intended as a first-step illustration of specialization rather than a complete utilization analysis across all experts and layers.
More in this topic

Object Detection

Object detection locates and classifies objects in images or video, balancing localization accuracy, robustness, and inference speed.

See all 624 papers

Explore similar work

CardsList
Apr 20, 2026

Domain-Specialized Object Detection via Model-Level Mixtures of Experts

Mixture-of-Experts (MoE) models provide a structured approach to combining specialized neural networks and offer greater interpretability than conventional ensembles. While MoEs have been successfully applied to image classification and semantic segmentation, their use in object detection remains limited due to challenges in merging dense and structured predictions. In this work, we investigate model-level mixtures of object detectors and analyze their suitability for improving performance and interpretability in object detection. We propose an MoE architecture that combines YOLO-based detectors trained on semantically disjoint data subsets, with a learned gating network that dynamically weights expert contributions. We study different strategies for fusing detection outputs and for training the gating mechanism, including balancing losses to prevent expert collapse. Experiments on the BDD100K dataset demonstrate that the proposed MoE consistently outperforms standard ensemble approaches and provides insights into expert specialization across domains, highlighting model-level MoEs as a viable alternative to traditional ensembling for object detection. Our code is available at https://github.com/KASTEL-MobilityLab/mixtures-of-experts/.

Svetlana Pavlitska, Malte Stüven, Beyza Keskin +1

FZI Research Center for Information Technology · Karlsruhe Institute of Technology (KIT)

Object DetectionMixture-of-Experts Models
AskPDF
May 15, 2026

When Does Sparse MoE Help in Vision? The Role of Backbone Compute Leverage in Sparse Routing

Mixture-of-Experts (MoE) networks promise favorable accuracy-compute trade-offs, yet practical vision deployments are hindered by expert collapse and limited end-to-end efficiency gains. We study when sparse top-kk routing with hard capacity constraints helps in vision classification, evaluated under multi-seed protocols on four benchmarks (CIFAR-10/100, Tiny-ImageNet, ImageNet-1K). We observe a compute-leverage pattern: positive accuracy gaps require a substantial fraction ρρ of total FLOPs to be routed; at ImageNet scale this is necessary but not sufficient, as multi-expert routing (k≥2k \geq 2) is additionally required. Two controlled experiments isolate these factors. A hidden-size sweep on CIFAR-10 yields both predicted sign reversals across standard and depthwise backbones, ruling out backbone family as the active variable. An ImageNet-1K ablation that varies only top-kk -- holding architecture, initialization, and ρρ fixed -- reverses the gap from positive to negative across all five seeds. A per-sample variant of Soft MoE that softmaxes over experts rather than the batch rescues CIFAR-100 above the dense baseline, identifying batch-axis dispatch as the dominant failure mode in per-sample CNN settings. Code and aggregate results: https://github.com/libophd/sparse-moe-vision-rho.

Libo Sun, Po-wei Harn, Peixiong He +1

Department of Computer Science and Software Engineering Auburn University, Auburn, AL 36849, USA · Department of Information Management National Central University, Taoyuan 320317, Taiwan · Department of Computer Science and Software Engineering Auburn University

Sparse Mixture-of-ExpertsExpert RoutingMixture-of-Experts ModelsImage Classification
AskPDF
Mar 21, 2026

Distilled Large Language Model-Driven Dynamic Sparse Expert Activation Mechanism

High inter-class similarity, extreme scale variation, and limited computational budgets hinder reliable visual recognition across diverse real-world data. Existing vision-centric and cross-modal approaches often rely on rigid fusion mechanisms and heavy annotation pipelines, leading to sub-optimal generalization. We propose the Distilled Large Language Model (LLM)-Driven Sparse Mixture-of-Experts (DS-MoE) framework, which integrates text-guided dynamic routing and lightweight multi-scale comprehension. The DS-MoE framework dynamically aligns textual semantics with defect-specific visual patterns through a sparse MoE architecture, where task-relevant experts are adaptively activated based on semantic relevance, resolving inter-class ambiguity. A lightweight MobileSAM encoder enables real-time inference while preserving multi-scale defect details. Extensive experiments on PCB, aluminum foil, and mold defect datasets demonstrate that our framework achieves superior performance compared to existing pure vision models. DS-MoE surpasses YOLOv8/YOLOX with gains of +13.9, +1.4, and +2.0 pp mAP@ 0.5:0.95 on BBMP, aluminum, and PCB, respectively, while also improving precision and recall.

Qinghui Chen, Zekai Zhang, Zaigui Zhang +5

School of Control Science and Engineering, Shandong University, Jinan, China · Laoshan Laboratory, Qingdao, China · Jinan Inspur Data Technology Co., Ltd, Jinan, China +2

Industrial Anomaly DetectionIndustrial InspectionSparse Mixture-of-Experts
AskPDF