Distilled Large Language Model-Driven Dynamic Sparse Expert Activation Mechanism

Qinghui Chen; Zekai Zhang; Zaigui Zhang; Kai Zhang; Dagang Li; Wenmin Wang; Jinglin Zhang; Cong Liu

cs.CVMar 21, 2026

Distilled Large Language Model-Driven Dynamic Sparse Expert Activation Mechanism

Authors: Qinghui Chen, Zekai Zhang, Zaigui Zhang, Kai Zhang, Dagang Li, Wenmin Wang, Jinglin Zhang, Cong Liu

Organizations: School of Control Science and Engineering, Shandong University, Jinan, China · Laoshan Laboratory, Qingdao, China · Jinan Inspur Data Technology Co., Ltd, Jinan, China · School of Computer Science and Engineering, Macau University of Science and Technology, Macau SAR, China ·

Abstract

High inter-class similarity, extreme scale variation, and limited computational budgets hinder reliable visual recognition across diverse real-world data. Existing vision-centric and cross-modal approaches often rely on rigid fusion mechanisms and heavy annotation pipelines, leading to sub-optimal generalization. We propose the Distilled Large Language Model (LLM)-Driven Sparse Mixture-of-Experts (DS-MoE) framework, which integrates text-guided dynamic routing and lightweight multi-scale comprehension. The DS-MoE framework dynamically aligns textual semantics with defect-specific visual patterns through a sparse MoE architecture, where task-relevant experts are adaptively activated based on semantic relevance, resolving inter-class ambiguity. A lightweight MobileSAM encoder enables real-time inference while preserving multi-scale defect details. Extensive experiments on PCB, aluminum foil, and mold defect datasets demonstrate that our framework achieves superior performance compared to existing pure vision models. DS-MoE surpasses YOLOv8/YOLOX with gains of +13.9, +1.4, and +2.0 pp mAP@ 0.5:0.95 on BBMP, aluminum, and PCB, respectively, while also improving precision and recall.

Figures & tables

Figure 1: Example of dataset for industrial detection. The first line and second line serve as the bottle-bottom mold point image and PCB defects image respectively. The aluminium defects image is shown in the third line. According to that, it is necessary to design an object detection in the industrial quality detection, especially for those with higher similarities and greater ranges of size.

Figure 2: DS-MoE Framework. DeepSeek-R1 generates defect-specific text prompts. MobileSAM encoder extracts multi-scale visual features. Hyperbolic manifold alignment fuses text and vision in curvature-aware space. Dynamic sparse MoE gates (top-k experts) activate task-relevant visual experts for fine-grained defect analysis. Dual-branch head outputs simultaneous classification and localization.

Figure 3: Hyperbolic Manifold Alignment. In the Poincaré ball, visual features are first lifted onto the manifold via the exponential map, while distilled text embeddings are anchored in the same space. Their logarithmic mappings are then fused with a learnable weight, preserving both the global defect taxonomy and local semantic nuances. The resulting geometrically aligned features enable downstream curvature-aware convolutional sampling.

Figure 4: A concise flowchart of Stages 8–10. Sparse MoE dynamically routes each input to only $\lfloor\log_{2}N_{e}\rfloor$ experts: two task-specific modules (anisotropic local patterns and global structure) and selected cross-modal experts that fuse vision with replicated text embeddings. Their outputs are ensembled via dilated convolutions and reweighted by channel-wise attention, yielding a compact feature map.

Figure 5: The glass bottle bottom detection device collects images. During this process, the glass bottle will be suspended by the conveyor belts on both sides and sent to the photographing. The photoelectric gate receives the signal to trigger the light source and the camera at the bottom to take pictures.

Panel of Figure 6 — Figure 6: Mold point images of different batches. Representative mold-point images from different production batches, illustrating the subtle height variations that make classification challenging.

Approach	Backbone	[email protected]	[email protected]:0.95	Precision	Recall	F1-score
SSD	VGG-16	70.44%	32.52%	78.94%	49.48%	0.25
Faster-R-CNN	ResNet-50	83.62%	38.73%	74.89%	75.40%	0.74
RetinaNet	ResNet-101	80.79%	49.89%	66.28%	77.56%	0.71
CenterNet	Hourglass-104	68.51%	43.18%	84.14%	48.74%	0.62
EfficientDet-D3	EfficientNet-B3	86.39%	55.67%	67.49%	89.90%	0.76
YOLOv3	DarkNet-53	91.37%	40.66%	84.39%	92.33%	0.87

Table 1: Contrast experiment on the BBMP dataset. Bold indicates best mean performance. DS-MoE statistics are reported as mean

\pm

95 % CI over five independent runs;

p

refers to a paired

t

-test against YOLOv8.

Figure 7: Real-time inspection interface on the glass-bottle production line. In the test system interface, the red box shows the detection accuracy and the time of reasoning a picture, and the blue box shows some detection results. After 30 samples, both precision and recall reach 100 %.

	A	B	C	D	E	F
DeepSeek-R1-Distill-Qwen	$\times$	✓	✓	✓	✓	✓
MobileSAM	$\times$ (Res50)	$\times$ (Res50)	✓	✓	✓	✓
Hyperbolic Alignment	$\times$	$\times$	$\times$	✓	✓	✓
Dynamic Experts	$\times$	$\times$	$\times$	$\times$	✓	✓
Loss Function	CE+IoU	CE+IoU	CE+IoU	CE+IoU	CE+IoU	QFocal+CIoU
[email protected]	89.2	92.7	94.1	95.8	97.3	99.5

Table 2: Ablation study on BBMP. Incremental gains confirm the necessity of text guidance, MobileSAM backbone, hyperbolic alignment, dynamic experts and the QFocal+CIoU loss.

Figure 8: Sample aluminum-surface defect image. The small defect (zhen kong) and large defect (zang wu) co-exist, demonstrating extreme scale variation.

Approach	Backbone	[email protected]	[email protected]:0.95	Precision	Recall	F1-score
SSD	VGG-16	86.58%	35.20%	99.80%	62.50%	0.77
Faster-R-CNN	ResNet-50	92.35%	42.30%	73.88%	97.50%	0.84
RetinaNet	ResNet-101	96.52%	46.70%	94.79%	93.32%	0.94
EfficientDet-D3	EfficientNet-B3	93.52%	50.20%	91.85%	90.91%	0.91
CenterNet	Hourglass-104	60.30%	38.60%	51.09%	60.61%	0.55
YOLOv3	DarkNet-53	95.90%	54.40%	94.70%	95.70%	0.95

Table 3: Contrast experiment on the aluminum-defect dataset. Bold indicates best mean performance. DS-MoE statistics are reported as mean

\pm

95 % CI over five independent runs;

p

refers to a paired

t

-test versus YOLOv8.

Figure 9: Qualitative detection results on 16 aluminum-defect images. Bounding boxes and class labels are overlayed to show the model’s ability to localize multi-scale defects accurately.

	A	B	C	D	E	F
DeepSeek-R1-Distill-Qwen	$\times$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
MobileSAM	$\times$ (Res50)	$\times$ (Res50)	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
Hyperbolic Alignment	$\times$	$\times$	$\times$	$\checkmark$	$\checkmark$	$\checkmark$
Dynamic Experts	$\times$	$\times$	$\times$	$\times$	$\checkmark$	$\checkmark$
Loss Function	CE+IoU	CE+IoU	CE+IoU	CE+IoU	CE+IoU	QFocal+CIoU
[email protected]	91.2	93.3	95.1	96.4	97.3	98.2

Table 4: Ablation study on aluminum defects. Each component systematically improves robust detection of pinholes, scratches, contamination and creases.

$k$	[email protected] (%)	Latency (ms)	GFLOPs	Recall (%)	F1-score
1	95.8	8.4	2.1	95.4	0.94
2	97.1	8.9	2.3	96.3	0.95
4	98.5	9.8	3.0	97.1	0.96
5 (log ${}_{2}N_{\mathrm{e}}$ )	99.5	10.2	3.4	98.1	0.97
8	99.3	11.5	4.4	98.2	0.97
16	99.2	13.9	6.9	97.8	0.97

Table 5: Sensitivity of top-

k

expert activation on BBMP.

k=5

(log

{}_{2}N_{\mathrm{e}}

) offers the best accuracy–efficiency trade-off.

Class	[email protected]	[email protected]	precision	recall
pinhole(zhen_kong)	94.20%	43.24%	89.30%	90.80%
scratch(ca_shang)	97.92%	56.37%	91.70%	99.80%
contamination(zang_wu)	99.50%	61.80%	99.80%	99.60%
crease(zhe_zhou)	98.90%	65.70%	97.90%	99.30%

Table 6: Experiments without tiny object detector. Class-wise results on aluminum defects without the tiny-object head, revealing recall deficits on sub-millimetre pinholes.

Class	[email protected]	[email protected]	precision	recall
pinhole(zhen_kong)	96.0%	44.1%	93.1%	91.3%
scratch(ca_shang)	98.1%	57.3%	94.7%	99.8%
contamination(zang_wu)	99.5%	62.6%	99.7%	99.8%
crease(zhe_zhou)	99.5%	65.9%	98.8%	99.6%

Table 7: Experiments with tiny object detector. Class-wise results on aluminum defects with the tiny-object head, showing consistent gains for pinhole detection.

Panel of Figure 10 — Figure 10: Example PCB defects: (left) missing hole, (right) mouse bite. Both are sub-millimeter anomalies that require high-resolution feature preservation for reliable detection.

Approach	Backbone	[email protected]	[email protected]:0.95	Precision	Recall	F1
SSD	VGG-16	17.10%	10.00%	19.93%	54.64%	0.29
Faster-R-CNN	ResNet-50	78.33%	50.24%	67.74%	74.51%	0.70
YOLOv3	DarkNet-53	73.32%	47.50%	90.85%	52.66%	0.67
YOLOv5-S	CSPDarkNet	91.20%	51.80%	81.82%	92.12%	0.86
EfficientDet-D3	EfficientNet-B3	71.80%	35.00%	99.44%	49.51%	0.66
YOLOv5-X	CSPDarkNet	91.80%	50.70%	95.20%	91.30%	0.93

Table 8: Contrast experiment on the PCB surface-defect dataset. Bold indicates best mean performance. DS-MoE statistics are reported as mean

\pm

95 % CI over five independent runs;

p

refers to a paired

t

-test versus YOLOv8.

	A	B	C	D	E	F
DeepSeek-R1-Distill-Qwen	$\times$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
MobileSAM	$\times$ (Res50)	$\times$ (Res50)	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
Hyperbolic Alignment	$\times$	$\times$	$\times$	$\checkmark$	$\checkmark$	$\checkmark$
Dynamic Experts	$\times$	$\times$	$\times$	$\times$	$\checkmark$	$\checkmark$
Loss Function	CE+IoU	CE+IoU	CE+IoU	CE+IoU	CE+IoU	QFocal+CIoU
[email protected]	81.2	85.3	87.6	89.1	91.8	94.2

Table 9: Ablation Study on PCB Defect Dataset. Text-guided dynamic experts and curvature-aware fusion jointly boost small-defect recall to 96.8 %.

Industrial Anomaly Detection

Industrial anomaly detection finds defective products and abnormal equipment behavior from visual, sensor, audio, and other production data.

See all 261 papers

VisMMOE: Exploiting Visual-Expert Affinity for Efficient Visual-Language MoE Offloading
May 7, 2026Cheng Xu, Xiaofeng Hou, Jiacheng Liu +1Expert Offloading Vision-Language Models
Shanghai Jiao Tong University Minhang Qu, Shanghai, China
PRISM: Synergizing Vision Foundation Models via Self-organized Expert Specialization
Jun 2, 2026Ying Tang, Dong Li, Youjia Zhang +3Vision Foundation Models Sparse Mixture-of-Experts
School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China.
HI-MoE: Hierarchical Instance-Conditioned Mixture-of-Experts for Object Detection
Apr 6, 2026Vadim Vashkelis, Natalia TrukhinaObject Detection Mixture-of-Experts Models

May 7, 2026J/K move · Enter open · S save

VisMMOE: Exploiting Visual-Expert Affinity for Efficient Visual-Language MoE Offloading

Cheng Xu, Xiaofeng Hou, Jiacheng Liu, Chao Li

Shanghai Jiao Tong University Minhang Qu, Shanghai, China

Expert Offloading Vision-Language Models Efficient VLM Inference Mixture-of-Experts Inference Visual Token Compression

Open paper AskPDF arXiv

Large-scale vision-language mixture-of-experts (VL-MoE) models provide strong multimodal capability, but efficient deployment on memory-constrained platforms remains difficult. Existing MoE offloading systems are largely designed for text-centric workloads and become much less effective for visual-heavy inputs, where large numbers of visual tokens induce broader and less predictable expert accesses. We present VisMMoE, a VL-MoE offloading system built on a single systems insight: pruning redundant visual tokens can improve offloading not only by reducing computation, but also by reshaping expert demand. We refer to this effect as visual-expert affinity: token pruning makes expert accesses more concentrated within layers and more stable across layers, producing a smaller and more predictable expert working set. Guided by this insight, VisMMoE combines affinity-aware token compression, lookahead expert prediction, and cache/pipeline orchestration to improve expert locality and prefetch effectiveness under tight memory budgets. We implement VisMMoE on multiple frameworks and evaluate it on representative VL-MoE models and benchmarks. VisMMoE improves end-to-end inference performance by up to 2.68x and 1.61x, respectively, over strong baselines for today's VL-MoE deployments while maintaining competitive accuracy.

Abstract

Industrial Anomaly Detection

Explore similar work

VisMMOE: Exploiting Visual-Expert Affinity for Efficient Visual-Language MoE Offloading

PRISM: Synergizing Vision Foundation Models via Self-organized Expert Specialization

HI-MoE: Hierarchical Instance-Conditioned Mixture-of-Experts for Object Detection