Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts

Haolei Xu; Haiwen Hong; Hongxing Li; Rui Zhou; Yang Zhang; Longtao Huang; Hui Xue; Yongliang Shen

cs.CVApr 9, 2026

Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts

Authors: Haolei Xu, Haiwen Hong, Hongxing Li, Rui Zhou, Yang Zhang, Longtao Huang, Hui Xue, Yongliang Shen, +2 more

Organizations: Zhejiang University · Alibaba Group

Abstract

Multimodal Mixture-of-Experts (MoE) models have achieved remarkable performance on vision-language tasks. However, we identify a puzzling phenomenon termed Seeing but Not Thinking: models accurately perceive image content yet fail in subsequent reasoning, while correctly solving identical problems presented as pure text. Through systematic analysis, we first verify that cross-modal semantic sharing exists in MoE architectures, ruling out semantic alignment failure as the sole explanation. We then reveal that visual experts and domain experts exhibit layer-wise separation, with image inputs inducing significant routing divergence from text inputs in middle layers where domain experts concentrate. Based on these findings, we propose the Routing Distraction hypothesis: when processing visual inputs, the routing mechanism fails to adequately activate task-relevant reasoning experts. To validate this hypothesis, we design a routing-guided intervention method that enhances domain expert activation. Experiments on three multimodal MoE models across six benchmarks demonstrate consistent improvements, with gains of up to 3.17% on complex visual reasoning tasks. Our analysis further reveals that domain expert identification locates cognitive functions rather than sample-specific solutions, enabling effective transfer across tasks with different information structures.

Figures & tables

Figure 1: Illustration of the Seeing but Not Thinking phenomenon. See Appendix B for details.

Figure 2: Overview of our work. We first conduct cross-modal concept intervention to verify semantic sharing in MoE architectures (left, § 3.1 ), then identify domain experts by comparing activation frequencies on domain-specific versus general data (middle, § 3.2 ), and finally analyze routing divergence across modalities and apply routing guidance to enhance domain expert activation (right, § 3.3 -§ 4 ).

Panel of Figure 3 — Figure 3: Analysis of routing mechanisms in multimodal MoE models. (a) Cross-modal semantic sharing verification showing inverted U-shaped intervention success rates. (b) Expert specialization quantification using Gini coefficients. (c) Routing divergence across modalities for three image versions.

Panel of Figure 4 — Figure 4: Layer-wise distribution of domain experts and visual experts. Left: Heatmap showing activation frequency differences (red: higher on math data; blue: higher on general data), with deep red concentrated in layers 6–42. Right: Expert counts per layer, where Overlap indicates experts identified as both math and visual experts.

Model	Intervention Layers	$\bm{\tau}$	$\bm{\lambda}$
Kimi-VL-16B-A3B	[0, 20]	0.3	0.5
Qwen3-VL-30B-A3B	[6, 42]	0.3	0.5
Llama4-Scout-109B-A17B	[8, 40]	0.3	0.2

Table 1: Intervention configurations for each model. Intervention layers are specified as inclusive ranges.

	Semantically Equivalent Scenarios			Natural Visual Scenarios
Method	Math	Chemistry	Physics	MathVerse	MATH-V	GSM8K-V	Average
Kimi-VL-16B-A3B-Instruct
Baseline	52.30	25.54	29.51	35.41	21.05	8.11	28.65
Random	51.64 -0.66	25.67 +0.13	28.78 -0.73	34.39 -1.02	19.74 -1.31	6.82 -1.29	27.84 -0.81
Hard	53.06 +0.76	25.81 +0.27	32.78 +3.27	35.79 +0.38	21.71 +0.66	7.13 -0.98	29.38 +0.73
Soft	54.54 +2.24	27.89 +2.35	32.49 +2.98	38.58 +3.17	23.36 +2.31	9.17 +1.06	31.01 +2.36

Table 2: Accuracy (%) across three multimodal MoE models and six benchmarks. Math, Chemistry, and Physics refer to image-rendered versions of MATH500 and GPQA-Diamond (chemistry and physics subsets) respectively. MATH-V denotes MATH-Vision.

Model	Text	Vision	w/ Guidance
Model	Text	Vision	$\text{Ref}_{\text{MathVerse}}$	$\text{Ref}_{\text{GSM8K}}$
Kimi-VL	45.94	35.41	38.58 +3.17	37.31 +1.90
Qwen3-VL	67.26	69.29	71.20 +1.91	69.54 +0.25
Llama4	63.32	56.09	57.11 +1.02	55.08 -1.01

Table 3: Text-only and vision-only accuracy on MathVerse, with guidance using experts identified from MathVerse text-only vs. GSM8K.

Model	Baseline	w/ Early	Middle	w/ Late
Kimi-VL	28.65	31.01	29.82	29.35
Qwen3-VL	59.24	59.76	60.72	58.97
Llama4	47.93	46.85	49.10	47.82

Table 4: Impact of intervention layer range on average accuracy. Middle: intervening only on middle layers; w/ Early: adding early layers; w/ Late: adding late layers. For Kimi-VL, middle layers refer to layers 6-20.

Figure 5: Effect of enhancement coefficient

\lambda

on reasoning accuracy gains across three models.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Baseline Response (Incorrect: Answer is 4)

Of course, let’s solve this problem step-by-step. The scenario involves a beanstalk that doubles in height every day and starts at 4 feet tall . We need to find out how many days it will take for the beanstalk to grow taller than a window located 20 feet off the ground . We can calculate the height of the beanstalk each day until its height exceeds 20 feet. - Day 1: The beanstalk is

4

feet tall. - Day 2: It doubles from Day 1:

4\times 2=8

feet tall. - Day 3: It doubles from Day 2:

8\times 2=16

feet tall. - Day 4: It doubles from Day 3:

16\times 2=32

feet tall. Now we compare these heights to the window: - On Day 3 , the beanstalk is 16 feet tall, which is less than 20 feet (the window). - On Day 4 , the beanstalk is 32 feet tall, which is greater than 20 feet. Therefore, on the fourth day , the beanstalk becomes taller than the window. It takes

\boxed{{\color[rgb]{1,0,0}4}}

days for the beanstalk to be taller than the window.

With Routing Intervention (Correct: Answer is 3)

Of course, let’s solve this problem step-by-step. The scenario involves a beanstalk that doubles in height every day and starts at 4 feet tall. We need to find out how many days it will take for the beanstalk to grow taller than a window located 20 feet off the ground. We can model the growth of the beanstalk using an exponential function: - Initial Height (Day 0):

h_{0}=4

feet. - Growth Rate: The plant doubles its size each day, so we multiply by 2 daily. - Height on Day n:

h_{n}=h_{0}\times 2^{n}

Let’s calculate the height of the beanstalk over several consecutive days until it surpasses the 20-foot mark: — Step 1: Calculate the height after each day - After Day 1:

h_{1}=4\text{ ft}\times 2^{1}=8\text{ ft}

- After Day 2:

h_{2}=4\text{ ft}\times 2^{2}=4\text{ ft}\times 4=16\text{ ft}

- After Day 3:

h_{3}=4\text{ ft}\times 2^{3}=4\text{ ft}\times 8=32\text{ ft}

— Step 2: Compare with the window height Now, compare these heights to the window’s height of 20 feet: - After Day 1 (

8

ft), the beanstalk is shorter than the window (

8<20

). - After Day 2 (

16

ft), the beanstalk is still shorter than the window (

16<20

). - After Day 3 (

32

ft), the beanstalk is now taller than the window (

32>20

). Therefore, the first time the beanstalk exceeds the height of the window is on the third day. It takes

\boxed{3}

days for the beanstalk to be taller than the window.

Appendix

Table 5: Complete model responses for the problem shown in Figure 1 . Despite successfully extracting all necessary information, the baseline response incorrectly counts the initial value as a separate day (Day 1), leading to an off-by-one error. The response with routing-guided intervention correctly starts the count from Day 0 and produces the correct answer.

Panel of Figure 6 — Figure 6: Examples of the three semantically equivalent image versions (v1, v2, v3 from top to bottom) for the same text problem.

Dataset	Acc	Acc (OCR first)	Perception Error	Reasoning Error
MATH500	92.8	-	-	-
MATH500-v1	89.0	87.4	31.8% (7/22)	68.2% (15/22)
MATH500-v2	88.2	86.8	26.9% (7/26)	73.1% (19/26)
MATH500-v3	87.4	86.8	31.0% (9/29)	69.0% (20/29)

Appendix

Table 6: Accuracy comparison and error analysis on MATH500 and its three image versions. “Acc (OCR first)” refers to performance when explicitly instructing the model to perform OCR before reasoning. Perception and reasoning error percentages are computed from samples where the model answered correctly on the text version but incorrectly on the image version.

Model	Layers	Routed Experts	Active Routed Experts	Shared Experts	Total Active Experts
Kimi-VL	27	64	6	2	8
Qwen3-VL	48	128	8	–	8
Llama4	48	16	1	1	2

Appendix

Table 7: Model architectures. All expert counts are specified per layer.

Figure 7: Sample image from MATH-Vision dataset.

Explore similar work

DecoMoE: Decoupling Visual Propagation and Expert Computation for Efficient Multimodal MoE Inference
Sep 30, 2026Xudong Tan, Peng Ye, Ming Xie +4Mixture-of-Experts Pruning Efficient Multimodal Inference
Fudan University · The Chinese University of Hong Kong
Beyond Routing: Characterising Expert Tuning and Representation in Vision Mixture-of-Experts
May 20, 2026Gene Tangtartharakul, Katherine R. StorrsVisual Representation Learning Mixture of Experts
School of Psychology · University of Auckland · Auckland, New Zealand
Before Thinking, Learn to Decide: Proactive Routing for Efficient Visual Reasoning
Jun 29, 2026Yinan Zhou, Haokun Lin, Yichen Wu +7Efficient Multimodal Inference Adaptive Model Routing
Xi’an Jiaotong University · ARC Lab, Tencent IEG · City University of Hong Kong +3

Sep 30, 2026J/K move · Enter open · S save

DecoMoE: Decoupling Visual Propagation and Expert Computation for Efficient Multimodal MoE Inference

Xudong Tan, Peng Ye, Ming Xie, Chenyu Huang+3

Fudan University · The Chinese University of Hong Kong

Mixture-of-Experts Pruning Efficient Multimodal Inference Mixture-of-Experts Inference Visual Token Pruning

Open paper AskPDF arXiv

Multimodal mixture-of-experts (MoE) models combine sparse expert activation with visual-language capabilities, yet their inference remains costly because long visual-token sequences repeatedly incur attention, routing, dispatch, and expert-MLP computation. Existing methods typically compress either the token or expert dimension, leaving redundancy along the other. Our analysis reveals two complementary regularities: the depth required for visual propagation varies across inputs, while text-token routing exhibits concentrated and recurrent expert-importance patterns. Based on these observations, we propose DecoMoE, a two-dimensional structured compression framework that decouples visual propagation from expert computation. The Sample-Adaptive Visual Boundary (SAVB) predicts an input-dependent visual-exit layer at which the visual-token block is removed. The Routing-Calibrated Expert Prefix (RCEP) reorders experts offline using text-token routed mass and, from this predicted exit layer onward, retains at each MoE layer the shortest contiguous prefix covering a target routed-mass fraction. We evaluate DecoMoE on Qwen3-VL-MoE and InternVL3.5-30B-A3B across six benchmarks. On Qwen3-VL-MoE, DecoMoE retains 97.91% of dense-baseline performance while reducing computation from 27.06 to 16.73 TFLOPs and latency from 0.44 to 0.26 seconds, yielding a 1.69x speedup. Code will be available at https://github.com/ShawnTan86/DecoMoE.

Abstract

Explore similar work

DecoMoE: Decoupling Visual Propagation and Expert Computation for Efficient Multimodal MoE Inference

Beyond Routing: Characterising Expert Tuning and Representation in Vision Mixture-of-Experts

Before Thinking, Learn to Decide: Proactive Routing for Efficient Visual Reasoning