Multimodal Mixture-of-Experts (MoE) reasoning models rely on long chain-of-thought (CoT) for complex tasks, yet often overthink easy inputs, increasing token cost and latency, and sometimes degrading accuracy. Existing methods optimize reasoning length during training or impose external control at inference, but leave the underlying reasoning mechanism a black box. We find that fast and slow thinking exhibit distinct expert-routing patterns: holding the input fixed and varying only the prompted reasoning tempo produces stable routing shifts. Suppressing slow-thinking experts shortens the reasoning chain, whereas suppressing fast-thinking experts lengthens it, providing causal evidence that these experts influence reasoning depth. Based on this finding, we propose TempoMoE, a training-free, two-stage framework. It identifies fast- and slow-thinking experts via tempo-contrastive routing analysis and redistributes routing weights within the original top-k set according to sample difficulty. It amplifies fast-thinking experts and suppresses slow-thinking experts on easy inputs and reverses this adjustment on hard ones, enabling difficulty-adaptive control over reasoning depth. We evaluate TempoMoE on Kimi-VL, Qwen3-VL, and ERNIE-4.5-VL across six benchmarks. Compared with Vanilla, TempoMoE consistently improves accuracy while reducing output tokens, with accuracy gains of up to 2.6 points and token reductions of up to 35.4%.
Figures & tables
Figure 1: Average accuracy and output tokens over six benchmarks. Tokens are relative to Vanilla; bottom values show maximum gains.
Figure 2: Overview of TempoMoE. (A) Fast-slow experts are identified offline by contrasting routing patterns under paired prompts. (B) Prefill routing entropy determines fast-slow gates that reweight the original top- k experts during decoding, preserving the selected expert set, total routing mass, and all model parameters.
Model
Experts
ΔM
Stab.
Cons.
Δ Tok
Kimi
F: (8,6), (12,46)
−0.037
100
94–100
+37.1
S: (1,13), (8,37)
+0.041
100
100
−51.7
Qwen
F: (28,63), (33,49)
−0.028
94
100
+32.1
S: (28,79), (40,79)
+0.023
100
100
−45.1
ERNIE
F: (9,8), (8,57)
−0.027
100
100
+24.0
S: (18,19), (21,34)
+0.024
98–100
94–100
−30.9
Table 1: Identified fast/slow experts and held-out validation. F/S denote fast/slow experts; each pair denotes (layer, expert); ΔM is averaged over the two experts; Stab./Cons. are percentages; Δ Tok is the mean token change after joint masking.
Method
MMStar
OCRBench
RealWorldQA
MathVista
ChartQA
MMMU
Acc ↑
Tok ↓
Acc ↑
Tok ↓
Acc ↑
Tok ↓
Acc ↑
Tok ↓
Acc ↑
Tok ↓
Acc ↑
Tok ↓
Kimi-VL-A3B-Thinking-2506
Vanilla
70.4
502.6
83.9
136.4
70.0
138.5
80.1
906.4
85.3
134.4
64.0
1934.8
NoThinking
66.1
174.8
81.7
60.1
66.8
3.4
74.2
670.2
84.5
36.1
56.4
398.4
TALE
65.6
457.5
74.6
119.6
67.8
127.2
74.9
836.3
80.9
122.0
60.8
1138.2
NoWait
67.7
456.1
85.9
98.5
70.0
89.9
75.3
700.3
85.0
99.6
62.2
1246.3
Table 2: Main results comparing accuracy and output token usage across three multimodal MoE reasoning models and six benchmarks. Acc denotes answer accuracy (%), and Tok denotes the average number of tokens generated per sample, including all tokens in the reasoning process and the final answer.
Figure 3: Visual localization case with Kimi-VL-A3B-Thinking-2506. TempoMoE gives the same correct answer with 35 reasoning tokens instead of 113 (69% fewer).
Method
Accuracy (%) ↑
Output Tokens ↓
Overall
Easy
Hard
Overall
Easy
Hard
Vanilla
75.6
80.2
64.5
625.5
282.6
1581.2
Random Experts
72.1
77.8
59.0
620.2
280.3
1510.2
Fixed-Fast
73.7
81.7
56.2
556.0
232.1
1408.8
Fixed-Slow
74.8
79.1
64.9
679.4
341.6
1464.6
Hard Mask
70.2
76.4
55.7
662.8
304.9
1600.0
Table 3: Ablation results on Kimi-VL-A3B-Thinking-2506, aggregated across six benchmarks. Easy and Hard use the difficulty split defined in Section 5.1 .
The introduction of long chain-of-thought (CoT) reasoning and external tools has substantially improved the performance of multimodal large language models (MLLMs) on general and reasoning tasks. However, existing fast-slow reasoning paradigms mainly regulate textual reasoning length, while visual computation introduced by external tools or visual tokens is often treated as an auxiliary component. This separation prevents MLLMs from jointly optimizing visual perception and textual reasoning. To address this issue, we propose L2T (adaptive Look, flexible Thought), a unified reasoning framework that jointly optimizes the form and depth of computation. L2T first discretizes tool-produced visual states into visual tokens, bringing visual perception and textual reasoning into a shared autoregressive space. Based on this unified representation, L2T introduces a modality-aware routing reward to select appropriate reasoning modes and a difficulty-aware fast-slow strategy to adapt reasoning depth according to task complexity. Experiments on general perception and reasoning-intensive tasks demonstrate the effectiveness of L2T. Compared with explicit tool-calling approaches, L2T maintains competitive performance while achieving 1.7--3.6 times higher throughput.
Quanlin Chen, Xiuwei Chen, Kun Xiang +5
Sun Yat-sen University · Shenzhen Loop Area Institute · Beijing Zhongguancun Academy +2
Multimodal Mixture-of-Experts (MoE) models have achieved remarkable performance on vision-language tasks. However, we identify a puzzling phenomenon termed Seeing but Not Thinking: models accurately perceive image content yet fail in subsequent reasoning, while correctly solving identical problems presented as pure text. Through systematic analysis, we first verify that cross-modal semantic sharing exists in MoE architectures, ruling out semantic alignment failure as the sole explanation. We then reveal that visual experts and domain experts exhibit layer-wise separation, with image inputs inducing significant routing divergence from text inputs in middle layers where domain experts concentrate. Based on these findings, we propose the Routing Distraction hypothesis: when processing visual inputs, the routing mechanism fails to adequately activate task-relevant reasoning experts. To validate this hypothesis, we design a routing-guided intervention method that enhances domain expert activation. Experiments on three multimodal MoE models across six benchmarks demonstrate consistent improvements, with gains of up to 3.17% on complex visual reasoning tasks. Our analysis further reveals that domain expert identification locates cognitive functions rather than sample-specific solutions, enabling effective transfer across tasks with different information structures.
Multimodal Large Language Models (MLLMs) often follow a fixed Think-then-Answer paradigm, which is inefficient in heterogeneous multitask settings because simple inputs may not require explicit reasoning while difficult ones can benefit substantially from it. Learning when to think is also unstable during post-training, where imbalanced rollouts can drive the model toward always-thinking or always-direct behavior. We propose Switch-Reasoner, a GRPO-based framework that learns to adaptively select reasoning modes for MLLMs. It treats thinking as a virtual tool invocation and allows the model to either answer directly or invoke explicit reasoning before answering. To stabilize this decision, we introduce a dual-level regulation mechanism that balances the overall use of Thinking Mode and Direct Mode while providing sample-level supervision based on the relative benefit of the two choices. Experiments on 11 multimodal tasks show that Switch-Reasoner reduces unnecessary reasoning while maintaining strong performance, achieving a better accuracy-efficiency trade-off.
Yiyang Fang, Pei Fu, Jinjie Li +7
Wuhan University · The Hong Kong University of Science and Technology · Xiaomi Inc +2