TempoMoE: Identifying and Steering Fast-Slow Thinking Experts for Efficient Multimodal Reasoning

Xiyue Zhang; Keru Zhou; Ziyi Wang; Bing Hu; Li Wang

cs.CLOct 19, 2026

TempoMoE: Identifying and Steering Fast-Slow Thinking Experts for Efficient Multimodal Reasoning

Authors: Xiyue Zhang, Keru Zhou, Ziyi Wang, Bing Hu, Li Wang

Organizations: Zhejiang University · Tsinghua University · Wenzhou Business College

Abstract

Multimodal Mixture-of-Experts (MoE) reasoning models rely on long chain-of-thought (CoT) for complex tasks, yet often overthink easy inputs, increasing token cost and latency, and sometimes degrading accuracy. Existing methods optimize reasoning length during training or impose external control at inference, but leave the underlying reasoning mechanism a black box. We find that fast and slow thinking exhibit distinct expert-routing patterns: holding the input fixed and varying only the prompted reasoning tempo produces stable routing shifts. Suppressing slow-thinking experts shortens the reasoning chain, whereas suppressing fast-thinking experts lengthens it, providing causal evidence that these experts influence reasoning depth. Based on this finding, we propose TempoMoE, a training-free, two-stage framework. It identifies fast- and slow-thinking experts via tempo-contrastive routing analysis and redistributes routing weights within the original top-k set according to sample difficulty. It amplifies fast-thinking experts and suppresses slow-thinking experts on easy inputs and reverses this adjustment on hard ones, enabling difficulty-adaptive control over reasoning depth. We evaluate TempoMoE on Kimi-VL, Qwen3-VL, and ERNIE-4.5-VL across six benchmarks. Compared with Vanilla, TempoMoE consistently improves accuracy while reducing output tokens, with accuracy gains of up to 2.6 points and token reductions of up to 35.4%.

Figures & tables

Figure 1: Average accuracy and output tokens over six benchmarks. Tokens are relative to Vanilla; bottom values show maximum gains.

Figure 2: Overview of TempoMoE. (A) Fast-slow experts are identified offline by contrasting routing patterns under paired prompts. (B) Prefill routing entropy determines fast-slow gates that reweight the original top- $k$ experts during decoding, preserving the selected expert set, total routing mass, and all model parameters.

Model	Experts	$\Delta M$	Stab.	Cons.	$\Delta$ Tok
Kimi	F: (8,6), (12,46)	$-0.037$	100	94–100	$+37.1$
Kimi	S: (1,13), (8,37)	$+0.041$	100	100	$-51.7$
Qwen	F: (28,63), (33,49)	$-0.028$	94	100	$+32.1$
Qwen	S: (28,79), (40,79)	$+0.023$	100	100	$-45.1$
ERNIE	F: (9,8), (8,57)	$-0.027$	100	100	$+24.0$
ERNIE	S: (18,19), (21,34)	$+0.024$	98–100	94–100	$-30.9$

Table 1: Identified fast/slow experts and held-out validation. F/S denote fast/slow experts; each pair denotes (layer, expert);

\Delta M

is averaged over the two experts; Stab./Cons. are percentages;

\Delta

Tok is the mean token change after joint masking.

Method	MMStar		OCRBench		RealWorldQA		MathVista		ChartQA		MMMU
Method	Acc $\uparrow$	Tok $\downarrow$	Acc $\uparrow$	Tok $\downarrow$	Acc $\uparrow$	Tok $\downarrow$	Acc $\uparrow$	Tok $\downarrow$	Acc $\uparrow$	Tok $\downarrow$	Acc $\uparrow$	Tok $\downarrow$
Kimi-VL-A3B-Thinking-2506
Vanilla	70.4	502.6	83.9	136.4	70.0	138.5	80.1	906.4	85.3	134.4	64.0	1934.8
NoThinking	66.1	174.8	81.7	60.1	66.8	3.4	74.2	670.2	84.5	36.1	56.4	398.4
TALE	65.6	457.5	74.6	119.6	67.8	127.2	74.9	836.3	80.9	122.0	60.8	1138.2
NoWait	67.7	456.1	85.9	98.5	70.0	89.9	75.3	700.3	85.0	99.6	62.2	1246.3

Table 2: Main results comparing accuracy and output token usage across three multimodal MoE reasoning models and six benchmarks. Acc denotes answer accuracy (%), and Tok denotes the average number of tokens generated per sample, including all tokens in the reasoning process and the final answer.

Figure 3: Visual localization case with Kimi-VL-A3B-Thinking-2506. TempoMoE gives the same correct answer with 35 reasoning tokens instead of 113 (69% fewer).

Method	Accuracy (%) $\uparrow$			Output Tokens $\downarrow$
Method	Overall	Easy	Hard	Overall	Easy	Hard
Vanilla	75.6	80.2	64.5	625.5	282.6	1581.2
Random Experts	72.1	77.8	59.0	620.2	280.3	1510.2
Fixed-Fast	73.7	81.7	56.2	556.0	232.1	1408.8
Fixed-Slow	74.8	79.1	64.9	679.4	341.6	1464.6
Hard Mask	70.2	76.4	55.7	662.8	304.9	1600.0

Table 3: Ablation results on Kimi-VL-A3B-Thinking-2506, aggregated across six benchmarks. Easy and Hard use the difficulty split defined in Section 5.1 .

Mixture-of-Experts Inference

Mixture-of-experts inference studies how to run sparsely activated expert models efficiently while preserving their quality.

See all 165 papers

Adaptive Look, Flexible Thought: Learning How and How Long to Think in Multimodal Reasoning
Oct 11, 2026Quanlin Chen, Xiuwei Chen, Kun Xiang +5Multimodal Large Language Models Multimodal Reasoning
Sun Yat-sen University · Shenzhen Loop Area Institute · Beijing Zhongguancun Academy +2
Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts
Apr 9, 2026Haolei Xu, Haiwen Hong, Hongxing Li +7VLM Reasoning Expert Routing
Zhejiang University · Alibaba Group
Switch-Reasoner: Learn When to Think in Multitask Mixtures via Reinforcement Learning
Jul 9, 2026Yiyang Fang, Pei Fu, Jinjie Li +7Adaptive Reasoning in Language Models Efficient Multimodal Inference
Wuhan University · The Hong Kong University of Science and Technology · Xiaomi Inc +2

Oct 11, 2026J/K move · Enter open · S save

Adaptive Look, Flexible Thought: Learning How and How Long to Think in Multimodal Reasoning

Quanlin Chen, Xiuwei Chen, Kun Xiang, Zehua Ma+4

Sun Yat-sen University · Shenzhen Loop Area Institute · Beijing Zhongguancun Academy+2

Multimodal Large Language Models Multimodal Reasoning Multimodal CoT Reasoning Efficient Multimodal Inference

Open paper AskPDF arXiv

The introduction of long chain-of-thought (CoT) reasoning and external tools has substantially improved the performance of multimodal large language models (MLLMs) on general and reasoning tasks. However, existing fast-slow reasoning paradigms mainly regulate textual reasoning length, while visual computation introduced by external tools or visual tokens is often treated as an auxiliary component. This separation prevents MLLMs from jointly optimizing visual perception and textual reasoning. To address this issue, we propose L2T (adaptive Look, flexible Thought), a unified reasoning framework that jointly optimizes the form and depth of computation. L2T first discretizes tool-produced visual states into visual tokens, bringing visual perception and textual reasoning into a shared autoregressive space. Based on this unified representation, L2T introduces a modality-aware routing reward to select appropriate reasoning modes and a difficulty-aware fast-slow strategy to adapt reasoning depth according to task complexity. Experiments on general perception and reasoning-intensive tasks demonstrate the effectiveness of L2T. Compared with explicit tool-calling approaches, L2T maintains competitive performance while achieving 1.7--3.6 times higher throughput.

Abstract

Mixture-of-Experts Inference

Explore similar work

Adaptive Look, Flexible Thought: Learning How and How Long to Think in Multimodal Reasoning

Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Experts

Switch-Reasoner: Learn When to Think in Multitask Mixtures via Reinforcement Learning