Chain-of-thought (CoT) reasoning has significantly improved the performance of large multimodal models in language-guided segmentation, yet its prohibitive computational cost, stemming from generating verbose rationales, limits real-world applicability. We introduce WISE (Wisdom from Internal Self-Exploration), a novel paradigm for efficient reasoning guided by the principle of thinking twice -- once for learning, once for speed. WISE trains a model to generate a structured sequence: a concise rationale, the final answer, and then a detailed explanation. By placing the concise rationale first, our method leverages autoregressive conditioning to enforce that the concise rationale acts as a sufficient summary for generating the detailed explanation. This structure is reinforced by a self-distillation objective that jointly rewards semantic fidelity and conciseness, compelling the model to internalize its detailed reasoning into a compact form. At inference, the detailed explanation is omitted. To address the resulting conditional distribution shift, our inference strategy, WISE-S, employs a simple prompting technique that injects a brevity-focused instruction into the user's query. This final adjustment facilitates the robust activation of the learned concise policy, unlocking the full benefits of our framework. Extensive experiments show that WISE-S achieves state-of-the-art zero-shot performance on the ReasonSeg benchmark with 58.3 cIoU, while reducing the average reasoning length by nearly \textbf{5×} -- from 112 to just 23 tokens. Code is available at WISE.
Figures & tables
Figure 1: WISE achieves a superior cost-performance trade-off by decoupling reasoning for learning and inference. Our framework trains the model with detailed explanations but uses a distilled, concise rationale at inference. The base model, WISE, reduces reasoning cost by an average of 4.7 × while outperforming the baseline. Our final model, WISE-S, applies an additional inference-time prompt, achieving even greater compression (up to 10.6 × on RefCOCO) and the highest overall performance. On ReasonSeg, WISE-S simultaneously improves performance by 3.9 cIoU and cuts token cost by 4.9 × .
Figure 2: Overview of the WISE framework. During training (orange), the reasoning model Freason generates a structured sequence: concise rationale ( τc ), answer ( A ), and detailed explanation ( τd ). The process is optimized via GRPO using a hierarchical reward, notably including a self-distillation term ( Rdistill ) that aligns τc with τd while enforcing brevity. The segmentation model Fseg remains frozen. At inference (blue), τd is omitted and a brevity prompt ( Tb ) is added to ensure highly efficient reasoning.
Figure 3: Prompting Strategy for WISE. The figure shows the basic instruction template and the specific components for generating concise ( Tc ) and detailed ( Td ) rationales. Different combinations of these components are used for the training phase, standard WISE inference, and the brevity-focused WISE-S inference.
ReasonSeg
val
test
Method
gIoU
cIoU
gIoU
cIoU
OVSeg ( Liang et al., 2023 )
28.5
18.6
26.1
20.8
ReLA ( Liu et al., 2023 )
22.4
19.9
21.3
22.0
Grounded-SAM ( Ren et al., 2024a )
26.0
14.5
21.3
16.4
LISA-7B-LLaVA1.5 ( Lai et al., 2024 )
53.6
52.3
48.7
48.8
Table 1: Comparison with state-of-the-art methods on the Zero-shot ReasonSeg benchmark. WISE-7B and WISE-7B-S significantly outperform prior methods. ∗ We re-evaluate with their official checkpoints. † Results reported in the original paper using different checkpoints.
RCO
RCO+
RCOg
Method
testA
testA
test
LAVT ( Yang et al., 2022 )
75.8
68.4
62.1
ReLA ( Liu et al., 2023 )
76.5
71.0
66.0
LISA-7B ( Lai et al., 2024 )
76.5
67.4
68.5
PixelLM-7B ( Ren et al., 2024b )
76.5
71.7
70.5
MagNet ( Chng et al., 2024 )
78.3
73.6
69.3
Table 2: Comparison with state-of-the-art methods on the Referring Expression Segmentation benchmark. Our models achieve competitive or superior performance against specialized methods while being more general.
RefCOCO testA
RefCOCO+ testA
RefCOCOg test
ReasonSeg (cIoU)
Method
#Tok ↓
cIoU ↑
#Tok ↓
cIoU ↑
#Tok ↓
cIoU ↑
#Tok ↓
val ↑
#Tok ↓
test ↑
Seg-Zero
165.1
78.8
159.9
74.8
159.8
71.3
117.5
57.4
111.9
54.4
Seg-Zero+L1-Exact
31.2
78.3
31.1
74.9
30.7
71.7
36.6
57.2
35.6
55.0
Seg-Zero+L1-Max
11.7
78.9
11.7
74.6
10.6
70.7
10.5
43.8
12.0
49.7
WISE (ours)
35.0
79.1
33.8
74.0
34.0
71.8
38.8
59.2
38.8
58.5
vs. Seg-Zero
(4.7 ×↓ )
(+0.3)
(4.7 ×↓ )
(-0.8)
(4.7 ×↓ )
(+0.5)
(3.0 ×↓ )
(+1.8)
(2.9 ×↓ )
(+4.1)
Table 3: Performance and Efficiency Comparison on Referring and Reasoning Segmentation Benchmarks. Our WISE models drastically reduce token overhead (#Tok) while improving or maintaining accuracy (cIoU) across all benchmarks compared to the Seg-Zero and brevity-focused reward shaping methods (L1-Exact, L1-Max ( Aggarwal and Welleck, 2025 ) ). The sub-rows show the token reduction factor ( ×↓ ) and the absolute performance change ( Δ ) against the Seg-Zero.
Method
Det. COCO
Seg ReaSeg
Count Pixmon
#Tok
mAP
#Tok
gIoU
#Tok
Acc
VisionReasoner-7B
62.5
38.8
80.1
64.0
74.5
70.1
VisionReasoner-7B-S
42.1
39.1
45.0
63.6
53.7
69.3
+WISE (Ours)
67.7
39.5
63.0
66.0
84.9
68.2
+WISE-S (Ours)
15.7
39.6
17.1
66.1
19.2
73.0
Table 4: Generalization to Unified Tasks. Validating WISE on the VisionReasoner across three distinct visual reasoning tasks. Note that VisionReasoner-7B-S applies our brevity prompt to the baseline, confirming the prompt’s generality but highlighting the superior compression of the full WISE training.
Order
RefCOCOg
ReasonSeg
#Tok ↓
cIoU ↑
#Tok ↓
cIoU ↑
A→τc→τd
74.6
70.1
75.7
54.4
τd→τc→A
107.5
68.2
106.5
55.4
τc→τd→A
134.7
70.1
132.7
55.2
τc→A→τd (Ours)
34.0
71.8
38.8
58.5
Table 5: Ablation on Generation Order. Placing the concise rationale first ( τc→A→τd ) is essential for enabling effective thought compression at inference time.
Components
RefCOCOg
ReasonSeg
I
Ssim
Sconcise
#Tok ↓
cIoU ↑
#Tok ↓
cIoU ↑
159.8
71.3
111.9
54.4
✓
✓
67.8
70.1
75.2
54.6
✓
✓
46.2
68.6
43.6
57.8
✓
✓
78.9
70.7
74.2
56.3
✓
✓
✓
34.0
71.8
38.8
58.5
Table 6: Component Analysis of Self-Distillation Reward. All three terms—the conditional indicator ( I ), semantic similarity ( Ssim ), and conciseness score ( Sconcise )—are requisite for optimal performance.
Inference Strategy
RefCOCOg
ReasonSeg
#Tok ↓
cIoU ↑
#Tok ↓
cIoU ↑
Seg-Zero (Baseline)
159.8
71.3
111.9
54.4
WISE-F (+ τc+τd )
188.5
71.3
197.5
58.8
Ablation: Prompt Modification
−τc−τd
51.5
70.3
59.9
54.8
−τc+τd
200.0
68.4
221.4
53.2
Table 7: Effect of Inference Strategies. While the default WISE setting ( +τc−τd ) offers substantial compression, the targeted brevity prompt in WISE-S achieves the optimal efficiency-accuracy balance.
Figure 4: Distribution of Reasoning Token Length. The KDE plot illustrates the dramatic reduction in both length and variance. WISE-S (green) demonstrates a tightly concentrated preference for conciseness compared to the long-tailed Seg-Zero baseline (purple).
Figure 5: Qualitative Comparison. Given a complex instruction, Seg-Zero produces a convoluted, 132-token rationale and fails. In contrast, WISE-S generates a concise, correct reasoning chain, successfully identifying the target with a fraction of the cost.
Method
RefCOCO testA
ReasonSeg test
#Tok
Time
cIoU
#Tok
Time
cIoU
Qwen2.5-VL-7B
0.0
11.2
77.8
0.0
7.6
41.2
Seg-Zero (Baseline)
165.1
153.1
78.8
111.9
50.0
54.4
Seg-Zero-S (Prompt)
85.6
116.1
79.0
57.9
20.3
55.0
WISE (Ours)
35.0
28.9
79.1
38.8
17.0
58.5
WISE-S (Ours)
24.0
16.1
79.1
22.7
10.4
58.3
Table 8: Wall-Clock Efficiency Analysis. Comparison of inference latency and token costs. While the raw model underperforms and prompting alone (Seg-Zero-S) yields limited gains, WISE-S achieves a 5 × speedup over the Seg-Zero baseline.
Method
ReasonSeg val
ReasonSeg test
#Tok
gIoU
cIoU
#Tok
gIoU
cIoU
Architecture: Qwen2-VL-7B
Qwen2-VL-7B
0.0
44.5
-
0.0
38.7
-
WISE (Ours)
32.7
45.8
42.6
31.2
44.2
40.4
WISE-S (Ours)
23.8
44.9
42.5
23.1
44.7
40.9
Scale: Qwen2.5-VL-3B
Table 9: Scalability across Architectures. Performance consistency of WISE across different model families (Qwen2 vs. Qwen2.5) and parameter scales (3B vs. 7B).
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Success Case Study Examples. The figures demonstrate how WISE-S acts as a sufficient statistic. (a) Shows the model efficiently identifying an object by its function (“take small sips”) without verbose description. (b) Shows the model correctly filtering out spatial redundancy to focus on discriminative visual attributes (“red and white obstacle”).
Figure 7: Analysis of Failure Cases: Semantic Fidelity amidst Grounding Errors. These examples illustrate instances where the model fails to produce an accurate mask (Low IoU). Crucially, however, the Concise Rationale ( τc ) remains a faithful summary of the Detailed Explanation ( τd ). (a) In the top example, both τc and τd correctly identify the semantic topic but struggle to ground the abstract concept to specific pixels. (b) In the bottom example, both rationales exhibit circular logic without identifying distinct visual attributes. This indicates that the failures stem from the underlying limitations of the base model’s spatial grounding capabilities or reasoning loops, rather than information loss caused by the WISE-S compression mechanism.
Reasoning segmentation aims to interpret implicit textual queries and enable fine-grained visual perception, which is critical for applications such as human-computer interaction and embodied agents. Existing methods typically generate explicit Chain-of-Thought (CoT) by multimodal large language models (MLLMs) before localizing the target. Although intuitive, such explicit verbal reasoning introduces substantial attention interference: redundant textual tokens disrupt attention during perception-token generation and also increase the effective distance between visual tokens. To address this issue, we propose LIRSeg, which fully replaces explicit CoT with a compact set of learnable latent tokens for reasoning segmentation. LIRSeg is trained in two stages: spatial alignment grounds the latent tokens in object-relevant visual evidence, and GRPO further optimizes them with segmentation rewards. To make these compact latent tokens more informative, we introduce three complementary mechanisms from an information perspective: extreme-advantage sampling for selecting informative training signals, decoupled exploration-stability updates for learning complementary representations, and latent diversity amplification for preventing representational collapse. Extensive experiments on benchmarks demonstrate that LIRSeg consistently improves both segmentation accuracy and reasoning efficiency. Compared with the VisionReasoner baseline, LIRSeg achieves absolute gIoU improvements of 4.9% on ReasonSeg, 7.1% on MUSE, and 4.7% on MMR, while achieving a approximately 16x reduction in reasoning tokens. Code is available in supplementary materials.
Tianhang Guo, Yulin He, Wei Chen +3
National University of Defense Technology, Changsha, China
Recent advances in Large Reasoning Models have significantly improved chain-of-thought (CoT) capabilities via reinforcement learning (RL). However, generated reasoning chains frequently suffer from structural redundancy (i.e., overthinking), incurring high computational overhead without improving answer correctness. Existing mitigation strategies typically rely on token-uniform length penalties, which provide coarse, segment-agnostic pressure toward shorter outputs and can inadvertently suppress useful reasoning alongside redundancy. To address this, we demonstrate that inefficiency concentrates in high-probability segments with low marginal utility. We derive a theoretical characterization of segment suboptimality under the correctness-length trade-off objective and propose SLAT (Segment-Level Adaptive Trimming), an RL framework that selectively suppresses redundant segments based on this criterion. Empirical results on standard benchmarks indicate that SLAT establishes a superior accuracy-efficiency Pareto frontier, reducing reasoning length by 50% relative to uncompressed baselines while maintaining competitive accuracy. Overall, our results suggest that theoretically grounded, segment-aware trimming is a promising direction for efficient CoT reasoning in large language models.
Jian Yao, Xiongcai Luo, Ran Cheng +1
Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University · Bytedance · The Hong Kong Polytechnic University-Daya Bay Technology and Innovation Research Institute, Huizhou,Guangdong Province, China.
Reasoning segmentation aims to segment target objects described by complex language through joint visual-textual reasoning. Existing methods typically rely on either learned semantic tokens to bridge Multimodal Large Language Models (MLLMs) and segmentation models, suffering from difficult cross-modal alignment, or explicit spatial prompts such as bounding boxes, which may lose holistic response semantics. To address these limitations, we propose Attention-Guided and CoT-Enhanced Coarse-to-Refined Reasoning Segmentation, termed CR-Seg, a two-stage framework for coarse-to-refined reasoning segmentation. Specifically, we design an Extract Attention Maps and Points (EAP) module to extract attention maps for coarse target localization and select informative points, both of which are fed into SAM for mask refinement. To alleviate reasoning--answer inconsistency, we further introduce Global-to-Local Chain-of-Thought (GLCoT), which guides the model to reason progressively from global scene context to local target details. Extensive experiments on reasoning segmentation benchmarks demonstrate the effectiveness of CR-Seg.
Yifan Cao, Xiaocui Yang, Faxian Wan +3
School of Computer Science and Engineering, Northeastern University Shenyang 110819, China