Chain-of-thought (CoT) reasoning has significantly improved the performance of large multimodal models in language-guided segmentation, yet its prohibitive computational cost, stemming from generating verbose rationales, limits real-world applicability. We introduce WISE (Wisdom from Internal Self-Exploration), a novel paradigm for efficient reasoning guided by the principle of thinking twice -- once for learning, once for speed. WISE trains a model to generate a structured sequence: a concise rationale, the final answer, and then a detailed explanation. By placing the concise rationale first, our method leverages autoregressive conditioning to enforce that the concise rationale acts as a sufficient summary for generating the detailed explanation. This structure is reinforced by a self-distillation objective that jointly rewards semantic fidelity and conciseness, compelling the model to internalize its detailed reasoning into a compact form. At inference, the detailed explanation is omitted. To address the resulting conditional distribution shift, our inference strategy, WISE-S, employs a simple prompting technique that injects a brevity-focused instruction into the user's query. This final adjustment facilitates the robust activation of the learned concise policy, unlocking the full benefits of our framework. Extensive experiments show that WISE-S achieves state-of-the-art zero-shot performance on the ReasonSeg benchmark with 58.3 cIoU, while reducing the average reasoning length by nearly \textbf{5×} -- from 112 to just 23 tokens. Code is available at WISE.
Figures & tables
Figure 1: WISE achieves a superior cost-performance trade-off by decoupling reasoning for learning and inference. Our framework trains the model with detailed explanations but uses a distilled, concise rationale at inference. The base model, WISE, reduces reasoning cost by an average of 4.7 × while outperforming the baseline. Our final model, WISE-S, applies an additional inference-time prompt, achieving even greater compression (up to 10.6 × on RefCOCO) and the highest overall performance. On ReasonSeg, WISE-S simultaneously improves performance by 3.9 cIoU and cuts token cost by 4.9 × .
Figure 2: Overview of the WISE framework. During training (orange), the reasoning model Freason generates a structured sequence: concise rationale ( τc ), answer ( A ), and detailed explanation ( τd ). The process is optimized via GRPO using a hierarchical reward, notably including a self-distillation term ( Rdistill ) that aligns τc with τd while enforcing brevity. The segmentation model Fseg remains frozen. At inference (blue), τd is omitted and a brevity prompt ( Tb ) is added to ensure highly efficient reasoning.
Figure 3: Prompting Strategy for WISE. The figure shows the basic instruction template and the specific components for generating concise ( Tc ) and detailed ( Td ) rationales. Different combinations of these components are used for the training phase, standard WISE inference, and the brevity-focused WISE-S inference.
ReasonSeg
val
test
Method
gIoU
cIoU
gIoU
cIoU
OVSeg ( Liang et al., 2023 )
28.5
18.6
26.1
20.8
ReLA ( Liu et al., 2023 )
22.4
19.9
21.3
22.0
Grounded-SAM ( Ren et al., 2024a )
26.0
14.5
21.3
16.4
LISA-7B-LLaVA1.5 ( Lai et al., 2024 )
53.6
52.3
48.7
48.8
Table 1: Comparison with state-of-the-art methods on the Zero-shot ReasonSeg benchmark. WISE-7B and WISE-7B-S significantly outperform prior methods. ∗ We re-evaluate with their official checkpoints. † Results reported in the original paper using different checkpoints.
RCO
RCO+
RCOg
Method
testA
testA
test
LAVT ( Yang et al., 2022 )
75.8
68.4
62.1
ReLA ( Liu et al., 2023 )
76.5
71.0
66.0
LISA-7B ( Lai et al., 2024 )
76.5
67.4
68.5
PixelLM-7B ( Ren et al., 2024b )
76.5
71.7
70.5
MagNet ( Chng et al., 2024 )
78.3
73.6
69.3
Table 2: Comparison with state-of-the-art methods on the Referring Expression Segmentation benchmark. Our models achieve competitive or superior performance against specialized methods while being more general.
RefCOCO testA
RefCOCO+ testA
RefCOCOg test
ReasonSeg (cIoU)
Method
#Tok ↓
cIoU ↑
#Tok ↓
cIoU ↑
#Tok ↓
cIoU ↑
#Tok ↓
val ↑
#Tok ↓
test ↑
Seg-Zero
165.1
78.8
159.9
74.8
159.8
71.3
117.5
57.4
111.9
54.4
Seg-Zero+L1-Exact
31.2
78.3
31.1
74.9
30.7
71.7
36.6
57.2
35.6
55.0
Seg-Zero+L1-Max
11.7
78.9
11.7
74.6
10.6
70.7
10.5
43.8
12.0
49.7
WISE (ours)
35.0
79.1
33.8
74.0
34.0
71.8
38.8
59.2
38.8
58.5
vs. Seg-Zero
(4.7 ×↓ )
(+0.3)
(4.7 ×↓ )
(-0.8)
(4.7 ×↓ )
(+0.5)
(3.0 ×↓ )
(+1.8)
(2.9 ×↓ )
(+4.1)
Table 3: Performance and Efficiency Comparison on Referring and Reasoning Segmentation Benchmarks. Our WISE models drastically reduce token overhead (#Tok) while improving or maintaining accuracy (cIoU) across all benchmarks compared to the Seg-Zero and brevity-focused reward shaping methods (L1-Exact, L1-Max ( Aggarwal and Welleck, 2025 ) ). The sub-rows show the token reduction factor ( ×↓ ) and the absolute performance change ( Δ ) against the Seg-Zero.
Method
Det. COCO
Seg ReaSeg
Count Pixmon
#Tok
mAP
#Tok
gIoU
#Tok
Acc
VisionReasoner-7B
62.5
38.8
80.1
64.0
74.5
70.1
VisionReasoner-7B-S
42.1
39.1
45.0
63.6
53.7
69.3
+WISE (Ours)
67.7
39.5
63.0
66.0
84.9
68.2
+WISE-S (Ours)
15.7
39.6
17.1
66.1
19.2
73.0
Table 4: Generalization to Unified Tasks. Validating WISE on the VisionReasoner across three distinct visual reasoning tasks. Note that VisionReasoner-7B-S applies our brevity prompt to the baseline, confirming the prompt’s generality but highlighting the superior compression of the full WISE training.
Order
RefCOCOg
ReasonSeg
#Tok ↓
cIoU ↑
#Tok ↓
cIoU ↑
A→τc→τd
74.6
70.1
75.7
54.4
τd→τc→A
107.5
68.2
106.5
55.4
τc→τd→A
134.7
70.1
132.7
55.2
τc→A→τd (Ours)
34.0
71.8
38.8
58.5
Table 5: Ablation on Generation Order. Placing the concise rationale first ( τc→A→τd ) is essential for enabling effective thought compression at inference time.
Components
RefCOCOg
ReasonSeg
I
Ssim
Sconcise
#Tok ↓
cIoU ↑
#Tok ↓
cIoU ↑
159.8
71.3
111.9
54.4
✓
✓
67.8
70.1
75.2
54.6
✓
✓
46.2
68.6
43.6
57.8
✓
✓
78.9
70.7
74.2
56.3
✓
✓
✓
34.0
71.8
38.8
58.5
Table 6: Component Analysis of Self-Distillation Reward. All three terms—the conditional indicator ( I ), semantic similarity ( Ssim ), and conciseness score ( Sconcise )—are requisite for optimal performance.
Inference Strategy
RefCOCOg
ReasonSeg
#Tok ↓
cIoU ↑
#Tok ↓
cIoU ↑
Seg-Zero (Baseline)
159.8
71.3
111.9
54.4
WISE-F (+ τc+τd )
188.5
71.3
197.5
58.8
Ablation: Prompt Modification
−τc−τd
51.5
70.3
59.9
54.8
−τc+τd
200.0
68.4
221.4
53.2
Table 7: Effect of Inference Strategies. While the default WISE setting ( +τc−τd ) offers substantial compression, the targeted brevity prompt in WISE-S achieves the optimal efficiency-accuracy balance.
Figure 4: Distribution of Reasoning Token Length. The KDE plot illustrates the dramatic reduction in both length and variance. WISE-S (green) demonstrates a tightly concentrated preference for conciseness compared to the long-tailed Seg-Zero baseline (purple).
Figure 5: Qualitative Comparison. Given a complex instruction, Seg-Zero produces a convoluted, 132-token rationale and fails. In contrast, WISE-S generates a concise, correct reasoning chain, successfully identifying the target with a fraction of the cost.
Method
RefCOCO testA
ReasonSeg test
#Tok
Time
cIoU
#Tok
Time
cIoU
Qwen2.5-VL-7B
0.0
11.2
77.8
0.0
7.6
41.2
Seg-Zero (Baseline)
165.1
153.1
78.8
111.9
50.0
54.4
Seg-Zero-S (Prompt)
85.6
116.1
79.0
57.9
20.3
55.0
WISE (Ours)
35.0
28.9
79.1
38.8
17.0
58.5
WISE-S (Ours)
24.0
16.1
79.1
22.7
10.4
58.3
Table 8: Wall-Clock Efficiency Analysis. Comparison of inference latency and token costs. While the raw model underperforms and prompting alone (Seg-Zero-S) yields limited gains, WISE-S achieves a 5 × speedup over the Seg-Zero baseline.
Method
ReasonSeg val
ReasonSeg test
#Tok
gIoU
cIoU
#Tok
gIoU
cIoU
Architecture: Qwen2-VL-7B
Qwen2-VL-7B
0.0
44.5
-
0.0
38.7
-
WISE (Ours)
32.7
45.8
42.6
31.2
44.2
40.4
WISE-S (Ours)
23.8
44.9
42.5
23.1
44.7
40.9
Scale: Qwen2.5-VL-3B
Table 9: Scalability across Architectures. Performance consistency of WISE across different model families (Qwen2 vs. Qwen2.5) and parameter scales (3B vs. 7B).
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Success Case Study Examples. The figures demonstrate how WISE-S acts as a sufficient statistic. (a) Shows the model efficiently identifying an object by its function (“take small sips”) without verbose description. (b) Shows the model correctly filtering out spatial redundancy to focus on discriminative visual attributes (“red and white obstacle”).
Figure 7: Analysis of Failure Cases: Semantic Fidelity amidst Grounding Errors. These examples illustrate instances where the model fails to produce an accurate mask (Low IoU). Crucially, however, the Concise Rationale ( τc ) remains a faithful summary of the Detailed Explanation ( τd ). (a) In the top example, both τc and τd correctly identify the semantic topic but struggle to ground the abstract concept to specific pixels. (b) In the bottom example, both rationales exhibit circular logic without identifying distinct visual attributes. This indicates that the failures stem from the underlying limitations of the base model’s spatial grounding capabilities or reasoning loops, rather than information loss caused by the WISE-S compression mechanism.
Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University · Bytedance · The Hong Kong Polytechnic University-Daya Bay Technology and Innovation Research Institute, Huizhou,Guangdong Province, China.