Attention Maps Know When to Think: Training-Free Difficulty-Aware Chain-of-Thought for Reasoning Segmentation

Chanseul Cho; Jaesung Jun; Dongheon Lee; Youngmin Ro

cs.CVOct 30, 2026

Attention Maps Know When to Think: Training-Free Difficulty-Aware Chain-of-Thought for Reasoning Segmentation

Authors: Chanseul Cho, Jaesung Jun, Dongheon Lee, Youngmin Ro

Organizations: Machine Intelligence Laboratory, University of Seoul Data Science Interdisciplinary Major

Abstract

Chain-of-Thought (CoT) prompting has become a standard reasoning strategy in Multi-modal Large Language Models (MLLMs), showing strong performance in complex vision-language tasks such as reasoning segmentation. However, our analysis reveals that indiscriminate CoT usage can lead to degraded performance by inducing unnecessary reasoning, a phenomenon we term overthinking. In reasoning segmentation, overthinking often leads to misalignment between the query's intent and visual grounding. To address this, we propose Difficulty-Aware Chain-of-Thought (DA-CoT), a training-free adaptive framework that selectively triggers CoT based on the input's intrinsic difficulty. We find that the entropy of the model's attention maps reliably reflects task difficulty: low entropy indicates confident, localized grounding, while high entropy suggests uncertainty and potential benefit from CoT. DA-CoT uses this entropy signal to determine whether to apply CoT prompting. Additionally, we introduce a Bidirectional Rethinking mechanism that enables the model to automatically flip its reasoning decision when the initial output fails, enhancing robustness by recovering from failures induced by overthinking and underthinking. DA-CoT demonstrates its effectiveness by achieving state-of-the-art performance on the Reasoning Segmentation benchmarks while significantly improving inference efficiency.

Figures & tables

Figure 1: Comparison of pretrained MLLMs applied to reasoning segmentation in a training-free manner, with and without Chain-of-Thought (CoT) prompting. The CoT-prompted model fails due to overthinking caused by excessive spatial reasoning, while the model without CoT correctly identifies the bicycle.

Figure 2: Qualitative examples illustrating the contrasting effects of CoT prompting in reasoning segmentation. Left : CoT-detrimental cases, where CoT leads to misgrounding by overthinking spatial reasoning; without CoT, the model correctly identifies the target with concentrated attention and low entropy. Right : CoT-beneficial cases, where CoT enhances performance in ambiguous scenarios by guiding reasoning towards the correct region, as reflected in high-entropy attention maps. The visualized attention entropy (bottom right of each map) provides an intrinsic signal that distinguishes when CoT is helpful or harmful.

Figure 3: Attention entropy predicts sample difficulty and per-sample CoT benefit using Qwen2.5VL-7B. (a) Mean attention entropy of easy vs. difficult samples on the proxy set ( $n=1{,}000$ ) at the three layers selected in Sec. 4 ; error bars: $\pm 1$ SEM; *: Mann–Whitney $U$ $p<0.05$ . (b) Per-sample $\Delta$ IoU vs. attention entropy (z-scored within each split) on held-out ReasonSeg val+test (invalid outputs score IoU $=0$ ); gray: individual samples, red: octile means with 95% CI. (c) ROC for predicting $\Delta\text{IoU}>0$ from entropy alone (ties excluded). (d) Mean IoU per entropy quartile with and without CoT, computed over each mode’s valid outputs (bars, left axis), and per-mode model-failure rates (dashed, right axis; Sec. C.1 ).

Figure 4: Overview of the proposed DA-CoT framework during inference. The aggregated attention entropy $\mathcal{H}_{\text{agg}}$ , computed over the pre-identified layers $\mathcal{L}$ , is used to adaptively decide whether to trigger CoT reasoning. If $\mathcal{H}_{\text{agg}}\geq\tau$ , the model follows the Difficulty-Aware Prompting Path with CoT for structured reasoning; otherwise, it takes the direct path without CoT for efficient inference. When the initial reasoning path fails to produce a valid output, a discrete flipping signal activates the Bidirectional Rethinking Path , enabling re-evaluation and recovery from overthinking or underthinking. The final bounding boxes or point prompts are then passed to the segmentation head to produce the reasoning-aware segmentation result.

Figure 5: Distribution of aggregated attention entropy $\mathcal{H}_{\text{agg}}$ over the proxy set. The curve exhibits an S-shaped trend with two elbow points that divide samples by reasoning difficulty. Samples below the first elbow correspond to easy and small-object cases with confident grounding, those between elbows represent medium-complexity examples with larger objects, and those beyond the second elbow correspond to difficult reasoning cases that require CoT prompting. The second elbow is selected as the decision threshold $\tau$ for adaptive CoT activation.

Threshold Strategy	All-CoT	First Elbow	Median	Second Elbow	All-Non-CoT
gIoU	66.7	67.3	67.8	68.2	62.1
CoT Usage (%)	100.0	90.8	49.2	21.2	0.0

Table 1: Comparative analysis of entropy threshold selection strategies. Performance (gIoU in %) and CoT usage are measured without Bidirectional Rethinking. Evaluation is performed on the proxy set described in Section 3.2 .

Training-based methods
Method	ReasonSeg
Method	val	test	Avg.
OVSeg [ 18 ]	28.5	26.1	27.3
ReLA [ 19 ]	22.4	21.3	21.9
SAM4MLLM [ 5 ]	46.7	-	-
LISA-7B-LLaVA1.5 [ 16 ]	53.6	48.7	51.2

Table 2: Performance comparison with zero-shot models on ReasonSeg benchmark. Performance is reported as gIoU. † denotes performance re-evaluated by [ 10 ] .

Training-based methods
Method	ReferringSeg
Method	COCO	COCO+	COCOg	Avg.
LAVT [ 43 ]	75.8	68.4	62.1	68.8
ReLA [ 19 ]	76.5	71.0	66.0	71.2
LISA-7B-LLaVA1.5 [ 16 ]	76.5	67.4	68.5	70.8
PixelLM-7B [ 31 ]	76.5	71.7	70.5	72.9

Table 3: Performance comparison on Referring Expression Segmentation benchmark. Performance is reported as cIoU. † denotes performance re-evaluated by [ 10 ] .

Method	LogicVista	MMVP
Qwen2.5VL-7B [ 2 ] (All-Non-CoT)	37.81	58.00
Qwen2.5VL-7B [ 2 ] (All-CoT)	44.30	55.33
VisionReasoner-7B [ 23 ]	38.93	56.00
Seg-Zero-7B [ 22 ]	38.93	58.67
DA-CoT-7B	45.64	59.33

Table 4: Performance comparison on LogicVista and MMVP. We report accuracy for LogicVista and pair accuracy for MMVP.

Method	ReasonSeg
Method	val	test	Avg.
Seg-Zero-7B † [ 22 ]	62.0	58.3	60.2
Seg-Zero-7B + Ours	66.2 ( $\uparrow$ 4.2)	64.6 ( $\uparrow$ 6.3)	65.4 ( $\uparrow$ 5.2)
VisionReasoner-7B [ 23 ]	66.3	63.6	65.0
VisionReasoner-7B + Ours	67.4 ( $\uparrow$ 1.1)	65.6 ( $\uparrow$ 2.0)	66.5 ( $\uparrow$ 1.5)
Qwen2.5VL-7B [ 2 ]	63.0	60.7	61.9

Table 5: Scalability and robustness of the proposed method on larger and more recent MLLMs. Performance is reported as gIoU. † denotes performance re-evaluated by [ 10 ] .

Method	ReasonSeg (gIoU)		ReferringSeg (cIoU)
Method	val	test	RefCOCO	RefCOCO+	RefCOCOg
Qwen2.5VL-7B	63.0	60.7	73.5	68.2	60.4
+ DA-CoT	65.2 ( $\uparrow$ 2.2)	64.0 ( $\uparrow$ 3.3)	77.6 ( $\uparrow$ 4.1)	72.0 ( $\uparrow$ 3.8)	69.6 ( $\uparrow$ 9.2)
+ BiReT	68.0 ( $\uparrow$ 2.8)	64.6 ( $\uparrow$ 0.6)	77.7 ( $\uparrow$ 0.1)	72.4 ( $\uparrow$ 0.4)	70.7 ( $\uparrow$ 1.1)

Table 6: Ablation study on our proposed components: DA-CoT and BiReT. The baseline applies the CoT prompt to all samples.

Method	time/img (sec)	gIoU (Test)
LISA-7B [ 16 ]	7.0	48.7
Seg-Zero-7B [ 22 ]	7.0	58.3
VisionReasoner-7B [ 23 ]	7.0	63.6
Qwen2.5VL-7B [ 2 ] (All-Non-CoT)	3.2	58.0
Qwen2.5VL-7B [ 2 ] (All-CoT)	9.9	60.7
DA-CoT-7B (w/o BiReT)	4.4	64.0

Table 7: Inference latency and performance on the ReasonSeg test. All-CoT and All-Non-CoT denote applying CoT or direct prediction to all samples, respectively.

Threshold Strategy	val	test	CoT Usage (%)
All-CoT	63.0	60.7	100.0
First Elbow	63.9	61.0	76.1
Median	64.6	61.9	51.6
Second Elbow	65.2	64.0	35.0
All-Non-CoT	59.9	58.0	0.0

Table 8: Ablation study on entropy threshold selection for ReasonSeg without Bidirectional Rethinking. Reported in gIoU.

Method	Selected Layers	Threshold ( $\tau$ )	ReasonSeg (gIoU)			LogicVista	MMVP
Method	Selected Layers	Threshold ( $\tau$ )	val	test	Avg.	LogicVista	MMVP
Qwen2.5VL-7B [ 2 ] (All-Non-CoT)	-	-	59.9	58.0	59.0	37.81	58.00
Qwen2.5VL-7B [ 2 ] (All-CoT)	-	-	63.0	60.7	61.9	44.30	55.33
DA-CoT (default)	23, 24, 25	6.21	68.0	64.6	66.3	45.64	59.33
DA-CoT (OK-VQA [ 26 ] )	16, 24, 25	6.14	67.2	65.0	66.1	44.52	62.00
DA-CoT (A-OKVQA [ 33 ] )	16, 23, 25	6.07	66.8	64.7	65.8	45.86	61.33

Table 9: Transfer of the selected layers and threshold across proxy sources. Parameters determined on each source via the procedure of Sec. 4.2 are fixed and directly applied to all benchmarks.

ReasonSeg	$k=1$	$k=2$	$k=3$	$k=4$	$k=5$
val	64.4	64.1	65.2	66.4	63.9
test	63.4	63.5	64.0	63.9	62.2
selected layers	23	23, 25	23-25	23-26	22-26

Table 10: Ablation study on the number of layers

k

without Bidirectional Rethinking.

Method	UniReT ( $t$ =0.5)	UniReT ( $t$ =0.7)	UniReT ( $t$ =1.0)	BiReT (Ours)
Valid Recovery Rate	35.4%	38.7%	35.5%	97.2%
Correct Recovery Rate	0.9%	2.8%	2.8%	22.2%
$\Delta$ gIoU	+0.1	+0.1	+0.1	+0.6

Table 11: Comparison of recovery behavior and performance change on the ReasonSeg test set.

Figure 6: CoT prompt for DA-CoT

Method	ReasonSeg (gIoU)		ReferringSeg (cIoU)
Method	val	test	RefCOCO	RefCOCO+	RefCOCOg
All-Non-CoT	59.9	58.0	63.2	58.0	57.2
All-CoT	63.0	60.7	73.5	68.2	60.4

Table 12: Quantitative validation of the structured CoT prompt design. Performance of the Qwen2.5VL-7B model is reported.

Formulation	Layers	$\tau$	ReasonSeg (gIoU)
Formulation	Layers	$\tau$	val / test
Default (head-averaged)	23, 24, 25	6.21	65.2 / 64.0
Per-head entropy	23, 24, 25	6.16	65.5 / 64.1

Table 13: Comparison between the default head-averaged entropy and the alternative per-head entropy formulation on ReasonSeg, reported without Bidirectional Rethinking. The two formulations yield identical layer selection and comparable performance.

Model	Selected Layers	Thresholds
Seg-Zero-7B [ 22 ]	8, 23, 24	5.95
VisionReasoner-7B [ 23 ]	23, 24, 25	6.32
Qwen2.5VL-7B [ 2 ]	23, 24, 25	6.21
Qwen2.5VL-32B [ 2 ]	44, 45, 58	5.15
Qwen2.5VL-72B [ 2 ]	63, 73, 74	5.93
Qwen3VL-8B [ 1 ]	8, 31, 33	5.09

Table 14: Layers and thresholds selected during the layer and threshold determination phase using the proxy set.

Setting	Layers	$\tau$	ReasonSeg (gIoU)
Setting	Layers	$\tau$	val	test
Default proxy	23, 24, 25	6.21	65.2	64.0
5-seed random	23, 24, 25	6.23 $\pm$ 0.1	65.0 $\pm$ 0.3	64.2 $\pm$ 0.1
Category-balanced proxy	23, 24, 25	6.38	66.3	64.6
Target-cat.-disjoint proxy	23, 24, 25	6.06	65.0	64.0

Table 15: Robustness of DA-CoT to proxy set construction on the ReasonSeg benchmark, reported without Bidirectional Rethinking. Despite varying the sampling seed, category balance, and target-category coverage, the selected layers, threshold

\tau

, and gIoU remain stable.

Method	ReasonSeg		Selected Layers ( $L$ )	Threshold ( $\tau$ )
Method	val	test	Selected Layers ( $L$ )	Threshold ( $\tau$ )
DA-CoT	68.0	64.6	23, 24, 25	6.21
DA-CoT (w/o proxy set)	67.6	64.2	23, 24, 25	6.24

Table 16: Comparison between DA-CoT and DA-CoT (w/o proxy set) on ReasonSeg benchmark. Performance is reported as gIoU.

Figure 7: Attention entropy vs. ground-truth target size on held-out ReasonSeg val+test. Annotations report the within-size-tercile AUC of entropy for per-sample CoT benefit.

Method	Venue	Base model	gIoU	cIoU	Training
LISA [ 16 ]	CVPR‘2024	LLaVA1.5-7B	48.7	48.8	✓
LISA [ 16 ]	CVPR‘2024	LLaVA1.5-13B	53.8	50.8	✓
Seg-Zero [ 22 ]	-	Qwen2.5-VL 3B	56.1	48.6	✓
RSVP [ 25 ]	ACL‘2025	Qwen2-VL 7B	56.6	51.6	$\times$
RSVP [ 25 ]	ACL‘2025	Gemini-Flash	57.1	59.2	$\times$
Seg-Zero † [ 22 ]	-	Qwen2.5-VL 7B	58.3	53.4	✓

Table 17: Performance comparison with zero-shot models on ReasonSeg test set. Performance is reported as gIoU and cIoU. † denotes performance re-evaluated by [ 10 ] .

Figure 8: Response format and rule-based failure detection criteria for BiReT. The mechanism triggers a flipping signal when structural components shown in red are incomplete or coordinate values shown in blue are missing.

Figure 9: Qualitative examples of BiReT.

Metric	val	test
Trigger Rate (%)	5.0	4.6
Valid Recovery Rate (%)	80.0	97.2
Correct Recovery Rate (%)	70.0	22.2

Table 18: Efficiency and Efficacy of the BiReT Mechanism. Both recovery rates are computed over the triggered samples.

Method	ReasonSeg (gIoU)		time/img (sec)
	val	test	time/img (sec)
Qwen 2.5-VL 7B	63.0	60.7	9.9
+ DA-CoT	65.2 ( $\uparrow$ 2.2)	64.0 ( $\uparrow$ 3.3)	4.4
+ BiReT	68.0 ( $\uparrow$ 2.8)	64.6 ( $\uparrow$ 0.6)	4.7

Table 19: Performance and latency are measured on the ReasonSeg Benchmark.

Failure Cause	Proportion (%)
Spatial confusion	50.3
Semantic drift	18.8
Visual hallucination	16.4
Format errors / others	14.5

Table 20: Failure-mode breakdown of 165 randomly sampled CoT-detrimental cross-over cases (

\mathrm{IoU}_{\mathrm{NonCoT}}>0.5\wedge\mathrm{IoU}_{\mathrm{CoT}}\leq 0.5

) on ReasonSeg. Spatial confusion is the dominant failure mode, indicating that unnecessary reasoning most often derails grounding through spatial-logic errors.

Figure 10: Overthinking of VisionReasoner

Adaptive Reasoning in Language Models

See all 49 papers

Attention-guided Fine-tuning of Multimodal Large Language Models Improves Chain-of-Thought Reasoning
Jun 1, 2026Sanchit Sinha, Guangzhi Xiong, Bohan Liu +2Multimodal CoT Reasoning Multimodal Large Language Models
University of Virginia
Efficient Reasoning via Thought Compression for Language Segmentation
Apr 2, 2026Qing Zhou, Shiyu Zhang, Yuyu Jia +4Reasoning Segmentation CoT Compression
Northwestern Polytechnical University, China · Northwest Institute of Nuclear Technology, China.
CR-Seg: Attention-Guided and CoT-Enhanced Coarse-to-Refined Reasoning Segmentation
Jun 2, 2026Yifan Cao, Xiaocui Yang, Faxian Wan +3Text-Guided Image Segmentation Promptable Image Segmentation
School of Computer Science and Engineering, Northeastern University Shenyang 110819, China

Jun 1, 2026J/K move · Enter open · S save

Attention-guided Fine-tuning of Multimodal Large Language Models Improves Chain-of-Thought Reasoning

Sanchit Sinha, Guangzhi Xiong, Bohan Liu, Zhenghao He+1

University of Virginia

Multimodal CoT Reasoning Multimodal Large Language Models VLM Reasoning Fine-Tuning Visually Grounded Reasoning Multimodal Understanding+1

Open paper AskPDF arXiv

The effectiveness of Chain-of-Thought (CoT) prompting in Multimodal Large Language Models (MLLMs) remains uncertain: across several visual reasoning benchmarks, CoT prompting often degrades performance compared to direct prompting. In this paper, we provide a systematic analysis of CoT behavior in three modern MLLM families across model scales on datasets requiring step-wise visual evidence. Our analysis identifies two recurring failure modes: premature answer commitment and limited direct visual-token access during rationale generation. We further find that standard CoT-style Supervised Fine-Tuning (CoT-SFT) can mitigate these issues only partially, while often increasing reliance on textual priors and reducing counterfactual visual dependence. Motivated by these findings, we propose Attentive-CoT (Att-CoT), an attention-guided fine-tuning objective that encourages CoT trajectories to delay answer commitment while maintaining sustained visual-token access. Att-CoT can be plugged into any CoT-SFT training run without architectural changes. Experiments on three visual reasoning benchmarks across six MLLMs show that Att-CoT enhances CoT performance over standard fine-tuning.

Abstract

Adaptive Reasoning in Language Models

Explore similar work

Attention-guided Fine-tuning of Multimodal Large Language Models Improves Chain-of-Thought Reasoning

Efficient Reasoning via Thought Compression for Language Segmentation

CR-Seg: Attention-Guided and CoT-Enhanced Coarse-to-Refined Reasoning Segmentation