Think, Compress, Answer: Reinforcement Learning for Multimodal Grounding via Reasoning Compression

Jinbae Seo; Hyeongjun Kwon; Jiyoung Lee; Hyunsung Jang; Namkoo Ha; Jungin Park; Kwanghoon Sohn

cs.AIOct 19, 2026

Think, Compress, Answer: Reinforcement Learning for Multimodal Grounding via Reasoning Compression

Authors: Jinbae Seo, Hyeongjun Kwon, Jiyoung Lee, Hyunsung Jang, Namkoo Ha, Jungin Park, Kwanghoon Sohn

Organizations: Yonsei University · School of AI and Software, Ewha Womans University · LIG Defense & Aerospace · Korea Institute of Science and Technology (KIST)

Abstract

Extended reasoning has substantially improved multimodal large language models (MLLMs) on tasks requiring compositional and relational inference. However, multimodal grounding imposes a distinct requirement: reasoning must ultimately converge on a target-specific compact representation that supports precise localization. As reasoning trajectories grow longer, they often expand into broad scene narratives and peripheral relational details, which can dilute grounded perceptual evidence and lead to ambiguous or incorrect predictions. Motivated by the information bottleneck principle, we propose Think-Compress-Answer (TCA), a reinforcement learning framework that decomposes multimodal grounding into three phases: free-form reasoning, target-aware context compression, and answer generation conditioned on the compressed context. TCA is optimized through a multi-stage Group Relative Policy Optimization (GRPO) with progressive rewards that establish the structured output protocol, minimize reasoning-specific information while preserving target-relevant content, and directly improve localization accuracy. This staged design enables MLLMs to retain the benefits of extended reasoning while distilling it into the minimal context required for grounding. Experiments across six multimodal grounding benchmarks show that TCA attains substantial zero-shot performance gains, demonstrating the effectiveness of reasoning compression.

Figures & tables

Panel of Figure 1 — Figure 1: Motivation of Think-Compress-Answer. Given multimodal inputs, ( b ) standard CoT and ( c ) answer-reward-only RL methods identify the correct entity but drift during localization. ( d ) Our Think-Compress-Answer distills reasoning into a concise, faithful cue, yielding accurate grounding. The predicted bounding boxes and center points in (a) and the output from each method (b–d) are matched in color.

Figure 2: Overview of Think-Compress-Answer. The model generates $N$ grouped responses $[\mathbf{T},\mathbf{c},\mathbf{a}]$ per input. Stage 1 establishes the required output protocol, Stage 2 encourages reasoning compression, Stage 3 uses a judge MLLM to promote compression fidelity and target relevance, and Stage 4 improves grounding accuracy. The stage-specific rewards are converted into group-relative advantages to progressively update the model.

Supervised, Fine-Tuning
Method	MLLM	Dec.	RefCOCO			RefCOCO+			RefCOCOg
Method	MLLM	Dec.	val	tA	tB	val	tA	tB	val	test
Qwen2.5-Omni-7B Qwen Team (2025a)	Qw2.5OM-7B	SAM2 ∗	76.2	78.0	71.3	69.4	75.1	62.7	71.5	70.5
Supervised, Embedding-Guided
LISA Lai et al. (2024)	LLaVA-7B	SAM	74.1	76.5	71.1	62.4	67.4	56.5	67.9	70.6
PixelLM Ren et al. (2024)	LLaVA-7B	Pix.Dec.	73.0	76.5	68.2	66.3	71.7	58.3	69.3	70.5

Table 1: Referring expression segmentation on RefCOCO, RefCOCO+, and RefCOCOg. We report the performance (cIoU) evaluated on validation, testA, and testB splits for RefCOCO and RefCOCO+, and validation and test splits for RefCOCOg. * employs a frozen decoder (Dec.) only for mask prediction.

Supervised, Fine-Tuning
Method	MLLM	Dec.	RY-VOS	RD17	MeViS
Qwen2.5-Omni-7B Qwen Team (2025a)	Qw2.5OM-7B	SAM2 ∗	69.3	76.8	53.3
Supervised, Embedding-Guided
VISA Yan et al. (2024)	ChatUniVi-7B	SAM †	61.5	69.4	43.5
ViLLa Zheng et al. (2025)	Vicuna-7B	E2E	67.5	74.3	49.4
Sa2VA Yuan et al. (2025)	IntVL2.5-8B	SAM2	70.7	75.2	46.0

Table 2: Referring video object segmentation on Ref-YouTube-VOS, Ref-DAVIS17, and MeViS. We report

\mathcal{J}\&\mathcal{F}

evaluated on the validation set of each benchmark. * employs a frozen decoder (Dec.) only for mask prediction.

\dagger

additionally employs an external XMem tracker for temporal mask propagation.

Figure 4: Qualitative comparisons for referring video object segmentation on Ref-YouTube-VOS.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Framework	ms-swift
Precision	bfloat16
Attention	FlashAttention
Learning rate	$1\times 10^{-6}$
Optimizer	AdamW
LR scheduler	Cosine

Appendix

Table A1: Hyperparameters for training

Panel of Figure 8 — Figure A1: Prompt for (a) the structured answer block, (b) Compression fidelity in Stage 3.

Method	MLLM	Dec.	ReVOS			RefAVS (Seen)			RefAVS (Unseen)
Method	MLLM	Dec.	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$
Qwen2.5-Omni-7B Qwen Team (2025a)	Qw2.5OM-7B	SAM2	38.4	37.2	39.6	35.5	30.1	40.9	64.8	64.8	64.9
Qwen2.5-Omni-7B-SFT Qwen Team (2025a)	Qw2.5OM-7B	SAM2	55.4	53.0	57.8	69.9	67.7	72.0	77.0	77.0	77.1
Omni-R1 Zhong et al. (2025)	Qw2.5OM-7B	SAM2	47.6	–	–	47.2	43.0	51.4	74.2	71.3	77.0
TCA ( Ours )	Qw2.5OM-7B	SAM2	60.5	57.7	62.2	70.7	68.0	73.4	78.8	78.7	78.8

Appendix

Table B1: Reasoning video object segmentation on the ReVOS Yan et al. (2024) benchmark and referring audio-visual segmentation on the RefAVS Wang et al. (2024) benchmark. We report region similarity (

\mathcal{J}

), contour accuracy (

\mathcal{F}

), and their average (

\mathcal{J}\&\mathcal{F}

). We use SAM2 as the decoder.

Oracle with GT
Method	MLLM	Dec.	RefCOCO			RefCOCO+			RefCOCOg
Method	MLLM	Dec.	val	tA	tB	val	tA	tB	val	test
SAM Kirillov et al. (2023)	–	SAM	82.6	84.1	81.3	82.0	84.1	81.3	80.4	81.4
SAM2 Ravi et al. (2024)	–	SAM2	83.2	84.9	81.9	83.0	84.9	81.9	81.0	81.8
SAM3 Carion et al. (2025)	–	SAM3	26.4	26.5	27.4	26.4	26.5	27.5	34.3	35.1
SAM3 + Carion et al. (2025)	–	SAM3 +	76.1	75.8	75.4	76.4	75.8	75.4	72.9	73.8

Appendix

Table B2: Referring expression segmentation on RefCOCO Yu et al. (2016) , RefCOCO+ Yu et al. (2016) , and RefCOCOg Mao et al. (2016) benchmarks. We report the performance (cIoU) evaluated on validation, testA, and testB splits for RefCOCO and RefCOCO+, and validation and test splits for RefCOCOg. For SAM3,

+

denotes joint text-and-bounding-box prompting, whereas the standard setting uses text-only prompting.

Method	MLLM	Dec.	ReVOS			RefAVS (Seen)			RefAVS (Unseen)
Method	MLLM	Dec.	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$
Oracle with GT
SAM Kirillov et al. (2023)	-	SAM	–	–	–	93.3	91.4	95.2	91.9	93.0	90.8
SAM2 Ravi et al. (2024)	-	SAM2	79.6	76.8	82.6	92.2	90.4	94.1	92.0	92.5	91.6
SAM3 Carion et al. (2025)	-	SAM3	25.5	24.1	26.8	66.6	65.3	67.9	71.4	71.4	71.3
SAM3 + Carion et al. (2025)	-	SAM3 +	30.9	30.1	31.6	71.6	70.3	72.0	76.5	75.1	77.8

Appendix

Table B3: Reasoning video object segmentation on the ReVOS Yan et al. (2024) benchmark and referring audio-visual segmentation on the RefAVS Wang et al. (2024) benchmark. We report region similarity (

\mathcal{J}

), contour accuracy (

\mathcal{F}

), and their average (

\mathcal{J}\&\mathcal{F}

Oracle with GT
Method	MLLM	Dec.	RY-V	RD17	MeViS
SAM2 Ravi et al. (2024)	-	SAM2	86.8	87.4	81.8
SAM3 Carion et al. (2025)	-	SAM3	43.3	51.3	34.5
SAM3 + Carion et al. (2025)	-	SAM3 +	46.7	52.7	40.5
LLM-3B with SAM2
Qwen2.5-Omni-3B Qwen Team (2025a)	Qw2.5OM-3B	SAM2	32.4	53.1	21.2

Appendix

Table B4: Referring video object segmentation on the Ref-YouTube-VOS (RY-V) Seo et al. (2020) , Ref-DAVIS17 (RD17) Khoreva et al. (2018) , and MeViS Ding et al. (2023a) benchmarks. We report the performance (

\mathcal{J}\&\mathcal{F}

) evaluated on the validation set of each benchmark. + indicates that the pre-trained frozen decoder (Dec.) takes both a text and bounding box coordinates as inputs.

LLM Judge	RY-V	RD17	MeViS	ReVOS	RefAVS(U)
Qw3VL-8B	72.3	78.9	55.8	60.5	78.8
Qw2.5VL-7B	71.9	78.1	54.5	58.5	77.6
Qw2.5VL-32B	71.4	78.5	53.6	58.7	77.2
Llama3.2Vi-11B	70.5	74.6	52.9	57.6	76.8

Appendix

Table B5: Ablation on the LLM judge. We report

\mathcal{J}\&\mathcal{F}

on the referring video object segmentation benchmarks, the RefAVS Unseen test split, and the ReVOS validation split. We use SAM2 as the decoder.

S1	S2	S3	S4	RefCOCO			RefCOCO+			RefCOCOg
S1	S2	S3	S4	val	tA	tB	val	tA	tB	val	test
✗	✗	✗	✗	74.5	75.8	69.1	66.5	71.2	60.5	71.8	71.2
✓	✗	✗	✗	75.6	77.2	70.6	68.2	73.5	61.8	72.9	72.1
✓	✓	✗	✗	76.2	77.9	71.0	68.4	74.1	62.4	73.2	72.8
✓	✓	✓	✗	76.8	78.5	70.8	69.5	74.8	63.1	72.9	73.3
✓	✗	✗	✓	77.5	79.6	71.6	70.8	76.2	64.2	73.9	73.8

Appendix

Table B6: Performance with respect to variants of the stages on the referring expression segmentation benchmarks. We report cIoU evaluated on validation, testA, and testB splits for RefCOCO and RefCOCO+, and validation and test splits for RefCOCOg. We use SAM2 as the decoder.

S1	S2	S3	S4	$\mathcal{J}\&\mathcal{F}$
S1	S2	S3	S4	ReVOS	RefAVS (U)
✗	✗	✗	✗	48.8	64.8
✓	✗	✗	✗	50.5	69.0
✓	✓	✗	✗	55.7	73.3
✓	✓	✓	✗	58.2	75.6
✓	✗	✗	✓	56.6	74.9

Appendix

Table B7: Performance with respect to variants of the stages on ReVOS and RefAVS. We report

\mathcal{J}\&\mathcal{F}

on the ReVOS validation set and the RefAVS Unseen test split. We use SAM2 as the decoder.

Panel of Figure 16 — Figure B1: Performance at each stage on (a) RefCOCO testA, (b) ReVOS, and (c) RefAVS Unseen.

Panel of Figure 17 — Figure B2: Compressed-context token distribution and reward dynamics. (a) Distribution of compressed-context lengths across optimization stages. (b) Compression-rate reward during Stage 2. (c) Compression-fidelity reward during Stage 3.

Panel of Figure 18 — Figure B3: Qualitative examples on RefAVS, ReVOS and R-VOS benchmarks.

Figure B4: Impact of compression. TCA extracts target-relevant context from the reasoning path, yielding a more precise mask.

Figure B5: Failure case from RefAVS. When the reasoning phase identifies a related but incorrect object, compression preserves the erroneous target context and cannot recover the correct localization.

Multimodal Grounding

Multimodal grounding links language and other signals to the specific objects, regions, events, or actions that support an interpretation.

See all 185 papers

Mixture-of-Thought-Tokens: Unifying Perception and Reasoning for Free-form Multimodal Grounding
Jul 27, 2026Tianyi Gao, Han Fang, Tianyi Ding +9Multimodal Grounding Referring Expression Comprehension
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University · Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd · Beijing University of Posts and Telecommunications +2
MCPO: Modality-Contrastive Preference Optimization for Multimodal Chain-of-Thought Compression
Sep 4, 2026Guangheng Yang, Zhenliang Ni, Zhenkai Wu +4Efficient Multimodal Inference Multimodal Large Language Models
Huawei Technologies Ltd. Shanghai, China
MG-Thinker: Bi-Axial Self-Reflection for Multi-Image Reasoning Grounding
Sep 29, 2026Heyu Huang, Chi Chen, Zonghao Guo +3Multimodal Grounding Multimodal CoT Reasoning
Huazhong University of Science and Technology Wuhan, China · Tsinghua University Beijing, China

Jul 27, 2026J/K move · Enter open · S save

Mixture-of-Thought-Tokens: Unifying Perception and Reasoning for Free-form Multimodal Grounding

Tianyi Gao, Han Fang, Tianyi Ding, Hao Li+8

State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University · Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd · Beijing University of Posts and Telecommunications+2

Multimodal Grounding Referring Expression Comprehension Multimodal Large Language Models Multimodal CoT Reasoning

Open paper AskPDF arXiv

Multimodal Large Language Models have made great progress in grounding tasks, yet existing methods still struggle to unify precise localization and complex reasoning. For one thing, text-based methods rely on coordinates or index prediction, severely limiting the perceptual capabilities of the model for dense visual objects. Meanwhile, latent token-based methods employ special tokens without inherent spatial references and use a decoding mechanism that lacks thinking steps, weakening high-level reasoning capabilities. Consequently, developing a unified framework that excels in both perception and reasoning remains challenging. To address this, we propose Mixture-of-Thought-Tokens (Motto), a new free-form multimodal grounding method that bridges the perception-reasoning gap, enabling MLLMs to empower diverse, arbitrary grounding queries. Specifically, we introduce Spatially-Grounded Thought Tokenization to explicitly align special tokens with spatial locations for clear spatial correspondence and visual interpretability. We further design a Context-Adaptive Chain-of-Tokens that dynamically switch grounding modes within an interleaved reasoning chain, achieving robust grounding across tasks of varying complexity. In addition, we construct PR-Bench, a new referring expression comprehension benchmark to evaluate the perception-reasoning gap. Extensive experiments demonstrate that Motto achieves state-of-the-art performance across diverse free-form grounding tasks.

Abstract

Multimodal Grounding

Explore similar work

Mixture-of-Thought-Tokens: Unifying Perception and Reasoning for Free-form Multimodal Grounding

MCPO: Modality-Contrastive Preference Optimization for Multimodal Chain-of-Thought Compression

MG-Thinker: Bi-Axial Self-Reflection for Multi-Image Reasoning Grounding