FIRM: Fine-Grained Intra-Token Representation of Masks for Remote Sensing Reasoning Segmentation

Weidong Tang; Kaiyu Li; Yikai Wang; Yanan Wu; Haotian Gan; Shihong Wang; Xiangyong Cao

cs.CVAug 14, 2026

FIRM: Fine-Grained Intra-Token Representation of Masks for Remote Sensing Reasoning Segmentation

Authors: Weidong Tang, Kaiyu Li, Yikai Wang, Yanan Wu, Haotian Gan, Shihong Wang, Xiangyong Cao

Organizations: Xi’an Jiaotong University, Xi’an, China · China Agricultural University, Beijing, China · Renmin University of China, Beijing, China · Shaanxi University of Science and Technology, Xi’an, China

Abstract

Reasoning segmentation requires multimodal large language models (MLLMs) to translate implicit instructions into precise pixel-level masks. MLLMs encode an image as visual tokens, each of which merges a group of image patches. In remote sensing images, small targets, thin structures, and adjacent instances can occupy different parts of the same visual token. Assigning a single binary mask label to such a token loses its internal spatial structure, causing nearby targets to merge and object boundaries to become coarse. To bridge this representational gap, we introduce FIRM, a Fine-grained Intra-token Representation of Masks. For each visual token, FIRM predicts a mask code that specifies an $r\times r$ binary sub-cell pattern rather than a single foreground/background label. Given a target identified by the MLLM, the complete grid of mask codes is predicted in one mask pass. Fixed lookup converts the predicted codes into a discrete sub-cell mask, while marginalizing the code distribution yields a soft structural field. To further recover fine-grained boundaries within each sub-cell, we introduce a lightweight continuous renderer that refines this field using pre-merge visual features and image details. Across five reasoning and referring segmentation benchmarks on satellite and UAV images, FIRM achieves leading results, including $70.5/80.5$ gIoU/cIoU on LaSeRS and a $3.0$ -point average gain on EarthReason. These results demonstrate the value of explicitly representing intra-token mask patterns for fine-grained MLLM segmentation.

Figures & tables

Figure 1: Comparison of MLLM-based segmentation paradigms. (a) Embedding Prediction (e.g., SegEarth-R2 ( Xin et al. 2025 ) ) relies on a separate mask decoder. (b) Mask-Token Prediction (e.g., STAMP ( Liu et al. 2026 ) ) represents each visual token with one foreground or background label. (c) FIRM represents each visual token with an $r\times r$ binary sub-cell pattern and refines the decoded mask with PACR.

Figure 2: Overview of the FIRM pipeline. The MLLM first identifies the targets and emits one [SEG] token for each target. Conditioned on the corresponding target representation, a grid of [MASK] queries predicts a distribution over mask codes at every visual-token position in one mask pass, with attention blocked across targets. Fixed lookup decodes the most probable codes into a discrete sub-cell mask, while marginalization yields a soft structural field. PACR further refines this field into the final pixel-level mask.

Method	Segmentation Granularity			Target Multiplicity		Reasoning Requirement		Expression Length		Avg.
Method	Semantic	Instance	Part	Single	Multiple	Explicit	Implicit	Short	Long	Avg.
With Mask Decoder
LISA-7B	26.4/23.2	20.5/25.0	16.1/11.6	37.3/32.2	18.2/22.4	27.1/24.3	21.5/25.6	34.1/27.8	38.4/33.9	26.6/25.1
LISA-13B	27.0/24.5	22.3/25.6	17.7/13.1	38.4/34.2	19.9/23.5	27.1/25.5	22.6/25.8	35.2/28.0	38.4/34.3	27.6/26.1
PixelLM-7B	32.0/32.8	26.6/30.0	13.2/16.5	44.3/40.4	20.2/23.5	25.0/23.1	23.9/21.9	41.6/38.9	37.1/34.5	29.3/29.1
PixelLM-13B	31.6/34.0	27.5/30.2	15.8/17.6	42.2/40.5	20.9/22.4	26.3/24.4	25.9/22.1	42.0/39.1	37.1/34.5	29.9/29.4

Table 1: Comparison on LaSeRS. Each entry is gIoU/cIoU (%); bold and underline mark the best and second-best value.

Figure 3: Qualitative comparisons on remote sensing reasoning segmentation. The last column shows the discrete sub-cell mask produced by the same FIRM model before continuous rendering.

Method	Validation		Test		Average
Method	gIoU	cIoU	gIoU	cIoU	Average
PixelLM	57.9	57.8	60.0	59.2	58.7
LISA	61.0	57.4	60.9	59.1	59.6
PSALM	66.6	62.0	68.3	64.6	65.4
Seg-Zero	67.2	65.4	67.9	62.4	65.7
VisionReasoner	66.6	67.1	67.2	65.4	66.6

Table 2: Comparison on the EarthReason dataset. Results are reported in %; Average is the arithmetic mean of validation and test gIoU/cIoU.

Zero-shot
Method	Attribute	Scene	Spatial
SegEarth-R1	20.58/11.18	26.33/16.11	25.83/14.87
LISA-13B	52.65/48.12	47.08/42.41	42.85/37.94
LISA-7B	49.21/45.15	45.55/41.59	42.00/37.50
LISA-7B (expl.)	48.40/43.77	42.79/37.86	38.69/34.97
LISAt-7B	43.80/46.36	42.13/44.20	39.91/37.45

Table 3: Comparison on the DRSeg UAV test set under zero-shot transfer and benchmark-specific SFT. Each entry is gIoU/cIoU (%).

Method	LaSeRS Avg. gIoU/cIoU $\uparrow$	Time (s) $\downarrow$	Peak VRAM (GB) $\downarrow$
Published native outputs
Text4Seg-4B	52.6/56.2	54.26	10.57
UFO-8B	50.6/61.2	2.50	17.79
SELF1E-2B	63.3/73.1	1.71	4.98
SELF1E-8B	68.7/77.1	1.92	17.76
STAMP-2B	46.7/64.2	0.99	4.91

Table 4: Ablation of intra-token subdivision granularity and PACR on LaSeRS. We report average gIoU/cIoU (%), mean inference time per sample, and peak GPU memory.

Method	RRSIS-D				RISBench
Method	Validation		Test		Validation		Test
Segmentation specialists
RIS-DMMI	60.7/	77.0	60.1/	76.2	62.6/	70.6	63.9/	74.8
LAVT	61.5/	77.6	61.0/	77.2	60.5/	69.4	61.9/	74.2
RMSIN	65.1/	78.3	64.2/	77.8	61.8/	69.5	63.1/	74.1
MLLM-based methods

Table 5: Comparison on explicit remote sensing referring segmentation. Each entry is gIoU/cIoU (%); a dash marks a metric the method does not report.

Method	LLM	LHRS-Bench		LaSeRS Avg.
Method	LLM	Base	Tuned	gIoU/cIoU
STAMP-7B	Qwen2-VL-7B	69.06	38.71	49.8/65.7
STAMP-2B	Qwen2-VL-2B	54.18	25.98	46.7/64.2
SegEarth-R2-3B	Mipha-3B	65.43	49.59	57.2/67.9
Text4Seg-4B	Qwen3-VL-4B	74.54	61.51	52.6/56.2
UFO-8B	InternVL2.5-8B	67.14	45.89	50.6/61.2

Table 6: LHRS-Bench micro-average accuracy (%) before and after LaSeRS segmentation tuning, together with the average LaSeRS gIoU/cIoU (%) of each tuned checkpoint.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

$\mathcal{L}_{\rm rep}$			$\mathcal{L}_{\rm pix}$		LaSeRS Avg.
$\mathcal{L}_{\rm CE}$	$\mathcal{L}_{\rm Ham}$	$\mathcal{L}_{\rm Dice}^{\rm str}$	$\mathcal{L}_{\rm BCE}^{\rm pix}$	$\mathcal{L}_{\rm Dice}^{\rm pix}$	gIoU/cIoU
$\times$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	68.59 / 78.97
$\checkmark$	$\times$	$\checkmark$	$\checkmark$	$\checkmark$	69.28 / 79.24
$\checkmark$	$\checkmark$	$\times$	$\checkmark$	$\checkmark$	69.70 / 80.10
$\checkmark$	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	69.94 / 80.05
$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	66.00 / 78.46

Appendix

Table 13: Ablation study of the loss components on LaSeRS. Results are reported in %.

Figure 4: Additional qualitative comparisons. Each row shows the input image, ground-truth mask, UFO-8B, STAMP-7B, and FIRM at $r=1,2,3$ , with and without PACR. The comparisons illustrate the effects of subdivision granularity and continuous rendering on the predicted masks.

Figure 5: Qualitative results of FIRM on LaSeRS.

Figure 6: Qualitative results of FIRM on EarthReason.

Figure 7: Qualitative results of FIRM on DRSeg.

Figure 8: Qualitative results of FIRM on RRSIS-D.

Figure 9: Qualitative results of FIRM on RISBench.

Figure 10: Qualitative multi-turn referring segmentation examples (Part I).

Figure 11: Qualitative multi-turn referring segmentation examples (Part II).

Remote Sensing Image Segmentation

Remote sensing image segmentation labels land cover and objects in satellite, aerial, radar, and multispectral imagery.

See all 213 papers

GeoReason: Bridging Logical Reasoning and Spatial Fidelity in Remote Sensing Segmentation
Oct 16, 2026Cheng Fan, Han Wu, Tingzhang Luo +3Reasoning Segmentation Remote Sensing Image Segmentation
City University of Hong Kong · School of Computer Science, Peking University · Samsung
GeoSeg: Training-Free Reasoning-Driven Segmentation in Remote Sensing Imagery
Mar 4, 2026Lifan Jiang, Yuhang Pei, oxi Wu +5Remote Sensing Image Segmentation Reasoning Segmentation
State Key lab of CAD&CG, Zhejiang University · UniTTEC Co. Ltd. · Wuchan Zhongda Chengtou (Ningbo) Holdings. Ltd.
HeatTok: Enhancing Remote Sensing Image Understanding via Thermodiffusion-based Tokenization
Aug 23, 2026Yingying Yan, Jiaqi Tang, Wei Wei +6Visual Tokenization Remote Sensing Image Understanding
Hong Kong University of Science and Technology Hong Kong, China

Oct 16, 2026J/K move · Enter open · S save

GeoReason: Bridging Logical Reasoning and Spatial Fidelity in Remote Sensing Segmentation

Cheng Fan, Han Wu, Tingzhang Luo, Minjing Dong+2

City University of Hong Kong · School of Computer Science, Peking University · Samsung

Reasoning Segmentation Remote Sensing Image Segmentation Remote Sensing VLMs Spatial Reasoning in Language Models

Open paper AskPDF arXiv

Multimodal Large Language Models (MLLMs) have significantly advanced instruction-driven perception through the "Embedding-as-Mask" paradigm. However, extending this capability to Remote Sensing (RS) reasoning segmentation remains exceptionally challenging under cluttered geographical contexts and extreme scale variations, often leading to Spatial Fidelity Degradation and Target Misidentification. We identify a critical yet underexplored bottleneck: the sequential alienation of the segmentation token [SEG] from its initial visual grounding within the autoregressive generation paradigm. Produced only at the end of long reasoning chains, [SEG] relies on visual evidence that has been progressively diluted by linguistic abstraction and autoregressive noise. This information decay is particularly catastrophic for RS targets with minuscule pixel footprints, where even marginal fidelity loss leads to localization failure. To address these bottlenecks, we propose GeoReason, a novel framework designed to fortify logical deduction and preserve spatial integrity. First, we introduce an Anticipatory Prior mechanism that injects the [SEG] token directly into the initial user query, shifting it from a terminal output to a primary condition. This paradigm shift transforms the token into a persistent spatial anchor, preventing information decay during long-form linguistic inference. Second, we enhance visual reasoning via a Location-aware Chain-of-Thought (CoT) that enforces coarse-to-fine localization using bounding boxes, coupled with a Latent Patch-to-Token Alignment (PTA) Loss that explicitly aligns the latent segmentation token with physical RS textures. Extensive experiments demonstrate that GeoReason consistently outperforms state-of-the-art methods on the Earthreason and RRSIS-D benchmarks. Code is available at https://github.com/Fanc0126/GeoReason.

Abstract

Remote Sensing Image Segmentation

Explore similar work

GeoReason: Bridging Logical Reasoning and Spatial Fidelity in Remote Sensing Segmentation

GeoSeg: Training-Free Reasoning-Driven Segmentation in Remote Sensing Imagery

HeatTok: Enhancing Remote Sensing Image Understanding via Thermodiffusion-based Tokenization