Organizations: Xi’an Jiaotong University, Xi’an, China · China Agricultural University, Beijing, China · Renmin University of China, Beijing, China · Shaanxi University of Science and Technology, Xi’an, China
Reasoning segmentation requires multimodal large language models (MLLMs) to translate implicit instructions into precise pixel-level masks. MLLMs encode an image as visual tokens, each of which merges a group of image patches. In remote sensing images, small targets, thin structures, and adjacent instances can occupy different parts of the same visual token. Assigning a single binary mask label to such a token loses its internal spatial structure, causing nearby targets to merge and object boundaries to become coarse. To bridge this representational gap, we introduce FIRM, a Fine-grained Intra-token Representation of Masks. For each visual token, FIRM predicts a mask code that specifies an r×r binary sub-cell pattern rather than a single foreground/background label. Given a target identified by the MLLM, the complete grid of mask codes is predicted in one mask pass. Fixed lookup converts the predicted codes into a discrete sub-cell mask, while marginalizing the code distribution yields a soft structural field. To further recover fine-grained boundaries within each sub-cell, we introduce a lightweight continuous renderer that refines this field using pre-merge visual features and image details. Across five reasoning and referring segmentation benchmarks on satellite and UAV images, FIRM achieves leading results, including 70.5/80.5 gIoU/cIoU on LaSeRS and a 3.0-point average gain on EarthReason. These results demonstrate the value of explicitly representing intra-token mask patterns for fine-grained MLLM segmentation.
Figures & tables
Figure 1: Comparison of MLLM-based segmentation paradigms. (a) Embedding Prediction (e.g., SegEarth-R2 ( Xin et al. 2025 ) ) relies on a separate mask decoder. (b) Mask-Token Prediction (e.g., STAMP ( Liu et al. 2026 ) ) represents each visual token with one foreground or background label. (c) FIRM represents each visual token with an r×r binary sub-cell pattern and refines the decoded mask with PACR.
Figure 2: Overview of the FIRM pipeline. The MLLM first identifies the targets and emits one [SEG] token for each target. Conditioned on the corresponding target representation, a grid of [MASK] queries predicts a distribution over mask codes at every visual-token position in one mask pass, with attention blocked across targets. Fixed lookup decodes the most probable codes into a discrete sub-cell mask, while marginalization yields a soft structural field. PACR further refines this field into the final pixel-level mask.
Method
Segmentation Granularity
Target Multiplicity
Reasoning Requirement
Expression Length
Avg.
Semantic
Instance
Part
Single
Multiple
Explicit
Implicit
Short
Long
With Mask Decoder
LISA-7B
26.4/23.2
20.5/25.0
16.1/11.6
37.3/32.2
18.2/22.4
27.1/24.3
21.5/25.6
34.1/27.8
38.4/33.9
26.6/25.1
LISA-13B
27.0/24.5
22.3/25.6
17.7/13.1
38.4/34.2
19.9/23.5
27.1/25.5
22.6/25.8
35.2/28.0
38.4/34.3
27.6/26.1
PixelLM-7B
32.0/32.8
26.6/30.0
13.2/16.5
44.3/40.4
20.2/23.5
25.0/23.1
23.9/21.9
41.6/38.9
37.1/34.5
29.3/29.1
PixelLM-13B
31.6/34.0
27.5/30.2
15.8/17.6
42.2/40.5
20.9/22.4
26.3/24.4
25.9/22.1
42.0/39.1
37.1/34.5
29.9/29.4
Table 1: Comparison on LaSeRS. Each entry is gIoU/cIoU (%); bold and underline mark the best and second-best value.
Figure 3: Qualitative comparisons on remote sensing reasoning segmentation. The last column shows the discrete sub-cell mask produced by the same FIRM model before continuous rendering.
Method
Validation
Test
Average
gIoU
cIoU
gIoU
cIoU
PixelLM
57.9
57.8
60.0
59.2
58.7
LISA
61.0
57.4
60.9
59.1
59.6
PSALM
66.6
62.0
68.3
64.6
65.4
Seg-Zero
67.2
65.4
67.9
62.4
65.7
VisionReasoner
66.6
67.1
67.2
65.4
66.6
Table 2: Comparison on the EarthReason dataset. Results are reported in %; Average is the arithmetic mean of validation and test gIoU/cIoU.
Method
Attribute
Scene
Spatial
Zero-shot
SegEarth-R1
20.58/11.18
26.33/16.11
25.83/14.87
LISA-13B
52.65/48.12
47.08/42.41
42.85/37.94
LISA-7B
49.21/45.15
45.55/41.59
42.00/37.50
LISA-7B (expl.)
48.40/43.77
42.79/37.86
38.69/34.97
LISAt-7B
43.80/46.36
42.13/44.20
39.91/37.45
Table 3: Comparison on the DRSeg UAV test set under zero-shot transfer and benchmark-specific SFT. Each entry is gIoU/cIoU (%).
Method
LaSeRS Avg. gIoU/cIoU ↑
Time (s) ↓
Peak VRAM (GB) ↓
Published native outputs
Text4Seg-4B
52.6/56.2
54.26
10.57
UFO-8B
50.6/61.2
2.50
17.79
SELF1E-2B
63.3/73.1
1.71
4.98
SELF1E-8B
68.7/77.1
1.92
17.76
STAMP-2B
46.7/64.2
0.99
4.91
Table 4: Ablation of intra-token subdivision granularity and PACR on LaSeRS. We report average gIoU/cIoU (%), mean inference time per sample, and peak GPU memory.
Method
RRSIS-D
RISBench
Validation
Test
Validation
Test
Segmentation specialists
RIS-DMMI
60.7/
77.0
60.1/
76.2
62.6/
70.6
63.9/
74.8
LAVT
61.5/
77.6
61.0/
77.2
60.5/
69.4
61.9/
74.2
RMSIN
65.1/
78.3
64.2/
77.8
61.8/
69.5
63.1/
74.1
MLLM-based methods
Table 5: Comparison on explicit remote sensing referring segmentation. Each entry is gIoU/cIoU (%); a dash marks a metric the method does not report.
Method
LLM
LHRS-Bench
LaSeRS Avg.
Base
Tuned
gIoU/cIoU
STAMP-7B
Qwen2-VL-7B
69.06
38.71
49.8/65.7
STAMP-2B
Qwen2-VL-2B
54.18
25.98
46.7/64.2
SegEarth-R2-3B
Mipha-3B
65.43
49.59
57.2/67.9
Text4Seg-4B
Qwen3-VL-4B
74.54
61.51
52.6/56.2
UFO-8B
InternVL2.5-8B
67.14
45.89
50.6/61.2
Table 6: LHRS-Bench micro-average accuracy (%) before and after LaSeRS segmentation tuning, together with the average LaSeRS gIoU/cIoU (%) of each tuned checkpoint.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Lrep
Lpix
LaSeRS Avg.
LCE
LHam
LDicestr
LBCEpix
LDicepix
gIoU/cIoU
×
✓
✓
✓
✓
68.59 / 78.97
✓
×
✓
✓
✓
69.28 / 79.24
✓
✓
×
✓
✓
69.70 / 80.10
✓
✓
✓
×
✓
69.94 / 80.05
✓
✓
✓
✓
×
66.00 / 78.46
Appendix
Table 13: Ablation study of the loss components on LaSeRS. Results are reported in %.
Figure 4: Additional qualitative comparisons. Each row shows the input image, ground-truth mask, UFO-8B, STAMP-7B, and FIRM at r=1,2,3 , with and without PACR. The comparisons illustrate the effects of subdivision granularity and continuous rendering on the predicted masks.
Figure 5: Qualitative results of FIRM on LaSeRS.
Figure 6: Qualitative results of FIRM on EarthReason.
Figure 7: Qualitative results of FIRM on DRSeg.
Figure 8: Qualitative results of FIRM on RRSIS-D.
Figure 9: Qualitative results of FIRM on RISBench.
Multimodal Large Language Models (MLLMs) have significantly advanced instruction-driven perception through the "Embedding-as-Mask" paradigm. However, extending this capability to Remote Sensing (RS) reasoning segmentation remains exceptionally challenging under cluttered geographical contexts and extreme scale variations, often leading to Spatial Fidelity Degradation and Target Misidentification. We identify a critical yet underexplored bottleneck: the sequential alienation of the segmentation token [SEG] from its initial visual grounding within the autoregressive generation paradigm. Produced only at the end of long reasoning chains, [SEG] relies on visual evidence that has been progressively diluted by linguistic abstraction and autoregressive noise. This information decay is particularly catastrophic for RS targets with minuscule pixel footprints, where even marginal fidelity loss leads to localization failure. To address these bottlenecks, we propose GeoReason, a novel framework designed to fortify logical deduction and preserve spatial integrity. First, we introduce an Anticipatory Prior mechanism that injects the [SEG] token directly into the initial user query, shifting it from a terminal output to a primary condition. This paradigm shift transforms the token into a persistent spatial anchor, preventing information decay during long-form linguistic inference. Second, we enhance visual reasoning via a Location-aware Chain-of-Thought (CoT) that enforces coarse-to-fine localization using bounding boxes, coupled with a Latent Patch-to-Token Alignment (PTA) Loss that explicitly aligns the latent segmentation token with physical RS textures. Extensive experiments demonstrate that GeoReason consistently outperforms state-of-the-art methods on the Earthreason and RRSIS-D benchmarks. Code is available at https://github.com/Fanc0126/GeoReason.
Cheng Fan, Han Wu, Tingzhang Luo +3
City University of Hong Kong · School of Computer Science, Peking University · Samsung
Recent advances in MLLMs are reframing segmentation from fixed-category prediction to instruction-grounded localization. While reasoning based segmentation has progressed rapidly in natural scenes, remote sensing lacks a generalizable solution due to the prohibitive cost of reasoning-oriented data and domain-specific challenges like overhead viewpoints. We present GeoSeg, a zero-shot, training-free framework that bypasses the supervision bottleneck for reasoning-driven remote sensing segmentation. GeoSeg couples MLLM reasoning with precise localization via: (i) bias-aware coordinate refinement to correct systematic grounding shifts and (ii) a dual-route prompting mechanism to fuse semantic intent with fine-grained spatial cues. We also introduce GeoSeg-Bench, a diagnostic benchmark of 810 image--query pairs with hierarchical difficulty levels. Experiments show that GeoSeg consistently outperforms all baselines, with extensive ablations confirming the effectiveness and necessity of each component.
Lifan Jiang, Yuhang Pei, oxi Wu +5
State Key lab of CAD&CG, Zhejiang University · UniTTEC Co. Ltd. · Wuchan Zhongda Chengtou (Ningbo) Holdings. Ltd.
Current visual tokenizers in Multimodal Large Language Models (MLLMs) predominantly rely on patch-based partitioning, which causes severe semantic mixture and object fragmentation in remote sensing imagery due to the irregular contours of geo-objects. Moreover, existing adaptive methods struggle to extract precise object-level tokens and lack dedicated geometric positional encodings for irregular regions. In this paper, we propose HeatTok, a semantic-aware tokenizer driven by thermodiffusion aggregation. Inspired by the physical principles of heat conduction, HeatTok adaptively merges adjacent homogeneous regions to generate semantically independent, object-aligned irregular tokens. To enable MLLMs to perceive these irregular shapes, we design the Gaussian Multimodal Rotary Positional Embedding (G-MRoPE), which models token spatial distributions via 2D Gaussians and explicitly injects center, scale, and orientation cues. Extensive evaluations on the VRSBench and EarthVQA datasets demonstrate that HeatTok effectively preserves object-level semantic integrity and achieves state-of-the-art performance under a reasonable token budget. The code is available: https://github.com/YingyingYan1/HeatTok.
Yingying Yan, Jiaqi Tang, Wei Wei +6
Hong Kong University of Science and Technology Hong Kong, China