Recent advances in MLLMs are reframing segmentation from fixed-category prediction to instruction-grounded localization. While reasoning based segmentation has progressed rapidly in natural scenes, remote sensing lacks a generalizable solution due to the prohibitive cost of reasoning-oriented data and domain-specific challenges like overhead viewpoints. We present GeoSeg, a zero-shot, training-free framework that bypasses the supervision bottleneck for reasoning-driven remote sensing segmentation. GeoSeg couples MLLM reasoning with precise localization via: (i) bias-aware coordinate refinement to correct systematic grounding shifts and (ii) a dual-route prompting mechanism to fuse semantic intent with fine-grained spatial cues. We also introduce GeoSeg-Bench, a diagnostic benchmark of 810 image--query pairs with hierarchical difficulty levels. Experiments show that GeoSeg consistently outperforms all baselines, with extensive ablations confirming the effectiveness and necessity of each component.
Figures & tables
Figure 1: Performance across reasoning difficulty levels. We visualize our proposed GeoSeg’s results on image–query pairs across three difficulty levels. The visualized masks illustrate that GeoSeg can handle varying instruction complexity and remain effective in challenging scenarios. Please refer to Appendix B for more results.
Figure 2: Overview of the GeoSeg pipeline. Given a remote sensing image I and a natural language query q , the pipeline operates in three stages: (1) Reasoning-Driven Grounding : The MLLM ( L ) generates a coarse bounding box b and extracts the object prompt p . (2) Bias-Aware Coordinate Refinement : To mitigate grounding bias, the box is adjusted via asymmetric expansion ( α,β ) to yield a refined RoI Ib′ . (3) Dual-Route Segmentation & Fusion : Within the RoI, we perform parallel segmentation using Route A (Visual Cues via CLIP Surgery) and Route B (Semantic Cues via SAM3 with prompt p ). The final prediction M^ is obtained by integrating both paths via Intersection-First Fusion.
Figure 3: Quantification of domain-specific grounding drift. We analyze coordinate offsets on a held-out calibration set comprising 1,000 images randomly sampled from LoveDA, NWPU-VHR-10, and DIOR datasets. The KDE visualization reveals a systematic bottom-right shift inherent to pre-trained MLLMs under overhead views, necessitating our statistically derived asymmetric expansion ( α=0.2,β=0.1 ).
Figure 4: Overview of GeoSeg-Bench. (a) Representative Domains : We showcase samples from the four domains defined in our scenario taxonomy: Urban , Rural , Traffic , and Nature . (b) Hierarchical Difficulty Design : Using the Traffic domain as a case study, we illustrate the progression across three levels: Basic (Level 1), Description (Level 2), and Reasoning (Level 3), corresponding to increasing reasoning requirements.
Figure 5: Distribution statistics of our GeoSeg-Bench. Left: Proportional breakdown of four scenario categories (Urban, Traffic, Rural, Nature). Right: Fixed ratio composition of three hierarchical difficulty levels (Basic, Description, Reasoning).
Figure 6: Qualitative comparison with multiple baselines. This figure demonstrates the superiority of our approach over three major categories of baseline models: generalist segmentation, reasoning segmentation, and open-source MLLMs. As shown, most baseline methods struggle to comprehend the query intent, resulting in segmentation failures or excessive noise, whereas our method successfully generates accurate masks. More examples are in Appendix B .
Figure 7: Ablation study on component effectiveness. We validate the contribution of the Bias-Aware Coordinate Refinement ( Box Refine ) and the Dual-Route strategy. Route A represents the Point-Prompt path (Visual Cues), and Route B denotes the Text-Prompt path (Semantic Cues). Removing any module significantly degrades performance, validating the necessity of our full pipeline. More examples in Appendix D .
Method
GeoSeg-Bench ( ↑ )
SegEarth-R2 training dataset ( ↑ )
IoU
Dice
Acc.
Prec.
Rec.
Spec.
BF
FPS
IoU
Dice
Acc.
Prec.
Rec.
Spec.
BF
Generalist segmentation
SAM3 ( Carion et al., 2025 )
19.9
26.8
89.3
21.7
58.6
91.8
12.9
-
10.5
14.3
77.0
11.2
47.3
78.0
10.9
Grounded SAM ( Ren et al., 2024a )
18.6
25.1
64.2
20.0
73.1
63.9
10.9
-
11.3
15.1
61.9
13.1
55.2
61.8
11.1
CLIP Surgery + SAM3 ( Li et al., 2023 )
24.7
31.4
73.9
26.8
69.8
74.2
10.4
-
14.5
19.0
69.0
15.5
55.7
69.3
10.1
Reasoning segmentation
Table 1: Quantitative comparison on GeoSeg-Bench and the SegEarth-R2 training dataset. We benchmark GeoSeg against representative methods from three categories: (1) Generalist segmentation , (2) Reasoning segmentation , and (3) Open-source MLLMs . We report seven pixel-level metrics (%) and inference speed (FPS). Best and second-best results are highlighted in bold and underlined , respectively. Methods that fail to segment any target (IoU=0) are excluded from Accuracy and Specificity rankings. Inference speed is evaluated on GeoSeg-Bench with a single A100 GPU.
Method
GeoSeg-Bench ( ↑ )
SegEarth-R2 training dataset ( ↑ )
Qwen-8B
Qwen-32B
User
Qwen-8B
Qwen-32B
User
F.
L.
R.
F.
L.
R.
F.
L.
R.
F.
L.
R.
F.
L.
R.
F.
L.
R.
Generalist segmentation
SAM3 ( Carion et al., 2025 )
1.78
1.72
1.81
2.03
1.74
2.00
2.75
2.30
2.68
1.39
1.36
1.40
1.50
1.37
1.46
2.00
1.87
1.99
Grounded SAM ( Ren et al., 2024a )
1.71
1.65
1.75
1.92
1.64
1.86
2.63
2.17
2.50
1.42
1.39
1.44
1.55
1.41
1.50
2.05
1.88
2.03
CLIP Surgery ( Li et al., 2023 )
1.98
1.91
2.02
2.15
1.88
2.05
2.86
2.51
2.78
1.57
1.52
1.61
1.74
1.54
1.66
2.35
2.05
2.25
Table 2: Model-based and human evaluation on GeoSeg-Bench and the SegEarth-R2 training dataset. We assess reasoning-driven segmentation quality using MLLM judges (Qwen3-VL-8B/32B) and human evaluators (User). Predictions are rated on a 1–5 scale and averaged. Metrics: Faithfulness (F.; compliance with constraints), Localization (L.; correctness of mask boundaries), and Robustness (R.; avoidance of distractors). Best and second-best are bold and underlined , respectively.
Components
GeoSeg-Bench ( ↑ )
SegEarth-R2 training dataset ( ↑ )
Box Refine
Route A
Route B
IoU
Dice
Acc.
Prec.
Rec.
Spec.
BF
IoU
Dice
Acc.
Prec.
Rec.
Spec.
BF
✗
✓
✓
51.1
59.9
96.5
62.7
64.3
97.8
21.7
13.4
17.7
93.9
22.9
21.1
96.2
9.7
✓
✗
✓
52.9
61.4
95.8
59.6
71.5
96.6
24.3
16.0
20.6
93.1
23.6
27.9
95.0
11.6
✓
✓
✗
43.2
49.0
96.5
52.4
48.9
99.0
21.2
15.2
19.1
95.1
22.7
22.6
97.6
11.4
✓
✓
✓
56.4
64.2
96.8
69.0
65.7
98.3
26.6
17.4
22.0
94.7
29.2
25.2
97.0
13.2
Table 3: Ablation study of component effectiveness on GeoSeg-Bench and the SegEarth-R2 training dataset. We evaluate the contribution of the Box Refinement module and the two parallel paths in our Dual-Route mechanism. Route A denotes the Point-Prompt path, and Route B denotes the Text-Prompt path. Removing any component degrades overall performance. Best and second-best are bold and underlined , respectively.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Extended qualitative comparison with multiple baselines. We present five additional challenging cases. Compared to generalist segmentation, reasoning segmentation, and open-source MLLMs, our method consistently interprets complex queries accurately and outputs precise masks. (Figure continued on next page.)
Figure 9: Extended qualitative comparison with multiple baselines. (Continued from previous page.)
Figure 10: Extended qualitative comparison with multiple baselines. (Continued from previous page.)
Figure 11: Extended qualitative ablation study. We showcase three additional cases comparing the full pipeline with its variants. Removing Route A , Route B , or the Box Refine module results in noticeable performance degradation, such as semantic misunderstanding or coarse boundaries, verifying the necessity of our complete architecture.
Multimodal Large Language Models (MLLMs) have significantly advanced instruction-driven perception through the "Embedding-as-Mask" paradigm. However, extending this capability to Remote Sensing (RS) reasoning segmentation remains exceptionally challenging under cluttered geographical contexts and extreme scale variations, often leading to Spatial Fidelity Degradation and Target Misidentification. We identify a critical yet underexplored bottleneck: the sequential alienation of the segmentation token [SEG] from its initial visual grounding within the autoregressive generation paradigm. Produced only at the end of long reasoning chains, [SEG] relies on visual evidence that has been progressively diluted by linguistic abstraction and autoregressive noise. This information decay is particularly catastrophic for RS targets with minuscule pixel footprints, where even marginal fidelity loss leads to localization failure. To address these bottlenecks, we propose GeoReason, a novel framework designed to fortify logical deduction and preserve spatial integrity. First, we introduce an Anticipatory Prior mechanism that injects the [SEG] token directly into the initial user query, shifting it from a terminal output to a primary condition. This paradigm shift transforms the token into a persistent spatial anchor, preventing information decay during long-form linguistic inference. Second, we enhance visual reasoning via a Location-aware Chain-of-Thought (CoT) that enforces coarse-to-fine localization using bounding boxes, coupled with a Latent Patch-to-Token Alignment (PTA) Loss that explicitly aligns the latent segmentation token with physical RS textures. Extensive experiments demonstrate that GeoReason consistently outperforms state-of-the-art methods on the Earthreason and RRSIS-D benchmarks. Code is available at https://github.com/Fanc0126/GeoReason.
Cheng Fan, Han Wu, Tingzhang Luo +3
City University of Hong Kong · School of Computer Science, Peking University · Samsung
Geospatial reasoning requires models to resolve complex spatial semantics and user intent into precise target locations for Earth observation. Recent progress has liberated the reasoning path from manual curation, allowing models to generate their own inference chains. Yet a final dependency remains: they are still supervised by human-annotated ground-truth coordinates. This leaves the reasoning process autonomous, but not its spatial endpoint, and prevents true self-evolution on abundant unlabeled remote sensing data. To break this bottleneck, we introduce RemoteZero, a box-supervision-free framework for geospatial reasoning. RemoteZero is motivated by a simple asymmetry: an MLLM is typically better at verifying whether a region satisfies a query than at directly generating precise coordinates. Leveraging this stronger discriminative ability, RemoteZero replaces geometric supervision with intrinsic semantic verification and enables GRPO training without box annotations. The resulting framework further supports iterative self-evolution, allowing the model to improve from unlabeled remote sensing imagery through its own verification signal. Experiments show that RemoteZero achieves competitive performance against strong supervised methods, demonstrating the potential of self-verifying training for geospatial reasoning localization.
Recent advances in Vision Language Models (VLMs) and Vision Foundation Models (VFMs) have opened new opportunities for zero-shot text-guided segmentation of remote sensing imagery. However, most existing approaches still rely on additional trainable components, limiting their generalisation and practical applicability. In this work, we investigate to what extent text-based remote sensing segmentation can be achieved without additional training, by relying solely on existing foundation models. We propose a simple yet effective approach that integrates contrastive and generative VLMs with the Segment Anything Model (SAM), enabling a fully training-free or lightweight LoRA-tuned pipeline. Our contrastive approach employs CLIP as mask selector for SAM's grid-based proposals, achieving state-of-the-art open-vocabulary semantic segmentation (OVSS) in a completely zero-shot setting. In parallel, our generative approach enables reasoning and referring segmentation by generating click prompts for SAM using GPT-5 in a zero-shot setting and a LoRA-tuned Qwen-VL model, with the latter yielding the best results. Extensive experiments across 19 remote sensing benchmarks, including open-vocabulary, referring, and reasoning-based tasks, demonstrate the strong capabilities of our approach. Code will be released at https://github.com/josesosajs/trainfree-rs-segmentation.