Current visual tokenizers in Multimodal Large Language Models (MLLMs) predominantly rely on patch-based partitioning, which causes severe semantic mixture and object fragmentation in remote sensing imagery due to the irregular contours of geo-objects. Moreover, existing adaptive methods struggle to extract precise object-level tokens and lack dedicated geometric positional encodings for irregular regions. In this paper, we propose HeatTok, a semantic-aware tokenizer driven by thermodiffusion aggregation. Inspired by the physical principles of heat conduction, HeatTok adaptively merges adjacent homogeneous regions to generate semantically independent, object-aligned irregular tokens. To enable MLLMs to perceive these irregular shapes, we design the Gaussian Multimodal Rotary Positional Embedding (G-MRoPE), which models token spatial distributions via 2D Gaussians and explicitly injects center, scale, and orientation cues. Extensive evaluations on the VRSBench and EarthVQA datasets demonstrate that HeatTok effectively preserves object-level semantic integrity and achieves state-of-the-art performance under a reasonable token budget. The code is available: https://github.com/YingyingYan1/HeatTok.
Figures & tables
Figure 1. Motivation and paradigm comparison of visual tokenizers. Patch-based fragment irregular geo-objects, harming Semantic Integrity. Dynamic Merging groups tokens but remains grid-confined, yielding blocky outlines. HeatTok leverages thermodiffusion to generate object-level tokens that adhere to true boundaries.
Figure 2. The overall framework of our method. Global branch extracts regular tokens with grid positions BPos(P1) . Semantic branch uses thermodiffusion to generate object-aligned irregular tokens. Their positions BPos(P2) are parameterized into 2D Gaussians—center ( Pc ), scale ( Pe ), and orientation ( ρ )—and explicitly injected via G-MRoPE before token concatenation.
Figure 3. Thermodiffusion-guided token merging. (a) Initial over-segmented regions act as graph nodes. (b) Heat diffuses via semantic consistency to merge correlated subgraphs. (c) Yields semantically complete, object-aligned regions.
Figure 4. (a) M-RoPE: Assigns discrete 2D integer coordinates to uniform grids, ignoring object boundaries. (b) G-MRoPE: Uses 2D Gaussian fitting to explicitly encode the continuous centers (x,y) , scales (w,h) , and orientations ρ of irregular regions, enabling precise perception of complex geo-objects.
Figure 5. Visual comparison of semantic integrity across different tokenization methods. HeatTok preserves crisp object boundaries and avoids over-segmentation.
Visual Tokenizer
Tokens
Throughput
Category
Presence
Quantity
Color
Shape
Size
Position
Direction
Scene
Reasoning
All
Patch Embed ( Dosovitskiy et al. 2020 )
324
24.6
83.86
90.94
57.25
72.06
75.12
63.60
65.56
51.99
81.82
72.95
75.03
Dynamic Grouping and Merging
ToMe ( Bolya et al. 2022 )
164
25.43
82.13
91.92
50.39
72.14
75.26
63.60
60.57
50.52
83.45
73.17
73.24
Chat-UniVi ( Jin et al. 2024 )
112
27.99
85.38
91.50
55.15
74.79
75.83
63.49
67.57
53.02
83.61
74.15
75.87
Quadtree ( Ronen et al. 2023 )
144
29.97
56.12
88.01
50.30
61.52
76.77
60.87
54.77
49.01
63.26
72.07
64.16
HOOK ( Shao et al. 2024 )
128
35.33
70.69
79.33
60.65
74.85
73.12
58.73
59.60
51.17
79.10
68.41
69.95
Table 1. Performance comparison of different tokenization methods across various dimensions on VRSBench. Tokens denotes average number of visual tokens per image, and Throughput denotes number of images processed per second (images/s). Red indicates the best performance, while blue denotes the second best.
Prompt: What are the water situations around the agricultural land? What are the needs for the renovation of villages?
Ground Truth: There are polluted waters near the agriculture land. The roads need to be improved , and waters need to be cleaned up .
Patch Embed: There are no water area. The roads need to be improved . Analysis: Uniform grid partitioning rigidly fragments the continuous polluted water body and agricultural land. This disruption of semantic integrity causes the model to fail entirely in recognizing the presence of the water body.
Dynamic Grouping and Merging
ToMe: There are clean waters near the agriculture land. Waters need to be cleaned up .
Table 3. Case study on remote sensing visual question answering. We show the visual question answering results of a specific scenario along with the tokenization visualizations for different methods. Red indicates hallucinations or incorrect perceptions; Green indicates accurate alignment with the ground truth.
Thermodiffusion
GMrope
Category
Presence
Quantity
Color
Shape
Size
Position
Direction
Scene
Reasoning
All
P(c)
P(e)
ρ
×
×
×
×
83.86
90.94
57.25
72.06
75.12
63.6
65.56
51.99
81.82
72.95
75.03
✓
×
×
×
83.47
90.65
56.41
77.17
75.98
60.94
65.38
52.06
82.45
68.18
75.15
✓
✓
×
×
83.82
91.14
55.96
76.92
76.15
61.72
67.34
52.94
82.81
73.63
75.72
✓
✓
✓
×
84.26
91.20
56.27
77.28
76.54
63.34
67.86
53.24
83.62
74.26
76.14
✓
✓
✓
✓
84.45
91.41
56.36
77.32
78.07
64.20
67.96
55.35
83.75
74.28
76.37
Table 4. Ablation on core components of HeatTok . P(c) , P(e) , and ρ denote the Gaussian center, scale, and orientation encodings.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Symbol
Value
Physical Interpretation & Impact
Base Diffusivity
κ0
1.0
Controls the global baseline speed of heat exchange between all adjacent regions.
Color Sensitivity Scale
σC
5.0
Controls sensitivity to color differences during diffusion. A larger value makes diffusion less sensitive to color variations, allowing heat to flow across minor spectral differences.
Temperature Scale
σT
5.0
Controls color sensitivity for initial temperature assignment ( T(0) ). A smaller value requires high neighbor similarity to form a "Heat Source".
Complexity Factor
α
0.5
Modulates the influence of texture complexity ( Ci ). A larger value accelerates heat exchange in high-frequency/complex regions.
Merging Threshold
τm
0.03
The threshold for temperature difference. A larger value leads to more aggressive merging and coarser tokens.
Appendix
Table 5. List of hyperparameters for the Thermodiffusion-Guided Token Merging algorithm, including their symbols, optimal values, and physical interpretations.
Experiment Set
κ0
σC
σT
α
τm
(Diffusivity)
(Color Sens.)
(Temp. Scale)
(Complexity)
(Threshold)
Optimal (Ours)
1.0
5.0
5.0
0.5
0.03
Set A
2.0
10.0
10.0
0.5
0.03
Set B
1.0
5.0
5.0
0.0
0.03
Set C
4.0
5.0
5.0
0.5
0.03
Appendix
Table 6. Hyperparameter settings for different experimental ablation sets.
Figure 6. Visual comparison of tokenization results under different hyperparameter configurations.
Proposal Generator
Frontend Latency (ms)
Frontend Params (M)
Category
Presence
Quantity
Color
Shape
Size
Position
Direction
Scene
Reasoning
All
FastSAM (w/o Thermo)
12.1
68.0
76.53
85.27
51.56
68.78
71.82
58.55
59.88
50.52
78.59
68.74
69.84
SLIC + Thermo
45.2
24.0
82.52
90.13
55.21
75.84
76.56
62.48
65.47
53.44
82.37
72.84
74.77
SAM-H + Thermo
1468.4
624.0
84.23
91.61
56.12
77.49
78.15
64.53
67.52
55.03
83.54
74.57
76.28
FastSAM + Thermo (Ours)
15.3
92.0
84.45
91.41
56.36
77.32
78.07
64.20
67.96
55.35
83.75
74.28
76.37
Appendix
Table 7. Ablation on region proposal dependency on VRSBench ( Li et al. 2024 ) . w/o Thermo indicates using raw proposals without our aggregation module. Frontend Params strictly includes the 24.0M parameters of the thermodiffusion module where applicable. All represents the sample-weighted average accuracy. Red indicates the best performance, while blue denotes the second best.
Proposal Generator
Frontend Latency (ms)
Frontend Params (M)
BasJu
RelJu
BasCo
RelCo
ObjAn
ComAn
All
FastSAM (w/o Thermo)
12.1
68.0
75.13
77.28
68.32
74.56
55.09
50.43
70.60
SLIC + Thermo
45.2
24.0
81.12
83.96
73.25
79.83
65.17
56.48
76.84
SAM-H + Thermo
1468.4
624.0
82.83
85.19
74.30
82.04
67.81
59.06
78.39
FastSAM + Thermo (Ours)
15.3
92.0
82.60
85.94
75.06
81.67
68.87
59.49
78.88
Appendix
Table 8. Ablation on region proposal dependency on EarthVQA ( Wang et al. 2024 ) , showing that gains stem from thermodiffusion rather than a specific frontend.
Figure 7. Visual grounding comparison between Patch Embed and HeatTok on Qwen2.5-VL, highlighting improved object-boundary alignment and more precise spatial localization.
Figure 8. Qualitative segmentation results on EarthVQA, showing more accurate boundaries and better preserved land-cover integrity.
Figure 9. Robustness analysis. HeatTok corrects over-segmentation by merging fragmented proposals into coherent regions (left), but performance degrades under severe haze (right).
Figure 10. Visual comparisons of tokenization results across different paradigms. Compared to baseline methods, HeatTok successfully preserves crisp object boundaries, maintains geometric continuity, and effectively avoids over-segmentation.
Prompt: What are the land use types in this scene? What is the situation of barren land?
Ground Truth: There are park, and agricultural areas . There is no barren land .
Patch Embed: I can observe that there are park, and agricultural areas within the scene. However, it seems the barren land is scattered across multiple small patches. Analysis: The uniform grid rigidly cuts across the curved boundaries of the park and fields. This destruction of geometric continuity causes the model to misinterpret local texture variations (like winding paths or dry grass) as scattered barren land.
Dynamic Grouping and Merging
ToMe: I can identify industrial zones alongside agricultural areas . The barren land appears to be distributed unevenly.
Appendix
Table 9. Case study on remote sensing visual question answering. We show the visual question answering results of a specific scenario along with the tokenization visualizations for different methods. Red indicates hallucinations or incorrect perceptions; Green indicates accurate alignment with the ground truth.
Prompt: What are the water situations around the agricultural land? What are the needs for the renovation of villages?
Ground Truth: There is clean water near the agriculture land. The roads need to be improved .
Patch Embed: Based on the scene, I can see dried-up water channels near the agriculture land. The roads need to be improved , and new residential buildings should be constructed . Analysis: The uniform grid partitioning rigidly bisects the meandering river and adjacent fields. This spatial fragmentation disrupts the clear water textures, causing the model to misinterpret the disjointed river segments as "dried-up channels" and hallucinate unrelated construction needs.
Dynamic Grouping and Merging
ToMe: There are polluted streams near the agriculture land. The roads need to be improved and residential areas expanded .
Appendix
Table 10. Case study on remote sensing visual question answering. We show the visual question answering results of a specific scenario along with the tokenization visualizations for different methods. Red indicates hallucinations or incorrect perceptions; Green indicates accurate alignment with the ground truth.
Reasoning segmentation requires multimodal large language models (MLLMs) to translate implicit instructions into precise pixel-level masks. MLLMs encode an image as visual tokens, each of which merges a group of image patches. In remote sensing images, small targets, thin structures, and adjacent instances can occupy different parts of the same visual token. Assigning a single binary mask label to such a token loses its internal spatial structure, causing nearby targets to merge and object boundaries to become coarse. To bridge this representational gap, we introduce FIRM, a Fine-grained Intra-token Representation of Masks. For each visual token, FIRM predicts a mask code that specifies an r×r binary sub-cell pattern rather than a single foreground/background label. Given a target identified by the MLLM, the complete grid of mask codes is predicted in one mask pass. Fixed lookup converts the predicted codes into a discrete sub-cell mask, while marginalizing the code distribution yields a soft structural field. To further recover fine-grained boundaries within each sub-cell, we introduce a lightweight continuous renderer that refines this field using pre-merge visual features and image details. Across five reasoning and referring segmentation benchmarks on satellite and UAV images, FIRM achieves leading results, including 70.5/80.5 gIoU/cIoU on LaSeRS and a 3.0-point average gain on EarthReason. These results demonstrate the value of explicitly representing intra-token mask patterns for fine-grained MLLM segmentation.
Weidong Tang, Kaiyu Li, Yikai Wang +4
Xi’an Jiaotong University, Xi’an, China · China Agricultural University, Beijing, China · Renmin University of China, Beijing, China +1
Multimodal Large Language Models (MLLMs) have demonstrated immense potential in Earth observation. However, the massive visual tokens generated when processing Ultra-High-Resolution (UHR) imagery introduce prohibitive computational overhead, severely bottlenecking their inference efficiency. Existing visual token compression methods predominantly adopt static and uniform compression strategies, neglecting the inherent "Semantic-Geometric Duality" in remote sensing interpretation tasks. Specifically, object semantic tasks focus on the abstract semantics of objects and benefit from aggressive background pruning, whereas scene geometric tasks critically rely on the integrity of spatial topology. To address this challenge, we propose DualComp, a task-adaptive dual-stream token compression framework. Dynamically guided by a lightweight pre-trained router, DualComp decouples feature processing into two dedicated pathways. In the object semantic stream, the Spatially-Contiguous Semantic Aggregator (SCSA) utilizes size-adaptive clustering to aggregates redundant background while protecting small object. In the scene geometric stream, the Instruction-Guided Structure Recoverer (IGSR) introduces a greedy path-tracing topology completion mechanism to reconstruct spatial skeletons. Experiments on the UHR remote sensing benchmark XLRS-Bench demonstrate that DualComp accomplishes high-fidelity remote sensing interpretation at an exceptionally low computational cost, achieving simultaneous improvements in both efficiency and accuracy.
Yueying Li, Fengxiang Wang, Yan Li +3
College of Computer Science and Technology, National University of Defense Technology, Changsha 410073, China · School of Computer Science, Beijing University of Posts and Telecommunications, Beijing 100876, China
Multimodal Large Language Models (MLLMs) have significantly advanced instruction-driven perception through the "Embedding-as-Mask" paradigm. However, extending this capability to Remote Sensing (RS) reasoning segmentation remains exceptionally challenging under cluttered geographical contexts and extreme scale variations, often leading to Spatial Fidelity Degradation and Target Misidentification. We identify a critical yet underexplored bottleneck: the sequential alienation of the segmentation token [SEG] from its initial visual grounding within the autoregressive generation paradigm. Produced only at the end of long reasoning chains, [SEG] relies on visual evidence that has been progressively diluted by linguistic abstraction and autoregressive noise. This information decay is particularly catastrophic for RS targets with minuscule pixel footprints, where even marginal fidelity loss leads to localization failure. To address these bottlenecks, we propose GeoReason, a novel framework designed to fortify logical deduction and preserve spatial integrity. First, we introduce an Anticipatory Prior mechanism that injects the [SEG] token directly into the initial user query, shifting it from a terminal output to a primary condition. This paradigm shift transforms the token into a persistent spatial anchor, preventing information decay during long-form linguistic inference. Second, we enhance visual reasoning via a Location-aware Chain-of-Thought (CoT) that enforces coarse-to-fine localization using bounding boxes, coupled with a Latent Patch-to-Token Alignment (PTA) Loss that explicitly aligns the latent segmentation token with physical RS textures. Extensive experiments demonstrate that GeoReason consistently outperforms state-of-the-art methods on the Earthreason and RRSIS-D benchmarks. Code is available at https://github.com/Fanc0126/GeoReason.
Cheng Fan, Han Wu, Tingzhang Luo +3
City University of Hong Kong · School of Computer Science, Peking University · Samsung