HeatTok: Enhancing Remote Sensing Image Understanding via Thermodiffusion-based Tokenization

Yingying Yan; Jiaqi Tang; Wei Wei; Qianzhou Wang; Jinjian Wu; Botong Geng; Jianmin Chen; Yuyang Xia

cs.CVAug 23, 2026

HeatTok: Enhancing Remote Sensing Image Understanding via Thermodiffusion-based Tokenization

Authors: Yingying Yan, Jiaqi Tang, Wei Wei, Qianzhou Wang, Jinjian Wu, Botong Geng, Jianmin Chen, Yuyang Xia,

Organizations: Hong Kong University of Science and Technology Hong Kong, China

Abstract

Current visual tokenizers in Multimodal Large Language Models (MLLMs) predominantly rely on patch-based partitioning, which causes severe semantic mixture and object fragmentation in remote sensing imagery due to the irregular contours of geo-objects. Moreover, existing adaptive methods struggle to extract precise object-level tokens and lack dedicated geometric positional encodings for irregular regions. In this paper, we propose HeatTok, a semantic-aware tokenizer driven by thermodiffusion aggregation. Inspired by the physical principles of heat conduction, HeatTok adaptively merges adjacent homogeneous regions to generate semantically independent, object-aligned irregular tokens. To enable MLLMs to perceive these irregular shapes, we design the Gaussian Multimodal Rotary Positional Embedding (G-MRoPE), which models token spatial distributions via 2D Gaussians and explicitly injects center, scale, and orientation cues. Extensive evaluations on the VRSBench and EarthVQA datasets demonstrate that HeatTok effectively preserves object-level semantic integrity and achieves state-of-the-art performance under a reasonable token budget. The code is available: https://github.com/YingyingYan1/HeatTok.

Figures & tables

Figure 1. Motivation and paradigm comparison of visual tokenizers. Patch-based fragment irregular geo-objects, harming Semantic Integrity. Dynamic Merging groups tokens but remains grid-confined, yielding blocky outlines. HeatTok leverages thermodiffusion to generate object-level tokens that adhere to true boundaries.

Figure 2. The overall framework of our method. Global branch extracts regular tokens with grid positions $BPos(P_{1})$ . Semantic branch uses thermodiffusion to generate object-aligned irregular tokens. Their positions $BPos(P_{2})$ are parameterized into 2D Gaussians—center ( $P_{c}$ ), scale ( $P_{e}$ ), and orientation ( $\rho$ )—and explicitly injected via G-MRoPE before token concatenation.

Figure 3. Thermodiffusion-guided token merging. (a) Initial over-segmented regions act as graph nodes. (b) Heat diffuses via semantic consistency to merge correlated subgraphs. (c) Yields semantically complete, object-aligned regions.

Figure 4. (a) M-RoPE: Assigns discrete 2D integer coordinates to uniform grids, ignoring object boundaries. (b) G-MRoPE: Uses 2D Gaussian fitting to explicitly encode the continuous centers $(x,y)$ , scales $(w,h)$ , and orientations $\rho$ of irregular regions, enabling precise perception of complex geo-objects.

Figure 5. Visual comparison of semantic integrity across different tokenization methods. HeatTok preserves crisp object boundaries and avoids over-segmentation.

Visual Tokenizer	Tokens	Throughput	Category	Presence	Quantity	Color	Shape	Size	Position	Direction	Scene	Reasoning	All
Patch Embed ( Dosovitskiy et al. 2020 )	324	24.6	83.86	90.94	57.25	72.06	75.12	63.60	65.56	51.99	81.82	72.95	75.03
Dynamic Grouping and Merging
ToMe ( Bolya et al. 2022 )	164	25.43	82.13	91.92	50.39	72.14	75.26	63.60	60.57	50.52	83.45	73.17	73.24
Chat-UniVi ( Jin et al. 2024 )	112	27.99	85.38	91.50	55.15	74.79	75.83	63.49	67.57	53.02	83.61	74.15	75.87
Quadtree ( Ronen et al. 2023 )	144	29.97	56.12	88.01	50.30	61.52	76.77	60.87	54.77	49.01	63.26	72.07	64.16
HOOK ( Shao et al. 2024 )	128	35.33	70.69	79.33	60.65	74.85	73.12	58.73	59.60	51.17	79.10	68.41	69.95

Table 1. Performance comparison of different tokenization methods across various dimensions on VRSBench. Tokens denotes average number of visual tokens per image, and Throughput denotes number of images processed per second (images/s). Red indicates the best performance, while blue denotes the second best.

Visual Tokenizer	BasJu	RelJu	BasCo	RelCo	ObjAn	ComAn	All
Patch Embed ( Dosovitskiy et al. 2020 )	80.44	84.56	76.56	79.15	66.66	58.15	77.86
Dynamic Grouping and Merging
ToMe ( Bolya et al. 2022 )	73.64	80.21	76.88	78.85	52.5	54.39	73.67
Chat-UniVi ( Jin et al. 2024 )	82.74	80.83	73.68	77.65	62.09	55.51	75.72
Quadtree ( Ronen et al. 2023 )	60.23	78.81	72.06	78.43	53.81	49.40	68.82
HOOK ( Shao et al. 2024 )	81.96	82.00	54.19	56.87	43.42	48.87	69.62

Table 2. Performance comparison on the EarthVQA.

Remote Sensing Image & Tokenization Visualization:

Prompt: What are the water situations around the agricultural land? What are the needs for the renovation of villages?

Ground Truth: There are polluted waters near the agriculture land. The roads need to be improved , and waters need to be cleaned up .

Patch Embed: There are no water area. The roads need to be improved . Analysis: Uniform grid partitioning rigidly fragments the continuous polluted water body and agricultural land. This disruption of semantic integrity causes the model to fail entirely in recognizing the presence of the water body.

Dynamic Grouping and Merging

ToMe: There are clean waters near the agriculture land. Waters need to be cleaned up .

Table 3. Case study on remote sensing visual question answering. We show the visual question answering results of a specific scenario along with the tokenization visualizations for different methods. Red indicates hallucinations or incorrect perceptions; Green indicates accurate alignment with the ground truth.

Thermodiffusion	GMrope			Category	Presence	Quantity	Color	Shape	Size	Position	Direction	Scene	Reasoning	All
Thermodiffusion	P(c)	P(e)	$\rho$	Category	Presence	Quantity	Color	Shape	Size	Position	Direction	Scene	Reasoning	All
$\times$	$\times$	$\times$	$\times$	83.86	90.94	57.25	72.06	75.12	63.6	65.56	51.99	81.82	72.95	75.03
$\mathord{\text{✓}}$	$\times$	$\times$	$\times$	83.47	90.65	56.41	77.17	75.98	60.94	65.38	52.06	82.45	68.18	75.15
$\mathord{\text{✓}}$	$\mathord{\text{✓}}$	$\times$	$\times$	83.82	91.14	55.96	76.92	76.15	61.72	67.34	52.94	82.81	73.63	75.72
$\mathord{\text{✓}}$	$\mathord{\text{✓}}$	$\mathord{\text{✓}}$	$\times$	84.26	91.20	56.27	77.28	76.54	63.34	67.86	53.24	83.62	74.26	76.14
$\mathord{\text{✓}}$	$\mathord{\text{✓}}$	$\mathord{\text{✓}}$	$\mathord{\text{✓}}$	84.45	91.41	56.36	77.32	78.07	64.20	67.96	55.35	83.75	74.28	76.37

Table 4. Ablation on core components of HeatTok .

P^{(c)}

P^{(e)}

, and

\rho

denote the Gaussian center, scale, and orientation encodings.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Symbol	Value	Physical Interpretation & Impact
Base Diffusivity	$\kappa_{0}$	1.0	Controls the global baseline speed of heat exchange between all adjacent regions.
Color Sensitivity Scale	$\sigma_{C}$	5.0	Controls sensitivity to color differences during diffusion. A larger value makes diffusion less sensitive to color variations, allowing heat to flow across minor spectral differences.
Temperature Scale	$\sigma_{T}$	5.0	Controls color sensitivity for initial temperature assignment ( $T^{(0)}$ ). A smaller value requires high neighbor similarity to form a "Heat Source".
Complexity Factor	$\alpha$	0.5	Modulates the influence of texture complexity ( $C_{i}$ ). A larger value accelerates heat exchange in high-frequency/complex regions.
Merging Threshold	$\tau_{m}$	0.03	The threshold for temperature difference. A larger value leads to more aggressive merging and coarser tokens.

Appendix

Table 5. List of hyperparameters for the Thermodiffusion-Guided Token Merging algorithm, including their symbols, optimal values, and physical interpretations.

Experiment Set	$\kappa_{0}$	$\sigma_{C}$	$\sigma_{T}$	$\alpha$	$\tau_{m}$
	(Diffusivity)	(Color Sens.)	(Temp. Scale)	(Complexity)	(Threshold)
Optimal (Ours)	1.0	5.0	5.0	0.5	0.03
Set A	2.0	10.0	10.0	0.5	0.03
Set B	1.0	5.0	5.0	0.0	0.03
Set C	4.0	5.0	5.0	0.5	0.03

Appendix

Table 6. Hyperparameter settings for different experimental ablation sets.

Figure 6. Visual comparison of tokenization results under different hyperparameter configurations.

Proposal Generator	Frontend Latency (ms)	Frontend Params (M)	Category	Presence	Quantity	Color	Shape	Size	Position	Direction	Scene	Reasoning	All
FastSAM (w/o Thermo)	12.1	68.0	76.53	85.27	51.56	68.78	71.82	58.55	59.88	50.52	78.59	68.74	69.84
SLIC + Thermo	45.2	24.0	82.52	90.13	55.21	75.84	76.56	62.48	65.47	53.44	82.37	72.84	74.77
SAM-H + Thermo	1468.4	624.0	84.23	91.61	56.12	77.49	78.15	64.53	67.52	55.03	83.54	74.57	76.28
FastSAM + Thermo (Ours)	15.3	92.0	84.45	91.41	56.36	77.32	78.07	64.20	67.96	55.35	83.75	74.28	76.37

Appendix

Table 7. Ablation on region proposal dependency on VRSBench ( Li et al. 2024 ) . w/o Thermo indicates using raw proposals without our aggregation module. Frontend Params strictly includes the 24.0M parameters of the thermodiffusion module where applicable. All represents the sample-weighted average accuracy. Red indicates the best performance, while blue denotes the second best.

Proposal Generator	Frontend Latency (ms)	Frontend Params (M)	BasJu	RelJu	BasCo	RelCo	ObjAn	ComAn	All
FastSAM (w/o Thermo)	12.1	68.0	75.13	77.28	68.32	74.56	55.09	50.43	70.60
SLIC + Thermo	45.2	24.0	81.12	83.96	73.25	79.83	65.17	56.48	76.84
SAM-H + Thermo	1468.4	624.0	82.83	85.19	74.30	82.04	67.81	59.06	78.39
FastSAM + Thermo (Ours)	15.3	92.0	82.60	85.94	75.06	81.67	68.87	59.49	78.88

Appendix

Table 8. Ablation on region proposal dependency on EarthVQA ( Wang et al. 2024 ) , showing that gains stem from thermodiffusion rather than a specific frontend.

Figure 7. Visual grounding comparison between Patch Embed and HeatTok on Qwen2.5-VL, highlighting improved object-boundary alignment and more precise spatial localization.

Figure 8. Qualitative segmentation results on EarthVQA, showing more accurate boundaries and better preserved land-cover integrity.

Figure 9. Robustness analysis. HeatTok corrects over-segmentation by merging fragmented proposals into coherent regions (left), but performance degrades under severe haze (right).

Figure 10. Visual comparisons of tokenization results across different paradigms. Compared to baseline methods, HeatTok successfully preserves crisp object boundaries, maintains geometric continuity, and effectively avoids over-segmentation.

Remote Sensing Image & Tokenization Visualization:

Prompt: What are the land use types in this scene? What is the situation of barren land?

Ground Truth: There are park, and agricultural areas . There is no barren land .

Patch Embed: I can observe that there are park, and agricultural areas within the scene. However, it seems the barren land is scattered across multiple small patches. Analysis: The uniform grid rigidly cuts across the curved boundaries of the park and fields. This destruction of geometric continuity causes the model to misinterpret local texture variations (like winding paths or dry grass) as scattered barren land.

Dynamic Grouping and Merging

ToMe: I can identify industrial zones alongside agricultural areas . The barren land appears to be distributed unevenly.

Appendix

Table 9. Case study on remote sensing visual question answering. We show the visual question answering results of a specific scenario along with the tokenization visualizations for different methods. Red indicates hallucinations or incorrect perceptions; Green indicates accurate alignment with the ground truth.

Remote Sensing Image & Tokenization Visualization:

Prompt: What are the water situations around the agricultural land? What are the needs for the renovation of villages?

Ground Truth: There is clean water near the agriculture land. The roads need to be improved .

Patch Embed: Based on the scene, I can see dried-up water channels near the agriculture land. The roads need to be improved , and new residential buildings should be constructed . Analysis: The uniform grid partitioning rigidly bisects the meandering river and adjacent fields. This spatial fragmentation disrupts the clear water textures, causing the model to misinterpret the disjointed river segments as "dried-up channels" and hallucinate unrelated construction needs.

Dynamic Grouping and Merging

ToMe: There are polluted streams near the agriculture land. The roads need to be improved and residential areas expanded .

Appendix

Table 10. Case study on remote sensing visual question answering. We show the visual question answering results of a specific scenario along with the tokenization visualizations for different methods. Red indicates hallucinations or incorrect perceptions; Green indicates accurate alignment with the ground truth.

Visual Tokenization

Visual tokenization turns images and video into compact, structured tokens for efficient perception, compression, and generation.

See all 68 papers

FIRM: Fine-Grained Intra-Token Representation of Masks for Remote Sensing Reasoning Segmentation
Aug 14, 2026Weidong Tang, Kaiyu Li, Yikai Wang +4Remote Sensing Image Segmentation Referring Image Segmentation
Xi’an Jiaotong University, Xi’an, China · China Agricultural University, Beijing, China · Renmin University of China, Beijing, China +1
Semantic-Geometric Dual Compression: Training-Free Visual Token Reduction for Ultra-High-Resolution Remote Sensing Understanding
Apr 13, 2026Yueying Li, Fengxiang Wang, Yan Li +3Visual Token Compression Efficient VLM Inference
College of Computer Science and Technology, National University of Defense Technology, Changsha 410073, China · School of Computer Science, Beijing University of Posts and Telecommunications, Beijing 100876, China
GeoReason: Bridging Logical Reasoning and Spatial Fidelity in Remote Sensing Segmentation
Oct 16, 2026Cheng Fan, Han Wu, Tingzhang Luo +3Reasoning Segmentation Remote Sensing Image Segmentation
City University of Hong Kong · School of Computer Science, Peking University · Samsung

Aug 14, 2026J/K move · Enter open · S save

FIRM: Fine-Grained Intra-Token Representation of Masks for Remote Sensing Reasoning Segmentation

Weidong Tang, Kaiyu Li, Yikai Wang, Yanan Wu+3

Xi’an Jiaotong University, Xi’an, China · China Agricultural University, Beijing, China · Renmin University of China, Beijing, China+1

Remote Sensing Image Segmentation Referring Image Segmentation Boundary-Aware Segmentation

Open paper AskPDF arXiv

Reasoning segmentation requires multimodal large language models (MLLMs) to translate implicit instructions into precise pixel-level masks. MLLMs encode an image as visual tokens, each of which merges a group of image patches. In remote sensing images, small targets, thin structures, and adjacent instances can occupy different parts of the same visual token. Assigning a single binary mask label to such a token loses its internal spatial structure, causing nearby targets to merge and object boundaries to become coarse. To bridge this representational gap, we introduce FIRM, a Fine-grained Intra-token Representation of Masks. For each visual token, FIRM predicts a mask code that specifies an

r\times r

binary sub-cell pattern rather than a single foreground/background label. Given a target identified by the MLLM, the complete grid of mask codes is predicted in one mask pass. Fixed lookup converts the predicted codes into a discrete sub-cell mask, while marginalizing the code distribution yields a soft structural field. To further recover fine-grained boundaries within each sub-cell, we introduce a lightweight continuous renderer that refines this field using pre-merge visual features and image details. Across five reasoning and referring segmentation benchmarks on satellite and UAV images, FIRM achieves leading results, including

70.5/80.5

gIoU/cIoU on LaSeRS and a

3.0

-point average gain on EarthReason. These results demonstrate the value of explicitly representing intra-token mask patterns for fine-grained MLLM segmentation.

Abstract

Visual Tokenization

Explore similar work

FIRM: Fine-Grained Intra-Token Representation of Masks for Remote Sensing Reasoning Segmentation

Semantic-Geometric Dual Compression: Training-Free Visual Token Reduction for Ultra-High-Resolution Remote Sensing Understanding

GeoReason: Bridging Logical Reasoning and Spatial Fidelity in Remote Sensing Segmentation