Organizations: School of GeoAI and Hindon STAI Institute, Key Laboratory of Geographic Information Science (Ministry of Education), East China Normal University, Shanghai, China · School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, China · Southeast University, China · Southwest Jiaotong University, China · Nanyang Technological University, Singapore · Wuhan University, China · the University of Waterloo, Cananda
During the deep learning era, innovations in remote sensing (RS) vision models primarily focused on optimizing network architectures for specific tasks and conducting end-to-end training. However, with the advent of the Vision-Language Models (VLMs) era, constrained by substantial computational resource demands, the mainstream research paradigm has shifted towards adapting general-purpose VLMs using RS vertical domain data for fine-tuning, rather than modifying their core architectures. This transition marks a change in the innovation pathway from model-centric to data-centric. Despite the increasing importance of data, the academic community still lacks a systematic, data-centric survey to guide researchers in constructing high-quality RS vision-language datasets for various application scenarios and in conducting effective model fine-tuning and performance evaluation based on these datasets. To address this gap, this paper presents a systematic review of the development status of RS datasets tailored for VLMs. It further provides a comprehensive practical guide for RS researchers in the large model era, covering the entire pipeline from data construction to fine-tuning and evaluation. This paper aims to encourage the academic community to build more diverse and higher-quality vision-language datasets for specialized RS sub-fields. Ultimately, the goal is to converge these efforts into a comprehensive RS vision-language corpus that offers extensive task coverage and a wide geographical distribution, thereby laying a solid data foundation for advancing towards general artificial intelligence in RS.
Figures & tables
Fig. 1: Representative Vision-Language Datasets in RS.
Fig. 2: Publication of RS Vision-Language Datasets
Fig. 3: Taxonomy of RS Vision-Language Data.
Fig. 5: Rapid Distinction of RS Captioning Tasks. The difficulty level increases with the rightward direction.
Fig. 6: Rapid Distinction of RS VQA Tasks. The difficulty level increases with the rightward direction.
Fig. 7: Rapid Distinction of RS VG Tasks. The three tasks on the left operate at the region level, while those on the right are at the pixel level, with difficulty increasing progressively from left to right on both sides.
Fig. 8: A Practical Guide to Data Choice, Curation, Fine-Tuning, and Evaluation. With GeoChef, researchers can master an iterative observe-refine cycle to train models that excel in practical RS scenarios.
Fig. 9: Our Open-source Project maintained on Github.
Fig. 10: A community-driven harmonica score library. Currently, 1,388 harmonica scores have been collected, 99% of which were contributed by community users.
Fig. 11: A mobile app developed based on our data warehouse. It can help researchers real-time grasp the dynamics of RS vision-language datasets.
Fig. 12: Distribution of Data Construction Methods Across the Entire and Different Types of Datasets.
Fig. 13: Distribution of Data Generation Models in Papers Using Semi-Automatic Data Construction.
Fig. 14: Multi-Agent Annotation Systems and Quality Control Paradigms.
Fig. 15: Frequency of Major RS Meta Datasets Used to Construct Vision-Language Datasets.
TABLE X: Comparison of mainstream RS MLLMs on 2 RS pixel-level VG benchmarks. Among them, EarthReason belongs to the dataset of reasoning segmentation.
School of Computer Science, Wuhan University · College of Resources and Environmental Sciences, Shanxi Agricultural University · School of Artificial Intelligence, Wuhan University +2
School of Artificial Intelligence and Robotics, Hunan University, Changsha, 410082, China · Intelligent Game and Decision Lab (IGDL), Beijing, 100091, China · cYuelushan Center for Industrial Innovation, Changsha, 410082, China