Knowledge-aware Visual Question Generation for Remote Sensing Images

Siran Li; Li Mi; Javiera Castillo-Navarro; Devis Tuia

cs.CVFeb 22, 2026

Knowledge-aware Visual Question Generation for Remote Sensing Images

Authors: Siran Li, Li Mi, Javiera Castillo-Navarro, Devis Tuia

Organizations: EPFL Switzerland

Abstract

With the rapid development of remote sensing image archives, asking questions about images has become an effective way of gathering specific information or performing image retrieval. However, automatically generated image-based questions tend to be simplistic and template-based, which hinders the real deployment of question answering or visual dialogue systems. To enrich and diversify the questions, we propose a knowledge-aware remote sensing visual question generation model, KRSVQG, that incorporates external knowledge related to the image content to improve the quality and contextual understanding of the generated questions. The model takes an image and a related knowledge triplet from external knowledge sources as inputs and leverages image captioning as an intermediary representation to enhance the image grounding of the generated questions. To assess the performance of KRSVQG, we utilized two datasets that we manually annotated: NWPU-300 and TextRS-300. Results on these two datasets demonstrate that KRSVQG outperforms existing methods and leads to knowledge-enriched questions, grounded in both image and domain knowledge.

Figures & tables

Figure 1: An example of image-based questions and a knowledge-aware question . In the questions, text from the image description is highlighted in green and text from external knowledge in orange.

Figure 2: The pipeline of our KRSVQG model comprises four components: an image encoder, a caption decoder, a text encoder, and a question decoder. With an image ( $I$ ) and a knowledge sentence ( $S$ ) as inputs, the model generates a knowledge-aware question ( $\widehat{Q}$ ) based on the caption ( $\widehat{C}$ ) and the knowledge sentence ( $S$ ). $\oplus$ means addition and normalization.

Figure 3: Example samples from the NWPU-300 and TextRS-300 datasets. In the questions, text from the caption description is highlighted in green, and text from external knowledge is highlighted in orange.

Dataset	Model	BLEU 1	BLEU 2	BLEU 3	BLEU 4	METEOR	ROUGE L	CIDEr
	IM-VQG [ 8 ]	32.46	15.91	7.93	0.00	8.96	31.95	0.17
NWPU- 300	AutoQG [ 18 ]	28.10	17.45	10.98	6.54	16.34	26.33	0.85
	KRSVQG	41.87	28.32	20.73	14.78	18.70	38.48	1.24
	IM-VQG [ 8 ]	40.51	23.02	14.16	9.79	19.41	40.85	0.46
TextRS-300	AutoQG [ 18 ]	32.89	24.53	19.01	14.42	22.54	32.35	1.47
	KRSVQG	44.26	33.78	27.63	22.90	19.64	42.90	1.47

Table 1: Question generation results on the NWPU-300 and TextRS-300 datasets.

Figure 4: Two generated samples from the KRSVQG model on the NWPU-300 dataset. The corresponding answer for $\widehat{Q}$ is marked in bold.

Remote Sensing

Remote sensing uses satellite, aerial, and ground-based observations to measure and map Earth and other planetary surfaces.

See all 528 papers

Questions beyond Pixels: Integrating Commonsense Knowledge in Visual Question Generation for Remote Sensing
Feb 22, 2026Siran Li, Li Mi, Javiera Castillo-Navarro +1Remote Sensing Remote Sensing Benchmarks
Ecole Polytechnique Fédérale de Lausanne (EPFL) · University of Tübingen · Conservatoire National des Arts et Métiers
TerraAgent: Demonstration-Grounded Multi-Agent Reasoning for Long-Context Remote Sensing Visual Question Answering
Oct 7, 2026Jingwen Huang, Yiming Fang, Hanxiao Wu +2Remote Sensing VQA Visual Question Answering
Yunnan University of Finance and Economics, Kunming, China
Benchmarking Answer-Grounding Consistency in Remote Sensing Visual Question Answering
Oct 9, 2026Ping Guo, Zhiqi Huang, Xinran Li +3Remote Sensing VQA VLM Evaluation
Graduate School of Information, Production and Systems, Waseda University, Kitakyushu, Japan · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China · School of Software, Dalian University of Technology, Dalian, China +1

Feb 22, 2026J/K move · Enter open · S save

Questions beyond Pixels: Integrating Commonsense Knowledge in Visual Question Generation for Remote Sensing

Siran Li, Li Mi, Javiera Castillo-Navarro, Devis Tuia

Ecole Polytechnique Fédérale de Lausanne (EPFL) · University of Tübingen · Conservatoire National des Arts et Métiers

Remote Sensing Remote Sensing Benchmarks

Open paper AskPDF arXiv

With the rapid development of remote sensing image archives, asking questions about images has become an effective way of gathering specific information or performing semantic image retrieval. However, current automatically generated questions tend to be simplistic and template-based, which hinders the deployment of question answering or visual dialogue systems for real-world applications. To enrich and diversify the questions with both image content and commonsense knowledge, we propose a Knowledge-aware Remote Sensing Visual Question Generation model (KRSVQG). The proposed model incorporates related knowledge triplets from external knowledge sources to broaden the question content, while employing image captioning as an intermediary representation to ground questions to the corresponding images. Moreover, KRSVQG utilizes a vision-language pre-training and fine-tuning strategy, enabling the model's adaptation to low data regimes. To evaluate the proposed KRSVQG model, we construct two knowledge-aware remote sensing visual question generation datasets: the NWPU-300 dataset and the TextRS-300 dataset. Evaluations, including metrics and human assessment, demonstrate that KRSVQG outperforms existing methods and leads to rich questions, grounded in both image and domain knowledge. As a key practice in vision-language research, knowledge-aware visual question generation advances the understanding of image content beyond pixels, facilitating the development of knowledge-enriched vision-language systems with vision-grounded human commonsense.

Abstract

Remote Sensing

Explore similar work

Questions beyond Pixels: Integrating Commonsense Knowledge in Visual Question Generation for Remote Sensing

TerraAgent: Demonstration-Grounded Multi-Agent Reasoning for Long-Context Remote Sensing Visual Question Answering

Benchmarking Answer-Grounding Consistency in Remote Sensing Visual Question Answering