Hierarchical Dual-Change Collaborative Learning for UAV Scene Change Captioning

Fuhai Chen; Pengpeng Huang; Junwen Wu; Hehong Zhang; Shiping Wang; Xiaoguang Ma; Xuri Ge

cs.CVMar 13, 2026

Hierarchical Dual-Change Collaborative Learning for UAV Scene Change Captioning

Authors: Fuhai Chen, Pengpeng Huang, Junwen Wu, Hehong Zhang, Shiping Wang, Xiaoguang Ma, Xuri Ge

Organizations: Fuzhou University, Fuzhou, China · Foshan Graduate School of Innovation, Northeastern University, Foshan, China · Shandong University, Jinan, China

Abstract

This paper proposes a novel task for UAV scene understanding - UAV Scene Change Captioning (UAV-SCC) - which aims to generate natural language descriptions of semantic changes in dynamic aerial imagery captured from a movable viewpoint. Unlike traditional change captioning that mainly describes differences between image pairs captured from a fixed camera viewpoint over time, UAV scene change captioning focuses on image-pair differences resulting from both temporal and spatial scene variations dynamically captured by a moving camera. The key challenge lies in understanding viewpoint-induced scene changes from UAV image pairs that share only partially overlapping scene content due to viewpoint shifts caused by camera rotation, while effectively exploiting the relative orientation between the two images. To this end, we propose a Hierarchical Dual-Change Collaborative Learning (HDC-CL) method for UAV scene change captioning. In particular, a novel transformer, i.e. Dynamic Adaptive Layout Transformer (DALT) is designed to adaptively model diverse spatial layouts of the image pair, where the interrelated features derived from the overlapping and non-overlapping regions are learned within the flexible and unified encoding layer. Furthermore, we propose a Hierarchical Cross-modal Orientation Consistency Calibration (HCM-OCC) method to enhance the model's sensitivity to viewpoint shift directions, enabling more accurate change captioning. To facilitate in-depth research on this task, we construct a new benchmark dataset, named UAV-SCC dataset, for UAV scene change captioning. Extensive experiments demonstrate that the proposed method achieves state-of-the-art performance on this task. The dataset and code will be publicly released upon acceptance of this paper.

Figures & tables

Figure 1: Comparison of three image captioning tasks. (a) Image Captioning generates semantic descriptions for a single image captured from a fixed camera viewpoint. (b) Change Captioning produces textual descriptions of differences between two images captured from the same viewpoint. (c) UAV Scene Change Captioning generates natural language descriptions of scene variations between image pairs captured from moving camera viewpoints. The gray boxes show example captions generated for each task.

Data	UAV-SCC Simple			UAV-SCC Rich
Data	Train	Val	Test	Train	Val	Test
Num. of image pairs	7213	901	903	5643	705	706
Avg. caption length	27.46	27.81	27.21	14.31	14.30	14.23

Table 1: Statistics of the UAV-SCC dataset. UAV-SCC Simple and UAV-SCC Rich share similar data organization but differ in annotation volume and linguistic diversity.

Figure 2: Overview of the HDC-CL framework. It consists of three steps: (1) image alignment for aligning the before–after image pair to correct spatial shifts and ensure consistent correspondence between regions, (2) scene change distillation , where GE, CE, and DE denote the global, common, and different encoders, and (3) caption generation based on distilled features.

Figure 3: An illustration of the Dynamic Adaptive Layout Transformer (DALT) pipeline. For clarity, this example uses $N=9$ patches per image.

Method	UAV-SCC Simple					UAV-SCC Rich
Method	B	M	R	C	S	B	M	R	C	S
DUDA Park et al. (2019)	7.36	17.69	24.39	–	21.70	4.17	8.89	18.32	–	14.19
SRDRLL Tu et al. (2021b)	8.28	18.78	24.27	–	25.48	3.64	5.29	15.19	–	10.26
SCORER+CBR Tu et al. (2023)	28.16	25.74	42.88	33.48	28.90	18.93	17.30	42.49	8.57	9.80
SMART Tu et al. (2024b)	25.08	24.55	40.78	30.13	26.44	16.37	17.66	40.35	8.93	10.16
DIRL+CCR Tu et al. (2024a)	24.99	23.81	40.20	30.43	26.61	16.88	15.98	42.13	8.77	11.41

Table 2: Comparison with existing methods on the UAV-SCC Simple and UAV-SCC Rich datasets. B: BLEU-4, M: METEOR, R: ROUGE-L, C: CIDEr, S: SPICE. GREEN indicates the best of all methods, and YELLOW indicates the second best.

$\mathcal{L}_{\text{global}}$	$\mathcal{L}_{\text{region}}$	$\mathcal{L}_{\text{HSIC}}$	UAV-SCC Rich					UAV-SCC Simple
$\mathcal{L}_{\text{global}}$	$\mathcal{L}_{\text{region}}$	$\mathcal{L}_{\text{HSIC}}$	B	M	R	C	S	B	M	R	C	S
$\times$	$\times$	$\times$	19.20	16.30	45.01	13.56	12.49	29.39	26.66	44.44	50.70	31.54
$\checkmark$	$\times$	$\times$	18.72	17.17	43.81	18.61	12.28	28.65	26.80	42.89	48.52	30.28
$\times$	$\checkmark$	$\times$	18.90	16.64	45.20	16.51	12.41	29.50	26.82	44.45	52.00	31.60
$\times$	$\times$	$\checkmark$	19.26	17.21	45.60	17.94	12.62	28.55	26.84	43.48	50.73	31.72
$\checkmark$	$\checkmark$	$\times$	18.63	17.44	43.87	18.92	12.58	29.75	27.14	43.88	50.29	32.34

Table 3: Joint ablation of loss components in HDC-CL on the UAV-SCC Rich and UAV-SCC Simple datasets. Bold numbers indicate the best performance in each column.

Method	UAV-SCC Simple					UAV-SCC Rich
Method	B	M	R	C	S	B	M	R	C	S
w/o HCM-OCC	29.62	26.84	43.33	51.47	31.82	18.86	16.99	44.03	18.66	12.42
HDC-CL	31.13 $\uparrow$ +1.51	27.34 $\uparrow$ +0.50	44.58 $\uparrow$ +1.25	54.68 $\uparrow$ +3.21	33.09 $\uparrow$ +1.27	19.26 $\uparrow$ +0.40	18.45 $\uparrow$ +1.46	44.32 $\uparrow$ +0.29	19.16 $\uparrow$ +0.50	13.00 $\uparrow$ +0.58
CARD	27.49	26.23	42.98	48.66	30.76	18.66	16.46	45.03	15.75	11.87
CARD+HCM-OCC	28.81 $\uparrow$ +1.32	26.89 $\uparrow$ +0.66	43.74 $\uparrow$ +0.76	47.26 $\downarrow$ -1.40	32.11 $\uparrow$ +1.35	21.48 $\uparrow$ +2.82	17.42 $\uparrow$ +0.96	47.15 $\uparrow$ +2.12	18.55 $\uparrow$ +2.80	12.62 $\uparrow$ +0.75
DIRL	24.99	24.81	40.20	30.43	26.61	16.88	15.98	42.13	8.77	11.41

Table 4: Ablation study of the proposed Hierarchical Cross-modal Orientation Consistency Calibration (HCM-OCC) module on the UAV-SCC datasets. The symbols

\uparrow

and

\downarrow

indicate performance improvements or drops compared with the corresponding baselines.

Mask strategy variants	UAV-SCC Simple					UAV-SCC Rich
Mask strategy variants	B	M	R	C	S	B	M	R	C	S
w/o Mask	25.60	25.56	41.27	46.10	29.97	18.22	17.09	43.54	18.66	12.31
Random Mask	28.00	26.48	43.31	52.12	31.26	18.41	17.21	42.95	18.33	12.08
Random Direction Window	29.68	26.89	43.80	52.20	31.95	18.70	17.41	42.85	18.51	12.74
No Direction Window	27.84	26.32	43.22	50.31	31.76	18.94	17.24	43.40	19.84	12.58
HDC-CL (ours)	31.13	27.34	44.58	54.68	33.09	19.26	18.45	44.32	19.16	13.00

Table 5: Ablation study of DALT under different mask generation strategies on the UAV-SCC datasets. Bold numbers indicate the best performance in each column.

Figure 4: Qualitative comparison of generated captions on the UAV-SCC Simple dataset. Text in different colors highlights the described objects or regions.

Figure 5: Performance comparison between GPT‑4o and our HDC-CL model on the UAV-SCC Simple dataset.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Statistical and linguistic analyses of the UAV-SCC dataset. (a) Word frequency distribution of captions in the UAV-SCC Simple dataset. (b) Word frequency distribution of captions in the UAV-SCC Rich dataset. (c) Distribution of noticeable and unnoticeable scene changes. (d) Heatmap of noun–verb co-occurrence frequencies, illustrating how object categories associate with specific change verbs. (e) Object-noun frequency distribution in the test-set captions generated by HDC-CL. (f) Object-noun frequency distribution in the test-set captions generated by CARD.

Object	Model	Pred-A	Pred-D	Pred-Desc	Corr-A	Corr-D	Co-Mentioned	Accuracy (%)
Car	CARD	1	5	24	1	0	8	30.0
Car	HDC-CL	6	4	14	2	0	6	33.3
Parking lot	CARD	0	5	56	0	0	25	41.0
Parking lot	HDC-CL	0	6	46	0	2	22	46.2
Road	CARD	19	13	3	1	1	3	14.3
Road	HDC-CL	23	12	17	4	1	8	25.0

Appendix

Table 6: Object-level prediction statistics for CARD and HDC-CL. “Pred-A” / “Pred-D” denote the number of images where the model predicts object appearance or disappearance; “Pred-Desc” denotes cases where the object is mentioned without a change verb. “Corr-A” / “Corr-D” count cases where appearance or disappearance predictions match the ground truth. “Co-Mentioned” counts images where both prediction and ground truth mention the object (regardless of change status). “Accuracy” is computed as: (Corr-A + Corr-D + Co-Mentioned) / (Pred-A + Pred-D + Pred-Desc).

Figure 7: Examples of image pairs and corresponding reference captions in the UAV-SCC dataset. (a) UAV-SCC Simple : each image pair shows clear and easily distinguishable scene changes, and the captions mainly provide straightforward descriptions with simple sentence structures. (b) UAV-SCC Rich : the scenes exhibit more complex spatial layouts and finer-grained variations, while the captions include richer linguistic details, describing diverse objects, attributes, and spatial relationships.

Period	Estimated Latency	Data Size
Image Transmission	$\approx$ 0.6 – 2 seconds	$\approx$ 10 MB
Text Transmission	$\approx$ 5 – 10 ms	$<$ 1 KB
Text Inference	$\approx$ 77 ms	—
Text Inference + Transmission	$\approx$ 82 – 87 ms	$<$ 1 KB

Appendix

Table 7: Comparison of image vs. text processing in terms of latency and data size (based on Matrice 4D specifications).

$\lambda$	UAV-SCC Simple					UAV-SCC Rich
$\lambda$	B	M	R	C	S	B	M	R	C	S
0	29.26	27.18	44.33	49.55	32.56	19.46	16.51	45.29	13.86	12.60
0.01	31.13	27.34	44.58	54.68	33.09	19.26	18.45	44.32	19.16	13.00
0.02	29.01	26.69	43.02	46.80	31.45	18.92	17.65	43.85	18.27	13.02
0.03	29.97	27.28	44.16	53.81	33.13	18.88	17.99	44.59	18.78	13.09
0.04	29.02	27.35	43.86	48.45	32.89	18.09	16.81	43.88	18.09	12.20

Appendix

Table 8: Effects of

\lambda

on UAV-SCC Simple and UAV-SCC Rich .

Figure 8: Quantitative analysis of the impact of the hyperparameter $\lambda$ on model performance. (a) UAV-SCC Simple and (b) UAV-SCC Rich show the variation of different evaluation metrics (BLEU4, METEOR, ROUGE-L, and CIDEr) with respect to $\lambda$ . A moderate value of $\lambda$ (around 0.01) generally yields better overall performance, while excessively large values lead to minor fluctuations or degradation, indicating the need for a balanced weighting among loss components.

Figure 9: Qualitative comparison of attention maps from CARD and our HDC-CL. The yellow box marks the approximate common region between the image pair; it is manually added for visualization and is not ground-truth annotation. HDC-CL yields more accurate and focused change responses than CARD.

Figure 10: Qualitative captioning examples on (a)UAV-SCC Simple and (b)UAV-SCC Rich . For each image pair, we show the ground-truth caption (GT), the output from CARD, and the caption generated by our proposed HDC-CL. HDC-CL produces change descriptions that are more consistent with the visual evidence and spatial relations in the before/after images, while CARD often yields incorrect object references or misaligned spatial descriptions.

$\mathcal{L}_{\text{global}}$	$\mathcal{L}_{\text{region}}$	$\mathcal{L}_{\text{HSIC}}$	UAV-SCC Rich					UAV-SCC Simple
$\mathcal{L}_{\text{global}}$	$\mathcal{L}_{\text{region}}$	$\mathcal{L}_{\text{HSIC}}$	B	M	R	C	S	B	M	R	C	S
$\times$	$\times$	$\times$	19.46	16.51	45.29	13.86	12.60	29.26	27.18	44.33	49.55	32.56
$\checkmark$	$\times$	$\times$	18.48	17.29	43.32	18.25	12.74	28.59	26.69	42.89	51.67	32.16
$\times$	$\checkmark$	$\times$	18.42	17.36	43.71	18.49	12.59	29.92	26.86	44.19	50.55	31.85
$\times$	$\times$	$\checkmark$	19.62	16.71	45.81	16.74	12.64	29.87	26.66	44.52	49.00	31.69
$\checkmark$	$\checkmark$	$\times$	19.67	17.68	44.51	17.63	12.80	28.76	26.75	43.78	51.24	32.10

Appendix

Table 9: Joint ablation of context distillation losses on UAV-SCC Rich and UAV-SCC Simple without the alignment loss

\mathcal{L}_{\text{align}}

. Bold numbers indicate the best performance in each column.

Explore similar work

UAV as Urban Construction Change Monitor: A New Benchmark and Change Captioning Model
May 6, 2026Yupeng Gao, Tianyu Li, Guoqing Wang +1Remote Sensing Image Understanding Remote Sensing Image Change Captioning
University of Electronic Science and Technology of China, Chengdu, China
UAV-OVO: Out-of-Viewpoint Generalization in UAV Action Recognition
May 25, 2026Yu Xia, Zhengbo Zhang, Shuaihu Zhang +1Test-Time Adaptation OOD Generalization
Wuhan University · Singapore University of Technology and Design
Imagine How To Change: Explicit Procedure Modeling for Change Captioning
Mar 6, 2026Jiayang Sun, Zixin Guo, Min Cao +2
School of Computer Science and Technology, Soochow University, Jiangsu, China · Department of Computer Science, Aalto University, Espoo, Finland · Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences, Beijing, China +2

May 6, 2026J/K move · Enter open · S save

UAV as Urban Construction Change Monitor: A New Benchmark and Change Captioning Model

Yupeng Gao, Tianyu Li, Guoqing Wang, Yang Yang

University of Electronic Science and Technology of China, Chengdu, China

Remote Sensing Image Understanding Remote Sensing Image Change Captioning Remote Sensing Change Detection Semantic Change Detection UAV Remote Sensing

Open paper AskPDF arXiv

Remote Sensing Image Change Captioning (RSICC) aims to generate spatially grounded natural language descriptions of scene evolution from bi-temporal imagery, moving beyond binary change masks toward semantic-level understanding. However, existing methods rely on implicit feature differencing without explicitly modeling structured change semantics, and struggle to reconcile the conflicting representation demands of change detection and caption generation. In addition, current benchmarks provide limited coverage of high-resolution urban construction scenarios. To address these challenges, we propose PTNet, a prototype-guided task-adaptive framework for joint change captioning and detection. PTNet explicitly models structured change semantics through a learnable prototype bank that guides cross-temporal interaction, disentangles task-specific representations via multi-head gating, and injects detection-derived spatial priors into caption generation, enabling coherent semantic correspondence while preserving fine-grained spatial sensitivity. Furthermore, we construct UCCD, a large-scale UAV-based benchmark comprising 9,000 high-resolution image pairs and 45,000 annotated sentences for urban construction monitoring. Extensive experiments on UCCD and WHU-CDC demonstrate that PTNet consistently outperforms existing methods. The dataset and source code are publicly available at https://github.com/G124556/ptnet.

Abstract

Explore similar work

UAV as Urban Construction Change Monitor: A New Benchmark and Change Captioning Model

UAV-OVO: Out-of-Viewpoint Generalization in UAV Action Recognition

Imagine How To Change: Explicit Procedure Modeling for Change Captioning