Hierarchical Dual-Change Collaborative Learning for UAV Scene Change Captioning

Fuhai Chen; Pengpeng Huang; Junwen Wu; Hehong Zhang; Shiping Wang; Xiaoguang Ma; Xuri Ge

cs.CVMar 13, 2026

Hierarchical Dual-Change Collaborative Learning for UAV Scene Change Captioning

Authors: Fuhai Chen, Pengpeng Huang, Junwen Wu, Hehong Zhang, Shiping Wang, Xiaoguang Ma, Xuri Ge

Organizations: Fuzhou University, Fuzhou, China · Foshan Graduate School of Innovation, Northeastern University, Foshan, China · Shandong University, Jinan, China

Abstract

This paper proposes a novel task for UAV scene understanding - UAV Scene Change Captioning (UAV-SCC) - which aims to generate natural language descriptions of semantic changes in dynamic aerial imagery captured from a movable viewpoint. Unlike traditional change captioning that mainly describes differences between image pairs captured from a fixed camera viewpoint over time, UAV scene change captioning focuses on image-pair differences resulting from both temporal and spatial scene variations dynamically captured by a moving camera. The key challenge lies in understanding viewpoint-induced scene changes from UAV image pairs that share only partially overlapping scene content due to viewpoint shifts caused by camera rotation, while effectively exploiting the relative orientation between the two images. To this end, we propose a Hierarchical Dual-Change Collaborative Learning (HDC-CL) method for UAV scene change captioning. In particular, a novel transformer, i.e. Dynamic Adaptive Layout Transformer (DALT) is designed to adaptively model diverse spatial layouts of the image pair, where the interrelated features derived from the overlapping and non-overlapping regions are learned within the flexible and unified encoding layer. Furthermore, we propose a Hierarchical Cross-modal Orientation Consistency Calibration (HCM-OCC) method to enhance the model's sensitivity to viewpoint shift directions, enabling more accurate change captioning. To facilitate in-depth research on this task, we construct a new benchmark dataset, named UAV-SCC dataset, for UAV scene change captioning. Extensive experiments demonstrate that the proposed method achieves state-of-the-art performance on this task. The dataset and code will be publicly released upon acceptance of this paper.

Figures & tables

Figure 1: Comparison of three image captioning tasks. (a) Image Captioning generates semantic descriptions for a single image captured from a fixed camera viewpoint. (b) Change Captioning produces textual descriptions of differences between two images captured from the same viewpoint. (c) UAV Scene Change Captioning generates natural language descriptions of scene variations between image pairs captured from moving camera viewpoints. The gray boxes show example captions generated for each task.

Data	UAV-SCC Simple			UAV-SCC Rich
Data	Train	Val	Test	Train	Val	Test
Num. of image pairs	7213	901	903	5643	705	706
Avg. caption length	27.46	27.81	27.21	14.31	14.30	14.23

Table 1: Statistics of the UAV-SCC dataset. UAV-SCC Simple and UAV-SCC Rich share similar data organization but differ in annotation volume and linguistic diversity.

Figure 2: Overview of the HDC-CL framework. It consists of three steps: (1) image alignment for aligning the before–after image pair to correct spatial shifts and ensure consistent correspondence between regions, (2) scene change distillation , where GE, CE, and DE denote the global, common, and different encoders, and (3) caption generation based on distilled features.

Figure 3: An illustration of the Dynamic Adaptive Layout Transformer (DALT) pipeline. For clarity, this example uses $N=9$ patches per image.

Method	UAV-SCC Simple					UAV-SCC Rich
Method	B	M	R	C	S	B	M	R	C	S
DUDA Park et al. (2019)	7.36	17.69	24.39	–	21.70	4.17	8.89	18.32	–	14.19
SRDRLL Tu et al. (2021b)	8.28	18.78	24.27	–	25.48	3.64	5.29	15.19	–	10.26
SCORER+CBR Tu et al. (2023)	28.16	25.74	42.88	33.48	28.90	18.93	17.30	42.49	8.57	9.80
SMART Tu et al. (2024b)	25.08	24.55	40.78	30.13	26.44	16.37	17.66	40.35	8.93	10.16
DIRL+CCR Tu et al. (2024a)	24.99	23.81	40.20	30.43	26.61	16.88	15.98	42.13	8.77	11.41

Table 2: Comparison with existing methods on the UAV-SCC Simple and UAV-SCC Rich datasets. B: BLEU-4, M: METEOR, R: ROUGE-L, C: CIDEr, S: SPICE. GREEN indicates the best of all methods, and YELLOW indicates the second best.

$\mathcal{L}_{\text{global}}$	$\mathcal{L}_{\text{region}}$	$\mathcal{L}_{\text{HSIC}}$	UAV-SCC Rich					UAV-SCC Simple
$\mathcal{L}_{\text{global}}$	$\mathcal{L}_{\text{region}}$	$\mathcal{L}_{\text{HSIC}}$	B	M	R	C	S	B	M	R	C	S
$\times$	$\times$	$\times$	19.20	16.30	45.01	13.56	12.49	29.39	26.66	44.44	50.70	31.54
$\checkmark$	$\times$	$\times$	18.72	17.17	43.81	18.61	12.28	28.65	26.80	42.89	48.52	30.28
$\times$	$\checkmark$	$\times$	18.90	16.64	45.20	16.51	12.41	29.50	26.82	44.45	52.00	31.60
$\times$	$\times$	$\checkmark$	19.26	17.21	45.60	17.94	12.62	28.55	26.84	43.48	50.73	31.72
$\checkmark$	$\checkmark$	$\times$	18.63	17.44	43.87	18.92	12.58	29.75	27.14	43.88	50.29	32.34

Table 3: Joint ablation of loss components in HDC-CL on the UAV-SCC Rich and UAV-SCC Simple datasets. Bold numbers indicate the best performance in each column.

Method	UAV-SCC Simple					UAV-SCC Rich
Method	B	M	R	C	S	B	M	R	C	S
w/o HCM-OCC	29.62	26.84	43.33	51.47	31.82	18.86	16.99	44.03	18.66	12.42
HDC-CL	31.13 $\uparrow$ +1.51	27.34 $\uparrow$ +0.50	44.58 $\uparrow$ +1.25	54.68 $\uparrow$ +3.21	33.09 $\uparrow$ +1.27	19.26 $\uparrow$ +0.40	18.45 $\uparrow$ +1.46	44.32 $\uparrow$ +0.29	19.16 $\uparrow$ +0.50	13.00 $\uparrow$ +0.58
CARD	27.49	26.23	42.98	48.66	30.76	18.66	16.46	45.03	15.75	11.87
CARD+HCM-OCC	28.81 $\uparrow$ +1.32	26.89 $\uparrow$ +0.66	43.74 $\uparrow$ +0.76	47.26 $\downarrow$ -1.40	32.11 $\uparrow$ +1.35	21.48 $\uparrow$ +2.82	17.42 $\uparrow$ +0.96	47.15 $\uparrow$ +2.12	18.55 $\uparrow$ +2.80	12.62 $\uparrow$ +0.75
DIRL	24.99	24.81	40.20	30.43	26.61	16.88	15.98	42.13	8.77	11.41

Table 4: Ablation study of the proposed Hierarchical Cross-modal Orientation Consistency Calibration (HCM-OCC) module on the UAV-SCC datasets. The symbols

\uparrow

and

\downarrow

indicate performance improvements or drops compared with the corresponding baselines.

Mask strategy variants	UAV-SCC Simple					UAV-SCC Rich
Mask strategy variants	B	M	R	C	S	B	M	R	C	S
w/o Mask	25.60	25.56	41.27	46.10	29.97	18.22	17.09	43.54	18.66	12.31
Random Mask	28.00	26.48	43.31	52.12	31.26	18.41	17.21	42.95	18.33	12.08
Random Direction Window	29.68	26.89	43.80	52.20	31.95	18.70	17.41	42.85	18.51	12.74
No Direction Window	27.84	26.32	43.22	50.31	31.76	18.94	17.24	43.40	19.84	12.58
HDC-CL (ours)	31.13	27.34	44.58	54.68	33.09	19.26	18.45	44.32	19.16	13.00

Table 5: Ablation study of DALT under different mask generation strategies on the UAV-SCC datasets. Bold numbers indicate the best performance in each column.

Figure 4: Qualitative comparison of generated captions on the UAV-SCC Simple dataset. Text in different colors highlights the described objects or regions.

Figure 5: Performance comparison between GPT‑4o and our HDC-CL model on the UAV-SCC Simple dataset.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Statistical and linguistic analyses of the UAV-SCC dataset. (a) Word frequency distribution of captions in the UAV-SCC Simple dataset. (b) Word frequency distribution of captions in the UAV-SCC Rich dataset. (c) Distribution of noticeable and unnoticeable scene changes. (d) Heatmap of noun–verb co-occurrence frequencies, illustrating how object categories associate with specific change verbs. (e) Object-noun frequency distribution in the test-set captions generated by HDC-CL. (f) Object-noun frequency distribution in the test-set captions generated by CARD.

Object	Model	Pred-A	Pred-D	Pred-Desc	Corr-A	Corr-D	Co-Mentioned	Accuracy (%)
Car	CARD	1	5	24	1	0	8	30.0
Car	HDC-CL	6	4	14	2	0	6	33.3
Parking lot	CARD	0	5	56	0	0	25	41.0
Parking lot	HDC-CL	0	6	46	0	2	22	46.2
Road	CARD	19	13	3	1	1	3	14.3
Road	HDC-CL	23	12	17	4	1	8	25.0

Appendix

Table 6: Object-level prediction statistics for CARD and HDC-CL. “Pred-A” / “Pred-D” denote the number of images where the model predicts object appearance or disappearance; “Pred-Desc” denotes cases where the object is mentioned without a change verb. “Corr-A” / “Corr-D” count cases where appearance or disappearance predictions match the ground truth. “Co-Mentioned” counts images where both prediction and ground truth mention the object (regardless of change status). “Accuracy” is computed as: (Corr-A + Corr-D + Co-Mentioned) / (Pred-A + Pred-D + Pred-Desc).

Figure 7: Examples of image pairs and corresponding reference captions in the UAV-SCC dataset. (a) UAV-SCC Simple : each image pair shows clear and easily distinguishable scene changes, and the captions mainly provide straightforward descriptions with simple sentence structures. (b) UAV-SCC Rich : the scenes exhibit more complex spatial layouts and finer-grained variations, while the captions include richer linguistic details, describing diverse objects, attributes, and spatial relationships.

Period	Estimated Latency	Data Size
Image Transmission	$\approx$ 0.6 – 2 seconds	$\approx$ 10 MB
Text Transmission	$\approx$ 5 – 10 ms	$<$ 1 KB
Text Inference	$\approx$ 77 ms	—
Text Inference + Transmission	$\approx$ 82 – 87 ms	$<$ 1 KB

Appendix

Table 7: Comparison of image vs. text processing in terms of latency and data size (based on Matrice 4D specifications).

$\lambda$	UAV-SCC Simple					UAV-SCC Rich
$\lambda$	B	M	R	C	S	B	M	R	C	S
0	29.26	27.18	44.33	49.55	32.56	19.46	16.51	45.29	13.86	12.60
0.01	31.13	27.34	44.58	54.68	33.09	19.26	18.45	44.32	19.16	13.00
0.02	29.01	26.69	43.02	46.80	31.45	18.92	17.65	43.85	18.27	13.02
0.03	29.97	27.28	44.16	53.81	33.13	18.88	17.99	44.59	18.78	13.09
0.04	29.02	27.35	43.86	48.45	32.89	18.09	16.81	43.88	18.09	12.20

Appendix

Table 8: Effects of

\lambda

on UAV-SCC Simple and UAV-SCC Rich .

Figure 8: Quantitative analysis of the impact of the hyperparameter $\lambda$ on model performance. (a) UAV-SCC Simple and (b) UAV-SCC Rich show the variation of different evaluation metrics (BLEU4, METEOR, ROUGE-L, and CIDEr) with respect to $\lambda$ . A moderate value of $\lambda$ (around 0.01) generally yields better overall performance, while excessively large values lead to minor fluctuations or degradation, indicating the need for a balanced weighting among loss components.

Figure 9: Qualitative comparison of attention maps from CARD and our HDC-CL. The yellow box marks the approximate common region between the image pair; it is manually added for visualization and is not ground-truth annotation. HDC-CL yields more accurate and focused change responses than CARD.

Figure 10: Qualitative captioning examples on (a)UAV-SCC Simple and (b)UAV-SCC Rich . For each image pair, we show the ground-truth caption (GT), the output from CARD, and the caption generated by our proposed HDC-CL. HDC-CL produces change descriptions that are more consistent with the visual evidence and spatial relations in the before/after images, while CARD often yields incorrect object references or misaligned spatial descriptions.

$\mathcal{L}_{\text{global}}$	$\mathcal{L}_{\text{region}}$	$\mathcal{L}_{\text{HSIC}}$	UAV-SCC Rich					UAV-SCC Simple
$\mathcal{L}_{\text{global}}$	$\mathcal{L}_{\text{region}}$	$\mathcal{L}_{\text{HSIC}}$	B	M	R	C	S	B	M	R	C	S
$\times$	$\times$	$\times$	19.46	16.51	45.29	13.86	12.60	29.26	27.18	44.33	49.55	32.56
$\checkmark$	$\times$	$\times$	18.48	17.29	43.32	18.25	12.74	28.59	26.69	42.89	51.67	32.16
$\times$	$\checkmark$	$\times$	18.42	17.36	43.71	18.49	12.59	29.92	26.86	44.19	50.55	31.85
$\times$	$\times$	$\checkmark$	19.62	16.71	45.81	16.74	12.64	29.87	26.66	44.52	49.00	31.69
$\checkmark$	$\checkmark$	$\times$	19.67	17.68	44.51	17.63	12.80	28.76	26.75	43.78	51.24	32.10

Appendix

Table 9: Joint ablation of context distillation losses on UAV-SCC Rich and UAV-SCC Simple without the alignment loss

\mathcal{L}_{\text{align}}

. Bold numbers indicate the best performance in each column.

Explore similar work

May 6, 2026

UAV as Urban Construction Change Monitor: A New Benchmark and Change Captioning Model

Remote Sensing Image Change Captioning (RSICC) aims to generate spatially grounded natural language descriptions of scene evolution from bi-temporal imagery, moving beyond binary change masks toward semantic-level understanding. However, existing methods rely on implicit feature differencing without explicitly modeling structured change semantics, and struggle to reconcile the conflicting representation demands of change detection and caption generation. In addition, current benchmarks provide limited coverage of high-resolution urban construction scenarios. To address these challenges, we propose PTNet, a prototype-guided task-adaptive framework for joint change captioning and detection. PTNet explicitly models structured change semantics through a learnable prototype bank that guides cross-temporal interaction, disentangles task-specific representations via multi-head gating, and injects detection-derived spatial priors into caption generation, enabling coherent semantic correspondence while preserving fine-grained spatial sensitivity. Furthermore, we construct UCCD, a large-scale UAV-based benchmark comprising 9,000 high-resolution image pairs and 45,000 annotated sentences for urban construction monitoring. Extensive experiments on UCCD and WHU-CDC demonstrate that PTNet consistently outperforms existing methods. The dataset and source code are publicly available at https://github.com/G124556/ptnet.

Yupeng Gao, Tianyu Li, Guoqing Wang +1

University of Electronic Science and Technology of China, Chengdu, China

Remote Sensing Image Understanding Remote Sensing Image Change Captioning Remote Sensing Change Detection Semantic Change Detection UAV Remote Sensing

Ask PDF

May 25, 2026

UAV-OVO: Out-of-Viewpoint Generalization in UAV Action Recognition

UAV action recognition faces a deployment shift that standard benchmarks often obscure: a model trained on UAV footage captured from low-depression viewpoints may be required to recognize the same action classes from high-depression viewpoints. While the action labels remain unchanged, this shift alters body visibility, motion projection, and scene context, encouraging models to rely on viewpoint-specific shortcuts. We introduce UAV-OVO, an Out-of-Viewpoint generalization benchmark for UAV action recognition. UAV-OVO derives view scores from uncalibrated videos, uses a view-isolation band to assign low-depression videos to the training and in-distribution test splits while reserving high-depression videos for out-of-distribution testing, and constructs ID/OOD test sets matched by class distribution so that performance differences reflect viewpoint shift rather than label imbalance. Across representative video recognizers, UAV-OVO reveals a substantial ID/OOD gap: models that fit the low-depression training distribution well often fail to transfer to held-out high-depression views, exposing viewpoint shortcuts hidden by aggregate accuracy. We further propose LATER, LoRA-Anchored Test-time Re-centering, which first adapts the recognizer with Low-Rank Adaptation (LoRA) and then uses the learned LoRA subspace as a semantic anchor for online feature re-centering. Specifically, LATER projects target-domain displacement onto the orthogonal complement of the LoRA subspace before re-centering features, reducing viewpoint-induced drift while preserving task-relevant semantics. Together, UAV-OVO and LATER provide a controlled testbed and a practical adaptation method for viewpoint-robust UAV video understanding.

Yu Xia, Zhengbo Zhang, Shuaihu Zhang +1

Wuhan University · Singapore University of Technology and Design

Test-Time Adaptation OOD Generalization Video Action Recognition

Ask PDF

Mar 6, 2026

Imagine How To Change: Explicit Procedure Modeling for Change Captioning

Change captioning generates descriptions that explicitly describe the differences between two visually similar images. Existing methods operate on static image pairs, thus ignoring the rich temporal dynamics of the change procedure, which is the key to understand not only what has changed but also how it occurs. We introduce ProCap, a novel framework that reformulates change modeling from static image comparison to dynamic procedure modeling. ProCap features a two-stage design: The first stage trains a procedure encoder to learn the change procedure from a sparse set of keyframes. These keyframes are obtained by automatically generating intermediate frames to make the implicit procedural dynamics explicit and then sampling them to mitigate redundancy. Then the encoder learns to capture the latent dynamics of these keyframes via a caption-conditioned, masked reconstruction task. The second stage integrates this trained encoder within an encoder-decoder model for captioning. Instead of relying on explicit frames from the previous stage -- a process incurring computational overhead and sensitivity to visual noise -- we introduce learnable procedure queries to prompt the encoder for inferring the latent procedure representation, which the decoder then translates into text. The entire model is then trained end-to-end with a captioning loss, ensuring the encoder's output is both temporally coherent and captioning-aligned. Experiments on three datasets demonstrate the effectiveness of ProCap. Code and pre-trained models are available at https://github.com/BlueberryOreo/ProCap

Jiayang Sun, Zixin Guo, Min Cao +2

School of Computer Science and Technology, Soochow University, Jiangsu, China · Department of Computer Science, Aalto University, Espoo, Finland · Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences, Beijing, China +2

Topic pending

Ask PDF