Understanding Dynamic Scenes at Gigapixel Scale: Wide-Area Spatio-Temporal Perception from UAVs

Yuhang Zhu; Meiyi Zhu; Yunkai Dang; Zhangnan Li; Yuxuan Wang; Wenbin Li; Hongbing Pan

cs.CVSep 16, 2026

Understanding Dynamic Scenes at Gigapixel Scale: Wide-Area Spatio-Temporal Perception from UAVs

Authors: Yuhang Zhu, Meiyi Zhu, Yunkai Dang, Zhangnan Li, Yuxuan Wang, Wenbin Li, Hongbing Pan

Organizations: Nanjing University

Abstract

UAV-borne imaging has advanced from megapixel to gigapixel sensors, shifting aerial perception from recognizing individual targets to understanding entire dynamic scenes. We characterize this demand as Wide-area Spatio-temporal Scene Understanding (WSTU), which requires wide-area coverage, per-target resolution, and temporal continuity at once, a combination existing datasets lack. To fill this gap, we introduce an ultra-High-resolution (12768x9564) Airborne Remote-sensing Dataset (HARD) annotated at three levels for object detection, multi-object tracking, and scene-level visual question answering. Ultra-high-resolution imagery raises per-frame processing time to seconds. At that scale latency can no longer be ignored in evaluation. Thus, we propose a latency-aware metric for multi-object tracking called streaming-HOTA (s-HOTA). Extensive baseline experiments show how ultra-high-resolution processing reshapes each task. For detection, the end-to-end pipeline affects accuracy and speed as much as the detector itself does. For tracking, high latency charges the association axis far more unevenly than the detection axis, and association is where pipelines diverge. As a result, the pipeline that performs best offline can lose its lead under s-HOTA. For VQA, vision-language models remain weak at cross-frame identity binding and cannot transfer their single-frame gains to it. Together these findings show that the baselines we evaluate fall short of WSTU. HARD provides the data and the systematic baselines to advance it.

Figures & tables

Figure 1: The dataset landscape. Average frame resolution is plotted against acquisition platform. Marker shape indicates temporal continuity and colour indicates whether tracking or VQA annotation is provided. HARD is the only benchmark that combines a UAV platform, 122 MP frames, temporal continuity, multi-class tracking and scene-level VQA.

Figure 2: Overview of the HARD benchmark. Left: dataset construction, covering 122 MP data collection over 21 sequences, 50–355 m altitudes, and four flight modes, the three-level annotation hierarchy (detection, tracking, scene-level VQA). Middle: the benchmark and evaluation framework, in which detectors, trackers, and vision-language models are evaluated on the three-level tasks. Right: the definition and computation of the latency-aware streaming-HOTA metric.

Detector	Backbone	Params	Total	Infer	Pedestrian		Car		Bus		Truck		mAP
Detector	Backbone	Params	Total	Infer	$\mathrm{AP}_{50}$	$\mathrm{AP}_{50:95}$	$\mathrm{AP}_{50}$	$\mathrm{AP}_{50:95}$	$\mathrm{AP}_{50}$	$\mathrm{AP}_{50:95}$	$\mathrm{AP}_{50}$	$\mathrm{AP}_{50:95}$	$\mathrm{AP}_{50}$	$\mathrm{AP}_{50:95}$
Two-stage detectors
Faster R-CNN	ResNet-50	41.8M	14.39	10.97	0.287	0.101	0.913	0.406	0.222	0.110	0.780	0.466	0.551	0.271
Cascade R-CNN	ResNet-50	69.2M	16.89	13.47	0.299	0.098	0.910	0.423	0.524	0.260	0.866	0.440	0.650	0.305
One-stage detectors
RetinaNet	ResNet-50	37.7M	15.07	11.65	0.313	0.113	0.915	0.426	0.267	0.146	0.144	0.072	0.410	0.189

Table 1: Detection performance of baseline detectors on HARD. Both

\mathrm{AP}_{50}

and

\mathrm{AP}_{50:95}

are reported per category and as mean AP (mAP) over all four categories. Light-red, light-orange, and light-yellow cells indicate the first-, second-, and third-ranked results in each numeric column. For Params, Total, and Infer, lower is better; for all AP metrics, higher is better. “Total” and “Infer” are in seconds. “Infer” denotes net model forward-pass time; “Total” denotes end-to-end per-frame latency.

Panel of Figure 3 — Figure 3: Level-1 detection results: (a) $\mathrm{AP}_{50:95}$ ranking of the twenty detectors; (b) accuracy–latency trade-off with Pareto frontier; (c) per-class $\mathrm{AP}_{50:95}$ by architectural family. For each object class, the three adjacent bars correspond, from left to right, to two-stage, one-stage, and transformer detectors. Bar heights give the mean AP of all detectors within each family, while the overlaid dots give the individual detector scores. The dots are slightly offset horizontally only to avoid visual overlap; (d) effect of tile overlap on accuracy and inference time.

Overlap	Tiles	Obj./tile	Empty tiles	mAP	Infer
0 px	300	0.130	$\geq$ 87.0%	0.166	3.31 s
50 px	374	0.105	$\geq$ 89.5%	0.178	4.32 s
100 px	432	0.091	$\geq$ 90.9%	0.204	5.25 s
150 px	520	0.075	$\geq$ 92.5%	0.213	5.90 s

Table 2: Spatial sparsity of the HARD tiling pipeline under different overlaps, including tile density, empty-tile ratio, accuracy averaged over all twenty detectors, and inference time averaged over the eighteen detectors with directly measured per-overlap timings.

Panel of Figure 4 — Figure 4: Level-2 streaming-tracking results: (a) s-HOTA and streaming penalty $-\Delta$ of StrongSORT vs. ByteTrack across representative detectors; (b) s-HOTA against the load ratio $\rho=\Delta t/P$ over all 80 detector–tracker pipelines.

Detector	StrongSORT			ByteTrack			DeepSORT			Deep-OC-SORT
Detector	HOTA $\uparrow$	s-HOTA $\uparrow$	Lat. $\downarrow$	HOTA $\uparrow$	s-HOTA $\uparrow$	Lat. $\downarrow$	HOTA $\uparrow$	s-HOTA $\uparrow$	Lat. $\downarrow$	HOTA $\uparrow$	s-HOTA $\uparrow$	Lat. $\downarrow$
Two-stage detectors
Faster R-CNN	40.6	17.7 $\downarrow$ 22.9	15.85	37.1	19.4 $\downarrow$ 17.7	14.36	11.7	7.0 $\downarrow$ 4.7	21.79	29.5	13.3 $\downarrow$ 16.2	15.80
Cascade R-CNN	54.5	21.9 $\downarrow$ 32.6	18.22	42.7	21.5 $\downarrow$ 21.2	17.01	20.3	10.9 $\downarrow$ 9.4	22.10	36.8	15.3 $\downarrow$ 21.4	18.39
One-stage detectors
RetinaNet	52.3	21.7 $\downarrow$ 30.6	16.45	40.9	20.2 $\downarrow$ 20.7	15.10	18.2	10.8 $\downarrow$ 7.5	20.46	37.5	15.5 $\downarrow$ 22.0	16.46

Table 3: Streaming-HOTA on HARD (Car,

P=2

s), aggregated over all sequences using TrackEval combined . For each detector–tracker pipeline, we report offline HOTA, streaming s-HOTA

\uparrow

, and end-to-end latency Lat.(s)

\downarrow

. The red downward value beside each s-HOTA score denotes its degradation relative to the corresponding offline HOTA. Light-red, light-orange, and light-yellow boxes respectively indicate the first-, second-, and third-ranked results in each metric column.

Panel of Figure 5 — Figure 5: Level-3 scene-level VQA results: (a) overall accuracy ranking of the twelve vision-language models; (b) per-task profiles across the eight question types.

Explore similar work

Apr 23, 2026cs.CV

UHR-DETR: Efficient End-to-End Small Object Detection for Ultra-High-Resolution Remote Sensing Imagery

Ultra-High-Resolution (UHR) imagery has become essential for modern remote sensing, offering unprecedented spatial coverage. However, detecting small objects in such vast scenes presents a critical dilemma: retaining the original resolution for small objects causes prohibitive memory bottlenecks. Conversely, conventional compromises like image downsampling or patch cropping either erase small objects or destroy context. To break this dilemma, we propose UHR-DETR, an efficient end-to-end transformer-based detector designed for UHR imagery. First, we introduce a Coverage-Maximizing Sparse Encoder that dynamically allocates finite computational resources to informative high-resolution regions, ensuring maximum object coverage with minimal spatial redundancy. Second, we design a Global-Local Decoupled Decoder. By integrating macroscopic scene awareness with microscopic object details, this module resolves semantic ambiguities and prevents scene fragmentation. Extensive experiments on the UHR imagery datasets (e.g., STAR and SODA-A) demonstrate the superiority of UHR-DETR under strict hardware constraints (e.g., a single 24GB RTX 3090). It achieves a 2.8% mAP improvement while delivering a 10

\times

inference speedup compared to standard sliding-window baselines on the STAR dataset. Our codes and models will be available at https://github.com/Li-JingFang/UHR-DETR.

Jingfang Li, Haoran Zhu, Wen Yang +4

School of Electronic Information, Wuhan University, Wuhan, 430072 China

Feature Pyramid Detection Transformer Clocks

Ask PDF

May 14, 2026cs.CV

GeoVista: Visually Grounded Active Perception for Vision-Language Understanding of Ultra-High-Resolution Remote Sensing Images

Interpreting ultra-high-resolution (UHR) remote sensing images requires models to search for sparse and tiny visual evidence across large-scale scenes. Existing remote sensing vision-language models can inspect local regions with zooming and cropping tools, but most exploration strategies follow either a one-shot focus or a single sequential trajectory. Such single-path exploration can lose global context, leave scattered regions unvisited, and revisit or count the same evidence multiple times. To this end, we propose GeoVista, a planning-driven active perception framework for UHR remote sensing interpretation. Instead of committing to one zooming path, GeoVista first builds a global exploration plan, then verifies multiple candidate regions through branch-wise local inspection, while maintaining an explicit evidence state for cross-region aggregation and de-duplication. To enable this behavior, we introduce APE-GRO, a cold-start supervised trajectory corpus that reformulates diverse UHR tasks as Global-Region-Object interactive reasoning processes with a unified, scale-invariant spatial representation. We further design an Observe-Plan-Track mechanism for global observation, adaptive region inspection, and evidence tracking, and align the model with a GRPO-based strategy using step-wise rewards for planning, localization, and final answer correctness. Experiments on RSHR-Bench, XLRS-Bench, and LRS-VQA show that GeoVista achieves state-of-the-art performance. Code and dataset are available at https://github.com/ryan6073/GeoVista.

Jiashun Zhu, Ronghao Fu, Jiasen Hu +3

College of Computer Science and Technology, Jilin University, Changchun 130021, China · Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Changchun 130021, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100000, China +1

Remote Sensing Geospatial Reasoning Geogs-Slam High-Resolution Vision-Language Foundation Models

Ask PDF

May 12, 2026cs.CV

UHR-Micro: Diagnosing and Mitigating the Resolution Illusion in Earth Observation VLMs

Vision-Language Models (VLMs) are increasingly used to analyze ultra-high-resolution (UHR) Earth observation imagery, yet they face a severe scale mismatch between broad scene context and micro-scale targets. We refer to this phenomenon as a "resolution illusion": higher input resolution provides access to more visual detail, but does not necessarily translate into reliable perception of task-relevant micro-evidence. To benchmark this challenge, we introduce UHR-Micro, a benchmark comprising 11,072 instructions grounded in 1,212 UHR images, designed to evaluate VLMs on micro-scale evidence in native-resolution Earth observation imagery. UHR-Micro spans diverse target scales, task families, and visual conditions, with each sample having an objectively verifiable target. Experiments with representative high-resolution VLMs show substantial failures in localizing and interpreting task-relevant evidence, despite access to high-resolution inputs. Further analysis shows that model scaling alone is insufficient, while localized evidence substantially improves performance, pointing to evidence access as a major bottleneck. Motivated by this finding, we propose Micro-evidence Active Perception (MAP), which constructs a spatially grounded evidence state from localized observations and supplements it when the current evidence is insufficient. Across two backbone VLMs, MAP improves UHR-Micro performance by 8.65 percentage points on average. UHR-Micro and MAP provide a framework for diagnosing and improving high-resolution reasoning in Earth observation VLMs. Datasets and source code were released at https://github.com/MiliLab/UHR-Micro.

Shuo Ni, Tong Wang, Jing Zhang +5

National Key Laboratory of Science and Technology on Space-Born Intelligent Information Processing, Beijing Institute of Technology, Beijing, China · Zhongguancun Academy, Beijing, China · School of Computer Science, Wuhan University, Wuhan, China +2

Illusions Agent2Agent Clocks

Ask PDF