RSLiveBench-UAV: A Continually Evolving Vision-Language Benchmark for Aerial Spatial Intelligence
Organizations: East China Normal University · Shanghai Jiao Tong University · University of California, Los Angeles · University of Waterloo ·
Abstract
Aerial spatial intelligence (ASI) requires models to understand objects and spatial relationships across changing viewpoints and scales. Evaluating these abilities as models evolve calls for fresh physical observations, yet producing new aerial spatial questions requires repeatedly establishing entity identities, geometric references, and cross-view correspondences. We introduce RSLiveBench-UAV, a continually evolving vision-language benchmark driven by recurring acquisition of real UAV imagery. Its update protocol admits images captured after the preceding release and delays answer disclosure. An entity-centered production workflow uses geometry-assisted association to link repeated observations of physical vehicles, reuse human-verified attributes, and combine them with view-specific annotations and spatial references for traceable question generation. All tasks use deterministic scoring without model judges. We evaluate 30 general-purpose, remote-sensing, and embodied models on 35,491 question records from ten scenes, covering 17 tasks across 2D perception, 3D perception, cross-view perception, and reasoning. All 18 tested embodied models outperform the three selected remote-sensing comparators in Overall, although the strongest embodied model scores 43.29 versus 59.88 for GPT-6-Astra. Fine-grained vehicle recognition remains challenging, and performance varies across heights and camera pitches. Retrospective scene-group comparisons and rolling-window replay characterize score-pattern consistency under scene renewal. RSLiveBench-UAV connects fresh physical observations, entity-centered answer production, and systematic model evaluation to support continual assessment of ASI. Benchmark will release at https://rslivebench.space
Figures & tables
| Year | Month | Scenes | Images | Total (h) | min/scene | s/image |
| 2023 | 9 | 11 | 623 | 24.99 | 136.31 | 144.40 |
| 2026 | 7 | 7 | 1,195 | 5.17 | 44.33 | 15.58 |
| 2026 | 8 | 26 | 3,529 | 15.08 | 34.80 | 15.38 |
| 2026 | 9 | 52 | 6,815 | 25.58 | 29.52 | 13.51 |
| Model | Overall | 2D | 3D | Cross-view | Reasoning |
| Remote sensing MLLMs | |||||
| VHM [ 19 ] | 4.38 | 9.83 | 0.33 | 6.48 | 0.89 |
| GeoChat-7B [ 10 ] | 8.31 | 8.63 | 7.15 | 7.64 | 9.82 |
| Embodied MLLMs | |||||
| SenseNova-SI-1.4-InternVL3-8B [ 3 ] | 18.95 | 22.16 | 12.01 | 23.36 | 18.30 |
| Cosmos-Reason2-8B [ 17 ] | 20.80 | 27.02 | 17.42 | 13.71 | 25.05 |
| Comparison setting | Correlation | ||
| Grouping | Pairs | Pearson | Spearman |
| 45 | 0.7449 | 0.7369 | |
| 630 | 0.8475 | 0.8263 | |
| 2,100 | 0.8962 | 0.8818 | |
| 1,575 | 0.9233 | 0.9144 | |
| 126 | 0.9393 | 0.9345 | |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Task | Components | Scoring family |
| BI | Brand; brand–model pair | Exact label matches |
| BV | Voted brand; voted pair | Exact label matches |
| SC | Count | Configured count score |
| SR | Grid; extreme position | HBB [email protected] |
| WU | Weather category | Exact category |
| VH | Heading | Angle thresholds |
| Model | Overall | 2D Perception | 3D Perception | Cross-View Perception | Reasoning | |||||||||||||||||
| BI | BV | SC | SR | WU | VH | Mean | LW | HT | RD | AD | Mean | CC | CG | RI | Mean | CP | PC | CL | CH | Mean | ||
| Remote Sensing MLLMs | ||||||||||||||||||||||
| GeoLLaVA-8K | 0.11 | 1.25 | 1.23 | 0.08 | 0.00 | 0.00 | 0.00 | 0.43 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| VHM | 4.38 | 1.08 | 0.77 | 30.62 | 0.00 | 20.00 | 6.54 | 9.83 | 0.06 | 1.25 | 0.00 | 0.00 | 0.33 | 19.43 | 0.00 | 0.00 | 6.48 | 0.80 | 2.77 | 0.00 | 0.00 | 0.89 |
| GeoChat-7B | 8.31 | 1.86 | 2.00 | 25.40 | 0.58 | 20.00 | 1.91 | 8.63 | 1.07 | 13.00 | 0.00 | 14.53 | 7.15 | 22.91 | 0.00 | 0.00 | 7.64 | 17.66 | 11.41 | 0.00 | 10.21 | 9.82 |
| Embodied MLLMs | ||||||||||||||||||||||
| Model | Overall | Generation Speed (questions/s) | Peak GPU Memory (MiB) |
| ACE-Brain-0.6-9B | 43.29 | 0.123 | 30,687 |
| Qwen3.5-9B | 36.08 | 0.119 | 25,090 |
| RynnBrain1.1-9B | 35.10 | 0.122 | 25,093 |
| RoboBrain2.5-8B-NV | 32.84 | 1.478 | 17,335 |
| RynnBrain-8B | 31.69 | 1.198 | 17,335 |
| Qwen3-VL-8B-Instruct | 31.16 | 0.167 | 24,474 |
| Input or unit | |||
| BI/BV shared views | 4,360 | 5,789 | 8,079 |
| BV vehicle groups | 324 | 320 | 315 |
| WU records | 175 | 147 | 158 |
| RD records | 56 | 30 | 9 |
| CP records | 495 | 238 | 227 |
| PC records | 235 | 307 | 264 |