Gravity and Flight: Learning to Navigate Fine-Grained Remote Sensing Scenes
Organizations: The Institute of Robotics and Automatic Information System (IRAIS), the Tianjin Key Laboratory of Intelligent Robotics (tjKLIR), Nankai University, Tianjin, 300071, China · The College of Artificial Intelligence, Nankai University, Tianjin, 300071, China · The Tianjin Key Laboratory of Intelligent Robotics, the Academy for Advanced Interdisciplinary Studies, and the College of Artificial Intelligence, Nankai University, Tianjin, 300071, China · The Department of Imaging Science and Arts & Metaverse Convergence, Chung-Ang University, South Korea
Abstract
Vision-language models are increasingly capable, yet their reasoning in complex remote sensing scenes can remain coarse-grained and insufficiently grounded in task-relevant visual evidence. We propose Gravity and Flight (G&F), a framework for scene specialization with limited supervision that separates visual and reasoning verification while coupling policy updates. It combines Evidence-of-Thought supervised fine-tuning (EoT-SFT) and Relational Transfer Policy Optimization (RTPO). EoT-SFT teaches fine-grained observation and faithful reasoning through description, analysis, and supported conclusions. RTPO independently samples responses to scene-matched image pairs with fine-grained visual differences under a shared task. Its rubric-based verification combines Gravity, which checks visual fidelity and description coverage, and Flight, which takes descriptions as premises to assess evidence use and relations between paired judgments. Their scores and task correctness yield a shared pair-level advantage that couples policy updates from both responses. Using 1,008 demonstrations and 500 training pairs for harbor specialization, G&F achieves a mean score of 69.12 across six evaluation sets, exceeding the backbone and supervised initialization by 9.76 and 4.56 points. Matched controls show gains over single-image and joint-input RL, and removing either verification view reduces performance. Applying G&F to urban specialization also improves on the base model and supervised initialization across three evaluation subsets. Project details and updates are available at https://github.com/Z0ngL1ng/Gravity-and-Flight.
Figures & tables
| Model | Params | Data Scale | Overall | HarborEval | VRSBench | RSVQA | OpenEval | OmniEarth | VLRS-Bench |
|---|---|---|---|---|---|---|---|---|---|
| General Vision-Language Models | |||||||||
| LLaVA-1.5 [CVPR’24] [ 45 ] | 7B | 1.2M | 36.87 | 46.22 | 37.44 | 27.26 | 45.75 | 32.32 | 32.20 |
| Qwen3.5 [Tech. Rep.’26] [ 43 ] | 4B | n/r | 59.36 | 73.15 | 37.21 | 58.69 | 67.34 | 80.39 | 39.39 |
| Remote Sensing VLMs | |||||||||
| GeoChat [CVPR’24] [ 3 ] | 7B | 318K | 41.61 | 47.49 | 53.44 | 51.46 | 21.78 | 41.42 | 34.09 |
| SkySenseGPT [arXiv’24] [ 12 ] | 7B | 3.00M | 39.25 | 47.24 | 42.99 | 42.98 | 35.78 | 29.38 | 37.12 |
| Setting | Overall | vs | Rel. cost ( ) |
|---|---|---|---|
| (base) | 59.36 | 0.00 | n/a |
| (+EoT-SFT) | 64.56 | +5.20 | 1.00 |
| Paired reward ablations | |||
| w/o G and F | 63.34 | +3.98 | 8.15 |
| w/o Flight | 65.11 | +5.75 | 10.13 |
| w/o Gravity | 66.32 | +6.96 | 15.83 |
| Stage | EarthVQA Urban | VRSBench Urban | RSVQA HR |
|---|---|---|---|
| Base | 11.00 | 23.33 | 15.00 |
| +EoT-SFT | 29.50 | 54.17 | 39.00 |
| +RTPO | 31.43 | 56.25 | 43.67 |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Relation | Pairs | Share |
|---|---|---|
| Decisive change | 372 | 74.4% |
| Invariance | 77 | 15.4% |
| Visibility scope | 51 | 10.2% |
| Total | 500 | 100.0% |
| View | Generated or assessed content | Available reference inputs |
|---|---|---|
| Policy | Response generated independently for one image | Own image and shared question with public conditions |
| Gravity G1 | Visual assertions across description, analysis, and answer | Own image and Evidence Card |
| Gravity G2 | Initial description only | Own image and the card’s coverage anchors |
| Flight F1 | Own description and analysis | Shared question and public task conditions |
| Flight F2 | Both descriptions and analyses | Shared question, public conditions, comparison applicability and type |
| Task Check | Extracted final answer | Question and side-specific answer contract |
| Score | F1: within-response evidence use | F2: paired task execution and adaptation |
|---|---|---|
| 5 | Sufficient prior evidence is correctly selected and applied; the supported judgment has no task-use defect. | Both sides execute their applicable conditions, and the relation between judgments follows their described evidence. |
| 4 | All required links are supported, with only a minor, nondecisive overstatement in the reasoning. | Both sides satisfy the decisive and requested conditions, with only a minor supporting overstatement. |
| 3 | The decisive chain is supported, but a requested secondary aspect is unfinished or an ancillary operation is wrong. | Decisive conditions are met, but a secondary contextual requirement is incomplete. |
| 2 | A necessary prior observation, primary scope, or decisive operation is missing, including support first supplied in analysis. | A decisive applicable condition or its descriptive witness is missing on either side. |
| 1 | The analysis performs a definite wrong core operation, target selection, ownership assignment, or eligibility decision. | Either side violates a decisive public condition or uses an inapplicable target, rule, owner, or scope; or the judgment relation contradicts the described premises. |
| 0 | There is no usable task-directed analysis. | There is no usable paired task execution. |
| Dimension | Items | Within one point of humans (%) | Repeat-change rate (%) |
|---|---|---|---|
| G1 | 48 | 91.67 | 22.92 |
| G2 | 48 | 100.00 | 12.50 |
| F1 | 48 | 89.58 | 27.08 |
| F2 | 24 | 83.33 | 29.17 |
| Full-credit sides | Pairs (%) | Reward | Advantage |
|---|---|---|---|
| 2 | 167 (27.8) | ||
| 1 | 190 (31.7) | ||
| 0 | 243 (40.5) |
| Setting | Advantage | ||||
|---|---|---|---|---|---|
| Single-image RL | .50 | 0 | Per image | ||
| Paired RL w/o F2 | .50 | 0 | Shared pair | ||
| Shuffled F2 feedback | .50 | Shared pair | |||
| Joint-input RL | .50 | Joint group | |||
| G&F (RTPO) | .50 | Shared pair |
| Setting | EoT-SFT | RTPO |
| Training data | 1,008 demonstrations | 500 image pairs |
| Initialization | Base checkpoint | Stage-1 checkpoint |
| Optimizer | Fused AdamW | AdamW |
| Learning rate | ||
| Schedule / warmup | Cosine / 5% | Constant / 0 |
| Training passes / updates | 3 epochs / 189 | 1 pass / 250 |
| Evaluation list | Images | Items | Input and task coverage |
| Harbor specialization | |||
| HarborEval-lite v1.2.1 | 288 | 1,245 | Eight tracks over 184 RGB, 55 SAR, 32 panchromatic, and 17 near-infrared images. |
| VRSBench harbor | 200 | 580 | RGB imagery with 430 scored VQA prompts and 150 auxiliary caption prompts. |
| RSVQA harbor v3 | 78 | 719 | USGS test imagery with counting, presence, and spatial questions. |
| OpenEval harbor | 100 | 100 | One open-ended prompt per image across 55 RGB, 25 SAR, 10 panchromatic, and 10 near-infrared images. |
| OmniEarth harbor | 240 | 240 | Nine tasks covering perception, reasoning, and robustness, with segmentation tasks excluded. |