Organizations: The Institute of Robotics and Automatic Information System (IRAIS), the Tianjin Key Laboratory of Intelligent Robotics (tjKLIR), Nankai University, Tianjin, 300071, China · The College of Artificial Intelligence, Nankai University, Tianjin, 300071, China · The Tianjin Key Laboratory of Intelligent Robotics, the Academy for Advanced Interdisciplinary Studies, and the College of Artificial Intelligence, Nankai University, Tianjin, 300071, China · The Department of Imaging Science and Arts & Metaverse Convergence, Chung-Ang University, South Korea
Vision-language models are increasingly capable, yet their reasoning in complex remote sensing scenes can remain coarse-grained and insufficiently grounded in task-relevant visual evidence. We propose Gravity and Flight (G&F), a framework for scene specialization with limited supervision that separates visual and reasoning verification while coupling policy updates. It combines Evidence-of-Thought supervised fine-tuning (EoT-SFT) and Relational Transfer Policy Optimization (RTPO). EoT-SFT teaches fine-grained observation and faithful reasoning through description, analysis, and supported conclusions. RTPO independently samples responses to scene-matched image pairs with fine-grained visual differences under a shared task. Its rubric-based verification combines Gravity, which checks visual fidelity and description coverage, and Flight, which takes descriptions as premises to assess evidence use and relations between paired judgments. Their scores and task correctness yield a shared pair-level advantage that couples policy updates from both responses. Using 1,008 demonstrations and 500 training pairs for harbor specialization, G&F achieves a mean score of 69.12 across six evaluation sets, exceeding the backbone and supervised initialization by 9.76 and 4.56 points. Matched controls show gains over single-image and joint-input RL, and removing either verification view reduces performance. Applying G&F to urban specialization also improves on the base model and supervised initialization across three evaluation subsets. Project details and updates are available at https://github.com/Z0ngL1ng/Gravity-and-Flight.
Figures & tables
Figure 1: Motivation for G&F. A harbor example contrasts recognizing scene objects with using task-relevant local evidence to judge vehicle-staging suitability.
Figure 2: Overview of G&F. EoT-SFT provides supervised initialization. RTPO jointly evaluates independently generated responses with Gravity and Flight, then updates the policy using a shared pair-level advantage.
Model
Params
Data Scale
Overall
HarborEval
VRSBench
RSVQA
OpenEval
OmniEarth
VLRS-Bench
General Vision-Language Models
LLaVA-1.5 [CVPR’24] [ 45 ]
7B
1.2M
36.87
46.22
37.44
27.26
45.75
32.32
32.20
Qwen3.5 [Tech. Rep.’26] [ 43 ]
4B
n/r
59.36
73.15
37.21
58.69
67.34
80.39
39.39
Remote Sensing VLMs
GeoChat [CVPR’24] [ 3 ]
7B
318K
41.61
47.49
53.44
51.46
21.78
41.42
34.09
SkySenseGPT [arXiv’24] [ 12 ]
7B
3.00M
39.25
47.24
42.99
42.98
35.78
29.38
37.12
Table 1: Comparison on six harbor evaluation sets. Overall is their unweighted mean. Data Scale follows source-reported adaptation counts, with n/r marking unreported values. G&F uses 1,008 demonstrations and 500 pairs. Bold and underlining indicate the best and second-best scores.
Figure 3: Qualitative comparison on held-out harbor images using the same image–question inputs. The cases illustrate functional inference and vessel–dock spatial reasoning, with relevant evidence highlighted.
Setting
Overall
Δ vs M0
Rel. cost ( × )
M0 (base)
59.36
0.00
n/a
M1 (+EoT-SFT)
64.56
+5.20
1.00
Paired reward ablations
w/o G and F
63.34
+3.98
8.15
w/o Flight
65.11
+5.75
10.13
w/o Gravity
66.32
+6.96
15.83
Table 2: Harbor training comparisons. RL starts from M1 . Cost is the measured training wall time, including Stage 1 and verifier waiting, normalized to M1 under the optimized implementation. Measurement details in Appendix G .
Figure 4: Token-level organization of six randomly sampled response pairs. Peach (Base) and blue (G&F) tracks share a content-token axis that excludes format-only tokens. Dark marks show evidence (E) or inference (I) phrase membership, not attention. NA marks a response with no analytical tokens.
Stage
EarthVQA Urban
VRSBench Urban
RSVQA HR
Base
11.00
23.33
15.00
+EoT-SFT
29.50
54.17
39.00
+RTPO
31.43
56.25
43.67
Table 3: Urban specialization on fixed evaluation subsets.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Relation
Pairs
Share
Decisive change
372
74.4%
Invariance
77
15.4%
Visibility scope
51
10.2%
Total
500
100.0%
Appendix
Table S1: Composition of the 500 harbor training pairs. Relations specify task-relevant comparisons, allowing varied wording for semantically equivalent judgments.
View
Generated or assessed content
Available reference inputs
Policy
Response generated independently for one image
Own image and shared question with public conditions
Gravity G1
Visual assertions across description, analysis, and answer
Own image and Evidence Card
Gravity G2
Initial description only
Own image and the card’s coverage anchors
Flight F1
Own description and analysis
Shared question and public task conditions
Flight F2
Both descriptions and analyses
Shared question, public conditions, comparison applicability and type
Task Check
Extracted final answer
Question and side-specific answer contract
Appendix
Table S2: Information supplied to generation and verification. Public conditions are included in the policy instruction. Each scoring call receives its own prescribed view.
Score
F1: within-response evidence use
F2: paired task execution and adaptation
5
Sufficient prior evidence is correctly selected and applied; the supported judgment has no task-use defect.
Both sides execute their applicable conditions, and the relation between judgments follows their described evidence.
4
All required links are supported, with only a minor, nondecisive overstatement in the reasoning.
Both sides satisfy the decisive and requested conditions, with only a minor supporting overstatement.
3
The decisive chain is supported, but a requested secondary aspect is unfinished or an ancillary operation is wrong.
Decisive conditions are met, but a secondary contextual requirement is incomplete.
2
A necessary prior observation, primary scope, or decisive operation is missing, including support first supplied in analysis.
A decisive applicable condition or its descriptive witness is missing on either side.
1
The analysis performs a definite wrong core operation, target selection, ownership assignment, or eligibility decision.
Either side violates a decisive public condition or uses an inapplicable target, rule, owner, or scope; or the judgment relation contradicts the described premises.
0
There is no usable task-directed analysis.
There is no usable paired task execution.
Appendix
Table S3: Summary of Flight score anchors. Both dimensions accept described observations conditionally. A missing decisive premise differs from an explicitly wrong task operation.
Dimension
Items
Within one point of humans (%)
Repeat-change rate (%)
G1
48
91.67
22.92
G2
48
100.00
12.50
F1
48
89.58
27.08
F2
24
83.33
29.17
Appendix
Table S4: Verifier reliability on 24 training units. G1, G2, and F1 cover 48 side responses; F2 covers 24 candidate pairs. Agreement counts a judge score as matching when it is within one point of the adjudicated reference, whereas the repeat-change rate counts any difference between two identical calls.
Full-credit sides
Pairs (%)
Reward Rk
Advantage A^k
2
167 (27.8)
0.90[0.85,0.94]
0.66[0.22,1.09]
1
190 (31.7)
0.60[0.55,0.70]
−0.15[−0.72,0.51]
0
243 (40.5)
0.53[0.41,0.61]
−0.41[−1.10,0.53]
Appendix
Table S5: Realized matches in the 100-unit training trace. A full-credit side has Task Check Ci,k=1 . Reward and advantage are medians with the 25th and 75th percentiles in brackets.
Setting
C
G
U
H
Advantage
Single-image RL
.50
λG
γU
0
Per image
Paired RL w/o F2
.50
λG
γU
0
Shared pair
Shuffled F2 feedback
.50
λG
γU
γH
Shared pair
Joint-input RL
.50
λG
γU
γH
Joint group
G&F (RTPO)
.50
λG
γU
γH
Shared pair
Appendix
Table S6: Matched reward definitions. C , G , and U are local scores; H is shared paired adaptation. Coefficients are expressed using the notation of the main method. Shuffled F2 feedback permutes the association of F2 scores with candidate pairs.
Setting
EoT-SFT
RTPO
Training data
1,008 demonstrations
500 image pairs
Initialization
Base checkpoint
Stage-1 checkpoint
Optimizer
Fused AdamW
AdamW
Learning rate
10−4
2×10−6
Schedule / warmup
Cosine / 5%
Constant / 0
Training passes / updates
3 epochs / 189
1 pass / 250
Appendix
Table S7: Recorded harbor training configuration (top) and reward and optimization hyperparameters (bottom). The Stage-2 batch unit is an annotated image pair; each image contributes six rollouts. Within-view weights refer to the setting where both dimensions apply.
Evaluation list
Images
Items
Input and task coverage
Harbor specialization
HarborEval-lite v1.2.1
288
1,245
Eight tracks over 184 RGB, 55 SAR, 32 panchromatic, and 17 near-infrared images.
VRSBench harbor
200
580
RGB imagery with 430 scored VQA prompts and 150 auxiliary caption prompts.
RSVQA harbor v3
78
719
USGS test imagery with counting, presence, and spatial questions.
OpenEval harbor
100
100
One open-ended prompt per image across 55 RGB, 25 SAR, 10 panchromatic, and 10 near-infrared images.
OmniEarth harbor
240
240
Nine tasks covering perception, reasoning, and robustness, with segmentation tasks excluded.
Appendix
Table S8: Evaluation input lists. Images count distinct source image file paths within each list. Items count prompts, including report and caption prompts. Sensor counts below are counts of images.
School of Electronics and Information Engineering, Harbin Institute of Technology, 150000 Harbin, China. · Huawei International PTE LTd, 486066 Singapore, Singapore.
College of Computer Science and Technology, Jilin University, Changchun 130012, China · Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education Jilin University
School of Mechanical and Electrical Engineering, Central South University, Changsha 410083, China · School of Geosciences and Info-Physics, Central South University, Changsha 410083, China