WeaveAgent: A Two-Stage Tool-Routing Agent for Ultra-High-Resolution Remote Sensing Imagery

Zhongyu Pang

cs.CVSep 25, 2026

WeaveAgent: A Two-Stage Tool-Routing Agent for Ultra-High-Resolution Remote Sensing Imagery

Authors: Zhongyu Pang

Organizations: Department of Electronics, National University of Defense Technology, Changsha, China

Abstract

Problem. Ultra-high-resolution (UHR) remote sensing with vague user intents has two bottlenecks: visual tokens are expensive, and tool calling must be format-reliable (pretrained models emit zero tool calls zero-shot). Method. WeaveAgent, a two-stage tool-routing agent, decouples routing from visual perception. Stage A is routing-first: emission is trained, not elicited. Stage B executes conditionally: intrinsic queries enter visual answering (full-scene thumbnail; a WeaveEarth-style evidence board as an optional fixed-budget, approx. 5k-token compression interface); extrinsic queries execute tool call on original full-resolution imagery, answering from tool observations in a second, observation-masked round. Training: alignment SFT, then GRPO under reward R_WA2. Results. Alignment SFT lifts extrinsic routing from 0% to 80.75% (323/400); GRPO suppresses 9 intrinsic mis-emissions while tool selection is unchanged. The trained 2B system does not beat the zero-shot 8B baseline overall (0.263 vs. 0.250), a diagnostic contribution. Oracle attribution separates two repair ingredients: loading the observation into context lifts extrinsic answer accuracy from 0.025 to 0.425 under marker-free cross-mode returns, and the two-turn SFT stage adds a further +9.3 points to 0.518 at a small routing cost. A +/- image ablation shows emission suppression is visually grounded, and a query-register matrix shows LLM-rewritten queries cost trained checkpoints 2-11 points. Scope. All training and evaluation use the 5,000 / 3,273 / 1,000-record VagueUHR corpus (600 intrinsic + 400 tool-requiring; the base seeds synthesis and is not used for optimization). Single-pass evidence construction runs at 7.31 s per image on an RTX 4090. Code, data, and evaluation protocols will be released.

Figures & tables

Figure 1: The problem and the answer at a glance. Left: a real test record, a vague query about a target that covers $0.14\%$ of a $5{,}000\times 5{,}000$ -px image (magnified inset); no thumbnail a VLM accepts resolves it. Right: WeaveAgent routes the query first (Stage A, a routing-first decision stage) and executes conditionally (Stage B): direct visual answering, or a T_call that is answered from its observation in a second round. Headline numbers: extrinsic routing $0\%\to 80.75\%$ after alignment training; answers on tool-bearing rows $0.025\to 0.425\to 0.518\to 0.965$ across the loading and repair stages ( Section 5.7 ).

Figure 2: WeaveAgent overview. Stage A performs intrinsic/extrinsic discrimination and tool selection from the query and the catalog (a routing-first decision stage whose emission suppression the

\pm

-image ablation shows to be visually grounded; Sections 3.2 and 5.6 ). Stage B executes conditionally: the intrinsic path answers from the thumbnail (optionally the TPEB/SEM compression interface); the extrinsic path executes the T_call on the original full-resolution imagery and answers from the tool observation in a second round.

Policy (on extrinsic tasks)	$R_{\mathrm{WA}}$	$R_{\mathrm{WA2}}$
Correct routing (right tool $+$ call)	0.35	2.25
Wrong tool call	0.05	0.25
Silent skip (no call, no answer)	0.06	0.0
Always answer directly: correct / wrong	1.11 / 0.11	1.25 / 0.25

Table 1: Per-case rewards on extrinsic tasks and the expectation algebra ( reward_wa2.py ; per-case values, no measured capability parameters). Weights:

R_{\mathrm{WA}}

=

1.0

answer

+

0.3

tool

+

0.1

format with a

0.2

no-call consolation;

R_{\mathrm{WA2}}

=

1.0

answer

+

1.0

tool

+

0.5

format, no consolation, and a correct route is automatically credited with the answer term (

r_{\mathrm{ans}}=\max(r_{\mathrm{ans}},1)

; Eq. ( 6 )). With

p_{\mathrm{ans}}

the policy’s answer accuracy on extrinsic tasks,

E_{R_{\mathrm{WA}}}[\text{always-answer}]=0.11+p_{\mathrm{ans}}

exceeds

E[\text{correct routing}]=0.35

once

p_{\mathrm{ans}}\geq 0.24

: all-direct-answer is RL-rational. Under

R_{\mathrm{WA2}}

the inversion (

2.25\gg 0.25+p_{\mathrm{ans}}

) is parameter-free; it holds for any

p_{\mathrm{ans}}\leq 1

Figure 3: The training recipe (left): alignment SFT and the E15 GRPO stage on the single-turn branch; right: the E17 two-turn stage, which branches from the E13 base. Turn 1 is byte-identical to the E13 aligned demonstrations; turn 2 loads the tool observation in the RemoteAgent inference format. The completion-only loss covers both assistant turns: emission (inherited) and observation

\to

answer conversion (new in E17).

Split	Records	Tool-traj. records	Tool pool	Task shapes
Train (template)	5,000	— (all intrinsic)	—	5 (training-visible)
Routing-train ( train_tools )	3,273	873 (870 rendered tool-call demos; 3 skipped: gt trajectory has no real tool step; 0 render failures)	2 visible: ref-seg 579 / skysense-det 291	2,400 intrinsic $+$ 873 tool
Test (template)	1,000	400	3: skysense 138 $+$ sm3det 62 (unseen) $+$ ref-seg 200	10 (5 unseen in training); 600 intrinsic $+$ 400 tool
Test (LLM-rewritten)	992 kept $+$ 8 removed (gt–image conflicts) $=$ 1,000	593 intrinsic $+$ 399 extrinsic (8 removed)	same	same

Table 2: VagueUHR dual splits. Tool-trajectory records carry explicit T_call steps; all answers derive from source annotations. Five of the ten evaluation task shapes never appear in training; the test tool pool contains one training-unseen tool (sm3det, 62 records).

Model / training stage	Intr.	Extr.	T-sel.	Calls	Mean
Qwen3-VL-8B zero-shot (E3a)	1.000 (600/600)	0.000 (0/400)	0.600 ∗	0	0.2630
GRPO $R_{\mathrm{RA}}$ from 2B base (E3b)	1.000	0.000	0.600 ∗	0	0.2420
GRPO $R_{\mathrm{WA}}$ from 2B base (E3b)	1.000	0.000	0.600 ∗	0	0.2570
Base 2B, native tools= channel (zero-shot)	0.015 (9/600)	0.12 (48/400)	0.057	986	—
Aligned SFT (E13, 2B)	0.970	0.8075 (323/400)	0.905	398	0.2430
$+$ GRPO $R_{\mathrm{WA2}}$ (E15)	0.985	0.8075 (323/400)	0.914	384	0.2500

Table 3: Routing and tool metrics on VagueUHR-test (1,000 records; 600 intrinsic

+

400 tool-requiring; five task shapes unseen in training; all greedy). Intr. = intrinsic routing accuracy; Extr. = extrinsic routing accuracy; T-sel. = fraction of all records whose tool decision is correct (a correct no-call on an intrinsic record counts as correct, as does an emitted call matching the ground-truth tool on an extrinsic record; hence E13’s

0.905=(0.970{\times}600+323)/1000

); Calls = number of T_call emissions; Mean = overall answer accuracy over all 1,000 records. Call accounting: E13’s 398

=

380 extrinsic single-calls

+

18 intrinsic mis-emissions; E15’s 384

=

375

+

9. The native-channel row emits through the official tools= channel rather than T_call ; its Mean is not measured. ∗ With zero emissions the tool-selection aggregate is a bookkeeping artifact (nothing to judge), not a measured capability.

Figure 4: Five-fold decomposition of the extrinsic pipeline on the 400 tool-bearing records ( Table 4 ). Argument folds are near-perfect while final answers stay broken for E13/E15: the layer-4 break is in converting tool text into answers; the two-turn SFT (E17) repairs exactly that fold.

Model	Family	Inst	Tool	ArgN	ArgV	Summ
Aligned SFT (E13)	detection	0.905	0.685	0.994	0.909	0.035
	segmentation	0.995	1.000	0.997	0.990	0.005
	ALL	0.950	0.850	0.996	0.951	0.020
$+$ GRPO $R_{\mathrm{WA2}}$ (E15)	detection	0.880	0.705	0.994	0.932	0.045
	segmentation	0.995	1.000	0.997	0.990	0.005
	ALL	0.938	0.861	0.996	0.963	0.025

Table 4: Five-fold decomposition (Inst / Tool / ArgN / ArgV / Summ) of the extrinsic pipeline, computed on the needs-tool records: detection and segmentation rows on the 200 records of each family, ALL on the 400 tool-bearing records. Fold denominators: Inst/ArgN/ArgV/Summ over the 400 needs-tool records; Tool over the emitted-call records (denominators 380 / 375 / 381 for E13/E15/E17). (Overall test accuracy, a different quantity, appears in Table 3 .) E17 is the two-turn model of Section 3.6 under the real (strict-return) protocol of Section 5.7 . Answers produced without any tool call ( answer_wo_tool_call ): E13

=

20, E15

=

25, E17

=

19; E17’s 381 emitted-call records are exactly those with a single well-formed call ( single_call_ok ). Family rows are computed within the decomposition pipeline over

n{=}200

per family; the ALL row is the routing-pipeline tool-selection score over 380 evaluated rows (20 execution-failure exclusions), so family and ALL rows use different denominators by construction.

Figure 5: GRPO training dynamics (per-log-interval values,

x

= training progress). On the gray curve (E12: cold-start SFT then

R_{\mathrm{WA}}

GRPO), the reward stays low and most rollout groups have zero advantage, giving no gradient signal for emission. After alignment SFT (E15, green), the reward starts high and zero-variance groups stay an order of magnitude lower: emission is seeded, and the reward can shape it.

Metric	GRPO from base (E7a)	SFT cold start $+$ GRPO $R_{\mathrm{WA}}$ (E12)	Aligned SFT $+$ GRPO $R_{\mathrm{WA2}}$ (E15)
Initial reward	${\approx}$ 0.058	${\approx}$ 0.55 (first-6: 0.5455)	1.18
frac_reward_zero_std	0.85–0.9	0.43–0.78	0–0.1 early, ${\leq}0.3$ for most of training (max 0.47)
Completions mean length (tokens)	59–69	11–27 (converges to direct answers; the $R_{\mathrm{RA}}$ arm collapses to 8.5–9.7)	165–180 (stable)

Table 5: Training dynamics of three configurations. Zero-variance groups are rollout groups whose four rewards are identical (zero advantage, zero gradient); mean length is the token length of generated completions. Ranges are min–max over logged steps of the respective runs.

Benchmark	Thumbnail 2048 (interface off)	Thumbnail $+$ board ( $k{=}6$ )	Paired $\Delta$
LRS-VQA (1,500 strat.)	33.20	32.20	$-1.0$
MME-RealWorld-RS (1,500 strat.)	39.33	35.93	$-3.4$
XLRS-Bench-lite (1,500 strat.)	41.47	38.60	$-2.9$

Table 6: Compression-interface paired characterization (Qwen3-VL-8B, greedy, identical 1,500-record stratified samples per benchmark; accuracy %). The evidence board’s token cost is fixed (thumbnail

+

board, two views

{\approx}5

k tokens) and does not grow with source size.

Variant	GCC	MSES	SEM	TPEB	Accuracy
w/o GCC	–	✓	✓	✓	0.3040
w/o MSES	✓	–	✓	✓	0.3140
w/o SEM	✓	✓	–	✓	0.3080
w/o TPEB	✓	✓	✓	–	0.3000
WeaveAgent (full)	✓	✓	✓	✓	0.2940

Table 7: Component ablations on LRS-VQA following the WeaveEarth protocol (accuracy;

n{=}500

stratified, greedy,

k{=}6

; the full-system row is the E4_full same-subset control).

$k$	2	4	6	8	10
Accuracy	0.3080	0.3040	0.2940 ∗	0.3100	0.3020

Table 8: Evidence-budget scan (

n{=}500

same-subset, greedy).

{}^{*}k{=}6

is the full-system same-subset control (E4_full).

Method	Hardware / protocol	s / sample	Relative
Thumbnail baseline 2048 (official script)	4090, $n{=}300$ , greedy	2.84	0.39 $\times$
Zoom baseline (ours, E6, multi-round)	4090, $n{=}50$	4.33	0.59 $\times$
Single-pass evidence construction (ours, E1a, greedy)	4090, $n{=}1500$	7.31	1 $\times$
WeaveAgent E15 single-pass (VagueUHR-test)	4090, $n{=}1000$	6.17	0.84 $\times$
WeaveEarth (paper-reported)	A100	7.59	1.04 $\times$
ZoomSearch (paper-reported, multi-round search)	—	54.25	7.4 $\times$

Table 9: End-to-end latency. Hardware and protocol differ across rows and are stated per row; paper-anchored rows are quoted from the respective papers and serve as order-of-magnitude references only. Relative times are normalized to our single-pass pipeline.

Figure 6: The query-register gradient: overall accuracy for three checkpoints under three query registers (filled circles: clear; open squares: template-vague; diamonds: LLM-rewritten; per-cell extrinsic routing in Table 10 below). The untrained row is flat: a zero-shot model cannot route, so register does not matter. Training creates a register gradient: template-vague queries behave like clear ones (

\leq

1.2 points), while LLM-rewritten queries cost the trained checkpoints

2

–

11

points. The vagueness axis is real, measured, and partially absorbed by the deeper recipe (E17 exceeds E15 in every register).

	Clear		Template-vague		LLM-rewritten		LLM per-type
Checkpoint	overall	extr	overall	extr	overall	extr	det	seg
8B zero-shot	0.268	0.000	0.263	0.000	0.296	0.0025	0.231	0.405
E15 (aligned)	0.241	0.800	0.250	0.8075	0.219	0.667	0.085	0.005
E17 (two-turn)	0.476	0.7875	0.488	0.800	0.379	0.659	0.382	0.235

Table 10: Query-register gradient on VagueUHR-test: overall accuracy (primary) and extrinsic routing (secondary) for the untrained 8B row and the two aligned checkpoints under three query registers. The E17 row uses the two-turn protocol with Eval-A oracle-grounded tool returns (return quality is an upper bound; the strict cross-mode protocol itself is not the performance upper bound, which is Eval B, Section 5.7 ); all LLM-rewritten cells use the 992-record paired rewritten split, clear/template cells the full 1,000-record test set. Det./Seg. give per-type accuracy on the rewritten register (8B from the E14 arm of Table 11 ); the two-turn repair’s largest per-type gains are exactly in these columns.

Split	Overall acc.	Intr. routing	Extr. routing	T-sel.	Calls
Template (992 paired)	0.2651	1.000	0.000	0.598 ∗	0
LLM-rewritten (992)	0.2964	0.998 (592/593)	0.0025 (1/399)	0.598 ∗	2

Table 11: Query-wording sensitivity, paired across splits (992 records; Qwen3-VL-8B zero-shot, greedy). Per-type

n

: 197 / 199 / 199 / 197 / 200 (classification/counting/detection/reasoning/segmentation). The template arm reproduces the E3a configuration on the paired subset; the LLM arm uses the visually rewritten queries.

\Delta=+3.12

points, McNemar exact

p=0.0433

. ∗ With (near-)zero emissions the T-sel. aggregate reduces to counting correct no-calls on intrinsic records, a bookkeeping artifact, not a measured capability ( Table 3 ).

Figure 7: The

\pm

-image ablation (E19, E15 checkpoint, 1,000 records, identical prompts). Removing image tokens keeps extrinsic routing (0.838 vs. 0.808) but collapses intrinsic routing to 0.022: the model emits a call on almost every record (987 vs. 384). Emission suppression, knowing when not to call, is visually grounded. The

15\times

latency gap makes a truly text-only router an attractive but unattained target.

Figure 8: Oracle attribution on the 100 routed-correct records ( Table 12 ): answer accuracy when the tool return is marker-free (strict) vs. answer-marked (embedded). The gap concentrates in segmentation, whose mask-type returns carry almost no textual information. Upper bound, not natural behavior.

Figure 9: The layer-4 repair, end to end. Answer accuracy on the 400 tool-bearing records across the five-point ladder: E15 (single-turn, no result loading), the two protocol-matched control arms (E13/E15 with the same strict returns loaded but no two-turn training), E17 under marker-free cross-mode returns, and E17 under training-distribution (embedded) returns. The ladder branches at the last two points: one trained checkpoint, two return protocols (the intrinsic task types and routing metrics are identical across them).

Arm	Detection	Segmentation	Overall	No-call
Strict (marker-free)	0.784 (29/37)	0.339 (21/62)	0.505	1/100
Embedded (answer-bearing)	0.974 (37/38)	0.758 (47/62)	0.840	0/100

Table 12: E16 oracle attribution (100 routed-correct records stratified by task type: 37–38 detection / 62 segmentation; two-round protocol, greedy; oracle returns). “Strict” strips answer markers from tool returns; “embedded” stuffs RemoteAgent-style trajectory text with answers in place. The detection denominator under strict is 37 (one no-call record).

Protocol	Overall	Class.	Count.	Reas.	Det.	Seg.	Extr.	Intr.	T-sel.	Calls
Eval A (strict fallback)	0.488	0.560	0.330	0.515	0.635	0.400	0.800	0.913	0.868	433
Eval B (embedded)	0.667	0.560	0.330	0.515	0.960	0.970	0.800	0.913	0.868	433
E13 $+$ loaded strict returns (control)	—	—	—	—	0.510	0.335	extr. ans. $0.4225$
E15 $+$ loaded strict returns (control)	—	—	—	—	0.510	0.340	extr. ans. $0.4250$

Table 13: E17 two-round evaluation under two protocols (full 1,000-record test set, greedy; oracle returns in both, no live service). Eval A: oracle-grounded strict marker-free returns via per-record fallback (fallback rate 1.0, disclosed); cross-mode generalization point. Eval B: embedded returns, training distribution; upper bound. Intrinsic types and routing metrics are identical across evals (protocol sanity). Class./Count./Reas./Det./Seg. = answer accuracy per task family (200 records each); Extr./Intr. = routing accuracies; T-sel. = tool selection. Control arms: same strict-return protocol and oracle results, no two-turn training; E15

\to

E17 difference on the same 400 rows is

+9.25

pt (McNemar

p=3.8\times 10^{-5}

Figure 10: Four qualitative cases from the E17 two-turn evaluation (Table S7; greedy, Eval-A protocol). Each band shows the input crop (ground-truth box in red where defined), the query, the Stage-A decision with the emitted T_call , the marker-free turn-2 tool return, and the final answer against the ground truth. C4 is the instructive conversion failure: the count is present in the return (“10 objects localized”) yet the model answers 3.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

#	Difference	Training prompt	Evaluation prompt
1	Opening sentence	“You are WeaveAgent answering…”	“Answer the question about…”
2	Dual-image description block	none	thumbnail $+$ evidence-board description
3	Category line	none	Category: <task_type>
4	Query prefix	User query:	Question:
5	Catalog position	before the query	after the query
6	Decision-instruction position	last line, mentions both options	mid-prompt, followed by 3 more blocks

Appendix

Table S1: Supplementary table (appendix A). Non-image differences between the unaligned training prompt and the evaluation prompt (byte-level audit; the evidence board is genuinely injected at evaluation time, ruling out “evidence not given”).

Prompt family	Thumbnail	TPEB board	SEM block
Routing-train prompts (E13/E15)	path (lazily decoded)	none	none (sem_text 0/3,273)
Main evaluation prompts	present	present (real injection)	present
E19 text-only arm (E15 ckpt.)	image removed; description text retained	image removed; description text retained	retained as text (byte-identical prompt)
E17 turn-1 / turn-2	present (E13 demo images)	present (evaluation-identical image format)	none (training side); turn-2 adds the observation text

Appendix

Table S2: Supplementary table (appendix A). Presence of the thumbnail, the TPEB board, and the SEM block across prompt families (consolidated from Table S1 , Sections 3.5 and 6.1, and Table S7 ; facts only, no new measurements). E17 turn-1 is byte-identical to the E13 aligned demonstrations, whose prompt and image formats are strictly identical to evaluation; turn-2 loads the stuffed tool observation in the RemoteAgent inference format.

ID	Configuration	Result
E1a	Single-pass evidence pipeline, interface on (thumbnail $+$ board, $k{=}6$ ); three benchmarks, stratified 1,500 each, 8B, greedy; timing anchor	Tables 6 and 9
E1b	Evidence interface off (2048 thumbnail), same samples	Table 6
E3a	Zero-shot Qwen3-VL-8B routing evaluation	Table 3
E3b	GRPO from the 2B base, $R_{\mathrm{RA}}$ / $R_{\mathrm{WA}}$ arms (merged E7a/E7b weights)	Table 3
E4 / E4_full	Component ablations with the same-subset full-system control ( $n{=}500$ , greedy)	Table 7
E5	Evidence-budget $k\in\{2,\dots,10\}$ scan	Table 8

Appendix

Table S3: Supplementary table (appendix A). Experiment register: one row per experiment ID used in this paper — configuration and result pointer (facts only). “Official 300” is the released-script 300-record stratified paired replication.

Case	Answer indicator	Tool indicator	Format indicator	Total
Intrinsic, direct answer correct	1.0	1.0	0.5	1.35
Intrinsic, direct answer wrong	0	1.0	0.5	0.35
Intrinsic, emits T_call	0	0.0	0.0	0.00
Extrinsic, correct T_call ( $=$ SFT behavior)	0	1.0	0.5	0.35
Extrinsic, wrong T_call	0	0.0	0.5	0.05
Extrinsic, no call $+$ wrong answer	0	0.2	0.5	0.11

Appendix

Table S4: Supplementary table (appendix B).

R_{\mathrm{WA}}

per-case scores (measured configuration:

w_{a}{=}1.0

w_{t}{=}0.3

w_{f}{=}0.1

, no-call consolation

0.2

). Total

=1.0\cdot

answer

+\;0.3\cdot

tool

+\;0.1\cdot

format (

R_{\mathrm{WA}}

weights); the three columns are unweighted component scores: a binary indicator (answer), a routing/consolation term (0.2 under

R_{\mathrm{WA}}

, 0 under

R_{\mathrm{WA2}}

), and the format term (0.5 per well-formed component). On intrinsic tasks a no-call earns the full tool term (there is no ground-truth tool); the structure makes bare T_call behavior — exactly what SFT teaches — nearly worthless relative to direct answering.

Measurement	Numbers	Source
WeaveEarth paper-reported (LRS-VQA full set)	26.68 $\to$ 33.38 ( $+6.70$ )	paper
Released script, 300-record stratified paired, both arms greedy	29.67 $\to$ 32.33 ( $+2.67$ , 24/32 discordant, McNemar exact $p=0.350$ )	this work, Section 5.1
Category task-type line asymmetry (released baseline lacks it, treatment arm has it)	paired control (E20): $\Delta=-1.7$ points, McNemar $p=0.51$ — no answer leakage; the earlier ${\approx}3.7$ -point cross-protocol inference is refuted	this work (E20)
MCQ option-instruction order (self-audit run, indicative)	25.47 $\to$ 35.93 ( ${\approx}10.5$ -point swing; 34% of outputs degrade to prose)	quarantined audit

Appendix

Table S5: Supplementary table (appendix B). Implementation differences and prompt sensitivity: neutral protocol-alignment records. The MCQ-order row comes from a quarantined self-audit run and is indicative only.

Figure S1: Supplementary figure (appendix C). Query-wording sensitivity on the 992 paired records (zero-shot Qwen3-VL-8B, greedy; Table 11): per-task-type accuracy on the template split vs. the LLM-visual-rewrite split. The movement is a per-type reshuffle rather than systematic damage.

Figure S2: Supplementary figure (appendix C). Category-line paired control (300 records, 60 per task type, E15 checkpoint, greedy): per-type answer accuracy with vs. without the Category row (classification

+13.3

, detection

-8.3

, reasoning

+8.3

points). The movement is a per-type reshuffle, not a systematic leak; the overall paired effect is

-1.7

points (

p=0.51

Task type	Recall@8 anchors	Recall@6 final board	$n$
Classification	0.285	0.240	200
Counting	0.0	0.0	2
Detection	0.323	0.258	62
Reasoning	0.296	0.204	196
Segmentation	0.285	0.240	200
All spatial records	0.291	0.230	660

Appendix

Table S6: Supplementary table (appendix C). GCC retrieval recall per task type on the 660 spatial-ground-truth test records (gt-box-center criterion; Section 6.3). Counting has no spatial ground truth (

n{=}2

) and is excluded from the uniformity statement. Classification and segmentation coincide exactly at this sample size (57/200 at

k{=}8

and 48/200 at

k{=}6

, identical for both types) — verified against the raw per-type output, not a transcription error.

Case	Query	Stage-A decision and T_call	Turn-2 tool return	Output vs. GT
C1 (det.)	“What is the amount of boat in the image?” ( 4498 )	extrinsic: T_call(skysense_detection, classes=[object], confidence_threshold=0.5)	“Detection finished; 3 objects localized.”	3 vs. 3 ✓
C2 (seg.)	“hey, what is the shape of the top-most tank?” ( 0177 )	extrinsic: T_call(ref-seg, prompt= $\langle$ query $\rangle$ )	“Segmentation mask generated.”	circular vs. circular ✓
C3 (intr.)	“hey, what is the activity of the top-most plane?” ( 983 )	intrinsic: no call; answers from thumbnail (+ evidence board)	— (single turn)	parked vs. parked ✓
C4 (fail)	“What is the amount of bridge in the image, roughly?” ( 11069 )	extrinsic: T_call(skysense_detection, classes=[bridge], confidence_threshold=0.5)	“Detection finished; 10 objects localized.”	3 vs. 10 $\times$

Appendix

Table S7: Supplementary table (appendix C). Four qualitative cases from the E17 two-turn evaluation (greedy; Section 5.7, Eval A protocol — tool returns are annotation-grounded fallbacks, marker-free). Question IDs are given for reproducibility. ref-seg

=

referring_expression_segmentation .

Explore similar work

Jul 28, 2026cs.CV

Beyond Zooming: Learning Multi-Tool Visual Reasoning for Ultra-High-Resolution Remote Sensing

Ultra-high-resolution (UHR) remote-sensing (RS) imagery provides fine-grained Earth-observation evidence over city-scale scenes, but poses a fundamental challenge for multimodal large language models (MLLMs): task-relevant evidence is often sparse, local, and spatially dispersed across extremely large visual contexts. A natural solution is to equip MLLMs with zoom-in tools for active local inspection. However, through a pilot study on XLRS-Bench, we find that zoom-in is only partially effective: it resolves easy and medium-level tasks with locally recoverable evidence, but saturates on hard cases requiring global search, multi-region comparison, path planning, or dispersed-evidence reasoning. Motivated by this finding, we move beyond single-tool zoom-in and introduce GeoMTVR, a large-scale Geospatial Multi-Tool Visual Reasoning dataset built from wide-area satellite imagery. GeoMTVR contains 13K UHR VQA samples with interleaved reasoning trajectories, diverse visual tool calls, and returned visual observations, enabling models to learn question decomposition, tool selection, regional inspection, object-level grounding, auxiliary visual reasoning, and cross-tool evidence integration. Beyond supervised fine-tuning, we propose a tool-attention-focused reinforcement learning algorithm that concentrates optimization on critical tool-use decisions, including when to invoke tools, which tool to select, where to apply it, and how to interpret tool outputs. By combining SFT on GeoMTVR with our RL algorithm, we develop GeoLens, a multi-tool visual reasoning MLLM for UHR RS. Experiments show that GeoLens consistently outperforms direct reasoning and single-tool zoom-in baselines, achieving stronger accuracy, better evidence grounding, and more efficient tool-use trajectories.

Fengxiang Wang, Jiangnan Huang, Mingshuo Chen +8

National University of Defense Technology · Wuhan University · Tsinghua University

Trustworthy Earth Observation Decisions Visual Reasoning

Ask PDF

Jul 11, 2026cs.CV

WeaveEarth: Structured Evidence Construction and Reasoning for Training-Free UHR Remote Sensing Understanding

Ultra-High-Resolution (UHR) remote sensing image understanding requires Vision-Language Models (VLMs) to capture both the global scene layout and sparse yet task-critical local details under limited computational budgets. Existing methods mainly follow two paradigms. One is passive perception, which relies on resolution expansion or token compression and may therefore discard fine-grained details. The other is active perception, which depends on multi-round zooming and search, but suffers from high latency, contextual fragmentation, and error accumulation. We argue that a more effective path toward UHR understanding lies not in accessing more, but in organizing better. To this end, we propose WeaveEarth, a training-free framework that reformulates UHR understanding as a problem of structured evidence construction and reasoning under global context constraints. Specifically, WeaveEarth first employs Global-Aware Evidence Construction to select a compact, low-redundancy, and spatially complementary Minimal Support Evidence Set. It then introduces Structured Evidence Reasoning, which weaves local evidence, spatial metadata, and relative topology into a unified reasoning interface, thereby enhancing the VLM's ability to perform global-local joint reasoning. Extensive experiments show that WeaveEarth consistently outperforms strong baselines and existing UHR methods across multiple UHR remote sensing benchmarks and multiple frozen VLM backbones. Code is available at https://github.com/XianZhi-Ma/WeaveEarth.

Xianzhi Ma, Shujun Wang, Xiaohan Li +3

School of Frontier Sciences, Nanjing University, Suzhou, China · Computer Network Information Center, Chinese Academy of Sciences, Beijing, China

High-Resolution

Ask PDF

Sep 29, 2026cs.CV

RS-OPSD: Reliable Privileged On-Policy-Self-Distillation for Ultra-High-Resolution Remote Sensing VQA

Ultra-high-resolution (UHR) remote sensing visual question answering (VQA) requires models to resolve small visual evidence within extremely large images. Existing approaches typically rely on token pruning, visual search, or tool-augmented reasoning at inference time. We instead investigate whether the benefit of zoom-in visual privilege can be internalized into the model. We introduce RS-OPSD, a reliable privileged on-policy self-distillation (OPSD) framework for UHR remote sensing VQA. To provide high-quality privileged information with explicit question-relevant evidence, we construct GeoEvidence-6K, containing 6,750 VQA samples across seven task categories with evidence-region annotations, and develop Human Feedback-Guided Skill Refinement (HF-SR) for scalable annotation. To address context loss from tight crops and conflicting signals from imperfect teachers, RS-OPSD introduces Context-Preserving Visual Privilege (CPVP) and Correctness-Aligned Distillation (CAD). Without any additional visual search or tool calls at inference time, RS-OPSD achieves state-of-the-art (SOTA) performance on XLRS-Bench, MME-RealWorld-RS, and LRS-VQA, outperforming previous SOTA models of comparable scale by an average of 4.0 percentage points. Moreover, our 2B variant, RS-OPD-Lite, surpasses most 8B-scale models while achieving the fastest measured inference speed. Our Code, GeoEvidence-6K, and the model weights for RS-OPSD and RS-OPD-Lite are publicly available.

Chengjie Jiang, Yunqi Zhou, Jiafeng Yan +3

Tsinghua University · Zhejiang University · Central University of Finance and Economics +2

Textvqa Remote Sensing Rp-Opsd Unsupervised On-Policy Self-Distillation Privileged Context

Ask PDF