From Sound to Sense: An Anatomy of Semantic Supervision for Speech-Driven Visual Grounding in Remote Sensing Imagery

Zhongyu Pang

cs.CVOct 12, 2026

From Sound to Sense: An Anatomy of Semantic Supervision for Speech-Driven Visual Grounding in Remote Sensing Imagery

Authors: Zhongyu Pang

Organizations: College of Electronic Science, National University of Defense Technology, Changsha 410073, China

Abstract

Speech-driven visual grounding in remote sensing (RSVG) removes the typing bottleneck of text-based interaction and is intended to preserve prosodic intent, enabling hands-free target acquisition in time-critical missions. The direct-audio paradigm, mapping raw speech straight to a bounding box and bypassing automatic speech recognition (ASR), was recently adopted by RSTalker, but what such grounders actually learn about language remains unmeasured. SemEcho grafts a frozen Whisper speech pathway onto a VLTVG grounding host so that speech and text share one decoder, making the paired text, free by construction since speech corpora in this task are synthesized from DIOR-RSVG expressions, a training-time teacher and a fair, same-host text upper bound. On this instrument we dissect three semantic injection channels (feature-mimicry distillation, word-level CTC supervision, and two-stage ASR pretraining) with linear semantic probes, and release Audio-DIOR-R (80,973 training clips; 10 timbres x K=3; Qwen3-TTS synthesis with online sox noise lambda in [0, 0.4]) and Audio-DIOR-Comp, a re-split holding out 17 attribute-category combinations under a 2x2 acoustic-semantic protocol. Our anatomy yields four findings. (i) Whether direct alignment beats a domain-trained ASR cascade is decided by recognizer quality: against a weak in-domain cascade (CTC 0.53) direct wins by 9.0 [email protected] points with transcription errors concentrated on spatial-relation words, but a strong in-domain cascade (CTC 0.029) reaches the text upper bound (80.8 vs. 81.0) and beats even the strongest direct model by 8.8 points; the anti-ASR premise holds only against weak recognizers.

Figures & tables

Fig. 1 : SemEcho at a glance. Spoken queries drive bounding-box prediction on remote-sensing imagery through a dual-stream host in which a frozen Whisper speech stream and a frozen BERT text stream share one decoder; three semantic distillation channels (A: feature mimicry, B: word-level CTC, C: two-stage ASR pretraining) inject language semantics at training time only. The full recipe reaches 64.32 [email protected] (

+11.04

over the matched baseline), closing 56% of the 19.7-point text–speech gap; 80,973 clips are released.

Fig. 2 : The SemEcho dual-stream host. A frozen Whisper-Base.en encoder feeds a lightweight speech adapter whose 40 projected tokens enter the same bert_proj projection and the same VLTVG multi-stage decoder as the frozen BERT text stream; the paired text is free by construction because every clip is synthesized from a DIOR-RSVG expression. Three semantic injection channels (feature mimicry, word-level CTC, two-stage ASR pretraining) attach to the speech stream and operate at training time only.

Generation	Front-end	Epochs	Final CTC	Wall-clock	Loaded by
v1 ( adapter.pt )	conv adapter	5	0.53	${\sim}1.4$ h	v6; the cascade recognizer of Section V-C
v2 ( connector_v2.pt )	transformer connector	20	0.006	${\sim}5.4$ h	v7
v3 ( connector_v3.pt )	transformer $+$ decoder-init	15	0.029	${\sim}11$ h	v8 / v8_ft2

Table I: Two-stage ASR pretraining: three generations. Each grounding run is reported with the generation it actually loaded.

Split	Queries	Clips	Hours	Notes
Train	26,991	80,973	53.5 h	10 timbres $\times\,K{=}3$ + online noise
Val	3,829	11,487	7.6 h	used for all reported val numbers
Test-Clean	7,500	22,500	15.4 h	seen-style timbres; synthesis complete
Test-Acoustic	7,500	22,493	—	2 held-out timbres + unseen noises; synthesis complete (7 missing)

Table II: Audio-DIOR-R splits. Train, val, Test-Clean, and Test-Acoustic audio are synthesized (Test-Acoustic: 22,493/22,500 clips; the 7 missing clips affect evaluated totals by at most 0.1 point).

	comp-train	comp-val	comp-test
Queries	26,090	3,698	1,284
(attribute, category) combinations	269 seen	—	17 held-out

Table III: Audio-DIOR-Comp re-split. Held-out combinations have atoms seen in training; only their composition is unseen.

	Combinations seen	Combinations unseen (Comp)
Acoustics seen (Test-Clean)	standard main table	Comp@Clean
Acoustics unseen (Test-Acoustic)	robustness table	Comp@Acoustic (hardest cell)

Table IV: The acoustic

\times

semantic 2

\times

2 evaluation protocol.

Model	Language supervision	Input	[email protected]	mIoU	$\Delta$ Acc
Fair text upper bound (same host)	full (paired text)	Text	72.97	62.31	$+19.69$
MGVLF [ 1 ] †	full	Text	76.78 †	78.42 †	—
RSTalker (reported) [ 2 ] †	none	Speech	79.85 †	78.29 †	—
Speech baseline	none	Speech	53.28	45.20	—
$+$ Channel A: feature mimicry (v4)	mimicry distillation	Speech	51.68	44.29	$\mathbf{-1.60}$
$+$ Channel B: CTC word-level from scratch (v5)	word-level	Speech	53.72	45.64	$+0.44$

Table V : Main results on the full Audio-DIOR-R validation set (3,829 queries, true pairwise IoU).

\Delta

Acc is computed against the speech baseline. † Reported in [ 1 , 2 ] on their own test protocol, host, and (unpublished) audio corpus — not comparable to our rows; shown only to locate the family.

Model	[email protected]	mIoU
Fair text upper bound (same host)	71.73	61.59
Speech baseline	51.75	44.39
$+$ Channel A: mimicry (v4)	51.28	43.82
$+$ Channel B: CTC (v5)	52.19	44.98
$+$ Channel C: ASR pretraining (v6)	52.53	45.06
$+$ Full strength intermediate (v7)	52.55	44.70

Table VI: Test-split confirmation: the full 7,500-query standard test split (Test-Clean audio, Section IV-A ), same systems as Table V .

Fig. 3 : The recipe gain and its test-split reproduction. a , Full-validation [email protected] for the seven systems of Table V : the two-stage recipe (v8_ft2, red) closes 56% of the 19.7-point gap to the fair text upper bound (8.65 remains). b , The same systems on the 7,500-query test split (Table VI ). The single-channel variants (v4–v6) sit within evaluation noise of the baseline at matched budgets (v4–v7).

Route	Input	[email protected]	mIoU
ASR cascade, v1 recognizer (CTC 0.53)	transcript $\to$ text path	53.00	46.34
ASR cascade, v3 recognizer (CTC 0.029)	transcript $\to$ text path	80.80	68.93
Direct speech (matched-budget baseline)	speech $\to$ shared decoder	61.20	52.34
Direct speech (full recipe, v8_ft2)	speech $\to$ shared decoder	72.00	61.27
Text upper bound (ground-truth expressions)	text	81.00	68.97

Table VII: Direct alignment vs. domain-trained ASR cascades vs. the text upper bound (first 500 val queries, identical samples; all routes in the same harness). The direct–baseline row is the 30-epoch matched-budget baseline; the direct–full-recipe row is v8_ft2 (the strongest direct model), evaluated in the same harness. The v1 rows are from the original run; its direct and text routes reproduced at 61.20/52.34 and 81.00/68.97 in the v3 run.

Fig. 4 : Cascade transcript analysis on identical queries: the v1-generation recognizer (CTC 0.53) corrupts or drops the spatial-relation words that select the target, while the v3-generation recognizer (CTC 0.029) preserves them — and the v3 cascade matches the text upper bound (80.80 vs. 81.00 [email protected]) where the v1 cascade loses 28.0 points.

Probe dimension	Chance	Baseline	v4 (mimicry)	v5 (CTC)	v6 (ASR-pre)	v7 (full)	v8 (final)
Category word (20-way)	5.0	31.0	33.0	53.0	34.5	66.0	41.5
Attribute word (9-way, attr subset)	12.5	34.1	41.5	36.6	39.0	31.7	31.7
Relation word (8-way, rel subset)	20.0	39.1	53.6	57.0	62.3	74.8	60.9
Bag-of-words (top-40, multi-label)	${\approx}15.8$	88.1	86.5	88.1	88.1	88.1	87.3

Table VIII : Linear semantic probes on the pooled speech representation (500 matched val queries); columns span the supervision dose.

Query bucket	Baseline	v4	v5 (CTC)	v6	v7 (full)	v8 (final)
Relation-bearing	60.9	59.0	63.0	60.9	59.8	70.5
Attribute-bearing	56.7	58.3	56.7	55.0	55.0	68.3
Simple (category-only)	71.9	65.6	65.6	70.3	68.8	76.6

Table IX : Bucketed localization ([email protected]); bucket sizes

n=376

/ 60 / 64 for relation / attribute / simple — the attribute and simple buckets are small, so we read them as trends and lean on the relation bucket for conclusions.

Probe (chance)	Frozen Whisper	Random adapter	Random proj.	Trained baseline
Category 20-way (5.0)	55.5	50.0	52.0	31.0
Attribute 9-way (12.5)	41.5	41.5	34.1	34.1
Relation 8-way (20.0)	64.2	58.9	58.3	39.1

Table X: Untrained probe controls: word-class decodability is already present in the frozen front-end.

Fig. 5 : Word-class decodability is inherited from the frozen front-end and recovered by the distillation channels. a , Relation-word probe (the hero): conventional grounding collapses the inheritance (39.1 vs. 64.2); the channels recover it monotonically and only v7 (74.8, green) crosses the inheritance line (gray dashed). b , Category probe. c , Attribute probe. Dotted line = chance; dark neutral bar = frozen front-end (the reference). v8 (red) trades probe decodability for localization (Fig. 7 , Table IX ).

Fig. 6 : Probe decodability vs. localization across the six systems (relation-word probe vs. relation-bucket [email protected]; Table VIII and Table IX ). The full-strength v8 sits below the frozen front-end’s inheritance line (dashed) on the probe yet leads every bucket — injected semantics sink from probe-detectable representation into localization behavior.

Fig. 7 : t-SNE of the v4 orthogonal-residual features (500 val queries, colored by the 10 synthesis timbres): timbre structure survives in the reserved subspace (fresh-probe decodability 69.0%, chance 10%), while collapsing to near chance in grounding-trained pooled representations without the reservation (Section V-E ).

Retrieval (speech $\to$ paired text)	R@1	R@10
Chance	${\approx}0.2$	${\approx}0.2$
Speech baseline	0.4	1.8
Mimicry-distilled (v4, ckpt ep21)	2.6	26.6

Table XI: Speech

\,\to\,

paired-text retrieval (earlier-protocol probe run, v4 checkpoint at epoch 21).

Loss family	Typical magnitude	Share of training loss
Box losses ( $\ell_{1}$ $+$ GIoU, deep supervision, teacher stream)	20–40	${\sim}98$ – $99\%$
Distillation terms $L_{q}+L_{\mathrm{attn}}+L_{\mathrm{sem}}+L_{\mathrm{orth}}$	${\approx}0.8$ (measured)	1–2%
CTC term (channel B, $\lambda=1.0$ )	—	${\approx}35\%$ at initialization, 3–8% at convergence

Table XII: Loss-share accounting during v4 training (loss-magnitude shares).

Arm	comp-val subset best	comp-test [email protected]	comp-test mIoU
Supervision-free baseline (conv adapter, Whisper-Base.en)	51.80	40.11	34.19
Full-strength recipe (transformer connector, Whisper-Small.en)	53.80	41.59	35.54

Table XIII: Audio-DIOR-Comp corrected two-arm chain (Comp@Clean cell). Both arms re-trained on comp-train (26,090 queries) under identical 20-epoch schedules, model selection on the comp-val monitoring subset (best epoch 19 in both arms), evaluation on the 1,284-query comp-test with a merged three-provenance audio bundle. The

+1.48

[email protected] difference is within the binomial 95% CI (

{\approx}{\pm}2.7

points at

n{=}1{,}284

): the preregistered

<2

-point branch applies — the two arms are statistically indistinguishable.

Connectionist Temporal Classification

See all 14 papers

Training-Free Open-Vocabulary Visual Grounding for Remote Sensing Images and Videos
Jun 15, 2026Ke Li, Di Wang, Yongshan Zhu +5Remote Sensing Image Understanding Temporal Video Grounding
School of Computer Science and Technology, Xidian University, Xi’an 710071, China · Interdisciplinary Institute of Artificial Intelligence, Xidian University, Xi’an, Shaanxi 710126, China · School of Artificial Intelligence, Xidian University, Xi’an 710071, China +2
ProVG: Progressive Visual Grounding via Language Decoupling for Remote Sensing Imagery
Apr 2, 2026Ke Li, Ting Wang, Di Wang +4Remote Sensing Visual Grounding Remote Sensing VLMs
Xidian University, China · UC San Diego, USA · SUST, China
ExACT: Exemplar-Driven Calibrated Refinement for Training-Free Visual Grounding in Remote Sensing Images
Jun 27, 2026Zixiao Zhang, Lingling Li, Pei He +2Vision-Language Grounding Visual Grounding
Xidian University

Jun 15, 2026J/K move · Enter open · S save

Training-Free Open-Vocabulary Visual Grounding for Remote Sensing Images and Videos

Ke Li, Di Wang, Yongshan Zhu, Ting Wang+4

School of Computer Science and Technology, Xidian University, Xi’an 710071, China · Interdisciplinary Institute of Artificial Intelligence, Xidian University, Xi’an, Shaanxi 710126, China · School of Artificial Intelligence, Xidian University, Xi’an 710071, China+2

Remote Sensing Image Understanding Temporal Video Grounding Vision-Language Grounding Visual Grounding Remote Sensing Visual Grounding Remote Sensing VLMs

Open paper AskPDF arXiv

Remote sensing visual grounding (RSVG) aims to localize a referred target in a remote sensing image or video according to a natural language expression. Existing RSVG methods usually rely on task-specific manual annotations, which are costly to collect and inevitably limited in covering the diversity of real-world geospatial scenarios. As a result, they often struggle to generalize to open-vocabulary queries involving novel objects, fine-grained attributes, complex spatial relationships, and functional semantics. In this paper, we propose RSVG-ZeroOV, a training-free framework that leverages frozen generic foundation models for zero-shot open-vocabulary RSVG. RSVG-ZeroOV follows an Overview-Focus-Evolve paradigm, which exploits the distinct yet complementary attention patterns of vision-language models (VLMs) and diffusion models (DMs) to progressively generate precise grounding results. Specifically, (i) Overview utilizes a VLM to extract cross-attention maps that capture semantic correlations between the referring expression and visual regions; (ii) Focus leverages the fine-grained modeling priors of a DM to compensate for object structure and shape information often overlooked by VLM attention; and (iii) Evolve introduces a simple yet effective attention evolution module to suppress irrelevant activations, yielding purified object masks. To handle video inputs, we further present Video RSVG-ZeroOV, which extends image-level grounding to spatio-temporal grounding through a query-relevant key-frame selector and a temporal propagator, enabling efficient and temporally coherent video grounding without video annotations or fine-tuning. Extensive experiments on six image and video grounding benchmarks show that RSVG-ZeroOV consistently outperforms existing zero-shot baselines and achieves competitive or superior performance compared with weakly- and fully-supervised methods.

Abstract

Connectionist Temporal Classification

Explore similar work

Training-Free Open-Vocabulary Visual Grounding for Remote Sensing Images and Videos

ProVG: Progressive Visual Grounding via Language Decoupling for Remote Sensing Imagery

ExACT: Exemplar-Driven Calibrated Refinement for Training-Free Visual Grounding in Remote Sensing Images