Commanding at the Speed of Sound: Audio-Centric Referring Expression Segmentation for Aerial Scenes
Authors: Ziyun Chen, Yijun Wang, Chuanyi Zhang, Liang Yao, Yuexuan An, Xiang Gu, Wei Zhou, Fan Liu
Organizations: Hohai University, Nanjing, China · Key Laboratory of Water Big Data Technology of Ministry of Water Resources, Hohai University, Nanjing, China
Referring expression segmentation (RES) aims to localize targets at the pixel level guided by human instructions. However, current models predominantly follow a text-centric paradigm that treats discrete text as the primary interaction medium for these human instructions. This assumption is especially limiting in unmanned aerial vehicle (UAV) applications, where rapid response, hands-free operation, and natural communication are often critical. In this paper, we reconstruct UAV referring expression segmentation through an audio-centric perspective. Grounded in information theory, we demonstrate that raw audio provides a higher upper bound for extracting rich semantic features than transcribed text. To achieve this, we construct an audio-driven UAV referring expression segmentation benchmark named Aerial-AudioRES by augmenting VisDrone with synthesized speech, forming "audio-text-image-mask" quadruples. Specifically, we employ an ambiguity filtering mechanism to ensure referential uniqueness, and incorporate eight distinct timbres alongside 4 realistic noises to simulate authentic environments. To drive the paradigm shift from text-centric to audio-centric, we propose the end-to-end Audio-centric Referring Expression Segmentation Network (ARES-Net). Distinct from discrete text-visual matching, our approach learns to establish a direct soft alignment between continuous acoustic manifolds and multi-scale visual representations. Comprehensive experiments show that our method obtains a peak accuracy of 64.23% in oIoU and demonstrates robust generalization on a collected real-human test set. Furthermore, it significantly reduces total interaction latency to one-third of the text-based method. Our code is available at https://github.com/LemonGauss/ARES-Net.
Figures & tables
Fig. 1: Comparison of RES paradigms: (a) Text-Centric: inefficient manual typing; (b) ASR+Text: prone to transcription errors (e.g., “truck” misrecognized as “track”) leading to failed segmentation; (c) Audio-Centric (Ours): robust, end-to-end, efficient perception without intermediate errors.
Fig. 2: Overview of the Aerial-AudioRES construction pipeline. It combines ambiguity-aware target selection and progressive caption generation with automated mask extraction and speech synthesis to construct aligned audio-text-image-mask quadruplets for aerial referring segmentation.
Fig. 3: Overview of the ARES-Net. The model aligns acoustic features with spatial visual representations and decodes the fused features into a target mask, enabling end-to-end prediction without intermediate ASR transcription.
TABLE I: Performance on the Aerial-AudioRES benchmark (clean synthetic set). ASR+Text baselines default to Whisper-Base. Best and second-best results are bold and underlined .
TABLE II: Evaluation on the clean real-human test set. ASR defaults to Whisper-Base.
Fig. 4: Performance comparison of ARES-Net and ASR + RMSIN across various noisy test sets. “WB” and “WL” denote Whisper-Base and Whisper-Large-V3, respectively.
Fig. 5: Visualization of ARES-Net predictions on the noisy synthetic test set.
Dataset (WER ↓ )
Clean
Microphone
Outdoor
Human
Indoor
Synthetic
4.22
5.50
5.72
5.98
5.53
Real-Human
34.82
56.95
41.05
42.98
39.99
TABLE III: Comparison of WER % across different noise scenarios on synthetic and real-human test sets.
Method
Params (M)
Latency (s)
ASR
RES Model
Total
Input
ASR
Inf.
Total
RMSIN
–
240
240
26.30
–
0.03
26.33
WB+RMSIN
74
240
314
8.72
0.26
0.03
9.01
WL+RMSIN
1550
240
1790
8.72
0.76
0.03
9.51
ARES-Net
–
249
249
8.72
–
0.13
8.85
TABLE IV: Comparison of interaction latency and parameter efficiency. “WB” and “WL” denote Whisper-Base and Whisper-Large-V3, respectively. “Inf.” stands for Inference Time.
TABLE VI: Ablation of HuBERT-base training strategies on the clean synthetic test set.
Fig. 6: Robustness against phonetic ambiguity. While phonetically similar words cause ASR transcription errors and RMSIN failure, ARES-Net successfully segments the query image.
TABLE VII: Impact of individual semantic components within instructions. Each configuration evaluates the addition of a single semantic attribute to the base target category.
Audio-based Referring Video Object Segmentation (ARVOS) requires grounding audio queries into pixel-level object masks over time, posing challenges in bridging acoustic signals with spatio-temporal visual representations. In this report, we present VIRST-Audio, a practical framework built upon a pretrained RVOS model integrated with a vision-language architecture. Instead of relying on audio-specific training, we convert input audio into text using an ASR module and perform segmentation using text-based supervision, enabling effective transfer from text-based reasoning to audio-driven scenarios. To improve robustness, we further incorporate an existence-aware gating mechanism that estimates whether the referred target object is present in the video and suppresses predictions when it is absent, reducing hallucinated masks and stabilizing segmentation behavior. We evaluate our approach on the MeViS-Audio track of the 5th PVUW Challenge, where VIRST-Audio achieves 3rd place, demonstrating strong generalization and reliable performance in audio-based referring video segmentation.
Jihwan Hong, Jaeyoung Do
AIDAS Laboratory, 1IPAI, Seoul National University · AIDAS Laboratory, 2ECE, Seoul National University
This report presents an Audio-aware Referring Video Object Segmentation (Ref-VOS) pipeline tailored to the MEVIS_Audio setting, where the referring expression is provided in spoken form rather than as clean text. Compared with a standard Sa2VA-based Ref-VOS pipeline, the proposed system introduces two additional front-end stages: speech transcription and visual existence verification. Specifically, we first employ VibeVoice-ASR to convert long-form spoken input into a structured textual transcript. Since audio-derived queries are inherently noisy and may describe entities that are not visually present in the video, we then introduce an Omni-based judgment module to determine whether the transcribed target can be grounded in the visual content. If the target is judged to be absent, the pipeline terminates early and outputs all-zero masks. Otherwise, the transcript is transformed into a segmentation-oriented prompt and fed into Sa2VA to obtain a coarse mask trajectory over the full video. Importantly, this trajectory is treated as an initial semantic hypothesis rather than a final prediction. On top of it, an agentic refinement layer evaluates query reliability, temporal relevance, anchor quality, and potential error sources, and may invoke SAM3 to improve spatial boundary precision and temporal consistency. The resulting framework explicitly decomposes the MEVIS_Audio task into audio-to-text conversion, visual existence verification, coarse video segmentation, and agent-guided refinement. Such a staged design is substantially more appropriate for audio-conditioned Ref-VOS than directly sending noisy ASR outputs into a segmentation model.
Deshui Miao, Yameng Gu, Chao Yang +3
Pengcheng Laboratory · Harbin Institute of Technology · University of California at Merced
Spoken language offers a natural, hands-free interface for specifying an arbitrary target in dense remote-sensing imagery, yet existing referring remote-sensing image segmentation benchmarks accept only written expressions. To bridge this gap, we introduce \dataset, a spoken-query benchmark derived from RISBench that adds accent- and voice-diverse speech while preserving the original image, mask, and data splits. Its hard evaluation sets combine rotor, wind, and mixed interference with three signal-to-noise levels. We also propose \model, an efficient bilateral network that combines a boundary-preserving visual path with token-preserving speech encoding, kernel linear cross-modal attention, and a resolution refinement head. The design conditions visual features at two scales without materializing a dense speech--visual affinity matrix, then restores fine boundaries using high-resolution visual features. On the clean test split, \model with Swin-Base achieves 62.09% mean intersection over union (mIoU) and 68.22% overall intersection over union (oIoU), outperforming the strongest audio-adapted remote-sensing baseline by 5.38 and 2.08 percentage points, respectively. It retains the best hard-set mIoU at 54.09%. To the best of our knowledge, this is the first benchmark and model study of full-sentence spoken-query referring segmentation for remote-sensing imagery. The code will be made publicly available.