Commanding at the Speed of Sound: Audio-Centric Referring Expression Segmentation for Aerial Scenes
Authors: Ziyun Chen, Yijun Wang, Chuanyi Zhang, Liang Yao, Yuexuan An, Xiang Gu, Wei Zhou, Fan Liu
Organizations: Hohai University, Nanjing, China · Key Laboratory of Water Big Data Technology of Ministry of Water Resources, Hohai University, Nanjing, China
Referring expression segmentation (RES) aims to localize targets at the pixel level guided by human instructions. However, current models predominantly follow a text-centric paradigm that treats discrete text as the primary interaction medium for these human instructions. This assumption is especially limiting in unmanned aerial vehicle (UAV) applications, where rapid response, hands-free operation, and natural communication are often critical. In this paper, we reconstruct UAV referring expression segmentation through an audio-centric perspective. Grounded in information theory, we demonstrate that raw audio provides a higher upper bound for extracting rich semantic features than transcribed text. To achieve this, we construct an audio-driven UAV referring expression segmentation benchmark named Aerial-AudioRES by augmenting VisDrone with synthesized speech, forming "audio-text-image-mask" quadruples. Specifically, we employ an ambiguity filtering mechanism to ensure referential uniqueness, and incorporate eight distinct timbres alongside 4 realistic noises to simulate authentic environments. To drive the paradigm shift from text-centric to audio-centric, we propose the end-to-end Audio-centric Referring Expression Segmentation Network (ARES-Net). Distinct from discrete text-visual matching, our approach learns to establish a direct soft alignment between continuous acoustic manifolds and multi-scale visual representations. Comprehensive experiments show that our method obtains a peak accuracy of 64.23% in oIoU and demonstrates robust generalization on a collected real-human test set. Furthermore, it significantly reduces total interaction latency to one-third of the text-based method. Our code is available at https://github.com/LemonGauss/ARES-Net.
Figures & tables
Fig. 1: Comparison of RES paradigms: (a) Text-Centric: inefficient manual typing; (b) ASR+Text: prone to transcription errors (e.g., “truck” misrecognized as “track”) leading to failed segmentation; (c) Audio-Centric (Ours): robust, end-to-end, efficient perception without intermediate errors.
Fig. 2: Overview of the Aerial-AudioRES construction pipeline. It combines ambiguity-aware target selection and progressive caption generation with automated mask extraction and speech synthesis to construct aligned audio-text-image-mask quadruplets for aerial referring segmentation.
Fig. 3: Overview of the ARES-Net. The model aligns acoustic features with spatial visual representations and decodes the fused features into a target mask, enabling end-to-end prediction without intermediate ASR transcription.
TABLE I: Performance on the Aerial-AudioRES benchmark (clean synthetic set). ASR+Text baselines default to Whisper-Base. Best and second-best results are bold and underlined .
TABLE II: Evaluation on the clean real-human test set. ASR defaults to Whisper-Base.
Fig. 4: Performance comparison of ARES-Net and ASR + RMSIN across various noisy test sets. “WB” and “WL” denote Whisper-Base and Whisper-Large-V3, respectively.
Fig. 5: Visualization of ARES-Net predictions on the noisy synthetic test set.
Dataset (WER ↓ )
Clean
Microphone
Outdoor
Human
Indoor
Synthetic
4.22
5.50
5.72
5.98
5.53
Real-Human
34.82
56.95
41.05
42.98
39.99
TABLE III: Comparison of WER % across different noise scenarios on synthetic and real-human test sets.
Method
Params (M)
Latency (s)
ASR
RES Model
Total
Input
ASR
Inf.
Total
RMSIN
–
240
240
26.30
–
0.03
26.33
WB+RMSIN
74
240
314
8.72
0.26
0.03
9.01
WL+RMSIN
1550
240
1790
8.72
0.76
0.03
9.51
ARES-Net
–
249
249
8.72
–
0.13
8.85
TABLE IV: Comparison of interaction latency and parameter efficiency. “WB” and “WL” denote Whisper-Base and Whisper-Large-V3, respectively. “Inf.” stands for Inference Time.
TABLE VI: Ablation of HuBERT-base training strategies on the clean synthetic test set.
Fig. 6: Robustness against phonetic ambiguity. While phonetically similar words cause ASR transcription errors and RMSIN failure, ARES-Net successfully segments the query image.
TABLE VII: Impact of individual semantic components within instructions. Each configuration evaluates the addition of a single semantic attribute to the base target category.