Audio-based Referring Video Object Segmentation (ARVOS) requires grounding audio queries into pixel-level object masks over time, posing challenges in bridging acoustic signals with spatio-temporal visual representations. In this report, we present VIRST-Audio, a practical framework built upon a pretrained RVOS model integrated with a vision-language architecture. Instead of relying on audio-specific training, we convert input audio into text using an ASR module and perform segmentation using text-based supervision, enabling effective transfer from text-based reasoning to audio-driven scenarios. To improve robustness, we further incorporate an existence-aware gating mechanism that estimates whether the referred target object is present in the video and suppresses predictions when it is absent, reducing hallucinated masks and stabilizing segmentation behavior. We evaluate our approach on the MeViS-Audio track of the 5th PVUW Challenge, where VIRST-Audio achieves 3rd place, demonstrating strong generalization and reliable performance in audio-based referring video segmentation.
Figures & tables
Figure 1: Overall architecture of VIRST-Audio.
Rank
Method
J&F
J
F
N-acc.
T-acc.
Final
1
yahooo
0.67
0.64
0.70
0.89
0.98
0.85
2
wangzhiyu918
0.64
0.61
0.67
0.83
0.95
0.81
3
Ours
0.54
0.52
0.56
0.70
0.82
0.68
4
vvv666
0.47
0.44
0.50
0.12
0.98
0.52
5
liyiying
0.48
0.45
0.50
0.09
0.97
0.51
6
eronguyen
0.24
0.23
0.26
0.18
0.91
0.45
Table 1: Results on the PVUW 2026 MeViS-Audio track. VIRST-Audio ranks 3rd among 13 teams. Our results are bolded .
Method
J&F
J
F
N-acc.
T-acc.
Final
VIRST (baseline)
0.49
0.47
0.51
0.62
0.84
0.65
w/o gating
0.52
0.49
0.54
0.58
0.89
0.66
gating ( τ=0.8 )
0.54
0.52
0.56
0.70
0.82
0.68
gating ( τ=0.9 )
0.53
0.51
0.55
0.71
0.78
0.67
Table 2: Effect of existence-aware gating with different thresholds τ . Final score is the average of J&F , N-acc., and T-acc. Best results are in bold .
Figure 2: Qualitative results of VIRST-Audio on the MeViS-Audio test set.
This report presents an Audio-aware Referring Video Object Segmentation (Ref-VOS) pipeline tailored to the MEVIS_Audio setting, where the referring expression is provided in spoken form rather than as clean text. Compared with a standard Sa2VA-based Ref-VOS pipeline, the proposed system introduces two additional front-end stages: speech transcription and visual existence verification. Specifically, we first employ VibeVoice-ASR to convert long-form spoken input into a structured textual transcript. Since audio-derived queries are inherently noisy and may describe entities that are not visually present in the video, we then introduce an Omni-based judgment module to determine whether the transcribed target can be grounded in the visual content. If the target is judged to be absent, the pipeline terminates early and outputs all-zero masks. Otherwise, the transcript is transformed into a segmentation-oriented prompt and fed into Sa2VA to obtain a coarse mask trajectory over the full video. Importantly, this trajectory is treated as an initial semantic hypothesis rather than a final prediction. On top of it, an agentic refinement layer evaluates query reliability, temporal relevance, anchor quality, and potential error sources, and may invoke SAM3 to improve spatial boundary precision and temporal consistency. The resulting framework explicitly decomposes the MEVIS_Audio task into audio-to-text conversion, visual existence verification, coarse video segmentation, and agent-guided refinement. Such a staged design is substantially more appropriate for audio-conditioned Ref-VOS than directly sending noisy ASR outputs into a segmentation model.
Deshui Miao, Yameng Gu, Chao Yang +3
Pengcheng Laboratory · Harbin Institute of Technology · University of California at Merced
Audio-based video object segmentation aims to locate and segment objects in videos conditioned on audio cues, requiring precise understanding of both appearance and motion. Recent audio-driven video segmentation methods extend MLLMs by fusing audio and visual features for end-to-end localization. Despite their promise, these approaches are computationally intensive, struggle with aligning temporal audio cues to dynamic video content, and depend on large paired audio-video datasets. To address these challenges, we present ASR-SaSaSa2VA, a resource-efficient framework for audio-guided video segmentation. The key idea is to convert audio inputs into textual motion descriptions via automatic speech recognition (ASR) models and then leverage pre-trained text-based referring video segmentation models (e.g., SaSaSa2VA) for pixel-level predictions. To further enhance robustness, we incorporate a no-target expression detection module, implemented by a fine-tuned audio-based MLLM, which filters out audio clips that do not refer to any target object. This design allows the system to exploit strong pre-trained models while effectively handling ambiguous or irrelevant audio inputs. Our approach achieves a final score of 80.7 in the 5th PVUW Challenge (MeViS-v2-Audio track), earning the second-place ranking.
In this technical report, we present a training-free framework for audio-guided video object segmentation, which integrates Multimodal Large Language Models (MLLMs) with SAM-based segmentation models. We decompose the task into several stages and identify suitable foundation models for each stage. Without introducing additional model training or task-specific fine-tuning, our approach leverages the strong multimodal reasoning capabilities of MLLMs to model text-visual correspondence and employs SAM-based models for accurate object mask generation. The proposed framework demonstrates the effectiveness of leveraging foundation models for audio-guided video segmentation and achieves competitive performance in the MeViS-Audio Track of the 8th LSVOS Challenge.
Liangtao Shi, Jinxia Xie, Xiantao Hu +1
Hefei University of Technology · Nanjing University of Science and Technology · Hunan Police College