3rd Place of MeViS-Audio Track of the 5th PVUW: VIRST-Audio

Jihwan Hong; Jaeyoung Do

cs.CVMar 24, 2026

3rd Place of MeViS-Audio Track of the 5th PVUW: VIRST-Audio

Authors: Jihwan Hong, Jaeyoung Do

Organizations: AIDAS Laboratory, 1IPAI, Seoul National University · AIDAS Laboratory, 2ECE, Seoul National University

Abstract

Audio-based Referring Video Object Segmentation (ARVOS) requires grounding audio queries into pixel-level object masks over time, posing challenges in bridging acoustic signals with spatio-temporal visual representations. In this report, we present VIRST-Audio, a practical framework built upon a pretrained RVOS model integrated with a vision-language architecture. Instead of relying on audio-specific training, we convert input audio into text using an ASR module and perform segmentation using text-based supervision, enabling effective transfer from text-based reasoning to audio-driven scenarios. To improve robustness, we further incorporate an existence-aware gating mechanism that estimates whether the referred target object is present in the video and suppresses predictions when it is absent, reducing hallucinated masks and stabilizing segmentation behavior. We evaluate our approach on the MeViS-Audio track of the 5th PVUW Challenge, where VIRST-Audio achieves 3rd place, demonstrating strong generalization and reliable performance in audio-based referring video segmentation.

Figures & tables

Figure 1: Overall architecture of VIRST-Audio.

Rank	Method	$\mathcal{J\&F}$	$\mathcal{J}$	$\mathcal{F}$	N-acc.	T-acc.	Final
1	yahooo	0.67	0.64	0.70	0.89	0.98	0.85
2	wangzhiyu918	0.64	0.61	0.67	0.83	0.95	0.81
3	Ours	0.54	0.52	0.56	0.70	0.82	0.68
4	vvv666	0.47	0.44	0.50	0.12	0.98	0.52
5	liyiying	0.48	0.45	0.50	0.09	0.97	0.51
6	eronguyen	0.24	0.23	0.26	0.18	0.91	0.45

Table 1: Results on the PVUW 2026 MeViS-Audio track. VIRST-Audio ranks 3rd among 13 teams. Our results are bolded .

Method	$\mathcal{J\&F}$	$\mathcal{J}$	$\mathcal{F}$	N-acc.	T-acc.	Final
VIRST (baseline)	0.49	0.47	0.51	0.62	0.84	0.65
w/o gating	0.52	0.49	0.54	0.58	0.89	0.66
gating ( $\tau=0.8$ )	0.54	0.52	0.56	0.70	0.82	0.68
gating ( $\tau=0.9$ )	0.53	0.51	0.55	0.71	0.78	0.67

Table 2: Effect of existence-aware gating with different thresholds

\tau

. Final score is the average of

\mathcal{J\&F}

, N-acc., and T-acc. Best results are in bold .

Figure 2: Qualitative results of VIRST-Audio on the MeViS-Audio test set.

Referring Video Object Segmentation

Referring video object segmentation grounds language descriptions in video frames and produces temporally consistent masks for the described objects.

See all 42 papers

APRVOS: 1st Place Winner of 5th PVUW MeViS-Audio Track
Apr 20, 2026Deshui Miao, Yameng Gu, Chao Yang +3Audio-Visual Understanding Video Object Segmentation
Pengcheng Laboratory · Harbin Institute of Technology · University of California at Merced
2nd of the 5th PVUW MeViS-Audio Track: ASR-SaSaSa2VA
Apr 27, 2026Zhiyu Wang, Xudong Kang, Shutao LiVideo Object Segmentation
1Hunan University
MLLM-Assisted Audio VOS: A 3rd Place Report for the MeViS-Audio Track, 8th LSVOS Challenge
Aug 24, 2026Liangtao Shi, Jinxia Xie, Xiantao Hu +1Video Object Segmentation
Hefei University of Technology · Nanjing University of Science and Technology · Hunan Police College

Apr 20, 2026J/K move · Enter open · S save

APRVOS: 1st Place Winner of 5th PVUW MeViS-Audio Track

Deshui Miao, Yameng Gu, Chao Yang, Xin Li+2

Pengcheng Laboratory · Harbin Institute of Technology · University of California at Merced

Audio-Visual Understanding Video Object Segmentation Referring Video Object Segmentation

Open paper AskPDF arXiv

This report presents an Audio-aware Referring Video Object Segmentation (Ref-VOS) pipeline tailored to the MEVIS_Audio setting, where the referring expression is provided in spoken form rather than as clean text. Compared with a standard Sa2VA-based Ref-VOS pipeline, the proposed system introduces two additional front-end stages: speech transcription and visual existence verification. Specifically, we first employ VibeVoice-ASR to convert long-form spoken input into a structured textual transcript. Since audio-derived queries are inherently noisy and may describe entities that are not visually present in the video, we then introduce an Omni-based judgment module to determine whether the transcribed target can be grounded in the visual content. If the target is judged to be absent, the pipeline terminates early and outputs all-zero masks. Otherwise, the transcript is transformed into a segmentation-oriented prompt and fed into Sa2VA to obtain a coarse mask trajectory over the full video. Importantly, this trajectory is treated as an initial semantic hypothesis rather than a final prediction. On top of it, an agentic refinement layer evaluates query reliability, temporal relevance, anchor quality, and potential error sources, and may invoke SAM3 to improve spatial boundary precision and temporal consistency. The resulting framework explicitly decomposes the MEVIS_Audio task into audio-to-text conversion, visual existence verification, coarse video segmentation, and agent-guided refinement. Such a staged design is substantially more appropriate for audio-conditioned Ref-VOS than directly sending noisy ASR outputs into a segmentation model.

Abstract

Referring Video Object Segmentation

Explore similar work

APRVOS: 1st Place Winner of 5th PVUW MeViS-Audio Track

2nd of the 5th PVUW MeViS-Audio Track: ASR-SaSaSa2VA

MLLM-Assisted Audio VOS: A 3rd Place Report for the MeViS-Audio Track, 8th LSVOS Challenge