Shift-Aware Identity-Guided Latent Refinement for Referring Audio--Visual Segmentation

Kun Li; Sami Sebastian Brandt; Michael Ying Yang

cs.CVOct 8, 2026

Shift-Aware Identity-Guided Latent Refinement for Referring Audio--Visual Segmentation

Authors: Kun Li, Sami Sebastian Brandt, Michael Ying Yang

Organizations: University of Liverpool, UK · IT University of Copenhagen, DK · University of Bath, UK

Abstract

Referring Audio--Visual Segmentation (Ref-AVS) is an emergent multimodal task requiring the precise identification and segmentation of a specific object based on a natural language expression grounded in both visual and auditory cues. Unlike traditional AVS, which segments generic sound sources, or referring video object segmentation (Ref-VOS), which relies solely on visual--textual grounding, Ref-AVS necessitates fine-grained tri-modal interaction across vision, audio, and language. However, existing methods often fail to maintain consistency when subjected to target shift, a phenomenon where the model erroneously transfers its focus between objects due to transient occlusions, acoustic fading, or identity ambiguities. To address this, we propose a novel Shift-Aware Identity-guided Latent (SAIL) refinement framework. It introduces a dual-token strategy that decouples frame-level dynamics from a global semantic anchor to explicitly model target-shift cues. We design a shift-aware refinement module that rectifies drift by aligning tokens with domain-specific evidence. Finally, we leverage a decoder that integrates the refined embeddings with a continuous memory update to maintain spatio-temporal coherence and identity consistency throughout the video sequence. Extensive experiments conducted on two Ref-AVS benchmarks demonstrate the effectiveness of our proposed method, significantly outperforming existing AVS, Ref-VOS, and Ref-AVS approaches. The code will be released after publication.

Figures & tables

Panel of Figure 1 — Figure 1: Illustration of Ref-AVS and target shift problems. (a) Given a tri-modal input (audio, video, and expression), a Ref-AVS model performs sequential mask prediction for the referred object. (b) Visual target shift caused by transient occlusion or overshadowing ( e.g. , shifting from the accordion to a guitar). (c) Auditory target shift caused by intermittent sound fading or pauses ( e.g. , shifting from the cello to a piano).

Figure 2: Overview of the proposed SAIL framework. Our MLLM with LoRA [ 11 ] reasons over the unified tri-modal tokens, generating frame-level [seg] tokens for local dynamics and a global [query] token as a semantic anchor. The shift-aware refinement module estimates and rectifies the latent shift by comparing the domain-specific evidence with the anchor’s semantics. The mask decoder utilizes the refined embeddings and a memory mechanism to generate consistent mask predictions for the entire video sequence. For brevity, projection units before the decoder are omitted.

Method	Task	Model	Seen $\uparrow$		Unseen $\uparrow$		Mix $\uparrow$		Null $\downarrow$
	Task	Model	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{S}$
AVSBench [ 53 ] eccv22	AVS	PVT-v2	23.2	51.1	32.4	54.7	27.8	52.9	0.208
AVSegFormer [ 8 ] aaai24		PVT-v2	33.5	47.0	36.1	50.1	34.8	48.6	0.171
GAVS [ 42 ] aaai24		SAM	28.9	49.8	29.8	49.7	29.4	49.8	0.190
SAMA [ 25 ] wacv24		SAM	39.2	56.2	47.5	56.6	43.4	56.4	0.130
ReferFormer [ 45 ] cvpr22	Ref-VOS	Video-Swin	31.3	50.1	30.4	48.8	30.9	49.5	0.176

Table 1: Comparison with existing methods on Ref-AVSBench [ 43 ] , where Mix represents the mean performance of Seen and Unseen subsets.

\dagger

denotes methods requiring additional data for instruction tuning;

\ddagger

indicates models that further necessitate external components in multiple stages. The best results are bold while the second best are underlined .

Figure 3: Qualitative results of SAIL and baseline from Ref-AVSBench [ 43 ] .

Method	Seen $\uparrow$			Unseen $\uparrow$			Mix $\uparrow$
	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}$ & $\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}$ & $\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}$ & $\mathcal{F}$
EEMC [ 43 ] eccv24	22.6	39.9	31.2	38.7	56.9	47.8	30.6	48.4	39.5
Grounded-SAM2 [ 34 ] arxiv24	21.4	30.1	25.7	44.9	53.4	49.1	33.2	41.7	37.4
Crab [ 6 ] cvpr25	19.6	32.1	25.9	36.9	52.0	44.4	28.2	42.1	35.2
TGS-Agent [ 55 ] aaai26	42.7	52.3	47.5	68.3	77.0	72.7	55.5	64.7	60.1
AURORA [ 29 ] $\ast$ aaai26	50.5	61.1	55.8	55.0	64.4	59.7	52.8	62.8	57.8

Table 2: Comparison on

\mathrm{R^{2}}

-AVSBench [ 55 ] . Results for existing methods are sourced from either TGS-Agent or our re-implementations (in

\ast

). Note that TGS-Agent employed external Grounding-DINO [ 26 ] for explicit object localization.

\mathcal{J}

\mathcal{F}

represents the average of

\mathcal{J}

and

\mathcal{F}

Table 6

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Seen $\uparrow$			Unseen $\uparrow$			Mix $\uparrow$
	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}$ & $\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}$ & $\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}$ & $\mathcal{F}$
EEMC [ 43 ]	17.4	30.5	24.0	34.2	52.6	43.4	25.8	41.6	33.7
Crab [ 6 ]	25.1	40.7	32.9	32.3	50.5	41.4	28.7	45.6	37.2
TGS-Agent [ 55 ]	37.5	47.9	42.7	56.7	64.2	60.5	47.1	56.1	51.6
AURORA [ 29 ]	40.7	50.0	45.4	52.6	61.9	57.3	46.7	56.0	51.4
SAIL (ours)	53.1	62.4	57.8	65.7	72.8	69.3	59.4	67.6	63.5

Appendix

Table 5: Comparison with existing methods on the selective samples for target shift from Ref-AVSBench [ 43 ] .

Fusion	Residual	Gating weights	Seen $\uparrow$			Unseen $\uparrow$			Mix $\uparrow$			Null $\downarrow$
Fusion	Residual	Gating weights	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}$ & $\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}$ & $\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}$ & $\mathcal{F}$	$\mathcal{S}$
✓			60.9	70.5	65.7	65.4	72.7	69.1	63.2	71.6	67.4	0.027
	✓		66.3	75.4	70.9	70.3	77.8	74.1	68.3	76.6	72.5	0.015
	✓	✓	68.4	77.5	73.0	72.0	79.5	75.8	70.2	78.5	74.4	0.010

Appendix

Table 6: Ablation study on the shift-aware refinement module.

Method	Seen $\uparrow$			Unseen $\uparrow$			Mix $\uparrow$			Null $\downarrow$
	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}$ & $\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}$ & $\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}$ & $\mathcal{F}$	$\mathcal{S}$
LLaMA2-7B [ 37 ]	65.1	73.7	69.4	69.2	76.7	73.0	67.2	75.2	71.2	0.028
InternVL2-8B [ 2 ]	67.6	76.8	72.2	70.6	77.4	74.0	69.1	77.1	73.1	0.014
Chat-UniVi-7B [ 13 ]	68.4	77.5	73.0	72.0	79.5	75.8	70.2	78.5	74.4	0.010

Appendix

Table 7: Ablation study on LMMs. We replaced the MLLM in our SAIL with LLaMA2 [ 37 ] and InternVL2 [ 2 ] , respectively.

Method	Seen $\uparrow$			Unseen $\uparrow$			Mix $\uparrow$			Null $\downarrow$
	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}$ & $\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}$ & $\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}$ & $\mathcal{F}$	$\mathcal{S}$
$\alpha=0.1$	68.4	77.5	73.0	72.0	79.5	75.8	70.2	78.5	74.4	0.010
$\alpha=0.2$	67.2	75.0	71.1	70.8	78.2	74.5	69.0	76.6	72.8	0.011
$\alpha=0.5$	66.7	74.3	70.5	70.1	77.6	73.9	68.4	76.0	72.2	0.012
$\alpha=1.0$	63.4	72.4	67.9	68.5	76.7	72.6	66.0	74.6	70.3	0.017

Appendix

Table 8: Ablation study on the fusion coefficient

\alpha

in the anchor alignment before the mask decoder.

Method	Seen $\uparrow$			Unseen $\uparrow$			Mix $\uparrow$			Null $\downarrow$
	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}$ & $\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}$ & $\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$	$\mathcal{J}$ & $\mathcal{F}$	$\mathcal{S}$
+ M2F	62.8	71.9	67.4	65.8	73.2	69.5	64.3	72.6	68.5	0.020
+ SAM	64.3	73.8	69.1	67.7	75.0	71.4	66.0	74.4	70.2	0.015
+ QP + SAM	66.9	76.1	71.5	71.0	78.5	74.8	69.0	77.3	73.2	0.010
+ SAM2	65.0	75.2	70.1	70.4	78.0	74.2	67.7	76.6	72.2	0.013
SAIL(ours)	68.4	77.5	73.0	72.0	79.5	75.8	70.2	78.5	74.4	0.010

Appendix

Table 9: Ablation study on the mask decoder. QP denotes the query propagation operation in OISA-1B [ 50 ] .

Panel of Figure 4 — Figure 4: Additional qualitative results from EEMC [ 43 ] , TGS-Agent [ 55 ] , and our proposed SAIL on the Seen split from the Ref-AVSBench [ 43 ] dataset.

Panel of Figure 5 — Figure 5: Additional qualitative results from EEMC [ 43 ] , TGS-Agent [ 55 ] , and our proposed SAIL on the Unseen split from the Ref-AVSBench [ 43 ] dataset.

Panel of Figure 6 — Figure 6: Qualitative results on samples from the $\mathrm{R^{2}}$ -AVSBench [ 55 ] dataset. The two examples are from the Seen and Unseen splits, respectively.

Panel of Figure 7 — Figure 7: Visualization of failure cases for the proposed SAIL on the Ref-AVSBench [ 43 ] dataset.

Method	Latency (ms)	FPS	Memory (GB)
SAM2-LOVE	54	18.5	12.4
Crab	115	8.7	18.2
SAIL	128	7.8	22.5

Appendix

Table 10: Latency and memory usage of different models.

Audio-Visual Understanding

Audio-visual understanding studies how AI combines sound and visual signals to recognize events, speech, people, and scenes over time.

See all 155 papers

PRIMED: Adaptive Modality Suppression for Referring Audio-Visual Segmentation via Biased Competition
May 8, 2026Yuchen He, Jing ZhangAudio-Visual Understanding Referring Expression Segmentation
School of Information Science and Engineering, East China University of Science and Technology, Shanghai, China
APRVOS: 1st Place Winner of 5th PVUW MeViS-Audio Track
Apr 20, 2026Deshui Miao, Yameng Gu, Chao Yang +3Audio-Visual Understanding Video Object Segmentation
Pengcheng Laboratory · Harbin Institute of Technology · University of California at Merced
3rd Place of MeViS-Audio Track of the 5th PVUW: VIRST-Audio
Mar 24, 2026Jihwan Hong, Jaeyoung DoReferring Video Object Segmentation
AIDAS Laboratory, 1IPAI, Seoul National University · AIDAS Laboratory, 2ECE, Seoul National University

May 8, 2026J/K move · Enter open · S save

PRIMED: Adaptive Modality Suppression for Referring Audio-Visual Segmentation via Biased Competition

Yuchen He, Jing Zhang

School of Information Science and Engineering, East China University of Science and Technology, Shanghai, China

Audio-Visual Understanding Referring Expression Segmentation Cross-Modal Attention Multimodal Fusion

Open paper AskPDF arXiv

Referring Audio-Visual Segmentation (Ref-AVS) seeks to localize and segment target objects in video frames based on visual, auditory, and textual referring cues. The task is challenging because the relevance of different modalities varies across referring expressions and scenes, while existing methods typically treat multimodal cues as homogeneous inputs for fusion, prompting, or reasoning, making them vulnerable to irrelevant or misleading modalities. To address this problem, we propose PRIMED, inspired by the biased competition theory in cognitive neuroscience, which explicitly models both visual perception and language-driven prior modulation, and enables more accurate Ref-AVS by adaptive modality suppression. Specifically, a Modality Prior Decoder first estimates whether the referring expression relies primarily on audio, vision, or their joint interaction, generating a modality prior to adaptively guide high-level attention. A Token Distiller further extracts compact global visual tokens from high-level features and shares them across Competition-aware Cross-modal Fusion modules to provide hierarchical global context. Additionally, we introduce a Spatial-Aware Semantic Alignment loss to further enhance foreground-background discrimination through contrastive learning. Extensive experiments on the Ref-AVS benchmark demonstrate that PRIMED achieves state-of-the-art overall performance.

Abstract

Audio-Visual Understanding

Explore similar work

PRIMED: Adaptive Modality Suppression for Referring Audio-Visual Segmentation via Biased Competition

APRVOS: 1st Place Winner of 5th PVUW MeViS-Audio Track

3rd Place of MeViS-Audio Track of the 5th PVUW: VIRST-Audio