TALENT: Target-aware Efficient Tuning for Referring Image Segmentation

Shuo Jin; Siyue Yu; Bingfeng Zhang; Chao Yao; Meiqin Liu; Jimin Xiao

cs.CVApr 1, 2026

TALENT: Target-aware Efficient Tuning for Referring Image Segmentation

Authors: Shuo Jin, Siyue Yu, Bingfeng Zhang, Chao Yao, Meiqin Liu, Jimin Xiao

Organizations: XJTLU · University of Liverpool · China University of Petroleum (East China) · University of Science and Technology Beijing · Beijing Jiaotong University

Abstract

Referring image segmentation aims to segment specific targets based on a natural text expression. Recently, parameter-efficient tuning (PET) has emerged as a promising paradigm. However, existing PET-based methods often suffer from the fact that visual features can't emphasize the text-referred target instance but activate co-category yet unrelated objects. We analyze and quantify this problem, terming it the non-target activation' (NTA) issue. To address this, we propose a novel framework, TALENT, which utilizes target-aware efficient tuning for PET-based RIS. Specifically, we first propose a Rectified Cost Aggregator (RCA) to efficiently aggregate text-referred features. Then, to calibrate NTA' into accurate target activation, we adopt a Target-aware Learning Mechanism (TLM), including contextual pairwise consistency learning and target-centric contrastive learning. The former uses the sentence-level text feature to achieve a holistic understanding of the referent and constructs a text-referred affinity map to optimize the semantic association of visual features. The latter further enhances target localization to discover the distinct instance while suppressing associations with other unrelated ones. The two objectives work in concert and address `NTA' effectively. Extensive evaluations show that TALENT outperforms existing methods across various metrics (e.g., 2.5% mIoU gains on G-Ref val set). Our codes will be released at: https://github.com/Kimsure/TALENT.

Figures & tables

Figure 1: Visual feature activation and segmentation maps. (a) Text descriptions. (b) Visual-text fusion in DETRIS [ 15 ] , which activates co-category foreground objects. (c) Our TALENT, which emphasizes the text-referred target instance. (d) Segmentation result of the ground truth (GT). (e)-(f) Corresponding segmentation results of DETRIS [ 15 ] and our TALENT.

Figure 2: Comparison of ‘NTA’ influence quantization among various methods on the selected subset of different benchmarks.

Figure 3: Framework pipeline of our TALENT. It contains four main modules: a frozen backbone building upon DINOv2-Reg and CLIP to encode the image and text, a rectified cost aggregator for vision-language interaction, a target-aware learning mechanism to strengthen feature representation, and a transformer decoder for final segmentation.

Methods	PET	Metirc	RefCOCO			RefCOCO+			G-Ref
Methods	PET	Metirc	val	testA	testB	val	testA	testB	val(u)	test(u)	val(g)
LAVT † [ 59 ]	✘	oIoU	72.7	75.8	68.8	62.1	68.4	55.1	61.2	62.1	60.5
DMMI [ 13 ]	✘		74.1	77.1	70.2	64.0	69.7	57.0	63.5	64.2	61.2
ReLA [ 28 ]	✘		73.8	76.5	70.2	66.0	71.0	57.7	65.0	66.0	62.7
CG-former [ 49 ]	✘		74.8	77.3	70.6	64.5	71.0	57.1	64.7	65.1	62.5
LISA-Vicuna-13B [ 25 ]	✘		71.7	74.7	68.1	59.4	64.2	52.9	65.2	66.1	-

Table 1: Quantitative comparison of our method against other PFT-based and PET-based methods on different benchmarks, evaluated using the oIoU and mIoU metrics. † denotes the re-implemented results, while others are directly sourced from the reported results. u: The UMD partition. g: The Google partition. The best results are marked with bold .

Methods	[email protected]	[email protected]	[email protected]
ETRIS [ 55 ]	83.4	72.7	17.4
ETOG [ 61 ]	83.2	73.0	26.2
CG-former [ 49 ]	87.2	78.7	38.8
Prompt-RIS [ 44 ]	85.6	76.9	26.2
DETRIS [ 15 ]	87.9	80.2	27.5
TALENT (Ours)	88.6	82.3	40.1

Table 2: Quantitative comparison using the Precision@X metric on the val-test set of RefCOCO.

Figure 4: Qualitative comparison. We compare our TALENT with ETRIS [ 55 ] , ETOG [ 61 ] , and DETRIS [ 15 ] . It’s observed that TALENT can accurately localize the target and generate more precise segmentation results.

Figure 5: Visualization of feature activation maps. We compare TALENT with the existing SOTA method DETRIS [ 15 ] .

Methods	Params. (backbone)	Params. (other modules)	Params. (overall)	mIoU (Avg.)
CRIS [ 54 ]	57.31M	103.94M	161.25M	63.8
LoRA † [ 11 ]	0.03M	23.98M	24.01M	60.4
Compacter † [ 21 ]	0.19M	23.98M	24.17M	61.3
ETRIS † [ 55 ]	1.94M	23.98M	25.92M	62.8
BarLeRIa [ 51 ]	2.21M	23.98M	26.19M	66.5
DETRIS † [ 15 ]	2.71M	22.84M	26.69M	70.4

Table 3: Comparison of parameters and average segment performance on three benchmarks. † indicates the results are calculated from the official implementation of DETRIS [ 17 ] and ETRIS [ 55 ] .

RCA	CPCL	TCCL	RefCOCO-mIoU(%)			AVG(%)	NTA-IoU(%)
RCA	CPCL	TCCL	val	testA	testB	AVG(%)	NTA-IoU(%)
✘	✘	✘	74.9	77.1	71.9	74.6	9.9
✔	✘	✘	76.0	77.9	73.3	75.7	8.2
✔	✔	✘	77.2	78.7	74.1	76.7	4.5
✔	✘	✔	76.8	78.4	73.8	76.3	3.7
✔	✔	✔	77.8	79.4	74.8	77.3	2.1

Table 4: Ablation results on each proposed module to validate the performance impact using the mIoU and NTA-IoU metrics.

Settings	Architecture	RefCOCO-mIoU(%)
Settings	Architecture	val	testA	testB
#1	$\nu_{i+1}(f_{vt}^{i})$	71.8	76.0	70.1
#2	$\nu_{i+1}(f_{vt}^{i})*\varphi$	71.1	74.5	68.9
#3	$cat(\nu_{i+1}(f_{v}^{i}),f_{vt}^{i})$	76.1	77.9	73.2
#4	$cat(\nu_{i+1}(f_{v}^{i}),f_{vt}^{i}*\varphi)$	76.6	78.8	73.9
#5	$\nu_{i+1}(f_{v}^{i})+f_{vt}^{i}$	77.1	78.7	74.1

Table 5: Ablation results on different feature aggregation architectures of Eq. 4 .

cat(\cdot)

means to concatenate the two input features and then use an MLP for channel reduction.

$\lambda_{cpcl}$	0.3	0.2	0.2	0.1	0.1	0.1	0.1	0.0
$\lambda_{tccl}$	0.1	0.2	0.1	0.3	0.2	0.1	0.0	0.1
val	76.9	77.2	77.5	77.4	77.6	77.8	77.2	76.8
testA	78.5	78.6	79.1	78.9	79.1	79.4	78.7	78.4
testB	74.3	74.3	74.4	74.2	74.6	74.8	74.1	73.8

Table 6: Ablation results on each loss scaling coefficient.

V-L interaction	TLM		RefCOCO			Avg
V-L interaction	CPCL	TCCL	val	testA	testB	Avg
✘	✘	✘	74.9	77.1	71.9	74.6
RCA	✘	✘	76.0	77.9	73.3	75.7
RCA	✔	✔	77.8	79.4	74.8	77.3
cross-attention	✘	✘	75.4	77.3	72.4	75.0
cross-attention	✔	✔	77.3	78.9	74.1	76.8

Table 7: Ablation experiments on applying CPCL and TCCL to different visual-text interaction mechanisms.

Figure 6: Ablation results of the evaluation of the loss scaling coefficient for our CPCL and TCCL. Each colored line corresponds to a specific CPCL coefficient.

Figure 7: Visualization of segmentation results. We compare our TALENT with existing PET-based methods. It’s observed that TALENT can generate more precise segmentation maps.

Figure 8: Visualization of feature activation maps. We compare TALENT with the SOTA PET-based method DETRIS [ 15 ] .

Explore similar work

AMLRIS: Alignment-aware Masked Learning for Referring Image Segmentation
Feb 26, 2026Tongfei Chen, Shuo Yang, Yuguang Yang +7Referring Image Segmentation Vision-Language Alignment
School of Artificial Intelligence, Beihang University, China · School of Electronic Information Engineering, Beihang University, China · AI Research, Qihoo 360, China +3
Progressive Prompt-Guided Cross-Modal Reasoning for Referring Image Segmentation
Mar 30, 2026Jiachen Li, Hongyun Wang, Jinyu Xu +5Referring Image Segmentation Promptable Image Segmentation
School of Computer Science and Artificial Intelligence, Wuhan University of Technology, Wuhan, China · School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan, China · University of Electronic Science and Technology of China, Chengdu, China +1
InstanceBench: Diagnosing Referential Reasoning and Target Identity in Referring Expression Segmentation
Oct 7, 2026Yuchen Li, Shaoyang Zhou, Yiran Wang +5Referring Expression Segmentation
The University of Sydney · The ATLAS Institute · Shanghai Artificial Intelligence Laboratory

Feb 26, 2026J/K move · Enter open · S save

AMLRIS: Alignment-aware Masked Learning for Referring Image Segmentation

Tongfei Chen, Shuo Yang, Yuguang Yang, Linlin Yang+6

School of Artificial Intelligence, Beihang University, China · School of Electronic Information Engineering, Beihang University, China · AI Research, Qihoo 360, China+3

Referring Image Segmentation Vision-Language Alignment

Open paper AskPDF arXiv

Referring Image Segmentation (RIS) aims to segment the object in an image uniquely referred to by a natural language expression. However, RIS training often contains hard-to-align and instance-specific visual signals; optimizing on such pixels injects misleading gradients and drives the model in the wrong direction. By explicitly estimating pixel-level vision-language alignment, the learner can suppress low-alignment regions, concentrate on reliable cues, and acquire more generalizable alignment features. In this paper, we propose Alignment-Aware Masked Learning (AML), a simple yet effective training strategy that quantifies region-referent alignment (PMME) and filters out unreliable pixels during optimization (AFM). Specifically, each sample first computes a similarity map between visual and textual features, and then masks out pixels falling below an adaptive similarity threshold, thereby excluding poorly aligned regions from the training process. AML does not require architectural changes and incurs no inference overhead, directing attention to the areas aligned with the textual description. Experiments on the RefCOCO (vanilla/+/g) datasets show that AML achieves state-of-the-art results across all 8 splits, and beyond improving RIS performance, AML also enhances the model's robustness to diverse descriptions and scenarios. Code is available at https://github.com/pipashu1/AMLRIS.

Abstract

Explore similar work

AMLRIS: Alignment-aware Masked Learning for Referring Image Segmentation

Progressive Prompt-Guided Cross-Modal Reasoning for Referring Image Segmentation

InstanceBench: Diagnosing Referential Reasoning and Target Identity in Referring Expression Segmentation