Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perception

Junjie Wang; Keyu Chen; Yulin Li; Bin Chen; Hengshuang Zhao; Xiaojuan Qi; Zhuotao Tian

Fig. 1: Comparison of the attention patterns of CLIP and VFM across different layers. The attention weights of CLIP and VFM are collected at different layers, averaged over heads, and upsampled to the original image resolution. The first row visualizes attention weights for all tokens, while the remaining rows use the [CLS] token and selected image tokens as queries. Red dots indicate selected query image tokens. The figure highlights CLIP’s deep-layer proxy-token behavior and VFM’s consistent dense correlations, which are discussed in Sec. 3.2 .

Fig. 2: (a) Comparison between DeCLIP and state-of-the-art methods across six OV dense perception tasks. These tasks include OV detection and semantic segmentation (OVD, OVSS), training-free OV semantic segmentation (TFOVSS), OV 3D and video instance segmentation (OV3DIS, OVVIS), and OV 6D pose estimation (OV6DPE). Experiments demonstrate the effectiveness of DeCLIP as a foundation model for OV dense perception. (b) Quantitative and qualitative comparisons between DeCLIP and CLIP. Compared to CLIP, DeCLIP’s image tokens attend to semantically relevant regions of query tokens, significantly improving dense perception performance. The query tokens are highlighted by red dots.

Fig. 3: Comparison between DeCLIP and existing methods. (a) Methods that combine CLIP with VFMs typically integrate CLIP into SAM and transform it into an OV segmenter, which often requires costly dense annotations to train adapters. (b) Pre-fine-tuning methods adapt CLIP by aligning CLIP’s local features with the CLS tokens of corresponding sub-images. However, these methods still fail to apply to image segmentation tasks. (c) Our DeCLIP enhances dense features’ local discriminability and spatial consistency via the content-context decoupled learning, enabling a unified OV dense perception foundation model.

Fig. 4: Loss and gradient analysis. (a) Training loss curves indicating superior stability and convergence for decoupled learning compared to the integrated approach. (b) Gradient cosine similarity between regional vision-language alignment and VFM distillation losses, which exhibit conflicting gradients for approximately 51.2% of the training duration.

Fig. 5: Illustration of the DeCLIP framework. (a) Decoupled Attention: The final attention module of CLIP is decoupled into context features (to enhance spatial consistency) and content features (to enhance local vision-language alignment). (b) Context Distillation: Enhancing context features by leveraging semantic affinity maps from the VFM. (c) SD-Guided Semantic Completion (SD-GSC): The self-attention maps of SD are leveraged to enhance the semantic affinity map of VFMs and further optimize the distillation signals for the context features. (d) Content Distillation: Enhancing content features by aligning region representations with the corresponding [CLS] tokens. (e) Region Correlation Constraint (RCC): Using VFM’s region correlations to prevent the collapse of CLIP’s dense correlations during content distillation.

Fig. 6: Visualization of VFM semantic affinity maps before and after semantic completion. The cosine similarity and attention between the query token (red dot) and the remaining image tokens are visualized. SD attention maps are highly sensitive to high-frequency features, especially object contours (second row). K-means clustering of SD attention maps demonstrates notable object integrity (third row). Before completion (fourth row), the VFM semantic affinity map is blurred at boundaries and contains internal gaps. After completion (last row), the affinity map exhibits improved object integrity.

Fig. 7: Comparison of feature semantic consistency. “V-L align (Image)” refers to the vanilla CLIP, while “V-L align (Region)” denotes existing fine-tuning approaches [ 9 , 10 ] . By performing K-means clustering on the cosine similarity maps of CLIP features, we observe that region-level fine-tuning may further weaken the pixel-level perception capability of CLIP.

Fig. 8: Qualitative comparison of OV 3D instance segmentation results between CLIP and DeCLIP on the ScanNet200 [ 62 ] dataset. The baseline method is Open3DIS [ 24 ] .

Fig. 9: Qualitative comparison of OV video instance segmentation results between CLIP and DeCLIP on the YTVIS21 [ 63 ] dataset. The baseline method is CLIP-VIS [ 26 ] .

Method	Backbone	Dataset	AR $\uparrow$	VSD $\uparrow$	MSSD $\uparrow$	MSPD $\uparrow$	mIoU $\uparrow$
Oryon $\dagger$	ViT-L/14	REAL275	46.5	32.1	50.9	56.7	100.0
Oryon ⋆	ViT-B/16		28.8	18.9	32.4	35.1	57.4
Oryon	ViT-L/14		32.2	23.6	36.6	36.4	66.5
DeCLIP	ViT-B/16		33.7	23.2	37.6	40.2	65.1
DeCLIP	ViT-L/14		37.6	27.5	42.2	43.2	72.2
$\Delta$ Score	ViT-B/16	REAL275	(+4.9)	(+4.3)	(+5.2)	(+5.1)	(+7.7)

TABLE 4: Results on OV object 6D pose estimation.

\dagger

denotes “oracle” performance using GT object masks for pose estimation. ⋆ indicates our ViT-B results based on the official code, as Oryon uses the ViT-L version of CLIP by default.

Fig. 10: Qualitative comparison of OV pose estimation results between CLIP and DeCLIP on REAL275 [ 74 ] and TOYL [ 75 ] . Following Oryon [ 23 ] , we color the object model by mapping its 3D coordinates to the RGB space for visualization.

Fig. 11: Comparison with existing regional V-L alignment methods on OV region classification. DeCLIP consistently outperforms existing methods across various resolutions.

Content Distill.	Context Distill.	RCC	SD-GSC	OV-COCO	VOC21	COCO-Obj	Avg.
				17.5	43.4	20.6	27.2
✓				37.6	42.3	23.2	34.4
	✓			37.4	59.7	36.4	44.5
✓	✓			41.1	59.9	36.5	45.8
✓	✓	✓		42.5	60.5	36.9	46.6
✓	✓		✓	41.9	63.7	38.5	48.0

TABLE 8: Component analysis of DeCLIP. We report OV detection performance on OV-COCO using mAP and training-free OVSS performance on PASCAL VOC and COCO-Obj using mIoU. Content+Context corresponds to the conference version. RCC and SD-GSC are newly added components.

Fig. 12: Comparison of semantic affinity maps between SD and VFM features. Both SD features and the channel concatenation of SD and VFM features exhibit lower semantic affinity capability than using VFM features alone.

Method	VOC20	Context59	Cityscapes	ADE	COCO-Stf
Concat	75.2	30.6	24.4	19.2	19.9
Concat ⋆	84.4	39.1	33.7	21.7	25.1
DirectAlign	79.6	32.0	28.5	18.0	21.3
SD-GSC	85.3	41.6	35.7	23.1	26.8

TABLE 9: Ablation Study on the collaboration strategies between VFM and SD. “Concat” combines VFM and SD features along the channel dimension. ⋆ represents the weighted feature concatenation strategy. “DirectAlign” refers to the direct distillation of SD self-attention maps.

Fig. 13: Qualitative comparison of attention maps. The comparison methods include: SAM [ 33 , 40 ] , trained on large-scale mask annotations; DINO [ 38 ] and DINOv2 [ 39 ] with registers [ 61 ] , trained using a self-supervised approach; AM-RADIO [ 99 ] , utilizing multi-teacher distillation from CLIP, SAM, and DINOv2 models; and SCLIP [ 42 ] , incorporating correlative self-attention mechanisms. The query image token is marked in red.

VLM Model	VOC21	Context60	COCO-Obj	Cityscapes	ADE	COCO-Stf
OpenAI-CLIP [ 7 ]	18.8	9.9	8.1	6.5	3.1	5.7
ClearCLIP [ 44 ]	51.8	32.6	33.0	30.0	16.7	23.9
DeCLIP	64.9	36.3	39.1	35.5	19.8	24.6
EVA-CLIP [ 60 ]	23.4	12.8	15.3	12.8	7.7	9.7
EVA-ClearCLIP †	47.0	29.7	30.2	26.3	16.7	20.4
DeCLIP	64.1	37.9	38.7	35.7	23.1	26.8

TABLE 10: Ablation studies on different VLM baselines.

\dagger

indicates re-implementation based on the official code.

Fig. 14: Ablation studies of DeCLIP method. (a), (b), (c), and (d) denote the weight of the context loss, the number of fine-tuned layers in CLIP, the domain and scale of the distillation dataset, and the impact of VFM selection on OV dense perception performance (detection and segmentation), respectively. The segmentation results are averaged over multiple datasets. All experiments are performed using the ViT-B version of DeCLIP.

Fig. 15: Analysis of decoupled learning and integrated distillation. Training loss curves for (a) content losses and (b) context losses. The proposed decoupled learning method (blue) demonstrates faster convergence, reduced final loss, and enhanced training stability.

Fig. 16: Gradient conflict analysis for content/context decoupling. We compare the gradient cosine similarity distributions of the content and context losses for (a) the integrated baseline and (b) DeCLIP. Red, orange, and blue bars denote conflicting, orthogonal, and aligned gradients, respectively.

Method	Final loss mean	Std.	CV	Gradient conflict
Integrated Distillation	1.0798	0.1598	14.8%	51.2%
DeCLIP	0.6500	0.0679	10.5%	25.6%

TABLE 11: Training stability comparison between integrated distillation and DeCLIP. Mean and standard deviation (Std.) are computed from the logged total loss over the final epoch. The coefficient of variation (CV) is computed as Std./Mean and measures relative loss fluctuation. Lower Std. and CV indicate more stable training.

Fig. 17: Large-scale proxy-token statistics. We measure Proxy-Token Mass (PT-Mass) and Same-Semantic-Region Attention Ratio (SSAR) on 5,000 COCO-Panoptic validation images. PT-Mass is the attention mass on the top-

k

proxy tokens, and SSAR is the attention mass within the same semantic region. Numbers are means.

Fig. 18: Attention interaction in CLIP. We select the top-10 proxy tokens at Layer 11 and trace them across layers over 500 COCO images. Rows and columns denote query and key groups, respectively. Values are mean attention mass.

Fig. 19: Proxy-token query attention. We compare CLIP and DINOv2 at Layer 11. CLIP shows vertical proxy-token stripes and concentrated proxy-query responses, while DINOv2 shows spatially distributed attention.

Method	Backbone	Detector	$\text{AP}_{50}^{\text{Base}}$	$\text{AP}_{50}^{\text{Novel}}$	$\text{AP}_{50}^{\text{All}}$
Integrated Learning	ViT-B/16 [ 60 ]	F-ViT [ 9 ]	53.6	36.5	49.1
DeCLIP	ViT-B/16 [ 60 ]	F-ViT [ 9 ]	56.7	43.3	53.2
Integrated Learning	ViT-L/14	F-ViT	62.7	43.2	57.6
DeCLIP	ViT-L/14	F-ViT	65.8	50.2	61.7

TABLE 12: Comparison of decoupled and integrated learning in OV detection. Results are reported as

\text{AP}_{50}

on base, novel, and all categories. Decoupled learning yields consistent gains over integrated distillation, with notably larger improvements on novel categories.

Fig. 20: Qualitative comparison of attention maps. We compare DeCLIP, integrated learning, and CLIP on representative samples. DeCLIP produces more localized and object-consistent attention responses, while integrated learning retains more background activation and CLIP shows diffuse noisy responses.

Method	With a background category			Without background category					Avg.
Method	VOC21	Context60	COCO-Obj	VOC20	Cityscapes	Context59	ADE	COCO-Stf	Avg.
Integrated Learning	58.5	33.8	34.6	78.3	28.7	32.4	20.1	23.4	38.7
DeCLIP	64.1	37.9	38.7	85.3	35.7	41.6	23.1	26.8	44.1

TABLE 13: Quantitative comparison of decoupled and integrated learning approaches for OV segmentation. Results are reported in mIoU across multiple benchmarks. Decoupled learning consistently outperforms integrated learning on all datasets, leading to a substantial improvement in average mIoU.

Fig. 21: Qualitative comparison of decoupled and integrated learning. From left to right are the input image, ground-truth mask, K-means clustering results, and PCA feature visualizations. Compared with integrated learning, the decoupled strategy produces cleaner segmentation masks with sharper object boundaries and closer alignment to ground truth. The PCA results further indicate that DeCLIP yields less noisy features than integrated learning.

Fig. 22: Qualitative comparison of semantic completion strategies. The red dot indicates the query token. From left to right, the columns display the input image, raw attention maps extracted from each model, and completed semantic affinity maps. In contrast to the boundary-focused SAM and fragmented I-JEPA, SD-GSC yields affinity maps with superior object integrity and spatial coherence.

Method	With a background category			Without background category					Avg.
Method	VOC21	Context60	COCO-Obj	VOC20	Cityscapes	Context59	ADE	COCO-Stf	Avg.
JEPA-GSC [ 100 ]	61.5	35.9	37.0	84.2	33.5	39.7	21.2	24.9	42.2
SAM-GSC [ 33 ]	61.7	36.1	37.2	83.9	34.1	39.9	21.9	25.1	42.5
SD-GSC	64.1	37.9	38.7	85.3	35.7	41.6	23.1	26.8	44.2

TABLE 14: Comparison of different semantic completion strategies. We evaluate the effectiveness of our Stable Diffusion-based approach (SD-GSC) against alternative methods, including transformer-based feature inpainting using I-JEPA and boundary refinement using SAM.

Method	Backbone	Detector	$\text{AP}_{50}^{\text{Base}}$	$\text{AP}_{50}^{\text{Novel}}$	$\text{AP}_{50}^{\text{All}}$
JEPA-GSC [ 100 ]	ViT-B/16 [ 60 ]	F-ViT [ 9 ]	54.3	39.5	50.4
SAM-GSC [ 33 ]			54.2	37.8	49.9
SD-GSC			56.7	43.3	53.2

TABLE 15: Comparison of different semantic completion strategies on OV detection. We report the Average Precision (AP) at IoU 0.5 for base categories, novel categories, and all categories on the OV-COCO benchmark.

Method	Clean	Noise (Avg.)		Blur (Avg.)		Snow		Frost		Fog		Brightness		Contrast		Elastic Trans.		Pixelate		JPEG Comp.
Method	mIoU	mPC	rPC	mPC	rPC	mPC	rPC	mPC	rPC	mPC	rPC	mPC	rPC	mPC	rPC	mPC	rPC	mPC	rPC	mPC	rPC
ClearCLIP [ 44 ]	37.5	24.1	63.4	26.1	68.5	26.1	66.8	25.7	66.1	32.9	87.0	35.6	93.9	30.5	80.2	27.6	73.7	32.3	85.6	31.2	82.4
SCLIP [ 42 ]	38.2	26.3	68.1	28.9	75.0	26.4	67.1	26.3	67.3	35.3	92.6	36.9	97.3	33.4	87.4	28.6	76.3	35.8	94.5	33.3	87.2
DeCLIP	44.2	33.6	75.8	34.8	77.7	33.4	73.2	33.2	72.8	41.3	93.2	43.3	97.8	40.4	91.0	36.7	83.2	41.9	94.9	40.4	90.9

TABLE 16: Corruption Robustness for OV Segmentation. Results are averaged over 5 severity levels for each of 15 corruption types and then aggregated over 8 segmentation benchmarks. Clean : average mIoU on clean images. For better readability, the results of the three noise corruptions (Gaussian/Shot/Impulse) and the four blur corruptions (Defocus/Glass/Motion/Zoom) are averaged and reported as Noise (Avg.) and Blur (Avg.) , respectively.

Method	$\text{AP}_{50}^{\text{Novel}}$			$\text{AP}_{50}^{\text{Base}}$			$\text{AP}_{50}^{\text{All}}$
Method	Clean	mPC	rPC	Clean	mPC	rPC	Clean	mPC	rPC
CLIP [ 7 ]	26.7	13.8	51.8	44.0	26.2	59.6	39.5	23.0	58.2
CLIPSelf [ 9 ]	37.5	29.3	78.0	54.9	40.8	74.3	50.4	37.8	75.0
DeCLIP	43.7	35.3	80.7	56.7	44.6	78.7	53.3	41.7	78.2

TABLE 17: Corruption robustness for OV detection. Experiments are conducted on OV-COCO. Clean is

\text{AP}_{50}

on clean inputs. mPC and rPC are averaged over 15 corruption types and 5 severity levels.

Method	Backbone	$\mathrm{AP}$	$\mathrm{AP}_{\mathrm{SO}}$	$\mathrm{AP}_{\mathrm{MO}}$	$\mathrm{AP}_{\mathrm{HO}}$
CLIP-VIS [ 26 ]	ResNet-50	14.1	24.6	14.1	3.1
CLIP-VIS	ConvNeXt-B	18.5	31.8	20.3	4.4
CLIP-VIS+DeCLIP	ViT-B/16	22.2	37.2	26.3	6.7
CLIP-VIS	ConvNeXt-L	20.0	36.6	22.5	5.5
CLIP-VIS+DeCLIP	ViT-L/14	29.3	47.0	33.9	11.4

TABLE 18: Occlusion-level evaluation on OVIS for OV video instance segmentation. We report overall

\mathrm{AP}

and the breakdown by occlusion level, including

\mathrm{AP}_{\mathrm{SO}}

,

\mathrm{AP}_{\mathrm{MO}}

, and

\mathrm{AP}_{\mathrm{HO}}

. Higher is better.

Method	OpenEarthMap	LoveDA	iSAID	Potsdam	Vaihingen	UAVid	UDD5	VDD	Avg.
CLIP [ 7 ]	12.0	12.4	7.5	14.5	10.3	10.9	9.5	14.2	11.4
MaskCLIP [ 86 ]	25.1	27.8	14.5	31.7	24.7	28.6	32.4	32.9	27.2
SCLIP [ 42 ]	29.3	30.4	16.1	36.6	28.4	31.4	38.7	37.9	31.1
GEM [ 112 ]	33.9	31.6	17.7	36.5	24.7	33.4	41.2	39.5	32.3
ClearCLIP [ 44 ]	31.0	32.4	18.2	40.9	27.3	36.2	41.8	39.3	33.4
DeCLIP	34.2	38.5	19.0	41.4	34.6	35.4	44.1	39.9	35.9

TABLE 19: Cross-domain generalization to remote sensing applications. We evaluated COCO-distilled DeCLIP on eight remote sensing segmentation datasets and report mIoU. Without exposure to any remote sensing images during decoupled distillation, DeCLIP achieved the best average performance, demonstrating robust transfer across a large domain gap.

Method	Size	Fwd. Time (ms)	Memory (MB)	FLOPs (G)	MACs (G)
CLIP [ 7 ]	ViT-B/16	11.79	658.45	57.18	28.59
CLIP	ViT-L/14	22.62	1842.28	262.92	131.46
DeCLIP	ViT-B/16	11.34	658.45	53.01	26.50
DeCLIP	ViT-L/14	21.81	1847.39	253.24	126.62

TABLE 20: Inference efficiency comparison of CLIP and DeCLIP. DeCLIP preserves the same inference-time architecture and parameter count as the original CLIP, resulting in comparable latency, memory footprint, and computational complexity across both backbones.

Method	Backbone	FPS	Latency breakdown (ms)
Method	Backbone	FPS	Pre.	Backbone	Seg Head	Post.
CLIP-VIS [ 26 ]	ConvNeXt-B	16.24	0.54	12.37	22.43	26.21
CLIP-VIS+CLIP	ViT-B/16	15.84	0.56	14.88	22.08	25.58
CLIP-VIS+DeCLIP	ViT-B/16	15.95	0.53	14.49	22.11	25.54

TABLE 21: Inference speed analysis. Inference time is decomposed into preprocessing, backbone, segmentation head, and post-processing phases in milliseconds. Tests use an RTX 4090 (512×512 input, batch size 1).

Method	With a background category			Without background category					Avg.
Method	VOC21	Context60	COCO-Object	VOC20	Cityscapes	Context59	ADE	COCO-Stuff	Avg.
SCLIP-Tiny [ 42 ]	32.6	14.3	8.0	59.8	3.9	19.0	5.7	9.8	19.1
ClearCLIP-Tiny [ 44 ]	54.2	29.8	28.9	79.8	20.3	32.5	16.7	22.3	35.6
DeCLIP-Tiny	59.3	32.2	32.2	82.8	31.9	35.8	18.2	24.1	39.6

TABLE 22: Generalization analysis on the lightweight TinyCLIP-ViT-39M/16 backbone. We implement state-of-the-art open-vocabulary segmentation methods on the capacity-constrained TinyCLIP model. DeCLIP consistently outperforms both SCLIP and ClearCLIP across eight diverse benchmarks, demonstrating the robustness of our decoupled learning approach in edge-deployment scenarios.

Model	Backend	Precision	Latency (ms) $\downarrow$	Throughput (FPS) $\uparrow$	Memory (MB) $\downarrow$	VOC21 (mIoU) $\uparrow$
CLIP [ 60 ]	PyTorch	FP32	10.29	97.19	1397	43.4
	PyTorch	FP16	8.41	118.87	725	43.4
	TensorRT	FP16	1.46	683.05	679	43.4
	TensorRT	FP8	1.41	711.73	531	43.9
DeCLIP	PyTorch	FP32	9.84	101.67	1397	64.1
	PyTorch	FP16	8.01	124.79	725	64.1

TABLE 23: Quantitative results of inference optimization. Experiments on an NVIDIA H100 GPU (batch size 1,

560\times 560

) demonstrate that DeCLIP is orthogonal to standard acceleration strategies. Existing optimization frameworks like TensorRT can significantly boost the inference speed of our method (e.g., reducing latency by 86% with FP16).

Method	Time	Memory	Teacher computation (ms/step)
Method	(min/epoch)	(GiB/GPU)	CLIP	DINOv2	Total
CLIPSelf-B	8.3	8.4	74.9	–	74.9
DeCLIP-B	8.6	10.3	74.5	16.7	91.2
DeCLIP-L	37.0	33.5	574.1	22.8	596.9

TABLE 24: Fine-tuning costs on eight NVIDIA A800 GPUs (batch size 8 per GPU). Memory denotes peak allocation per GPU. SD incurs no computational overhead during fine-tuning, as its attention maps can be cached beforehand.

Fig. 23: Qualitative comparison of feature representations using K-means and PCA. Columns 3 through 5 display token-level segmentation masks generated via K-means clustering. Columns 6 through 8 show the principal components of the features. Region-level alignment causes CLIPSelf to exhibit more severe feature fragmentation than the original CLIP. Conversely, DeCLIP generates integral object masks with robust activations that align closely with the ground truth.

Fig. 24: Qualitative comparison of feature correlations between DeCLIP and existing methods [ 9 , 10 ] . Specifically, the input image resolution is 336336. We extract the output features from each attention block of CLIP, where each feature $\mathbf{F}\in\mathbb{R}^{441\times D}$ . Then, we compute the feature correlations $\mathbf{FC}\in\mathbb{R}^{441\times 441}$ between the image tokens within $\mathbf{F}$ using cosine similarity. We randomly select a specific image token’s feature correlation (indicated by the red dots) and upsample it to a resolution of 336336 for visualization.

Method	Size	ADE847	Context459	ADE150	Context59	VOC20	VOC21
CLIP	ViT-B	12.0	19.0	31.8	57.5	94.6	77.3
CLIP	ViT-L	16.0	23.8	37.9	63.3	97.0	82.5
EVA-CLIP	ViT-B	11.9	17.6	30.4	52.3	94.2	74.2
EVA-CLIP	ViT-L	14.2	21.3	34.8	56.2	95.8	80.1
DeCLIP	ViT-B	15.3	21.4	36.3	60.6	96.6	81.3
DeCLIP	ViT-L	17.6	25.9	40.7	63.9	97.7	83.9

TABLE 25: Sanity check on OV semantic segmentation to eliminate potential bias introduced by EVA-CLIP. The evaluation metric is mIoU.

Method	VOC21	Context60	COCO-Obj	Cityscapes	ADE	COCO-Stf
CLIP [ 7 ]	18.8	9.9	8.1	6.5	3.1	5.7
EVA-CLIP [ 60 ]	23.4	12.8	15.3	12.8	7.7	9.7
ClearCLIP [ 44 ]	51.8	32.6	33.0	30.0	16.7	23.9
EVA-ClearCLIP	47.0	29.7	30.2	26.3	16.7	20.4
DeCLIP	64.1	37.9	38.7	35.7	23.1	26.8

TABLE 26: Sanity checks on training-free OV semantic segmentation to eliminate potential bias introduced by EVA-CLIP. The evaluation metric is mIoU.

Fig. 25: Quantitative comparison of vision-language alignment across various resolutions. Classification accuracy on COCO-Panoptic for (a) “thing” categories using Ground Truth (GT) boxes, (b) “thing” categories using GT masks, and (c) “stuff” categories using GT masks. CLIP results are limited to 560 resolution due to the absence of high-resolution training.

Method	Novel $\mathrm{AP}_{50}$	BBox mAP	$\mathrm{AP}_{s}$	$\mathrm{AP}_{m}$	$\mathrm{AP}_{l}$
CLIP [ 60 ]	30.0	25.5	15.6	28.2	35.6
CLIPSelf [ 9 ]	44.5	33.1	22.8	37.2	42.2
DeCLIP	50.3	34.9	25.5	38.3	44.6

TABLE 27: Object-scale detection analysis on OV-COCO. We report novel

\mathrm{AP}_{50}

, bbox mAP, and AP for small, medium, and large objects.

Fig. 26: Qualitative comparison of improvements and failure cases. Rows 1-2 show OV detection on OV-LVIS, while Rows 3-4 display video segmentation on OVIS. Even columns present zoomed-in views of the predicted regions. (a) Improvement Cases: DeCLIP corrects baseline classification errors for small objects and refines segmentation boundaries for large, overlapping instances. (b)-(c) Failure Cases: The model still faces difficulties in detecting tiny objects from rare categories and segmenting instances under severe occlusion.

Crop range	VOC20	VOC21	Context59	Context60	COCO-Obj	COCO-Stuff	ADE	Cityscapes	Avg.
$m,n\in[1,3]$	86.4	64.0	41.5	37.8	39.7	26.7	23.0	33.4	44.1
$m,n\in[1,6]$	85.3	65.1	42.2	38.5	39.0	27.1	23.2	35.9	44.5
$m,n\in[1,8]$	84.6	64.7	42.2	38.4	37.6	27.0	23.2	37.2	44.4

TABLE 28: Sensitivity to the crop sampling range. Results are reported as mIoU on eight training-free OV semantic segmentation benchmarks.

Model	SD setting	With a background category			Without background category					Avg.
Model	SD setting	VOC21	Context60	COCO-Obj	VOC20	Cityscapes	Context59	ADE	COCO-Stf	Avg.
CLIP	–	18.8	9.9	8.1	49.4	6.5	11.1	3.1	5.7	14.1
DeCLIP	t45_up3 (default)	65.1	38.5	39.0	85.3	35.9	42.2	23.2	27.1	44.5
DeCLIP	t45_up2	55.4	32.3	28.3	80.3	30.4	35.4	18.1	22.0	37.8
DeCLIP	t30_up3	65.0	38.5	38.7	85.2	36.0	41.9	23.0	27.0	44.4

TABLE 29: Stable Diffusion layer and timestep ablation for SD-GSC on training-free OVSS. We report mIoU on eight training-free OVSS benchmarks. The CLIP row is included as a reference baseline.

Fig. 27: Qualitative comparison under extreme lighting. We present qualitative training-free OV semantic segmentation results on Cityscapes and ADE20K under high-brightness degradation. DeCLIP demonstrates superior semantic coherence compared to existing methods such as ClearCLIP and SCLIP. However, the third and fourth rows highlight a limitation where extreme overexposure leads to significant information loss, resulting in segmentation artifacts.

Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perception

Abstract

Figures & tables

Open-Vocabulary Object Detection

SynCLIP: Synonym-Coherent Language-Image Pretraining for Robust Open-Vocabulary Dense Perception

SCI-CLIP: Segment-Centric Inference with Reference Memory for Training-Free Open-Vocabulary Segmentation

TraceCLIP: Recovering Local Semantics from Patch-to-CLS Contributions

Abstract

Open-Vocabulary Object Detection

Explore similar work

SynCLIP: Synonym-Coherent Language-Image Pretraining for Robust Open-Vocabulary Dense Perception

SCI-CLIP: Segment-Centric Inference with Reference Memory for Training-Free Open-Vocabulary Segmentation

TraceCLIP: Recovering Local Semantics from Patch-to-CLS Contributions