MixSIS: Mixed-supervised instance segmentation

Christian Wilms; Emre Gülsoylu; Sven Schreiber; Simone Frintrop

cs.CVOct 27, 2026

MixSIS: Mixed-supervised instance segmentation

Authors: Christian Wilms, Emre Gülsoylu, Sven Schreiber, Simone Frintrop

Organizations: Computer Vision Group, University of Hamburg, Hamburg, Germany

Abstract

Weakly-supervised instance segmentation (WSIS) based on either box-, point-, or image-level annotations has been proposed to overcome the labor-intensive annotation process in fully-supervised instance segmentation. Despite decent results, most WSIS methods only focus on a single annotation level, ignoring that some objects are easy to spot from point annotations, while others require box or even mask annotations. Therefore, we propose the Mixed-Supervised Instance Segmentation (MixSIS) task, which allows a flexible mix of annotation levels (image, point, box, and mask) within a dataset. MixSIS has a high practical relevance, as it allows optimizing the annotation process by annotating challenging objects in more detail, while saving time on simpler instances with less detailed annotations. To address MixSIS, we propose AnnotationsMixer (AnnsMixer) as a baseline. AnnsMixer is a truly mixed-supervised instance segmentation system, which generates pseudo mask annotations from all annotation levels, even mixed within a single image. We evaluate AnnsMixer on both WSIS and MixSIS tasks, largely outperforming state-of-the-art methods. In three case studies, we also demonstrate how the unique features of MixSIS and AnnsMixer can substantially reduce the annotation time (-90%) or improve the results (+34%) in instance segmentation with a simple annotation rule or leveraging heterogeneous data. Our code is available at https://github.com/chwilms/MixSIS

Figures & tables

Figure 1: Comparison of the FSIS and WSIS tasks (left), where all instances are labeled on the same annotation level, and the MixSIS task (right), where all annotation levels are combined even within the same image.

Panel of Figure 2 — Figure 2: Pseudo mask annotations from box (left) or point (right) annotations generated with our AnnsMixer. While the mask for the person is accurate from a box or a point, the bench needs a box, as the point annotation falls within the person, making it ambiguous.

Figure 3: Overview of our pioneering baseline system AnnsMixer for MixSIS and WSIS. Prompts are derived from box-, point-, and image-level annotations to generate pseudo mask annotations using SAM. After combining them with original mask annotations, a vanilla Mask R-CNN is trained and, subsequently, applied during inference.

	System	Mask R-CNN [ 9 ]	MAL [ 17 ]	SAPNet [ 26 ]	BoxLevelSet [ 21 ]	AnnsMixer (our)
	Backbone	R-50	R-50	R-50	R-101	R-50
Setup	Supervision	$\mathcal{M}$	$\mathcal{M^{\dagger},B}$	$\mathcal{M^{\dagger},B^{\dagger},P}$	$\mathcal{M^{\dagger},B}$	$\mathcal{M,B,P,I}$
Mix 1 (25/25/25/25)	mAP	0.239	0.317	0.297	0.277	0.323
	mAP 50	0.405	0.517	0.500	0.489	0.523
	mAP 75	0.249	0.335	0.304	0.277	0.344
Mix 2 (10/20/30/40)	mAP	0.196	0.288	0.285	0.234	0.295

Table 1: MixSIS results on the COCO val split. The tuple (

\mathcal{M}

\mathcal{B}

\mathcal{P}

\mathcal{I}

) indicates the percentage of mask-, box-, point-, and image-level annotations. Best results are in bold. † denotes annotations transformed to boxes or points to match the systems’ inputs.

System	Supervision	mAP	mAP 50	mAP 75
Mask R-CNN	25% $\mathcal{M}$	0.239	0.405	0.249
AnnsMixer (our)	25% $\mathcal{M}$ +75% $\mathcal{I}$	0.267	0.429	0.288
AnnsMixer (our)	25% $\mathcal{M}$ +75% $\mathcal{P}$	0.298	0.494	0.313
AnnsMixer (our)	25% $\mathcal{M}$ +75% $\mathcal{B}$	0.346	0.561	0.367
AnnsMixer (our)	25% $\mathcal{M}$ +25% $\mathcal{B}$ +25% $\mathcal{P}$ +25% $\mathcal{I}$	0.323	0.523	0.344

Table 2: MisSIS results on COCO val split with different combinations of annotations levels. The first and last row are the respective systems’ setups from Mix 1 in Tab. 1 .

System	Backbone	Supervision	mAP	mAP 50	mAP 75
Mask R-CNN [ 9 ]	ResNet-50	$\mathcal{M}$	35.5	57.0	37.8
BoxInst [ 24 ]	ResNet-50	$\mathcal{B}$	32.1	55.1	32.4
DiscoBox [ 18 ]	ResNet-50	$\mathcal{B}$	32.0	53.6	32.6
BoxTeacher [ 5 ]	ResNet-50	$\mathcal{B}$	32.9	54.1	34.2
MAL [ 17 ]	ResNet-50	$\mathcal{B}$	35.7	-	-
BoxSeg [ 16 ]	ResNet-101	$\mathcal{B}$	33.3	53.4	35.0

Table 3: WSIS results on the COCO test-dev split.

\mathcal{M},\mathcal{B},\mathcal{P},\text{ and }\mathcal{I}

denote mask-, box-, point-, and image-level supervision. Best results per level are in bold. † : Results on the COCO val split.

Supervision	IoU	Recall	Precision
$\mathcal{B}$	70.5	84.1%	95.7%
$\mathcal{P}$	56.6	62.8%	66.8%
$\mathcal{I}$	27.1	29.3%	48.5%

Table 4: Comparison of original ground truth masks and generated pseudo mask annotations in our proposed AnnMixer given weak supervision levels (

\mathcal{B},\mathcal{P},\text{ and }\mathcal{I}

) on COCO train split. Recall and precision are calculated based on an IoU of 50 [ 7 ] .

Panel of Figure 4 — Figure 4: Examples of original COCO mask annotations and pseudo mask annotations from box-, point-, or image-level supervision generated in our proposed AnnsMixer. We use the centroid of the COCO mask annotation as point supervision.

Panel of Figure 5 — Figure 5: Examples for objects with low good continuation (left), which will be annotated with boxes in our Mix efficient , and objects with high good continuation (right), which will be annotated with a point in our Mix efficient .

Dataset	Metric	$\mathcal{M}$	$\mathcal{B}+\mathcal{P}$	$\mathcal{B}+\mathcal{P}$
		Mask R-CNN	AnnsMixer with Mix efficient	AnnsMixer with Mix random
COCO	mAP	0.366	0.347 ( -5.2% )	0.325 ( -11.2% )
COCO	Ann. time	650.8 s	114.7 s ( -82.4% )	114.7 s ( -82.4% )
VOC	mAP	0.302	0.275 ( -8.9% )	0.267 ( 11.6% )
VOC	Ann. time	207.8 s	32.1 s ( -84.6% )	32.1 s ( -84.6% )
UVO	mAP	0.125	0.127 ( +1.6% )	0.113 ( -9.6% )

Table 5: Results of Mask R-CNN and AnnsMixer trained with full supervision (

\mathcal{M}

), our proposed efficient annotation scheme Mix efficient , and the random annotation scheme Mix random on four different datasets. Annotation times are estimated per image based on [ 12 ] . Colors of relative differences indicate improvement (green) or decline (red).

Figure 6: Results of AnnsMixer trained various annotation schemes evaluated during our study. Annotation times are estimated per image based on [ 12 ] .

System	Supervision	mAP	mAP 50	mAP 75
Mask R-CNN	$\mathcal{M}$	0.288	0.530	0.288
AnnsMixer (our)	$\mathcal{M}$ + $\mathcal{B}$	0.385	0.660	0.401

Table 6: Results of Mask R-CNN and AnnsMixer using mask and box annotations on the PASCAL VOC dataset. Best results are in bold.

System	Supervision	mAP	mAP 50	mAP 75
Mask R-CNN	$\mathcal{M}$	0.229	0.423	0.216
AnnsMixer (our)	$\mathcal{M}$ + $\mathcal{B}$	0.263	0.455	0.261

Table 7: Results of Mask R-CNN and AnnsMixer using mask and box annotations on the BDD100k dataset. Best results are in bold.

System	Datasets	mAP	mAP 50	mAP 75
Mask R-CNN	MinneApple ( $\mathcal{M}$ )	0.360	0.722	0.322
AnnsMixer (our)	MinneApple ( $\mathcal{M}$ ) + MAD ( $\mathcal{B}$ )	0.367	0.723	0.330

Table 8: Results of Mask R-CNN using MinneApple’s mask annotations and AnnsMixer using MinneApple’s mask and MAD’s box annotations on the MinneApple test set. Best results are in bold.

System	Datasets	mAP	mAP 50	mAP 75
Mask R-CNN	PASCAL VOC ( $\mathcal{M}$ )	0.294	0.513	0.306
AnnsMixer (our)	PASCAL VOC ( $\mathcal{M}$ ) + COCO ( $\mathcal{I}$ )	0.325	0.546	0.341

Table 9: Results of Mask R-CNN using PASCAL VOC’s mask annotations and AnnsMixer using PASCAL VOC’s mask and COCO’s image-level annotations on the PASCAL VOC validation set. Best results are in bold.

Instance Segmentation

Instance segmentation identifies each object and assigns it a separate pixel-level mask in images or video.

See all 159 papers

Hybrid supervised instance segmentation by learning label noise suppression
Oct 14, 2026Linwei Chen, Ying Fu, Shaodi You +1Instance Segmentation Pseudo-Labeling
School of Computer Science and Technology, Beijing Institute of Technology, Beijing 100081, China · Computer Vision Research Group, Insititute of Informatics, University of Amsterdam, Amsterdam 1098 XH, Netherlands · Beijing Key Laboratory of Information Service Engineering, Beijing Union University, Beijing 100101, China
Do Instance Priors Help Weakly Supervised Semantic Segmentation?
Apr 13, 2026Anurag Das, Anna Kukleva, Xinting Hu +2Weakly Supervised Semantic Segmentation Promptable Image Segmentation
Max Planck Institute for Informatics, Saarland Informatics Campus · University of Technology Nuremberg
SAPNet++: Evolving Point-Prompted Instance Segmentation with Semantic and Spatial Awareness
Feb 25, 2026Zhaoyang Wei, Xumeng Han, Xuehui Yu +4Instance Segmentation Weakly Supervised Image Segmentation
the School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Science (UCAS), Beijing, 100049, China. · School of Automation and Intelligent Sensing, Shanghai Jiao Tong University (SJTU), Shanghai, 200240, China. · the School of Computer Science and Technology, University of Chinese Academy of Science (UCAS), Beijing, 100049, China.

Oct 14, 2026J/K move · Enter open · S save

Hybrid supervised instance segmentation by learning label noise suppression

Linwei Chen, Ying Fu, Shaodi You, Hongzhe Liu

School of Computer Science and Technology, Beijing Institute of Technology, Beijing 100081, China · Computer Vision Research Group, Insititute of Informatics, University of Amsterdam, Amsterdam 1098 XH, Netherlands · Beijing Key Laboratory of Information Service Engineering, Beijing Union University, Beijing 100101, China

Instance Segmentation Pseudo-Labeling Learning with Noisy Labels

Open paper AskPDF arXiv

To reach top accuracy, current fully supervised instance segmentation methods severely rely on large-scale pixel-wise labeled datasets. They are usually expensive and time-consuming to obtain. Though weakly or semi-supervised methods utilize cheap bounding box labeled, image-level labeled or unlabeled samples to save the labeling cost, their performance is largely sacrificed. To save labeling cost without losing much performance, in this paper, we present a pipeline that can utilize economical bounding box labels and accurate pixel-wise labels in a hybrid way. Specifically, we design two ancillary models to learn label noise suppression and obtain accurate pseudo pixel-wise labels from bounding box labels for training. One is designed to suppress mislabeling between foreground and background, and the other is designed to suppress noise from mislabeling of instances. Moreover, we exploit category-aware spatial attention module, category constraint module, instance constraint module, and self-learning training approach to improve the accuracy of pseudo labels. Experiments on the PASCAL VOC 2012 and the Cityscapes datasets show that our method can achieve competitive performance with much less labeling cost.

Abstract

Instance Segmentation

Explore similar work

Hybrid supervised instance segmentation by learning label noise suppression

Do Instance Priors Help Weakly Supervised Semantic Segmentation?

SAPNet++: Evolving Point-Prompted Instance Segmentation with Semantic and Spatial Awareness