Rewis3d: Reconstruction Improves Weakly-Supervised Semantic Segmentation

Jonas Ernst; Wolfgang Boettcher; Lukas Hoyer; Jan Eric Lenssen; Bernt Schiele

cs.CVMar 6, 2026

Rewis3d: Reconstruction Improves Weakly-Supervised Semantic Segmentation

Authors: Jonas Ernst, Wolfgang Boettcher, Lukas Hoyer, Jan Eric Lenssen, Bernt Schiele

Organizations: Saarland University · Max Planck Institute for Informatics, SIC · ETH Zurich

Abstract

We present Rewis3d, a framework that leverages recent advances in feed-forward 3D reconstruction to significantly improve weakly supervised semantic segmentation on 2D images. Obtaining dense, pixel-level annotations remains a costly bottleneck for training segmentation models. Alleviating this issue, sparse annotations offer an efficient weakly-supervised alternative. However, they still incur a performance gap. To address this, we introduce a novel approach that leverages 3D scene reconstruction as an auxiliary supervisory signal. Our key insight is that 3D geometric structure recovered from 2D videos provides strong cues that can propagate sparse annotations across entire scenes. Specifically, a dual student-teacher architecture enforces semantic consistency between 2D images and reconstructed 3D point clouds, using state-of-the-art feed-forward reconstruction to generate reliable geometric supervision. Extensive experiments demonstrate that Rewis3d achieves state-of-the-art performance in sparse supervision, outperforming existing approaches by 2-7% without requiring additional labels or inference overhead.

Figures & tables

Figure 2: Conceptual overview of weakly-supervised segmentation approaches. (a) Traditional methods rely solely on sparse 2D annotations, limiting supervision propagation. (b) Our proposed method Rewis3d introduces a 3D branch, enforcing cross-modal consistency (CMC) between 2D predictions and 3D predictions from reconstructed geometry.

Figure 3: Overview of the training pipeline. Our framework operates in two stages. Base Training (blue and green) establishes independent student-teacher setups for each modality using sparse supervision. Cross-Modal Consistency (orange) introduces our core contribution: bidirectional knowledge transfer where the teacher of one modality supervises the student of the other, weighted by our dual confidence mechanism leveraging prediction certainty and reconstruction quality.

Figure 4: Sparse label accumulation. Firstly, an image sequence is unprojected to a 3D point cloud via a multi-view reconstruction model. Subsequently, we establish correspondences between the 3D points and the 2D pixels in the source images. This allows for label accumulation in the 3D space, and by projection, also in the 2D images.

Method	Ours	Backbone	3D Supervision	Waymo		KITTI-360		NYUv2
Method	Ours	Backbone	3D Supervision	mIoU	SS/FS (%)	mIoU	SS/FS (%)	mIoU	SS/FS (%)
Fully Supervised		Segformer-B4	–	$59.0$	–	$68.4$	–	$51.1$	–
EMA		Segformer-B4	–	$49.4$	$83.7$	$60.3$	$88.2$	$42.9$	$84.0$
SASFormer [ 22 ]		Segformer-B4	–	$37.8$	$64.1$	$46.4$	$67.8$	44.7	$87.5$
TEL [ 15 ]		DeepLabV3+	–	$42.4$	$71.9$	$59.2$	$86.6$	$38.3$	$75.0$
Ours (Real 3D)	✓	Segformer-B4	LiDAR/Depth	51.8	87.8	61.7	90.2	44.7	87.6

Table 1: Semantic segmentation results with scribble supervision. We report mean Intersection-over-Union (mIoU, %) and the percentage of the supervision gap closed between scribble-supervised and fully-supervised baselines (SS/FS). Our proposed Rewis3d framework, which leverages reconstructed 3D geometry for cross-modal consistency, achieves state-of-the-art performance and consistently outperforms existing weakly supervised semantic segmentation (WSSS) methods across all datasets. Bold indicates the best scribble-supervised result, and underline the second best.

Figure 5: Qualitative comparison across outdoor and indoor datasets. Rewis3d produces sharper boundaries, more accurate fine-grained predictions, and better long-range segmentation compared to the Mean Teacher baseline (EMA) and TEL, even in regions where 3D reconstruction is uncertain. Colormaps are provided in the appendix.

Method	Annotation Type
Method	Points	Scribbles	Coarse
Fully Supervised	77.6	77.6	77.6
TEL [ 15 ]	53.1	64.4	64.9
SASFormer [ 22 ]	42.7	55.6	42.8
EMA (Baseline)	50.5	61.2	66.5
Ours	56.5	68.1	68.6

Table 2: Generalization across annotation types on Cityscapes (mIoU %). Rewis3d consistently outperforms all competing methods, including the EMA baseline, across three sparse annotation types—points, scribbles, and coarse masks—demonstrating strong versatility and broad applicability. Reported improvements (+6.0, +6.9, +2.1) are relative to the EMA baseline, with the largest gains observed under minimal supervision (points and scribbles), confirming the effectiveness of geometric consistency in diverse weak supervision scenarios.

Configuration	Confidence Filtering			Sampling		3D Source	mIoU
Configuration	None	Pred.	Recon.	Random	View-Aware	3D Source	mIoU
EMA Baseline (2D only)	—	—	—	—	—	—	49.4
Effect of Confidence Filtering (with View-Aware sampling, Multi-view Recon.):
No filtering	✓				✓	Multi-view	51.9
+ Prediction conf.		✓			✓	Multi-view	52.7
+ Reconstruction conf.			✓		✓	Multi-view	52.1

Table 3: Ablation study on Waymo (mIoU %). We systematically validate each component of our framework. Base configuration uses EMA baseline (49.4 mIoU). Each row progressively adds components, demonstrating their individual and cumulative contributions.

Figure 6: Scribble length ablation on KITTI-360. Rewis3d (red) maintains strong performance across varying scribble lengths compared to baselines, with largest gains in sparse settings, demonstrating the value of geometric supervision when annotations are scarce.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Waymo		KITTI-360		NYUv2
Method	mIoU	$\Delta$ vs EMA	mIoU	$\Delta$ vs EMA	mIoU	$\Delta$ vs EMA
EMA (3D only)	$41.8$	–	$44.3$	–	$24.4$	–
Ours (Recon)	45.5	+3.7	44.9	+0.6	28.5	+4.1

Appendix

Table A1 : 3D Semantic Segmentation Performance (mIoU %). Cross-modal consistency also improves 3D segmentation through bidirectional knowledge transfer from 2D. The column

\Delta

vs EMA denotes the absolute performance improvement in percentage points (pp) of our method compared to the 3D-only Mean Teacher baseline (EMA).

Figure A1 : 3D Segmentation and Error Maps. Rewis3d improves also improves the 3D segmentation noticeably. The bidirectional knowledge transfer through CMC especially improves the 3D segmentation quality with respect to errors emanating from misclassifications of objects. This highlights how segmentation models in 2D and 3D domain possess different advantages. While the structure of the 3D space lends itself to more accurate separation of objects, the correct class assignment appears to be easier to learn in 2D.

Figure B1 : Qualitative comparison with point supervision. Even with minimal supervision (one point per object), Rewis3d successfully propagates labels to the full object extent. Note the improved segmentation of the vehicle and road markers in Waymo (top row) and the clearer delineation of the sidewalk in KITTI-360 (bottom row) compared to the EMA and TEL baselines. Additionally, we observe that the KITTI-360 ground truth contains occasional labeling errors (e.g., the bicycle); consequently, our model’s visually correct predictions in these regions may be penalized during quantitative evaluation against the imperfect ground truth.

Figure B2 : Additional Qualitative Results with Scribble Supervision. We provide further qualitative examples of Rewis3d predictions on the Waymo (top), Cityscapes (middle), and KITTI-360 (bottom) datasets. These visualizations demonstrate the model’s capability to generate spatially coherent segmentations with precise boundaries across diverse outdoor urban environments, solely trained with sparse scribble supervision.

Method	mIoU	SS/FS (%)	bicycle	bird	building	bus	car	construction cone pole	cyclist	ground	ground animal	lane marker	motorcycle	motorcyclist	other large vehicle	pedestrian	pedestrian object	pole	road	road marker	sidewalk	sign	sky	traffic light	trailer	truck	vegetation
Fully Supervised	59.1	-	79.3	10.0	93.4	16.0	93.9	65.4	57.9	74.6	0.0	64.3	73.7	0.0	16.0	83.0	9.0	69.7	94.8	64.9	84.3	73.7	95.3	74.0	28.1	66.3	89.2
Point annotations
EMA	43.4	73.4	58.1	0.0	84.8	15.7	80.7	39.6	10.9	50.3	0.0	30.9	54.0	0.0	10.8	58.7	7.0	43.9	85.3	43.1	60.8	49.4	92.0	54.3	22.4	54.3	78.8
SASFormer	37.4	63.2	52.9	0.0	80.7	16.4	75.5	11.9	15.4	49.8	0.0	32.7	47.1	0.0	0.1	52.9	7.0	23.9	89.9	29.9	57.4	44.0	93.8	32.0	1.7	40.8	78.3
TEL	35.4	59.9	0.0	0.0	78.5	41.9	54.3	21.2	9.4	45.7	0.0	53.9	28.5	0.0	2.3	49.3	0.0	34.8	86.4	39.7	55.1	45.4	92.9	41.2	1.9	29.8	73.3
Ours (Recon)	48.8	82.6	59.9	0.0	88.0	6.1	89.5	44.3	23.7	64.1	0.0	53.6	60.2	0.0	7.6	66.5	7.3	49.9	91.9	54.4	72.9	57.7	91.0	55.9	28.8	64.6	81.2

Appendix

Table C1 : Comparison of 2D mIoU Results and Relative Scores on Waymo Dataset Bold numbers indicate the best and underlined values the second-best performance among non-fully supervised approaches.

Method	mIoU	SS/FS (%)	background	bag	bathtub	bed	blinds	books	bookshelf	box	cabinet	ceiling	chair	clothes	counter	curtain	desk	door	dresser	floor	floor mat	lamp	mirror	nightstand	otherfurniture	otherprop	otherstructure	paper	person	picture	pillow	refrigerator	shelves	sink	sofa	table	television	toilet	towel	wall	whiteboard	window
Fully Supervised	51.1	-	80.4	12.5	48.0	63.8	51.0	35.4	68.3	19.0	71.7	64.1	54.4	22.8	61.4	62.3	25.0	49.2	55.8	64.5	27.8	45.5	47.8	51.5	20.6	41.4	37.2	38.3	81.8	66.7	45.9	64.1	11.2	61.1	45.2	43.0	68.4	80.7	46.0	85.6	78.4	46.1
Scribble annotations
EMA	42.9	84.0	66.6	11.0	39.3	52.8	31.0	31.9	53.6	11.8	63.8	47.7	49.0	21.9	59.0	47.1	20.5	48.5	45.0	57.3	34.3	31.3	36.1	42.8	15.1	31.7	28.6	31.3	66.5	56.7	36.4	53.3	11.5	50.0	38.6	36.8	57.9	69.5	36.3	80.2	77.1	37.0
SASFormer	45.2	88.5	71.2	15.6	50.8	59.3	40.3	34.0	56.0	13.6	69.0	51.8	50.0	23.2	59.5	52.1	18.0	42.5	40.8	56.9	32.5	34.8	34.3	38.7	14.3	34.2	29.8	33.0	75.6	53.4	40.3	59.7	12.4	57.5	36.9	35.5	61.4	69.9	40.1	79.2	72.8	38.4
TEL	39.1	76.5	69.0	16.3	28.9	47.2	18.6	30.4	56.7	10.4	57.5	49.3	39.7	16.0	53.4	36.1	10.7	40.7	33.7	54.2	15.8	33.5	32.0	28.4	28.3	33.6	16.3	35.4	69.9	51.1	35.3	51.6	6.7	47.5	31.3	27.3	41.5	63.4	25.8	76.6	74.4	38.6
Ours (Real 3D)	44.7	87.6	72.7	9.9	47.4	55.9	39.7	31.4	57.3	14.0	62.8	54.1	45.6	21.6	60.7	58.3	17.7	45.9	50.5	61.8	28.1	35.5	41.3	44.1	14.4	34.1	34.3	33.8	73.9	59.7	36.4	54.1	9.8	55.7	37.4	42.9	51.6	68.3	40.0	80.9	70.1	38.8

Appendix

Table C2 : Detailed Class-wise IoU Comparison on NYUv2 Dataset Bold numbers indicate the best and underlined values the second-best performance among non-fully supervised approaches.

Method	mIoU	SS/FS (%)	bicycle	building	car	fence	motorcycle	person	pole	road	sidewalk	sky	terrain	traffic light	traffic sign	truck	vegetation	wall
Fully Supervised	68.4	-	49.1	89.0	94.5	53.4	60.6	66.3	43.4	96.4	85.5	94.3	77.7	0.0	47.9	78.6	89.8	67.8
Point annotations
EMA	52.2	76.3	33.1	78.2	78.6	42.0	38.7	43.3	22.8	84.4	59.9	84.7	56.7	0.0	33.6	51.5	80.0	48.1
SASFormer	27.0	39.5	21.0	54.6	22.2	28.2	3.8	6.7	0.1	65.6	27.5	53.2	26.0	0.0	8.2	23.5	60.9	30.5
TEL	48.9	71.5	24.8	79.4	70.6	39.5	37.0	37.9	16.0	81.1	54.1	86.9	56.8	0.0	34.8	42.9	76.3	43.7
Ours (Recon)	58.2	85.1	43.7	83.9	89.7	49.5	48.8	46.4	25.3	91.0	70.8	72.2	61.8	0.0	37.9	63.9	83.8	63.1

Appendix

Table C3 : Comparison of mIoU Results and Relative Scores on KITTI-360 Dataset Bold numbers indicate the best and underlined values the second-best performance among non-fully supervised approaches.

Method	mIoU	SS/FS (%)	bicycle	building	bus	car	fence	motorcycle	person	pole	rider	road	sidewalk	sky	terrain	traffic light	traffic sign	train	truck	vegetation	wall
Fully Supervised	77.6	-	74.5	92.4	86.4	94.5	60.1	62.7	78.9	58.9	60.4	98.2	85.4	94.8	65.7	65.5	74.8	80.4	82.0	92.3	66.3
Point annotations
EMA	50.5	65.1	49.0	74.6	57.8	82.9	29.1	25.0	51.8	25.9	30.3	94.1	64.9	78.4	37.7	16.9	30.0	52.3	49.4	77.2	32.7
SASFormer	42.7	55.0	35.5	67.5	53.1	70.1	33.8	17.8	38.3	12.1	10.4	91.8	41.1	75.8	18.6	10.8	25.4	45.8	54.2	69.4	41.5
TEL	53.0	68.3	59.0	81.2	39.7	80.6	36.7	31.3	62.9	32.8	44.2	93.2	62.7	80.5	41.7	36.5	57.8	30.9	23.9	84.1	26.8
Ours (Recon)	56.5	72.8	54.0	82.9	69.6	85.3	33.1	31.1	55.3	32.4	31.8	94.6	67.1	84.2	43.1	25.5	40.6	62.1	55.3	81.6	43.1

Appendix

Table C4 : Comparison of mIoU Results and Relative Scores on Cityscapes Dataset Bold numbers indicate the best and underlined values the second-best performance among non-fully supervised approaches.

Figure D1 : Visualization of sampling strategies for cross-modal consistency. The rows display the 3D point cloud (middle) and its projection onto the 2D target view (bottom). Green points indicate valid 2D-3D correspondences for the target image, while black points represent the remaining scene geometry. Left: Random Sampling (120k points global) results in sparse correspondences (avg. 167) in the view, insufficient for dense consistency learning. Center: Sampling strictly from correspondences (100% target view) maximizes alignment but results in a fragmented 3D scene (note the missing car geometry), preventing the 3D network from learning global context. Right: Our View-Aware Sampling Strategy (60% target view, 40% spatial context) ensures dense correspondences ( $\sim$ 72K) while maintaining the holistic scene structure required for robust 3D segmentation.

Source	Density	Confidence Weighting	mIoU (%)
Ours (Real 3D)	Sparse (LiDAR)	Single (Pred. Only)	51.8
Ours (Recon)	Sparse (Simulated)	Single (Pred. Only)	51.7
Ours (Recon)	Dense (Multi-view)	Dual (Pred. + Rec.)	53.3

Appendix

Table E1 : Validation of Real vs. Reconstructed Gap (Waymo). We simulate the characteristics of Real LiDAR (sparsity and lack of confidence scores) using our Reconstructed data. When matching the constraints of Real LiDAR, our performance aligns with the Real 3D baseline, confirming that the gains of Rewis3d come from the increased density and dual-confidence weighting enabled by the reconstruction.

Parameter	Waymo Open Dataset Value	NYUv2 Dataset Value
height distortion	0.9	1
min binary erosion	2	2
max binary erosion	40	20
it extra erosion	5	2
min erosion area share	0.002	0.003
max erosion area share	0.05	0.15

Appendix

Table F1 : Scribble Generation Configurations for Waymo Open Dataset and NYUv2

2D Branch		3D Branch
Config	Value	Config	Value
Optimizer	AdamW	Optimizer	AdamW
Scheduler	PolyLR	Scheduler	OneCycleLR
Criteria	CrossEntropy	Criteria	CrossEntropy
	Student Teacher Loss		Student Teacher Loss
Learning Rate	0.00005	Learning Rate	0.001

Appendix

Table G1 : Overview of core training hyperparameters for 2D and 3D branches

Augmentation	Parameters	Student	Teacher
Random Horizontal Flip	p: 0.5	✓	✓
Scale and Distort	scale: [0.5, 1.2], distort: [0.9, 1.1]	✓	✓
Random Crop	KITTI-360: (376, 512)	✓	✓
	Waymo: (640, 640)
	NYUv2: (480, 480)
Gaussian Blur	kernel size: 7; p: 0.5	✓

Appendix

Table G2 : Overview of Applied Data Augmentations in 2D. The first two columns list the augmentations and their corresponding parameters. The third and fourth columns indicate whether each augmentation is applied to the inputs of both the student and teacher models, or to only one of them.

Augmentation	Parameters	Student	Teacher
Random Rotation	axis: z, angle [-1, 1]; p: 0.5	✓	✓
Random Scale	scale: [0.9, 1.1]	✓	✓
Random Flip	p: 0.5	✓	✓
Random Jitter	sigma: 0.005; clip: 0.02	✓
PointClip	[100, 40, 100]	✓

Appendix

Table G3 : Overview of Applied Data Augmentations in 3D. The first two columns list the augmentations and their corresponding parameters. The third and fourth columns indicate whether each augmentation is applied to the inputs of both the student and teacher models, or to only one of them.

Figure H1 : Semantic class color coding for all evaluated datasets. We visualize results using consistent color mappings across each dataset: Cityscapes (19 classes), KITTI-360 (19 classes), NYUv2 (40 classes), and Waymo Open Dataset (25 classes). Color assignments follow the official dataset conventions where available. Note that while some classes share similar names across datasets (e.g., ’road’, ’car’, ’person’), their definitions and label protocols may differ between datasets.

Weakly Supervised Semantic Segmentation

Weakly supervised semantic segmentation learns pixel-level class masks from image-level labels, points, scribbles, or other sparse annotations.

See all 33 papers

Geometry-Guided Semantic Reconstruction for 3D Instance Segmentation
Oct 12, 2026Yuanhao Su, Shaofeng Zhang3D Instance Segmentation Feature Fusion
University of Science and Technology of China · Fuzhou University
UniSem: Generalizable Semantic 3D Reconstruction from Sparse Unposed Images
Mar 18, 2026Guibiao Liao, Qian Ren, Kaimin Liao +43D Reconstruction Sparse-View 3D Reconstruction
Moore Threads AI
RelationVGGT: Visual Geometry Transformers for 3D Spatial Relation Segmentation
Oct 1, 2026Minsu Kim, Jaesung Choe, Jiwoo Lee +23D ViTs 3D Spatial Reasoning
Yonsei University · NVIDIA

Oct 12, 2026J/K move · Enter open · S save

Geometry-Guided Semantic Reconstruction for 3D Instance Segmentation

Yuanhao Su, Shaofeng Zhang

University of Science and Technology of China · Fuzhou University

3D Instance Segmentation Feature Fusion

Open paper AskPDF arXiv

3D instance segmentation requires representations that are both geometrically structured and semantically discriminative. Pure 3D encoders model spatial structure well but often lack strong appearance semantics, while directly lifting 2D priors into 3D by concatenation injects rich semantics without ensuring that they align well with the underlying 3D structure. In this paper, we address this problem by formulating 2D-to-3D semantic transfer as a masked semantic reconstruction task. Specifically, we mask a subset of background lifted 2D priors and train the 3D encoder to recover the missing semantic priors, providing explicit supervision for distilling rich 2D semantics into the 3D backbone. We further propose geometry-modulated semantic injection, which uses raw 3D geometry to modulate lifted 2D priors before sparse encoding, so that geometry explicitly controls semantic injection and alleviates the distribution gap between 2D semantic features and 3D geometric features. Experiments on two baseline frameworks demonstrate that the proposed method achieves state-of-the-art results in both full-data and few-shot settings on ScanNetV2 and ScanNet200. Code is publicly available at https://github.com/yharoldsu0627/Geometry_Guided_Semantic_Reconstruction

Abstract

Weakly Supervised Semantic Segmentation

Explore similar work

Geometry-Guided Semantic Reconstruction for 3D Instance Segmentation

UniSem: Generalizable Semantic 3D Reconstruction from Sparse Unposed Images

RelationVGGT: Visual Geometry Transformers for 3D Spatial Relation Segmentation