We present Rewis3d, a framework that leverages recent advances in feed-forward 3D reconstruction to significantly improve weakly supervised semantic segmentation on 2D images. Obtaining dense, pixel-level annotations remains a costly bottleneck for training segmentation models. Alleviating this issue, sparse annotations offer an efficient weakly-supervised alternative. However, they still incur a performance gap. To address this, we introduce a novel approach that leverages 3D scene reconstruction as an auxiliary supervisory signal. Our key insight is that 3D geometric structure recovered from 2D videos provides strong cues that can propagate sparse annotations across entire scenes. Specifically, a dual student-teacher architecture enforces semantic consistency between 2D images and reconstructed 3D point clouds, using state-of-the-art feed-forward reconstruction to generate reliable geometric supervision. Extensive experiments demonstrate that Rewis3d achieves state-of-the-art performance in sparse supervision, outperforming existing approaches by 2-7% without requiring additional labels or inference overhead.
Figures & tables
Figure 2: Conceptual overview of weakly-supervised segmentation approaches. (a) Traditional methods rely solely on sparse 2D annotations, limiting supervision propagation. (b) Our proposed method Rewis3d introduces a 3D branch, enforcing cross-modal consistency (CMC) between 2D predictions and 3D predictions from reconstructed geometry.
Figure 3: Overview of the training pipeline. Our framework operates in two stages. Base Training (blue and green) establishes independent student-teacher setups for each modality using sparse supervision. Cross-Modal Consistency (orange) introduces our core contribution: bidirectional knowledge transfer where the teacher of one modality supervises the student of the other, weighted by our dual confidence mechanism leveraging prediction certainty and reconstruction quality.
Figure 4: Sparse label accumulation. Firstly, an image sequence is unprojected to a 3D point cloud via a multi-view reconstruction model. Subsequently, we establish correspondences between the 3D points and the 2D pixels in the source images. This allows for label accumulation in the 3D space, and by projection, also in the 2D images.
Method
Ours
Backbone
3D Supervision
Waymo
KITTI-360
NYUv2
mIoU
SS/FS (%)
mIoU
SS/FS (%)
mIoU
SS/FS (%)
Fully Supervised
Segformer-B4
–
59.0
–
68.4
–
51.1
–
EMA
Segformer-B4
–
49.4
83.7
60.3
88.2
42.9
84.0
SASFormer [ 22 ]
Segformer-B4
–
37.8
64.1
46.4
67.8
44.7
87.5
TEL [ 15 ]
DeepLabV3+
–
42.4
71.9
59.2
86.6
38.3
75.0
Ours (Real 3D)
✓
Segformer-B4
LiDAR/Depth
51.8
87.8
61.7
90.2
44.7
87.6
Table 1: Semantic segmentation results with scribble supervision. We report mean Intersection-over-Union (mIoU, %) and the percentage of the supervision gap closed between scribble-supervised and fully-supervised baselines (SS/FS). Our proposed Rewis3d framework, which leverages reconstructed 3D geometry for cross-modal consistency, achieves state-of-the-art performance and consistently outperforms existing weakly supervised semantic segmentation (WSSS) methods across all datasets. Bold indicates the best scribble-supervised result, and underline the second best.
Figure 5: Qualitative comparison across outdoor and indoor datasets. Rewis3d produces sharper boundaries, more accurate fine-grained predictions, and better long-range segmentation compared to the Mean Teacher baseline (EMA) and TEL, even in regions where 3D reconstruction is uncertain. Colormaps are provided in the appendix.
Method
Annotation Type
Points
Scribbles
Coarse
Fully Supervised
77.6
77.6
77.6
TEL [ 15 ]
53.1
64.4
64.9
SASFormer [ 22 ]
42.7
55.6
42.8
EMA (Baseline)
50.5
61.2
66.5
Ours
56.5
68.1
68.6
Table 2: Generalization across annotation types on Cityscapes (mIoU %). Rewis3d consistently outperforms all competing methods, including the EMA baseline, across three sparse annotation types—points, scribbles, and coarse masks—demonstrating strong versatility and broad applicability. Reported improvements (+6.0, +6.9, +2.1) are relative to the EMA baseline, with the largest gains observed under minimal supervision (points and scribbles), confirming the effectiveness of geometric consistency in diverse weak supervision scenarios.
Configuration
Confidence Filtering
Sampling
3D Source
mIoU
None
Pred.
Recon.
Random
View-Aware
EMA Baseline (2D only)
—
—
—
—
—
—
49.4
Effect of Confidence Filtering (with View-Aware sampling, Multi-view Recon.):
No filtering
✓
✓
Multi-view
51.9
+ Prediction conf.
✓
✓
Multi-view
52.7
+ Reconstruction conf.
✓
✓
Multi-view
52.1
Table 3: Ablation study on Waymo (mIoU %). We systematically validate each component of our framework. Base configuration uses EMA baseline (49.4 mIoU). Each row progressively adds components, demonstrating their individual and cumulative contributions.
Figure 6: Scribble length ablation on KITTI-360. Rewis3d (red) maintains strong performance across varying scribble lengths compared to baselines, with largest gains in sparse settings, demonstrating the value of geometric supervision when annotations are scarce.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Waymo
KITTI-360
NYUv2
mIoU
Δ vs EMA
mIoU
Δ vs EMA
mIoU
Δ vs EMA
EMA (3D only)
41.8
–
44.3
–
24.4
–
Ours (Recon)
45.5
+3.7
44.9
+0.6
28.5
+4.1
Appendix
Table A1 : 3D Semantic Segmentation Performance (mIoU %). Cross-modal consistency also improves 3D segmentation through bidirectional knowledge transfer from 2D. The column Δ vs EMA denotes the absolute performance improvement in percentage points (pp) of our method compared to the 3D-only Mean Teacher baseline (EMA).
Figure A1 : 3D Segmentation and Error Maps. Rewis3d improves also improves the 3D segmentation noticeably. The bidirectional knowledge transfer through CMC especially improves the 3D segmentation quality with respect to errors emanating from misclassifications of objects. This highlights how segmentation models in 2D and 3D domain possess different advantages. While the structure of the 3D space lends itself to more accurate separation of objects, the correct class assignment appears to be easier to learn in 2D.
Figure B1 : Qualitative comparison with point supervision. Even with minimal supervision (one point per object), Rewis3d successfully propagates labels to the full object extent. Note the improved segmentation of the vehicle and road markers in Waymo (top row) and the clearer delineation of the sidewalk in KITTI-360 (bottom row) compared to the EMA and TEL baselines. Additionally, we observe that the KITTI-360 ground truth contains occasional labeling errors (e.g., the bicycle); consequently, our model’s visually correct predictions in these regions may be penalized during quantitative evaluation against the imperfect ground truth.
Figure B2 : Additional Qualitative Results with Scribble Supervision. We provide further qualitative examples of Rewis3d predictions on the Waymo (top), Cityscapes (middle), and KITTI-360 (bottom) datasets. These visualizations demonstrate the model’s capability to generate spatially coherent segmentations with precise boundaries across diverse outdoor urban environments, solely trained with sparse scribble supervision.
Method
mIoU
SS/FS (%)
bicycle
bird
building
bus
car
construction cone pole
cyclist
ground
ground animal
lane marker
motorcycle
motorcyclist
other large vehicle
pedestrian
pedestrian object
pole
road
road marker
sidewalk
sign
sky
traffic light
trailer
truck
vegetation
Fully Supervised
59.1
-
79.3
10.0
93.4
16.0
93.9
65.4
57.9
74.6
0.0
64.3
73.7
0.0
16.0
83.0
9.0
69.7
94.8
64.9
84.3
73.7
95.3
74.0
28.1
66.3
89.2
Point annotations
EMA
43.4
73.4
58.1
0.0
84.8
15.7
80.7
39.6
10.9
50.3
0.0
30.9
54.0
0.0
10.8
58.7
7.0
43.9
85.3
43.1
60.8
49.4
92.0
54.3
22.4
54.3
78.8
SASFormer
37.4
63.2
52.9
0.0
80.7
16.4
75.5
11.9
15.4
49.8
0.0
32.7
47.1
0.0
0.1
52.9
7.0
23.9
89.9
29.9
57.4
44.0
93.8
32.0
1.7
40.8
78.3
TEL
35.4
59.9
0.0
0.0
78.5
41.9
54.3
21.2
9.4
45.7
0.0
53.9
28.5
0.0
2.3
49.3
0.0
34.8
86.4
39.7
55.1
45.4
92.9
41.2
1.9
29.8
73.3
Ours (Recon)
48.8
82.6
59.9
0.0
88.0
6.1
89.5
44.3
23.7
64.1
0.0
53.6
60.2
0.0
7.6
66.5
7.3
49.9
91.9
54.4
72.9
57.7
91.0
55.9
28.8
64.6
81.2
Appendix
Table C1 : Comparison of 2D mIoU Results and Relative Scores on Waymo Dataset Bold numbers indicate the best and underlined values the second-best performance among non-fully supervised approaches.
Method
mIoU
SS/FS (%)
background
bag
bathtub
bed
blinds
books
bookshelf
box
cabinet
ceiling
chair
clothes
counter
curtain
desk
door
dresser
floor
floor mat
lamp
mirror
nightstand
otherfurniture
otherprop
otherstructure
paper
person
picture
pillow
refrigerator
shelves
sink
sofa
table
television
toilet
towel
wall
whiteboard
window
Fully Supervised
51.1
-
80.4
12.5
48.0
63.8
51.0
35.4
68.3
19.0
71.7
64.1
54.4
22.8
61.4
62.3
25.0
49.2
55.8
64.5
27.8
45.5
47.8
51.5
20.6
41.4
37.2
38.3
81.8
66.7
45.9
64.1
11.2
61.1
45.2
43.0
68.4
80.7
46.0
85.6
78.4
46.1
Scribble annotations
EMA
42.9
84.0
66.6
11.0
39.3
52.8
31.0
31.9
53.6
11.8
63.8
47.7
49.0
21.9
59.0
47.1
20.5
48.5
45.0
57.3
34.3
31.3
36.1
42.8
15.1
31.7
28.6
31.3
66.5
56.7
36.4
53.3
11.5
50.0
38.6
36.8
57.9
69.5
36.3
80.2
77.1
37.0
SASFormer
45.2
88.5
71.2
15.6
50.8
59.3
40.3
34.0
56.0
13.6
69.0
51.8
50.0
23.2
59.5
52.1
18.0
42.5
40.8
56.9
32.5
34.8
34.3
38.7
14.3
34.2
29.8
33.0
75.6
53.4
40.3
59.7
12.4
57.5
36.9
35.5
61.4
69.9
40.1
79.2
72.8
38.4
TEL
39.1
76.5
69.0
16.3
28.9
47.2
18.6
30.4
56.7
10.4
57.5
49.3
39.7
16.0
53.4
36.1
10.7
40.7
33.7
54.2
15.8
33.5
32.0
28.4
28.3
33.6
16.3
35.4
69.9
51.1
35.3
51.6
6.7
47.5
31.3
27.3
41.5
63.4
25.8
76.6
74.4
38.6
Ours (Real 3D)
44.7
87.6
72.7
9.9
47.4
55.9
39.7
31.4
57.3
14.0
62.8
54.1
45.6
21.6
60.7
58.3
17.7
45.9
50.5
61.8
28.1
35.5
41.3
44.1
14.4
34.1
34.3
33.8
73.9
59.7
36.4
54.1
9.8
55.7
37.4
42.9
51.6
68.3
40.0
80.9
70.1
38.8
Appendix
Table C2 : Detailed Class-wise IoU Comparison on NYUv2 Dataset Bold numbers indicate the best and underlined values the second-best performance among non-fully supervised approaches.
Method
mIoU
SS/FS (%)
bicycle
building
car
fence
motorcycle
person
pole
road
sidewalk
sky
terrain
traffic light
traffic sign
truck
vegetation
wall
Fully Supervised
68.4
-
49.1
89.0
94.5
53.4
60.6
66.3
43.4
96.4
85.5
94.3
77.7
0.0
47.9
78.6
89.8
67.8
Point annotations
EMA
52.2
76.3
33.1
78.2
78.6
42.0
38.7
43.3
22.8
84.4
59.9
84.7
56.7
0.0
33.6
51.5
80.0
48.1
SASFormer
27.0
39.5
21.0
54.6
22.2
28.2
3.8
6.7
0.1
65.6
27.5
53.2
26.0
0.0
8.2
23.5
60.9
30.5
TEL
48.9
71.5
24.8
79.4
70.6
39.5
37.0
37.9
16.0
81.1
54.1
86.9
56.8
0.0
34.8
42.9
76.3
43.7
Ours (Recon)
58.2
85.1
43.7
83.9
89.7
49.5
48.8
46.4
25.3
91.0
70.8
72.2
61.8
0.0
37.9
63.9
83.8
63.1
Appendix
Table C3 : Comparison of mIoU Results and Relative Scores on KITTI-360 Dataset Bold numbers indicate the best and underlined values the second-best performance among non-fully supervised approaches.
Method
mIoU
SS/FS (%)
bicycle
building
bus
car
fence
motorcycle
person
pole
rider
road
sidewalk
sky
terrain
traffic light
traffic sign
train
truck
vegetation
wall
Fully Supervised
77.6
-
74.5
92.4
86.4
94.5
60.1
62.7
78.9
58.9
60.4
98.2
85.4
94.8
65.7
65.5
74.8
80.4
82.0
92.3
66.3
Point annotations
EMA
50.5
65.1
49.0
74.6
57.8
82.9
29.1
25.0
51.8
25.9
30.3
94.1
64.9
78.4
37.7
16.9
30.0
52.3
49.4
77.2
32.7
SASFormer
42.7
55.0
35.5
67.5
53.1
70.1
33.8
17.8
38.3
12.1
10.4
91.8
41.1
75.8
18.6
10.8
25.4
45.8
54.2
69.4
41.5
TEL
53.0
68.3
59.0
81.2
39.7
80.6
36.7
31.3
62.9
32.8
44.2
93.2
62.7
80.5
41.7
36.5
57.8
30.9
23.9
84.1
26.8
Ours (Recon)
56.5
72.8
54.0
82.9
69.6
85.3
33.1
31.1
55.3
32.4
31.8
94.6
67.1
84.2
43.1
25.5
40.6
62.1
55.3
81.6
43.1
Appendix
Table C4 : Comparison of mIoU Results and Relative Scores on Cityscapes Dataset Bold numbers indicate the best and underlined values the second-best performance among non-fully supervised approaches.
Figure D1 : Visualization of sampling strategies for cross-modal consistency. The rows display the 3D point cloud (middle) and its projection onto the 2D target view (bottom). Green points indicate valid 2D-3D correspondences for the target image, while black points represent the remaining scene geometry. Left: Random Sampling (120k points global) results in sparse correspondences (avg. 167) in the view, insufficient for dense consistency learning. Center: Sampling strictly from correspondences (100% target view) maximizes alignment but results in a fragmented 3D scene (note the missing car geometry), preventing the 3D network from learning global context. Right: Our View-Aware Sampling Strategy (60% target view, 40% spatial context) ensures dense correspondences ( ∼ 72K) while maintaining the holistic scene structure required for robust 3D segmentation.
Source
Density
Confidence Weighting
mIoU (%)
Ours (Real 3D)
Sparse (LiDAR)
Single (Pred. Only)
51.8
Ours (Recon)
Sparse (Simulated)
Single (Pred. Only)
51.7
Ours (Recon)
Dense (Multi-view)
Dual (Pred. + Rec.)
53.3
Appendix
Table E1 : Validation of Real vs. Reconstructed Gap (Waymo). We simulate the characteristics of Real LiDAR (sparsity and lack of confidence scores) using our Reconstructed data. When matching the constraints of Real LiDAR, our performance aligns with the Real 3D baseline, confirming that the gains of Rewis3d come from the increased density and dual-confidence weighting enabled by the reconstruction.
Parameter
Waymo Open Dataset Value
NYUv2 Dataset Value
height distortion
0.9
1
min binary erosion
2
2
max binary erosion
40
20
it extra erosion
5
2
min erosion area share
0.002
0.003
max erosion area share
0.05
0.15
Appendix
Table F1 : Scribble Generation Configurations for Waymo Open Dataset and NYUv2
2D Branch
3D Branch
Config
Value
Config
Value
Optimizer
AdamW
Optimizer
AdamW
Scheduler
PolyLR
Scheduler
OneCycleLR
Criteria
CrossEntropy
Criteria
CrossEntropy
Student Teacher Loss
Student Teacher Loss
Learning Rate
0.00005
Learning Rate
0.001
Appendix
Table G1 : Overview of core training hyperparameters for 2D and 3D branches
Augmentation
Parameters
Student
Teacher
Random Horizontal Flip
p: 0.5
✓
✓
Scale and Distort
scale: [0.5, 1.2], distort: [0.9, 1.1]
✓
✓
Random Crop
KITTI-360: (376, 512)
✓
✓
Waymo: (640, 640)
NYUv2: (480, 480)
Gaussian Blur
kernel size: 7; p: 0.5
✓
Appendix
Table G2 : Overview of Applied Data Augmentations in 2D. The first two columns list the augmentations and their corresponding parameters. The third and fourth columns indicate whether each augmentation is applied to the inputs of both the student and teacher models, or to only one of them.
Augmentation
Parameters
Student
Teacher
Random Rotation
axis: z, angle [-1, 1]; p: 0.5
✓
✓
Random Scale
scale: [0.9, 1.1]
✓
✓
Random Flip
p: 0.5
✓
✓
Random Jitter
sigma: 0.005; clip: 0.02
✓
PointClip
[100, 40, 100]
✓
Appendix
Table G3 : Overview of Applied Data Augmentations in 3D. The first two columns list the augmentations and their corresponding parameters. The third and fourth columns indicate whether each augmentation is applied to the inputs of both the student and teacher models, or to only one of them.
Figure H1 : Semantic class color coding for all evaluated datasets. We visualize results using consistent color mappings across each dataset: Cityscapes (19 classes), KITTI-360 (19 classes), NYUv2 (40 classes), and Waymo Open Dataset (25 classes). Color assignments follow the official dataset conventions where available. Note that while some classes share similar names across datasets (e.g., ’road’, ’car’, ’person’), their definitions and label protocols may differ between datasets.
3D instance segmentation requires representations that are both geometrically structured and semantically discriminative. Pure 3D encoders model spatial structure well but often lack strong appearance semantics, while directly lifting 2D priors into 3D by concatenation injects rich semantics without ensuring that they align well with the underlying 3D structure. In this paper, we address this problem by formulating 2D-to-3D semantic transfer as a masked semantic reconstruction task. Specifically, we mask a subset of background lifted 2D priors and train the 3D encoder to recover the missing semantic priors, providing explicit supervision for distilling rich 2D semantics into the 3D backbone. We further propose geometry-modulated semantic injection, which uses raw 3D geometry to modulate lifted 2D priors before sparse encoding, so that geometry explicitly controls semantic injection and alleviates the distribution gap between 2D semantic features and 3D geometric features. Experiments on two baseline frameworks demonstrate that the proposed method achieves state-of-the-art results in both full-data and few-shot settings on ScanNetV2 and ScanNet200. Code is publicly available at https://github.com/yharoldsu0627/Geometry_Guided_Semantic_Reconstruction
Yuanhao Su, Shaofeng Zhang
University of Science and Technology of China · Fuzhou University
Semantic-aware 3D reconstruction from sparse, unposed images remains challenging for feed-forward 3D Gaussian Splatting (3DGS). Existing methods often predict an over-complete set of Gaussian primitives under sparse-view supervision, leading to unstable geometry and inferior depth quality. Meanwhile, they rely solely on 2D segmenter features for semantic lifting, which provides weak 3D-level and limited generalizable supervision, resulting in incomplete 3D semantics in novel scenes. To address these issues, we propose UniSem, a unified framework that jointly improves depth accuracy and semantic generalization via two key components. First, Error-aware Gaussian Dropout (EGD) performs error-guided capacity control by suppressing redundancy-prone Gaussians using rendering error cues, producing meaningful, geometrically stable Gaussian representations for improved depth estimation. Second, we introduce a Mix-training Curriculum (MTC) that progressively blends 2D segmenter-lifted semantics with the model's own emergent 3D semantic priors, implemented with object-level prototype alignment to enhance semantic coherence and completeness. Extensive experiments on ScanNet and Replica show that UniSem achieves superior performance in depth prediction and open-vocabulary 3D segmentation across varying numbers of input views. Notably, with 16-view inputs, UniSem reduces depth Rel by 15.2% and improves open-vocabulary segmentation mAcc by 3.7% over strong baselines.
Recent advances in 3D reconstruction have progressed from per-scene optimization to feed-forward inference, and semantic scene understanding has followed suit -- yet existing methods remain confined to object-centric perception, neglecting spatial relations between objects. We formulate 3D spatial relation segmentation in a feed-forward, pose-free multi-view setting: given a visually specified subject and a relational text query, the model segments the target across views without receiving its category name. To this end, we propose RelationVGGT, a novel feed-forward framework that integrates semantic features from a visual foundation model with geometry-aware representations from a 3D geometry foundation model and leverages a relation transformer for subject-conditioned, cross-view relation prediction -- requiring neither per-scene optimization nor known camera poses. We additionally provide a fully automated annotation pipeline built on ScanNet++ with VLMs and LLMs, enabling scalable training data generation for this new task.