Humans can easily recognize the same objects across different viewpoints of a 3D scene, whereas teaching models such multi-view-consistent perception typically requires videos or calibrated multi-view captures, which are considerably more expensive and less diverse than large-scale monocular data.~In this paper, we introduce Feature Puzzles, a novel framework for learning multi-view-consistent segmentation using only single-image semantic segmentation supervision. Specifically, we propose a simple yet effective pipeline that constructs multi-view-consistent segmentation pseudo-labels through image cropping and geometric transformations. We further introduce a lightweight, plug-and-play network that leverages the multi-view-consistent image features learned by existing pretrained 3D foundation models and the generated pseudo-labels to perform multi-view-consistent segmentation. Importantly, our flexible framework supports various types of prompts provided from one or multiple views and can be readily extended to open-vocabulary 3D segmentation. Under this single-image supervision, our feed-forward model achieves the best mIoU on NVOS, remains competitive on SPIn-NeRF and ScanNet++ against baselines using per-scene optimization or in-domain multi-view training, and demonstrates zero-shot transfer to open-vocabulary 3D segmentation. Visual comparisons are available on our project page: https://featurepuzzles.github.io/
Figures & tables
Figure 1: Feature Puzzles learns multi-view-consistent segmentation from single images , extending feed-forward reconstruction to (A) promptable segmentation that propagates to all views in one pass, (B) spatial tracking, and (C) open-vocabulary 3D segmentation.
Figure 2: Feature Puzzles pipeline. From a single RGB(-D) parent image ( A ), we sample overlapping crops as a puzzle ( B ), optionally calibrate their cameras and apply 3D augmentation ( C ), and pull back SAM masks ( D ) and DINO features ( E ) into every crop (bottom).
Figure 3: Feature Puzzles network architecture. A shared Image Encoder + Aggregator produce tokens feeding three decoders: Geometry (poses, 3D points), prompt-conditioned Mask (mask/point/box → masks), and Feature ( ⊕ text-prompt → open-vocab masks). Mask and feature decoders share cross-view attention and a weighted pixel decoder.
Category
Method
Train
NVOS
SPIn
mIoU (%)
mAcc (%)
mIoU (%)
mAcc (%)
Per-scene opt.
SPIn-NeRF ( Mirzaei et al., 2023 )
NeRF
–
–
90.7
98.8
SA3D ( Cen et al., 2023 )
NeRF+SAM
91.1
98.4
92.4
98.8
SAGA ( Cen et al., 2025a )
3DGS+SAM
92.6
98.6
93.7
99.2
SA3D-GS ( Cen et al., 2025b )
3DGS+SAM
92.7
98.5
93.4
99.1
OmniSeg3D ( Ying et al., 2024 )
3DGS+SAM
92.8
98.6
94.5
99.3
Table 1: Promptable multi-view segmentation on NVOS and SPIn-NeRF. Per-scene methods fit each test scene; generalization methods use one forward pass. Ours uses only SA-1B-style masks for task-specific segmentation training.
Method
Training Data
ScanNet++
nuScenes-mini
mIoU (%)
T-SR (%)
T-mIoU all
T-mIoU vis
T-SR all
T-SR vis
SpaTracker+SAM ( Li et al., 2026 )
Kubric synth. (video)
16.15
23.68
–
–
–
–
CoTracker3+SAM ( Karaev et al., 2025 )
Kubric synth. (video)
–
–
4.99
18.15
0.00
17.41
SAM2* ( Ravi et al., 2025 )
SA-V (video)
44.16
57.89
4.48
20.71
0.00
28.70
IGGT ( Li et al., 2026 )
ScanNet++ RGB-D
73.02
98.90
2.34
9.46
59.26
78.89
Ours
SA-1B (single image)
63.60
87.50
4.78
20.97
47.13
88.89
Table 2: Instance spatial tracking on ScanNet++ and nuScenes-mini. IGGT is in-distribution on ScanNet++ but OOD on nuScenes. Ours is OOD with respect to task-specific training on both benchmarks and uses only SA-1B for task-specific segmentation training. T-SR denotes temporal success rate; nuScenes reports all-frame (all) and visible-frame (vis) metrics.
Method
ScanNet
ScanNet200
ScanNet++
Recon. time
Per-Scene/ Gen.
on/ off
10 cls
15 cls
19 cls
70 cls
20 cls
mIoU
mAcc
mIoU
mAcc
mIoU
mAcc
mIoU
mAcc
mIoU
mAcc
In-domain / benchmark-specific methods
LangSplat ( Qin et al., 2024 )
6.52
20.11
3.25
13.16
1.34
7.44
0.72
4.39
2.21
10.18
∼ 6 hr
Per-Scene
Offline
LEGaussians ( Shi et al., 2024 )
6.79
18.13
4.13
15.49
2.53
5.67
1.39
5.45
2.93
9.34
∼ 6 hr
Online-LangSplat ( Katragadda et al., 2025 )
7.13
21.56
3.89
14.52
3.45
8.97
2.45
4.12
4.51
11.34
5.4 min
Online
Table 3: 3D semantic segmentation on the EmbodiedSplat benchmark. † zero-shot evaluation with SA-1B task-specific training only; ‡ closed-set evaluation after in-domain fine-tuning on the evaluated semantic classes. “cls” denotes class. Bold = best, underline = second best.
Figure 4: Ablation study. (a, b) Precision/recall, memory, and runtime versus view count. The long-view stress test shows near-linear memory/runtime scaling, while precision degrades at large view counts. (c, d) Component ablations for open-vocabulary (ScanNet-10 mAcc) and promptable segmentation (SceneNN-MV mIoU); numbers indicate the drop from the full model.
Backbone
Multi-view architecture
Open-vocabulary 3D segmentation
Instance spatial tracking
ScanNet
ScanNet200
nuScenes-mini
10 cls
15 cls
19 cls
70 cls
T-mIoU all
T-mIoU vis
T-SR all
T-SR vis
mIoU
mAcc
mIoU
mAcc
mIoU
mAcc
mIoU
mAcc
LingBot-Map
Online, streaming
27.38
40.26
19.45
29.88
15.69
27.21
5.67
11.38
3.68
13.21
41.98
78.90
VGGT-Omega (default)
Offline, all-view attention
41.09
58.55
35.87
50.73
30.50
45.93
13.79
23.79
4.78
20.97
47.13
88.89
Table 4: Multi-backbone generalization. The same head and task-specific SA-1B training are applied to an offline and a streaming reconstruction backbone, without in-domain fine-tuning.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Image-to-Clips (Puzzles)
Feature Puzzles (ours)
Parent annotation
RGB-D (GT or predicted depth)
Instance masks / semantic map / frozen teacher field
View-to-parent map
Fixed K + extrinsics from PnP-RANSAC + centroid rotation
Closed-form affine πn
Label construction
Per-patch depth and pose, re-rendered
Pull-back ℓ∘πn
Error sources
Depth, PnP residual, rendering holes
Resampling only
Validity handling
FrontCov / ImgCov tests, reject or hole-fill
Foreground coverage + cross-view support
Cross-view agreement
Approximate
Exact by construction
Appendix
Table 5: Term-by-term comparison between the Image-to-Clips augmentation of Puzzles ( Ma et al., 2025a ) and the Feature Puzzles construction.
Aspect
VGGT-Segmentor
Feature Puzzles
Training unit
Image pair
Multi-view sequence
Inference
Source–target pairs
All views jointly
Object modeling
Pairwise correspondence
Joint multi-view representation
Prompt propagation
Source → target
Arbitrary view(s) → all views
Explicit correspondence
VGGT point tracking
Not required
Intermediate-view context
Pairwise only
Jointly exploited
Appendix
Table 6: Detailed comparison between VGGT-Segmentor ( Gao et al., 2026 ) and Feature Puzzles.
Variant
ScanNet
ScanNet200
Δ mAcc 10
10 cls
15 cls
19 cls
70 cls
mIoU
mAcc
mIoU
mAcc
mIoU
mAcc
mIoU
mAcc
Full model
15.70
29.69
10.01
19.95
8.46
18.73
3.32
7.59
–
w/o Lwarp
15.60
29.49
9.86
19.87
8.22
18.73
3.64
7.87
−0.20
w/o cross-view attention
14.07
27.77
8.89
18.57
7.35
17.48
3.37
7.42
−1.92
w/o Lgram
12.85
26.91
8.10
18.06
7.05
17.60
2.74
6.48
−2.78
Appendix
Table 7: Component ablations for open-vocabulary 3D segmentation on EmbodiedSplat (Stage-1 only, 2,500-step budget, no in-domain fine-tuning). Δ is the ScanNet 10-class mAcc change relative to the full model. Bold = best, underline = second best.
Variant
Vis mIoU
Obj mIoU
Vis Prec.
Vis Rec.
Δ Vis mIoU
Full model (LoRA + multi-view prompting)
55.19
56.80
65.35
76.79
–
w/o Lcons
55.15
56.36
65.11
76.74
−0.04
w/o multi-view prompting
52.17
52.01
60.53
77.97
−3.02
w/o cross-view attention
51.12
54.66
67.80
67.37
−4.07
Hard overlap [0.10,0.50]
40.23
41.39
51.20
61.75
−14.96
w/o LoRA
38.11
36.40
47.10
64.37
−17.08
Appendix
Table 8: Component ablations for promptable multi-view segmentation on SceneNN-MV (GT-prompted, 6 views, 2,500-step budget). “Vis” metrics are averaged over views where the object is visible; “Obj” is object-level mIoU. Δ is the Vis mIoU change relative to the full model. Bold = best, underline = second best.
Figure 5: Efficiency and camera-pose preservation. (a–b) GPU memory and runtime of VGGT- Ω with and without Feature Puzzles as the number of views increases. (c) Camera pose accuracy on 7-Scenes, showing that Feature Puzzles largely preserves the original camera-pose performance.
Figure 6: Promptable multi-view segmentation. Results on ScanNet and an out-of-distribution AI-generated video, showing consistent cross-view mask propagation from sparse input prompts.
Figure 7: Open-vocabulary 3D segmentation. Zero-shot evaluation on ScanNet and FARM-Scenes, illustrating coherent semantic reconstruction across diverse large-scale environments.