Semantic and panoptic occupancy prediction for road scene analysis provides a dense 3D representation of the ego vehicle's surroundings. Current camera-only approaches typically rely on costly dense 3D supervision or require training models on data from the target domain, limiting deployment in unseen environments. We propose FreeOcc, a training-free pipeline that leverages pretrained foundation models to recover both semantics and geometry from multi-view images. FreeOcc extracts per-view panoptic priors with a promptable foundation segmentation model and prompt-to-taxonomy rules, and reconstructs metric 3D points with a reconstruction foundation model. Depth- and confidence- aware filtering lifts reliable labels into 3D, which are fused over time and voxelized with a deterministic refinement stack. For panoptic occupancy, instances are recovered by fitting and merging robust current-view 3D box candidates, enabling instance-aware occupancy without any learned 3D model. On Occ3D-nuScenes, FreeOcc achieves 16.9 mIoU and 16.5 RayIoU train-free, on par with state-of-the-art weakly supervised methods. When employed as a pseudo-label generation pipeline for training downstream models, it achieves 21.1 RayIoU, surpassing the previous state-of-the-art weakly supervised baseline. Furthermore, FreeOcc sets new baselines for both train-free and weakly supervised panoptic occupancy prediction, achieving 3.1 RayPQ and 3.9 RayPQ, respectively. These results highlight foundation-model-driven perception as a practical route to training-free 3D scene understanding.
Figures & tables
Figure 1: FreeOcc: Training-free panoptic occupancy prediction from foundation models. The pipeline operates directly from camera images and poses without any training. It combines a Semantic Branch powered by the SegmentAnything foundation model (SAM3) and a Geometric Branch powered by the MapAnything 3D reconstruction model to obtain 3D semantically labeled point clouds with instance priors from SAM3’s masks. An Instance Identification module then refines instance candidates and re-assigns wrongly labeled points to the correct instance and semantic class. Finally, the fused point cloud is voxelized and refined to produce the final panoptic occupancy grid.
Figure 2: Semantic branch. A set of easily handcrafted prompts is fed to SegmentAnything3 [ 9 ] , which yields 2D mask candidates with scores; we fuse them into per-view semantic and instance priors and use rules to remap prompt classes to the target taxonomy. Multiple prompts can be used for each target class, e.g. synonyms.
Figure 3: geometric branch. In parallel, MapAnything outputs per-pixel 3D points along with depth and confidence maps. We filter points by depth/confidence and keep the remaining labeled 3D points, yielding a sparse point cloud.
Figure 4: instance identification. Prompted SAM3 yields 2D mask candidates with scores; we fuse them into per-view semantic and instance priors and remap prompts to a canonical taxonomy with a simple lookup. MapAnything yields metric depth with a confidence signal; after reliability filtering we lift pixels to labeled 3D points. We optionally fuse points over time (non-causal or causal by frame selection), regularize thing instances via current-sample 3D box candidates, then voxelize and refine to produce semantic and panoptic occupancy grids.
Figure 5: voxelization branch. We voxelize the labeled point cloud and apply a lightweight multi-stage refinement step to produce semantic and panoptic occupancy grids. Notice that small holes, ego vehicle noise and backgrounded ghost artifacts are removed in this stage.
Method
mIoU
occupied
barrier
bicycle
bus
car
cons. veh.
motorcycle
pedestrian
traffic cone
trailer
truck
driv. surf.
sidewalk
terrain
manmade
vegetation
ShelfOcc* [ 2 ]
9.6
26.0
6.6
3.3
7.0
8.8
2.6
4.7
5.0
8.1
0.1
5.4
34.6
14.6
18.2
10.9
14.5
FreeOcc (Ours)
16.9
37.2
16.1
7.0
12.2
17.6
13.0
8.4
9.7
4.6
2.1
14.0
51.3
31.1
29.0
17.7
19.4
Table 1: Training-free semantic occupancy prediction performance on Occ3D-nuScenes validation set. IoU per class and mIoU are reported. ’others’ and ’other_flat’ classes are not included in the mIoU calculation. * means that the method is from a preprint.
Method
Train Free
Cam. Masks
mIoU
IoUocc
RayIoU
1m
2m
4m
EasyOcc* [ 5 ]
×
✓
16.0
38.9
-
-
-
-
GaussianFlowOcc [ 1 ]
×
✓
17.1
13.9
16.5
11.8
16.6
21.0
ShelfOcc* [ 2 ]
×
×
9.6
26.0
-
-
-
-
ShelfOcc* [ 2 ] (+STCOcc [ 10 ] )
×
✓
22.9
56.1
20.0
14.4
20.1
25.5
FreeOcc (Ours)
✓
×
16.9
37.2
16.5
11.7
16.7
21.6
FreeOcc (+STCOcc [ 10 ] )
×
×
22.8
43.8
21.1
13.6
21.3
28.5
Table 2: Weakly-supervised semantic occupancy prediction performance on Occ3D-nuScenes validation set. IoU per class and mIoU are reported. ’others’ and ’other_flat’ classes are not included in the mIoU calculation. * means that the method is from a preprint.
Method
Sup.
RayPQ
RayPQ1m
RayPQ2m
RayPQ4m
SparseOcc [ 11 ]
Full
14.1
10.2
14.5
17.6
Panoptic-FlashOcc [ 16 ]
Full
16.0
11.9
16.3
19.7
FreeOcc (Ours)
Train-free
3.1
0.8
2.6
5.8
FreeOcc (+STCOcc [ 10 ] )
Weak
3.9
1.0
3.0
7.6
Table 3: Panoptic occupancy prediction on Occ3D-nuScenes’ validation set. Measured with RayPQ from [ 11 ] .
Stage
Prompts &Rules
Conf. Filter
Occ Refine
Inst. Ident.
mIoU
RayIoU
RayPQ
(I)
10.9
11.2
1.1
(II)
✓
13.6
14.2
1.4
(III)
✓
✓
13.7
14.2
1.4
(IV)
✓
✓
✓
15.8
14.6
1.5
(V)
✓
✓
✓
✓
16.9
16.5
2.5
Table 4: Ablation study. Five stages of ablation are presented where features are added incrementally. (I) only includes the Semantic and Geometric branches. (V) is the final FreeOcc method. All values are computed on the Occ3D-nuScenes validation set.
Extrinsics
Causal
mIoU
RayIoU
RayPQ
✓
✓
16.9
16.5
2.5
×
✓
7.9
9.2
1.1
✓
×
20.4
16.8
3.5
Table 5: Impact of ablating prior extrinsics (no camera poses given to MapAnything as input) and causality (using future frames for current prediction) on panoptic occupancy prediction. Values are reported for Occ3D-nuScenes’ validation set.
Existing learning-based occupancy prediction methods rely on large-scale 3D annotations and generalize poorly across environments. We present FreeOcc, a training-free framework for open-vocabulary occupancy prediction from monocular or RGB-D sequences. Unlike prior approaches that require voxel-level supervision and ground-truth camera poses, FreeOcc operates without 3D annotations, pose ground truth, or any learning stage. FreeOcc incrementally builds a globally consistent occupancy map via a four-layer pipeline: a SLAM backbone estimates poses and sparse geometry; a geometrically consistent Gaussian update constructs dense 3D Gaussian maps; open-vocabulary semantics from off-the-shelf vision-language models are associated with Gaussian primitives; and a probabilistic Gaussian-to-occupancy projection produces dense voxel occupancy. Despite being entirely training-free and pose-agnostic, FreeOcc achieves over 2× improvements in IoU and mIoU on EmbodiedOcc-ScanNet compared to prior self-supervised methods. We further introduce ReplicaOcc, a benchmark for indoor open-vocabulary occupancy prediction, and show that FreeOcc transfers zero-shot to novel environments, substantially outperforming both supervised and self-supervised baselines. Project page: https://the-masses.github.io/freeocc-web/.
Zeyu Jiang, Changqing Zhou, Xingxing Zuo +1
The Hong Kong University of Science and Technology (Guangzhou) · MBZUAI
Weakly supervised 3D occupancy prediction reduces the reliance on costly 3D annotations by learning from 2D pseudo-labels generated by vision foundation models. However, existing methods typically use these imperfect pseudo-labels directly as supervision, making occupancy learning vulnerable to erroneous geometric and semantic targets. We observe that agreement across repeated observations provides an inexpensive and reliable cue for assessing pseudo-label reliability. Based on this observation, we propose LetOccVote, a weakly supervised Gaussian-based occupancy framework that leverages cross-frame voting to improve both geometric and semantic supervision. For geometry, Depth Vote exploits cross-frame geometric agreement to refine supported pseudo depth and reject contradictory estimates before volumetric lifting and depth supervision. For semantics, Semantic Vote aggregates pseudo-semantic observations in a shared 3D space to identify reliable and contested evidence, strengthening reliable semantic supervision while filtering unreliable pseudo-label segments. The entire framework is trained solely with 2D pseudo-label supervision without requiring 3D occupancy annotations. On Occ3D-nuScenes, LetOccVote achieves 53.27 IoU and 20.39 mIoU, establishing state-of-the-art performance among methods with 2D pseudo-label supervision.
Chi Zhang, Qi Song, Feifei Li +2
Tsinghua University · Shenzhen Polytechnic University
Relying on in-domain annotations and precise sensor-rig priors, existing 3D occupancy prediction methods are limited in both scalability and out-of-domain generalization. While recent visual geometry foundation models exhibit strong generalization capabilities, they were mainly designed for general purposes and lack one or more key ingredients required for urban occupancy prediction, namely metric prediction, geometry completion in cluttered scenes and adaptation to urban scenarios. We address this gap and present OccAny, the first unconstrained urban 3D occupancy model capable of operating on out-of-domain uncalibrated scenes to predict and complete metric occupancy coupled with segmentation features. OccAny is versatile and can predict occupancy from sequential, monocular, or surround-view images. Our contributions are three-fold: (i) we propose the first generalized 3D occupancy framework with (ii) Segmentation Forcing that improves occupancy quality while enabling mask-level prediction, and (iii) a Novel View Rendering pipeline that infers novel-view geometry to enable test-time view augmentation for geometry completion. Extensive experiments demonstrate that OccAny outperforms all visual geometry baselines on 3D occupancy prediction task, while remaining competitive with in-domain self-supervised methods across three input settings on two established urban occupancy prediction datasets. Our code is available at https://github.com/valeoai/OccAny .