Semantic and panoptic occupancy prediction for road scene analysis provides a dense 3D representation of the ego vehicle's surroundings. Current camera-only approaches typically rely on costly dense 3D supervision or require training models on data from the target domain, limiting deployment in unseen environments. We propose FreeOcc, a training-free pipeline that leverages pretrained foundation models to recover both semantics and geometry from multi-view images. FreeOcc extracts per-view panoptic priors with a promptable foundation segmentation model and prompt-to-taxonomy rules, and reconstructs metric 3D points with a reconstruction foundation model. Depth- and confidence- aware filtering lifts reliable labels into 3D, which are fused over time and voxelized with a deterministic refinement stack. For panoptic occupancy, instances are recovered by fitting and merging robust current-view 3D box candidates, enabling instance-aware occupancy without any learned 3D model. On Occ3D-nuScenes, FreeOcc achieves 16.9 mIoU and 16.5 RayIoU train-free, on par with state-of-the-art weakly supervised methods. When employed as a pseudo-label generation pipeline for training downstream models, it achieves 21.1 RayIoU, surpassing the previous state-of-the-art weakly supervised baseline. Furthermore, FreeOcc sets new baselines for both train-free and weakly supervised panoptic occupancy prediction, achieving 3.1 RayPQ and 3.9 RayPQ, respectively. These results highlight foundation-model-driven perception as a practical route to training-free 3D scene understanding.
Figures & tables
Figure 1: FreeOcc: Training-free panoptic occupancy prediction from foundation models. The pipeline operates directly from camera images and poses without any training. It combines a Semantic Branch powered by the SegmentAnything foundation model (SAM3) and a Geometric Branch powered by the MapAnything 3D reconstruction model to obtain 3D semantically labeled point clouds with instance priors from SAM3’s masks. An Instance Identification module then refines instance candidates and re-assigns wrongly labeled points to the correct instance and semantic class. Finally, the fused point cloud is voxelized and refined to produce the final panoptic occupancy grid.
Figure 2: Semantic branch. A set of easily handcrafted prompts is fed to SegmentAnything3 [ 9 ] , which yields 2D mask candidates with scores; we fuse them into per-view semantic and instance priors and use rules to remap prompt classes to the target taxonomy. Multiple prompts can be used for each target class, e.g. synonyms.
Figure 3: geometric branch. In parallel, MapAnything outputs per-pixel 3D points along with depth and confidence maps. We filter points by depth/confidence and keep the remaining labeled 3D points, yielding a sparse point cloud.
Figure 4: instance identification. Prompted SAM3 yields 2D mask candidates with scores; we fuse them into per-view semantic and instance priors and remap prompts to a canonical taxonomy with a simple lookup. MapAnything yields metric depth with a confidence signal; after reliability filtering we lift pixels to labeled 3D points. We optionally fuse points over time (non-causal or causal by frame selection), regularize thing instances via current-sample 3D box candidates, then voxelize and refine to produce semantic and panoptic occupancy grids.
Figure 5: voxelization branch. We voxelize the labeled point cloud and apply a lightweight multi-stage refinement step to produce semantic and panoptic occupancy grids. Notice that small holes, ego vehicle noise and backgrounded ghost artifacts are removed in this stage.
Method
mIoU
occupied
barrier
bicycle
bus
car
cons. veh.
motorcycle
pedestrian
traffic cone
trailer
truck
driv. surf.
sidewalk
terrain
manmade
vegetation
ShelfOcc* [ 2 ]
9.6
26.0
6.6
3.3
7.0
8.8
2.6
4.7
5.0
8.1
0.1
5.4
34.6
14.6
18.2
10.9
14.5
FreeOcc (Ours)
16.9
37.2
16.1
7.0
12.2
17.6
13.0
8.4
9.7
4.6
2.1
14.0
51.3
31.1
29.0
17.7
19.4
Table 1: Training-free semantic occupancy prediction performance on Occ3D-nuScenes validation set. IoU per class and mIoU are reported. ’others’ and ’other_flat’ classes are not included in the mIoU calculation. * means that the method is from a preprint.
Method
Train Free
Cam. Masks
mIoU
IoUocc
RayIoU
1m
2m
4m
EasyOcc* [ 5 ]
×
✓
16.0
38.9
-
-
-
-
GaussianFlowOcc [ 1 ]
×
✓
17.1
13.9
16.5
11.8
16.6
21.0
ShelfOcc* [ 2 ]
×
×
9.6
26.0
-
-
-
-
ShelfOcc* [ 2 ] (+STCOcc [ 10 ] )
×
✓
22.9
56.1
20.0
14.4
20.1
25.5
FreeOcc (Ours)
✓
×
16.9
37.2
16.5
11.7
16.7
21.6
FreeOcc (+STCOcc [ 10 ] )
×
×
22.8
43.8
21.1
13.6
21.3
28.5
Table 2: Weakly-supervised semantic occupancy prediction performance on Occ3D-nuScenes validation set. IoU per class and mIoU are reported. ’others’ and ’other_flat’ classes are not included in the mIoU calculation. * means that the method is from a preprint.
Method
Sup.
RayPQ
RayPQ1m
RayPQ2m
RayPQ4m
SparseOcc [ 11 ]
Full
14.1
10.2
14.5
17.6
Panoptic-FlashOcc [ 16 ]
Full
16.0
11.9
16.3
19.7
FreeOcc (Ours)
Train-free
3.1
0.8
2.6
5.8
FreeOcc (+STCOcc [ 10 ] )
Weak
3.9
1.0
3.0
7.6
Table 3: Panoptic occupancy prediction on Occ3D-nuScenes’ validation set. Measured with RayPQ from [ 11 ] .
Stage
Prompts &Rules
Conf. Filter
Occ Refine
Inst. Ident.
mIoU
RayIoU
RayPQ
(I)
10.9
11.2
1.1
(II)
✓
13.6
14.2
1.4
(III)
✓
✓
13.7
14.2
1.4
(IV)
✓
✓
✓
15.8
14.6
1.5
(V)
✓
✓
✓
✓
16.9
16.5
2.5
Table 4: Ablation study. Five stages of ablation are presented where features are added incrementally. (I) only includes the Semantic and Geometric branches. (V) is the final FreeOcc method. All values are computed on the Occ3D-nuScenes validation set.
Extrinsics
Causal
mIoU
RayIoU
RayPQ
✓
✓
16.9
16.5
2.5
×
✓
7.9
9.2
1.1
✓
×
20.4
16.8
3.5
Table 5: Impact of ablating prior extrinsics (no camera poses given to MapAnything as input) and causality (using future frames for current prediction) on panoptic occupancy prediction. Values are reported for Occ3D-nuScenes’ validation set.