TriO: Tri-Modal Unsupervised Occupancy World Model for Anything Perception

Quinlan Sykora; Sourav Biswas; Christopher Diehl; Andrew Cunningham; Thomas Gilles; Raquel Urtasun

Figure 1 : We present TriO , an unsupervised tri-modal occupancy world model designed to perceive and forecast anything that could be considered path-blocking.

Figure 2 : TriO ’s multi-modal architecture fusing LiDAR, RADAR, and camera inputs via voxel-attention. An implicit decoder generates occupancy, segmentation, and flow for a set of query points

\mathcal{Q}

supervised by pseudo-labels derived from sensor data.

Figure 3 : Obstacle and ground pseudo-label generation process.

Figure 4 : Visualization of TriO on AV2. Each row shows (left to right): perspective-view occupancy, perspective-view obstacle segmentation, camera image, rendered depth, and rendered obstacle segmentation. TriO captures key elements of the scenes: A) excavator, B) fallen cone, C) inside of truck, D) open car door, E) pedestrians.

Table 5

Figure 5 : Segmentation results on the STU validation set. Each scene contains an unexpected obstacle: A) two pigeons and B) chair, C) ball, and D) bookbag. An obstacle segmentation is correct if the road is labeled traversable ( blue ) and the obstacle is fully segmented as an obstacle ( orange ). ISL robustly segments long-tail obstacles.

Figure 6 : Scene forecasting results of TriO on AV2. LiDAR and camera sensor inputs are displayed at present time for reference. TriO ’s forecasts maintain coherent obstacle segmentation throughout time, distinguishing occupied from free space even as agents move. TriO captures a bicyclist A) and predicts its future multimodality B) .

	L1 ( $\text{\,}\mathrm{m}$ ) $\downarrow$	AbsRel ( $\text{\,}\mathrm{\%}$ ) $\downarrow$	NFCD ( $\text{\,}{\mathrm{m}}^{2}$ ) $\downarrow$	CD ( $\text{\,}{\mathrm{m}}^{2}$ ) $\downarrow$
RayTracing	4.88	35.00	3.62	17.03
$\text{occformer}\_\text{ep15}$	3.57	22.00	3.35	91.61
4D-Occ	3.22	19.00	2.45	72.74
occformer	2.97	17.00	2.03	71.33
Progressive ARM	2.32	13.00	1.81	71.41
UnO	2.24	12.00	0.86	8.10

Table 3 : LiDAR forecasting on the Argoverse 2 leaderboard.

	AV2			LRR
				All			200m+
Method	mAP $\uparrow$	Soft-IoU $\uparrow$	EPE $\downarrow$	mAP $\uparrow$	Soft-IoU $\uparrow$	EPE $\downarrow$	mAP $\uparrow$	Soft-IoU $\uparrow$	EPE $\downarrow$
Implicit-O	45.3	18.9	3.04	49.1	22.6	5.69	29.5	10.7	10.2
UnO	59.0	29.2	2.04	51.6	21.2	4.92	27.2	8.30	8.29
DIO	57.6	26.2	1.43	58.0	31.5	3.01	36.2	13.8	4.40
TriO	63.3	31.9	1.97	60.5	35.7	2.86	42.6	21.4	4.01

Table 4 : Fine-tuning downstream performance on AV2 and LRR .

	All			200m+
Method	mAP $\uparrow$	Soft-IoU $\uparrow$	EPE $\downarrow$	mAP $\uparrow$	Soft-IoU $\uparrow$	EPE $\downarrow$
TriO (no pretraining)	59.7	33.9	4.09	41.0	19.8	4.70
TriO (no radar input)	57.7	31.9	3.04	34.5	14.3	4.59
TriO (no camera input)	60.5	35.7	2.68	42.2	20.9	3.64
TriO (no radar loss)	60.5	35.6	4.35	42.6	21.3	6.02
TriO	60.5	35.7	2.86	42.6	21.4	4.01

Table 5 : Ablation studies of TriO on LRR .

Figure 7 : Occupancy fine-tuning under a varying number of labelled frames.

Method	Input			Supervision			Semantics	Continuous	4D	Unsupervised
	Camera	LiDAR	RADAR	Camera	LiDAR	RADAR
ImplicitO [ 2 ]	✗	✓	✗	✗	✗	✗	✓	✓	✗	✗
Occ4D [ 43 ]	✗	✓	✗	✗	✓	✗	✗	✓	✓	✓
UnO [ 1 ]	✗	✓	✗	✗	✓	✗	✗	✓	✓	✓
DIO [ 18 ]	✗	✓	✗	✗	✓	✗	✓	✓	✓	(✗) ∗
GASP [ 60 ]	✗	✓	✗	✓	✓	✗	✗	✓	✓	✗

Table 6 : Comparison of TriO against the most closely related methods. * Only DIO-

\varnothing

does not use human labels. † QueryOcc+ uses both camera and LiDAR supervision. ‡ The used segmentation models rely annotations like per pixel semantic masks or image-text pairs annotations. ⋄ Continuous Gaussian coordinates discretized via voxelization

Symbol	Description	Value
$\tau_{n}$	Normal orientation transition threshold	$0.85$
$k_{n}$	Normal orientation sigmoid steepness	$20.0$
$\tau_{g}$	Normal smoothness transition threshold	$0.05$
$k_{g}$	Normal smoothness sigmoid steepness	$70.0$
$\tau_{c}$	Depth confidence transition threshold	$0.7$
$k_{c}$	Depth confidence sigmoid steepness	$100.0$

Table 7 : Obstacle segmentation hyperparameters.

Class	Text prompts
Super classes
vehicle	vehicle, car, truck, bicycle, motorcycle, other-vehicle, jeep, SUV, van,
vehicle	bike, moped, pickup truck, caravan, trailer, bus, tram, train,
	construction vehicle, crane, excavator
human	human, person, bicyclist, motorcyclist, pedestrian, bicycle rider,
human	motorcycle rider

Table 8 : Default terms/ text prompts used for open-vocabulary label mapping. Those colored as orange correspond to an obstacle class, and those colored as blue correspond to a ground class. Table design inspired from [ 69 ] .

Category	Oracle term/ text prompt
Seating	Chair, Office chair, Roller chair, Gaming chair
Bags	Dior bag, Shopping bag, Paper bag, Plastic bag, Polkadot bag,
	Banana bag, Zebra bag, Inflatable bag, Briefcase, Sleeping bag,
	Workout bag, Backpack
Inflatables	Inflatable toy, Inflatable ball, Inflatable, Pool noodle
Balls	Ball, Beach ball

Table 9: Oracle obstacle text prompts/ terms hand picked for STU, separated by category. These prompts are added to

\mathcal{P}_{\text{obstacle}}

got the oracle baselines.

Method	mAP $\uparrow$	Soft-IoU $\uparrow$	EPE $\downarrow$	mAP $\uparrow$	Soft-IoU $\uparrow$	EPE $\downarrow$	# Params	Mem (MB) $\downarrow$	Runtime (ms) $\downarrow$	Runtime (ms) $\downarrow$
	All			200m+					Planner [ 6 ]	Eval
UnO	51.6	21.2	4.92	27.2	8.30	8.29	39,742,221	2,493	63.92	1774.41
DIO	58.0	31.5	3.01	36.2	13.8	4.40	8,205,109	2,930	68.21	1776.86
TriO (no camera)	60.5	35.7	2.68	42.2	20.9	3.64	12,930,483	3,449	60.81	1622.54
TriO (no radar)	57.7	31.9	3.04	34.5	14.3	4.59	54,224,584	9,907	488.06	2048.88
TriO	60.5	35.7	2.86	42.6	21.4	4.01	54,224,967	9,915	489.32	2054.63

Table 10 : Model parameters, runtime, and GPU memory for two query-point settings (Planner [ 6 ] : 20k points; Eval: 2048k points), and fine-tuning downstream performance on LRR .

	4D Semantic Occupancy			LiDAR Forecasting
Model	mAP (%) $\uparrow$	SoftIoU (%) $\uparrow$	EPE (m) $\downarrow$	L1 (m) $\downarrow$	AbsRel (%) $\downarrow$	NFCD (m 2 ) $\downarrow$	CD (m 2 ) $\downarrow$
TRiO (2 Cam)	63.3	31.9	1.97	1.82	10.63	0.93	10.90
TRiO (4 Cam)	63.3	32.3	1.91	1.82	10.48	0.90	11.06

Table 11: Camera Ablation on AV2

	All			200m+
Method	EPE $\downarrow$	EPE (static) $\downarrow$	EPE (dynamic) $\downarrow$	EPE $\downarrow$	EPE (static) $\downarrow$	EPE (dynamic) $\downarrow$
DIO	27.8	0.01	28.8	28.4	0.01	29.3
TriO	19.4	0.17	20.3	19.6	0.16	20.5

Table 12 : Unsupervised flow forecasting results on LRR .

Model	NF-L1 $\downarrow$	FF-L1 $\downarrow$	NF-AbsRel (%) $\downarrow$	FF-AbsRel (%) $\downarrow$	NFCD $\downarrow$	FFCD $\downarrow$	CD $\downarrow$
UnO	2.07	2.07	11.69	3.01	1.03	79.28	11.41
DIO	1.87	2.20	11.28	3.28	0.97	87.07	12.43
TRiO	1.82	2.13	10.63	3.31	0.93	76.52	10.90

Table 13: AV2 LiDAR forecasting. Leaderboard metrics .

Figure 8 : Sensitivity to fusion parameter

\alpha_{\text{fusion}}

(STU)

Figure 9 : a) Warped occupancy by flow (hard scenario). b) Flow degeneracy: longitudinal bias failing to resolve orthogonal motion.

Figure 10 : TriO qualitative results on AV2 across a variety of diverse scenes: 1) large group of pedestrians in the snow, 2) excavator, 3) pedestrian at street corner and detailed structures, 4) pedestrian with trees, 5) different vehicle types and pedestrian walking on street, 6) pedestrians crossing street at intersection.

Figure 11 : Additional TriO qualitative results on AV2: 7) closeby pedestrians and detailed static structures, 8) scooter in middle of road, 9) pedestrian pulling a box, 10) vehicles and pedestrians on sidewalk, 11) pylons, parked vehicles, and pedestrians on sidewalk.

Figure 12 : Pseudo-labels of our method in comparison to baselines across a variety of scenarios. Oracle denotes upper-performance bound baselines with additional text prompts of Table 9 . To fit all the labels into the image we use the following naming convention: MCPP + SAM

\rightarrow

MSAM, MCPP + FC-CLIP

\rightarrow

MFCC, MCPP + MAFT+

\rightarrow

MAFT, Grounded SAM

\rightarrow

GSAM, OWLv2 + SAM

\rightarrow

OSAM. In this figure, there are a number of scenes including: 1) a pedestrian with debris on the road, 2) a box on the road, 3) a pedestrian with trash on the road, 4) umbrella with object behind.

Figure 13 : Additional pseudo-labels of our method in comparison to baselines, following the same naming convention as in Figure 12 : 5) fallen object and pedestrian crouching, 6) toy car in the road, 7) pile of garbage, 8) box with debris on it.

Figure 14 : Zero-shot TriO outputs on a roll-out of frames from a STU validation sequence. The columns include (from left to right): the reference image, the TriO rendered depth image, and the TriO rendered obstacle segmentation image. This example showcases TriO ’s ability to generalize to unseen country road scenes, even though it was trained on urban data (AV2).

Figure 15 : Zero-shot TriO outputs on a roll-out of frames from a STU validation sequence. The columns include (from left to right): the reference image, the TriO rendered depth image, and the TriO rendered obstacle segmentation image. This figure showcases an example of small debris being perceived as an obstacle - where even moving debris is properly segmented.

Figure 16 : A qualitative visualization of TriO outputs on AV2 across a rolled out sequence. The columns include (from left to right): the reference image, the TriO rendered depth image, and the TriO rendered obstacle segmentation image. This figure showcases an example of TriO perceiving a vulnerable road user.

Figure 17 : Ground truth (GT) vs predicted LiDAR of TriO on AV2.

Figure 18 : Visualization of failure cases of TriO on AV2 . A) a false positive obstacle detected from vehicular exhaust. B) false positive obstacles detected from snow.

Figure 19 : Visualization of a failure case of TriO on STU. A) what appears to be foam pool noodles that are segmented as ground - a false negative.

Figure 20 : TriO inference results for 1. speed bump, 2. hilly roads.

TriO: Tri-Modal Unsupervised Occupancy World Model for Anything Perception

Abstract

Figures & tables

Explore similar work

FreeOcc: Training-Free Embodied Open-Vocabulary Occupancy Prediction

O3N: Omnidirectional Open-Vocabulary Occupancy Prediction

CascadeOcc: Rethinking 3D Occupancy World Models with Cascaded VQ Representations