TriO: Tri-Modal Unsupervised Occupancy World Model for Anything Perception

Quinlan Sykora; Sourav Biswas; Christopher Diehl; Andrew Cunningham; Thomas Gilles; Raquel Urtasun

cs.CVSep 25, 2026

TriO: Tri-Modal Unsupervised Occupancy World Model for Anything Perception

Authors: Quinlan Sykora, Sourav Biswas, Christopher Diehl, Andrew Cunningham, Thomas Gilles, Raquel Urtasun

Organizations: Waabi · University of Toronto

Abstract

We present TriO, a multi-modal unsupervised world model that predicts 4D occupancy, obstacle segmentation, flow and LiDAR. In contrast to prior work, TriO utilizes three distinct sensor modalities (camera, LiDAR, and RADAR) as both inputs and sources of self-supervision, eliminating the need for additional human annotations. Thanks to its novel supervision, the model is able to segment any occupancy from the drivable surface, overcoming the limitations of existing open-set methods in handling long-tail objects. TriO achieves state-of-the-art results in multiple 3D and 4D tasks, including occupancy, flow, and LiDAR prediction, as well as zero-shot road obstacle segmentation across multiple datasets such as Argoverse 2, and Spotting the Unexpected.

Figures & tables

Figure 1 : We present TriO , an unsupervised tri-modal occupancy world model designed to perceive and forecast anything that could be considered path-blocking.

Figure 2 : TriO ’s multi-modal architecture fusing LiDAR, RADAR, and camera inputs via voxel-attention. An implicit decoder generates occupancy, segmentation, and flow for a set of query points $\mathcal{Q}$ supervised by pseudo-labels derived from sensor data.

Figure 3 : Obstacle and ground pseudo-label generation process.

Panel of Figure 4 — Figure 4 : Visualization of TriO on AV2. Each row shows (left to right): perspective-view occupancy, perspective-view obstacle segmentation, camera image, rendered depth, and rendered obstacle segmentation. TriO captures key elements of the scenes: A) excavator, B) fallen cone, C) inside of truck, D) open car door, E) pedestrians.

Table 5

Panel of Figure 5 — Figure 5 : Segmentation results on the STU validation set. Each scene contains an unexpected obstacle: A) two pigeons and B) chair, C) ball, and D) bookbag. An obstacle segmentation is correct if the road is labeled traversable ( blue ) and the obstacle is fully segmented as an obstacle ( orange ). ISL robustly segments long-tail obstacles.

Panel of Figure 6 — Figure 6 : Scene forecasting results of TriO on AV2. LiDAR and camera sensor inputs are displayed at present time for reference. TriO ’s forecasts maintain coherent obstacle segmentation throughout time, distinguishing occupied from free space even as agents move. TriO captures a bicyclist A) and predicts its future multimodality B) .

	L1 ( $\text{\,}\mathrm{m}$ ) $\downarrow$	AbsRel ( $\text{\,}\mathrm{\%}$ ) $\downarrow$	NFCD ( $\text{\,}{\mathrm{m}}^{2}$ ) $\downarrow$	CD ( $\text{\,}{\mathrm{m}}^{2}$ ) $\downarrow$
RayTracing	4.88	35.00	3.62	17.03
$\text{occformer}\_\text{ep15}$	3.57	22.00	3.35	91.61
4D-Occ	3.22	19.00	2.45	72.74
occformer	2.97	17.00	2.03	71.33
Progressive ARM	2.32	13.00	1.81	71.41
UnO	2.24	12.00	0.86	8.10

Table 3 : LiDAR forecasting on the Argoverse 2 leaderboard.

	AV2			LRR
				All			200m+
Method	mAP $\uparrow$	Soft-IoU $\uparrow$	EPE $\downarrow$	mAP $\uparrow$	Soft-IoU $\uparrow$	EPE $\downarrow$	mAP $\uparrow$	Soft-IoU $\uparrow$	EPE $\downarrow$
Implicit-O	45.3	18.9	3.04	49.1	22.6	5.69	29.5	10.7	10.2
UnO	59.0	29.2	2.04	51.6	21.2	4.92	27.2	8.30	8.29
DIO	57.6	26.2	1.43	58.0	31.5	3.01	36.2	13.8	4.40
TriO	63.3	31.9	1.97	60.5	35.7	2.86	42.6	21.4	4.01

Table 4 : Fine-tuning downstream performance on AV2 and LRR .

	All			200m+
Method	mAP $\uparrow$	Soft-IoU $\uparrow$	EPE $\downarrow$	mAP $\uparrow$	Soft-IoU $\uparrow$	EPE $\downarrow$
TriO (no pretraining)	59.7	33.9	4.09	41.0	19.8	4.70
TriO (no radar input)	57.7	31.9	3.04	34.5	14.3	4.59
TriO (no camera input)	60.5	35.7	2.68	42.2	20.9	3.64
TriO (no radar loss)	60.5	35.6	4.35	42.6	21.3	6.02
TriO	60.5	35.7	2.86	42.6	21.4	4.01

Table 5 : Ablation studies of TriO on LRR .

Panel of Figure 7 — Figure 7 : Occupancy fine-tuning under a varying number of labelled frames.

Method	Input			Supervision			Semantics	Continuous	4D	Unsupervised
	Camera	LiDAR	RADAR	Camera	LiDAR	RADAR
ImplicitO [ 2 ]	✗	✓	✗	✗	✗	✗	✓	✓	✗	✗
Occ4D [ 43 ]	✗	✓	✗	✗	✓	✗	✗	✓	✓	✓
UnO [ 1 ]	✗	✓	✗	✗	✓	✗	✗	✓	✓	✓
DIO [ 18 ]	✗	✓	✗	✗	✓	✗	✓	✓	✓	(✗) ∗
GASP [ 60 ]	✗	✓	✗	✓	✓	✗	✗	✓	✓	✗

Table 6 : Comparison of TriO against the most closely related methods. * Only DIO-

\varnothing

does not use human labels. † QueryOcc+ uses both camera and LiDAR supervision. ‡ The used segmentation models rely annotations like per pixel semantic masks or image-text pairs annotations. ⋄ Continuous Gaussian coordinates discretized via voxelization

Symbol	Description	Value
$\tau_{n}$	Normal orientation transition threshold	$0.85$
$k_{n}$	Normal orientation sigmoid steepness	$20.0$
$\tau_{g}$	Normal smoothness transition threshold	$0.05$
$k_{g}$	Normal smoothness sigmoid steepness	$70.0$
$\tau_{c}$	Depth confidence transition threshold	$0.7$
$k_{c}$	Depth confidence sigmoid steepness	$100.0$

Table 7 : Obstacle segmentation hyperparameters.

Class	Text prompts
Super classes
vehicle	vehicle, car, truck, bicycle, motorcycle, other-vehicle, jeep, SUV, van,
vehicle	bike, moped, pickup truck, caravan, trailer, bus, tram, train,
	construction vehicle, crane, excavator
human	human, person, bicyclist, motorcyclist, pedestrian, bicycle rider,
human	motorcycle rider

Table 8 : Default terms/ text prompts used for open-vocabulary label mapping. Those colored as orange correspond to an obstacle class, and those colored as blue correspond to a ground class. Table design inspired from [ 69 ] .

Category	Oracle term/ text prompt
Seating	Chair, Office chair, Roller chair, Gaming chair
Bags	Dior bag, Shopping bag, Paper bag, Plastic bag, Polkadot bag,
	Banana bag, Zebra bag, Inflatable bag, Briefcase, Sleeping bag,
	Workout bag, Backpack
Inflatables	Inflatable toy, Inflatable ball, Inflatable, Pool noodle
Balls	Ball, Beach ball

Table 9: Oracle obstacle text prompts/ terms hand picked for STU, separated by category. These prompts are added to

\mathcal{P}_{\text{obstacle}}

got the oracle baselines.

Method	mAP $\uparrow$	Soft-IoU $\uparrow$	EPE $\downarrow$	mAP $\uparrow$	Soft-IoU $\uparrow$	EPE $\downarrow$	# Params	Mem (MB) $\downarrow$	Runtime (ms) $\downarrow$	Runtime (ms) $\downarrow$
	All			200m+					Planner [ 6 ]	Eval
UnO	51.6	21.2	4.92	27.2	8.30	8.29	39,742,221	2,493	63.92	1774.41
DIO	58.0	31.5	3.01	36.2	13.8	4.40	8,205,109	2,930	68.21	1776.86
TriO (no camera)	60.5	35.7	2.68	42.2	20.9	3.64	12,930,483	3,449	60.81	1622.54
TriO (no radar)	57.7	31.9	3.04	34.5	14.3	4.59	54,224,584	9,907	488.06	2048.88
TriO	60.5	35.7	2.86	42.6	21.4	4.01	54,224,967	9,915	489.32	2054.63

Table 10 : Model parameters, runtime, and GPU memory for two query-point settings (Planner [ 6 ] : 20k points; Eval: 2048k points), and fine-tuning downstream performance on LRR .

	4D Semantic Occupancy			LiDAR Forecasting
Model	mAP (%) $\uparrow$	SoftIoU (%) $\uparrow$	EPE (m) $\downarrow$	L1 (m) $\downarrow$	AbsRel (%) $\downarrow$	NFCD (m 2 ) $\downarrow$	CD (m 2 ) $\downarrow$
TRiO (2 Cam)	63.3	31.9	1.97	1.82	10.63	0.93	10.90
TRiO (4 Cam)	63.3	32.3	1.91	1.82	10.48	0.90	11.06

Table 11: Camera Ablation on AV2

	All			200m+
Method	EPE $\downarrow$	EPE (static) $\downarrow$	EPE (dynamic) $\downarrow$	EPE $\downarrow$	EPE (static) $\downarrow$	EPE (dynamic) $\downarrow$
DIO	27.8	0.01	28.8	28.4	0.01	29.3
TriO	19.4	0.17	20.3	19.6	0.16	20.5

Table 12 : Unsupervised flow forecasting results on LRR .

Model	NF-L1 $\downarrow$	FF-L1 $\downarrow$	NF-AbsRel (%) $\downarrow$	FF-AbsRel (%) $\downarrow$	NFCD $\downarrow$	FFCD $\downarrow$	CD $\downarrow$
UnO	2.07	2.07	11.69	3.01	1.03	79.28	11.41
DIO	1.87	2.20	11.28	3.28	0.97	87.07	12.43
TRiO	1.82	2.13	10.63	3.31	0.93	76.52	10.90

Table 13: AV2 LiDAR forecasting. Leaderboard metrics .

Panel of Figure 8 — Figure 8 : Sensitivity to fusion parameter $\alpha_{\text{fusion}}$ (STU)

Panel of Figure 9 — Figure 9 : a) Warped occupancy by flow (hard scenario). b) Flow degeneracy: longitudinal bias failing to resolve orthogonal motion.

Panel of Figure 10 — Figure 10 : TriO qualitative results on AV2 across a variety of diverse scenes: 1) large group of pedestrians in the snow, 2) excavator, 3) pedestrian at street corner and detailed structures, 4) pedestrian with trees, 5) different vehicle types and pedestrian walking on street, 6) pedestrians crossing street at intersection.

Panel of Figure 11 — Figure 11 : Additional TriO qualitative results on AV2: 7) closeby pedestrians and detailed static structures, 8) scooter in middle of road, 9) pedestrian pulling a box, 10) vehicles and pedestrians on sidewalk, 11) pylons, parked vehicles, and pedestrians on sidewalk.

Panel of Figure 12 — Figure 12 : Pseudo-labels of our method in comparison to baselines across a variety of scenarios. Oracle denotes upper-performance bound baselines with additional text prompts of Table 9 . To fit all the labels into the image we use the following naming convention: MCPP + SAM $\rightarrow$ MSAM, MCPP + FC-CLIP $\rightarrow$ MFCC, MCPP + MAFT+ $\rightarrow$ MAFT, Grounded SAM $\rightarrow$ GSAM, OWLv2 + SAM $\rightarrow$ OSAM. In this figure, there are a number of scenes including: 1) a pedestrian with debris on the road, 2) a box on the road, 3) a pedestrian with trash on the road, 4) umbrella with object behind.

Panel of Figure 13 — Figure 13 : Additional pseudo-labels of our method in comparison to baselines, following the same naming convention as in Figure 12 : 5) fallen object and pedestrian crouching, 6) toy car in the road, 7) pile of garbage, 8) box with debris on it.

Panel of Figure 14 — Figure 14 : Zero-shot TriO outputs on a roll-out of frames from a STU validation sequence. The columns include (from left to right): the reference image, the TriO rendered depth image, and the TriO rendered obstacle segmentation image. This example showcases TriO ’s ability to generalize to unseen country road scenes, even though it was trained on urban data (AV2).

Panel of Figure 15 — Figure 15 : Zero-shot TriO outputs on a roll-out of frames from a STU validation sequence. The columns include (from left to right): the reference image, the TriO rendered depth image, and the TriO rendered obstacle segmentation image. This figure showcases an example of small debris being perceived as an obstacle - where even moving debris is properly segmented.

Panel of Figure 16 — Figure 16 : A qualitative visualization of TriO outputs on AV2 across a rolled out sequence. The columns include (from left to right): the reference image, the TriO rendered depth image, and the TriO rendered obstacle segmentation image. This figure showcases an example of TriO perceiving a vulnerable road user.

Panel of Figure 17 — Figure 17 : Ground truth (GT) vs predicted LiDAR of TriO on AV2.

Panel of Figure 18 — Figure 18 : Visualization of failure cases of TriO on AV2 . A) a false positive obstacle detected from vehicular exhaust. B) false positive obstacles detected from snow.

Figure 19 : Visualization of a failure case of TriO on STU. A) what appears to be foam pool noodles that are segmented as ground - a false negative.

Panel of Figure 20 — Figure 20 : TriO inference results for 1. speed bump, 2. hilly roads.

Explore similar work

Apr 30, 2026cs.RO

FreeOcc: Training-Free Embodied Open-Vocabulary Occupancy Prediction

Existing learning-based occupancy prediction methods rely on large-scale 3D annotations and generalize poorly across environments. We present FreeOcc, a training-free framework for open-vocabulary occupancy prediction from monocular or RGB-D sequences. Unlike prior approaches that require voxel-level supervision and ground-truth camera poses, FreeOcc operates without 3D annotations, pose ground truth, or any learning stage. FreeOcc incrementally builds a globally consistent occupancy map via a four-layer pipeline: a SLAM backbone estimates poses and sparse geometry; a geometrically consistent Gaussian update constructs dense 3D Gaussian maps; open-vocabulary semantics from off-the-shelf vision-language models are associated with Gaussian primitives; and a probabilistic Gaussian-to-occupancy projection produces dense voxel occupancy. Despite being entirely training-free and pose-agnostic, FreeOcc achieves over

2\times

improvements in IoU and mIoU on EmbodiedOcc-ScanNet compared to prior self-supervised methods. We further introduce ReplicaOcc, a benchmark for indoor open-vocabulary occupancy prediction, and show that FreeOcc transfers zero-shot to novel environments, substantially outperforming both supervised and self-supervised baselines. Project page: https://the-masses.github.io/freeocc-web/.

Zeyu Jiang, Changqing Zhou, Xingxing Zuo +1

The Hong Kong University of Science and Technology (Guangzhou) · MBZUAI

3D Semantic Occupancy Prediction Occupancy 3D Gaussian

Ask PDF

Mar 12, 2026cs.CV

O3N: Omnidirectional Open-Vocabulary Occupancy Prediction

Understanding and reconstructing the 3D world through omnidirectional perception is becoming increasingly important for autonomous agents and embodied systems. However, existing 3D occupancy prediction methods are constrained by limited perspective inputs and a predefined training distribution, making them difficult to apply to embodied agents that require comprehensive and safe perception of scenes in open-world exploration. To address this, we present O3N, the first framework for open-vocabulary occupancy prediction from a single omnidirectional RGB image. O3N embeds omnidirectional voxels in a polar-spiral topology via the Polar-spiral Mamba (PsM) module, enabling continuous spatial representation and long-range context modeling across 360°. The Occupancy Cost Aggregation (OCA) module introduces a principled mechanism for unifying geometric and semantic supervision within the voxel space, ensuring consistency between the reconstructed geometry and the underlying semantic structure. Moreover, Natural Modality Alignment (NMA) establishes a gradient-free alignment pathway that harmonizes visual features, voxel embeddings, and text semantics, forming a consistent ``pixel-voxel-text'' representation triad. Extensive experiments on multiple models demonstrate that our method not only achieves state-of-the-art performance on QuadOcc and Human360Occ benchmarks but also exhibits remarkable cross-scene generalization and semantic scalability, highlighting the potential of O3N for scalable open-world 3D scene understanding. The source code will be made publicly available at https://github.com/MengfeiD/O3N.

Mengfei Duan, Hao Shi, Fei Teng +4

School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, Changsha, China · Ant Group, Hangzhou, China · State Key Laboratory of Extreme Photonics and Instrumentation, Zhejiang University, Hangzhou, China

3D Semantic Occupancy Prediction 3D Scene Understanding Voxel

Ask PDF

Jun 26, 2026cs.CV

CascadeOcc: Rethinking 3D Occupancy World Models with Cascaded VQ Representations

This letter proposes CascadeOcc, a novel occupancy world model that prioritizes intrinsic structural hierarchy over extrinsic auxiliary modalities for autonomous driving. Occupancy world models -- forecasting the future driving environment and planning the driving trajectory -- effectively bridge perception and planning, but current approaches often heavily rely on external modalities or large language models, failing to fully exploit the inherent structural potential of occupancy representations themselves. To enhance representational capacity for complex 3D scenes, we integrate a cascaded Vector Quantized (VQ) mechanism into an autoregressive framework. Following a coarse-to-fine principle, CascadeOcc progressively refines fine-grained details from global structures through a multi-scale architecture. Additionally, we incorporate a TimeMixer to capture multi-scale temporal dependencies, establishing a dual-hierarchy mechanism in both space and time. Experimental results on 4D occupancy forecasting and motion planning benchmarks demonstrate that CascadeOcc achieves superior performance among vision-centric approaches, validating that optimizing inherent representations is a powerful alternative to relying on external foundation models.

Kyumin Hwang, Wonhyeok Choi, Jaeyeul Kim +3

Daegu Gyeongbuk Institute of Science and Technology (DGIST), Daegu 42988, South Korea

3D Semantic Occupancy Prediction Occupancy Autonomous Driving Latent World Models Motion Planning Residual Vector Quantization

Ask PDF