IGLOSS: Image Generation for Lidar Open-vocabulary Semantic Segmentation

Nermin Samet; Gilles Puy; Renaud Marlet

cs.CVApr 1, 2026

IGLOSS: Image Generation for Lidar Open-vocabulary Semantic Segmentation

Authors: Nermin Samet, Gilles Puy, Renaud Marlet

Organizations: Valeo.ai, Paris, France · LIGM, Ecole des Ponts, Univ Gustave Eiffel, CNRS, Marne-la-Vallée, France

Abstract

This paper presents a new method for the zero-shot open-vocabulary semantic segmentation (OVSS) of 3D automotive lidar data. To circumvent the recognized image-text modality gap that is intrinsic to approaches based on Vision Language Models (VLMs) such as CLIP, our method relies instead on image generation from text, to create prototype images. Given a 3D network distilled from a 2D Vision Foundation Model (VFM), we then label a point cloud by matching 3D point features with 2D image features of these prototypes. Our method is state-of-the-art for OVSS on nuScenes and SemanticKITTI. Code, pre-trained models and generated images are available at https://github.com/valeoai/IGLOSS.

Figures & tables



Wheelchair	Stroller	Crosswalk

Figure 1: Examples of segment retrieval for arbitrary classes. As IGLOSS is OV, it can segment any class defined by a free text prompt. We present (points reprojected onto images) OV results for usually-ignored classes wheelchair , stroller , and crosswalk , which are not included in the 16 official nuScenes classes [ 5 ] . Crosswalks are not even identified in nuScenes and are just labeled as any driveable surface . Although crosswalk points are geometrically on the road plane, their features identifies them specifically because the 2D-3D distillation takes lidar intensity into account [ 80 ] .

Figure 2: IGLOSS vs typical 3D OVSS methods. Closed-set 3D OVSS methods (top) train a 3D network on pseudo-labels provided by a 2D OVSS method. Typical 3D OVSS methods (middle) distill a 2D VLM into a 3D network, leveraging some mask generator at training time; inference consists in matching text features to 3D features, on a nearest prompt basis (NN). Our method, IGLOSS (bottom), exploits a 2D VFM distilled into a 3D network, making it a 3D VFM. Inference consists in generating prototype images from text prompts and matching 2D features of these images to 3D features, using test-time multinomial logistic regression (LR).

		Image	Extra	Label	OV	No img	No scan	Code/
3D OVSS		-text	foundation	assign-	at test	at test	seq. at	chkpt
method		bridge	model(s) used	ment	time	time	test time	avail.
CNS	[ 9 ]	$\text{{\color[rgb]{0,0,1}{i}}}\,{\Rightarrow}\,\text{{\color[rgb]{0,0,0}$ f $}}\,{\Leftarrow}\,\text{{\color[rgb]{1,0,1}{t}}}$	$\text{{\color[rgb]{0,0,1}{i}}}\,{\Rightarrow}\,\text{{\color[rgb]{0.5,0,0.5}{m}}}$	NN	✗	✓	✓	$\Circle$
HICL	[ 41 ]	$\text{{\color[rgb]{0,0,1}{i}}}\,{\Rightarrow}\,\text{{\color[rgb]{0,0,0}$ f $}}\,{\Leftarrow}\,\text{{\color[rgb]{1,0,1}{t}}}$		NN	✗	✓	✓	$\Circle$
AdaCo	[ 124 ]	$\text{{\color[rgb]{0,0,1}{i}}}\,{\Rightarrow}\,\text{{\color[rgb]{1,0,1}{t}}}$	$\text{{\color[rgb]{0,0,1}{i}}}\,{\Rightarrow}\,\text{{\color[rgb]{0.5,0,0.5}{m}}}$ , $\text{{\color[rgb]{1,0,1}{t}}}\,{\Rightarrow}\,\text{{\color[rgb]{0,0,0}$ f $}}$	NN	✗	✓	✓	$\Circle$
AFOV	[ 93 ]	$\text{{\color[rgb]{0,0,1}{i}}}\,{\Rightarrow}\,\text{{\color[rgb]{0,0,0}$ f $}}\,{\Leftarrow}\,\text{{\color[rgb]{1,0,1}{t}}}$	$\text{{\color[rgb]{0,0,1}{i}}}\,{\Rightarrow}\,\text{{\color[rgb]{0.5,0,0.5}{m}}}$	CP	✗	(✓)	✓	( $\CIRCLE$ )

Table 1: Characteristics of 3D OVSS methods. Image-text bridge and Extra foundation models used: “

\text{{\color[rgb]{0,0,1}{i}}}\,{\Rightarrow}\,\text{{\color[rgb]{0,0,0}

}}\,{\Leftarrow}\,\text{{\color[rgb]{1,0,1}{t}}}

” = image and text encoders, producing aligned features; “

\text{{\color[rgb]{0,0,1}{i}}}\,{\Rightarrow}\,\text{{\color[rgb]{0.5,0,0.5}{m}}}

” = image mask generator; “

\text{{\color[rgb]{0,0,1}{i}}}\,{\Rightarrow}\,\text{{\color[rgb]{1,0,1}{t}}}

” = image captioner; “

(\text{{\color[rgb]{0,0,1}{i}}},\text{{\color[rgb]{1,0,1}{t}}})\,{\Rightarrow}\,\text{{\color[rgb]{0,1,1}{b}}}

” (resp. “

(\text{{\color[rgb]{0,0,1}{i}}},\text{{\color[rgb]{1,0,1}{t}}})\,{\Rightarrow}\,\text{{\color[rgb]{0.5,0,0.5}{m}}}

”) = image object detector producing bounding boxes (resp. masks) given classes defined with text prompts; “

\text{{\color[rgb]{1,0,1}{t}}}\,{\Rightarrow}\,\text{{\color[rgb]{0,0,1}{i}}}

” = image generator conditioned on text; “

\text{{\color[rgb]{0,0.88,0}{p}}}\,{\Rightarrow}\,\text{{\color[rgb]{0,0,0}

}}

” = point encoder. Label assignment: nearest neighbor (NN) to prototype features, class probability (CP) of underlying model, or multinomial logistic regression (LR) on prototype features. OV at test time: ✗ = the model is closed-set;

\boldsymbol{\pm}

= the model is biased at training time with a pre-defined vocabulary or with text prompts, but remains functionally OV; (✓) = the model is biased by a captioner but remains OV; ✓ = the model is fully OV, the classes of interest are decided at test time. No image at test time: ✓ = input is only 3D points; (✓) = 2D-3D ensembling for best results. No scan sequence at test time: ✓ = yes; ✗ = no. Code/checkpoint available:

\CIRCLE

= yes for both nuScenes (NS) and SemanticKITTI (SK); (

\CIRCLE

) = for NS, not for SK;

\LEFTcircle

= partial or with long-standing open issues;

\Circle

= no; † : upon publication.

Figure 3: Examples of generated images (see Section 4 ) with ChatGPT (top), Qwen Image (middle) and Gemini (bottom). Web images are not shown due to copyright restrictions.

(a) Open-vocabulary semantic segmentation
Method		OV	3D	ens	NS	SK
MaskCLIP $\rightarrow$ 3D	[ 124 ]	✓	✗	✗	16.6	8.1
3D-AVS (I+L)	[ 102 ]	(✓)	✗	✗	36.2	-
OpenScene (I+L)	[ 76 ]	✓	✗	✗	42.1	-
CLIP2Scene	[ 10 ]	$\boldsymbol{\pm}$	✓	✗	20.8	-
3D-AVS (L)	[ 102 ]	(✓)	✓	✗	33.4	-

Table 2: Test-time and train-time OVSS results (mIoU%) on the nuScenes (NS) and SemanticKITTI (SK) validation sets. We denote IGLOSS = IGLOSS(ScaLR + (DINOv2)), IGLOSS mix = IGLOSS(ScaLR + (DINOv2)

\varoplus

OpenScene), OpenScene is distilled from OpenSeg. OV: ✗ = not OV at test time;

\boldsymbol{\pm}

= bias at training time with a predefined vocabulary or text prompts, but functionally OV; (✓) = bias at training time from a captioner, but still OV; ✓ = fully OV. 3D: ✓ = input is only 3D points; ✗ = input includes images in addition to 3D points. ens: ✗ = no ensembling; ✓ = VFM ensembling (at train or test time). “-” = no such experiment reported.

Prototype	bus	truck	car	motorcycle	bicycle	road	pedestrian	terrain	sidewalk	vegetation	mIoU (%)
Text	27.7	26.2	26.1	25.2	24.9	24.4	23.9	23.4	23.1	22.0	24.7
Image	55.7	47.2	38.2	31.2	26.4	43.4	28.7	45.5	52.7	47.1	41.6

Table 3: CLIP alignment of text or generated-image prototypes with Cityscapes. We measure the cosine similarity of CLIP embeddings of class prototypes with image masks in Cityscapes val.

(a) 2D VFM evaluation
2D VFM		NN	NN ∗
OVDiff	[ 42 ]	N/A	23.5
SigLIP 2	[ 95 ]	18.7	23.7
OpenSeg	[ 29 ]	19.8	23.7
DiNOv2	[ 70 ]	23.3	25.1

Table 4: 2D-3D VFM evaluation. We generate one image per prompt with ChatGPT, except for OVDiff that generates 32 images using Stable Diffusion [ 84 ] . (a) We match 2D VFM features of prototypes and of nuScenes val set images, with nearest neighbor (NN), unproject classes into seen 3D points, and measure mIoU%. ∗ : OVDiff-like evaluation, i.e., with all foreground classes and one merged background class. (b) We match 2D VFM prototype features with point features of a distilled 3D VFM, using NN or LR, and measure mIoU%. As OpenSeg, hence also OpenScene, have the same feature space as CLIP, we also simply experimented with CLIP visual features.

Method		Drop	Non lin.	Head	Resol.	Epochs	LP	NN	LR
ScaLR	[ 80 ]	0	ReLU	lin.	224 $\times$ 448	25	64.3	37.1	40.1
		0.2	ReLU	lin.	224 $\times$ 448	25	65.4	39.9	42.3
$\vdots$		0.2	GELU	lin.	224 $\times$ 448	25	66.2	40.7	43.7
$\vdots$		0.2	GELU	MLP	448 $\times$ 896	25	67.9	43.1	45.7
		0.2	GELU	MLP	448 $\times$ 896	45	68.7	43.3	46.3
ScaLR +	(ours)	0.2	GELU	MLP	448 $\times$ 896	65	68.7	43.7	46.5

Table 5: ScaLR + ablation study. We study ScaLR + distillation of DINOv2 in WaffleIron [ 79 ] (WI-48-256) on the mini-train and mini-val sets of nuScenes defined in [ 87 , 80 ] , varying: the drop-path proportion (Drop), the type of non linearity (Non lin.), the backbone head (Head), the image resolution (Resol.) and the number of epochs (Epochs). We use a 2-layer MLP. Hidden dimension is 2048 as in [ 31 ] . We measure the quality of features extracted before the head with linear probing (LP) and the impact on IGLOSS when classifying with NN or LR.

Distill. method		ChatGPT		Qwen		Gemini		Web
Distill. method		NN	LR	NN	LR	NN	LR	NN	LR
ScaLR	[ 80 ]	34.8	39.7	34.7	39.1	32.9	34.5	32.1	32.3
ScaLR +	(ours)	42.9	46.6	38.9	43.5	42.2	42.3	38.4	37.3

Table 6: Impact of image source on IGLOSS (mIoU%) with NN or LR on NS val set. We evaluate 1 image per prompt .

Prototypes per class	(LR) mIoU%
1 $\times$ ChatGPT	46.6
2 $\times$ ChatGPT	47.5
ChatGPT + Qwen	47.3
ChatGPT + Gemini	47.9
ChatGPT + Web	47.7
3 $\times$ ChatGPT	47.2

Table 7: Ensembling prototypes from various sources. Tested on NS val set.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Method	mIoU%	barrier	bicycle	bus	car	const. veh.	motorcycle	pedestrian	traffic cone	trailer	truck	driv. surf.	other flat	sidewalk	terrain	manmade	vegetation
IGLOSS	47.5	53.4	21.7	63.5	67.4	39.9	60.9	52.2	37.0	15.8	31.6	86.0	6.8	45.3	46.7	68.4	63.7
IGLOSS mix	51.1	42.9	18.5	74.1	73.4	32.6	71.0	58.1	39.7	16.7	43.4	86.5	4.8	46.9	55.5	75.6	78.5
IGLOSS clo	49.6	56.7	23.6	64.6	69.0	41.9	63.8	58.6	39.9	16.3	32.7	87.2	7.3	46.6	49.1	70.2	66.2
IGLOSS mix,clo	54.1	43.7	21.3	75.8	74.3	38.0	76.0	66.7	51.3	16.8	46.4	88.1	4.6	49.6	56.8	75.8	79.5

Appendix

Table 7: Classwise semantic segmentation results (IoU%) of IGLOSS and its variants on nuScenes val set .

Method	mIoU%	car	bicycle	motorcycle	truck	other-vehicle	person	bicyclist	motorcyclist	road	parking	sidewalk	other-ground	building	fence	vegetation	trunk	terrain	pole	traffic-sign
IGLOSS	34.3	76.0	14.4	40.5	20.2	10.5	55.5	36.4	0.0	83.4	0.0	54.3	0.6	62.5	0.7	79.1	33.0	64.9	0.0	19.8
IGLOSS clo	39.4	87.8	21.6	46.2	19.8	11.2	70.6	53.2	0.0	89.6	0.0	64.9	0.7	68.3	0.6	84.6	39.3	69.3	0.0	20.7

Appendix

Table 8: Classwise semantic segmentation results (IoU%) of IGLOSS and its closed-set variant on SemanticKITTI val set . (There is no result on IGLOSS mix and IGLOSS mix,clo as there is no OpenScene training on SemanticKITTI.)

Template	Text prompt
$T_{\text{obj}}$ ( $c$ )	generate an image of $c$ with white background
$T_{\text{stuff}}$ ( $c$ )	generate an image of $c$ covering the whole image
$T_{\text{drive}}$ ( $p$ )	$p$ , similar to what you see along roadsides and in cities
$T_{\text{dr-obj}}$ ( $c$ )	$T_{\text{drive}}$ ( $T_{\text{obj}}$ ( $c$ ) )
$T_{\text{dr-stuff}}$ ( $c$ )	$T_{\text{drive}}$ ( $T_{\text{stuff}}$ ( $c$ ) )

Appendix

Table 9: Our context prompt templates used to generate prototype images.

Class	Templ.	Text prompts
car		nuScenes prompts/images for car
bicycle		nuScenes prompts/images for bicycle
motorcycle		nuScenes prompts/images for motorcycle
truck		nuScenes prompts/images for truck
other vehicle		nuScenes prompts/images for bus, trailer, construction vehicle
person		nuScenes prompts/images for pedestrian

Appendix

Table 11: Our classwise text prompts used to generate prototype images for SemanticKITTI. Templates

T_{\text{obj}}

and

T_{\text{dr-stuff}}

are defined in Table 10 . Only a few prompts are specific to SemanticKITTI; for other classes, the prompts are the same as in nuScenes for similar classes. In practice, we do not generate new images; we just reuse the same prototype images already generated for nuScenes.

		Code/	No.
3D OVSS		chkpt	templ.	No.
method		avail.	/subcl.	subcl.	Remark
CNS	[ 9 ]	$\Circle$	85	-
HICL	[ 41 ]	$\Circle$	85	-
AdaCo	[ 124 ]	$\Circle$	-	55
AFOV	[ 93 ]	( $\CIRCLE$ )	14	175

Appendix

Table 12: Number of prompt templates and subclass prompts used by 3D OVSS methods for nuScenes. Code/checkpoint available:

\CIRCLE

= yes for both nuScenes (NS) and SemanticKITTI (SK); (

\CIRCLE

) = for NS, not for SK;

\LEFTcircle

= partial or with long-standing open issues;

\Circle

= no; † : upon publication. No templ./subcl.: number of context prompt templates per subclass name. No subcl.: number of subclass names used to make the 16 classes of nuScenes explicit.

Prototype Number	(LR) mIoU%
1 $\times$ ChatGPT	46.6
2 $\times$ ChatGPT	47.3
3 $\times$ ChatGPT	47.2
4 $\times$ ChatGPT	46.9
5 $\times$ ChatGPT	46.5
8 $\times$ ChatGPT	47.4

Appendix

Table 13: Increasing the number of generated ChatGPT images per prompt. Performance (on NS val set) plateaus at two.

Random pair id	(LR) mIoU%
v1	47.3
v2	46.8
v3	47.5
v4	47.0
v5	47.0
Variance	0.08

Appendix

Table 14: Randomly selecting two prototypes for each class/prompt . The variance is very small between the selected random sets which shows the robustness of our method against generated templates.

Panel of Figure 6 — Figure 6: Qualitative results of semantic segmentation from the validation sets of nuScenes and SemanticKITTI. The color code used to represent each class is provided in Table 15 .

Panel of Figure 7 — Figure 7: Qualitative comparison of semantic segmentation on the nuScenes validation set using our method and OpenScene [ 76 ] . The color code used to represent each class is provided in Table 15 . Our method produces more faithful results by detecting other flat in the first row, segmenting trailer in the second row, and segmenting road and pedestrians in the third row.

Color	nuScenes class
	barrier	trailer
	bicycle	truck
	bus	driveable surface
	car	manmade
	construction vehicle	other flat
	motorcycle	sidewalk

Appendix

Table 15: color codes used to represent each class of nuScenes or SemanticKITTI in Figure 6 and Figure 7 .

Open-Vocabulary 3D Segmentation

Open-vocabulary 3D segmentation assigns language-queryable semantic and instance masks to 3D scenes, including objects unseen during training.

See all 74 papers

Lang3DSeg: Annotation-Free Open-Vocabulary 3D Segmentation with Point Transformers
Oct 1, 2026Cigdem Kokenoz, Amir Salarpour, Alkim Domeke +5Open-Vocabulary 3D Segmentation Transformer
Department of Automotive Engineering, Clemson University, Greenville, SC, USA. · School of Computing, Clemson University, Clemson, SC, USA.
dinov3.seg: Open-Vocabulary Semantic Segmentation with DINOv3
Mar 19, 2026Saikat Dutta, Biplab Banerjee, Hamid RezatofighiOpen-Vocabulary Semantic Segmentation Vision-Language Models
IITB-Monash Research Academy · IIT Bombay · Monash University
VaViT: Vanilla ViT for Automotive Point Cloud Semantic Segmentation
May 29, 2026Gilles Puy, Nermin Samet, Alexandre Boulch +33D Point Cloud Segmentation 3D ViTs
valeo.ai, Paris, France · LIGM, CNRS, Univ Gustave Eiffel, ENPC, IP Paris, France

Oct 1, 2026J/K move · Enter open · S save

Lang3DSeg: Annotation-Free Open-Vocabulary 3D Segmentation with Point Transformers

Cigdem Kokenoz, Amir Salarpour, Alkim Domeke, Christopher Salas+4

Department of Automotive Engineering, Clemson University, Greenville, SC, USA. · School of Computing, Clemson University, Clemson, SC, USA.

Open-Vocabulary 3D Segmentation Transformer 3D Semantic Segmentation Point Transformer

Open paper AskPDF arXiv

Accurate 3D semantic perception is critical for safe autonomous navigation. However, supervised LiDAR segmentation remains tied to closed taxonomies and to the cost of point-wise manual annotation. Open-vocabulary methods avoid that cost by projecting the output of 2D vision-language models onto LiDAR and distilling it into a 3D network. These methods rely almost exclusively on voxel-based sparse convolutions, and point transformers have so far been limited to indoor environments, where 3D data is dense and bounded. We present Lang3DSeg, which establishes a point transformer as the backbone for annotation-free open-vocabulary segmentation of outdoor 3D LiDAR, and is trained from scratch without geometric pre-training. This training paradigm necessitates addressing the inherent noise in 2D-to-3D label projections; specifically, naive projection often suffers from depth ambiguity, where points behind an object are erroneously assigned its semantic label. We therefore composite masks using an explicit class-priority rule and truncate each projected instance at the first gap in its depth distribution, correcting the projection error directly rather than averaging it over registered sequences. Lang3DSeg achieves 52.8% mIoU on nuScenes validation and 41.4% on SemanticKITTI, the highest among published annotation-free methods on both benchmarks. Every 3D semantic segmentation is on a single LiDAR sweep, and inference operates in real-time without running vision-language models.

Abstract

Open-Vocabulary 3D Segmentation

Explore similar work

Lang3DSeg: Annotation-Free Open-Vocabulary 3D Segmentation with Point Transformers

dinov3.seg: Open-Vocabulary Semantic Segmentation with DINOv3

VaViT: Vanilla ViT for Automotive Point Cloud Semantic Segmentation