Hard-Region Supervision: #1 on the Waymo Open Dataset 2D Video Panoptic Segmentation Leaderboard

Jinghan Yang

cs.CVSep 30, 2026

Hard-Region Supervision: #1 on the Waymo Open Dataset 2D Video Panoptic Segmentation Leaderboard

Authors: Jinghan Yang

Abstract

We describe our winning entry to the Waymo Open Dataset 2D Video Panoptic Segmentation Challenge. The task asks for a semantic class at every pixel of every frame and, for countable objects, an identity that holds across 100 frames and across five overlapping cameras. We build on DVIS++, a cascade of a segmenter, a tracker, and a refiner, as our baseline. We propose hard region supervision (HRS) to improve the baseline. In particular, we use the baseline to define the hard region as where it makes mistakes, and design a loss and an auxiliary prediction head for this region. The auxiliary head is used only in training and removed at test time, so at inference the model trained with HRS has the same architecture as the baseline. In addition, we propose three test-time steps that further improve the results: a two-model ensemble, a merge of the segmenter's output into the final panoptic map, and cross-camera identity linking. On the challenge test set, our entry reaches 0.3547 wSTQ, 0.2071 wAQ, and 0.6075 mIoU, ranking first on all three metrics. It is 3.6 wSTQ points ahead of the second entry and 2.4 points ahead of our DVIS++ baseline.

Figures & tables

1:	initialise a union–find structure $\mathcal{U}$ over all thing instances
2:	for each adjacent pair $(A,B)\in\mathcal{P}$ do
3:		$H\leftarrow\textsc{Homography}\big(\textsc{OrbMatch}(I_{A},I_{B})\big)$ // geometry, from RGB only
4:		$P_{A}^{\prime}\leftarrow\textsc{Warp}(P_{A},H)$ // $A$ ’s predicted map, now in $B$ ’s view
5:		for each thing instance $a$ in $P_{A}^{\prime}$ do
6:			for each thing instance $b$ in $P_{B}$ do
7:				if $\mathrm{class}(a)=\mathrm{class}(b)$ and $\|\mathrm{mask}(a)\cap\mathrm{mask}(b)\|\geq\tau$ then

Algorithm 1 Cross-camera alignment of overlapping objects

Figure 1: Pixel distribution of the 28 semantic categories, reproduced from Figure 2 of Mei et al. [ 30 ] . The vertical axis is the number of pixels per class on a logarithmic scale; classes are grouped by super-class, and diamonds mark the classes for which instance IDs are provided.

method	wSTQ	wAQ	mIoU
DVIS_Plus_M1_NN_fusion (ours)	0.3547	0.2071	0.6075
Pano-CTVPS	0.3187	0.1781	0.5702
Jay_Street_PVPS_v0.1	0.2929	0.1486	0.5772
AmapNet-v_230523181741	0.2757	0.1466	0.5185
AmapNet-v_230501183044	0.2428	0.1039	0.5672
ViP-DeepLab Baseline S	0.1750	0.1062	0.2883

Table 1: Waymo Open Dataset 2D Video Panoptic Segmentation leaderboard on the test set, ordered by wSTQ.

\dagger

our internal baseline, not a leaderboard entry.

	Baseline-segmenter	$+$ SE	$+$ SE, HRT f (ours)
mIoU	0.6290	0.6353	0.6445
$\Delta$	—	$+0.0063$	$+0.0092$

Table 2: The baseline, the two-model ensemble, and the two-model ensemble whose second model is trained with the hard-region loss. Each

\Delta

is against the column to its left, so the first isolates ensembling and the second isolates the hard region.

class	Baseline-segmenter	Ours	$\Delta$	GT%
trailer	0.6432	0.6641	$+0.0209$	0.022
construction cone	0.5536	0.5875	$+0.0339$	0.062
ego vehicle	0.0000	0.0000	$+0.0000$	0.004
cyclist	0.8404	0.8583	$+0.0179$	0.018
motorcycle	0.7653	0.7828	$+0.0175$	0.020
bicycle	0.7319	0.7502	$+0.0183$	0.028

Table 3: Per-class IoU on the rare classes

\mathcal{R}

: the baseline versus ours, the SE with the HRT f -trained second model. Classes are ordered by pixel count. GT% is the class’s share of all ground-truth pixels. The last three rows compare

\mathcal{R}

with its complement and with all 28 classes.

model	wSTQ	wAQ	mIoU
baseline	0.3379	0.1929	0.5919
$+$ DIM (ours)	0.3590	0.2049	0.6290
$+$ SE-HRT f , DIM (ours)	0.3654	0.2071	0.6445

Table 4: Official Waymo metric on the 20 validation videos, cross-camera aligned, offline setting . baseline : DVIS++ trained and run as released.

+

DIM : the baseline model with decoupled inference merge at test time.

+

DIM (ours), HRT f (ours) : the segmenter trained with the hard-region loss and the restart ensemble, with DIM at test time.

Figure 2: The panoptic prediction on one frame of the front camera, coloured by semantic class. Left to right: the camera image, the ground truth, the baseline, and ours; three regions—a pedestrian group, a bus and car, and a road marker—are magnified beside each panel, with the class legend below. Black is VOID:undef , which a method emits where no retained query claims the pixel; it can appear only under the VPS readout of Section 2.1 , never under VSS. The baseline leaves large VOID:undef regions over the vehicles and across the pedestrian group, where ours commits to a class.

method	wSTQ	wAQ	mIoU
baseline	0.2712	0.1297	0.5669
$+$ HRT g (ours)	0.2843	0.1449	0.5579
$+$ SE-HRT f ,DIM (ours)	0.3113	0.1503	0.6445
$+$ HRT g , SE-HRT f , DIM (ours)	0.3225	0.1614	0.6445

Table 5: Hard-region loss on the referring tracker (HRL T ) and decoupled inference, online setting ; the temporal refiner is not used. Each “

+

” row adds the named component to the baseline. Best wSTQ in bold.

Figure 3: Cross-camera overlap objects on one frame. Top: the five camera images, arranged left to right in panoramic order. Middle: the ground-truth overlap objects. Bottom: the overlap objects recovered by Algorithm 1 , that is, the objects found to span more than one camera; each is shaded by its global id, so an object keeps a single color across every camera it appears in. The two lower rows use matched colors, so an object present in both appears in the same color in each.

		IoU			Precision			Recall
Class	GT %	Baseline	Ours	$\Delta$	Baseline	Ours	$\Delta$	Baseline	Ours	$\Delta$
truck	0.53	0.5970	0.7110	+0.1140	0.6964	0.8147	+0.1183	0.8071	0.8481	+0.0410
bird	0.00	0.4877	0.5468	+0.0592	0.6326	0.7317	+0.0991	0.6804	0.6840	+0.0036
cone	0.04	0.5524	0.5860	+0.0337	0.6257	0.6479	+0.0222	0.8251	0.8598	+0.0347
lane-marker	0.44	0.6493	0.6755	+0.0262	0.8016	0.8254	+0.0238	0.7736	0.7881	+0.0145
road-marker	0.60	0.6233	0.6492	+0.0259	0.8271	0.8345	+0.0074	0.7167	0.7451	+0.0284

Table 6: Per-class IoU, precision and recall on the validation split: the baseline against ours, the two-model ensemble with the hard-region second model. Classes are ordered by change in IoU; the two below the rule are the only ones whose IoU the ensemble degrades. GT % is each class’s share of the ground-truth pixels.

model	mIoU
baseline-segmenter	0.6290
baseline-refiner	0.6130

Table 7: Video semantic segmentation (VSS) inference for both modules, on the 20 validation videos.

Group	#classes	VPS	VSS	$\Delta$
Thing	8	0.5294	0.5515	$+$ 0.0222
Stuff	20	0.6465	0.6376	$-$ 0.0089
All	28	0.5919	0.6130	$+$ 0.0211

Table 8: mIoU of the same refiner under VPS and VSS inference, on the 20 validation videos.

\Delta

is VSS minus VPS.

Explore similar work

Scene-Centric Unsupervised Video Panoptic Segmentation
Jun 3, 2026Christoph Reich, Oliver Hahn, Nikita Araslanov +4Video Object Segmentation Panoptic Segmentation
TU Munich · TU Darmstadt · MCML +3
Open-World Panoptic Segmentation
Dec 17, 2024Matteo Sodano, Federico Magistri, Jens Behley +1Panoptic Segmentation Scene Reasoning
Center for Robotics, University of Bonn, Germany, and the Lamarr Institute for Machine Learning and Artificial Intelligence, Germany
CRF Loss is How Networks Should Learn Boundaries in Weakly Supervised Segmentation
Sep 28, 2026Joshua Li, Yuri BoykovHigh Confidence Pseudo-Labels Boundaries
Cheriton School of Computer Science University of Waterloo

Jun 3, 2026, cs.CVJ/K move · Enter open · S save

Scene-Centric Unsupervised Video Panoptic Segmentation

Christoph Reich, Oliver Hahn, Nikita Araslanov, Laura Leal-Taixé+3

TU Munich · TU Darmstadt · MCML+3

Video Object Segmentation Panoptic Segmentation

Open paper AskPDF arXiv

Video panoptic segmentation (VPS) aims to jointly detect, segment, and track all objects while partitioning the video into semantically consistent regions. We introduce the task setting of unsupervised VPS, omitting any human supervision. Existing unsupervised scene understanding works mainly focused on image segmentation tasks; the video domain remains underexplored. We propose VideoCUPS, the first unsupervised VPS approach. VideoCUPS generates temporally consistent panoptic video pseudo-labels from scene-centric videos by exploiting unsupervised depth, motion, and visual cues. Training on these pseudo-labels using a novel Video DropLoss yields an accurate, unsupervised VPS model. To benchmark progress, we introduce a comprehensive evaluation protocol and four competitive baselines, extending state-of-the-art unsupervised panoptic image and instance video segmentation models to VPS. VideoCUPS outperforms all baselines and demonstrates strong label-efficient learning. With VideoCUPS, our evaluation protocol, and baselines, we provide a strong foundation for future research on unsupervised VPS.

Abstract

Explore similar work

Scene-Centric Unsupervised Video Panoptic Segmentation

Open-World Panoptic Segmentation

CRF Loss is How Networks Should Learn Boundaries in Weakly Supervised Segmentation