Every View Counts: View-Consistent Panoptic Quality for Multi-view Panoptic Segmentation

Youngmin Lee; Byungha Ko; Guhnoo Yun; Dong Hwan Kim

cs.CVOct 5, 2026

Every View Counts: View-Consistent Panoptic Quality for Multi-view Panoptic Segmentation

Authors: Youngmin Lee, Byungha Ko, Guhnoo Yun, Dong Hwan Kim

Organizations: Korea University · Korea Institute of Science and Technology

Abstract

Multi-view panoptic segmentation assigns a semantic class and a scene-level instance ID to every pixel of an unordered set of images, and recent feed-forward 3D models predict these labels for the input views in a single forward pass. Their predictions, however, have been evaluated with the scene-level PQ (PQ^scene) borrowed from per-scene optimization methods, typically on rendered held-out views. PQ^scene tiles all views of a scene into a single image, so that a missed appearance or a change of ID lowers the score of the matched pair only in proportion to its area. We propose View-Consistent Panoptic Quality (VC-PQ), which extends PQ from a single image to a set of input views, counts equally every view in which an instance is visible, and penalizes a prediction that is not visible in the same views as its ground truth. A decomposition of VC-PQ attributes the score a method loses to mask accuracy, view consistency, and the matching threshold. A single additional parameter recovers the area weighting of tiling for comparison. Under a fixed evaluation protocol on ScanNet++ and ScanNetv2, recent feed-forward methods are evaluated with VC-PQ and PQ^scene, and the decomposition shows where each of them loses its score. Controlled perturbations of the ground truth show that VC-PQ responds to the number of views in which an instance is missed or changes ID, whereas PQ^scene responds to their area. The aim of this work is to make view consistency part of the evaluation of multi-view panoptic segmentation, with VC-PQ reported alongside PQ^scene.

Figures & tables

Figure 1: (a) Multi-view panoptic segmentation: every view of a scene is labeled with a class and a scene-level instance ID. (b) A toy example of an input set of three views: the orange ball is predicted only in the view in which it appears large, so PQ ${}^{\text{scene}}$ matches it by area whereas VC-PQ counts the two missed views and leaves it unmatched.

		PQ ${}^{\text{scene}}$				VC-PQ
Dataset	Method	$N{=}5$	$N{=}10$	$N{=}20$	$N{=}40$	$N{=}5$	$N{=}10$	$N{=}20$	$N{=}40$
ScanNet++	PanSt3R v1	43.7	45.7	48.9	50.4	22.0	19.2	17.9	17.4
	PanSt3R v2	46.8	51.3	54.7	57.7	24.5	22.2	20.7	20.0
	EPS3D	13.2	13.6	–	–	2.4	2.0	–	–
	IGGT	17.8	17.1	–	–	5.7	4.2	–	–
ScanNetv2	PanSt3R v2	50.1	52.5	55.2	57.4	43.7	41.5	40.6	40.6

Table 1: PQ

{}^{\text{scene}}

and VC-PQ of the same predictions for input sets of

N

views, all classes; ScanNet++ pools the DSLR and iPhone runs. –: out of memory at

N\geq 20

Figure 2: A door (PanSt3R v2, ScanNet++ DSLR, $N{=}10$ ) visible in five views and predicted in the two largest: PQ ${}^{\text{scene}}$ matches the pair at $\mathrm{IoU}_{\text{concat}}=0.95$ , VC-PQ leaves it unmatched at $q=0.39$ . The union area $a_{9}$ in $I_{9}$ is 84% of the total, so this view alone contributes 0.82 to $\mathrm{IoU}_{\text{concat}}$ , whereas VC-PQ counts the door as missed in three of its five views, $\mathrm{VC}=2/5$ .

	VC-PQ							PQ ${}^{\text{scene}\dagger}$	PQ ${}^{\text{scene}\dagger}$ $-$ VC-PQ, Shapley
area weighting	–	✓	–	–	✓	✓	–	✓		✓
per-scene avg.	–	–	✓	–	✓	–	✓	✓			✓
0.5% filter	–	–	–	✓	–	✓	✓	✓				✓
PanSt3R v1	19.2	25.0	24.2	26.0	29.7	35.6	37.2	45.8	+26.6	+7.3	+7.7	+11.5
PanSt3R v2	22.2	28.9	25.6	32.7	31.2	42.7	42.3	51.4	+29.2	+7.8	+6.0	+15.3
EPS3D	2.0	2.8	7.0	2.5	8.8	3.4	11.1	13.9	+11.9	+1.7	+7.6	+2.7

Table 2: VC-PQ with the three properties of PQ

{}^{\text{scene}}

switched on in every combination (ScanNet++,

N{=}10

, all classes). † Differs from PQ

{}^{\text{scene}}

of Table 1 only by the visibility threshold. Right block: Shapley attribution of the difference between the two named columns.

ScanNet++
	$N{=}5$				$N{=}10$				$N{=}20$				$N{=}40$
Method	VC-PQ	$\varepsilon_{\overline{\mathrm{IoU}}}$	$\varepsilon_{\mathrm{VC}}$	$\varepsilon_{q}$	VC-PQ	$\varepsilon_{\overline{\mathrm{IoU}}}$	$\varepsilon_{\mathrm{VC}}$	$\varepsilon_{q}$	VC-PQ	$\varepsilon_{\overline{\mathrm{IoU}}}$	$\varepsilon_{\mathrm{VC}}$	$\varepsilon_{q}$	VC-PQ	$\varepsilon_{\overline{\mathrm{IoU}}}$	$\varepsilon_{\mathrm{VC}}$	$\varepsilon_{q}$
PanSt3R v1	21.8	73.1	4.1	1.0	18.7	74.0	4.7	2.5	17.4	74.0	4.3	4.3	16.9	73.6	3.9	5.6
PanSt3R v2	24.7	68.7	5.2	1.4	22.3	67.5	7.2	3.0	20.6	66.7	7.5	5.2	19.9	66.3	7.0	6.8
EPS3D	0.6	98.8	0.5	0.1	0.3	99.1	0.4	0.2	–	–	–	–	–	–	–	–
IGGT	4.9	93.0	1.8	0.3	3.7	93.7	1.9	0.7	–	–	–	–	–	–	–	–

Table 3: Decomposition of

1-\text{VC-PQ}

by Eq. ( 7 ) for input sets of

N

views, thing classes; VC-PQ and the three terms sum to 100.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Scored as its own class (19)
bathtub	counter	picture	table
bed	curtain	refrigerator	toilet
bookshelf	desk	shower curtain	wall
cabinet	door	sink	window
chair	floor	sofa
Scored as otherfurniture (22)

Appendix

Table A1: The 41 prompts given to the open-vocabulary methods on ScanNetv2, in alphabetical order within each group (read down each column). The first group are the names of the other 19 classes, each scored as its own class. The second group are the 22 ScanNet200 categories ( Rozenberszki et al., 2022 ) that the official label mapping assigns to otherfurniture, all scored as otherfurniture.

	DSLR		iPhone
Method	PQ ${}^{\text{scene}}$	VC-PQ	PQ ${}^{\text{scene}}$	VC-PQ
PanSt3R v1	48.8	20.7	42.5	17.5
PanSt3R v2	54.1	22.1	48.5	22.5
EPS3D	13.2	1.9	14.1	2.1
IGGT	16.6	3.8	17.7	4.7

Appendix

Table A2: PQ

{}^{\text{scene}}

and VC-PQ on ScanNet++ at

N{=}10

for each camera, aggregated as in Table 1 . The DSLR and iPhone runs capture the same 50 scenes, and the iPhone values exclude the scene without usable ground truth (Sec. 5.1 ).

	ScanNet++		ScanNetv2
Method	PQ ${}^{\text{scene}}$	VC-PQ	PQ ${}^{\text{scene}}$	VC-PQ
PanSt3R v1	45.7 [43.3, 48.1]	19.2 [17.2, 21.2]	–	–
PanSt3R v2	51.3 [48.9, 53.8]	22.2 [20.5, 24.1]	52.5 [51.0, 54.0]	41.5 [39.8, 43.3]
EPS3D	13.6 [12.3, 15.2]	2.0 [1.8, 2.1] ∗	21.5 [20.8, 22.2]	8.2 [8.0, 8.4]
IGGT	17.1 [15.1, 19.7]	4.2 [3.6, 4.8]	–	–

Appendix

Table A3: 95% BCa confidence intervals ( Efron, 1987 ) for the scores of Table 1 at

N{=}10

. On ScanNet++ the bootstrap draws the 50 physical scenes with replacement and keeps the DSLR and iPhone runs of a drawn scene together; on ScanNetv2 it draws the 312 scenes. ∗ The bound lies at the most extreme of the

B{=}10{,}000

draws.

ScanNet++
	PQ ${}^{\text{scene}}$			VC-PQ
Method	all	things	stuff	all	things	stuff
PanSt3R v1	45.7	42.1	55.6	19.2	18.7	21.7
PanSt3R v2	51.3	48.4	59.4	22.2	22.3	21.8
EPS3D	13.6	1.3	45.5	2.0	0.3	10.7
IGGT	17.1	11.8	32.5	4.2	3.7	6.7

Appendix

Table A4: The scores of Table 1 at

N{=}10

over all classes, thing classes, and stuff classes. The identity of an instance across views is defined only for things, since each stuff class forms a single instance, so that on stuff the view-consistency factor measures only the mismatch between the sets of views in which a class appears. SegVGGT predicts instances only and enters in the thing columns.

Method	$\lvert\mathrm{TP}\rvert$	$\lvert\mathrm{FP}\rvert$	$\lvert\mathrm{FN}\rvert$	pred. / GT	class-agnostic VC-PQ	class agrees (%)	GT split (%)
PanSt3R v1	11,044	11,905	39,334	0.46	29.1	56	21
PanSt3R v2	15,351	21,625	35,027	0.73	31.9	52	23
EPS3D	346	38,581	50,032	0.77	4.5	22	49
IGGT	2,756	18,758	47,622	0.43	14.0	31	17

Appendix

Table A5: Predicted instances of thing classes on ScanNet++ (

N{=}10

, DSLR and iPhone). Counts of matched and unmatched instances under VC-PQ, the number of predicted instances per ground-truth instance (both above the visibility threshold), VC-PQ with all classes, things and stuff, merged, the share of predicted instances whose class agrees with the ground-truth instance they overlap most, and the share of ground-truth instances split over two or more predicted instances.

Method	not covered (%)	covered by another ID (%)
PanSt3R v1	63.9	0.4
PanSt3R v2	54.8	1.1
EPS3D	92.0	0.6
IGGT	80.7	1.2

Appendix

Table A6: Errors of the predictions on the runs of Table A7 (ScanNet++ DSLR,

N{=}10

, thing classes), counted over the same appearances. An appearance is covered when a predicted instance overlaps it with an IoU above 0.5 in that view. The first column is the share of appearances that are not covered, and the second the share of appearances covered by a predicted instance other than the one that covers the instance in the largest number of views of the same input set.

Perturbation	Variant	PQ ${}^{\text{scene}}$	VC-PQ	VC-PQ 1 ( $\gamma{=}1$ )	per-image PQ
none	–	100.0	100.0	100.0	100.0
missed appearance	smallest area first	91.4	80.9	92.9	86.7
	largest area first	39.3	80.9	50.7	86.7
	random	72.9	80.9	77.2	86.7
identity split	smallest area first	65.2	63.2	72.3	100.0
	largest area first	51.6	63.2	61.9	100.0

Appendix

Table A7: Ground truth of ScanNet++ DSLR (

N{=}10

, thing classes) perturbed at one operating point and scored against the original, so that the unperturbed label maps score 100. In each group the same number of appearances, 28% of all, is removed (missed appearance) or given a new ID (identity split), and the three variants differ only in which views are chosen: the appearances of smallest area first, of largest area first, or at random (mean of three seeds). Per-image PQ scores each view on its own.

erosion radius $r$ (px)	1	2	4	8
PQ ${}^{\text{scene}}$	85.9	77.0	63.3	45.2
VC-PQ	61.3	40.2	20.3	8.7
VC-PQ 1 ( $\gamma{=}1$ )	70.3	51.3	30.0	15.2
per-image PQ	61.7	41.9	24.3	12.3
appearances below $\eta$ (%)	13.2	23.4	38.1	55.0

Appendix

Table A8: Scores of the ground truth of Table A7 with every thing mask eroded by

r

pixels, scored against the unperturbed ground truth. Eroded pixels become void, and the image border is not eroded. The last row is the share of appearances that fall below the visibility threshold after erosion.

Score	0.5% filter	aggregation	$N{=}5$	$N{=}10$	$N{=}20$	$N{=}40$
PQ ${}^{\text{scene}}$	on	per-scene	46.8	51.3	54.7	57.7
PQ ${}^{\text{scene}}$	on	pooled	41.3	42.7	40.9	44.2
VC-PQ	off	per-scene	25.3	25.6	26.9	28.2
VC-PQ	off	pooled	24.5	22.2	20.7	20.0
PQ ${}^{\text{scene}}$	off	pooled	28.6	28.6	28.6	29.1

Appendix

Table A9: PQ

{}^{\text{scene}}

and VC-PQ of PanSt3R v2 on ScanNet++ for input sets of

N

views under two aggregations. Per-scene averages the class-averaged value of each scene run, and pooled sums the counts of all input sets before the class average. The first and fourth rows are the values of Table 1 .

	$\eta{=}50$	$\eta{=}100$	$\eta{=}200$
PanSt3R v1	18.5	19.2	20.2
PanSt3R v2	21.5	22.2	23.4
EPS3D	2.0	2.0	2.1
IGGT	4.0	4.2	4.5
ground-truth instances excluded (%)	0.0	2.5	6.9

Appendix

Table A10: VC-PQ on ScanNet++ at

N{=}10

for three values of the visibility threshold

\eta

, aggregated as in Table 1 . The last row is the share of ground-truth instances whose mask has fewer than

\eta

pixels in every view of the set and that are therefore excluded from evaluation.

Figure A1: VC-PQ γ of PanSt3R v2 against

\gamma

in Eq. ( 6 ), one curve per input set size

N

, over all classes. The predictions are identical along each curve; only the weighting of views changes.

\gamma=0

is VC-PQ and

\gamma=1

applies the area weighting of tiling.

	VC-PQ							PQ ${}^{\text{scene}\dagger}$	PQ ${}^{\text{scene}\dagger}$ $-$ VC-PQ, Shapley
area weighting	–	✓	–	–	✓	✓	–	✓		✓
per-scene avg.	–	–	✓	–	✓	–	✓	✓			✓
0.5% filter	–	–	–	✓	–	✓	✓	✓				✓
PanSt3R v1	18.7	24.5	20.8	25.2	26.1	35.0	33.5	42.2	+23.5	+7.3	+4.8	+11.4
PanSt3R v2	22.3	28.9	23.1	33.1	28.3	42.6	39.8	48.5	+26.1	+7.5	+3.2	+15.4
EPS3D	0.3	0.7	0.4	0.5	0.8	1.1	0.8	1.7	+1.4	+0.6	+0.3	+0.5

Appendix

Table A11: Table 2 restricted to thing classes. VC-PQ with the three properties of PQ

{}^{\text{scene}}

switched on in every combination, on ScanNet++ at

N{=}10

. † With all three switched on, the score reproduces PQ

{}^{\text{scene}}

to within about 0.4 over thing classes of Table A4 ; unlike PQ

{}^{\text{scene}}

, it applies the visibility threshold. The right block splits the difference between the two named columns over the three properties by Shapley attribution.

Figure A2: The three properties of PQ

{}^{\text{scene}}

of Sec. 3.2 on real data. (a) Area weighting: ground truth of the ScanNet++ DSLR runs (

N{=}10

, thing classes) with the same number of appearances removed, in the views in which an instance appears smallest or in those in which it appears largest, scored against the unperturbed ground truth; the vertical line marks the operating point of Table A7 . (b) Averaging over scenes: the counts of PanSt3R v2 on ScanNet++ (

N{=}10

, all classes, 0.5% filter on), averaged over classes after pooling all input sets and after averaging per scene run (Table 1 ), split into the shares of floor, wall, and ceiling, of the small classes (median share of the labelled pixels of an input set below 0.5% at

N{=}10

), and of the other classes. (c) The 0.5% class filter at

N{=}40

: for the 20 classes present in the most input sets, the share of the labelled pixels of an input set that the class covers in the ground truth (box: quartiles, line: 5th to 95th percentile), and on the right the share of the input sets containing the class in which the filter removes it.

Figure A3: Share of matched candidates as a function of the fraction

\lvert V\rvert/N

of the input set in which the instance is visible, for

\gamma=0

(VC-PQ), one curve per

N

, over thing classes visible in at least two views. Top: ground-truth instances, bottom: predicted instances. (a)–(c) and (e)–(g) are PanSt3R v2 on the three captures, (d) and (h) SegVGGT on ScanNetv2. A candidate is an instance with a same-class counterpart at

\overline{\mathrm{IoU}}(p,g)>0.5

, and bins with fewer than 30 instances are omitted. The fraction is binned, with bin edges at 0.1, 0.2, 0.3, 0.4, 0.5, and 0.7, and each marker sits at the centre of its bin.

Figure A4: Distribution of the fraction

\lvert V(g)\rvert/N

of the input set in which a ground-truth instance is visible, at each

N

, over thing classes of the three captures. Top: all scored instances, bottom: instances visible in at least two views.

Figure A5: The plane of the two factors of Eq. ( 4 ),

\overline{\mathrm{IoU}}(p,g)

and

\mathrm{VC}(p,g)

, divided by the three tests of Sec. 4.3 , for PanSt3R v2 on ScanNet++ at

N{=}10

(thing classes): (a) ground-truth instances, (b) predicted instances. Each region is labelled with the term of Eq. ( 7 ) that its unmatched instances enter, with their number and share of all instances; the dashed curve is

q(p,g)=0.5

. Matched pairs also enter

\varepsilon_{\overline{\mathrm{IoU}}}

with

1-\overline{\mathrm{IoU}}(p,g)

and

\varepsilon_{\mathrm{VC}}

with

\overline{\mathrm{IoU}}(p,g)\,(1-\mathrm{VC}(p,g))

, which the regions do not show.

ScanNet++
	$N{=}5$				$N{=}10$				$N{=}20$				$N{=}40$
Method	VC-PQ	$\varepsilon_{\overline{\mathrm{IoU}}}$	$\varepsilon_{\mathrm{VC}}$	$\varepsilon_{q}$	VC-PQ	$\varepsilon_{\overline{\mathrm{IoU}}}$	$\varepsilon_{\mathrm{VC}}$	$\varepsilon_{q}$	VC-PQ	$\varepsilon_{\overline{\mathrm{IoU}}}$	$\varepsilon_{\mathrm{VC}}$	$\varepsilon_{q}$	VC-PQ	$\varepsilon_{\overline{\mathrm{IoU}}}$	$\varepsilon_{\mathrm{VC}}$	$\varepsilon_{q}$
PanSt3R v1	22.0	72.8	4.0	1.1	19.2	73.6	4.6	2.6	17.9	73.7	4.1	4.2	17.4	73.6	3.7	5.3
PanSt3R v2	24.5	68.8	5.1	1.5	22.2	67.6	7.0	3.1	20.7	67.0	7.1	5.2	20.0	66.6	6.6	6.8
EPS3D	2.4	96.8	0.6	0.2	2.0	97.1	0.5	0.3	–	–	–	–	–	–	–	–
IGGT	5.7	92.2	1.8	0.4	4.2	93.2	1.8	0.7	–	–	–	–	–	–	–	–

Appendix

Table A12: The decomposition of Table 3 over all classes, for input sets of

N

views (ScanNet++ pools the DSLR and iPhone runs). VC-PQ and the three terms sum to 100 in each block, and VC-PQ equals Table 1 . SegVGGT predicts thing classes only and is reported over thing classes alone (Table 3 ); EPS3D on ScanNetv2 is added here. –: as in Table 1 .

		instances		unmatched ground truth			unmatched predictions
Method	$N$	GT	pred.	$\lvert\mathrm{FN}_{\overline{\mathrm{IoU}}}\rvert$	$\lvert\mathrm{FN}_{\mathrm{VC}}\rvert$	$\lvert\mathrm{FN}_{q}\rvert$	$\lvert\mathrm{FP}_{\overline{\mathrm{IoU}}}\rvert$	$\lvert\mathrm{FP}_{\mathrm{VC}}\rvert$	$\lvert\mathrm{FP}_{q}\rvert$
ScanNet++
PanSt3R v1	5	80,635	38,468	56,078	2,970	915	14,310	2,632	854
	10	50,378	22,949	36,256	1,828	1,250	9,114	1,619	1,172
	20	28,608	12,633	21,033	781	1,017	5,218	690	948
	40	15,058	6,493	11,235	312	659	2,772	268	601

Appendix

Table A13: Counts behind Table 3 (thing classes): ground-truth and predicted instances, and the unmatched ones grouped by the first of the three tests,

\overline{\mathrm{IoU}}(p,g)>0.5

\mathrm{VC}(p,g)>0.5

, and

q(p,g)>0.5

, that fails (Sec. 4.3 ).

		share by $\lvert V(p)\rvert$ (%)
$N$	$\lvert\mathrm{FP}_{\mathrm{VC}}\rvert$	1	2	$\geq 3$
5	4,406	77.4	19.2	3.4
10	3,241	65.2	24.0	10.8
20	1,763	45.1	28.1	26.8
40	811	26.6	19.5	53.9

Appendix

Table A14: Predicted instances that fail the view-consistency test (

\mathrm{FP}_{\mathrm{VC}}

) for PanSt3R v2 on ScanNet++ (thing classes), by the number of views

\lvert V(p)\rvert

in which they are visible.

Figure A6: As Fig. A3 , for

\gamma=1

		VC-PQ				VC-SQ				VC-RQ
Dataset	Method	$N{=}5$	10	20	40	$N{=}5$	10	20	40	$N{=}5$	10	20	40
ScanNet++	PanSt3R v1	22.0	19.2	17.9	17.4	65.8	62.9	56.6	53.3	28.7	25.7	24.5	24.2
	PanSt3R v2	24.5	22.2	20.7	20.0	64.8	61.8	57.1	52.9	32.0	29.6	28.3	27.9
	EPS3D	2.4	2.0	–	–	15.2	14.4	–	–	3.3	2.9	–	–
	IGGT	5.7	4.2	–	–	36.4	32.4	–	–	7.4	5.7	–	–
ScanNetv2	PanSt3R v2	43.7	41.5	40.6	40.6	78.9	76.6	74.7	73.7	54.9	53.5	53.6	54.2

Appendix

Table A15: VC-PQ and its two factors VC-SQ and VC-RQ for input sets of

N

views, all classes, aggregated as in Table 1 . The product

\text{VC-PQ}=\text{VC-SQ}\times\text{VC-RQ}

holds for each class and not for the class averages shown here, and a class without a matched pair enters the average of VC-SQ with zero, as in PQ. –: as in Table 1 .

Explore similar work

May 29, 2026cs.CV

Redefining Instance Matching: A Unified Framework for Part-Aware Matching in Panoptic Segmentation Evaluation

The Panoptic Quality (PQ) metric is the standard for jointly evaluating instance and semantic segmentation. However, its original definition relies on a One-to-One matching between predicted and ground truth segments, which is only straightforward when the IoU threshold exceeds 0.5. Below 0.5, multiple matching strategies emerge in a poorly explored problem space. We systematically elucidate this space by recasting segment matching as a constrained bipartite assignment problem. Independently bounding the prediction- and ground-truth-side degrees yields four matching strategies: One-to-One, Many-to-One, One-to-Many, and Many-to-Many. We show that the first three are well-defined within the PQ framework, while Many-to-Many falls outside it. These strategies become relevant when instances are fragmented, adjacent objects are difficult to delineate, or annotations are noisy. Central to our framework is a vertex-based accounting of TP, FN, and FP, anchored to ground truth and predicted segments rather than to matching edges. We further show that the framework extends naturally to part-aware panoptic segmentation, and we explore part-aware evaluation on biomedical data. Across configurable case studies we report how different combinations of thresholds and matching strategies behave in practice. We release a unified open-source package built on Panoptica. It exposes Voronoi-based region-wise analysis, part-aware evaluation, and Area Under Threshold Curve computations as configurable options.

Erik Großkopf, Soumya Snigdha Kundu, Hendrik Möller +9

Hertie Institute for AI in Brain Health, University of Tübingen, Germany · King’s College London, UK · Technical University of Munich, Germany +1

Panoptic Segmentation Ground Truth Unified Framework Biomedical

Ask PDF

Jul 22, 2026cs.CV

Extending a Large View Synthesis Model for Multi-view Panoptic Segmentation

Large view synthesis models synthesize novel views through cross-view attention without explicit 3D representations, and recent studies have shown that they learn accurate spatial correspondence from RGB supervision alone. We observe that this correspondence generalizes beyond appearance. When non-photorealistic signals such as binary encoded panoptic labels are passed through the model, they are propagated to novel views with consistent spatial structure. These results indicate that the correspondence learned for RGB view synthesis can also propagate view-independent per-pixel labels. From this observation, we present the first work to extend large view synthesis models beyond appearance rendering to 3D scene understanding. We propose a panoptic segmentation pipeline that reuses a frozen view synthesis model to propagate panoptic labels from input views to novel views, without 3D reconstruction or any segmentation-specific training of the view synthesis model. Given panoptic labels on the input views, we encode them into binary channel representations and pass them through the same model to render target-view segmentation. On ScanNet, our method achieves segmentation quality on par with Gaussian based approaches requiring explicit 3D reconstruction, while outperforming them in novel view synthesis by more than 7 dB. The label propagation also transfers across datasets, surpassing these approaches on Replica without any fine-tuning.

Kwonyoung Ryu, In-Jae Lee, Jonghyun Jin +3

POSTECH · Seoul National University · POSCO DX +1

Panoptic Segmentation Novel View Synthesis Omnidirectional Images Label Propagation Process-Level Supervision

Ask PDF

Jul 20, 2026cs.CV

MuViSeg: Multi-View Segment Correspondences from Dense Geometry Priors

Classical image correspondence is solved at the level of sparse keypoints or dense pixels, but the systems that consume these matches - object-level mapping, topological navigation, scene-graph maintenance - reason about whole objects. Recent work narrows this gap by matchng directly at the level of instance segments: a class-agnostic segmenter partitions each image, and per-segment descriptors are obtained by pooling features from large 3D foundation models over the masks. We build on this segment-level matching paradigm and propose three learned matching heads: a LightGlue-style attention head with DoubleSoftmax scoring on frozen MASt3R descriptors; a DPT-style multi-scale fusion module that exposes layered spatial detail from the VGGT foundation model before pooling; and - as our main contribution - a multi-view extension that performs joint self-attention over segments drawn from several views at once, recovering transitive correspondences that strictly pairwise matchers cannot reach. Under a stratified zero-shot protocol on Replica and Virtual KITTI 2 with controlled viewpoint baselines from 0 deg to 180 deg, the LightGlue-style head improves over a parameter-free Sinkhorn matcher on the same MASt3R backbone by +4.85 AUPRC on Replica and +25.9 AUPRC on Virtual KITTI 2. Dropped into the RoboHop topological navigation pipeline on the Habitat-Matterport 3D (HM3D) Instance Image Navigation benchmark without retraining, our multi-view variant raises success rate from 50% to 70%, and our LightGlue-style head raises SPL from 45.7 to 59.1.

Denis Fatykhoph, Timur Akhtyamov, Konstantin Pakulev +2

Applied AI Institute, Moscow, Russia

Multi-View Amodal Segmentation 3D Generation Multi-Image Editing Benchmarks

Ask PDF