When Predicting Nothing Beats SAM 3: Revisiting Evaluation in Video Object Segmentation

Jihwan Hong; Woohyeon Park; Jaeik Kim; Jaeyoung Do

cs.CVOct 2, 2026

When Predicting Nothing Beats SAM 3: Revisiting Evaluation in Video Object Segmentation

Authors: Jihwan Hong, Woohyeon Park, Jaeik Kim, Jaeyoung Do

Organizations: AIDAS Laboratory, 1IPAI & 2ECE, Seoul National University

Abstract

Video Object Segmentation (VOS) in complex and long videos is increasingly important for real-world applications, where target objects often appear only intermittently within long temporal horizons. However, existing benchmarks largely focus on temporally salient objects that remain visible for most of the video. To address this gap, we introduce FaVOS (A Benchmark for Video Object Segmentation with Fractional Temporal Visibility), a benchmark designed to evaluate VOS methods under low temporal visibility. We show that, in this regime, the standard J&F metric can collapse VOS evaluation into absence classification, because empty predictions receive high rewards on target-absent frames. Consequently, even a trivial empty-mask predictor can outperform strong models such as SAM 3, revealing a fundamental mismatch between current metrics and practical VOS performance. To mitigate this issue, we propose Volumetric J&F, which evaluates mask sequences as spatio-temporal volumes and reduces the dominance of target-absence rewards while preserving sensitivity to segmentation quality and temporal structure. Project page: https://aidaslab.github.io/FaVOS.

Figures & tables

Figure 1: Temporal absence ratio (

1-\tau

) across VOS benchmarks. Higher values indicate a larger fraction of frames where the target is absent, reflecting lower temporal visibility in recent datasets.

Figure 2: The standard VOS metric, $\mathcal{J}\&\mathcal{F}$ , assigns perfect scores to empty predictions, allowing a trivial “predict nothing” baseline to outperform strong models under low temporal visibility.

Benchmark	Year	Videos	Objects	Duration (s)	Disappearance Rate	Avg. Temporal Visibility
Tracking benchmarks (VOT)
GOT-10k Huang et al. (2019)	2019	180	180	12.7	0.00%	1.00
LaSOT Fan et al. (2019)	2019	280	280	81.6	58.6%	0.97
VOT2022 Kristan et al. (2022)	2022	62	62	10.7	19.4%	1.00
DiDi Videnovic et al. (2025)	2025	180	180	254.5*	42.8%	0.97
Segmentation benchmarks (VOS)

Table 1: Comparison of video benchmarks in terms of scale and temporal dynamics. Metrics are computed on evaluation splits (test if available; otherwise validation). Disappearance rate Ding et al. (2023b) denotes the fraction of objects absent in at least one frame. * indicates values estimated with fps=6.

Figure 3: Example videos from the FaVOS benchmark. Each row shows sampled frames from a video with annotated target masks. The bars below each sequence indicate temporal visibility, where colored segments denote frames in which the target object is visible and gray segments denote absence. See Appendix A.1 for additional examples.

Figure 4: Contribution ratio of classification and segmentation terms in

\mathcal{J}\&\mathcal{F}

as a function of temporal visibility

\tau

Model	Venue	FaVOS-20					FaVOS-40
		$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}\&\mathcal{F}_{v}$	$p_{0}$	$p_{1}$	$s$	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}\&\mathcal{F}_{v}$	$p_{0}$	$p_{1}$	$s$
Empt. pred.	–	80.0	0.0	1.00	0.00	0.00	59.9	0.0	1.00	0.00	0.00
STM Oh et al. (2019)	ICCV’19	71.9	35.5	0.82	0.62	0.65	65.9	44.9	0.85	0.64	0.68
STCN Cheng et al. (2021)	NeurIPS’21	50.6	26.0	0.52	0.81	0.60	53.9	37.7	0.57	0.85	0.63
XMem Cheng and Schwing (2022)	ECCV’22	63.8	36.8	0.69	0.74	0.66	62.0	45.3	0.71	0.78	0.69
DeAOT-L Yang and Yang (2022)	NeurIPS’22	59.9	38.0	0.61	0.80	0.73	62.7	49.2	0.65	0.83	0.77

Table 2: Performance comparison on the FaVOS benchmark.

Figure 5: Object-level dependency of

\mathcal{J}\&\mathcal{F}

and

\mathcal{J}\&\mathcal{F}_{v}

p_{0}

p_{1}

, and

s

, across (a) FaVOS-20, (b) FaVOS-40, and (c) SA-V.

Model	FaVOS-full						SA-V test
	$\mathcal{J}\&\mathcal{F}$ $\uparrow$	$\mathcal{J}\&\mathcal{F}_{v}$ $\uparrow$	$\mathrm{tIoU}\uparrow$	$\mathcal{T}\downarrow$	$\mathcal{J}\&\dot{\mathcal{F}}_{d}\uparrow$	$\mathcal{J}\&\dot{\mathcal{F}}_{r}\uparrow$	$\mathcal{J}\&\mathcal{F}$ $\uparrow$	$\mathcal{J}\&\mathcal{F}_{v}$ $\uparrow$	$\mathrm{tIoU}\uparrow$	$\mathcal{T}\downarrow$	$\mathcal{J}\&\dot{\mathcal{F}}_{d}\uparrow$	$\mathcal{J}\&\dot{\mathcal{F}}_{r}\uparrow$
Empt. pred.	69.2	0.0	0.00	–	100.0	0.0	25.3	0.0	0.00	–	100.0	0.0
STM Oh et al. (2019)	68.6	40.5	0.45	0.57	82.3	16.3	46.6	34.4	0.50	0.56	82.9	21.1
STCN Cheng et al. (2021)	52.4	32.3	0.40	0.56	54.4	36.4	60.4	55.1	0.70	0.51	70.8	43.5
XMem Cheng and Schwing (2022)	62.8	41.4	0.47	0.56	68.8	28.5	58.0	48.9	0.63	0.52	81.6	28.4
DeAOT-L Yang and Yang (2022)	61.4	44.0	0.44	0.54	61.2	35.7	62.2	55.9	0.75	0.51	63.2	44.7

Table 3: Performance comparison on FaVOS-full and SA-V Ravi et al. (2025) test split.

Figure 6: Relationship between temporal overlap (tIoU) and segmentation metrics on (a) FaVOS-full and (b) SA-V test.

	FaVOS-20				FaVOS-40				SA-V test
Model	Linear	Sqrt	Log	Uniform	Linear	Sqrt	Log	Uniform	Linear	Sqrt	Log	Uniform
SAM 3	56.0	55.5	55.8	54.5	72.7	71.7	72.1	70.0	82.6	82.3	82.5	81.2
SeC	55.7	55.2	55.4	54.1	70.9	70.5	70.8	69.4	77.8	77.6	77.8	76.7
SAM2Long-L	49.8	49.4	49.7	48.5	64.8	64.5	64.7	63.7	76.4	76.3	76.5	75.5
Cutie-B	49.0	48.5	48.7	47.2	58.6	58.4	58.6	57.6	53.4	53.2	53.4	52.6
SAM 2.1-L	46.7	45.7	46.0	44.2	60.5	60.3	60.5	59.6	74.3	74.4	74.5	73.8

Table 4:

\mathcal{J}\&\mathcal{F}_{v}

under different frame weightings

w(u_{t})

. Arrows mark rank changes from the shaded linear weighting used in our main experiments, computed from unrounded scores. Rows are sorted by the linear-weighting score on FaVOS-20.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Additional examples from the FaVOS benchmark.

Figure 8: Temporal visibility distributions of different VOS benchmarks. The center value indicates the average temporal visibility, and

n

denotes the number of annotated objects. Since MOSEv2 test annotations are not publicly available, we report statistics on the training split for MOSEv2. Compared to existing benchmarks, FaVOS contains substantially more low-visibility objects.

Statistic	FaVOS-20	FaVOS-40
Videos	100	100
Objects	129	152
Objects per video	1.29	1.52
Videos from YFCC100M / self-collected	67 / 33	66 / 34
Mean temporal visibility $\tau$	0.21	0.42
Mean disappearance intervals per object	1.54	1.90

Appendix

Table 5: Statistics of the FaVOS-20 and FaVOS-40 splits.

Figure 9: Relative contributions of the absence and segmentation terms in

\mathcal{J}\&\mathcal{F}

across different

p_{0}

p_{1}

, and

s

settings. The dashed line marks the crossover visibility

\tau^{*}

where the two contributions are equal.

Model	DAVIS17 val			OVOS val
	$\mathcal{J}\&\mathcal{F}$ $\uparrow$	$\mathcal{J}\&\mathcal{F}_{v}$ $\uparrow$	$\Delta$	$\mathcal{J}\&\mathcal{F}$ $\uparrow$	$\mathcal{J}\&\mathcal{F}_{v}$ $\uparrow$	$\Delta$
SAM 3 [ 4 ]	91.47	93.15	+1.68	86.67	87.00	+0.33
SeC [ 47 ]	89.87	91.28	+1.42	81.62	81.81	+0.19
SAM 2.1-L [ 35 ]	89.80	91.38	+1.58	80.98	81.22	+0.24
SAM2Long-L [ 14 ]	89.80	91.35	+1.55	81.45	81.77	+0.32
SAMURAI-L [ 43 ]	89.57	91.19	+1.62	79.68	80.29	+0.61

Appendix

Table 6: Comparison of

\mathcal{J}\&\mathcal{F}

and

\mathcal{J}\&\mathcal{F}_{v}

on high-temporal-visibility benchmarks.

\Delta

denotes

\mathcal{J}\&\mathcal{F}_{v}

-

\mathcal{J}\&\mathcal{F}

. Rows are sorted by

\mathcal{J}\&\mathcal{F}

on DAVIS17.

	FaVOS-full					SA-V test
	Single score			Two-number		Single score			Two-number
Model	$\mathcal{J}\&\mathcal{F}$	Bal.	$\mathcal{J}\&\mathcal{F}_{v}$	Present	F1	$\mathcal{J}\&\mathcal{F}$	Bal.	$\mathcal{J}\&\mathcal{F}_{v}$	Present	F1
SAM 3	81.8 (2)	81.3 (1)	65.0 (1)	78.4	74.5	84.7 (1)	82.9 (1)	82.6 (1)	81.5	94.0
SeC	82.4 (1)	81.2 (2)	63.9 (2)	74.6	72.7	81.7 (2)	81.3 (2)	77.8 (2)	76.1	91.2
SAM2Long-L	76.3 (5)	76.2 (4)	57.9 (3)	73.1	70.3	80.4 (3)	79.2 (4)	76.4 (4)	75.7	91.3
DAM4SAM-L	74.1 (6)	75.8 (5)	54.5 (4)	75.4	66.8	80.1 (4)	79.6 (3)	76.6 (3)	74.9	90.5

Appendix

Table 7: Comparison with alternative evaluation protocols on FaVOS-full and SA-V [ 35 ] test split. Single-score protocols: standard

\mathcal{J}\&\mathcal{F}

, visibility-balanced

\mathcal{J}\&\mathcal{F}

(Bal.), and

\mathcal{J}\&\mathcal{F}_{v}

, with ranks in parentheses. Two-number protocol:

\mathcal{J}\&\mathcal{F}

on target-present frames only (Present) and frame-level presence F1. Higher is better for all columns. Rows are sorted by

\mathcal{J}\&\mathcal{F}_{v}

on FaVOS-full.

Video Object Segmentation

Video object segmentation assigns pixel-level masks to objects across frames while preserving their identities through motion, occlusion, and change.

See all 114 papers

Toward Open-World Video Segmentation over Long Horizons
Jun 14, 2026Qing Su, Kaiyang Li, Yuan Zhuang +2Video Object Segmentation Multi-Object Tracking
University of Connecticut
SAM3Dual: A 3rd Place Solution to the MOSEv2 Track, 8th LSVOS Challenge
Aug 23, 2026JeongRae Kim, Chaehyun Kim, Changwon LimVideo Object Segmentation
Chung-Ang University, Seoul, Republic of Korea
VOS-Agent: The 1st Place Solution for the 8th LSVOS Challenge (MOSEv2 Track)
Aug 13, 2026Canyang Wu, Jinrong Zhang, Xusheng He +3Video Object Segmentation Visual Object Tracking
Harbin Institute of Technology, Shenzhen · Nanyang Technological University · Shenzhen Loop Area Institute

Jun 14, 2026J/K move · Enter open · S save

Toward Open-World Video Segmentation over Long Horizons

Qing Su, Kaiyang Li, Yuan Zhuang, Fei Miao+1

University of Connecticut

Video Object Segmentation Multi-Object Tracking

Open paper AskPDF arXiv

Long videos challenge open-world segmentation systems to continually discover objects and preserve their identities through disappearance and reappearance. Savvy, a zero-shot, semi-online, class-agnostic system for persistent object discovery and identity maintenance, and OGA, an evaluation suite that credits coherent part-level predictions when their granularity differs from the reference annotations. Savvy combines modular mask discovery, deferred admission based on accumulated evidence, and track consolidation to maintain an evolving object set. OGA allows multiple predictions to support one reference while preserving prediction IDs, pairing granularity-aware fidelity with diagnostics of identity bleeding and fragmented support. Across 142 ScanNet videos and 106 HM3D trajectories, evaluated using adaptively sampled prefixes capped at 1,500 source frames, Savvy outperforms DEVA+SAM and EntitySAM in VPQ_inf, STQ, and AQ under both conventional and OGA evaluation. Conventional VPQ_inf reaches 18.44 versus DEVA+SAM's 11.52 on ScanNet and 17.31 versus 6.58 on HM3D. VIPSeg comparisons reveal conventional one-to-one VPQ's sensitivity to annotation granularity. Controlled identity-severing and flicker tests show that OGA detects temporal failures even when frame-level masks remain unchanged. Together, these results demonstrate complementary advances: Savvy improves long-horizon segmentation and association, and OGA distinguishes coherent part-level support from temporal identity failures. Project website: https://github.com/QingSuML/savvy

Abstract

Video Object Segmentation

Explore similar work

Toward Open-World Video Segmentation over Long Horizons

SAM3Dual: A 3rd Place Solution to the MOSEv2 Track, 8th LSVOS Challenge

VOS-Agent: The 1st Place Solution for the 8th LSVOS Challenge (MOSEv2 Track)