VideoMaMa++: Temporally Consistent Video Matting via Preserve-and-Refine Token

Sangbeom Lim; Seoung Wug Oh; Heeji Yoon; Seungryong Kim; Joon-Young Lee

cs.CVOct 19, 2026

VideoMaMa++: Temporally Consistent Video Matting via Preserve-and-Refine Token

Authors: Sangbeom Lim, Seoung Wug Oh, Heeji Yoon, Seungryong Kim, Joon-Young Lee

Organizations: Korea University · Adobe Research · KAIST AI

Abstract

Diffusion-based video matting has recently emerged as a leading approach, leveraging video diffusion priors from internet-scale pretraining to achieve strong zero-shot generalization from limited synthetic data. However, the heavy compute of these diffusion backbones limits inference to a short chunk of just 6 to 24 frames, so even ordinary videos span multiple chunks. Because each chunk is matted independently from coarse mask guidance, existing methods fail to maintain consistency across chunk boundaries, producing flickering edges and discontinuous transparency. We address this with a mask-guided diffusion-matting framework built on two ideas. The RGB and mask conditioning are represented as separate token streams that interact through full spatio-temporal self-attention, letting the model adaptively combine fine RGB texture with coarse mask layout rather than committing to a fixed channel-wise fusion. On top of this, we propose a novel pair of learnable embeddings, the preserve token and the refine token, that act as a per-frame conditioning interface and enable a sliding-window inference scheme in which already-generated mattes are propagated across chunk boundaries for local temporal consistency, complemented by a globally shared reference anchor at every chunk to prevent error accumulation in long sequences without any additional training. Across standard video matting benchmarks and in-the-wild long footage, our framework outperforms prior diffusion-based methods in both fine-detail accuracy and temporal consistency, and uniquely maintains consistency across chunk boundaries on long videos.

Figures & tables

Figure 1: We introduce VideoMaMa++ , a mask-guided video matting framework that recovers fine-grained details (e.g., hair strands) and scales reliably to long videos. (a) Per-frame qualitative results on diverse in-the-wild videos. (b) Long-video matting results: VideoMaMa Lim et al. (2026) , which processes videos in chunks, fails to maintain consistency across chunk boundaries (red boxes), whereas VideoMaMa++ produces mattes that are both finely detailed and temporally consistent.

Figure 2: Overview of the VideoMaMa++ training and inference. (Top) During training, RGB frames are paired with either a binary mask or the ground-truth alpha matte as guide input, encoded into separate token streams by a frozen VAE encoder $\mathcal{E}$ , and tagged with a refine token ( R ) or preserve token ( P ). The DiT processes both modes jointly within a single clip, supervised against the ground-truth matte. (Bottom) At inference, long videos are processed in a sliding-window manner. Each chunk receives the refine token on its non-overlapping frames and the preserve token on its overlap frames, which carry the previous chunk’s predicted matte. A globally selected reference anchor frame is additionally supplied as preserve input to every chunk, providing a stable anchor that prevents error accumulation across the sequence.

Figure 3: Long-video temporal consistency. Qualitative comparison between VideoMaMa++ and other methods. The comparison methods exhibits visible flickering at chunk boundaries or struggles to capture fine-grained details, while VideoMaMa++ produces smooth transitions across chunks while still capturing fine-grained details that emerge in later chunks.

Panel of Figure 4 — Figure 4: (Left) Interactive refinement examples. (Right) MAD across chunk index.

VideoMatte (1920 $\times$ 1080)
Setting	MAD $\downarrow$	MAD-T $\downarrow$	MSE $\downarrow$	GRAD $\downarrow$	CONN $\downarrow$	DTSSD $\downarrow$
w/o r. anchor	3.9505	4.7366	0.1518	2.2398	1.2817	0.8576
w/ r. anchor	3.8633	4.6354	0.1240	1.7887	1.0755	0.8206
YoutubeMatte (1920 $\times$ 1080)
w/o r. anchor	1.2691	31.6900	0.2861	4.4151	2.3294	1.1803
w/ r. anchor	1.1526	29.4655	0.2288	3.2741	2.0606	1.1663

Table 2: Ablation study on reference anchor.

Method	MAD $\downarrow$	Gradient $\downarrow$
w/o preserve	0.7128	0.6811
w/ preserve	0.2583	0.1235

Table 4: Chunk boundary consistency.

	MAD $\downarrow$		Gradient $\downarrow$
Input Mask	Input	Refined	Input	Refined
XMem	5.5587	3.1119	17.7265	3.6025
Cutie	3.6998	1.9430	13.5395	2.9448
SAM2	4.0844	1.8994	19.3595	3.2741
SAM3	3.9106	1.6924	18.3659	2.9196

Table 5: Robustness to mask generators.

Table 6: Ablation studies on video matting design choices. We evaluate the impact of our proposed components across different configurations. The ablations analyze (a) the advantage of incorporating SAM logits, and (b) channel-wise versus token-wise concatenation. All evaluations are performed on the V-Him60 (Hard) benchmark. The best results are highlighted.

Steps	MAD $\downarrow$	MAD-T $\downarrow$	MSE $\downarrow$	GRAD $\downarrow$	CONN $\downarrow$	DTSSD $\downarrow$
1	1.6680	41.7354	0.4027	2.9674	2.6769	1.9546
2	1.7221	42.0193	0.4138	2.9708	2.7389	1.9479
3	2.7264	50.3829	0.5093	3.8081	3.3320	2.1253
5	10.7571	74.6095	2.0941	5.1494	17.5623	2.6155
10	16.6343	101.5099	3.9117	6.1175	31.1540	3.1936

Table 7: Ablation on denoising steps.

Chunk	MAD $\downarrow$	MAD-T $\downarrow$	GRAD $\downarrow$	CONN $\downarrow$	DTSSD $\downarrow$
3	2.1206	46.2131	3.5778	3.5776	2.2120
6	1.6942	41.2279	2.9196	2.7194	1.9796
12	1.7906	41.6290	3.0313	2.8858	2.0693

Table 8: Ablation on chunk size.

Rank	MAD $\downarrow$	MAD-T $\downarrow$	MSE $\downarrow$	GRAD $\downarrow$	CONN $\downarrow$	DTSSD $\downarrow$
64	2.4294	56.5292	0.8798	4.2180	4.2180	2.6284
128	2.3018	53.3100	0.8108	4.7875	3.9483	2.5732
512	1.6942	41.2279	0.4487	2.9196	2.7194	1.9796

Table 9: Ablation on LoRA rank.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Datasets
Image	DVM Sun et al. (2021) , AM-2k Li et al. (2022) , DAViD Saleh et al. (2025) , P3M-10k Li et al. (2021) , RefMatte Li et al. (2023)
Video	CRGNN Wang et al. (2021) , VideoMatte240K Lin et al. (2021) , DVM Sun et al. (2021)
Background	VideoMatting108 Zhang et al. (2021) , DVM Sun et al. (2021)

Appendix

Table 10: Datasets used to train VideoMaMa and SAM2-Matte.

Iter.	MAD $\downarrow$	MAD-T $\downarrow$	GRAD $\downarrow$	CONN $\downarrow$	DTSSD $\downarrow$	MSE $\downarrow$
1	1.6924	41.9167	2.9813	2.7228	1.9758	0.4234
2	1.6688	41.8064	2.9677	2.6761	1.9589	0.3992
3	1.6634	41.7927	2.9633	2.6655	1.9545	0.3949

Appendix

Table 11: Ablation on interactive refinement.

Method	P3M-500-NP				AM-2k
Method	MAD $\downarrow$	MSE $\downarrow$	Grad $\downarrow$	Conn $\downarrow$	MAD $\downarrow$	MAD-T $\downarrow$	Grad $\downarrow$
Mask free methods
RVM	10.89	6.57	14.97	18.81	-	-	-
SparseMat	11.14	6.36	13.47	18.13	-	-	-
GVM	8.39	4.89	12.89	14.21	-	-	-
Mask guided methods

Appendix

Table 12: Quantitative evaluation on P3M-500 and AM-2k datasets.

Figure 6: Annotation noise in AM-2K Li et al. (2022) . The ground-truth alpha mattes in AM-2K often miss fine-grained internal details, that VideoMaMa++ correctly recovers. Because MAD averages pixel-wise error over the entire image, these missing details penalize accurate predictions and inflate the reported error. MAD-T, computed only within the transition region, isolates the matting quality that actually matters and avoids the bias introduced by such annotation gaps.

Figure 7: Re-initialization of SAM3 on YouTubeMatte static_0017. Initializing SAM3 Carion et al. (2025) on the first frame fails to track a sheet of paper that the subject begins holding mid-sequence, since the object is not yet visible at initialization. Initializing on a later frame where both are simultaneously visible recovers the correct mask track.

Metric	Init.	MGM	MGM-IW	MaGGIe	MaGGIe-FT	VideoMaMa	Ours
MAD	First-frame	3.87	2.87	2.29	2.30	1.88	1.61
MAD	Re-init	3.59	2.41	1.67	1.52	1.41	1.15
MSE	First-frame	0.78	0.83	0.93	0.95	0.76	0.69
MSE	Re-init	0.53	0.38	0.50	0.37	0.30	0.23
Grad	First-frame	4.37	4.37	7.02	6.42	4.20	3.31
Grad	Re-init	4.27	4.25	5.11	5.36	4.08	3.27

Appendix

Table 13: Effect of re-initializing YouTubeMatte static_0017. “Re-init”: SAM3 re-initialized at frame 60 for static_0017, as used in Table 1 . “First-frame”: SAM3 initialized on the first frame for every sequence. All metrics

\downarrow

	YouTubeMatte		In-the-wild (no GT)
	Seam-elevated clips		Autocorr.		Seam-elevated
Model	MAX	fg-mean	lag 4	lag 8	clips
VideoMaMa Lim et al. (2026)	62/64	63/64	$+0.35$	$+0.34$	14/16
	$p{=}1.1{\times}10^{-16}$	$p{=}3.5{\times}10^{-18}$	$p{=}5{\times}10^{-3}$	$p{=}6{\times}10^{-3}$	$p{=}2.1{\times}10^{-3}$
GT (no chunking)	34/64	32/64	–	–	–
	$p{=}0.35$	$p{=}0.55$

Appendix

Table 14: Annotation-free seam test. Both models run at identical chunk seams. Left: ground-truth validation on YouTubeMatte (64 clips; 24 seam vs. 75 interior transitions per clip). Entries count clips whose mean seam transition exceeds their mean interior transition (chance: 32/64). Right: 16 in-the-wild clips (fg-mean), for which no ground truth exists. Higher autocorrelation at the seam lag and more seam-elevated clips indicate a stronger chunk-boundary discontinuity. p-values for seam-elevated clips are from an exact one-sided sign test against chance.

	FPS $\uparrow$					Peak memory (GB) $\downarrow$					TFLOPs $\downarrow$
Method	256	512	768	1K	2K	256	512	768	1K	2K	256	512	768	1K	2K
MGMatting	66.71	66.23	63.56	60.44	18.05	0.58	0.84	1.37	1.98	5.41	0.036	0.115	0.240	0.411	1.55
MaGGIe	23.2	19.3	20.0	15.0	7.5	0.52	1.18	2.52	4.88	16.01	0.021	0.083	0.187	0.332	1.33
MatAnyone	43.6	42.0	38.9	34.6	7.5	0.33	0.67	1.32	3.43	22.40	0.030	0.124	0.289	0.542	2.94
VideoMaMa	27.56	13.64	6.06	3.41	0.71	4.75	5.36	6.41	7.88	17.05	1.60	6.55	15.4	28.9	159
VideoMaMa++	16.4	6.9	2.6	1.17	0.11	3.73	5.17	7.86	11.33	35.59	4.25	24.8	85.4	225	2920

Appendix

Table 15: Computational cost on a single H100 at resolutions

256^{2}

2048^{2}

(1K =

1024^{2}

, 2K =

2048^{2}

), excluding mask generation for all methods. For VideoMaMa++, the cost of each 6-frame window is amortized over its 4 new frames, so FPS accounts for the recomputed overlap frames. MatAnyone/MatAnyone 2 and MGM/MGM-ITW share architectures and are listed once.

Res.	SAM3	VAE Enc.	DiT (1-step)	VAE Dec.	Matting total	Matting FPS	E2E FPS
256 2	45.4	24.9	20.1	15.9	60.9	16.4	9.4
512 2	45.4	37.9	79.2	27.5	144.6	6.9	5.3
768 2	45.4	78.3	254.3	55.8	388.4	2.6	2.3
1024 2	45.4	139.0	616.1	97.7	852.8	1.17	1.11
2048 2	45.4	681.0	7602.1	448.6	8731.7	0.11	0.11

Appendix

Table 16: Per-output-frame cost of VideoMaMa++ (ms) on a single H100, including SAM3 mask propagation. Matting-stage costs are measured per 6-frame window and divided by its 4 new frames. E2E FPS

=1000\,/

(SAM3 + matting total).

Temporal Consistency

Temporal consistency studies how AI systems preserve coherent states, predictions, and meanings across time.

See all 46 papers

SAM2Matting: Generalized Image and Video Matting
Jun 25, 2026Ruiqi Shen, Guangquan Jie, Chang Liu +1Fine-Grained Visual Perception
Fudan University · Shanghai University of Finance and Economics
Unified Long Video Inpainting and Outpainting via Overlapping High-Order Co-Denoising
Nov 5, 2025Shuangquan Lyu, Jian Mao, Yue MaVideo Diffusion Models Long Video Generation
Carnegie Mellon University · Jilin University · Tsinghua University
FrameDiT: Diffusion Transformer with Matrix Attention for Efficient Video Generation
Mar 10, 2026Minh Khoa Le, Kien Do, Duc Thanh Nguyen +1Efficient Video Generation Video Diffusion Models
Applied Artificial Intelligence Initiative, Deakin University, Australia · FPT Smart Cloud, Vietnam · Deakin University, Australia

Jun 25, 2026J/K move · Enter open · S save

SAM2Matting: Generalized Image and Video Matting

Ruiqi Shen, Guangquan Jie, Chang Liu, Henghui Ding

Fudan University · Shanghai University of Finance and Economics

Fine-Grained Visual Perception

Open paper AskPDF arXiv

Despite impressive advances in image matting, video matting remains challenging due to the inherent gap between high-level tracking, which requires frame-wise understanding, and low-level matting, which focuses on extremely fine-grained details. Existing methods attempt this with expensive and narrowly-scoped video matting datasets, which may limit out-of-domain generalization and compromise tracking robustness. We rethink the paradigm with SAM2Matting, a tracker-to-matting framework that advances VOS trackers to high-fidelity video matting. Specifically, it decouples the task by enhancing a foundational tracker (e.g., SAM2, SAM3) with a region-proposal bridge and dedicated matting heads, enabling the uncompromised tracker to handle temporal consistency while the matting components resolve fine-grained details. Notably, despite being trained only on images, SAM2Matting establishes new state-of-the-art performance on video matting, supports diverse prompt types, maintains strong temporal consistency, and demonstrates robust generalization across both human-centric and in-the-wild scenarios.

Abstract

Temporal Consistency

Explore similar work

SAM2Matting: Generalized Image and Video Matting

Unified Long Video Inpainting and Outpainting via Overlapping High-Order Co-Denoising

FrameDiT: Diffusion Transformer with Matrix Attention for Efficient Video Generation