MFVSR: A Multi-Style and Fine-Grained Benchmark for Video Subtitle Removal

Xiyan Huang; Qingdong He; Peng Tang; Bo Yin; Chaoyi Wang; Xiaobin Hu

cs.CVOct 13, 2026

MFVSR: A Multi-Style and Fine-Grained Benchmark for Video Subtitle Removal

Authors: Xiyan Huang, Qingdong He, Peng Tang, Bo Yin, Chaoyi Wang, Xiaobin Hu

Organizations: National University of Singapore · Tencent Youtu Lab · University of Chinese Academy of Sciences

Abstract

Video subtitle removal (VSR) is an important video restoration task, yet existing evaluation remains limited by low-resolution datasets, fragmented test sets, and generic video-inpainting protocols that fail to adequately capture localized restoration errors, unintended modifications, temporal instability, and incomplete outputs. In this paper, we introduce MFVSR, a large-scale multi-style benchmark for fine-grained and reliability-aware video subtitle removal evaluation. MFVSR distinguishes itself through three key features: (1) Large-scale and diverse data, comprising 3,000 paired videos from Real-HQ, Real-LQ, and AI-Generated sources, with an auxiliary set of 953 in-the-wild real videos for real-world coverage analysis; (2) Controlled subtitle diversity, spanning 16 languages and four factors covering spatial layout, temporal behavior, visual style, and language mode; (3) Fine-grained evaluation, introducing an information-conditioned, spatially decoupled protocol that separates blind from mask-guided removal and measures Restoration Fidelity, Preservation Fidelity, Temporal Fidelity, coverage, and failures. Experiments on six representative methods reveal substantial discrepancies between local restoration, outside-region preservation, temporal consistency, and full-set reliability that are obscured by conventional full-frame metrics. MFVSR therefore provides a more comprehensive and diagnostic foundation for evaluating and advancing video subtitle removal methods.

Figures & tables

Figure 1: Overview of MFVSR. MFVSR contains 3,000 paired videos from Real-HQ, Real-LQ, and AI-generated sources, together with 953 auxiliary in-the-wild videos. It spans 16 languages and four subtitle factors, and evaluates blind and mask-guided removal through five spatial views and three complementary dimensions: Restoration Fidelity, Preservation Fidelity, and Temporal Fidelity.

Figure 2: Dataset construction and unified processing of MFVSR consists of: (1) Multi-source collection and filtering to provide a clean video pool (Section 2.1 ); (2) Controllable subtitle rendering to generate diverse subtitled videos with different attributes (Section 2.2 ); (3) Preprocessing to construct paired supervision assets (Section 2.3 ); (4) Execution routing to assign each sample to the corresponding model (Section 2.4 ).

Figure 3: Controlled subtitle-rendering taxonomy in MFVSR. The two-level ring summarizes four independently configurable factors: Spatial Layout, Temporal Behavior, Visual Style, and Language Mode and their major options, forming 24 configuration cells. Surrounding examples are color-coded by factor to illustrate finer-grained within-cell variations.

Category	Method	Mask	Backbone	Temporal mechanism
Inpainting	ProPainter ( Zhou et al., 2023 )	Seq.	Flow + Sparse ViT	Flow-guided propagation
Inpainting	DiffuEraser ( Li et al., 2025 )	MP4	SD 1.5 + BrushNet	Motion module w/ prior
Object	MiniMax-Remover ( Zi et al., 2026 )	MP4	Wan DiT	Full-seq. attention
Object	OmnimatteZero ( Samuel et al., 2025 )	MP4	LTX DiT	Attn. guidance
Object	EffectErase ( Fu et al., 2026 )	MP4	Wan DiT	Region guidance
Subtitle	CLEAR ( He et al., 2026 )	None	Wan DiT	Spatio-temporal attention

Table 1: Characteristics and MFVSR input interfaces of evaluated methods. Detailed inference configurations are in Table 6 .

Figure 4: MFVSR demonstrates rich diversity and balanced coverage. Subtitle Statistics: (a) shows a sunburst chart of language coverage across 16 languages, five script groups, and monolingual/bilingual settings; (e) shows balanced distributions across spatial, temporal, visual style, and language factors, whose full factorial combination forms 24 configurations with 123-126 videos each. Video Statistics: (b) indicates diverse video resolutions across AI-Gen, Real-LQ, and Real-HQ; (c) presents frame-count distributions with substantial variation in video length; (d) shows video durations ranging from short clips to sequences longer than 30 seconds; (f) illustrates video content distributed across nine semantic categories; (g) presents a word cloud of video captions, covering diverse objects, scenes, attributes, and activities.

Benchmark	Resource		Task setting		Evaluation design					Venue
Benchmark	Videos	Paired GT	Input	Sep.	Strat.	Region	Temp.	Failure	Coverage	Venue
Video Inpainting Evaluation
FVI test set ( Chang et al., 2019 )	100	$\boldsymbol{\checkmark}$	M	–	$\boldsymbol{\checkmark}$	$\boldsymbol{\times}$	$\boldsymbol{\triangle}$	$\boldsymbol{\times}$	$\boldsymbol{\times}$	ICCV 2019
DEVIL ( Szeto and Corso, 2022 )	1,250	$\boldsymbol{\checkmark}$	M	–	$\boldsymbol{\checkmark}$	$\boldsymbol{\triangle}$	$\boldsymbol{\checkmark}$	$\boldsymbol{\times}$	$\boldsymbol{\times}$	CVPR 2022
BVINet evaluation ( Wu et al., 2025 )	568	$\boldsymbol{\checkmark}$	B+M	$\boldsymbol{\checkmark}$	$\boldsymbol{\times}$	$\boldsymbol{\times}$	$\boldsymbol{\checkmark}$	$\boldsymbol{\times}$	$\boldsymbol{\times}$	ICCV 2025
Video Object Removal

Table 2: Comparison with related evaluation resources.

\checkmark

\triangle

\times

, and – denote full, partial, unsupported, and not-applicable settings, respectively.

Dimension	Central question	Primary views	Representative evidence
Restoration Fidelity	Is content covered by the subtitle reconstructed toward the paired clean target?	Subtitle	PSNR and LPIPS; conditioned DISTS
Preservation Fidelity	Does content outside the operational edit box remain close to the paired clean target?	Outside	PSNR, LPIPS
Temporal Fidelity	Does the output preserve legitimate target dynamics without model-induced instability?	BBox, Outside	Temporal Change Error; TWE-Excess

Table 3: Metric–view design. Bold views and metrics form the three primary dimensions; the remaining views are diagnostic or compatibility measurements. TC denotes flow-free Temporal Change Error, and TWE-X denotes RAFT-based TWE-Excess.

Setting	Method	Restoration: Subtitle			Preservation: Outside		Temporal Fidelity				Joint cov. (%) $\uparrow$
Setting	Method	PSNR $\uparrow$	LPIPS $\downarrow$	cDISTS $\downarrow$	PSNR $\uparrow$	LPIPS $\downarrow$	BBox TC $\downarrow$	Out. TC $\downarrow$	BBox TWE-X $\downarrow$	Out. TWE-X $\downarrow$	Joint cov. (%) $\uparrow$
Blind	CLEAR	11.99	0.621	0.236	21.48	0.221	5.00	1.77	15.042	0.076	85.4
Mask-guided	ProPainter	19.66	0.364	0.138	34.11	0.050	2.57	0.70	0.683	0.072	100.0
	DiffuEraser	21.36	0.269	0.116	32.87	0.062	2.53	0.73	0.686	0.068	99.9
	MiniMax-Remover	20.32	0.321	0.132	30.73	0.086	2.52	1.12	1.142	0.028	99.4
	OmnimatteZero	17.69	0.369	0.160	26.00	0.131	2.77	1.51	0.653	0.048	98.5

Table 4: Raw metrics on

\mathcal{A}_{300}

. PSNR is in dB; TC and TWE-X are scaled by

10^{2}

and

10^{3}

. Dark and light shading mark the best and second-best mask-guided means. ∗ EffectErase is evaluated on its verified prefix.

Figure 5: Decoupled quality on Audit300. (a) Subtitle versus Outside LPIPS. (b) BBox TWE-Excess versus BBox TC. Points show video-level macro means with video-bootstrap 95% CIs; lower-left is better. CLEAR (blind) is plotted separately with independent axis ranges; cross-facet distances are not comparable. The dashed line marks zero TWE-Excess; negative values are retained. ∗ EffectErase is evaluated on its verified prefix.

Figure 6: Performance across Audit300 dimensions. (a) Subtitle LPIPS, (b) Outside LPIPS, and (c) BBox TC (

\times 10^{2}

) represent Restoration, Preservation and Temporal fidelity. Points are video-level macro means and colored sectors indicate attribute domains. CLEAR is blind; ∗ EffectErase uses its verified prefix.

Method	Comp.	Part.	Non-eval.	Time cov.
CLEAR	5.6	94.4	$<0.1$	86.2
ProPainter	98.3	0.0	1.7	100.0
DiffuEraser	72.4	26.6	1.0	99.8
MiniMax-Remover	30.8	64.9	4.3	99.3
OmnimatteZero	21.0	75.2	3.8	98.1
EffectErase	3.9	93.7	2.4	47.1

Table 5: Full-set delivery (%). Rates use all 3,000 videos; temporal coverage is averaged over evaluable outputs.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Mask adapter	Condition	Spatial	Temporal / sampling	Numerics
CLEAR	None	Fixed removal prompt	$0.92$ Mpx; $a=16$ Scale-to-cover; center crop	Window 81; overlap 16 Denoising steps: 5; CFG 1.0	Precision logged; seed unset; tiled VAE
ProPainter	PNG sequence; NN; dilation 4	None	$2.07$ Mpx AR; $a=8$	RAFT: 20 iter.; subvideo 80 Neighbor 10; ref. stride 10	FP16
DiffuEraser	Video; binary; dilation 8	ProPainter prior from $\mathbf{X},\mathbf{Q}$	$2.07$ Mpx; $a=8$ Long side $\leq 1920$	Prior: subvideo 50, neighbor 10, ref. stride 10 2-Step checkpoint	FP16
MiniMax- Remover	Video; threshold $>20$ ; NN	No text	$2.07$ Mpx AR; $a=16$	Full supplied clip Sampling: 12 steps, 6 iter.	FP16; seed unset
OmnimatteZero	Video; threshold 128	Fixed Empty + negative prompts	$2.07$ Mpx AR; $a=32$	Full supplied clip Denoising steps: 30	BF16; seed 1; tiled VAE
EffectErase	Video; bilinear resizing	Fixed prompts; $224{\times}224$ crop *	$2.07$ Mpx AR; $a=16$	$\leq 81$ frames; output $4n+1$ Denoising steps: 50; CFG 1.0	BF16; seed 2025; tiled inference

Appendix

Table 6: Frozen inference configurations in MFVSR. Mpx: per-frame pixel budget; AR: aspect ratio retained;

a

: dimension alignment; NN: nearest-neighbor resizing. All video masks are lossless.

Attribute	Groups (videos)
Source	Real-HQ (123), Real-LQ (88), AI-generated (89)
Resolution	720p-class (177), 1080p-class (123)
Duration	$\leq 10$ s (168), 10–30 s (122), $>30$ s (10)
Style complexity	Easy (93), Medium (97), Hard (110)

Appendix

Table 7: Audit300 composition. Resolution labels denote evaluation classes.

Metric / tier	Overall	BBox	Subtitle	BBox-NG	Outside
PSNR / Lite	Compat.	Diag.	Primary	Diag.	Primary
SSIM / Audit	Compat.	Diag.	Diag.	Diag.	Diag.
LPIPS / Lite	Compat.	Diag.	Primary	Diag.	Primary
DISTS / Audit	Compat.	–	cDISTS	cDISTS	–
TWE / TWE-Excess Audit	–	Primary	–	–	Primary
TC / Lite	–	Primary	–	–	Primary

Appendix

Table 8: Metric–view matrix. Lite measurements cover Lite3000; Audit adds measurements on Audit300. Bold entries define the fidelity components. Overall is retained for compatibility; BBox-NG is a mixed local diagnostic.

Panel of Figure 7 — Figure 7: Human-evaluation interface for held-out protocol validation.

Method	Subtitle				Outside
	PSNR	SSIM	LPIPS	cDISTS	PSNR	LPIPS
CLEAR	11.99 [11.31, 12.75]	0.356 [0.322, 0.391]	0.621 [0.594, 0.651]	0.236 [0.225, 0.248]	21.48 [21.05, 21.90]	0.221 [0.211, 0.232]
ProPainter	19.66 [18.94, 20.41]	0.714 [0.694, 0.734]	0.364 [0.343, 0.390]	0.138 [0.130, 0.148]	34.11 [33.78, 34.44]	0.050 [0.047, 0.054]
DiffuEraser	21.36 [20.74, 21.94]	0.746 [0.726, 0.766]	0.269 [0.253, 0.287]	0.116 [0.110, 0.124]	32.87 [32.59, 33.16]	0.062 [0.057, 0.066]
MiniMax- Remover	20.32 [19.61, 20.96]	0.725 [0.700, 0.748]	0.321 [0.297, 0.346]	0.132 [0.123, 0.141]	30.73 [30.37, 31.12]	0.086 [0.080, 0.092]
OmnimatteZero	17.69 [17.09, 18.31]	0.651 [0.628, 0.673]	0.369 [0.352, 0.388]	0.160 [0.151, 0.168]	26.00 [25.53, 26.47]	0.131 [0.124, 0.139]

Appendix

Table 9: Audit300 spatial results: mean above video-bootstrap 95% CI. PSNR is in dB; LPIPS and cDISTS are lower-is-better. CLEAR is blind; other methods are mask-guided.

Method	BBox		Outside		Joint
	TC	TWE-X	TC	TWE-X	coverage (%)
CLEAR	5.00 [4.54, 5.43]	15.042 [12.605, 17.707]	1.77 [1.59, 1.98]	0.076 [ $-0.013$ , 0.175]	85.4 [83.9, 86.8]
ProPainter	2.57 [2.36, 2.81]	0.683 [0.481, 0.891]	0.70 [0.65, 0.75]	0.072 [0.017, 0.160]	100.0 [100.0, 100.0]
DiffuEraser	2.53 [2.28, 2.79]	0.686 [0.491, 0.923]	0.73 [0.68, 0.79]	0.068 [0.012, 0.152]	99.9 [99.8, 99.9]
MiniMax- Remover	2.52 [2.25, 2.85]	1.142 [0.508, 2.058]	1.12 [1.01, 1.24]	0.028 [ $-0.041$ , 0.117]	99.4 [99.3, 99.5]
OmnimatteZero	2.77 [2.49, 3.10]	0.653 [0.429, 0.899]	1.51 [1.34, 1.70]	0.048 [ $-0.032$ , 0.119]	98.5 [98.3, 98.7]

Appendix

Table 10: Audit300 temporal results: mean above video-bootstrap 95% CI. TC is scaled by

10^{2}

and TWE-X by

10^{3}

. CLEAR is blind; other methods are mask-guided.

Metric relation	Reversed pairs	Kendall $\tau_{b}$
Subtitle LPIPS vs. Outside LPIPS	5/10	0.00
BBox TWE-Excess vs. BBox TC	6/10	$-0.20$
Overall LPIPS vs. Subtitle LPIPS	4/10	0.20
Overall LPIPS vs. Outside LPIPS	1/10	0.80

Appendix

Table 11: Pairwise rank agreement across mask-guided methods. Metric-induced orderings are compared over five methods (10 pairs), including prefix-conditioned EffectErase; reversals indicate opposite pairwise preferences

Figure 8: Subtitle-factor contrasts on Audit300. Decorated–Plain (96/107), Bilingual–Monolingual (150/150), Dynamic–Fixed (145/155), and Animated–Static (150/150). Positive differences indicate higher error in the first group; bars are 95% bootstrap intervals. EffectErase uses its verified prefix.

Figure 9: Box-style stress results with video-bootstrap 95% intervals. Each measurement uses 464 videos except EffectErase’s Subtitle metrics (

n=463

, owing to one empty glyph support). Quality uses verified outputs, including partial prefixes; missing values are not imputed.

Method	C	P	IF	MA	EF
CLEAR	168	2,831	0	0	1
ProPainter	2,949	0	50	0	1
DiffuEraser	2,171	799	29	0	1
MiniMax-Remover	924	1,947	128	0	1
OmnimatteZero	630	2,255	91	23	1
EffectErase	117	2,810	72	0	1

Appendix

Table 12: Terminal-state counts: complete (C), verified partial (P), inference failure (IF), missing artifact (MA), and evaluation failure (EF). Each row sums to 3,000.

Figure 10: Delivery across resolution and duration groups. Rates use all scheduled videos in each group, including failed and missing outputs. Terminal-state mixtures distinguish verified truncation from failure.

Figure 11: Quality–coverage relationship across evaluation scopes. Left: temporal coverage over all scheduled videos. Right: Audit300 Subtitle LPIPS versus Lite3000 complete-delivery rate. Quality intervals are video-bootstrap 95% CIs. EffectErase quality is computed on verified prefixes, while delivery is assessed against full targets.

Method	Native set	Additional videos
CLEAR	2,499	201
ProPainter	2,456	158
DiffuEraser	2,474	176
MiniMax-Remover	2,392	94
OmnimatteZero	2,407	109
EffectErase	2,436	138

Appendix

Table 13: Method-native evaluable scope in the non-box primary pool. All methods share 2,298 videos; native sets include verified partial outputs.

Figure 12: Conditional quality on common and additional videos. Points are medians, and segments extend to the 90th percentile, not confidence limits. Values are multiplied by 100.

Task	Decisive $n$	Undecided (%)	$\kappa$
Restoration	83	17	0.544
Preservation	58	42	0.517
Temporal	39	61	0.123
Overall	84	16	0.578

Appendix

Table 14: Human-response decisiveness and agreement. Each task contains 100 pairs;

\kappa

is Fleiss’ inter-annotator agreement.

Human task	Candidate	Accuracy (%)	95% CI	$\tau_{b}$
Restoration	Restoration component	81.9	[72.3, 88.7]	0.471
$(n=83)$	Subtitle LPIPS	88.0	[79.2, 93.3]	0.501
	Subtitle PSNR	81.9	[72.3, 88.7]	0.443
	Subtitle cDISTS	75.9	[65.7, 83.8]	0.391
Preservation	Preservation component	91.4	[81.4, 96.3]	0.559
$(n=58)$	Outside LPIPS	91.4	[81.4, 96.3]	0.519

Appendix

Table 15: Held-out metric–human alignment. Accuracy uses decisive preferences; intervals are Wilson 95% CIs. Kendall’s

\tau_{b}

uses signed aggregate preferences.

Figure 13: Human alignment and scalar comparison. (a) Matched-task agreement with Wilson 95% intervals;

n

counts decisive pairs and the dashed line marks 50%. (b) SR-Q minus baseline accuracy on the same 84 decisive Overall pairs, with paired-bootstrap 95% intervals. Component and Overall rows address different judgments and should be interpreted within their tasks.

Method	Timed records	GPU-hours
CLEAR	3,000	844.6
ProPainter	3,000	289.6
DiffuEraser	3,000	738.9
MiniMax-Remover	2,909	4,500.0
OmnimatteZero	2,977	2,021.4
EffectErase	2,928	1,742.9

Appendix

Table 16: Recorded inference workload. Each method has 3,000 scheduled videos.

Video Restoration

Video restoration reconstructs clearer, more faithful, temporally consistent footage from blur, noise, compression, weather, and other degradations.

See all 58 papers

CLEAR: Context-Aware Learning with End-to-End Mask-Free Inference for Adaptive Video Subtitle Removal
Mar 23, 2026Qingdong He, Chaoyi Wang, Peng Tang +2Video Editing Video Diffusion Models
University of Electronic Science and Technology of China · University of Chinese Academy of Sciences · Technical University of Munich +2
SEDiT: Mask-Free Video Subtitle Erasure via One-step Diffusion Transformer
May 14, 2026Zheng Hui, Yunlong BaiDiffusion Transformer Video Editing
Baidu Inc., Beijing, China.
DeOverlay: A Benchmark and Source-Anchored Flow for Video Overlay Removal
Oct 19, 2026Deshui Miao, Yuhang Ma, Shaoyong Jia +4Flow Matching
ByteDance · Southern University of Science and Technology

Mar 23, 2026J/K move · Enter open · S save

CLEAR: Context-Aware Learning with End-to-End Mask-Free Inference for Adaptive Video Subtitle Removal

Qingdong He, Chaoyi Wang, Peng Tang, Yifan Yang+1

University of Electronic Science and Technology of China · University of Chinese Academy of Sciences · Technical University of Munich+2

Video Editing Video Diffusion Models

Open paper AskPDF arXiv

Video subtitle removal aims to distinguish text overlays from background content while preserving temporal coherence. Existing diffusion-based methods necessitate explicit mask sequences during both training and inference phases, which restricts their practical deployment. In this paper, we present CLEAR (Context-aware Learning for End-to-end Adaptive Video Subtitle Removal), a mask-free framework that achieves truly end-to-end inference through context-aware adaptive learning. Our two-stage design decouples prior extraction from generative refinement: Stage I learns disentangled subtitle representations via self-supervised orthogonality constraints on dual encoders, while Stage II employs LoRA-based adaptation with generation feedback for dynamic context adjustment. Notably, our method only requires 0.77% of the parameters of the base diffusion model for training. On Chinese subtitle benchmarks, CLEAR outperforms mask-dependent baselines by + 6.77dB PSNR and -74.7% VFID, while demonstrating superior zero-shot generalization across six languages (English, Korean, French, Japanese, Russian, German), a performance enabled by our generation-driven feedback mechanism that ensures robust subtitle removal without ground-truth masks during inference.

Abstract

Video Restoration

Explore similar work

CLEAR: Context-Aware Learning with End-to-End Mask-Free Inference for Adaptive Video Subtitle Removal

SEDiT: Mask-Free Video Subtitle Erasure via One-step Diffusion Transformer

DeOverlay: A Benchmark and Source-Anchored Flow for Video Overlay Removal