DeOverlay: A Benchmark and Source-Anchored Flow for Video Overlay Removal

Deshui Miao; Yuhang Ma; Shaoyong Jia; Wenzhao Gao; Yue Zhang; Shaohui Jiao; Xin Li

cs.CVOct 19, 2026

DeOverlay: A Benchmark and Source-Anchored Flow for Video Overlay Removal

Authors: Deshui Miao, Yuhang Ma, Shaoyong Jia, Wenzhao Gao, Yue Zhang, Shaohui Jiao, Xin Li

Organizations: ByteDance · Southern University of Science and Technology

Abstract

Video overlay removal aims to recover scenes obscured by subtitles, titles, and logos while preserving surrounding content. Existing methods use temporal propagation or source-conditioned generation, but can leave residual overlays and distort scene structure when useful source information is obscured. We propose DeOverlay, a source-anchored flow model that reconstructs the clean video from the source latent with fixed source guidance. Specifically, we train a shared velocity network along paired source-to-clean paths using region-weighted flow matching, enabling successive cor- rections from intermediate states. To select local source cues, we develop visibility-normalized temporal aggregation that excludes masked positions and averages the remaining observations. Global source context and edit-region embeddings complement these cues through additive condi- tioning, retaining a single video-token sequence. To evaluate scene recovery across diverse overlay conditions, we introduce DVTE-Bench, comprising 1,631 paired videos across eight text-overlay categories, with OCR-screened clean targets, frame-aligned pixel masks, and a common evaluation protocol. Experiments on the complete release show that DeOverlay outperforms four baselines on all five reconstruction and perceptual metrics, reducing Mask LPIPS and Mask VFID by 58.3% and 70.3%, respectively, relative to the strongest evaluated baseline. Additional qualitative examples demonstrate the removal of graphic logos and stickers. We will make the complete benchmark, evaluation code, and test outputs availabl. Project page:https://yahooo-m.github.io/DVTE-Bench/

Figures & tables

Figure 1: Starting state and source conditioning. Channel- and sequence-conditioned editors start from noise, whereas ours starts from $z_{s}$ and receives fixed source features at each update. On a shared patch grid, the three designs use $N$ , $2N$ , and $N$ video tokens per call, with auxiliary tokens and codecs omitted from the diagram.

Benchmark	Clips	Release	Ground Truth		Overlay Diversity					Evaluation
Benchmark	Clips	Release	Clean	Mask	Types	Multilingual	Spatial	Temporal	Appearance	Regional	Temporal
CLEAR test set	400	✗	✓	✗	1	✗	✓	✗	✓	✗	✓
VSR-Bench-400	400	✗	✓	✓	1	✓	✓	✗	✓	✗	✗
DVTE-Bench (ours)	1,631	✓	✓	✓	8	✓	✓	✓	✓	✓	✓

Table 1: Comparison of video text erasure benchmarks. CLEAR [ 5 ] and VSR-Bench-400 [ 8 ] are compared with DVTE-Bench. The DVTE-Bench release checkmark denotes its planned public release.

Figure 2: DVTE-Bench construction. Text plans and type-controlled rendering produce aligned source videos, clean targets, and overlay masks. Target-frame OCR screening rejects 569 of 2,200 candidates and retains 1,631 clips. The eight thumbnails show distinct released clips, while the lower-right source, target, and mask form one aligned example.

Figure 3: DeOverlay overview. (a) Shared-weight Euler updates start from the source latent, with the upper bypass carrying the current state and the lower loop updating state and time with fixed source cues. (b) Visibility-aware source aggregation (VSA), global source conditioning (GSC), and mask embeddings form the source condition. (c) Source cues are added to the current state tokens before joint video–text processing. Tensor grids and intermediate states are schematic, whereas RGB frames are matched source/output pairs.

(a) Regional reconstruction and perceptual quality
Method	PSNR ${}^{\dagger}\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	DISTS $\downarrow$	VFID $\downarrow$
CLEAR	16.02	0.5172	0.3209	0.2158	1.4736
ProPainter	25.90	0.7832	0.1166	0.0957	0.3879
MiniMax-Remover	26.17	0.8044	0.0977	0.0856	0.2806
DiffuEraser	24.84	0.7723	0.1130	0.0952	0.3285
DeOverlay, $K=6$	30.80	0.8822	0.0408	0.0507	0.0835

Table 2: DVTE-Bench results on all 1,631 videos , using the released pixel-mask protocol. (a) PSNR † excludes infinite values, and VFID is a global video-feature distance. (b) TWE Gap is

|\overline{\mathrm{TWE}}_{\rm pred}-\overline{\mathrm{TWE}}_{\rm GT}|

, with clean-target TWE 9.3228 and diagnostics defined in Appendix I . Bold marks best values in (a), while bold/underline in (b) mark the closest/second-closest TWE Pred to GT and the lowest/second-lowest values otherwise.

Figure 4: Paired benchmark comparisons. Five frame-40 examples with shared yellow crop boxes compare four baselines, DeOverlay, and clean targets, with inference settings and expanded seven-category comparisons in Appendix E .

Figure 5: Removal across overlay appearances. Twelve matched source/output crops cover subtitles, titles, watermarks, a sticker, and a television logo, with full frames and crop boxes in Appendix E .

Variant	PSNR $\uparrow$	MAE $\downarrow$	Crop SSIM $\uparrow$	Outside MAE $\downarrow$	TWE gap $\downarrow$	$\Delta$ PSNR
DeOverlay, $K=4$	32.461	4.299	0.92765	2.541	0.180	0.000
Direct residual, $K=1$	31.300	4.520	0.91120	2.548	0.195	-1.161
Noise init. + source tokens	29.600	6.500	0.89200	3.050	0.320	-2.861
Mask only (no VSA/GSC)	32.230	4.440	0.92590	2.552	0.197	-0.231
Ordinary temporal mean	32.360	4.360	0.92725	2.542	0.183	-0.101
Without VSA	32.370	4.350	0.92705	2.543	0.185	-0.091

Table 3: Core design ablations on 200 randomly sampled videos. Component variants use

K=4

, and the direct-residual reference uses

K=1

. PSNR and MAE use overlay pixels, with

\Delta

PSNR measured relative to the full four-step model. VSA supplies local source context and GSC supplies global context, while the noise-start variant concatenates noise and source video tokens. Bold marks the best value in each quality column.

$K$	Mask PSNR $\uparrow$	Mask MAE $\downarrow$	MaskCrop SSIM $\uparrow$	TWE abs. gap $\downarrow$	Rel. latency
2	32.410	4.330	0.92710	0.184	$0.82\times$
4	32.461	4.299	0.92765	0.180	$1.00\times$
6	32.467	4.294	0.92772	0.178	$1.25\times$
8	32.468	4.293	0.92774	0.178	$1.50\times$

Table 4: Flow step count on 200 randomly sampled videos. Only the inference update count changes under a fixed checkpoint, with latency normalized to

K=4

and the best quality metric marked in bold.

Method	E2E s/frame $\downarrow$	E2E FPS $\uparrow$	Inference s/frame $\downarrow$	Peak GiB $\downarrow$
CLEAR	0.6030	1.6584	0.5983	18.354
ProPainter	0.4271	2.3413	0.4134	34.101
MiniMax-Remover	0.3787	2.6405	0.3530	10.776
DiffuEraser	1.1677	0.8564	1.1524	23.950
DeOverlay, $K=1$	0.3242	3.0847	0.3001	12.236
DeOverlay, $K=4$	0.4828	2.0712	0.4587	12.236

Table 5: Speed and memory on the same A100. All rows process the same eight clips (650 frames), with E2E denoting end-to-end time excluding model initialization. Inference time follows each runner’s CUDA-synchronized interval, and peak memory uses a common external GPU-memory sampler. Bold marks the best value in each column.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Backbone	LingBot-Video dense 1.3B configuration
Latent channels / patch size	16 / $(1,2,2)$
Transformer blocks / width / heads	24 / 2048 / 16
Feed-forward width / text embedding width	6144 / 2560
LoRA rank / scale	256 / 1.0
Local / global branch hidden width	512 / 512

Appendix

Table 6: Configured model and optimization settings.

Challenge type	Overlay characteristic	Release	Evaluated
Standard subtitle	Lower-third captions and style variation	232	232
Artistic title	Display typography, outlines, and glow	218	218
Moving text	Frame-varying trajectories	232	232
Vertical text	Vertical writing at varied positions	215	215
Non-bottom layout	Placement outside the bottom region	223	223
Multi-region	Distributed simultaneous text regions	225	225

Appendix

Table 7: DVTE-Bench coverage and complete evaluation. All released clips are evaluated by five methods under the protocol defined in Section 3.3 .

Figure 6: DVTE-Bench release distributions. All panels count the 1,631 released clips: (a) assigned overlay type, (b) language metadata, (c) frame orientation, and (d) the per-clip number of rendered overlay elements. “Other labels” combines Italian (2) and mixed, Spanish, Portuguese, Polish, and Welsh (1 each). Overlay-element count does not measure spatial regions or pixel coverage.

Text type	Clips	PSNR ${}^{\dagger}\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	DISTS $\downarrow$	$N_{\infty}$
Standard subtitle	232	30.39	0.8718	0.0285	0.0395	1
Artistic title	218	26.24	0.8409	0.0674	0.0668	0
Moving text	232	30.73	0.8800	0.0209	0.0325	4
Vertical text	215	30.46	0.8736	0.0975	0.1015	3
Non-bottom layout	223	30.92	0.8843	0.0414	0.0628	2
Multi-region	225	29.88	0.8834	0.0103	0.0167	2

Appendix

Table 8: DeOverlay across all eight DVTE-Bench types. All 1,631 clips are included, with the 40K checkpoint using GT bounding-box masks, six steps, and the released seamless paste-back setting. Scoring uses pixel masks, and

N_{\infty}

counts clips omitted only from finite-mean PSNR.

Figure 7: Expanded comparison across seven overlay categories. Every row shows an identical frame and common crop for the source, four baselines, DeOverlay ( $K=4$ ), and clean target, with yellow boxes locating the crops in small source frames.

Figure 8: Benchmark context: titles, moving text, and spatial layouts. Each row uses the same clip and frame for all methods, with yellow boxes marking the source-coordinate crops in Figures 4 and 7 .

Figure 9: Benchmark context: watermarks, subtitles, and vertical text. Full frames retain the surrounding scene and all overlaid regions, while yellow boxes locate the identical crops in Figure 7 .

Figure 10: Application context for Figure 5 (a–f). Adjacent source/output frames preserve native orientation, with yellow boxes locating the matching main-paper crops and zero-based indices identifying the frames.

Figure 11: Application context for Figure 5 (g–l). Full source/output pairs include titles, watermarks, captions, a sticker, and the selected television-logo edit, which retains unrelated program graphics in the logo-only case. Yellow boxes locate the main-paper crops.

Figure 12: Multilingual generalization on application videos. Six language settings include Thai, Indonesian, Japanese–Chinese, Vietnamese–Chinese, German, and Portuguese overlays. Each row contains two matched source/DeOverlay pairs, with timestamps identifying the displayed frames. For these application clips without clean targets, input and output use identical crop coordinates and display scales.

Figure 13: Additional source/output examples. Paired full frames show subtitles, a text watermark, a name caption, and mixed text overlays from the earlier export collection, with native orientations preserved for these application clips without clean targets.

Input organization	Video tokens	Total tokens	Time (s) $\downarrow$	Memory (GiB) $\downarrow$
Source-initialized stream	13,248	13,346	5.266	1.744
Noise and source streams	26,496	26,594	13.088	3.479

Appendix

Table 9: One versus two video-token streams. The microbenchmark uses real 32K weights in bfloat16 and a latent tensor of shape

1\times 16\times 23\times 36\times 64

, corresponding to an 81-frame input with eight mirror-padding frames at the 768s bucket. Both inputs include 98 text tokens, and timing covers four transformer forwards while excluding VAE encoding, decoding, and video processing. Memory is incremental peak allocation above the loaded runtime.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	DISTS $\downarrow$	$N_{\infty}$
Standard subtitle (232 clips)
CLEAR	17.98	0.5726	0.2192	0.1567	0
ProPainter	25.17	0.7668	0.0890	0.0778	1
MiniMax-Remover	24.92	0.7752	0.0838	0.0755	0
DiffuEraser	24.17	0.7501	0.0899	0.0783	1
DeOverlay ( $K=6$ )	30.39	0.8718	0.0285	0.0395	1

Appendix

Table 10: Reconstruction across overlay categories (1/2). All five methods are evaluated on every clip in each category under the same pixel-mask scoring protocol, with DeOverlay using the 40K checkpoint and six steps. PSNR is the finite mean,

N_{\infty}

counts infinite-valued video scores, and bold marks the best quality score within each category.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	DISTS $\downarrow$	$N_{\infty}$
Non-bottom layout (223 clips)
CLEAR	17.76	0.5941	0.3264	0.2282	0
ProPainter	24.84	0.7627	0.1462	0.1269	2
MiniMax-Remover	25.31	0.7909	0.1195	0.1137	1
DiffuEraser	24.22	0.7582	0.1356	0.1229	2
DeOverlay ( $K=6$ )	30.92	0.8843	0.0414	0.0628	2

Appendix

Table 11: Reconstruction across overlay categories (2/2). Settings and notation follow Table 10 , with the two tables together covering all 1,631 released clips.

Flow Matching

Flow matching learns continuous-time vector fields that transport simple noise distributions into data for efficient generative modeling.

See all 650 papers

Mar 23, 2026

CLEAR: Context-Aware Learning with End-to-End Mask-Free Inference for Adaptive Video Subtitle Removal

Video subtitle removal aims to distinguish text overlays from background content while preserving temporal coherence. Existing diffusion-based methods necessitate explicit mask sequences during both training and inference phases, which restricts their practical deployment. In this paper, we present CLEAR (Context-aware Learning for End-to-end Adaptive Video Subtitle Removal), a mask-free framework that achieves truly end-to-end inference through context-aware adaptive learning. Our two-stage design decouples prior extraction from generative refinement: Stage I learns disentangled subtitle representations via self-supervised orthogonality constraints on dual encoders, while Stage II employs LoRA-based adaptation with generation feedback for dynamic context adjustment. Notably, our method only requires 0.77% of the parameters of the base diffusion model for training. On Chinese subtitle benchmarks, CLEAR outperforms mask-dependent baselines by + 6.77dB PSNR and -74.7% VFID, while demonstrating superior zero-shot generalization across six languages (English, Korean, French, Japanese, Russian, German), a performance enabled by our generation-driven feedback mechanism that ensures robust subtitle removal without ground-truth masks during inference.

Qingdong He, Chaoyi Wang, Peng Tang +2

University of Electronic Science and Technology of China · University of Chinese Academy of Sciences · Technical University of Munich +2

Video Editing Video Diffusion Models

Ask PDF

Oct 13, 2026

MFVSR: A Multi-Style and Fine-Grained Benchmark for Video Subtitle Removal

Video subtitle removal (VSR) is an important video restoration task, yet existing evaluation remains limited by low-resolution datasets, fragmented test sets, and generic video-inpainting protocols that fail to adequately capture localized restoration errors, unintended modifications, temporal instability, and incomplete outputs. In this paper, we introduce MFVSR, a large-scale multi-style benchmark for fine-grained and reliability-aware video subtitle removal evaluation. MFVSR distinguishes itself through three key features: (1) Large-scale and diverse data, comprising 3,000 paired videos from Real-HQ, Real-LQ, and AI-Generated sources, with an auxiliary set of 953 in-the-wild real videos for real-world coverage analysis; (2) Controlled subtitle diversity, spanning 16 languages and four factors covering spatial layout, temporal behavior, visual style, and language mode; (3) Fine-grained evaluation, introducing an information-conditioned, spatially decoupled protocol that separates blind from mask-guided removal and measures Restoration Fidelity, Preservation Fidelity, Temporal Fidelity, coverage, and failures. Experiments on six representative methods reveal substantial discrepancies between local restoration, outside-region preservation, temporal consistency, and full-set reliability that are obscured by conventional full-frame metrics. MFVSR therefore provides a more comprehensive and diagnostic foundation for evaluating and advancing video subtitle removal methods.

Xiyan Huang, Qingdong He, Peng Tang +3

National University of Singapore · Tencent Youtu Lab · University of Chinese Academy of Sciences

Video Restoration Video Editing

Ask PDF

Sep 30, 2026

TripleFlow: Training-Free Video Object Removal by Bridging Residual Editing and Native Generation

Video object removal presents a uniquely difficult editing challenge. Because a removal prompt specifies only what to erase rather than what to generate, the model must infer and reconstruct a highly specific occluded background entirely from the surrounding context. Existing training-free methods struggle with this because their editing mechanisms act primarily as localized erasers. They fail to actively synthesize the missing background details and often leave behind ghosting artifacts. To solve this, we propose TripleFlow, a training-free framework that tightly couples erasure and generation. It coordinates a source flow, a residual flow, and a synthesis flow throughout the entire process. By reusing a single target prediction, the residual flow isolates and suppresses the object, while the synthesis flow independently reconstructs the occluded background. Crucially, TripleFlow injects this newly synthesized background back into the editing trajectory at every step. This continuous feedback loop ensures that the generated structures actively guide the removal process, achieving seamless completion that is spatiotemporally consistent with the unedited scene. Extensive evaluations across five challenging benchmarks demonstrate that TripleFlow establishes a new state-of-the-art, significantly outperforming existing baselines in both reconstruction fidelity and temporal consistency.

Songhe Wang, Lifu Wei, Shuolin Xu +2

CSE Department, Penn State University · Department of Computer Science, The University of British Columbia, Canada · Department of Computing, Bournemouth University, United Kingdom +2

Video Editing Image Inpainting Temporal Consistency in Video Generation Object Removal

Ask PDF

Abstract

Flow Matching

Explore similar work

CLEAR: Context-Aware Learning with End-to-End Mask-Free Inference for Adaptive Video Subtitle Removal

MFVSR: A Multi-Style and Fine-Grained Benchmark for Video Subtitle Removal

TripleFlow: Training-Free Video Object Removal by Bridging Residual Editing and Native Generation