CLEAR: Context-Aware Learning with End-to-End Mask-Free Inference for Adaptive Video Subtitle Removal

Qingdong He; Chaoyi Wang; Peng Tang; Yifan Yang; Xiaobin Hu

cs.CVMar 23, 2026

CLEAR: Context-Aware Learning with End-to-End Mask-Free Inference for Adaptive Video Subtitle Removal

Authors: Qingdong He, Chaoyi Wang, Peng Tang, Yifan Yang, Xiaobin Hu

Organizations: University of Electronic Science and Technology of China · University of Chinese Academy of Sciences · Technical University of Munich · Shanghai Jiao Tong University ·

Abstract

Video subtitle removal aims to distinguish text overlays from background content while preserving temporal coherence. Existing diffusion-based methods necessitate explicit mask sequences during both training and inference phases, which restricts their practical deployment. In this paper, we present CLEAR (Context-aware Learning for End-to-end Adaptive Video Subtitle Removal), a mask-free framework that achieves truly end-to-end inference through context-aware adaptive learning. Our two-stage design decouples prior extraction from generative refinement: Stage I learns disentangled subtitle representations via self-supervised orthogonality constraints on dual encoders, while Stage II employs LoRA-based adaptation with generation feedback for dynamic context adjustment. Notably, our method only requires 0.77% of the parameters of the base diffusion model for training. On Chinese subtitle benchmarks, CLEAR outperforms mask-dependent baselines by + 6.77dB PSNR and -74.7% VFID, while demonstrating superior zero-shot generalization across six languages (English, Korean, French, Japanese, Russian, German), a performance enabled by our generation-driven feedback mechanism that ensures robust subtitle removal without ground-truth masks during inference.

Figures & tables

Figure 2: CLEAR framework overview. Two-stage training enables mask-free inference: Stage I learns self-supervised priors $\mathcal{M}^{prior}$ through disentangled feature learning; Stage II trains LoRA-adapted diffusion with adaptive weighting optimized by distillation, generation feedback, and sparsity losses. Inference requires only subtitled video input.

Figure 3: CLEAR Pipeline Details. Stage I: Dual encoders extract disentangled features with orthogonality and adversarial losses. Stage II: Occlusion head predicts adaptive weights from DiT encoder features, modulating generation through focal weighting $(\epsilon^{gen})^{\gamma}$ with gradient backflow.

Figure 4: Illustration of our progressive mask refinement. Despite noisy pseudo-labels from Stage I ( $\mathcal{M}^{prior}$ ), Stage II learns adaptive weights ( $\mathcal{M}^{pred}$ ) that dynamically adjust based on generation difficulty, enabling robust context-aware modulation, which are internal to training and not predicted during inference.

Method	Reconstruction			Perceptual		Temporal		Flow		Time
Method	PSNR↑	SSIM↑	LPIPS↓	DISTS↓	VFID↓	TWE↓	TC↓	Mean↓	Var↓	s/frame↓
ProPainter ( Zhou et al., 2023 )	17.24	0.658	0.3291	0.245	98.46	1.286	1.160	0.952	0.885	2.36
Minimax-Remover ( Zi et al., 2025 )	20.03	0.773	0.166	0.119	95.39	4.222	3.016	0.841	0.415	4.90
DiffuEraser ( Li et al., 2025 )	17.85	0.672	0.458	0.228	72.51	1.523	1.174	0.840	0.630	3.47
CLEAR (Ours)	26.80	0.894	0.101	0.075	20.37	1.227	1.049	0.209	0.029	4.86

Table 1: Quantitative comparison with state-of-the-art methods on Chinese subtitle test set. CLEAR uses default configuration (rank=64, steps=5, cfg=1.0, lora_scale=1.0).

Figure 5: Qualitative comparison with baseline methods. Unlike ProPainter, Minimax-Remover, and DiffuEraser that require explicit masks during inference, CLEAR achieves mask-free end-to-end removal with clean subtitle elimination and fine-grained detail preservation.

Configuration	Reconstruction			Perceptual		Temporal		Flow
Configuration	PSNR↑	SSIM↑	LPIPS↓	DISTS↓	VFID↓	TWE↓	TC↓	Mean↓	Var↓
Baseline (LoRA-only)	21.62	0.855	0.131	0.088	34.74	1.320	1.137	0.217	0.032
+ M1: Stage I Prior with Focal Weighting	23.11	0.868	0.130	0.091	38.21	1.303	1.123	0.216	0.031
+ M2: Context Distillation	24.72	0.890	0.110	0.083	31.73	1.279	1.102	0.212	0.030
+ M3: Context-Aware Adaptation	25.09	0.891	0.109	0.083	31.56	1.257	1.082	0.211	0.029
+ M4: Context Consistency (CLEAR)	26.80	0.894	0.101	0.075	20.37	1.227	1.049	0.209	0.029

Table 2: Ablation study on CLEAR’s modular design. Each row progressively adds one component.

Configuration	Reconstruction			Perceptual		Temporal		Flow		Time
Configuration	PSNR↑	SSIM↑	LPIPS↓	DISTS↓	VFID↓	TWE↓	TC↓	Mean↓	Var↓	s/frame↓
Denoising Steps (lora_scale=1.0, cfg=1.0)
steps=5	26.80	0.894	0.101	0.075	20.37	1.227	1.049	0.209	0.029	4.86
steps=10	29.43	0.884	0.101	0.081	35.70	1.358	1.243	0.215	0.030	9.92
CFG Scale (lora_scale=1.0, steps=5)
cfg=0.8	25.24	0.899	0.109	0.083	30.05	1.257	1.078	0.210	0.028	4.86

Table 3: Results of key inference hyperparameters on performance.

Figure 6: Visual comparison of output quality between the LoRA-only baseline and CLEAR (Ours). The baseline struggles with residual subtitles and background blurring, while our two-stage framework achieves clean subtitle removal and faithful preservation of structural details.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A1 : Qualitative comparison with baseline methods. Unlike ProPainter, Minimax-Remover, and DiffuEraser that require explicit masks during inference, CLEAR achieves mask-free end-to-end removal with clean subtitle elimination and fine-grained detail preservation.

Figure A2 : Qualitative comparison with baseline methods. Unlike ProPainter, Minimax-Remover, and DiffuEraser that require explicit masks during inference, CLEAR achieves mask-free end-to-end removal with clean subtitle elimination and fine-grained detail preservation.

Figure A3 : Zero-shot cross-lingual generalization. CLEAR achieves robust subtitle removal across English, Korean, French, Japanese, Russian, and Arabic without language-specific training. The mask overlays visualize the learned context-aware guidance ability during Stage II training, demonstrating accurate subtitle localization; notably, these masks are not predicted during inference.

Figure A4 : Zero-shot cross-lingual generalization. CLEAR achieves robust subtitle removal across English, Korean, French, Japanese, Russian, and Arabic without language-specific training. The mask overlays visualize the learned context-aware guidance ability during Stage II training, demonstrating accurate subtitle localization; notably, these masks are not predicted during inference.

Figure A5 : Zero-shot cross-lingual generalization. CLEAR achieves robust subtitle removal across English, Korean, French, Japanese, Russian, and Arabic without language-specific training. The mask overlays visualize the learned context-aware guidance ability during Stage II training, demonstrating accurate subtitle localization; notably, these masks are not predicted during inference.

Video Editing

Video editing uses AI to alter, compose, relight, or reorganize moving footage while controlling how changes persist across frames.

See all 178 papers

May 14, 2026

SEDiT: Mask-Free Video Subtitle Erasure via One-step Diffusion Transformer

Recent breakthroughs in video diffusion models have significantly accelerated the development of video editing techniques. However, existing methods often rely on inpainting video frames based on masked input, which requires extracting the target video mask in advance, and the precision of the segmentation directly affects the quality of the completion. In this paper, we present SEDiT, a novel one-stage video Subtitle Erasure approach via One-step Diffusion Transformer. We introduce a mask-free inference approach that enables direct erasure of the targeted subtitle. The proposed one-stage framework mitigates the sub-optimality inherent in the two-stage processing of prior models. Since subtitle removal is a localized editing task in which most pixels remain unchanged, the underlying distribution shift is minimal, making it well-suited to one-step generation under rectified flow. We empirically validate the reliability of one-step denoising and further provide a formal theoretical justification. Under the localized-editing structure of subtitle removal, the conditional optimal transport (OT) map and its induced rectified flow velocity field are Lipschitz continuous with respect to the latent variable, which underpins the theoretical feasibility of one-step sampling. To address the challenge of long-term temporal consistency, we adopt a hybrid training strategy by occasionally conditioning the model with a clean first-frame latent. This facilitates temporal continuity, allowing each segment during inference to leverage the output of its predecessor. To avoid visible seams caused by cropping and reinserting processed targets, particularly in scenarios involving substantial motion, we feed the original video directly into SEDiT. Thanks to one-step and chunk-wise streaming inference, our method can efficiently handle native 1440p video with infinite length.

Zheng Hui, Yunlong Bai

Baidu Inc., Beijing, China.

Diffusion Transformer Video Editing Diffusion Models

Ask PDF

Oct 13, 2026

MFVSR: A Multi-Style and Fine-Grained Benchmark for Video Subtitle Removal

Video subtitle removal (VSR) is an important video restoration task, yet existing evaluation remains limited by low-resolution datasets, fragmented test sets, and generic video-inpainting protocols that fail to adequately capture localized restoration errors, unintended modifications, temporal instability, and incomplete outputs. In this paper, we introduce MFVSR, a large-scale multi-style benchmark for fine-grained and reliability-aware video subtitle removal evaluation. MFVSR distinguishes itself through three key features: (1) Large-scale and diverse data, comprising 3,000 paired videos from Real-HQ, Real-LQ, and AI-Generated sources, with an auxiliary set of 953 in-the-wild real videos for real-world coverage analysis; (2) Controlled subtitle diversity, spanning 16 languages and four factors covering spatial layout, temporal behavior, visual style, and language mode; (3) Fine-grained evaluation, introducing an information-conditioned, spatially decoupled protocol that separates blind from mask-guided removal and measures Restoration Fidelity, Preservation Fidelity, Temporal Fidelity, coverage, and failures. Experiments on six representative methods reveal substantial discrepancies between local restoration, outside-region preservation, temporal consistency, and full-set reliability that are obscured by conventional full-frame metrics. MFVSR therefore provides a more comprehensive and diagnostic foundation for evaluating and advancing video subtitle removal methods.

Xiyan Huang, Qingdong He, Peng Tang +3

National University of Singapore · Tencent Youtu Lab · University of Chinese Academy of Sciences

Video Restoration Video Editing

Ask PDF

Oct 19, 2026

DeOverlay: A Benchmark and Source-Anchored Flow for Video Overlay Removal

Video overlay removal aims to recover scenes obscured by subtitles, titles, and logos while preserving surrounding content. Existing methods use temporal propagation or source-conditioned generation, but can leave residual overlays and distort scene structure when useful source information is obscured. We propose DeOverlay, a source-anchored flow model that reconstructs the clean video from the source latent with fixed source guidance. Specifically, we train a shared velocity network along paired source-to-clean paths using region-weighted flow matching, enabling successive cor- rections from intermediate states. To select local source cues, we develop visibility-normalized temporal aggregation that excludes masked positions and averages the remaining observations. Global source context and edit-region embeddings complement these cues through additive condi- tioning, retaining a single video-token sequence. To evaluate scene recovery across diverse overlay conditions, we introduce DVTE-Bench, comprising 1,631 paired videos across eight text-overlay categories, with OCR-screened clean targets, frame-aligned pixel masks, and a common evaluation protocol. Experiments on the complete release show that DeOverlay outperforms four baselines on all five reconstruction and perceptual metrics, reducing Mask LPIPS and Mask VFID by 58.3% and 70.3%, respectively, relative to the strongest evaluated baseline. Additional qualitative examples demonstrate the removal of graphic logos and stickers. We will make the complete benchmark, evaluation code, and test outputs availabl. Project page:https://yahooo-m.github.io/DVTE-Bench/

Deshui Miao, Yuhang Ma, Shaoyong Jia +4

ByteDance · Southern University of Science and Technology

Flow Matching

Ask PDF

Abstract

Video Editing

Explore similar work

SEDiT: Mask-Free Video Subtitle Erasure via One-step Diffusion Transformer

MFVSR: A Multi-Style and Fine-Grained Benchmark for Video Subtitle Removal

DeOverlay: A Benchmark and Source-Anchored Flow for Video Overlay Removal