Hint, Not Hole: Localization-Robust Video Overlay Removal
Organizations: Malanshan Audio & Video Laboratory
Abstract
Video overlay removal recovers the scene beneath burned-in subtitles, karaoke lyrics, titles, credits, logos and watermarks. Generative video models now fill a known region convincingly, but in practice the region comes from a detector or a segmenter, and it is wrong in both directions: it misses strokes, halos, whole frames and whole objects, and it covers background between characters, around boxes and on genuine scene text. Erasers that treat the mask as a hole, regenerating inside and copying outside, inherit every one of these errors. We present HintErase, which treats localization as a hint, not a hole. The model sees the full, unmasked video together with a hint channel; during training the hint is corrupted the way real localizers fail, while supervision follows the true overlay. The result is a single model that removes overlay the hint missed, keeps background the hint wrongly covered, and is trained to keep genuine scene text even when the hint lands on it. For full-length releases, a streaming pipeline handles interlaced sources, shot cuts, missed and jittering hints, paste-back and audio, with memory independent of the length of the video. This report describes the approach, the pipeline that turns it into full-length deliveries, and qualitative results; video results, with input, hint and output side by side, are on the project page.
Figures & tables
| Stage | Problem | Solution |
|---|---|---|
| Input | Sources are interlaced, heavily compressed, variable-frame-rate, or stored with non-square pixels. Small text in low-resolution sources covers only a few latent cells. | Interlacing is detected by a majority vote over an initial window of decoded frames and removed with a field-parity-aware deinterlacer; the sample aspect ratio and frame timing are carried through to the output. Each run is cropped around the union of its hints and resized to a fixed working size chosen for the overlay type, so small text is magnified before encoding. |
| Shots | Windows that cross a cut mix two shots; fades, flashes and dissolves trigger false cuts; detectors leave fragments of a few frames. | A shot-boundary detector proposes cuts with a probability. Confident cuts are kept; the others are kept only if the frames on either side differ in both a gray thumbnail and a color histogram, with a looser test for low-probability cuts. Shots shorter than a minimum length are merged into a neighbor. Runs, windows and temporal hint extension never cross a remaining cut. |
| Hints | Segmenters miss frames, especially during fades; masks jitter; small marks such as logos and title cards are missed by a text segmenter. | The character-level subtitle mask is combined with boxes from a logo and title-card detector. Missed frames are filled in time: a gap between two hinted frames, up to a bounded length, is filled with the union of the hints within a fixed radius on either side, and a hint is extended by a bounded number of frames before an overlay first appears and after it last disappears. Hints need no spatial dilation, since the model reads them as evidence. |
| Planning | A full video does not fit in GPU memory; most frames carry no overlay at all. | A first decoding pass records frame times, thumbnails and per-frame hint statistics. Frames with a hint are grouped into runs, padded with context frames and merged when close enough together; only runs enter the model. Each run is covered by overlapping windows of fixed length. A second pass streams the video, and each frame is written as soon as no later window needs it; memory is bounded by the window length, independent of the length of the video. |
| Paste-back | Replacing whole windows softens untouched areas; residual text on dark frames differs from the background by only a few gray levels. | Only pixels the model changed are pasted back: a pixel is replaced if it differs from the input by more than an adaptive threshold that drops with the mean luma of the frame, so fades and night scenes are not under-replaced. The replaced region is restricted to a fixed neighborhood of the hint at the working resolution and feathered at its border. |
| Output | Dropped or duplicated frames; audio drifts out of sync. | Frames outside runs are written without entering the model, and the output has exactly as many frames as the source. The source audio is muxed back, compensated for the offset between the video stream and the container start, and trimmed to the video duration; truncating to the shorter stream instead can drop the last frame. |
Image Inpainting
Image inpainting reconstructs missing or corrupted image regions using surrounding context, learned visual priors, and optional structural or reference guidance.
See all 98 papers