Enhanced Video Text Editing with Trajectory-Aligned Glyph Rendering

Shulian Zhang; Xiangyu Shu; Wenbo Li; Jian Chen; Yong Guo

cs.CVSep 28, 2026

Enhanced Video Text Editing with Trajectory-Aligned Glyph Rendering

Authors: Shulian Zhang, Xiangyu Shu, Wenbo Li, Jian Chen, Yong Guo

Organizations: South China University of Technology · The Chinese University of Hong Kong · Huawei

Abstract

Video text editing aims to replace or add text in a video while keeping the rest of the video unchanged, which requires the edited text to be correct in every frame and to move coherently with the scene. Despite the remarkable progress of video diffusion models, they struggle to reproduce exact stroke structures and often produce garbled or wrong characters, especially for characters with complex strokes. To address this, we propose a trajectory-aligned glyph rendering reference that provides explicit per-frame glyph guidance following the position and perspective of the text, and a depth-normalized recognizer feature supervision that supervises the generated text on multi-depth features of a frozen text recognizer with per-depth normalized errors, targeting stroke errors overlooked by the diffusion loss. We further build VTEdit, a benchmark of 288 real-scene clips with 440 annotated text trajectories covering text replacement and text addition, which will be publicly released to facilitate future research. Experiments on VTEdit show that our method outperforms image text editing methods, video editing methods, and commercial models in text accuracy and background preservation, achieving a sentence accuracy of 0.9408, and receives the highest preference in a user study.

Figures & tables

Figure 1: Overview of our framework with the trajectory-aligned glyph rendering reference. The target text $y$ is rendered into a glyph image ${\bm{G}}$ , which is warped into the text box of every frame along the trajectory ${\bm{P}}$ . The warped glyphs $\mathcal{W}({\bm{G}};{\bm{P}})$ replace the content of the source video ${\bm{V}}$ inside the mask ${\bm{M}}$ , forming the rendered video ${\bm{R}}$ (Eq. 1 ). The latent ${\mathcal{E}}({\bm{R}})$ and the downsampled mask ${\bm{m}}$ are concatenated with the noisy latent ${\bm{x}}_{t}$ along the channel dimension, and the glyph embedding ${\bm{e}}_{g}$ is concatenated with the prompt tokens ${\bm{e}}_{c}$ along the sequence dimension, as the inputs of the diffusion transformer. The VAE decoder ${\mathcal{D}}$ decodes the generated latent into the edited video $\hat{{\bm{V}}}$ .

Figure 2: Visualization of perceptual features at the three supervised depths of the text recognizer PP-OCRv3. For clarity, we manually erase a stroke (red box) to observe the differences. At the shallow depth, the feature difference is sharp and localized to the missing stroke; at the middle and deep depths, it becomes diffuse and covers a larger part of the character.

Figure 3: Distribution of the activation values at the three supervised depths, measured on the text regions. Left: in the text recognizer, the three depths lie on very different scales, with the shallow depth reaching about 5 while the middle and deep depths stay below 1. Right: in VGG, the three depths lie on comparable scales.

Figure 4: Overview of the depth-normalized recognizer feature supervision. During training, the frozen VAE decoder ${\mathcal{D}}$ decodes the estimated clean latent $\hat{{\bm{x}}}_{0}$ into the predicted video $\hat{{\bm{V}}}$ . The text regions of $\hat{{\bm{V}}}$ and the ground truth ${\bm{V}}$ are cropped by $\mathcal{C}$ and passed through a frozen text recognizer, which extracts the features $\phi_{1}$ , $\phi_{2}$ , and $\phi_{3}$ at the shallow, middle, and deep depths. The error at each depth is normalized by the magnitude of the ground-truth features, and the three normalized errors are averaged into ${\mathcal{L}}_{\text{rec}}$ (Eq. 3 ), which is added to the flow-matching loss.

Figure 5: Examples of text replacement and text addition in VTEdit. Each row shows five frames sampled from a clip, with the annotated text boxes in green, and the text below describes the edit.

Method	Text Accuracy		Temporal	Background Preservation		User Study
Method	Sen.ACC $\uparrow$	NED $\uparrow$	$E_{warp}$ $\downarrow$	PSNR bg $\uparrow$	SSIM bg $\uparrow$	Preference (%) $\uparrow$
FLUX-Text	0.8393	0.9307	4.1516	34.55	0.9678	0.3
VACE	0.1532	0.2991	1.5593	34.67	0.9669	1.2
VideoPainter	0.0395	0.0945	2.0111	26.72	0.9423	0.3
UniVideo	0.0002	0.0710	1.7399	20.04	0.6689	0.2
Kiwi-Edit	0.0000	0.0862	1.5849	20.20	0.7143	0.0

Table 1: Quantitative comparison with state-of-the-art methods. The best and second-best results are highlighted in bold and underlined , respectively. † denotes a closed-source commercial model. Our method achieves the best text accuracy, background preservation, and user preference.

Figure 6: Qualitative comparison of video text editing. The top two examples show text addition, and the bottom example shows text replacement. For each example, the source frame is shown on the left, and the results of all methods are shown on the right. Green boxes in the source frames indicate the editing regions. Our results show legible text with scene-compatible layout and appearance.

Variant	Sen.ACC $\uparrow$	NED $\uparrow$
w/o reference	0.9013	0.9658
Ours	0.9408	0.9862

Table 2: Ablation on the glyph rendering reference. The reference improves text accuracy.

Metric	w/o supervision	VGG	Recognizer
Metric	w/o supervision	VGG	Shallow	Middle	Deep	Multi (Ours)
Sen.ACC $\uparrow$	0.9172	0.9238	0.9245	0.9282	0.9241	0.9408
NED $\uparrow$	0.9685	0.9798	0.9810	0.9815	0.9804	0.9862

Table 3: Ablation on the features used for supervision. VGG uses features from multiple depths with the same normalization. Multi-depth text recognizer features achieve the best text accuracy.

Variant	Sen.ACC $\uparrow$	NED $\uparrow$
w/o normalization	0.9270	0.9819
Normalized (Ours)	0.9408	0.9862

Table 4: Ablation on depth-wise normalization in the recognizer feature supervision. Normalizing each depth by its ground-truth feature magnitude improves text accuracy.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Qualitative comparison of text accuracy. Green boxes mark the editing regions in the source frame, and Red boxes highlight unsuccessful edits and character errors. Our method can produce clear and accurate replacement text.

Figure 8: Qualitative comparison of temporal consistency. Columns show frames 1, 6, 11, and 21. Green boxes mark the source editing regions, and Red boxes highlight failed edits and inconsistent text appearance. Our method maintains legible lettering and a consistent layout across the frames.

Figure 9: Qualitative comparison of background preservation. Green boxes mark the text to be edited. The blue inset enlarges the background outside the editing region. Red boxes highlight unintended changes to the background. Our method more faithfully preserves the source content outside the editing region.

Explore similar work

Jun 22, 2026cs.CV

SteerVTE: Seamless Video Text Editing with Style and Glyph Control

Visual text editing aims to precisely modify text in images and videos while preserving stylistic consistency and visual realism. Despite significant advances in the image domain, video text editing remains largely unexplored: it is a localized task demanding stroke-level precision within small text regions, which compounds the challenges of cross-frame accuracy, temporal coherence, and stylistic fidelity. We introduce SteerVTE, a unified framework that \underline{\textbf{steer}}s a frozen video diffusion model to perform precise \underline{\textbf{V}}ideo \underline{\textbf{T}}ext \underline{\textbf{E}}diting through style and glyph control. Built on a frozen diffusion transformer, SteerVTE attaches a lightweight text context adapter with two complementary modules: a style encoder capturing the original text's visual attributes, and dual-granularity glyph encoders encoding the target text at both the line and character levels. To overcome the inherently weak text rendering priors of video foundation models, we further propose a glyph-aware spatial-focal loss and a three-stage progressive training curriculum that scales from image to video data. To support large-scale training, we also develop an automatic synthesis pipeline and construct SteerVTE-1M, a dataset of one million triplets spanning diverse scenes, fonts, and stylistic effects. Extensive experiments demonstrate that SteerVTE substantially outperforms existing video editing baselines across text accuracy, style consistency, and temporal coherence.

Kai Zeng, Moran Li, Zhengwei Wang +6

Peking University · ByteDance China · ByteDance Singapore

Video Editing Text-To-Video Models Diffusion Editing Typography Generation Encoders Steering+1

Ask PDF

Sep 30, 2026cs.CV

ViTeX-Bench: Benchmarking High-Fidelity Video Scene Text Editing

Recent video generation is increasingly realistic and controllable, yet video editing remains less developed, particularly for precise local edits that must preserve the original scene dynamics. Video scene text editing replaces text on scene surfaces, such as storefront signs, whiteboards, and product labels, while preserving the surrounding content, motion, and camera dynamics. Although scene text editing is well studied for images, video scene text editing that achieves high visual quality, temporal consistency, and edit locality remains underexplored. Existing resources offer limited paired real-video data, and general video-editing metrics do not directly measure whether the requested text remains correct over time. We introduce ViTeX-Bench, a benchmark suite comprising ViTeX-Dataset and a three-axis evaluation protocol. The dataset contains 387 real-world 720p videos with text-region masks and editing instructions: 230 provide reviewed, pipeline-generated paired edits for training, and 157 form a frozen evaluation split. The protocol evaluates text correctness, visual and temporal quality, and edit locality through 13 metrics, with one primary metric per axis and a Pareto comparison of their trade-offs. OCR calibration, human evaluation, and annotation-sensitivity analyses support the interpretation of these scores. Across eight baselines from four editing families, accurate text, temporal stability, and scene preservation remain difficult to achieve together. We also release ViTeX-Edit-14B, an open-source reference editor fine-tuned on the paired training split with motion-aligned glyph-video conditioning. It achieves CharAcc 0.688, the highest mean among the evaluated video-native editors, and the lowest comparable text-crop Warp among raw editor outputs. ViTeX-Bench provides a reproducible foundation for studying these trade-offs in video scene text editing.

Xinghao Chen, Xiangbo Gao, Jiongze Yu +2

Texas A&M University

Video Editing Text-To-Video Models Longvideobench

Ask PDF

Sep 29, 2026cs.CV

Beyond Legibility: Benchmarking Visual Text Rendering and In-Place Editing in Unified Video Generation

A video can exhibit convincing motion and photorealism yet fail immediately when visual text collapses. Unlike generic scene content, visual text is unforgiving in video generation: minor stroke corruption, temporal instability, or editing errors instantly break legibility and realism. Existing benchmarks overlook this challenge by treating text as incidental or using static OCR metrics that ignore temporal dynamics. We introduce VidScribe, a unified diagnostic benchmark spanning four generation regimes: writing from language (T2V), transferring text identity from a reference (R2V), sustaining text under dynamics (I2V), and localized text editing (V2V). VidScribe contains 803 human-verified samples across a 12-axis conditionally orthogonal factor space covering Intrinsic Text Properties, Physical Imaging Conditions, and Temporal Behavior. For reliable evaluation, we build a track-grounded, gated suite with 11 shared metrics and 2 task-specific probes under strict measurability conditions. Benchmarking 11 commercial and open-source systems shows that video text capability is non-monolithic, with content recognition decoupled from stroke-level glyph correctness. Performance is highly task-asymmetric: I2V sustains text most reliably, whereas V2V editing is the primary bottleneck. Counter-intuitively, degradation concentrates on a small subset of text-centric structural and temporal factors rather than adverse imaging conditions. Further probes show that visual references improve glyph and typographic fidelity rather than content accuracy, while localized editing fails to isolate target text without corrupting undeclared source text. Beyond evaluation, VidScribe also provides an actionable training signal, where benchmark-aligned preference optimization measurably improves visual text generation. https://huggingface.co/datasets/Vicky0720/VidScribe.

Ziying Zhang, Litao Li, Junchao Liao +4

Alibaba Group · Shanghai Jiao Tong University · Fudan University

Text-To-Video Models

Ask PDF