Enhanced Video Text Editing with Trajectory-Aligned Glyph Rendering

Shulian Zhang; Xiangyu Shu; Wenbo Li; Jian Chen; Yong Guo

cs.CVSep 28, 2026

Enhanced Video Text Editing with Trajectory-Aligned Glyph Rendering

Authors: Shulian Zhang, Xiangyu Shu, Wenbo Li, Jian Chen, Yong Guo

Organizations: South China University of Technology · The Chinese University of Hong Kong · Huawei

Abstract

Video text editing aims to replace or add text in a video while keeping the rest of the video unchanged, which requires the edited text to be correct in every frame and to move coherently with the scene. Despite the remarkable progress of video diffusion models, they struggle to reproduce exact stroke structures and often produce garbled or wrong characters, especially for characters with complex strokes. To address this, we propose a trajectory-aligned glyph rendering reference that provides explicit per-frame glyph guidance following the position and perspective of the text, and a depth-normalized recognizer feature supervision that supervises the generated text on multi-depth features of a frozen text recognizer with per-depth normalized errors, targeting stroke errors overlooked by the diffusion loss. We further build VTEdit, a benchmark of 288 real-scene clips with 440 annotated text trajectories covering text replacement and text addition, which will be publicly released to facilitate future research. Experiments on VTEdit show that our method outperforms image text editing methods, video editing methods, and commercial models in text accuracy and background preservation, achieving a sentence accuracy of 0.9408, and receives the highest preference in a user study.

Figures & tables

Figure 1: Overview of our framework with the trajectory-aligned glyph rendering reference. The target text $y$ is rendered into a glyph image ${\bm{G}}$ , which is warped into the text box of every frame along the trajectory ${\bm{P}}$ . The warped glyphs $\mathcal{W}({\bm{G}};{\bm{P}})$ replace the content of the source video ${\bm{V}}$ inside the mask ${\bm{M}}$ , forming the rendered video ${\bm{R}}$ (Eq. 1 ). The latent ${\mathcal{E}}({\bm{R}})$ and the downsampled mask ${\bm{m}}$ are concatenated with the noisy latent ${\bm{x}}_{t}$ along the channel dimension, and the glyph embedding ${\bm{e}}_{g}$ is concatenated with the prompt tokens ${\bm{e}}_{c}$ along the sequence dimension, as the inputs of the diffusion transformer. The VAE decoder ${\mathcal{D}}$ decodes the generated latent into the edited video $\hat{{\bm{V}}}$ .

Figure 2: Visualization of perceptual features at the three supervised depths of the text recognizer PP-OCRv3. For clarity, we manually erase a stroke (red box) to observe the differences. At the shallow depth, the feature difference is sharp and localized to the missing stroke; at the middle and deep depths, it becomes diffuse and covers a larger part of the character.

Figure 3: Distribution of the activation values at the three supervised depths, measured on the text regions. Left: in the text recognizer, the three depths lie on very different scales, with the shallow depth reaching about 5 while the middle and deep depths stay below 1. Right: in VGG, the three depths lie on comparable scales.

Figure 4: Overview of the depth-normalized recognizer feature supervision. During training, the frozen VAE decoder ${\mathcal{D}}$ decodes the estimated clean latent $\hat{{\bm{x}}}_{0}$ into the predicted video $\hat{{\bm{V}}}$ . The text regions of $\hat{{\bm{V}}}$ and the ground truth ${\bm{V}}$ are cropped by $\mathcal{C}$ and passed through a frozen text recognizer, which extracts the features $\phi_{1}$ , $\phi_{2}$ , and $\phi_{3}$ at the shallow, middle, and deep depths. The error at each depth is normalized by the magnitude of the ground-truth features, and the three normalized errors are averaged into ${\mathcal{L}}_{\text{rec}}$ (Eq. 3 ), which is added to the flow-matching loss.

Figure 5: Examples of text replacement and text addition in VTEdit. Each row shows five frames sampled from a clip, with the annotated text boxes in green, and the text below describes the edit.

Method	Text Accuracy		Temporal	Background Preservation		User Study
Method	Sen.ACC $\uparrow$	NED $\uparrow$	$E_{warp}$ $\downarrow$	PSNR bg $\uparrow$	SSIM bg $\uparrow$	Preference (%) $\uparrow$
FLUX-Text	0.8393	0.9307	4.1516	34.55	0.9678	0.3
VACE	0.1532	0.2991	1.5593	34.67	0.9669	1.2
VideoPainter	0.0395	0.0945	2.0111	26.72	0.9423	0.3
UniVideo	0.0002	0.0710	1.7399	20.04	0.6689	0.2
Kiwi-Edit	0.0000	0.0862	1.5849	20.20	0.7143	0.0

Table 1: Quantitative comparison with state-of-the-art methods. The best and second-best results are highlighted in bold and underlined , respectively. † denotes a closed-source commercial model. Our method achieves the best text accuracy, background preservation, and user preference.

Figure 6: Qualitative comparison of video text editing. The top two examples show text addition, and the bottom example shows text replacement. For each example, the source frame is shown on the left, and the results of all methods are shown on the right. Green boxes in the source frames indicate the editing regions. Our results show legible text with scene-compatible layout and appearance.

Variant	Sen.ACC $\uparrow$	NED $\uparrow$
w/o reference	0.9013	0.9658
Ours	0.9408	0.9862

Table 2: Ablation on the glyph rendering reference. The reference improves text accuracy.

Metric	w/o supervision	VGG	Recognizer
Metric	w/o supervision	VGG	Shallow	Middle	Deep	Multi (Ours)
Sen.ACC $\uparrow$	0.9172	0.9238	0.9245	0.9282	0.9241	0.9408
NED $\uparrow$	0.9685	0.9798	0.9810	0.9815	0.9804	0.9862

Table 3: Ablation on the features used for supervision. VGG uses features from multiple depths with the same normalization. Multi-depth text recognizer features achieve the best text accuracy.

Variant	Sen.ACC $\uparrow$	NED $\uparrow$
w/o normalization	0.9270	0.9819
Normalized (Ours)	0.9408	0.9862

Table 4: Ablation on depth-wise normalization in the recognizer feature supervision. Normalizing each depth by its ground-truth feature magnitude improves text accuracy.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Qualitative comparison of text accuracy. Green boxes mark the editing regions in the source frame, and Red boxes highlight unsuccessful edits and character errors. Our method can produce clear and accurate replacement text.

Figure 8: Qualitative comparison of temporal consistency. Columns show frames 1, 6, 11, and 21. Green boxes mark the source editing regions, and Red boxes highlight failed edits and inconsistent text appearance. Our method maintains legible lettering and a consistent layout across the frames.

Figure 9: Qualitative comparison of background preservation. Green boxes mark the text to be edited. The blue inset enlarges the background outside the editing region. Red boxes highlight unintended changes to the background. Our method more faithfully preserves the source content outside the editing region.

Explore similar work

SteerVTE: Seamless Video Text Editing with Style and Glyph Control
Jun 22, 2026Kai Zeng, Moran Li, Zhengwei Wang +6Video Editing Text-To-Video Models
Peking University · ByteDance China · ByteDance Singapore
ViTeX-Bench: Benchmarking High-Fidelity Video Scene Text Editing
Sep 30, 2026Xinghao Chen, Xiangbo Gao, Jiongze Yu +2Video Editing Text-To-Video Models
Texas A&M University
Beyond Legibility: Benchmarking Visual Text Rendering and In-Place Editing in Unified Video Generation
Sep 29, 2026Ziying Zhang, Litao Li, Junchao Liao +4Text-To-Video Models
Alibaba Group · Shanghai Jiao Tong University · Fudan University

Jun 22, 2026, cs.CVJ/K move · Enter open · S save

SteerVTE: Seamless Video Text Editing with Style and Glyph Control

Kai Zeng, Moran Li, Zhengwei Wang, Yingchen Yu+5

Peking University · ByteDance China · ByteDance Singapore

Video Editing Text-To-Video Models Diffusion Editing Typography Generation Encoders Steering+1

Open paper AskPDF arXiv

Visual text editing aims to precisely modify text in images and videos while preserving stylistic consistency and visual realism. Despite significant advances in the image domain, video text editing remains largely unexplored: it is a localized task demanding stroke-level precision within small text regions, which compounds the challenges of cross-frame accuracy, temporal coherence, and stylistic fidelity. We introduce SteerVTE, a unified framework that \underline{\textbf{steer}}s a frozen video diffusion model to perform precise \underline{\textbf{V}}ideo \underline{\textbf{T}}ext \underline{\textbf{E}}diting through style and glyph control. Built on a frozen diffusion transformer, SteerVTE attaches a lightweight text context adapter with two complementary modules: a style encoder capturing the original text's visual attributes, and dual-granularity glyph encoders encoding the target text at both the line and character levels. To overcome the inherently weak text rendering priors of video foundation models, we further propose a glyph-aware spatial-focal loss and a three-stage progressive training curriculum that scales from image to video data. To support large-scale training, we also develop an automatic synthesis pipeline and construct SteerVTE-1M, a dataset of one million triplets spanning diverse scenes, fonts, and stylistic effects. Extensive experiments demonstrate that SteerVTE substantially outperforms existing video editing baselines across text accuracy, style consistency, and temporal coherence.

Abstract

Explore similar work

SteerVTE: Seamless Video Text Editing with Style and Glyph Control

ViTeX-Bench: Benchmarking High-Fidelity Video Scene Text Editing

Beyond Legibility: Benchmarking Visual Text Rendering and In-Place Editing in Unified Video Generation