TextVBench: Evaluating Text Rendering and Preservation in Video Generation

Ruiqiang Zhang; Hengyi Wang; Chang Liu; Guanjie Wang; Zehua Ma; Weiming Zhang

cs.CVOct 9, 2026

TextVBench: Evaluating Text Rendering and Preservation in Video Generation

Authors: Ruiqiang Zhang, Hengyi Wang, Chang Liu, Guanjie Wang, Zehua Ma, Weiming Zhang

Organizations: Anhui Province Key Laboratory of Digital Security, University of Science and Technology of China

Abstract

Modern video generators can synthesize convincing scenes while rendering a few characters incorrectly, omitting requested text, or changing it across frames. Measuring these failures is harder than applying OCR independently to each frame: when a scene contains multiple text targets and multiple noisy OCR regions, the evaluator must first determine which observation belongs to which intended text. We introduce TextVBench, a benchmark and evaluation framework that formulates visual-text evaluation as a multi-target, multi-instance spatiotemporal correspondence problem. TextVBench contains two complementary tracks: TextVBench-T2V measures the creation and maintenance of prompt-specified text, while TextVBench-Ref2V measures preservation of text from reference images. Both tracks use target-conditioned candidate selection rather than unconstrained VLM transcription, support one target mapping to multiple OCR regions, and separate frame-level spatial fidelity from cross-frame temporal fidelity. A length-weighted harmonic score summarizes these factors, while extra-text diagnostics expose failure modes hidden by a single OCR score. The benchmark is organized around controlled text, layout, carrier, and motion attributes so that aggregate scores can be traced back to specific capability limits. Across the evaluated model runs, MiniMax-H3 reaches a T2V TextVScore of 68.32, while the strongest Ref2V score is 56.51; attribute slices reveal a pronounced drop as the number of simultaneous reference text regions increases.

Figures & tables

Figure 1: Why visual-text evaluation is difficult. The aligned cases progress from single-target recognition to multi-target correspondence, fragmented OCR observations, and identity-aware tracking over time. A reliable evaluator must associate OCR candidates with semantic targets before scoring spatial and temporal fidelity.

Figure 2: TextVBench overview and shared evaluator. A target is intended text from a T2V prompt or a Ref2V reference region; numbered boxes are OCR observations. In the T2V example, one target spans Box 1 and Box 2, whose text is joined without rewriting. Spatial fidelity compares observations with the target; temporal fidelity compares successive observations of the same target. Omitting “EVERY DAY” removes 8 of 24 normalized characters, giving the illustrated frame and transition similarities. Multi-reference Ref2V skips inactive views and compares text on reappearance. Length-weighted target aggregation (followed by equal reference averaging for Ref2V) yields $S$ and $T$ for TextVScore; extra text is reported as a separate diagnostic. Scenes, trajectories, and numerical examples are illustrative.

Figure 3: TextVBench-T2V construction pipeline. Seven stages turn source video prompts into controlled bilingual specifications: eligibility filtering, scene and carrier analysis, coverage and difficulty sampling, carrier–slot planning, text instantiation, prompt composition, and quality control. The example binds three text slots to a storefront sign and a window poster. Scene images are synthetic illustrations.

Figure 4: Structured Ref2V construction. Preservation difficulty is controlled by combining text complexity, spatial structure, carrier, view/motion, and visibility. Generated scenes illustrate three attribute combinations.

Figure 5: Model-wise scaling diagnostics. Each heatmap cell is a video-level TextVScore mean; the highlighted pooled row aggregates the detailed attribute-slice export. The final column reports the last-band minus first-band change, making robustness differences explicit. T2V difficulty is fixed by construction, while Ref2V load is the number of OCR target slots in the reference. The bands are descriptive slices, not matched single-factor experiments.

Diagnostic	T2V	Ref2V
Spatial failure ( $<0.10$ )	49.8%	14.2%
Temporal failure ( $<0.10$ )	36.5%	16.4%
TextVScore $=0$	29.4%	4.9%
TextVScore $<50$	76.4%	50.4%
ExtraTextRatio $>0.20$	32.8%	19.8%
Temporal $>$ Spatial	58.4%	26.2%

Table 2: Pooled failure and complementarity diagnostics. The first two rows use the common low-fidelity threshold shown in parentheses; ExtraTextRatio captures unassigned readable text and is not folded into TextVScore.

Figure 6: Correct and stable T2V text. Both targets remain matched across all sampled observations (Spatial/Temporal $1.000$ ; TextVScore $100.0$ ).

Figure 7: Stable OCR does not imply correct target text. The carrier is present throughout, but the requested “Bakery Corner” is repeatedly read as “Corner”.

Figure 8: Intermittent correspondence. “Field Watch” is matched in the first displayed observation and then loses correspondence while the carrier remains visible.

Video Generation Evaluation

Video generation evaluation measures visual quality, prompt adherence, temporal consistency, physical plausibility, and task-specific behavior in synthesized videos.

See all 182 papers

VTR-Bench: A Systematic Benchmark for Evaluating Visual Text Rendering in Video Generation
Oct 1, 2026Yu Huang, Jungang Li, Zhiyuan Wang +8Agentic Video Generation Video Generation
Department of Data Science, City University of Hong Kong · The Hong Kong University of Science and Technology (Guangzhou) · Westlake University +3
Beyond Legibility: Benchmarking Visual Text Rendering and In-Place Editing in Unified Video Generation
Sep 29, 2026Ziying Zhang, Litao Li, Junchao Liao +4Video Editing Benchmarks Video Editing
Alibaba Group · Shanghai Jiao Tong University · Fudan University
ViTeX-Bench: Benchmarking High-Fidelity Video Scene Text Editing
Sep 30, 2026Xinghao Chen, Xiangbo Gao, Jiongze Yu +2Video Editing Benchmarks Video Editing
Texas A&M University

Oct 1, 2026J/K move · Enter open · S save

VTR-Bench: A Systematic Benchmark for Evaluating Visual Text Rendering in Video Generation

Yu Huang, Jungang Li, Zhiyuan Wang, Yonghua Hei+7

Department of Data Science, City University of Hong Kong · The Hong Kong University of Science and Technology (Guangzhou) · Westlake University+3

Agentic Video Generation Video Generation Video Generation Evaluation

Open paper AskPDF arXiv

Recent video generation models can produce highly realistic videos from natural language instructions, with visual quality approaching cinematic standards. Existing evaluation benchmarks, however, predominantly assess visual quality, aesthetic appeal and physical plausibility, while paying limited attention to text, an essential medium for conveying information in everyday scenes. A generated video may appear visually compelling and feature lifelike subjects, yet still render the text within the scene incorrectly. To address this overlooked dimension, we introduce VTR-Bench, a systematic benchmark for evaluating the Visual Text Rendering capabilities of video generation models. VTR-Bench situates text within concrete application scenarios, such as advertisements and scientific videos, with 300 carefully constructed prompts spanning five scenario categories. We develop an automated evaluation pipeline with human alignments that separately assesses text fidelity through carrier-specific transcription and scene and motion requirements through a prompt-specific chain of query. Beyond evaluation, we introduce a Keyframe-Guided Agentic Framework in which a Director agent coordinates image and video generation with visual evaluation, guiding iterative refinement and candidate selection through visual feedback. Experiments on 11 state-of-the-art models reveal widespread difficulties in accurately rendering scene text, with the best-performing model recording an overall word error rate (WER) of 0.250. We further analyze text rendering failures to characterize the challenges faced by current video generation models. These findings highlight visual text rendering as a key challenge for video generation and demonstrate a practical path toward improvement. Code is available at https://github.com/hardenyu21/VTR-Bench.

Abstract

Video Generation Evaluation

Explore similar work

VTR-Bench: A Systematic Benchmark for Evaluating Visual Text Rendering in Video Generation

Beyond Legibility: Benchmarking Visual Text Rendering and In-Place Editing in Unified Video Generation

ViTeX-Bench: Benchmarking High-Fidelity Video Scene Text Editing