Beyond Legibility: Benchmarking Visual Text Rendering and In-Place Editing in Unified Video Generation

Ziying Zhang; Litao Li; Junchao Liao; Tianyi Zeng; Siyu Zhu; Long Qin; Zhenghao Zhang

cs.CVSep 29, 2026

Beyond Legibility: Benchmarking Visual Text Rendering and In-Place Editing in Unified Video Generation

Authors: Ziying Zhang, Litao Li, Junchao Liao, Tianyi Zeng, Siyu Zhu, Long Qin, Zhenghao Zhang

Organizations: Alibaba Group · Shanghai Jiao Tong University · Fudan University

Abstract

A video can exhibit convincing motion and photorealism yet fail immediately when visual text collapses. Unlike generic scene content, visual text is unforgiving in video generation: minor stroke corruption, temporal instability, or editing errors instantly break legibility and realism. Existing benchmarks overlook this challenge by treating text as incidental or using static OCR metrics that ignore temporal dynamics. We introduce VidScribe, a unified diagnostic benchmark spanning four generation regimes: writing from language (T2V), transferring text identity from a reference (R2V), sustaining text under dynamics (I2V), and localized text editing (V2V). VidScribe contains 803 human-verified samples across a 12-axis conditionally orthogonal factor space covering Intrinsic Text Properties, Physical Imaging Conditions, and Temporal Behavior. For reliable evaluation, we build a track-grounded, gated suite with 11 shared metrics and 2 task-specific probes under strict measurability conditions. Benchmarking 11 commercial and open-source systems shows that video text capability is non-monolithic, with content recognition decoupled from stroke-level glyph correctness. Performance is highly task-asymmetric: I2V sustains text most reliably, whereas V2V editing is the primary bottleneck. Counter-intuitively, degradation concentrates on a small subset of text-centric structural and temporal factors rather than adverse imaging conditions. Further probes show that visual references improve glyph and typographic fidelity rather than content accuracy, while localized editing fails to isolate target text without corrupting undeclared source text. Beyond evaluation, VidScribe also provides an actionable training signal, where benchmark-aligned preference optimization measurably improves visual text generation. https://huggingface.co/datasets/Vicky0720/VidScribe.

Figures & tables

Figure 1: Motivating failure cases and headline findings of VidScribe. (a) Representative failure cases across T2V, R2V, I2V, and V2V. (b) Model-wise comparison across shared metrics, showing decoupling between Content Accuracy and Glyph Correctness. (c) Task-wise overall performance aggregated over all 7 commercial unified video generators, highlighting strong task asymmetry. (d) Factor-wise damage ranking aggregated over all 7 commercial unified video generators.

Figure 2: Data distribution of VidScribe across (a) generation regimes and V2V edit operations, (b) content domains, (c) visual styles and text materials, (d) the 12 factor axes, (e) spatial layout, and (f) linguistic and semantic properties.

Figure 3: Overview of the VidScribe evaluation framework. Video frames are grounded through OCR and text tracking, then evaluated through 3 pathways: Deterministic Expert Scoring, Hybrid Scoring, and Rubric-Grounded VLM verification, with gating applied to enforce measurability.

		Text Fidelity		Temporal Stability		Instruction Compliance						Non-Text Preservation
Video generators	Overall $\uparrow$	A1 $\uparrow$	A2 $\uparrow$	B1 $\uparrow$	B2 $\uparrow$	C1 $\uparrow$	C2 $\uparrow$	C3 $\uparrow$	C4 $\uparrow$	C5 $\uparrow$	C6 $\uparrow$	D1 $\uparrow$
MiniMax H3	79.2	82.1	57.0	74.8	83.5	83.5	74.3	75.3	75.2	74.1	83.4	90.6
Wan 3.0	78.5	82.7	45.4	76.7	86.0	84.4	75.6	79.7	77.7	76.8	83.9	88.8
Seedance 2.5	78.1	77.6	58.2	72.5	83.4	84.5	72.9	70.9	75.0	71.1	80.9	90.7
Gemini Omni 1.1 Flash	77.8	79.0	49.9	73.8	84.4	82.6	72.9	76.6	73.1	77.1	83.7	90.1
HappyHorse	75.0	72.5	48.7	66.7	82.5	84.1	72.7	70.3	73.5	71.0	82.2	89.1

Table 1: Results of 7 commercial unified video generators on VidScribe. “Overall” denotes the group-balanced mean over the 11 shared metrics.

Table 2: Task-wise performance and V2V-specific editing diagnostics across unified video generators.

Factor Axis	Metric	Regular	Hard	$\Delta$	Regular	Hard	$\Delta$
	Assigned Metric				Overall Score
F12: Content Evolution	B1: Content Stability	70.8	49.7	$-21.1$	77.3	69.4	$-7.9$
F11: Foreground Occlusion	A2: Glyph Correctness	52.5	37.4	$-15.1$	76.8	71.8	$-5.0$
F2: Layout Hierarchy	C6: Hierarchy Fidelity	87.7	50.9	$-36.8$	76.7	72.6	$-4.1$
F10: Motion Pattern	C5: Placement Accuracy	75.0	67.9	$-7.1$	77.0	73.9	$-3.1$
F1: Text Amount	A1: Content Accuracy	74.7	66.0	$-8.7$	76.7	73.6	$-3.1$

Table 3: Factor sensitivity under Regular and Hard settings. Factors are ordered by task-balanced Overall Cliff’s

\delta

( Hard vs. Regular ), where more negative values indicate larger degradation under Hard . Regular/Hard definitions are given in Appendix A.1 .

Figure 5: Further diagnostic examples. (a) Matched R2V–T2V comparisons. (b) A representative V2V failure case with over-erasure or residual source text.

Evaluator Backbone	Correlation	A1	A2	B1	B2	C1	C2	C3	C4	C5	C6	D1
Qwen3.7-Plus	Spearman ( $\rho$ ) $\uparrow$	0.87	0.84	0.82	0.81	0.78	0.84	0.91	0.72	0.88	0.79	0.96
Qwen3.7-Plus	Pearson ( $r$ ) $\uparrow$	0.90	0.90	0.85	0.83	0.77	0.86	0.86	0.72	0.89	0.76	0.94
Gemini-3.8-Flash	Spearman ( $\rho$ ) $\uparrow$	0.82	0.80	0.83	0.79	0.76	0.84	0.87	0.70	0.82	0.79	0.95
Gemini-3.8-Flash	Pearson ( $r$ ) $\uparrow$	0.85	0.84	0.85	0.80	0.76	0.85	0.86	0.71	0.85	0.77	0.94

Table 4: Correlation with blinded human expert consensus on 950 sampled videos (

\rho

and

r\in[-1,1]

, all

p<0.001

). Qwen3.7-Plus serves as our default evaluator for reporting results.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Axis	Factor	Baseline	Non-baseline values	Definition
Intrinsic text properties ( F1–F6 )
F1	Text Amount	minimal	moderate, heavy , multi_instance	Character load and instance count in one frame: 1–8, 9–30, or 31+ characters, or 5+ separate blocks.
F2	Layout Hierarchy	single_level	two_level, multi_level_plus , table_like	Number of typographic levels, from single level to title–body, multi-level, or tabular.
F3	String Type	natural_language	non_semantic	Whether the string follows natural language or is a non-semantic code such as a serial number or plate.
F4	Typographic Style	sans	serif, script , monospace	Coarse typeface category.
F5	Stroke Weight	regular	light , bold	Ordered stroke-weight class.

Appendix

Table 5: VidScribe’s 12 factor axes and value definitions. Bold marks baseline values; underlining marks Hard values, with all remaining values treated as Regular .

Panel of Figure 6 — Figure 6: Expert annotation interfaces. Sensitive source text and personal information are masked in the visualization.

Affected pair	Gate	Reason
F1 $\times$ F12	typing excludes heavy; handwriting_stroke requires minimal	The progression must remain visually resolvable within a nominal 5-second clip.
F1 $\times$ F2	minimal permits only single_level or two_level	Minimal text cannot support richer hierarchy patterns.
F5 $\times$ F6	tiny excludes light in specification-driven streams	At 720p, extremely thin strokes become visually unreliable after encoding.
F10 $\times$ F11	camera_motion_screen_locked permits none or off_screen_crop, but not scene-object occlusion	Scene-object occlusion is incompatible with screen-locked anchoring.
F4 $\times$ F5	In R2V, light is retained only for font families backed by an actual light face	Unsupported font–weight combinations are excluded from reference construction.

Appendix

Table 6: Principal feasibility constraints on the joint benchmark support.

Figure 7: Scores of 7 commercial unified video generators across the 11 shared metrics, evaluated with (a) Qwen3.7-Plus and (b) Gemini-3.8-Flash.

System	Human Overall	Human Rank	Overall Rank
Wan 3.0	80.4	1	2
MiniMax H3	80.1	2	1
Seedance 2.5	79.5	3	3
Gemini Omni 1.1 Flash	79.3	4	4
HappyHorse	77.6	5	5
PixVerse V6	74.3	6	6

Appendix

Table 7: Human Overall scores and system rankings on the same 560 generated videos. Overall Rank is determined by the group-balanced VidScribe Overall score.

Figure 8: Screenshot of the expert questionnaire. Left: the generated video. Right: the survey questions. A $3\times 3$ grid is overlaid on the video to assist position judgments.

Panel of Figure 9 — Figure 9: Comparison of text style and stroke weight classification performance on held-out rendered text. Top-1 denotes overall accuracy, and worst-cls. denotes the lowest per-class accuracy. The VLM baseline uses Qwen3.7-Plus ( Qwen Team, 2026 ) .

		Text Fidelity		Temporal Stability		Instruction Compliance						Non-Text Preservation
Input videos	Overall $\uparrow$	A1 $\uparrow$	A2 $\uparrow$	B1 $\uparrow$	B2 $\uparrow$	C1 $\uparrow$	C2 $\uparrow$	C3 $\uparrow$	C4 $\uparrow$	C5 $\uparrow$	C6 $\uparrow$	D1 $\uparrow$
Ideal synthetic	97.9	100	100	100	99.7	99.6	91.6	90.5	99.5	95.2	95.7	96.2

Appendix

Table 8: Metric sanity check on 500 ideal synthetic videos.

Rank	Factor	$\delta_{\mathrm{TB}}$	$p_{\mathrm{FDR}}(\beta)$
1	F12 Content Evolution	$-0.347$	$<0.001$
2	F11 Foreground Occlusion	$-0.245$	$<0.001$
3	F2 Layout Hierarchy	$-0.208$	$0.010$
4	F10 Motion Pattern	$-0.193$	$<0.001$
5	F1 Text Amount	$-0.154$	$<0.001$
6	F6 Text Scale	$-0.139$	$0.095^{\ddagger}$

Appendix

Table 9: Factor sensitivity on 803 VidScribe samples, ranked by task-balanced Cliff’s

\delta_{\mathrm{TB}}

for Overall ( Hard vs. Regular ; more negative indicates larger degradation).

p_{\mathrm{FDR}}(\beta)

reports BH-adjusted

p

-values for task- and co-factor-adjusted OLS effects; ‡ marks

p_{\mathrm{FDR}}(\beta)\geq 0.05

Figure 10: Pooled Cliff’s $\delta$ ( Hard vs. Regular ) across VidScribe’s 12 factors and 11 shared metrics.

		Text Fidelity		Temporal Stability		Instruction Compliance						Non-Text Preservation
Model	Overall $\uparrow$	A1 $\uparrow$	A2 $\uparrow$	B1 $\uparrow$	B2 $\uparrow$	C1 $\uparrow$	C2 $\uparrow$	C3 $\uparrow$	C4 $\uparrow$	C5 $\uparrow$	C6 $\uparrow$	D1 $\uparrow$
Wan2.2-TI2V-5B	45.92	16.83	21.90	14.50	51.23	51.70	50.70	27.57	44.00	41.19	54.00	86.60
+ benchmark-aligned DPO	48.42	21.00	26.90	18.37	51.31	49.70	50.85	25.90	48.00	45.81	63.33	87.64
$\Delta$	$+2.50$	$+4.17$	$+5.00$	$+3.87$	$+0.08$	$-2.00$	$+0.15$	$-1.67$	$+4.00$	$+4.62$	$+9.33$	$+1.04$

Appendix

Table 10: Wan2.2-TI2V-5B and its benchmark-aligned DPO variant on the VidScribe T2V split. Values and

\Delta

are percentages and changes.

Dimension	DPO wins	DPO Win Rate
A: Text Fidelity	172	68.8%
B: Temporal Stability	137	54.8%
C: Instruction Compliance	135	54.0%
D: Non-Text Preservation	129	51.6%

Appendix

Table 11: Blinded user-study results for Wan2.2-TI2V-5B and its benchmark-aligned DPO variant on 250 prompt pairs. Green bars show the DPO win rate in each dimension.

Figure 11: Qualitative comparison of glyph rendering by Wan2.2-TI2V-5B (top) and its benchmark-aligned DPO variant (bottom).

		Text Fidelity		Temporal Stability		Instruction Compliance						Non-Text Preservation
Video generators	Overall $\uparrow$	A1 $\uparrow$	A2 $\uparrow$	B1 $\uparrow$	B2 $\uparrow$	C1 $\uparrow$	C2 $\uparrow$	C3 $\uparrow$	C4 $\uparrow$	C5 $\uparrow$	C6 $\uparrow$	D1 $\uparrow$
Gemini Omni 1.1 Flash	80.9	86.0	49.7	83.9	91.7	85.4	70.1	74.5	59.3	76.3	89.7	92.3
Seedance 2.5	80.2	81.5	63.4	79.4	89.7	83.2	68.9	62.0	65.7	65.3	84.1	92.4
MiniMax H3	80.2	85.0	56.6	82.1	88.4	79.7	70.1	68.8	63.1	69.7	87.7	91.5
Wan 3.0	79.6	89.6	38.7	83.1	88.7	81.3	72.6	80.6	68.3	76.2	90.5	90.2
HappyHorse	76.6	73.3	52.3	71.1	90.3	87.6	66.2	56.8	59.7	70.3	86.5	91.6

Appendix

Table 12: T2V results on VidScribe for commercial unified and task-specific open-source video generators.

		Text Fidelity		Temporal Stability		Instruction Compliance						Non-Text Preservation
Video generators	Overall $\uparrow$	A1 $\uparrow$	A2 $\uparrow$	B1 $\uparrow$	B2 $\uparrow$	C1 $\uparrow$	C2 $\uparrow$	C3 $\uparrow$	C4 $\uparrow$	C5 $\uparrow$	C6 $\uparrow$	D1 $\uparrow$
MiniMax H3	81.9	83.9	68.5	77.6	89.0	72.2	84.1	86.6	77.7	68.6	74.6	90.9
Seedance 2.5	80.9	82.9	65.5	77.7	88.8	80.7	81.7	79.9	71.7	63.7	72.4	91.3
Wan 3.0	80.4	86.0	54.7	81.8	89.0	73.6	84.8	85.3	79.1	72.8	73.6	87.7
HappyHorse	78.3	75.6	63.7	73.3	90.9	67.9	83.4	82.4	72.7	55.3	70.3	89.7
Gemini Omni 1.1 Flash	76.3	73.3	52.4	67.9	86.0	67.6	80.5	78.8	70.3	73.5	75.3	91.3

Appendix

Table 13: R2V results on VidScribe for commercial unified video generators.

		Text Fidelity		Temporal Stability		Instruction Compliance						Non-Text Preservation
Video generators	Overall $\uparrow$	A1 $\uparrow$	A2 $\uparrow$	B1 $\uparrow$	B2 $\uparrow$	C1 $\uparrow$	C2 $\uparrow$	C3 $\uparrow$	C4 $\uparrow$	C5 $\uparrow$	C6 $\uparrow$	D1 $\uparrow$
MiniMax H3	84.2	90.4	62.5	85.2	90.1	92.6	76.0	77.6	83.6	78.1	80.9	91.3
Kling3.0-Omni	83.5	88.0	57.6	84.8	92.1	92.2	78.1	79.5	81.5	78.5	81.2	91.0
Seedance 2.5	83.5	90.2	62.7	84.4	88.5	92.0	76.0	74.8	83.3	76.6	79.9	90.6
Wan 3.0	81.8	87.4	54.6	82.5	88.7	93.4	77.4	78.6	84.9	78.6	80.3	88.4
Gemini Omni 1.1 Flash	81.6	87.5	56.1	81.4	86.2	91.8	74.3	80.3	84.4	77.5	80.6	89.3

Appendix

Table 14: I2V results on VidScribe for commercial unified video generators.

		Text Fidelity		Temporal Stability		Instruction Compliance						Non-Text Preservation
Video generators	Overall $\uparrow$	A1 $\uparrow$	A2 $\uparrow$	B1 $\uparrow$	B2 $\uparrow$	C1 $\uparrow$	C2 $\uparrow$	C3 $\uparrow$	C4 $\uparrow$	C5 $\uparrow$	C6 $\uparrow$	D1 $\uparrow$
Wan 3.0	70.3	62.8	34.8	54.3	75.3	85.8	69.9	75.0	80.9	78.7	87.5	88.1
Gemini Omni 1.1 Flash	69.3	61.4	39.6	52.7	69.7	78.5	69.1	72.7	80.7	81.0	85.9	87.5
MiniMax H3	68.7	64.8	40.4	47.0	62.0	86.1	70.3	72.2	80.0	80.0	87.5	88.4
Seedance 2.5	64.9	49.6	38.0	40.5	61.9	79.0	67.4	71.2	80.3	78.4	84.4	87.9
PixVerse V6	64.6	63.8	33.9	40.0	55.1	74.4	65.2	62.9	66.7	84.3	87.7	88.3

Appendix

Table 15: V2V results on VidScribe for commercial unified and task-specific open-source video generators.

Figure 12: Group-balanced Overall scores by the number of Hard factors per sample. Black circles show pooled means with 95% within-bin bootstrap confidence intervals; their labels give the means and sample counts

n

. Colored curves show task-wise means.

Figure 13: T2V failure cases: a rare-glyph target (top) and a multi-line archive sign (bottom).

Figure 14: R2V failure cases: a low-contrast Free Wi-Fi poster (top, F9) and stacked ink-brush codes (bottom, F5).

Figure 15: I2V failure cases: a mid-clip chart switch (top, F12) and a persistently occluded character (bottom, F11).

Figure 16: V2V failure cases: a style-only recolor edit (top) and a targeted code-line replacement (bottom).

Explore similar work

Sep 30, 2026cs.CV

ViTeX-Bench: Benchmarking High-Fidelity Video Scene Text Editing

Recent video generation is increasingly realistic and controllable, yet video editing remains less developed, particularly for precise local edits that must preserve the original scene dynamics. Video scene text editing replaces text on scene surfaces, such as storefront signs, whiteboards, and product labels, while preserving the surrounding content, motion, and camera dynamics. Although scene text editing is well studied for images, video scene text editing that achieves high visual quality, temporal consistency, and edit locality remains underexplored. Existing resources offer limited paired real-video data, and general video-editing metrics do not directly measure whether the requested text remains correct over time. We introduce ViTeX-Bench, a benchmark suite comprising ViTeX-Dataset and a three-axis evaluation protocol. The dataset contains 387 real-world 720p videos with text-region masks and editing instructions: 230 provide reviewed, pipeline-generated paired edits for training, and 157 form a frozen evaluation split. The protocol evaluates text correctness, visual and temporal quality, and edit locality through 13 metrics, with one primary metric per axis and a Pareto comparison of their trade-offs. OCR calibration, human evaluation, and annotation-sensitivity analyses support the interpretation of these scores. Across eight baselines from four editing families, accurate text, temporal stability, and scene preservation remain difficult to achieve together. We also release ViTeX-Edit-14B, an open-source reference editor fine-tuned on the paired training split with motion-aligned glyph-video conditioning. It achieves CharAcc 0.688, the highest mean among the evaluated video-native editors, and the lowest comparable text-crop Warp among raw editor outputs. ViTeX-Bench provides a reproducible foundation for studying these trade-offs in video scene text editing.

Xinghao Chen, Xiangbo Gao, Jiongze Yu +2

Texas A&M University

Video Editing Text-To-Video Models Longvideobench

Ask PDF

Sep 28, 2026cs.CV

Enhanced Video Text Editing with Trajectory-Aligned Glyph Rendering

Video text editing aims to replace or add text in a video while keeping the rest of the video unchanged, which requires the edited text to be correct in every frame and to move coherently with the scene. Despite the remarkable progress of video diffusion models, they struggle to reproduce exact stroke structures and often produce garbled or wrong characters, especially for characters with complex strokes. To address this, we propose a trajectory-aligned glyph rendering reference that provides explicit per-frame glyph guidance following the position and perspective of the text, and a depth-normalized recognizer feature supervision that supervises the generated text on multi-depth features of a frozen text recognizer with per-depth normalized errors, targeting stroke errors overlooked by the diffusion loss. We further build VTEdit, a benchmark of 288 real-scene clips with 440 annotated text trajectories covering text replacement and text addition, which will be publicly released to facilitate future research. Experiments on VTEdit show that our method outperforms image text editing methods, video editing methods, and commercial models in text accuracy and background preservation, achieving a sentence accuracy of 0.9408, and receives the highest preference in a user study.

Shulian Zhang, Xiangyu Shu, Wenbo Li +2

South China University of Technology · The Chinese University of Hong Kong · Huawei

Text-To-Video Models Diffusion Editing Typography Generation

Ask PDF

Jun 4, 2026cs.CV

V2V-Bench: A Comprehensive Benchmark for Video-to-Video Generation Evaluation

Video-to-video (V2V) generation is difficult to evaluate because outputs must both follow editing instructions and preserve frame-level correspondence with the source video, which existing T2V and I2V metrics do not capture. We introduce V2V-Bench, a 11-dimension benchmark organized into five categories: temporal alignment, structural fidelity, transformation quality, video quality, and semantic alignment. V2V-Bench pairs diverse source videos with challenging editing tasks and evaluates two commercial models, Grok Imagine and Gemini Veo3, and one open-source model, Open Sora 2. Results show complementary model strengths: Grok performs better on editing fidelity, while Veo3 achieves stronger visual quality. On six V2V-specific dimensions, V2V-Bench reaches a Spearman correlation of 0.905 with human judgments.

Tao Liu, Leela Krishna, Gouti Pavan Kumar +2

Centific Global Solutions Inc.

Video Quality Videos

Ask PDF