Frozen Scenes, Shifting Winners: Configuration Fragility in Text-to-3D Evaluation

Anson Y. Lam; Shuqing Li; Michael R. Lyu

cs.AISep 30, 2026

Frozen Scenes, Shifting Winners: Configuration Fragility in Text-to-3D Evaluation

Authors: Anson Y. Lam, Shuqing Li, Michael R. Lyu

Organizations: Department of Computer Science and Engineering The Chinese University of Hong Kong Hong Kong, China

Abstract

Can a text-to-3D leaderboard change when every generated scene stays fixed? We audit this question for rendered-image evaluation, where camera settings and caption wording become part of the measurement protocol. Across 300 frozen scenes from six generators, we vary eight render and caption factors for 19 alignment evaluators plus one perceptual-quality control, then test four targeted scene degradations. Peak configuration variance exceeds between-generator variance for 17/19 alignment evaluators, with prompt-bootstrap lower bounds above 1 for 11/19. Rankings are more stable than scores, yet 18/19 evaluators change their point-estimate winner under some configuration. Pairwise protocol margin envelopes show which comparisons keep their direction across the tested settings. Selected pairs have opposite pointwise intervals, but no reversal survives simultaneous inference over the full search. Thus the observed winner changes are descriptive, not confirmed changes in generator superiority. Sensitivity remains separate: no evaluator, even the prompt-free control, exceeds 67% tie-adjusted directional discrimination on layout scrambling, which is diagnostic rather than human-validated ground truth. The audit separates score stability, decision uncertainty, and targeted sensitivity, and recommends reporting (generator, score, card ID) with protocol-dependent comparisons and selection-aware uncertainty.

Figures & tables

Figure 1: The artifact stays fixed, but the measurement changes. Three camera pitches reveal different evidence from one frozen room. Our audit holds all 300 scene artifacts fixed and varies the rendering or caption protocol. Winner changes are measured across all 50 prompts, not inferred from this pictured room, and are separated from statistical confirmation in Section 4.3 .

Audit question	Result	Scope and interpretation
Does configuration variance exceed generator separation?	17/19	Peak $\rho>1$ for the six-generator field.
Does this persist under prompt resampling?	11/19	Bootstrap lower bound for peak $\rho$ exceeds 1.
Does the point-estimate winner change?	18/19	Across tested factor levels, without claiming a confirmed reversal.
Is an opposite pairwise ordering confirmed after search?	0/19	Evaluators with a selected reversal under a band over 110,295 contrasts.
Which pairwise orders survive all tested protocols?	4–11/15	Strict orders per alignment evaluator over a finite set of 387 protocols.
Does scrambling lower the score?	$\leq 67\%$	Tie-adjusted ceiling across evaluators, including the render-only control.

Table 1: Decision-level results from 50 prompts, six generators, and 300 fixed scenes, scored by 19 alignment evaluators and one render-only control. Configuration sweeps change one factor at a time from a default, so these results do not measure factor interactions.

Evaluator	template	peak	min $W$	#top-1	Content acc. (%)	Best view
	$\bm{\rho}$		Rank stability		Content acc. (%)	Best view
CLIP ViT-B/32	2.05	3.04 †	0.75	2	67	1.17 $\times$
CLIP ViT-B/16	2.30	2.30 †	0.64	3	68	1.14 $\times$
CLIP ViT-L/14	3.04	3.04 †	0.70	2	71	1.14 $\times$
CLIP ViT-L/14@336	1.85	1.85 †	0.77	2	72	1.12 $\times$
OpenCLIP ViT-L/14	1.08	1.08	0.75	2	70	1.10 $\times$

Table 2: Per-evaluator audit across four distinct questions. Template and peak

\rho

compare configuration variance with between-generator variance in the six-generator field. Rank stability gives the minimum Kendall’s

W

and maximum number of distinct point-estimate winners over factors. Content accuracy averages four directional probes and is not a human-validity score. Best view is the median score inflation from selecting the highest-scoring tested camera. † marks a per-evaluator prompt-bootstrap lower bound above one for peak

\rho

in 11 alignment rows and the prompt-free CLIP-IQA control (Appendix K ).

Figure 2: Score dispersion (normalized range, %) across evaluators and configuration factors in default-conditioned one-factor-at-a-time sweeps. Cells show the true values; colors are clipped at $100\%$ . Caption template is the largest factor for 18 of 19 alignment evaluators, while pitch and lighting dominate the render factors. The prompt-free CLIP-IQA control has no caption measurements.

Slot	Disp.	Choices
Viewpoint	$19\%$	top-down $+7\%$ , bird-eye $+5\%$ , (none) $-12\%$
Head noun	$11\%$	rendering $+6\%$ , view $+1\%$ , image $-1\%$ , picture $-2\%$ , photo $-4\%$
Scene frame	$14\%$	a virtual scene: $+9\%$ , a scene: $-2\%$ , scene: $-2\%$ , (none) $-5\%$
Connective	$3\%$	depicting $+1\%$ , about $0\%$ , of $0\%$
Article	$4\%$	a $+1\%$ , (none) $0\%$ , an $-1\%$

Table 3: Per-slot caption-template effects over the 19 text-swept evaluators. Dispersion is the range of choice means as a percentage of the slot mean, and shifts are relative to that mean. Each slot is measured on a matched subset of captions to limit confounding by co-occurring slots. (none) omits an optional slot.

Evaluator / factor	Method pair	Levels	Gap [pointwise 95%]	Simultaneous 95%
CLIP ViT-B/32 Lighting	Scenethesis $-$ I-Design	city night	$+0.91\ [+0.12,+1.72]$ $-0.91\ [-1.62,-0.20]$	$[-0.88,+2.70]$ $[-2.56,+0.73]$
CLIP ViT-L/14@336 Pitch	Scenethesis $-$ Reason-3D	15 ∘ 90 ∘	$+1.53\ [+0.52,+2.62]$ $-1.39\ [-2.57,-0.23]$	$[-0.79,+3.85]$ $[-4.07,+1.29]$
MetaCLIP ViT-L/14 Pitch	Scenethesis $-$ LayoutVLM	0 ∘ 75 ∘	$+2.31\ [+1.03,+3.67]$ $-1.57\ [-2.88,-0.22]$	$[-0.63,+5.26]$ $[-4.54,+1.41]$
BLIP-VQA Template	Scenethesis $-$ Holodeck	T1 T2	$+4.33\ [+0.13,+8.93]$ $-7.52\ [-11.81,-3.67]$	$[-5.63,+14.29]$ $[-17.01,+1.97]$

Table 4: Selected same-pair point-estimate reversals at two configurations. Gaps have the same first-method-minus-second-method orientation within each row, but their scales differ across evaluators. Pointwise

95\%

intervals exclude zero in opposite directions. Every full-search simultaneous interval includes zero. The BLIP-VQA pointwise result changes under a second bootstrap seed.

Figure 3: Tie-adjusted directional discrimination against each intact scene, with the

50\%

chance line dashed. Removing objects is easier to detect than scrambling their layout across this fixed dataset. The scramble ceiling belongs to the prompt-free CLIP-IQA control.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Work	Artifact held fixed	Render factors isolated	Caption wording isolated	Decision stability	Controlled degradation	Reporting artifact
AutoMetrics ( Ryan et al., 2026 )	Equivalent-quality outputs, not one frozen artifact	Not applicable to 3D rendering	Output rephrasing, not a 3D caption wrapper	Metric validity, not generator-winner stability	LLM-generated worse-quality outputs	Metric Cards, not a 3D configuration card
T2IScoreScore ( Saxon et al., 2024 )	Related 2D image families under a fixed prompt	Not applicable to 3D rendering	Semantic error graphs, not caption wrappers	Metric ordering, not generator-winner stability	Semantic error graphs	No analogous card in the stated contribution
3D-DefectBench ( Zhao et al., 2026 )	Fixed assets across 84 inference designs	Camera protocol and visual input crossed	Prompt schema crossed	Factor effects and judge comparisons for defect detection	Nine human-labeled defect categories	Released data and metadata, not a reporting card
Cross-Model VLM-Judge ( Asaria et al., 2026 )	Fixed mesh pairs across 24 rendered views	Fixed multi-view rig, not render-factor isolation	Fixed judge prompt, not a caption-wording audit	Pairwise asset preference, not generator-winner stability	Face-drop degradation, not room-layout scrambling	Reproducible judge protocol, not a configuration card
SceneCritic ( Sengupta et al., 2026 )	Yes across views and repeated VLM calls	Viewpoint and repeated calls, not 8 OFAT factors	Prompt sensitivity motivated, not experimentally swept	View-dependent method-ranking reversals for one VLM judge	No fixed-artifact degradation audit	SceneOnto and a symbolic critic, not a configuration card
T3Bench and Eval3D ( He et al., 2023 ; Duggal et al., 2025 )	Generated assets, not a same-artifact audit	Many views, not a render-factor audit	Caption or QA pipelines, not a wording audit	Leaderboards, not stability under configurations	Janus or inconsistency probes, not room-layout scrambling	No analogous card in the stated contribution

Appendix

Table 5: Closest-work boundary. The cited work improves or meta-evaluates text, 2D, and 3D metrics, while the table distinguishes adjacent components from our combined intervention. “Related” marks an adjacent design rather than the same intervention.

Card field	Required report content
Schema and status	card_type , schema_version , evidence status, and the validator-produced card identifier.
Scene artifact source	Dataset, generator set, scene and prompt counts, frozen-scene rule, and artifact provenance.
Render protocol	Renderer, camera pose, focal length, resolution, lighting, background, random seeds, and post-processing.
Caption protocol	Prompt template, paraphrase rule, prompt source, and whether wording was frozen before scoring.
View policy	Fixed, averaged, or selected view, declared views, and selection rule.
Evaluator identity	Model or metric name, checkpoint/API snapshot, instruction, decoding settings, and call date.

Appendix

Table 6: Paper-ready Evaluator Configuration Card checklist. A complete JSON card records these fields, validates against schema version 1.0 , and receives a deterministic ecc-v1-sha256:… identifier.

Panel of Figure 4 — Figure 4: The same scene under two configuration factors. Top: focal length $16$ / $50$ / $200$ mm at a fixed, auto-framed top-down view. Bottom: environment-map lighting (studio/city/sunset). Figure 1 shows the pitch sweep in the main paper.

Slot	Status	Choices
Article	optional	a, an
Viewpoint	optional	top-down, bird-eye
Head noun	optional	image, photo, picture, rendering, view
Connective	with a noun	of, about, depicting
Scene frame	optional	scene:, a scene:, a virtual scene:
Prompt	required	{prompt}

Appendix

Table 7: The template grammar as ordered slots. A caption picks at most one entry from each slot in order (subject to the adjacency in Figure 5 ) and always ends with the prompt. The connective is present exactly when a head noun is. The default template “a top-down view of { prompt } ” takes the article, the top-down viewpoint, view , the connective of , no scene frame, and the prompt.

Figure 5: The template grammar as a directed acyclic token graph. Each of the

337

templates is exactly one source-to-sink path (START

\rightarrow\cdots\rightarrow

{prompt}

\rightarrow

END). Each edge is colored by the node it points into, and every node is outlined in its own color, so an edge and its target always match. The dashed edge from START to {prompt} is the degenerate identity template.

Environment map	EV	CCT (K)	Mean $z$ -shift
studio	-6.41	7802	-0.067
night	-5.13	5051	-0.173
courtyard	-3.70	5759	-0.017
sunrise	-3.24	6004	-0.044
forest	-2.22	7513	+0.075
interior	-2.20	5420	+0.077

Appendix

Table 8: The eight lighting maps ordered dark to bright. EV is a log-average exposure value (lower is darker). CCT is correlated color temperature (lower is warmer). The default map “city” is the brightest. Mean

z

-shift is the map’s average standardized score shift across evaluators and scenes.

Evaluator	Protocols	Strict ordering	Sign crossing	Tie boundary	Stable winner
CLIP ViT-B/32	387	5/15	10/15	0/15	none
CLIP ViT-B/16	387	4/15	11/15	0/15	none
CLIP ViT-L/14	387	4/15	11/15	0/15	none
CLIP ViT-L/14@336	387	5/15	10/15	0/15	none
OpenCLIP ViT-L/14	387	5/15	10/15	0/15	none
OpenCLIP ViT-H/14	387	5/15	10/15	0/15	none

Appendix

Table 9: Pairwise ordering over the finite tested protocol set. Strict ordering means that a generator pair keeps the same nonzero margin sign throughout the set. A sign crossing includes both positive and negative margins, while a tie boundary reaches zero without crossing signs. A stable winner beats all five alternatives at every tested protocol. These descriptive ranges are not confidence intervals or statements of human-grounded superiority. The hosted judge’s zero-score handling is documented in Appendix I , and the image-only control, marked qc , lacks a caption sweep and is summarized over its 50 render-only protocols.

Outcome	Count	Percent [95% interval]
Strict drop	124	41.3 [35.0, 47.7]
Exact tie	77	25.7 [20.7, 31.0]
Rise	99	33.0 [27.0, 38.7]
Tie-adjusted discrimination		54.2 [48.5, 59.5]

Appendix

Table 10: GPT-4o scramble outcomes from the same 300 intact and scrambled score pairs. Intervals use the joint prompt bootstrap of Appendix H and are pointwise descriptive summaries. Tie-adjusted discrimination credits a drop as one, a tie as one-half, and a rise as zero.

Factor	#	Levels
Resolution (pixel)	9	196, 224, 256, 336, 384, 448, 512 ⋆ , 768, 1024
Focal length (mm)	7	16, 24, 35, 50 ⋆ , 85, 100, 200
Lighting (environment map)	8	city ⋆ , courtyard, forest, interior, night, studio, sunrise, sunset
Background (RGB)	10	0, 65, 118, 128, 186, 204, 255 ⋆ gray, plus red, green, blue
Pitch (degrees)	7	0 ⋆ , 15, 30, 45, 60, 75, 90
Yaw (degrees, at pitch $45^{\circ}$ )	8	0 ⋆ , 45, 90, 135, 180, 225, 270, 315

Appendix

Table 11: Evaluator-configuration factors, each swept independently with all others held at the default

c_{0}

(

\star

). The cross-factor baseline uses a white background. The background sweep ranges over the listed gray and chromatic fills.

Group	Members
CLIP (OpenAI)	ViT-B/32, ViT-B/16, ViT-L/14, ViT-L/14@336 ( Radford et al., 2021 )
CLIP (open data)	OpenCLIP ViT-L/14, ViT-H/14, ViT-bigG/14 ( Cherti et al., 2023 ) , MetaCLIP ( Xu et al., 2024 ) , DFN ( Fang et al., 2024 ) , EVA-CLIP ( Sun et al., 2023 ) , and CLIPA ( Li et al., 2023b )
CLIP (long-text)	Long-CLIP ( Zhang et al., 2024 )
Sigmoid objective	SigLIP ( Zhai et al., 2023 ) and SigLIP 2 ( Tschannen et al., 2025 )
BLIP-2 ( Li et al., 2023a )	ITM and ITC heads
VQA-based	BLIP-VQA ( Huang et al., 2023 ) and VQAScore ( Lin et al., 2024 )

Appendix

Table 12: The 20 evaluators under audit, grouped by family. CLIP-IQA is reported apart from the alignment metrics because it does not condition on the prompt.

Figure 6: Peak configuration-to-signal ratio for the fixed six-generator field, with per-evaluator prompt-bootstrap

95\%

intervals (50 prompts with generators fixed). The lower bound exceeds one for 11 of 19 alignment evaluators. CLIP-IQA is shown separately as a prompt-free quality control. A high ratio alone does not establish a changed pairwise ordering.

Evaluator	Res.	Focal	Light	Bg.	Pitch	Yaw	Tmpl.	Para.
CLIP ViT-B/32	7	13	13	8	20	16	28	13
CLIP ViT-B/16	9	12	12	7	18	14	31	12
CLIP ViT-L/14	15	21	20	12	26	23	51	18
CLIP ViT-L/14@336	14	16	17	10	24	19	51	18
OpenCLIP ViT-L/14	20	19	21	12	31	26	56	16
OpenCLIP ViT-H/14	15	16	20	14	31	25	60	18

Appendix

Table 13: Score dispersion (normalized range, %) per evaluator and factor. Larger is worse. Res. is resolution, Bg. is background, Tmpl. is template, and Para. is paraphrase.

Evaluator	Res.	Focal	Light	Bg.	Pitch	Yaw	Tmpl.	Para.
CLIP ViT-B/32	0.34	1.34	1.09	0.46	3.04	2.01	2.05	0.97
CLIP ViT-B/16	0.36	0.88	0.86	0.24	2.07	1.35	2.30	0.72
CLIP ViT-L/14	0.57	1.35	1.05	0.35	2.10	1.37	3.04	0.78
CLIP ViT-L/14@336	0.30	0.54	0.49	0.17	1.13	0.64	1.85	0.47
OpenCLIP ViT-L/14	0.28	0.36	0.34	0.11	0.90	0.50	1.08	0.20
OpenCLIP ViT-H/14	0.25	0.34	0.44	0.23	1.16	0.56	2.04	0.34

Appendix

Table 14: Configuration-to-signal ratio

\rho

per evaluator and factor. Values

\geq 1

mean that within-scene configuration variance equals or exceeds the variance between generator mean scores on that evaluator’s scale. Res. is resolution, Bg. is background, Tmpl. is template, and Para. is paraphrase.

Evaluator	Peak $\rho$	Prompt 95% CI	Leave-one-gen range	Top-1/2 gap [95% CI]
CLIP ViT-B/32	3.04	[1.92, 4.41]	[2.51, 9.06]	+0.91 [+0.12, +1.73] ∗
CLIP ViT-B/16	2.30	[1.51, 3.25]	[1.91, 7.92]	+0.33 [-0.64, +1.28]
CLIP ViT-L/14	3.04	[1.97, 4.61]	[2.54, 9.60]	+1.12 [-0.24, +2.38]
CLIP ViT-L/14@336	1.85	[1.31, 2.69]	[1.52, 5.94]	+1.71 [+0.46, +2.87] ∗
OpenCLIP ViT-L/14	1.08	[0.76, 1.49]	[0.89, 6.45]	+2.59 [+1.11, +4.03] ∗
OpenCLIP ViT-H/14	2.04	[1.44, 2.84]	[1.70, 12.69]	+1.49 [+0.01, +2.90] ∗

Appendix

Table 15: Uncertainty and generator-set sensitivity of the configuration-to-signal ratio, and baseline generator separability. Peak

\rho

is the point estimate of Table 2 . Prompt 95% CI resamples the

N=50

prompts with the six generators fixed. Leave-one-gen range recomputes peak

\rho

after dropping each generator. Top-1/2 gap is the paired-bootstrap difference between the two baseline leaders, and ∗ marks an interval that excludes zero. This column does not identify the actual winner pair at every configuration level. CLIP-IQA has no alignment Top-1/2 gap because it never conditions on the prompt, and its other rank columns are quality-control pseudo-rankings.

Evaluator	Res.	Focal	Light	Bg.	Pitch	Yaw	Tmpl.
CLIP ViT-B/32	0.91/1	0.89/2	0.83/2	0.95/1	0.75/2	0.87/2	0.94/1
CLIP ViT-B/16	0.97/1	0.89/1	0.75/3	0.97/2	0.64/2	0.73/1	0.89/2
CLIP ViT-L/14	0.91/1	0.89/1	0.70/2	0.95/1	0.82/2	0.94/2	0.91/2
CLIP ViT-L/14@336	0.87/1	0.95/1	0.84/2	0.97/1	0.85/2	0.77/1	0.89/1
OpenCLIP ViT-L/14	0.85/1	0.94/1	0.75/2	0.93/1	0.80/2	0.89/1	0.97/1
OpenCLIP ViT-H/14	0.87/1	0.95/1	0.72/2	0.95/1	0.81/3	0.95/1	0.94/1

Appendix

Table 16: Rank stability per factor, reported as Kendall’s

W

/ number of distinct top-1 winners across levels.

W=1

with a single winner is perfect stability. Paraphrases are excluded because their levels are per-scene strings that cannot be aligned across scenes. Res. is resolution, Bg. is background, and Tmpl. is template.

Evaluator	Keep-half	Biggest-only	Scrambled	Worst-objects	Mean (4)
CLIP ViT-B/32	64	74	56	72	67
CLIP ViT-B/16	64	75	57	77	68
CLIP ViT-L/14	69	78	57	80	71
CLIP ViT-L/14@336	71	79	58	81	72
OpenCLIP ViT-L/14	66	85	54	76	70
OpenCLIP ViT-H/14	70	86	62	81	75

Appendix

Table 17: Tie-adjusted directional discrimination (%): a strict drop receives 1, an exact tie 0.5, and a rise 0 (chance

=50\%

). The intended order is a dataset-level diagnostic expectation rather than per-pair human ground truth. The final column, “Mean (4),” is the plain average of the four probe columns, and is the value reported as Content acc. in Table 2 .

Evaluator	Resolution	Focal	Lighting	Background	Pitch	Yaw
CLIP ViT-B/32	224/1024	85/16	interior/studio	204/green	45/0	180/45
CLIP ViT-B/16	336/224	200/16	forest/night	255/65	45/0	180/135
CLIP ViT-L/14	448/224	200/16	city/night	255/green	30/90	90/180
CLIP ViT-L/14@336	1024/384	200/16	forest/night	255/green	15/90	45/180
OpenCLIP ViT-L/14	1024/196	200/16	city/night	128/green	15/90	0/225
OpenCLIP ViT-H/14	768/196	200/16	city/night	128/red	15/90	0/225

Appendix

Table 18: Highest- and lowest-scoring level ( best/worst ) of each render factor per evaluator, by mean score over the 300 scenes. Background is given as its gray value (

0

black to

255

white) or color name. No level is established as human-grounded ground truth, yet the best entries are dominated by telephoto, bright, and shallow-pitch levels and the worst by wide-angle, dark, and eye-level.

Explore similar work

VTR-Bench: A Systematic Benchmark for Evaluating Visual Text Rendering in Video Generation
Oct 1, 2026Yu Huang, Jungang Li, Zhiyuan Wang +8Text-To-Video Generation Model Video Generation
Department of Data Science, City University of Hong Kong · The Hong Kong University of Science and Technology (Guangzhou) · Westlake University +3
DynEval: Holistic Evaluations of T2I Generative Models in the Wild
Jul 13, 2026Shyam Marjit, Dheeraj Baiju, Anuj Shikarkhane +3Modern Text-To-Image Models Text-To-Image
Indian Institute of Science · VCL, IISc · Hugging Face
3D-DefectBench: A Controlled Factorial Study of Vision-Language Model Evaluation Pipelines for Fine-Grained 3D Generation Defects
Jul 12, 2026Zhenyu Zhao, Nanshan Jia, Jihyeon Je +7Controlled Study
1Roblox Corporation · 2Berkeley AI Research Lab & Department of Industrial Engineering and Operations Research, University of California, Berkeley · * This work was done while Nanshan Jia, Jihyeon Je, and Yifu Tang were interns at Roblox Corporation. +2

Oct 1, 2026, cs.CVJ/K move · Enter open · S save

VTR-Bench: A Systematic Benchmark for Evaluating Visual Text Rendering in Video Generation

Yu Huang, Jungang Li, Zhiyuan Wang, Yonghua Hei+7

Department of Data Science, City University of Hong Kong · The Hong Kong University of Science and Technology (Guangzhou) · Westlake University+3

Text-To-Video Generation Model Video Generation Cinematic Ideal Text-Driven

Open paper AskPDF arXiv

Recent video generation models can produce highly realistic videos from natural language instructions, with visual quality approaching cinematic standards. Existing evaluation benchmarks, however, predominantly assess visual quality, aesthetic appeal and physical plausibility, while paying limited attention to text, an essential medium for conveying information in everyday scenes. A generated video may appear visually compelling and feature lifelike subjects, yet still render the text within the scene incorrectly. To address this overlooked dimension, we introduce \textbf{VTR-Bench}, a systematic benchmark for evaluating the \textbf{V}isual \textbf{T}ext \textbf{R}endering capabilities of video generation models. VTR-Bench situates text within concrete application scenarios, such as advertisements and scientific videos, with 300 carefully constructed prompts spanning five scenario categories. We develop an automated evaluation pipeline with human alignments that separately assesses text fidelity through carrier-specific transcription and scene and motion requirements through a prompt-specific chain of query. Beyond evaluation, we introduce a \textbf{Keyframe-Guided Agentic Framework} in which a Director agent coordinates image and video generation with visual evaluation, guiding iterative refinement and candidate selection through visual feedback. Experiments on 11 state-of-the-art models reveal widespread difficulties in accurately rendering scene text, with the best-performing model recording an overall word error rate (WER) of 0.250. We further analyze text rendering failures to characterize the challenges faced by current video generation models. These findings highlight visual text rendering as a key challenge for video generation and demonstrate a practical path toward improvement. Code is available at https://github.com/hardenyu21/VTR-Bench.

Abstract

Explore similar work

VTR-Bench: A Systematic Benchmark for Evaluating Visual Text Rendering in Video Generation

DynEval: Holistic Evaluations of T2I Generative Models in the Wild

3D-DefectBench: A Controlled Factorial Study of Vision-Language Model Evaluation Pipelines for Fine-Grained 3D Generation Defects