EVF-Bench: A Comprehensive Benchmark for E-commerce Visual Effects in Video Generation
Authors: Run Ling, Ke Cao, Runze He, Jie Gao, Ao Ma, Wei Feng, Zhanjie Zhang, Hao Wang, Jinghe Hu, Guibing Guo
Organizations: JD.com, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Software College, Northeastern University, Shenyang, China
E-commerce visual effects (EVF) are dynamic, product-centered visual presentations in e-commerce videos, often accompanied by promotional text. They play an important role in attracting viewer attention and conveying product information. However, general measures of visual quality and text relevance alone do not establish whether generated videos preserve product appearance, maintain readable text, and produce coherent, contextually appropriate effects. To address this issue, we introduce EVF-Bench, a dedicated benchmark spanning six evaluation dimensions: visual quality and text relevance as general dimensions, and motion, product consistency, text stability, and creative rationality as EVF-specific dimensions. To complement the benchmark and support task-aligned research and model adaptation, we further construct EVF-32K, a dataset of over 32000 curated image-text-video triples tailored to e-commerce scenarios. Using this suite, we systematically evaluate representative video generation models and their domain-adapted variants. The results reveal complementary strengths, with no evaluated configuration leading all six dimensions. Together, EVF-Bench and EVF-32K provide a task-oriented evaluation framework and a supporting data resource for e-commerce video generation.
Figures & tables
Fig. 1: Examples of EVF videos across six product domains. Each group shows a sequence of sampled frames from a product-centered presentation, illustrating changes in product pose or surrounding effects alongside promotional text.
Benchmark
Visual quality
Text relevance
Motion
Product consistency
Text stability
Creative rationality
Make-a-Video-Eval [ 7 ]
✓
✓
✗
✗
✗
✗
FETV [ 8 ]
✓
✓
△
✗
✗
✗
T2VScore [ 9 ]
✓
✓
△
✗
✗
✗
EvalCrafter [ 5 ]
✓
✓
△
✗
✗
✗
VBench [ 3 ]
✓
✓
△
✗
✗
✗
VBench++ [ 4 ]
✓
✓
△
✗
✗
✗
TABLE I: Comparison of EVF-Bench with existing benchmarks across six evaluation dimensions. ✓ denotes assessment of the dimension; △ denotes related assessment that is not fully tailored to EVF; ✗ denotes no dedicated assessment of the corresponding EVF requirement, rather than the absence of all related general-purpose measurements.
Dataset
Scene
Resolution
Video clips
Average length (s)
Video duration (h)
Panda-70M [ 12 ]
Open
720P
70M
8.6
167K
ChronoMagic-Pro [ 13 ]
Time-Lapse
720p
460K
234.8
30K
OpenHumanVid [ 14 ]
Human
720P
52.3M
4.9
70K
ActivityNet [ 15 ]
Action
–
27.8K
110.0
849
YouCook2 [ 16 ]
Cooking
–
2K
316.0
176
How2 [ 17 ]
Instruct
–
80K
90.0
2000
TABLE II: Comparison of EVF-32K with existing video datasets in terms of content focus, resolution, clip count, average clip length, and total video duration. All entries follow each dataset’s own reported statistics at the video level, so that the three columns are mutually consistent; average lengths are as reported or derived from the reported duration and clip count; – indicates a value not reported.
Fig. 2: Design of the four EVF-specific metrics in EVF-Bench. ProductScore compares the reference product with its generated appearance; TSScore follows rendered text across sampled frames; MotionXScore characterizes motion through optical flow; and CRScore uses the video, generation prompt, and evaluation instruction to judge creative rationality. AesScore and GmeScore provide complementary assessments of visual quality and text relevance and are not shown here.
Fig. 3: Representative product reference images from the six domains of EVF-Bench. The examples illustrate the variation in product appearance, promotional text, and presentation context covered by the evaluation cases.
Fig. 4: The six-stage construction pipeline of EVF-32K. Product and image filtering provide reference inputs for prompt construction and video generation; video filtering selects candidate clips, and metadata augmentation adds captions, quantitative indicators, and structural attributes.
Fig. 5: Examples of retained and excluded reference images in EVF-32K construction. The upper panels, labeled Good Images , show retained product-centered presentations. The lower panels, labeled Bad Images , show excluded references with dense specifications, instructional content, or multi-panel layouts. These labels indicate suitability for the present task.
Fig. 6: Product-category taxonomy of EVF-32K. The inner ring shows six product domains, and the outer ring lists their 44 fine-grained categories. Sector sizes are schematic and do not represent sample proportions.
Fig. 7: Statistics of EVF-32K: (a) distributions of AesScore, GmeScore, and MotionXScore; (b) product-domain distribution; (c) prompt word cloud; and (d) prompt-length distribution.
Fig. 8: Qualitative comparison of commercial systems, open-source backbones, and adapted variants on two EVF cases. The top panels contain the reference images and prompts; each row below shows sampled frames from one configuration. LTX-* and Wan2.2-* denote LTX-EVF and Wan2.2-EVF, respectively.
Type
Method
AesScore ↑
GmeScore ↑
MotionX ↑
ProductScore ↑
TSScore ↑
CRScore ↑
Commercial
Taobao Xingchen [ 52 ]
26.90%
59.03%
54.33%
47.31%
78.26%
60.31%
Hailuo 02 [ 53 ]
27.11%
58.32%
55.13%
48.42%
77.36%
65.14%
Seedance 1.0 Mini [ 50 ]
27.52%
57.17%
53.02%
45.16%
75.19%
62.00%
Kling 2.1 Std [ 49 ]
31.76%
58.14%
56.58%
50.64%
80.80%
66.25%
Open-source
LTX-Video-13B [ 54 ]
20.20%
56.83%
49.11%
33.79%
64.70%
54.87%
HunyuanVideo [ 2 ]
28.16%
54.50%
48.01%
39.31%
69.76%
54.19%
TABLE III: Quantitative comparison on EVF-Bench across six evaluation dimensions. Higher scores indicate better performance within each column. Values retain the percentage-style reporting scale of the source results. Best values are bold and second-best values are underlined.
Fig. 9: Human evaluation and motion analysis. (a) Reported agreement between automatic metrics and human preferences for four EVF-specific dimensions. (b) A case comparing burst and sustained motion, with scores shown before and after adding the activation-ratio component. The illustrated human preference favors sustained motion.
Recent video generation models can produce highly realistic videos from natural language instructions, with visual quality approaching cinematic standards. Existing evaluation benchmarks, however, predominantly assess visual quality, aesthetic appeal and physical plausibility, while paying limited attention to text, an essential medium for conveying information in everyday scenes. A generated video may appear visually compelling and feature lifelike subjects, yet still render the text within the scene incorrectly. To address this overlooked dimension, we introduce VTR-Bench, a systematic benchmark for evaluating the Visual Text Rendering capabilities of video generation models. VTR-Bench situates text within concrete application scenarios, such as advertisements and scientific videos, with 300 carefully constructed prompts spanning five scenario categories. We develop an automated evaluation pipeline with human alignments that separately assesses text fidelity through carrier-specific transcription and scene and motion requirements through a prompt-specific chain of query. Beyond evaluation, we introduce a Keyframe-Guided Agentic Framework in which a Director agent coordinates image and video generation with visual evaluation, guiding iterative refinement and candidate selection through visual feedback. Experiments on 11 state-of-the-art models reveal widespread difficulties in accurately rendering scene text, with the best-performing model recording an overall word error rate (WER) of 0.250. We further analyze text rendering failures to characterize the challenges faced by current video generation models. These findings highlight visual text rendering as a key challenge for video generation and demonstrate a practical path toward improvement. Code is available at https://github.com/hardenyu21/VTR-Bench.
Yu Huang, Jungang Li, Zhiyuan Wang +8
Department of Data Science, City University of Hong Kong · The Hong Kong University of Science and Technology (Guangzhou) · Westlake University +3
Video-to-video (V2V) generation is difficult to evaluate because outputs must both follow editing instructions and preserve frame-level correspondence with the source video, which existing T2V and I2V metrics do not capture. We introduce V2V-Bench, a 11-dimension benchmark organized into five categories: temporal alignment, structural fidelity, transformation quality, video quality, and semantic alignment. V2V-Bench pairs diverse source videos with challenging editing tasks and evaluates two commercial models, Grok Imagine and Gemini Veo3, and one open-source model, Open Sora 2. Results show complementary model strengths: Grok performs better on editing fidelity, while Veo3 achieves stronger visual quality. On six V2V-specific dimensions, V2V-Bench reaches a Spearman correlation of 0.905 with human judgments.
Recent studies suggest that video generation models can exhibit certain forms of zero-shot visual reasoning through generated frames. Yet reliable evaluation remains challenging: benchmarks should adopt inputs aligned with the visual priors of current video models, require valid evolving processes rather than only plausible final states, and calibrate task difficulty to remain challenging yet partly feasible. To this end, we introduce VGI-bench, containing 27 tasks and 810 instances, organized by a two-level taxonomy of task domains and skill tags for fine-grained evaluation of visual reasoning capabilities of video generation models. Our evaluations show that current generative systems can solve a subset of visually grounded reasoning tasks, but remain far from reliable, with even the strongest model, Seedance 2.0, achieving only 51.0% under our evaluation criteria. Our analysis further explore the output failure modes, input condition sensitivity, performance transfer boundary from synthetic fine-tuning, and internal denoising perspective revealing limited self-correction, where later steps mainly refine early hypotheses rather than correct reasoning errors. We hope VGI-bench will help stimulate the development of next-generation video generation models. Website: https://hexuan21.github.io/VGI-Bench/
Xuan He, Cong Wei, Yuhao Cheng +20
University of Illinois Urbana Champaign · University of Waterloo · Vector Institute +6