SLVMEval: Synthetic Meta Evaluation Benchmark for Text-to-Long Video Generation

Ryosuke Matsuda; Keito Kudo; Haruto Yoshida; Nobuyuki Shimizu; Jun Suzuki

cs.CVMar 31, 2026

SLVMEval: Synthetic Meta Evaluation Benchmark for Text-to-Long Video Generation

Authors: Ryosuke Matsuda, Keito Kudo, Haruto Yoshida, Nobuyuki Shimizu, Jun Suzuki

Organizations: Tohoku University · LY Corporation

Abstract

This paper proposes the synthetic long-video meta-evaluation (SLVMEval), a benchmark for meta-evaluating text-to-video (T2V) evaluation systems. The proposed SLVMEval benchmark focuses on assessing these systems on videos of up to 10,486 s (approximately 3 h). The benchmark targets a fundamental requirement, namely, whether the systems can accurately assess video quality in settings that are easy for humans to assess. We adopt a pairwise comparison-based meta-evaluation framework. Building on dense video-captioning datasets, we synthetically degrade source videos to create controlled "high-quality versus low-quality" pairs across 10 distinct aspects. Then, we employ crowdsourcing to filter and retain only those pairs in which the degradation is clearly perceptible, thereby establishing an effective final testbed. Using this testbed, we assess the reliability of existing evaluation systems in ranking these pairs. Experimental results demonstrate that human evaluators can identify the better long video with 84.7%-96.8% accuracy, and in nine of the 10 aspects, the accuracy of these systems falls short of human assessment, revealing weaknesses in text-to-long-video evaluation.

Figures & tables

	Human Annotation	Aspects	Videos	Unique Prompts	Max Duration (sec)	Avg. Prompt Length (chars)
UVE-Bench [ 28 ]	$\checkmark$	15	1,045	293	6.1	73.68
VBench [ 15 ]	$\checkmark$	16	21,110	968	3.3	41.32
VBench Long [ 16 ]		16	N/A	944	N/A	41.00
SLVMEval ( ours )	$\checkmark$	10	3,932	1,461	10,486.0	57,883.52

Table 1: Statistics of existing and proposed benchmarks. VBench-Long [ 16 ] only provides a framework for automatic evaluation (it does not include human annotations of video quality). It supplies prompts as inputs to T2V models, assuming generated videos of approximately 1 min [ 14 ] .

	Video Quality				Video-Text Consistency
	Aesthetics	Technical Quality	Appearance Style	Background Consistency	Object Integrity	Color	Dynamics Degree	Comprehen- siveness	Spatial Relationship	Temporal Flow
Video-based
GPT-5	90.1 $\pm$ 2.5	85.8 $\pm$ 4.2	88.9 $\pm$ 2.5	98.9 $\pm$ 0.8	72.0 $\pm$ 6.2	84.3 $\pm$ 3.5	35.3 $\pm$ 3.6	51.3 $\pm$ 4.5	59.7 $\pm$ 4.4	50.3 $\pm$ 4.1
GPT-5-mini	84.0 $\pm$ 3.0	48.1 $\pm$ 6.1	78.0 $\pm$ 3.2	95.2 $\pm$ 1.6	66.5 $\pm$ 6.5	69.4 $\pm$ 4.5	31.5 $\pm$ 3.5	45.7 $\pm$ 4.5	51.1 $\pm$ 4.5	43.7 $\pm$ 4.1
Qwen3	55.7 $\pm$ 4.1	51.9 $\pm$ 6.1	55.3 $\pm$ 3.9	49.7 $\pm$ 3.7	38.5 $\pm$ 6.7	48.8 $\pm$ 4.9	50.0 $\pm$ 3.8	52.6 $\pm$ 4.5	51.7 $\pm$ 4.5	50.2 $\pm$ 4.1
Text-based

Table 2: Accuracy (%) of each baseline system on SLVMEval. Numbers are accuracy %

\pm

95% CI (percentage points). Blue bold = best per aspect; green = second best. The chance level is 50%.

Figure 3: Relationship between video duration and accuracy. First, we sorted the dataset by video duration, divided it into four bins (intervals), and computed the accuracy within each bin. The x-axis and y-axis represents the average video duration in each bin and the corresponding accuracy, respectively. For each aspect and automatic evaluation system (excluding the human evaluators), we computed the Spearman rank correlation coefficient

\rho_{\mathrm{S}}

between video duration and accuracy. Here, we sorted the samples by video duration, divided them into 50 bins, computed the accuracy within each bin, and then measured Spearman \CJK@punctchar \CJK@uniPunct 0”80”99s

\rho_{\mathrm{S}}

using these 50 accuracy values. The per-aspect Spearman \CJK@punctchar \CJK@uniPunct 0”80”99s

\rho_{\mathrm{S}}

values and associated

p

-values for each automatic evaluation system are reported in the supplementary material.

Figure 4: Relationship between accuracy values before and after filtering on the degraded SLVMEval data. For each aspect, we plot the accuracy of each evaluation system before versus after filtering and compute the Pearson correlation coefficient

\rho_{\mathrm{P}}

from these points. Each marker corresponds to one evaluation system. The horizontal axis and vertical axis show the accuracy on the unfiltered data and the filtered test set, respectively. The solid line is the fitted linear regression line visualizing the correlation.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Aspect	Initial Samples	Excluded ( $C\neq 0$ )	Excluded ( $A\leq B$ )	Final Samples	Retention Rate (%)
Aesthetics	446	92	72	282	63.3
Technical Quality	446	243	72	131	29.4
Appearance Style	446	112	22	312	70.0
Background Consistency	446	75	17	354	79.4
Temporal Flow	444	115	44	285	64.2
Comprehensiveness	430	83	112	235	54.7

Appendix

Table 3: Filtering statistics for the annotated degraded pairs. For each aspect, we report the number of candidate samples before filtering (Initial), the number of samples excluded because at least one worker selected option C (Excluded (

C\neq 0

)), the number of samples excluded because the number of votes for option A did not exceed that for option B (Excluded (

A\leq B

)), the number of samples retained after filtering (Final), and the resulting retention rate (Retention Rate). The filtering conditions follow the human annotation procedure defined in Section 4.4 .

Aspect	Japanese Instruction	English Instruction
Aesthetics	「美的品質」とは，各フレームが構図・調和・写真的品質を含む総合的な視覚的魅力を有しているかどうかを表し，「シーン説明」との色合いに矛盾があったり，動画の見た目の好ましさが劣っている場合は品質が低いとみなします．	“Aesthetics” indicates whether each frame possesses overall visual appeal, including composition, harmony, and photographic quality. Quality is considered low if there is a tonal contradiction with the “scene description” or if the visual appeal of the video is inferior.
Technical Quality	「技術的品質」とは，技術的な映像のミス（例：ぼけ・ブレ，ノイズ，低画質，輪郭歪み）がないかどうかを表し，技術的な映像ミスが存在する場合は品質が低いとみなします．	“Technical Quality” indicates whether the video is free from technical artifacts (e.g., blur/shake, noise, low resolution, contour distortion). Quality is considered low if such technical video errors exist.
Appearance Style	「スタイル」とは，「シーン説明」で指定された外観スタイル（例：油絵風，漫画風，水彩画風，スケッチ風など）が動画内で適切に再現されているかどうかを表し，指定されたスタイルと異なる表現にとなっている場合は品質が低いとみなします．	“Appearance Style” indicates whether the visual style specified in the “scene description” (e.g., oil painting, cartoon, watercolor, sketch) is appropriately reproduced in the video. Quality is considered low if the representation differs from the specified style.
Background Consistency	「背景の一貫性」とは，フレーム間で背景が視覚的に安定かつ一貫して保たれているかどうかを表し，連続フレームで背景が一貫していない動画は品質が低いとみなします．	“Background Consistency” indicates whether the background remains visually stable and consistent across frames. Quality is considered low if the background is inconsistent across consecutive frames.
Object Integrity	「物体の完全性」とは，「シーン説明」の文章で言及されている物体が不足なく動画に写っているかどうかを表し，「シーン説明」に記載された物体が動画に写っていない場合は品質が低いとみなします．	“Object Integrity” indicates whether the objects mentioned in the “scene description” are present in the video without omission. Quality is considered low if objects described in the “scene description” do not appear in the video.
Color	「色」とは，各フレーム内における色が「シーン説明」で指定された色（例：物体の色など）と一致しているかどうかを表し，指定された色と異なる場合は品質が低いとみなします．	“Color” indicates whether the colors within frames match those specified in the “scene description” (e.g., object colors). Quality is considered low if the colors differ from the specifications.

Appendix

Table 4: Evaluation criteria and instructions provided to crowd workers for each aspect. The Japanese instructions were presented to the workers, and the English instructions are their translations.

Figure 7: Rank consistency on FETV.

Figure 8: Distribution of video content categories in SLVMEval, showing a balanced representation across 15 categories.

Figure 9: Distribution of video durations in SLVMEval. We divide the video duration into 100-second intervals and plot the number of videos included in each interval.

	Video Quality				Video-Text Consistency
	Aesthetics	Technical Quality	Appearance Style	Background Consistency	Object Integrity	Color	Dynamics Degree	Comprehen- siveness	Spatial Relationship	Temporal Flow
IgnMax	48.9 $\pm$ 5.8	76.2 $\pm$ 7.3	62.8 $\pm$ 5.4	70.9 $\pm$ 4.7	71.0 $\pm$ 8.9	66.2 $\pm$ 6.5	55.3 $\pm$ 5.3	51.1 $\pm$ 6.4	60.2 $\pm$ 6.2	56.1 $\pm$ 5.8
EachTrunc	54.3 $\pm$ 5.8	72.3 $\pm$ 7.7	66.0 $\pm$ 5.3	76.6 $\pm$ 4.4	75.0 $\pm$ 8.5	68.1 $\pm$ 6.4	55.0 $\pm$ 5.3	60.4 $\pm$ 6.3	56.4 $\pm$ 6.3	54.4 $\pm$ 5.8

Appendix

Table 5: Accuracy (%) of CLIPScore under different prompt-handling strategies. Numbers are accuracy %

\pm

95% CI (percentage points). The chance level is 50%.

aspect	dimension	description
Aesthetics	aesthetics	This viewpoint indicates whether each frame of the video exhibits overall visual appeal, including composition, harmony, and photographic quality.
Technical Quality	technical_quality	This viewpoint indicates whether each frame is free from technical artifacts (e.g., blur, noise, compression artifacts, over- or under-exposure).
Appearance Style	appearance_style	This viewpoint indicates whether the required appearance style in the text prompt (e.g., oil painting style, black-and-white style, watercolor style, cyberpunk style) is appropriately represented.
Background Consistency	background_consistency	This viewpoint indicates whether the background remains visually stable and consistent across frames.
Object Integrity	object_removal	This viewpoint indicates whether a specific object (e.g., dog, car) mentioned in the text prompt is correctly generated.
Color	color	This viewpoint indicates whether the colors specified in the text prompt (for instance, the color of an object) are correctly reproduced.

Appendix

Table 6: Correspondence between aspect, dimension, and description. The dimension and description are included in the prompt.

	Video Quality				Video-Text Consistency
	Aesthetics	Technical Quality	Appearance Style	Background Consistency	Object Integrity	Color	Dynamics Degree	Comprehen- siveness	Spatial Relationship	Temporal Flow
Video-based
GPT-5	-0.6806	-0.4930	-0.1150	0.0487	0.0541	-0.1198	0.1446	-0.2200	-0.1216	-0.4927
GPT-5-mini	-0.5422	-0.3558	-0.2473	-0.2874	-0.2409	-0.3055	-0.0402	-0.2239	-0.1750	-0.3623
Qwen3	-0.1368	-0.0572	-0.0636	-0.2014	0.3848	-0.0683	0.1418	0.0200	-0.2934	0.1239
Text-based

Appendix

Table 7: Spearman rank correlation coefficients

\rho_{\mathrm{S}}

between video duration and accuracy for each evaluation system (rows) and evaluation aspect (columns). For each aspect and system, we sort the test samples by video duration, divide them into 50 bins, compute the accuracy within each bin, and then compute Spearman’s

\rho_{\mathrm{S}}

between the bin index (corresponding to video duration) and the bin-wise accuracy. Negative values indicate that accuracy tends to decrease as video duration increases.

	Video Quality				Video-Text Consistency
	Aesthetics	Technical Quality	Appearance Style	Background Consistency	Object Integrity	Color	Dynamics Degree	Comprehen- siveness	Spatial Relationship	Temporal Flow
Video-based
GPT-5	$\mathbf{0.0000}$	$\mathbf{0.0003}$	$0.4263$	$0.7371$	$0.7090$	$0.4073$	$0.3164$	$0.1248$	$0.4002$	$\mathbf{0.0003}$
GPT-5-mini	$\mathbf{0.0000}$	$\mathbf{0.0112}$	$0.0834$	$\mathbf{0.0430}$	$0.0919$	$\mathbf{0.0310}$	$0.7818$	$0.1181$	$0.2241$	$\mathbf{0.0097}$
Qwen3	$0.3434$	$0.6934$	$0.6606$	$0.1607$	$\mathbf{0.0058}$	$0.6373$	$0.3259$	$0.8902$	$\mathbf{0.0387}$	$0.3915$
Text-based

Appendix

Table 8: Two-sided

p

-values for the Spearman rank correlations in Table 7 . Each entry is rounded to four decimal places. Bold values indicate statistically significant correlations with

p<0.05

	CPU resource	GPU resource
CPU Model	2 x Intel Xeon Platinum 8490H (Sapphire Rapids)	2 x Intel Xeon Platinum 8490H (Sapphire Rapids)
GPU Model	None	4 x NVIDIA H100 (HBM2e)
System Memory (per node)	512 GiB (DDR5)	1 TiB (DDR5)
GPU Memory (per node)	N/A	94 GiB per GPU
Operating System	Rocky Linux	Rocky Linux

Appendix

Table 9: Computing resources used in our experiments.

Our aspect	VideoScore aspect
Aesthetics	visual quality
Technical Quality	visual quality
Appearance Style	visual quality
Background Consistency	factual consistency
Object Integrity	text-to-video alignment
Color	visual quality

Appendix

Table 10: Mapping between our defined aspects and those defined in VideoScore.

Video Generation Evaluation

Video generation evaluation measures visual quality, prompt adherence, temporal consistency, physical plausibility, and task-specific behavior in synthesized videos.

See all 191 papers

A Closer Look at Evaluation of Text-to-Video Models Under Camera Motion
Oct 8, 2025Nithin C. Babu, Aniruddha Mahapatra, Harsh Rangwani +2Video Generation Evaluation Temporal Consistency in Video Generation
Indian Institute of Science · Adobe Research
DirectorBench: Diagnosing Long-Form Video Generation with Personalized Multi-Agent Evaluation
May 28, 2026Jiamin Chen, Qianben Chen, Jiawen Zhang +5Long Video Generation Video Generation Evaluation
1ByteDance Inc. · 2City University of Hong Kong
TextVBench: Evaluating Text Rendering and Preservation in Video Generation
Oct 9, 2026Ruiqiang Zhang, Hengyi Wang, Chang Liu +3Video Generation Evaluation T2V Generation
Anhui Province Key Laboratory of Digital Security, University of Science and Technology of China

Oct 8, 2025J/K move · Enter open · S save

A Closer Look at Evaluation of Text-to-Video Models Under Camera Motion

Nithin C. Babu, Aniruddha Mahapatra, Harsh Rangwani, Rajiv Soundararajan+1

Indian Institute of Science · Adobe Research

Video Generation Evaluation Temporal Consistency in Video Generation

Open paper AskPDF arXiv

Existing text-to-video (T2V) evaluation benchmarks, such as VBench and EvalCrafter, predominantly contain either subject-centric prompts or static camera scenes, overlooking the importance of camera motion, essential for producing cinematic shots with realistic scenes. Due to this, the metrics developed in these works often produce incorrect results in the presence of camera motion. To fill this gap, we introduce DynamicEval, a benchmark comprising of systematically curated prompts that emphasize dynamic camera motion, paired with 45k human annotations on video pairs. DynamicEval evaluates two key dimensions of video quality: the consistency of the foreground subjects and the consistency of the background scene. To address the shortcomings of existing metrics under camera motion, we introduce two novel metrics: 1) Tracker-FG, which evaluates subject consistency by assessing the keypoint tracks of the subject. 2) Debiased Motion Smoothness, for background consistency, devoid from issues of occlusions/disocclusions arising from camera and foreground subject movements. Extensive experiments on the large-scale DynamicEval benchmark demonstrate that our proposed metrics achieve stronger correlations with human preferences at both the video level, and the model level with an increase of 0.32 in average Pearson Linear Correlation.

Abstract

Video Generation Evaluation

Explore similar work

A Closer Look at Evaluation of Text-to-Video Models Under Camera Motion

DirectorBench: Diagnosing Long-Form Video Generation with Personalized Multi-Agent Evaluation

TextVBench: Evaluating Text Rendering and Preservation in Video Generation