Existing text-to-video (T2V) evaluation benchmarks, such as VBench and EvalCrafter, predominantly contain either subject-centric prompts or static camera scenes, overlooking the importance of camera motion, essential for producing cinematic shots with realistic scenes. Due to this, the metrics developed in these works often produce incorrect results in the presence of camera motion. To fill this gap, we introduce DynamicEval, a benchmark comprising of systematically curated prompts that emphasize dynamic camera motion, paired with 45k human annotations on video pairs. DynamicEval evaluates two key dimensions of video quality: the consistency of the foreground subjects and the consistency of the background scene. To address the shortcomings of existing metrics under camera motion, we introduce two novel metrics: 1) Tracker-FG, which evaluates subject consistency by assessing the keypoint tracks of the subject. 2) Debiased Motion Smoothness, for background consistency, devoid from issues of occlusions/disocclusions arising from camera and foreground subject movements. Extensive experiments on the large-scale DynamicEval benchmark demonstrate that our proposed metrics achieve stronger correlations with human preferences at both the video level, and the model level with an increase of 0.32 in average Pearson Linear Correlation.
Figures & tables
Figure 1: Existing metrics (e.g., VB-Subject Consistency) favor low-motion videos even when they contain subject distortions. Our motion-invariant Tracker-FG metric reliably evaluates foreground quality regardless of camera or object motion. On our large-scale DynamicEval benchmark, prior metrics show substantial performance drops (top right), while our proposed metrics (Tracker-FG and MS-Debias) achieve significantly higher Pearson’s Linear Correlation Coefficient (PLCC) with model-level human preferences (bottom right).
Figure 2: Prompt curation: Scene elements from databases (orange) are sampled into a metadata (JSON format), which GPT-4o converts into descriptive prompts. Dataset: Video pairs generated from a common prompt are annotated via a subjective study.
Figure 3: Dataset analysis: The plot on the left shows the average win rates for both evaluation dimensions. The plot on right illustrates the percentage of video samples in each model that are static and dynamic.
Figure 4: Tracker Motivation: Deviation of neighbor tracks on real video vs generated video with subject distortions. Blue plot: Distance between a nearest neighbor and a candidate point. Red plot: Moving average of blue. Green plot: the deviation between blue and red.
Figure 5: Tracker-FG framework: We track the points inside an object mask across the video. TrackerFG is obtained by computing the average of distances between neighboring tracking points.
Figure 6: Motion Smoothness error maps: The zoomed-in regions show localized distortions visible across frames. VB-MS shows large errors near edges and foreground objects, suppressing the localized distortions. After debiasing, the localized distortions are visible.
Figure 7: MS-Debias: obtains debiased motion smoothness error maps by masking out foreground subjects and occlusions. It is applied at multiple scales with Gaussian pyramid downsampling, for improving the robustness of the metric.
Table 2: Performance Comparisons: The first table shows pairwise preference accuracy on the DynamicEval database, and the second table evaluates the Top-k video selection accuracy. On the full dataset, our method, MS-Debias outperforms the VB-BG by 2.2 % points and on the full agreement subset by 2.4 % points. Similar boosts can be seen for subject consistency pairwise accuracy.
Subject Consistency
Background Consistency
Method
static-dynamic
dynamic-dynamic
Method
static-dynamic
dynamic-dynamic
VB-SC
57.1
57.3
VB-BG
55.2
56.1
Tracker-FG
60.3
58.1
MS-Debias
58.0
56.7
Table 3: Video-Pair Configurations: Performance of each metric when the pair contains different configurations of static-dynamic pairs. The improved metrics ensure no bias, for static videos over the dynamic ones, in the proposed Tracker-FG and MS-Debias metrics.
Method (FG)
PLCC
Method (BG)
PLCC
VB-SC
0.334
VB-BG
0.551
Tracker-FG
0.772
MS-Debias
0.743
Table 4: PLCC of model-level win ratios between the metrics and human preference. We evaluate these on the full agreement subset.
”In an auto factory, a lone dog drinks water from a puddle amidst hulking machinery and assembly lines. The ground shot dolly camera moves forward, revealing industrial surroundings with the dog in sharp focus.”
Appendix
Table 6: Structured metadata (left) and its descriptive narrative (right).
Figure 8: A screenshot of the User Interface for pairwise comparison
Figure 9: Dataset analysis: Percentage of categories of scene elements in the prompts suite, and the prompt wordcloud. A majority of the words focus towards camera movement.
Figure 10: Effect of camera motion on feature-level metrics: We observe that with large camera motions in the scene, the feature level metrics get affected despite having high quality video.
Figure 11: Comparison of ratio static videos in DynamicEval (DE) vs VBench (VB) dataset on matching models. The ratio of static scenes generated from VBench prompts are significantly higher than the ones generated from DynamicEval.
Figure 12: Multi-scale processing in MS-Debias: Different down-sampled versions of the input video are processed independently through the MS-Debias pipeline to obtain background consistency scores at multiple levels. The scores are then averaged using weights proportional to the scales to obtain the final metric score. This mimics the multi-scale processing capabilities of the human visual system.
Figure 13: Baseline metrics (VB-SC and VB-BG) exhibit a strong bias with respect to camera motion. As camera motion increases, their scores decrease even for high-quality videos, which is consistent with the visualizations in Fig. 10 . In foreground subject consistency, even though video V1 is of high quality and V2 shows a cat morphing (distortion), VB-SC gives a lower score to V1 because it has a larger camera motion. Similarly, in background consistency, VB-BG gives a low score to V1 , which has more camera motion even though V2 exhibits significant localized distortions. Our metrics are independent of camera motion bias.
Table 9: Statistical significance tests ( 95% confidence level) across all pairs of foreground subject consistency metrics. Table (a) shows the χ2 and p values of McNemar’s statistical test that tests whether the pairwise accuracy differences are significant. A value of p<0.05 for a pair of methods means that the accuracy differences are statistically significant. Table (c) shows the low-high bounds of 95% bootstrap percentile confidence interval of accuracy differences. The accuracy differences are statistically significant if the interval does not contain 0 . Table (b) and (d) show the statistical significance matrix, in which, ‘1’: metric in the row has statistically significant performance gains over the metric mentioned in the corresponding column; ‘0’: row metric has statistically significantly poorer performance; ‘-’: the accuracy difference is not statistically significant.
Table 10: Statistical significance tests ( 95% confidence level) across all pairs of background consistency metrics. Details are already discussed in Table 9 .
Background Consistency
Comparison Type
VB-BG
MS-Debias
Inter-model Comparisons
54.8
58.3
Intra-model Comparisons
57.7
58.2
Subject Consistency
VB-SC
Tracker-FG
Inter-model Comparisons
54.3
56.5
Appendix
Table 13: Performance of the methods on inter-model vs intra-model video comparisons. The proposed metrics perform better on both comparisons, with higher performance on intra-model pairs, as the videos contain similar content.
Camera-prompted text-to-video (T2V) models are increasingly used to synthesize virtual camera captures, such as orbiting objects or moving through static scenes. For these outputs, visual plausibility is insufficient: the generated frames should also provide coherent multi-view evidence for a single static 3D scene. We introduce GeoT2V-Bench, a reconstruction-based diagnostic benchmark for evaluating whether camera-prompted T2V clips can support explicit rigid 3D reconstruction. Our pipeline estimates per-frame camera intrinsics and poses with VGGT-style geometry estimation, fits DeformableGS, derives a static MedianGS proxy by temporal-median aggregation, and renders this proxy along the estimated camera path. Instead of producing a pass/fail label or a single scalar score, GeoT2V-Bench reports a continuous reconstruction profile covering apparent image motion, estimated trajectory behavior, MedianGS static rendering error, static-render flow agreement, and the gap between flexible and static fits. On a fair-format four-seed evaluation with 3,840 completed reconstructions from 12 open-weight model configurations and 80 GeCo-Eval static-scene prompts, we find that visible motion, static rendering error, flow agreement, and flexible-vs-static behavior often disagree. GeoT2V-Bench therefore captures complementary failure modes that emerge when generated videos are tested as global static-scene acquisitions.
Chenrui Fan, Paolo Favaro
Computer Vision Group, Institute of Computer Science, University of Bern, Switzerland
Recent video generation models can produce highly realistic videos from natural language instructions, with visual quality approaching cinematic standards. Existing evaluation benchmarks, however, predominantly assess visual quality, aesthetic appeal and physical plausibility, while paying limited attention to text, an essential medium for conveying information in everyday scenes. A generated video may appear visually compelling and feature lifelike subjects, yet still render the text within the scene incorrectly. To address this overlooked dimension, we introduce VTR-Bench, a systematic benchmark for evaluating the Visual Text Rendering capabilities of video generation models. VTR-Bench situates text within concrete application scenarios, such as advertisements and scientific videos, with 300 carefully constructed prompts spanning five scenario categories. We develop an automated evaluation pipeline with human alignments that separately assesses text fidelity through carrier-specific transcription and scene and motion requirements through a prompt-specific chain of query. Beyond evaluation, we introduce a Keyframe-Guided Agentic Framework in which a Director agent coordinates image and video generation with visual evaluation, guiding iterative refinement and candidate selection through visual feedback. Experiments on 11 state-of-the-art models reveal widespread difficulties in accurately rendering scene text, with the best-performing model recording an overall word error rate (WER) of 0.250. We further analyze text rendering failures to characterize the challenges faced by current video generation models. These findings highlight visual text rendering as a key challenge for video generation and demonstrate a practical path toward improvement. Code is available at https://github.com/hardenyu21/VTR-Bench.
Yu Huang, Jungang Li, Zhiyuan Wang +8
Department of Data Science, City University of Hong Kong · The Hong Kong University of Science and Technology (Guangzhou) · Westlake University +3
Reliable reward models are essential for text-to-video evaluation and alignment. However, the trade-off between evaluation accuracy and inference efficiency places high demands on the quality of training supervision. Existing approaches often rely on holistic judges with fixed rubrics or open-ended reasoning, leading to incomplete inspection, unfaithful justification, and entangled attribution. We introduce FIRM-Video, a unified checklist-driven data construction framework based on a check-before-score principle: construct dimension-specific checklists, verify each criterion against temporal visual evidence, and aggregate only verified decisions. For Instruction Following, FIRM-Video decomposes prompts into weighted atomic requirements; for World Coherence, it constructs prompt-calibrated, target-specific checks grounded in visible entities and actions; and for Perceptual Quality, it applies a generic taxonomy of visual defects. The verified criteria and scores are further transformed into natural-language analyses for end-to-end reward modeling. Subsequently, we construct FIRM-Video-90K with 88,044 dimension-specific instances from 29,348 videos, and introduce FIRM-Video-Bench with 750 point-wise human annotations across 250 videos. The Qwen3-VL-based FIRM-Video-8B achieves the best overall MAE on FIRM-Video-Bench while consistently delivering the highest VBench Total, Quality, and Semantic Scores in Best-of-8 sampling across three video generators.
Peiyuan Zhang, Xiangyu Zhao, Hongbo Liu +8
Shanghai Jiao Tong University · Tencent Youtu Lab · Tongji University