cs.CVOct 8, 2025
SaveA Closer Look at Evaluation of Text-to-Video Models Under Camera Motion
Organizations: Indian Institute of Science · Adobe Research
Abstract
Existing text-to-video (T2V) evaluation benchmarks, such as VBench and EvalCrafter, predominantly contain either subject-centric prompts or static camera scenes, overlooking the importance of camera motion, essential for producing cinematic shots with realistic scenes. Due to this, the metrics developed in these works often produce incorrect results in the presence of camera motion. To fill this gap, we introduce DynamicEval, a benchmark comprising of systematically curated prompts that emphasize dynamic camera motion, paired with 45k human annotations on video pairs. DynamicEval evaluates two key dimensions of video quality: the consistency of the foreground subjects and the consistency of the background scene. To address the shortcomings of existing metrics under camera motion, we introduce two novel metrics: 1) Tracker-FG, which evaluates subject consistency by assessing the keypoint tracks of the subject. 2) Debiased Motion Smoothness, for background consistency, devoid from issues of occlusions/disocclusions arising from camera and foreground subject movements. Extensive experiments on the large-scale DynamicEval benchmark demonstrate that our proposed metrics achieve stronger correlations with human preferences at both the video level, and the model level with an increase of 0.32 in average Pearson Linear Correlation.
Figures & tables
Figure 1: Existing metrics (e.g., VB-Subject Consistency) favor low-motion videos even when they contain subject distortions. Our motion-invariant Tracker-FG metric reliably evaluates foreground quality regardless of camera or object motion. On our large-scale DynamicEval benchmark, prior metrics show substantial performance drops (top right), while our proposed metrics (Tracker-FG and MS-Debias) achieve significantly higher Pearson’s Linear Correlation Coefficient (PLCC) with model-level human preferences (bottom right).
Figure 2: Prompt curation: Scene elements from databases (orange) are sampled into a metadata (JSON format), which GPT-4o converts into descriptive prompts. Dataset: Video pairs generated from a common prompt are annotated via a subjective study.
Figure 3: Dataset analysis: The plot on the left shows the average win rates for both evaluation dimensions. The plot on right illustrates the percentage of video samples in each model that are static and dynamic.
Figure 4: Tracker Motivation: Deviation of neighbor tracks on real video vs generated video with subject distortions. Blue plot: Distance between a nearest neighbor and a candidate point. Red plot: Moving average of blue. Green plot: the deviation between blue and red.
Figure 5: Tracker-FG framework: We track the points inside an object mask across the video. TrackerFG is obtained by computing the average of distances between neighboring tracking points.
Figure 6: Motion Smoothness error maps: The zoomed-in regions show localized distortions visible across frames. VB-MS shows large errors near edges and foreground objects, suppressing the localized distortions. After debiasing, the localized distortions are visible.
Figure 7: MS-Debias: obtains debiased motion smoothness error maps by masking out foreground subjects and occlusions. It is applied at multiple scales with Gaussian pyramid downsampling, for improving the robustness of the metric.
| Subject Consistency (Pairwise Acc.) Method Full Dataset Full Agreement VB-SC 56.2 58.8 Tracker-FG (Ours) 58.2 62.7 Background Consistency (Pairwise Acc.) VB-BG 56.0 59.3 VB-MS 53.7 56.8 MS-Debias (Ours) 58.2 62.7 | Subject Consistency (Top-k) Method Top-1 Top-2 Top-3 Top-4 Top-5 VB-SC 16.2 29.1 39.5 50.0 59.3 Tracker-FG (Ours) 19.7 31.4 41.9 55.8 62.8 Background Consistency (Top-k) Method Top-1 Top-2 Top-3 Top-4 Top-5 VB-BG 10.1 31.5 41.6 55.1 65.2 MS-Debias (Ours) 14.6 34.8 44.9 57.3 69.7 |
Table 2: Performance Comparisons: The first table shows pairwise preference accuracy on the DynamicEval database, and the second table evaluates the Top-k video selection accuracy. On the full dataset, our method, MS-Debias outperforms the VB-BG by % points and on the full agreement subset by % points. Similar boosts can be seen for subject consistency pairwise accuracy.
| Subject Consistency | Background Consistency | ||||
|---|---|---|---|---|---|
| Method | static-dynamic | dynamic-dynamic | Method | static-dynamic | dynamic-dynamic |
| VB-SC | 57.1 | 57.3 | VB-BG | 55.2 | 56.1 |
| Tracker-FG | 60.3 | 58.1 | MS-Debias | 58.0 | 56.7 |
Table 3: Video-Pair Configurations: Performance of each metric when the pair contains different configurations of static-dynamic pairs. The improved metrics ensure no bias, for static videos over the dynamic ones, in the proposed Tracker-FG and MS-Debias metrics.
| Method (FG) | PLCC | Method (BG) | PLCC |
|---|---|---|---|
| VB-SC | 0.334 | VB-BG | 0.551 |
| Tracker-FG | 0.772 | MS-Debias | 0.743 |
Table 4: PLCC of model-level win ratios between the metrics and human preference. We evaluate these on the full agreement subset.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Scene meta-data | Generated prompt |
| { "setting": "indoor", "action_dataset": "comclip", "metadata": { "scene": "auto factory", "subject": { "name": "dog", "number_of_subjects": "one", "action": "drinking the water" }, "camera": { "type": "ground shot", "movement": "dolly shot" }, "extra attibutes": "" }, } | ”In an auto factory, a lone dog drinks water from a puddle amidst hulking machinery and assembly lines. The ground shot dolly camera moves forward, revealing industrial surroundings with the dog in sharp focus.” |
Table 6: Structured metadata (left) and its descriptive narrative (right).
Figure 8: A screenshot of the User Interface for pairwise comparison
Figure 9: Dataset analysis: Percentage of categories of scene elements in the prompts suite, and the prompt wordcloud. A majority of the words focus towards camera movement.
Figure 10: Effect of camera motion on feature-level metrics: We observe that with large camera motions in the scene, the feature level metrics get affected despite having high quality video.
Figure 11: Comparison of ratio static videos in DynamicEval (DE) vs VBench (VB) dataset on matching models. The ratio of static scenes generated from VBench prompts are significantly higher than the ones generated from DynamicEval.
Figure 12: Multi-scale processing in MS-Debias: Different down-sampled versions of the input video are processed independently through the MS-Debias pipeline to obtain background consistency scores at multiple levels. The scores are then averaged using weights proportional to the scales to obtain the final metric score. This mimics the multi-scale processing capabilities of the human visual system.
Figure 13: Baseline metrics (VB-SC and VB-BG) exhibit a strong bias with respect to camera motion. As camera motion increases, their scores decrease even for high-quality videos, which is consistent with the visualizations in Fig. 10 . In foreground subject consistency, even though video V1 is of high quality and V2 shows a cat morphing (distortion), VB-SC gives a lower score to V1 because it has a larger camera motion. Similarly, in background consistency, VB-BG gives a low score to V1 , which has more camera motion even though V2 exhibits significant localized distortions. Our metrics are independent of camera motion bias.
Table 9: Statistical significance tests ( confidence level) across all pairs of foreground subject consistency metrics. Table (a) shows the and values of McNemar’s statistical test that tests whether the pairwise accuracy differences are significant. A value of for a pair of methods means that the accuracy differences are statistically significant. Table (c) shows the low-high bounds of bootstrap percentile confidence interval of accuracy differences. The accuracy differences are statistically significant if the interval does not contain . Table (b) and (d) show the statistical significance matrix, in which, ‘1’: metric in the row has statistically significant performance gains over the metric mentioned in the corresponding column; ‘0’: row metric has statistically significantly poorer performance; ‘-’: the accuracy difference is not statistically significant.
Table 10: Statistical significance tests ( confidence level) across all pairs of background consistency metrics. Details are already discussed in Table 9 .
| Background Consistency | ||
| Comparison Type | VB-BG | MS-Debias |
| Inter-model Comparisons | 54.8 | 58.3 |
| Intra-model Comparisons | 57.7 | 58.2 |
| Subject Consistency | ||
| VB-SC | Tracker-FG | |
| Inter-model Comparisons | 54.3 | 56.5 |
Table 13: Performance of the methods on inter-model vs intra-model video comparisons. The proposed metrics perform better on both comparisons, with higher performance on intra-model pairs, as the videos contain similar content.
More in this topic
Video Generation Evaluation
Video generation evaluation measures visual quality, prompt adherence, temporal consistency, physical plausibility, and task-specific behavior in synthesized videos.
See all 191 papers