cs.CVMar 20, 2026
SaveMME-CoF-Pro: Evaluating Reasoning Coherence in Video Generative Models with Text and Visual Hints
Organizations: Northeastern University · The Chinese University of Hong Kong · Westlake University · ByteDance Seed ·
Abstract
Video generative models show emerging reasoning behaviors. It is essential to ensure that generated events remain causally consistent across frames for reliable deployment, a property we define as reasoning coherence. To bridge the gap in literature for missing reasoning coherence evaluation, we propose MME-CoF-Pro, a comprehensive video reasoning benchmark to assess reasoning coherence in video models. Specifically, MME-CoF-Pro contains 303 samples across 16 categories, ranging from visual logical to scientific reasoning. It introduces Reasoning Score as evaluation metric for assessing process-level necessary intermediate reasoning steps, and includes three evaluation settings, (a) no hint (b) text hint and (c) visual hint, enabling a controlled investigation into the underlying mechanisms of reasoning hint guidance. Evaluation results in 7 open and closed-source video models reveals insights including: (1) Video generative models exhibit weak reasoning coherence, decoupled from generation quality. (2) Text hints boost apparent correctness but often cause inconsistency and hallucinated reasoning (3) Visual hints benefit structured perceptual tasks but struggle with fine-grained perception. Website: https://video-reasoning-coherence.github.io/
Figures & tables
Figure 1: MME-CoF-Pro evaluates video model capacity to model world behaviour. This evaluation is shown under the numbered multi steps section (shown in darker green), this matches with “reasoning steps as rubric” section (human generated to match expected behaviour). Using different prompting mechanisms, our benchmark evaluates video model behaviour at a fine grain. Red box indicates input image.
Figure 2: MME-CoF-Pro consists of 303 samples from 16 basic reasoning categories, covering from Visual Detail Reasoning to Visual Trace Reasoning . Each sample is provided with no-hint and text-hint evaluation prompt. In text-hint setting, models are provided with additional textual details to facilitate reasoning.
Figure 3: Visual Hints in MME-CoF-Pro . For controlled comparison, we additionally add visual hints to 8 categories which requires visual reasoning capabilities, using bounding boxes and arrows to indicate positions, trajectories to facilitate reasoning.
Figure 4: Statistics, Distribution and Evaluation Results of MME-CoF-Pro.
Figure 5: Setting Comparisons. All three settings share the same evaluation metrics and identical instructions except for the hint components.
Figure 6: Process-level Evaluation Metric for Reasoning Score in MME-CoF-Pro. Videos are evaluated against fine-grained reasoning steps, enabling interpretable assessment beyond final-answer accuracy.
| Category | Metric | Closed-Source Models | Open-Source Models | |||||
|---|---|---|---|---|---|---|---|---|
| Veo3.1 [ 69 ] | Veo3.1-fast [ 69 ] | Sora2 [ 53 ] | Seedance 1.0-pro [ 17 ] | Seedance 1.0-fast [ 17 ] | Kling-v2.1 [ 57 ] | Cosmos Predict [ 1 ] | ||
| Visual Detail Reasoning | RS | 61.5( +18.6 ) | 65.7( +6.0 ) | 40.3( +27.8 ) | 19.8( +29.3 ) | 26.1( +11.7 ) | 2.0( -2.0 ) | 30.2( +8.1 ) |
| CS | 51.2(0.0) | 47.6( -1.8 ) | 22.8( +13.3 ) | 50.0( +7.6 ) | 55.6( -13.3 ) | 64.7( -16.6 ) | 36.1( -10.6 ) | |
| Avg | 56.8( -1.1 ) | 56.7( -3.2 ) | 30.7( +13.7 ) | 54.8( +6.6 ) | 58.1( -5.3 ) | 72.3( -7.3 ) | 37.9( -8.6 ) | |
| Embodied Reasoning | RS | 81.7( -0.2 ) | 78.5( +0.8 ) | 38.4( -3.9 ) | 48.7( +6.8 ) | 57.0( -6.4 ) | 2.5( -1.4 ) | 49.4( +20.1 ) |
| CS | 72.9( +1.1 ) | 62.5( +8.2 ) | 61.8( -1.1 ) | 56.9( -3.3 ) | 52.8( +0.1 ) | 44.1( -3.8 ) | 58.6( -6.6 ) | |
Table 1: Evaluation results of MME-CoF-Pro. We report the percentage of Reasoning Score (RS) and Consistency Score (CS) in no-hint settings , with text-hint performance shown in parentheses. Color indicates the value of change brought by text-hint : improvement and degradation . We additionally evaluate Hallucination Score , Visual Stability , and Physics Grounding , and present the overall average score with Consistency Score here. Details are provided in the Appendix.
Figure 7: Failure Examples of Video Generative Models. Subfigure (a) shows that the Kling model prioritizes perceptual fidelity (e.g., wind and forest dynamics) over the intended reasoning trajectory of gradual zoom-in. Subfigures (b) to (d) reveal hallucinations and inconsistencies arising under different hint conditions.
Figure 8: Human study spearman correlation map.
| Category | Metric | Closed-Source Models | Open-Source Models | |||||
|---|---|---|---|---|---|---|---|---|
| Veo3.1 [ 69 ] | Veo3.1-fast [ 69 ] | Sora2 [ 53 ] | Seedance 1.0-pro [ 17 ] | Seedance 1.0-fast [ 17 ] | Kling-v2.1 [ 57 ] | Cosmos Predict [ 1 ] | ||
| 4D Dynamics Reasoning | RS | 51.7( -0.4 ) | 70.5( +8.5 ) | 62.0( -3.9 ) | 67.4( -25.8 ) | 51.7( -6.6 ) | 9.0( -5.6 ) | 50.8( -7.0 ) |
| CS | 63.3( -15.6 ) | 63.3( +2.2 ) | 74.4( -12.2 ) | 53.3( -4.4 ) | 63.3( -21.1 ) | 60.0( +3.3 ) | 56.7( -2.2 ) | |
| Avg | 62.4( -10.9 ) | 62.2( +5.1 ) | 71.6( -17.1 ) | 55.3( -5.8 ) | 62.4( -23.8 ) | 61.8( -1.3 ) | 52.0( +0.7 ) | |
| Embodied Reasoning | RS | 60.8( +22.9 ) | 91.2( -17.5 ) | 68.8( -8.3 ) | 50.4( +8.3 ) | 63.8( +12.4 ) | 9.2( -2.9 ) | 22.4( +20.5 ) |
| CS | 60.0( +17.5 ) | 80.0( -3.8 ) | 70.0( +5.0 ) | 51.2( +8.8 ) | 57.1( +17.1 ) | 23.8( +2.5 ) | 55.7( +12.9 ) | |
Table 2: Evaluation results (visual-hint). We report Reasoning Score (RS) and Consistency Score (CS) in no-hint settings for items that have visual-hint, with visual-hint change in parentheses. Color: improvement , degradation . Only categories with visual-hint data are shown.
Figure 9: Scaling case study on Frozen Lake.
| Category | Benchmark |
|---|---|
| Visual Detail Reasoning | V* Bench [ 27 ] |
| 3D Geometry Reasoning | STARE [ 36 ] |
| SpatialViz-Bench [ 65 ] | |
| Real-world Spatial Reasoning | MMSI-Bench [ 73 ] |
| OmniSpatial [ 28 ] | |
| Visual Trace Reasoning | MVoT [ 32 ] |
Table 3: Overview of MME-CoF-Pro by Reasoning Category
| Model | Resolution | Duration | FPS |
|---|---|---|---|
| Veo-3.1 [ 69 ] | 1280 720 | 8s | 24 |
| Veo-3.1-fast [ 69 ] | 1280 720 | 8s | 24 |
| Sora-2 [ 53 ] | 1280 720 | 4s | 30 |
| Seedance-1.0-pro [ 17 ] | 1920 1280 | 5s | 24 |
| Seedance-1.0-fast [ 17 ] | 1280 720 | 5s | 24 |
| Kling-v2.1 [ 57 ] | 1280 720 | 5s | 24 |
Table 4: Default inference configurations for evaluated video generation models. When configurable, we follow the recommended settings from the official implementations or APIs.
| Category | Metric | Closed-Source Models | Open-Source Models | |||||
|---|---|---|---|---|---|---|---|---|
| Veo3.1 [ 69 ] | Veo3.1-fast [ 69 ] | Sora2 [ 53 ] | Seedance 1.0-pro [ 17 ] | Seedance 1.0-fast [ 17 ] | Kling-v2.1 [ 57 ] | Cosmos Predict | ||
| Visual Detail Reasoning | TC | 64.8( -7.4 ) | 55.6( -2.0 ) | 22.8( +11.7 ) | 60.1( +9.2 ) | 64.8( -7.4 ) | 76.4( -6.2 ) | 37.7( -8.0 ) |
| VS | 61.7( -1.9 ) | 51.6( -2.0 ) | 36.4( +5.6 ) | 58.2( +7.2 ) | 61.7( -1.9 ) | 88.2( -17.4 ) | 43.8( -7.4 ) | |
| Hallu | 66.7( -4.9 ) | 74.5( -6.5 ) | 38.3( +30.9 ) | 66.0( +1.3 ) | 66.7( -4.9 ) | 88.2( +4.2 ) | 45.1( -9.3 ) | |
| PG | 67.9( -0.6 ) | 80.4( -5.2 ) | 47.5( +13.0 ) | 64.7( +10.5 ) | 67.9( -0.6 ) | 79.2( -4.9 ) | 43.8( -11.1 ) | |
| Embodied Reasoning | TC | 65.9( -4.8 ) | 68.3( +11.5 ) | 76.6( -8.7 ) | 66.7( -3.2 ) | 65.9( -4.8 ) | 72.0( -11.9 ) | 56.8( -1.7 ) |
Table 5: Evaluation results (text-hint): Temporal Consistency, Visual Stability, Hallucination, Physics Grounding. We report no-hint score (0-9 mean as %); parenthetical change is text-hint minus no-hint. improvement , degradation .
| Category | Metric | Closed-Source Models | Open-Source Models | |||||
|---|---|---|---|---|---|---|---|---|
| Veo3.1 [ 69 ] | Veo3.1-fast [ 69 ] | Sora2 [ 53 ] | Seedance 1.0-pro [ 17 ] | Seedance 1.0-fast [ 17 ] | Kling-v2.1 [ 57 ] | Cosmos Predict | ||
| 4D Dynamics Reasoning | TC | 67.9( -9.9 ) | 67.9( +2.5 ) | 74.1( -16.0 ) | 63.0( -8.6 ) | 67.9( -22.2 ) | 88.9( -23.5 ) | 51.9( +6.2 ) |
| VS | 77.8( -7.4 ) | 77.8( +4.9 ) | 87.7( -21.0 ) | 76.5( -4.9 ) | 77.8( -21.0 ) | 88.9( +11.1 ) | 69.1( +1.2 ) | |
| Hallu | 75.3( -12.3 ) | 72.8( +8.6 ) | 86.4( -22.2 ) | 64.2( -4.9 ) | 75.3( -37.0 ) | 98.8( +1.2 ) | 67.9( -3.7 ) | |
| PG | 55.6( -13.6 ) | 56.8( +9.9 ) | 66.7( -22.2 ) | 44.4( -8.6 ) | 55.6( -28.4 ) | 0.0(0.0) | 37.0( +2.5 ) | |
| Embodied Reasoning | TC | 69.4( +12.5 ) | 86.1( -9.7 ) | 70.8( +5.6 ) | 63.9( +4.2 ) | 68.3( +19.0 ) | 50.0( -20.8 ) | 60.3( +12.7 ) |
Table 6: Evaluation results (visual-hint): Temporal Consistency, Visual Stability, Hallucination, Physics Grounding. No-hint score (%); parenthetical change is visual-hint minus no-hint. improvement , degradation . Only categories with visual-hint data.
More in this topic
Video Reasoning
Video reasoning studies how AI systems use video evidence to infer events, actions, temporal relations, causes, and future states.
See all 210 papers