FVScore: Specification-Driven Neuro-Symbolic Video Evaluation
Organizations: Google Cloud AI Research · National University of Singapore
Abstract
We introduce Faithfulness Video Score (FVSCORE), a neuro-symbolic framework for specification-driven video generation evaluation. Existing evaluators rely on holistic MLLM judgments or vision-language metrics, yet aligning their scores with human judgments and tracing them to specific temporal violations remain open challenges. FVSCORE addresses these limitations by compiling each video into explicit propositions capturing prompt requirements and implicit propositions capturing unstated physical, temporal, and cross-modal constraints. It grounds each proposition using specialized perceptual tools and evaluates the resulting confidence traces with soft temporal operators inspired by Linear Temporal Logic. A Significance-Aware Calibration step then aligns verification with human perception by retaining only perceptually significant violations before producing a timestamp- and evidence-grounded report. We conduct extensive experiments across public and complex multi-event benchmarks with the strongest video models to date, demonstrating that FVSCORE outperforms holistic and agent-based evaluators in human correlation while providing highly accurate and interpretable feedback.
Figures & tables
| Text-Video Align. | Consistency | Commonsense | Video-Audio Sync. | |||||
| Method | P | S | P | S | P | S | P | S |
| VBench ( Huang et al., 2024 ) | 0.112 | 0.181 | 0.149 | 0.176 | ||||
| NeuS-V ( Sharan et al., 2025 ) | 0.263 | 0.375 | ||||||
| MLLM-Judge w/ SC ( Wang et al., 2023 ) | 0.360 | 0.390 | 0.182 | 0.174 | 0.240 | 0.234 | -0.110 | -0.133 |
| Self-Refine ( Madaan et al., 2023 ) | 0.333 | 0.378 | 0.230 | 0.261 | 0.168 | 0.154 | 0.189 | 0.196 |
| Multi-Agent Debate ( Du et al., 2024 ) | 0.370 | 0.410 | 0.237 | 0.249 | 0.135 | 0.082 | 0.167 | 0.116 |
| EvalCrafter | VideoFeedback | VideoFeedback2 | ||||||||
| ( Liu et al., 2024 ) | ( He et al., 2024 ) | ( He et al., 2025 ) | ||||||||
| Text-Video Align. | Consistency | Text-Video Align. | Consistency | Text-Video Align. | ||||||
| Method | P | S | P | S | P | S | P | S | P | S |
| VBench ( Huang et al., 2024 ) | -0.162 | -0.150 | 0.479 | 0.533 | -0.112 | -0.079 | 0.421 | 0.502 | -0.138 | -0.120 |
| NeuS-V ( Sharan et al., 2025 ) | 0.217 | 0.210 | 0.151 | 0.123 | 0.164 | 0.157 | ||||
| MLLM-Judge w/ SC ( Wang et al., 2023 ) | 0.691 | 0.699 | 0.571 | 0.585 | 0.428 | 0.433 | 0.427 | 0.432 | 0.615 | 0.629 |
| Method | Accuracy | Groundedness | Completeness | Helpfulness | Avg. | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Win | Tie | Loss | Win | Tie | Loss | Win | Tie | Loss | Win | Tie | Loss | ||||||
| MLLM-Judge w/ SC ( Wang et al., 2023 ) | 65.3 | 17.4 | 17.3 | +48.0 | 75.5 | 11.2 | 13.3 | +62.2 | 73.3 | 12.7 | 14.0 | +59.3 | 69.8 | 14.4 | 15.8 | +54.0 | +55.9 |
| Self-Refine ( Madaan et al., 2023 ) | 52.6 | 21.2 | 26.2 | +26.4 | 56.3 | 17.1 | 26.6 | +29.7 | 53.1 | 20.3 | 26.6 | +26.5 | 51.6 | 19.9 | 28.5 | +23.1 | +26.4 |
| Multi-Agent Debate ( Du et al., 2024 ) | 56.3 | 19.8 | 23.9 | +32.4 | 62.4 | 16.8 | 20.8 | +41.6 | 62.3 | 15.7 | 22.0 | +40.3 | 59.5 | 18.3 | 22.2 | +37.3 | +37.9 |
| VISTA ( Long et al., 2026b ) | 48.0 | 24.9 | 27.1 | +20.9 | 54.5 | 19.6 | 25.9 | +28.6 | 53.2 | 22.5 | 24.3 | +28.9 | 49.6 | 24.9 | 25.5 | +24.1 | +25.6 |
| VQQA ( Song et al., 2026 ) | 58.0 | 20.0 | 22.0 | +36.0 | 60.4 | 21.2 | 18.4 | +42.0 | 65.9 | 17.3 | 16.8 | +49.1 | 55.9 | 21.7 | 22.4 | +33.5 | +40.2 |
| FVBench | VideoFeedback ( He et al., 2024 ) | |||||
| Text-Video Align. | Consist. | Commonsense | Video-Audio Sync. | Text-Video Align. | Consist. | |
| FVScore w/o PAC (Ours) | 0.550 / 0.533 | 0.364 / 0.293 | 0.326 / 0.344 | 0.657 / 0.734 | 0.558 / 0.478 | 0.464 / 0.508 |
| FVScore (Neural-only) | 0.441 / 0.437 | 0.305 / 0.286 | 0.312 / 0.327 | 0.588 / 0.640 | 0.503 / 0.525 | 0.450 / 0.484 |
| FVScore (Graph-only) | 0.085 / 0.085 | 0.149 / 0.236 | 0.088 / 0.066 | N/A | 0.329 / 0.324 | 0.269 / 0.247 |
| FVScore (w/ Original LTL) | 0.091 / 0.142 | 0.113 / 0.140 | -0.089 / -0.105 | 0.368 / 0.415 | 0.204 / 0.237 | |
| FVScore (w/o Temporal Logic) | 0.183 / 0.197 | 0.148 / 0.159 | 0.535 / 0.530 | |||
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| FVBench | VideoFeedback ( He et al., 2024 ) | ||||||
|---|---|---|---|---|---|---|---|
| Backbone MLLM | PAC | Text-Video Align. | Consist. | Commonsense | Video-Audio Sync. | Text-Video Align. | Consist. |
| Gemini 3 Flash | No | 0.550 / 0.533 | 0.364 / 0.293 | 0.326 / 0.344 | 0.657 / 0.734 | 0.558 / 0.478 | 0.464 / 0.508 |
| Gemini 3 Flash | Yes | 0.579 / 0.566 | 0.441 / 0.452 | 0.558 / 0.528 | 0.678 / 0.752 | 0.564 / 0.564 | 0.502 / 0.528 |
| Qwen3.8-Omni | No | 0.434 / 0.492 | 0.258 / 0.307 | 0.282 / 0.297 | 0.378 / 0.415 | 0.503 / 0.523 | 0.446 / 0.498 |
| Qwen3.8-Omni | Yes | 0.519 / 0.508 | 0.341 / 0.364 | 0.415 / 0.446 | 0.462 / 0.480 | 0.532 / 0.509 | 0.542 / 0.610 |
| Qwen3.5-9B | No | 0.387 / 0.430 | 0.242 / 0.294 | 0.191 / 0.169 | 0.509 / 0.506 | 0.473 / 0.500 | |
| Judge | Acc. | Ground. | Compl. | Help. | Avg. |
|---|---|---|---|---|---|
| Human Agreement (Leave-One-Out) | 73.1% | 80.5% | 87.7% | 78.6% | 80.0% |
| Gemini 3.1 Pro ( Google, 2026 ) | 74.2% | 80.8% | 80.8% | 80.0% | 79.0% |
| Minimax-M3 ( Lai et al., 2026 ) | 65.8% | 76.7% | 78.3% | 76.7% | 74.4% |
| Qwen 3.8 Max (0902) ( Qwen Team, 2026 ) | 69.7% | 72.1% | 75.4% | 76.3% | 73.4% |
| Method | Average tokens | Average tokens (normalized) |
|---|---|---|
| MLLM Judge | 210.38 | 431.18 |
| VISTA ( Long et al., 2026b ) | 324.18 | 428.43 |
| Multi-Agent Debate ( Du et al., 2024 ) | 183.21 | 381.21 |
| Self-Refine ( Madaan et al., 2023 ) | 248.20 | 461.76 |
| VQQA ( Song et al., 2026 ) | 123.11 | 344.90 |
| FVScore (Ours) | 459.91 | 459.91 |
| Method | Accuracy | Groundedness | Completeness | Helpfulness | Avg. | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Win | Tie | Loss | Win | Tie | Loss | Win | Tie | Loss | Win | Tie | Loss | ||||||
| MLLM-Judge | 60.5 | 21.2 | 18.3 | +42.2 | 70.6 | 15.4 | 14.0 | +56.6 | 70.6 | 14.3 | 15.1 | +55.5 | 59.8 | 25.1 | 15.1 | +44.7 | +49.8 |
| Self-Refine | 54.8 | 23.7 | 21.5 | +33.3 | 63.4 | 20.5 | 16.1 | +47.3 | 61.3 | 19.3 | 19.4 | +41.9 | 54.8 | 24.8 | 20.4 | +34.4 | +39.2 |
| Multi-Agent Debate | 53.8 | 23.6 | 22.6 | +31.2 | 65.6 | 19.3 | 15.1 | +50.5 | 67.7 | 17.2 | 15.1 | +52.6 | 55.9 | 24.7 | 19.4 | +36.5 | +42.7 |
| VISTA | 48.5 | 20.2 | 31.3 | +17.2 | 55.6 | 15.6 | 28.8 | +26.8 | 60.8 | 18.2 | 21.0 | +39.8 | 50.5 | 30.1 | 19.4 | +31.1 | +28.7 |
| Avg. | +31.0 | +45.3 | +47.5 | +36.7 | +40.1 | ||||||||||||
Video Generation Evaluation
Video generation evaluation measures visual quality, prompt adherence, temporal consistency, physical plausibility, and task-specific behavior in synthesized videos.
See all 205 papers