Organizations: College of Computing and Data Science, Nanyang Technological University · Department of Computer Science, South China Normal University, China · School of Computer Science, City University of Hong Kong
Existing AI-generated video quality assessment (AIGVQA) methods mainly focus on global perceptual realism and coarse text-video alignment, while overlooking a critical requirement in educational scenarios: concept correctness. In early mathematics education, subtle errors in numerical quantities, geometric relations, or spatial configurations may fundamentally alter the conveyed knowledge despite visually plausible generation. To address this problem, we introduce EduAVQABench, the first benchmark for concept-aware educational AIGV assessment, containing 1,130 videos generated by ten state-of-the-art T2V models together with over 310,650 fine-grained human annotations spanning perceptual quality and semantic alignment. Built upon this benchmark, we further propose EduVQA, a concept-aware AIGVQA framework equipped with a Structured 2D Mixture-of-Experts (S2D-MoE) architecture. By jointly modeling fine-grained concept assessment and overall quality prediction through shared experts and adaptive two-dimensional routing, EduVQA effectively captures subtle concept-level inconsistencies overlooked by conventional global scoring methods. Extensive experiments demonstrate that EduVQA consistently outperforms existing AIGVQA approaches across both perceptual and semantic evaluation tasks while exhibiting strong generalization capability on unseen benchmarks. Code and dataset will be publicly available at: https://github.com/EduVQA/EduVQA.
Figures & tables
Fig. 1: Annotation structure of our constructed EduAVQABench dataset. Each educational video is annotated with spatial and temporal fidelity and word-level semantic consistency, enabling fine-grained assessment of perceptual quality and prompt alignment. The red and blue elliptical regions indicate temporal inconsistencies that negatively impact temporal quality.
Fig. 2: An overview of our dataset, divided into four categories: Numbers , Geometry , Measurement , and Probability .
Metric
Avg. SRCC
Avg. PLCC
#Annotators (SRCC > 0.8)
Perceptual Quality
0.810
0.803
12
Prompt Alignment
0.759
0.762
9
TABLE I: Annotation consistency analysis across evaluators.
Fig. 3: Annotation Analysis. (a)-(e): MOS distributions across five dimensions; (f): Average MOS of each scene category.
Fig. 4: Overview of EduVQA framework. We jointly predict five quality dimensions via a dual-path framework equipped with 2D MoE.
Metric
Setting
Method
SRCC ↑
PLCC ↑
KRCC ↑
RMSE ↓
Perceptual Quality
Zero-shot
ImageReward [ 43 ]
0.409
0.432
0.282
3.369
Q-Align [ 44 ]
0.644
0.669
–
–
Fine-tuned
IPCE [ 45 ]
0.822
0.822
0.631
0.494
IP-IQA [ 46 ]
0.852
0.863
0.666
0.434
DOVER [ 47 ]
0.832
0.840
0.645
0.479
BVQA [ 48 ]
0.848
0.862
–
–
TABLE II: Comparison on Perceptual Quality and Prompt Alignment of EduAVQABench. ↑ : higher is better, ↓ : lower is better.
Method
LGVQ [ 54 ] Dataset
EvalCrafter [ 27 ] Dataset
Spatial
Temporal
Alignment
Perceptual
Alignment
SRCC / PLCC
SRCC / PLCC
SRCC / PLCC
SRCC / PLCC
SRCC / PLCC
BVQA [ 48 ]
0.518 / 0.608
0.235 / 0.550
– / –
0.161 / 0.172
– / –
CLIPVQA [ 49 ]
0.509 / 0.480
0.355 / 0.287
– / –
0.319 / 0.325
– / –
FasterVQA [ 42 ]
0.511 / 0.539
0.380 / 0.427
– / –
0.378 / 0.359
– / –
GSTVQA [ 50 ]
0.507 / 0.555
0.459 / 0.529
– / –
0.276 / 0.281
– / –
TABLE III: Cross-dataset performance comparison. All models are trained on our EduAVQABench dataset.
Train Dataset
Test Dataset
Perceptual Quality
Prompt Alignment
SRCC
PLCC
SRCC
PLCC
EduAVQABench (Ours)
LGVQ [ 54 ]
0.5558
0.6346
0.5745
0.5941
AIGVE-Bench [ 55 ]
LGVQ [ 54 ]
0.2111
0.1986
0.0759
0.0777
TVGE [ 56 ]
LGVQ [ 54 ]
0.1488
0.1688
0.4950
0.5223
EduAVQABench (Ours)
TVGE [ 56 ]
0.3950
0.3866
0.5580
0.5343
AIGVE-Bench [ 55 ]
TVGE [ 56 ]
0.2532
0.2630
0.1772
0.1685
TABLE IV: Cross-dataset training analysis using different training datasets. All models are evaluated on the same target benchmark without fine-tuning.
Fig. 5: Comparison between predicted scores and MOS across multiple quality dimensions and quality levels.
Fig. 6: Qualitative comparison of perceptual quality (top row) and prompt alignment (bottom row). We compare our EduVQA model against state-of-the-art baselines, IP-IQA and T2VQA, in each quality dimensions. In each video pair, the right video exhibits superior perceptual quality or prompt alignment compared to the left. EduVQA consistently aligns with human judgments, while IP-IQA and T2VQA produce rankings contrary to the MOS.
Fig. 7: gMAD competition results between T2VQA and our EduVQA. Top: perceptual quality comparison; Bottom: prompt alignment comparison. zoomed-in view for clarity.
ID
Fusion
MoE
Sub-dim.
Perception
Alignment
Vanilla
Per.
Aln.
ST
WL
PLCC
SRCC
PLCC
SRCC
1
✗
✗
✗
✗
✗
✗
0.845
0.836
0.744
0.737
2
✓
✗
✗
✗
✗
✗
0.863
0.851
0.756
0.750
3
✓
✗
✗
✗
✓
✓
0.869
0.859
0.760
0.751
4
✓
✗
✗
✓
✗
✓
0.860
0.849
0.762
0.754
5
✓
✗
✓
✗
✓
✗
0.873
0.862
0.747
0.741
TABLE V: Ablation results on cross-modality fusion, MoE configurations, and fine-grained sub-dimension supervision. “ST” and “WL” denote spatial–temporal perceptual and word-level alignment branches, respectively.
Num. Experts
Top-K
Perceptual Quality
Prompt Alignment
SRCC
PLCC
SRCC
PLCC
4
2
0.845
0.858
0.751
0.759
8
1
0.856
0.869
0.746
0.753
8
2
0.869
0.879
0.757
0.768
8
4
0.850
0.863
0.737
0.745
8
8
0.853
0.864
0.756
0.761
TABLE VI: Impact of Expert Number and Top-K Selection in MoE.