Existing text-to-video retrieval benchmarks often permit matching from static appearance, making it difficult to determine whether models capture state changes and final outcomes. We introduce GenState-AI, a benchmark comprising 956 AI-generated videos and 195 verified state-change queries across Toy, Cartoon, Home, and Manipulation domains. Its construction uses prompt-controlled variants that target either conflicting temporal/state outcomes or semantic substitutions. Following human verification, all valid videos contribute to subset-level retrieval pools, while complete query-specific triplets support diagnostic hard-negative ordering. We evaluate VCF-Lik, Qwen3-VL, and LanguageBind using both standard Recall@K and triplet-based consistency metrics. Qwen3-VL (8B) achieves the best macro-average R@1 and triplet consistency, while LanguageBind obtains the highest semantic discrimination accuracy. However, stronger retrieval does not consistently yield correct hard-negative ordering, and Qwen3-VL and LanguageBind remain weak on Manipulation triplets, where VCF-Lik performs substantially better. GenState-AI therefore provides a controlled diagnostic testbed for distinguishing global retrieval quality from fine-grained temporal and semantic grounding in AI-generated videos.
Figures & tables
Figure 1: Overview of the GenState-AI construction and evaluation pipeline, including prompt-triplet construction, AI-based video generation, human verification and filtering, and retrieval evaluation.
Category
Number of Queries
Avg. Tokens
Home
127
20.1
Toy
21
26.8
Cartoon
24
30.7
Manipulation
23
14.7
Total
195
21.5
Table 1: Number of query descriptions per video category in our GenState-AI dataset, along with the average token length, computed using the Qwen2.5-1.5B tokenizer.
Category
#Videos
Resolution
FPS
Duration (s)
Size
Home
368
1280×704
24
5
2.53 GB
Toy
221
1280×704
24
5
1.01 GB
Cartoon
150
1280×704
24
5
522 MB
Manipulation
217
1280×704
24
5
1.92 GB
Total
956
5.98 GB
Table 2: Statistics of our video dataset. Each video has a resolution of 1280×704 (720p) at 24 FPS and lasts about 5 seconds (121 frames).
Subset
Baseline
R@1
R@5
R@10
Acc temp
Acc sem
Acc tri
Toy
VCF-Lik
0.381
0.714
0.857
0.619
0.810
0.524
Qwen3-VL (2B)
0.571
0.952
0.952
0.667
0.905
0.667
Qwen3-VL (8B)
0.667
0.952
1.000
0.762
0.905
0.714
LanguageBind
0.571
0.905
0.952
0.619
0.857
0.619
Cartoon
VCF-Lik
0.250
0.708
0.708
0.500
0.833
0.417
Qwen3-VL (2B)
0.708
1.000
1.000
0.792
0.875
0.750
Table 3: Performance comparison of the baselines across different subsets on GenState-AI. The Avg. row reports the macro average over the four subsets. Best results are highlighted in bold . Recall@K uses all videos in each corresponding subset pool and retains all 195 queries; the three triplet accuracies use the 154 complete triplets and exclude 41 incomplete groups.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Subset
Model
Context
R@1
R@5
R@10
Acc temp
Acc sem
Acc tri
Cartoon
Qwen3-VL (2B)
Full
0.708
1.000
1.000
0.792
0.875
0.750
Qwen3-VL (2B)
Last
0.625
0.833
0.917
0.708
0.917
0.667
Qwen3-VL (8B)
Full
0.667
1.000
1.000
0.792
0.958
0.750
Qwen3-VL (8B)
Last
0.625
0.958
1.000
0.667
0.958
0.667
LanguageBind
Full
0.833
0.958
1.000
0.833
1.000
0.833
LanguageBind
Last
0.750
0.958
0.958
0.750
0.917
0.750
Appendix
Table 4: Full subset-level ablation comparing full-video context (5 frames) against static last-frame context. Recall@K uses all videos in each corresponding subset pool and retains all 195 queries, whereas the triplet accuracies use the 154 complete query-specific triplets; 41 incomplete query groups are excluded.
Subset
Frames
R@1
R@5
R@10
Acc temp
Acc sem
Acc tri
Toy
5
0.571
0.905
0.952
0.619
0.857
0.619
Toy
8
0.571
0.905
0.952
0.619
0.857
0.619
Cartoon
5
0.833
0.958
1.000
0.833
1.000
0.833
Cartoon
8
0.750
1.000
1.000
0.792
1.000
0.792
Home
5
0.441
0.890
0.969
0.716
0.989
0.716
Home
8
0.504
0.921
0.984
0.750
0.989
0.750
Appendix
Table 5: LanguageBind frame-sampling sensitivity. The 5-frame setting is used in the main tables for protocol consistency. Recall@K uses each corresponding complete subset pool; triplet accuracies use the 154 complete triplets.
Subset pool
956-video pool
Subset
R@1
R@5
R@10
R@1
R@5
R@10
Toy
0.381
0.714
0.857
0.381
0.714
0.857
Cartoon
0.250
0.708
0.708
0.208
0.417
0.625
Home
0.386
0.512
0.591
0.386
0.512
0.591
Manip.
0.478
0.870
0.957
0.478
0.870
0.957
Appendix
Table 6: VCF-Lik Recall using the Table 3 subset pools versus all 956 videos.
Figure 3: Comparison of relative-order distributions across subsets under LanguageBind . Counts and percentages are computed from the complete triplets available for each subset (154 in total).
Figure 4: Representative triplet-ordering failures under VCF-Lik ; higher-ranked negatives appear above the main video. In Toy , Rank 1 is a temporal negative showing the same rubber duck moving from left to right beside a gift box, while the Rank 2 target shows the duck moving from right to left; Rank 3 is a semantic negative featuring a jumping cartoon owl. Thus, the same-object clip with opposite motion outranks the target. In Home , Rank 1 is a semantic negative showing a girl moving a potted plant to a windowsill, Rank 2 is a temporal negative showing her lifting and laying down a blue blanket, and the Rank 3 target shows her picking up a book. This ordering illustrates difficulty distinguishing the target action from hard negatives in a similar visual context.