OSCBench: Benchmarking Object State Change in Text-to-Video Generation

Xianjing Han; Bin Zhu; Shiqi Hu; Franklin Mingzhe Li; Patrick Carrington; Roger Zimmermann; Jingjing Chen

cs.CVMar 12, 2026

OSCBench: Benchmarking Object State Change in Text-to-Video Generation

Authors: Xianjing Han, Bin Zhu, Shiqi Hu, Franklin Mingzhe Li, Patrick Carrington, Roger Zimmermann, Jingjing Chen

Organizations: National University of Singapore · Singapore Management University · Carnegie Mellon University · Fudan University

Abstract

Text-to-video (T2V) generation models have made rapid progress in producing visually high-quality and temporally coherent videos. However, existing benchmarks primarily focus on perceptual quality, text-video alignment, or physical plausibility, leaving a critical aspect of action understanding largely unexplored: object state change (OSC) explicitly specified in the text prompt. OSC refers to the transformation of an object's state induced by an action, such as peeling a potato or slicing a lemon. In this paper, we introduce OSCBench, a benchmark specifically designed to assess OSC performance in T2V models. OSCBench is constructed from instructional cooking data and systematically organizes action-object interactions into regular, novel, and compositional scenarios to probe both in-distribution performance and generalization. We evaluate six representative open-source and proprietary T2V models using both human user study and multimodal large language model (MLLM)-based automatic evaluation. Our results show that, despite strong performance on semantic and scene alignment, current T2V models consistently struggle with accurate and temporally consistent object state changes, especially in novel and compositional settings. These findings position OSC as a key bottleneck in text-to-video generation and establish OSCBench as a diagnostic benchmark for advancing state-aware video generation models.

Figures & tables

Figure 1: Overview of OSCBench evaluation. (a) Representative failure cases from regular, novel, and compositional object state change scenarios. In the regular case, the red box marks an implausible state change of the lemon during slicing. In the novel case, the model misinterprets the instructed action, resulting in a wrong object transformation. In the compositional case, the yellow box indicates an incomplete state change where the pear remains unpeeled. (b) Human-evaluated multi-dimensional performance of T2V models on OSCBench.

Figure 2: Overview of the OSCBench construction and evaluation pipeline. We build unified action and object categories from instructional cooking data via a human-in-the-loop process, and construct regular, novel, and compositional OSC scenarios as text prompts for video generation. The generated videos are evaluated by humans and MLLMs across multiple criteria, and we analyze their correlations to assess automatic evaluation reliability.

Open-source models
	Semantic Adherence			Object State Change		Scene Alignment	Perceptual Quality
Model	Subject	Object	Action	Accuracy	Consistency		Realism	Aesthetics
Open-Sora-2.0 ( Peng et al., 2025 )	0.860	0.734	0.518	0.380	0.428		0.740	0.416	0.540
HunyuanVideo ( Kong et al., 2024 )	0.868	0.826	0.494	0.402	0.510	0.834	0.526	0.688
HunyuanVideo-1.5 ( Team, 2025 )	0.914	0.902	0.656	0.524	0.608	0.876	0.618	0.730
Wan-2.2 ( Wan et al., 2025 )	0.904	0.842	0.616	0.560	0.668	0.894	0.702	0.818

Table 1: Human evaluation results of different T2V models across multiple evaluation dimensions.

Open-source models
	Semantic Adherence			Object State Change		Scene Alignment	Perceptual Quality
Model	Subject	Object	Action	Accuracy	Consistency		Realism	Aesthetics
Open-Sora-2.0 ( Peng et al., 2025 )	0.910	0.722	0.616	0.512	0.658		0.892	0.634	0.712
HunyuanVideo ( Kong et al., 2024 )	0.898	0.764	0.562	0.466	0.730	0.948	0.752	0.782
HunyuanVideo-1.5 ( Team, 2025 )	0.982	0.788	0.642	0.546	0.708	0.936	0.736	0.778
Wan-2.2 ( Wan et al., 2025 )	0.950	0.774	0.570	0.518	0.710	0.974	0.768	0.798

Table 2: GPT-5.2–based evaluation results of T2V models on OSCBench across multiple evaluation dimensions.

Figure 3: Overall performance comparison of T2V models based on aggregated evaluation scores from human evaluator and MLLM-based evaluators (Qwen3-VL-30B and GPT-5.2).

Metrics	Semantic Adherence						Object State Change				Scene Alignment		Perceptual Quality
Metrics	Subject		Object		Action		Accuracy		Consistency				Realism		Aesthetics
	$\tau$	$\rho$	$\tau$	$\rho$	$\tau$	$\rho$	$\tau$	$\rho$	$\tau$	$\rho$	$\tau$	$\rho$	$\tau$	$\rho$	$\tau$	$\rho$
ViCLIP	0.106	0.132	0.195	0.245	0.288	0.386	-	-	-	-	-	-	-	-	-
Qwen3-VL-30B	0.406	0.413	0.412	0.429	0.542	0.624	0.426	0.503	0.289	0.341	0.145	0.149	0.269	0.297	0.407	0.426
GPT-5-mini	0.433	0.439	0.428	0.441	0.478	0.543	0.342	0.392	0.243	0.259	0.200	0.206	0.303	0.338	0.514	0.541
GPT-5.2 (w/o CoT)	0.295	0.318	0.409	0.444	0.623	0.703	0.415	0.493	0.303	0.343	0.425	0.447	0.323	0.355	0.393	0.411

Table 3: Correlation between human and MLLM-based automatic evaluations in terms of Kendall’s

\tau

and Spearman’s

\rho

. The last row reports the mean inter-human correlation for reference.

Figure 4: Sampled video frames generated by different T2V models. State change consistency or noticeable artifacts are highlighted in boxes.

Open-source models
Models	Object State Change Scenario
Models	Regular	Novel	Compositional
Open-Sora-2.0 ( Peng et al., 2025 )	0.410	0.389	0.416
HunyuanVideo ( Kong et al., 2024 )	0.472	0.405	0.437
HunyuanVideo-1.5 ( Team, 2025 )	0.572	0.559	0.556
Wan-2.2 ( Wan et al., 2025 )	0.635	0.531	0.594

Table 4: Human-evaluated object state change scores of T2V models across regular, novel, and compositional scenarios, averaged over accuracy and consistency.

Figure 5: Object state change performance across action categories by human evaluation. Scores are averaged over accuracy and consistency.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmarks	#Prompt	SA	PQ	PC
VBench ( Huang et al., 2024 )	1362	✓	✓
EvalCrafter ( Liu et al., 2024 )	700	✓	✓
T2V-CompBench ( Sun et al., 2025 )	1400	✓
VideoPhy ( Bansal et al., 2024 )	688	✓		✓
PhyGenBench ( Meng et al., 2024 )	160	✓		✓
PhyWorldBench ( Gu et al., 2025 )	1050	✓		✓

Appendix

Table 5: Number of prompts and evaluation dimensions in different T2V generation benchmarks. We abbreviate semantic adherence (SA), perceptual quality (PQ), physical commonsense (PC), and object state change (OSC).

Models	Resolution	Frames	FPS	Duration (s)
Open-Sora-2.0	768 $\times$ 768	129	25	5
HunyuanVideo	1280 $\times$ 720	129	25	5
HunyuanVideo-1.5	1280 $\times$ 720	121	24	5
Wan-2.2	1280 $\times$ 720	81	16	5
Kling-2.5-Turbo	1920 $\times$ 1080	121	24	5
Veo-3.1-Fast	1280 $\times$ 720	144	24	6

Appendix

Table 6: Generation settings of T2V models in terms of resolution, total frames, FPS, and duration.

Panel of Figure 6 — Figure 6: Data abstraction results and word cloud in OSCBench.

Suppose you are an expert in judging and evaluating the quality of AI-generated videos. Such videos may exhibit anomalies such as unnatural object appearance or disappearance, physically implausible state changes, and temporal inconsistencies across frames. They may also contain visual artifacts or unnatural textures. You are given 20 frames evenly sampled from a 5-second AI-generated video.

Video Prompt:

“A chef with a white apron is slicing leek at a street food stand.”

Your Task:

Analyze these frames chronologically and evaluate the video using the following criteria.

{Criteria}

Appendix

Table 7: Prompt for MLLM to generate the evidence and score for each sampled video. Criteria are the same as those used in the human-evaluation interface.

Open-source models
	Semantic Adherence			Object State Change		Scene Alignment	Perceptual Quality
Model	Subject	Object	Action	Accuracy	Consistency		Realism	Aesthetics
Open-Sora-2.0 ( Peng et al., 2025 )	0.978	0.932	0.718	0.656	0.722		0.980	0.796	0.816
HunyuanVideo ( Kong et al., 2024 )	0.966	0.954	0.682	0.648	0.752	0.988	0.846	0.858
HunyuanVideo-1.5 ( Team, 2025 )	0.986	0.958	0.780	0.756	0.808	0.990	0.864	0.864
Wan-2.2 ( Wan et al., 2025 )	0.992	0.952	0.738	0.712	0.820	0.992	0.882	0.910

Appendix

Table 8: Qwan3-VL-30B-based evaluation results of different T2V models across multiple evaluation dimensions.

Open-source models
	Semantic Adherence			Object State Change		Scene Alignment	Perceptual Quality
Model	Subject	Object	Action	Accuracy	Consistency		Realism	Aesthetics
Open-Sora-2.0 ( Peng et al., 2025 )	0.918	0.838	0.712	0.794	0.814		0.930	0.722	0.774
HunyuanVideo ( Kong et al., 2024 )	0.960	0.942	0.792	0.718	0.956	0.996	0.908	0.900
HunyuanVideo-1.5 ( Team, 2025 )	0.934	0.914	0.794	0.582	0.788	0.962	0.834	0.788
Wan-2.2 ( Wan et al., 2025 )	0.946	0.944	0.750	0.744	0.970	0.966	0.930	0.946

Appendix

Table 9: GPT-5-mini-based evaluation results of different T2V models across multiple evaluation dimensions.

Figure 7: Example of human and MLLM evaluation on a generated video. Original human evaluation scores averaged over three evaluators are provided for reference. The evidence is generated by GPT-5.2 when scoring with CoT.

Figure 8: Task instructions and evaluation criteria in the human evaluation interface.

Figure 9: Sampled videos of different models in regular OSC scenario.

Figure 10: Sampled videos of different models in novel OSC scenario.

Figure 11: Sampled videos of different models in compositional OSC scenario.

Figure 12: Sampled videos of different models with minimal OSC prompt.

T2V Generation

Text-to-video generation creates moving visual scenes from natural-language prompts, with growing control over motion, identity, events, and style.

See all 110 papers

TextVBench: Evaluating Text Rendering and Preservation in Video Generation
Oct 9, 2026Ruiqiang Zhang, Hengyi Wang, Chang Liu +3Video Generation Evaluation T2V Generation
Anhui Province Key Laboratory of Digital Security, University of Science and Technology of China
VTR-Bench: A Systematic Benchmark for Evaluating Visual Text Rendering in Video Generation
Oct 1, 2026Yu Huang, Jungang Li, Zhiyuan Wang +8Agentic Video Generation Video Generation
Department of Data Science, City University of Hong Kong · The Hong Kong University of Science and Technology (Guangzhou) · Westlake University +3
GenState-AI: State-Aware Dataset for Text-to-Video Retrieval on AI-Generated Videos
Mar 15, 2026Minghan Li, Tongna Chen, Tianrui Lv +3Video Retrieval Temporal Video Understanding
Soochow University China

Oct 9, 2026J/K move · Enter open · S save

TextVBench: Evaluating Text Rendering and Preservation in Video Generation

Ruiqiang Zhang, Hengyi Wang, Chang Liu, Guanjie Wang+2

Anhui Province Key Laboratory of Digital Security, University of Science and Technology of China

Video Generation Evaluation T2V Generation Video Generation Image-to-Video Generation Temporal Consistency in Video Generation

Open paper AskPDF arXiv

Modern video generators can synthesize convincing scenes while rendering a few characters incorrectly, omitting requested text, or changing it across frames. Measuring these failures is harder than applying OCR independently to each frame: when a scene contains multiple text targets and multiple noisy OCR regions, the evaluator must first determine which observation belongs to which intended text. We introduce TextVBench, a benchmark and evaluation framework that formulates visual-text evaluation as a multi-target, multi-instance spatiotemporal correspondence problem. TextVBench contains two complementary tracks: TextVBench-T2V measures the creation and maintenance of prompt-specified text, while TextVBench-Ref2V measures preservation of text from reference images. Both tracks use target-conditioned candidate selection rather than unconstrained VLM transcription, support one target mapping to multiple OCR regions, and separate frame-level spatial fidelity from cross-frame temporal fidelity. A length-weighted harmonic score summarizes these factors, while extra-text diagnostics expose failure modes hidden by a single OCR score. The benchmark is organized around controlled text, layout, carrier, and motion attributes so that aggregate scores can be traced back to specific capability limits. Across the evaluated model runs, MiniMax-H3 reaches a T2V TextVScore of 68.32, while the strongest Ref2V score is 56.51; attribute slices reveal a pronounced drop as the number of simultaneous reference text regions increases.

Abstract

T2V Generation

Explore similar work

TextVBench: Evaluating Text Rendering and Preservation in Video Generation

VTR-Bench: A Systematic Benchmark for Evaluating Visual Text Rendering in Video Generation

GenState-AI: State-Aware Dataset for Text-to-Video Retrieval on AI-Generated Videos