SPATIALALIGN: Aligning Dynamic Spatial Relationships in Video Generation

Fengming Liu; Tat-Jen Cham; Chuanxia Zheng

cs.CVFeb 26, 2026

SPATIALALIGN: Aligning Dynamic Spatial Relationships in Video Generation

Authors: Fengming Liu, Tat-Jen Cham, Chuanxia Zheng

Organizations: College of Computing and Data Science, Nanyang Technological University, 50 Nanyang Avenue, Singapore 639798.

Abstract

Most text-to-video (T2V) generators prioritize aesthetic quality, but often ignoring the spatial constraints in the generated videos. In this work, we present SPATIALALIGN, a self-improvement framework that enhances T2V models capabilities to depict Dynamic Spatial Relationships (DSR) specified in text prompts. We present a zeroth-order regularized Direct Preference Optimization (DPO) to fine-tune T2V models towards better alignment with DSR. Specifically, we design DSR-SCORE, a geometry-based metric that quantitatively measures the alignment between generated videos and the specified DSRs in prompts, which is a step forward from prior works that rely on VLM for evaluation. We also conduct a dataset of text-video pairs with diverse DSRs to facilitate the study. Extensive experiments demonstrate that our fine-tuned model significantly out performs the baseline in spatial relationships. The code will be released in Link. Project page: https://fengming001ntu.github.io/SpatialAlign/

Figures & tables

Figure 2: Overview of SpatialAlign . (A) Given a text prompt, the pre-trained T2V model generates several video samples. For each sample, we first use GroundedSAM to obtain the bboxes of the animal and the object in each frame. (B) Then, for each frame, we compute the Static Spatial Relationship (SSR) Score based on the bboxes. From the SSR Score Sequence (all frames), we derive four metric components that are aggregated into the DSR-Score , which quantifies how well the video aligns with the dynamic spatial relationship (DSR). (C) During DPO training, we identify winner/loser pairs based on the DSR-Score using a threshold. We then train a LoRA to enhance the model’s ability to accurately represent DSR in generated videos, using our proposed zero-order regularized DPO.

	DSR Type	Initial SSR $\textbf{p}_{\text{init}}$	Final SSR $\textbf{p}_{\text{f}}$
$\mathbb{A}$	left-to-top	on the left of	to the top of
$\mathbb{B}$	top-to-left	on the top of	to the left of
$\mathbb{C}$	right-to-top	on the right of	to the top of
$\mathbb{D}$	left-to-right	on the left of	to the right of
$\mathbb{E}$	top-to-right	on the top of	to the right of
$\mathbb{F}$	right-to-left	on the right of	to the left of

Table 1: Overview of DSR types and the corresponding keywords.

Figure 3: Ablation on loss . With only the DPO loss $\mathcal{L}_{\text{DPO}}$ , the model degrades after 800 steps. With SFT loss $\mathcal{L}_{\text{SFT}}$ , the color saturation is strong. With our zeroth-order regularization $\mathcal{L}_{\text{ZO}}$ , the training is stable.

Figure 4: VLM vs DSR-Score . Each plot contains bins at each SSR Score value. Each bin shows the portion where the VLM gives the YES answer or NO answer at the particular SSR Score location. The VLM gives a significant portion of YES on the low SSR Score interval, especially for the evaluation on the final SSR.

Figure 5: Qualitative comparisons with the state-of-the-art T2V models on DSR-Dataset . Our fine-tuned model (right columns) generates videos that correctly reflect the desired change in spatial relationships, while the baseline models (left columns) fail to do so.

Method	[email protected] $\uparrow$	ID $\uparrow$	CLIP-IQA $\uparrow$	IQ $\uparrow$
CogVideoX1.5-5B	0.053	0.6404	0.8874	0.5894
OpenSora v1.2	0.018	0.7151	0.8712	0.6616
LTX-Video-2B	0.058	0.8028	0.9061	0.6786
HunyuanVideo v1.5-8B	0.490	0.7565	0.9299	0.6662
Wan2.1-1.3B	0.125	0.7046	0.8481	0.7000
Wan2.1-1.3B + LoRA (Ours)	0.585	0.6934	0.8243	0.6909

Table 2: Quantitative Results on DSR-Dataset . “[email protected]” denotes Correctness at threshold 0.7. “ID” denotes ID Consistency. “CLIP-IQA” denotes CLIP-based Image Quality Assessment. “IQ” denotes Imaging Quality. The best and second-best results are bolded and underlined , respectively. Our method achieves the best correctness while maintaining decent image quality.

Table 7

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure A7 : Correctness curve . The percentage of samples having DSR-Score $>=\tau_{\text{test}}$ under different $\tau_{\text{test}}$ . The emphasized focus (grey area) is on $\tau_{\text{test}}=0.7$ . (a) The training significantly enhances the model’s ability in aligning with DSR prompts. (b) We compare the result on the test set to a subset of training set (also 30 prompts), and indicate that the fine-tuned model is not overfit to the training set.

		Animal	Object	Initial SSR	Final SSR	Others
Baseline	Initial SSR	0.395	0.280	1.000	0.837	0.878
Baseline	Final SSR	0.435	0.278	0.837	1.000	0.892
+ LoRA	Initial SSR	0.456	0.277	1.000	0.866	0.894
+ LoRA	Final SSR	0.489	0.274	0.866	1.000	0.903

Appendix

Table A4 : CAMAP Analysis . We compute the average CAMAP correlation between different types of tokens: Animal-related tokens, Object-related tokens, Initial SSR-related tokens, Final SSR-related tokens, and Other tokens. The fine-tuned model shows a stronger correlation between the

\langle\text{\emph{initial SSR}}\rangle\text{\emph{-to-}}\langle\text{\emph{animal}}\rangle

and

\langle\text{\emph{final SSR}}\rangle\text{\emph{-to-}}\langle\text{\emph{animal}}\rangle

pairs, indicating better capturing of the spatial relationships specified in the DSR prompts.

Figure A8 : Training curves of $\mathcal{L}_{\text{DPO}}+\lambda_{\text{ZO}}\mathcal{L}_{\text{ZO}}$ (black) and $\mathcal{L}_{\text{DPO}}$ ( blue ). With only $\mathcal{L}_{\text{DPO}}$ , the training exhibits likelihood displacement : the winner-side implicit reward score_pos drops catastrophically and the loser-side reward score_neg drops even further, raising up the score_margin in an incorrect way. In contrast, the existence of $\mathcal{L}_{\text{ZO}}$ keeps score_pos at slight growth and score_neg at slight drop, giving a healthier growing of score_margin .

Panel of Figure 11 — Figure A9 : Ablation Studies . This is an extended version of Table 3 in the main paper, showing the correctness curves of various ablations.

T2V Generation

Text-to-video generation creates moving visual scenes from natural-language prompts, with growing control over motion, identity, events, and style.

See all 111 papers

Plan-and-Verify Video Reward Reasoning with Spatio-Temporal Scene Graph Grounding
Jun 10, 2026Hyomin Kim, Junghye Kim, Joanie Hayoun Chung +4Reward Modeling T2V Generation
Department of Artificial Intelligence, Korea University · Department of Statistics, Korea University
Retrieval, Refinement, and Ranking for Text-to-Video Generation via Prompt Optimization and Test-Time Scaling
Mar 2, 2026Zillur Rahman, Alex Sheng, Cristian MeoPrompt Optimization for T2V Generation Retrieval-Augmented Video Generation
Algoverse AI
NoisEasier: Test-Time Noise Optimization for Text-to-Video Generation
Aug 31, 2026Yujiang Pu, Yu KongDiffusion Model Guidance Video Diffusion Models
Michigan State University, East Lansing, MI 48824, USA

Jun 10, 2026J/K move · Enter open · S save

Plan-and-Verify Video Reward Reasoning with Spatio-Temporal Scene Graph Grounding

Hyomin Kim, Junghye Kim, Joanie Hayoun Chung, Yoonjin Oh+3

Department of Artificial Intelligence, Korea University · Department of Statistics, Korea University

Reward Modeling T2V Generation Video Reasoning Visually Grounded Reasoning Video Generation Evaluation

Open paper AskPDF arXiv

Reward models for text-to-video (T2V) generation guide post-training but often fail at fine-grained semantic alignment. We trace this to two structural weaknesses in existing reasoning-based reward models: they do not systematically verify every condition described in the prompt, and the visual evidence supporting each judgment remains implicit in their free-form reasoning. We propose SG-PVR, a video reward model that addresses these limitations through plan-and-verify reasoning grounded in spatio-temporal scene graphs. The verification plan decomposes the prompt into atomic claims, making the set of requirements to be checked explicit. The spatio-temporal scene graph, encoding entities, attributes, and temporally grounded relations, is extracted from the video and maintained as a persistent structured visual reference throughout reasoning. Each claim is verified against both the video and the scene graph, anchoring judgments in explicit visual evidence. SG-PVR achieves strong performance on semantic alignment, including fine-grained temporal semantics. As a test-time reranker, it further enhances compositional alignment in T2V generation.

Abstract

T2V Generation

Explore similar work

Plan-and-Verify Video Reward Reasoning with Spatio-Temporal Scene Graph Grounding

Retrieval, Refinement, and Ranking for Text-to-Video Generation via Prompt Optimization and Test-Time Scaling

NoisEasier: Test-Time Noise Optimization for Text-to-Video Generation