Most text-to-video (T2V) generators prioritize aesthetic quality, but often ignoring the spatial constraints in the generated videos. In this work, we present SPATIALALIGN, a self-improvement framework that enhances T2V models capabilities to depict Dynamic Spatial Relationships (DSR) specified in text prompts. We present a zeroth-order regularized Direct Preference Optimization (DPO) to fine-tune T2V models towards better alignment with DSR. Specifically, we design DSR-SCORE, a geometry-based metric that quantitatively measures the alignment between generated videos and the specified DSRs in prompts, which is a step forward from prior works that rely on VLM for evaluation. We also conduct a dataset of text-video pairs with diverse DSRs to facilitate the study. Extensive experiments demonstrate that our fine-tuned model significantly out performs the baseline in spatial relationships. The code will be released in Link. Project page: https://fengming001ntu.github.io/SpatialAlign/
Figures & tables
Figure 2: Overview of SpatialAlign . (A) Given a text prompt, the pre-trained T2V model generates several video samples. For each sample, we first use GroundedSAM to obtain the bboxes of the animal and the object in each frame. (B) Then, for each frame, we compute the Static Spatial Relationship (SSR) Score based on the bboxes. From the SSR Score Sequence (all frames), we derive four metric components that are aggregated into the DSR-Score , which quantifies how well the video aligns with the dynamic spatial relationship (DSR). (C) During DPO training, we identify winner/loser pairs based on the DSR-Score using a threshold. We then train a LoRA to enhance the model’s ability to accurately represent DSR in generated videos, using our proposed zero-order regularized DPO.
DSR Type
Initial SSR pinit
Final SSR pf
A
left-to-top
on the left of
to the top of
B
top-to-left
on the top of
to the left of
C
right-to-top
on the right of
to the top of
D
left-to-right
on the left of
to the right of
E
top-to-right
on the top of
to the right of
F
right-to-left
on the right of
to the left of
Table 1: Overview of DSR types and the corresponding keywords.
Figure 3: Ablation on loss . With only the DPO loss LDPO , the model degrades after 800 steps. With SFT loss LSFT , the color saturation is strong. With our zeroth-order regularization LZO , the training is stable.
Figure 4: VLM vs DSR-Score . Each plot contains bins at each SSR Score value. Each bin shows the portion where the VLM gives the YES answer or NO answer at the particular SSR Score location. The VLM gives a significant portion of YES on the low SSR Score interval, especially for the evaluation on the final SSR.
Figure 5: Qualitative comparisons with the state-of-the-art T2V models on DSR-Dataset . Our fine-tuned model (right columns) generates videos that correctly reflect the desired change in spatial relationships, while the baseline models (left columns) fail to do so.
Table 2: Quantitative Results on DSR-Dataset . “[email protected]” denotes Correctness at threshold 0.7. “ID” denotes ID Consistency. “CLIP-IQA” denotes CLIP-based Image Quality Assessment. “IQ” denotes Imaging Quality. The best and second-best results are bolded and underlined , respectively. Our method achieves the best correctness while maintaining decent image quality.
Table 7
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A7 : Correctness curve . The percentage of samples having DSR-Score >=τtest under different τtest . The emphasized focus (grey area) is on τtest=0.7 . (a) The training significantly enhances the model’s ability in aligning with DSR prompts. (b) We compare the result on the test set to a subset of training set (also 30 prompts), and indicate that the fine-tuned model is not overfit to the training set.
Animal
Object
Initial SSR
Final SSR
Others
Baseline
Initial SSR
0.395
0.280
1.000
0.837
0.878
Final SSR
0.435
0.278
0.837
1.000
0.892
+ LoRA
Initial SSR
0.456
0.277
1.000
0.866
0.894
Final SSR
0.489
0.274
0.866
1.000
0.903
Appendix
Table A4 : CAMAP Analysis . We compute the average CAMAP correlation between different types of tokens: Animal-related tokens, Object-related tokens, Initial SSR-related tokens, Final SSR-related tokens, and Other tokens. The fine-tuned model shows a stronger correlation between the ⟨initial SSR⟩-to-⟨animal⟩ and ⟨final SSR⟩-to-⟨animal⟩ pairs, indicating better capturing of the spatial relationships specified in the DSR prompts.
Figure A8 : Training curves of LDPO+λZOLZO (black) and LDPO ( blue ). With only LDPO , the training exhibits likelihood displacement : the winner-side implicit reward score_pos drops catastrophically and the loser-side reward score_neg drops even further, raising up the score_margin in an incorrect way. In contrast, the existence of LZO keeps score_pos at slight growth and score_neg at slight drop, giving a healthier growing of score_margin .
Figure A9 : Ablation Studies . This is an extended version of Table 3 in the main paper, showing the correctness curves of various ablations.
Reward models for text-to-video (T2V) generation guide post-training but often fail at fine-grained semantic alignment. We trace this to two structural weaknesses in existing reasoning-based reward models: they do not systematically verify every condition described in the prompt, and the visual evidence supporting each judgment remains implicit in their free-form reasoning. We propose SG-PVR, a video reward model that addresses these limitations through plan-and-verify reasoning grounded in spatio-temporal scene graphs. The verification plan decomposes the prompt into atomic claims, making the set of requirements to be checked explicit. The spatio-temporal scene graph, encoding entities, attributes, and temporally grounded relations, is extracted from the video and maintained as a persistent structured visual reference throughout reasoning. Each claim is verified against both the video and the scene graph, anchoring judgments in explicit visual evidence. SG-PVR achieves strong performance on semantic alignment, including fine-grained temporal semantics. As a test-time reranker, it further enhances compositional alignment in T2V generation.
Hyomin Kim, Junghye Kim, Joanie Hayoun Chung +4
Department of Artificial Intelligence, Korea University · Department of Statistics, Korea University
While large-scale datasets have driven significant progress in Text-to-Video (T2V) generative models, these models remain highly sensitive to input prompts, demonstrating that prompt design is critical to generation quality. Current methods for improving video output often fall short: they either depend on complex, post-editing models, risking the introduction of artifacts, or require expensive fine-tuning of the core generator, which severely limits both scalability and accessibility. In this work, we introduce 3R, a novel RAG based prompt optimization framework. 3R utilizes the power of current state-of-the-art T2V diffusion model and vision language model. It can be used with any T2V model without any kind of model training. The framework leverages three key strategies: RAG-based modifiers extraction for enriched contextual grounding, diffusion-based Preference Optimization for aligning outputs with human preferences, and temporal frame interpolation for producing temporally consistent visual contents. Together, these components enable more accurate, efficient, and contextually aligned text-to-video generation. Experimental results demonstrate the efficacy of 3R in enhancing the static fidelity and dynamic coherence of generated videos, underscoring the importance of optimizing user prompts.
Diffusion models have recently advanced text-to-video (T2V) generation, yet they still struggle with fine-grained compositional alignment, such as attribute binding, spatial relations, and object interactions. While reward-based fine-tuning improves alignment, it is susceptible to reward hacking and adapts poorly to new prompt distributions. In this work, we propose NoisEasier, a test-time scaling framework that improves T2V generation through differentiable reward-guided noise optimization without modifying the underlying model. By combining efficient short-step generators with a multi-objective reward formulation, NoisEasier enables stable and practical test-time optimization under realistic inference budgets. Our key insight is that jointly optimizing the entire stochastic trajectory accelerates reward convergence and improves compositional alignment over optimizing only the initial latent, with negligible additional computational and time cost. Experiments on VBench and T2V-CompBench demonstrate consistent improvements across multiple backbones, achieving over 10% average gains on challenging dimensions such as attribute binding, object interaction, and numeracy. Overall, NoisEasier serves as both a flexible alternative and a complementary enhancement to reward-based fine-tuning, establishing test-time scaling as an effective paradigm for controllable text-to-video generation.
Yujiang Pu, Yu Kong
Michigan State University, East Lansing, MI 48824, USA