With the recent drastic advancements in text-to-video diffusion models, controlling their generations has drawn interest. A popular way for control is through bounding boxes or layouts. However, enforcing adherence to these control inputs is still an open problem. In this work, we show that by slightly adjusting user-provided bounding boxes we can improve both the quality of generations and the adherence to the control inputs. This is achieved by simply optimizing the bounding boxes to better align with the internal attention maps of the video diffusion model while carefully balancing the focus on foreground and background. In a sense, we are modifying the bounding boxes to be at places where the model is familiar with. Surprisingly, we find that even with small modifications, the quality of generations can vary significantly. To do so, we propose a smooth mask to make the bounding box position differentiable and an attention-maximization objective that we use to alter the bounding boxes. We conduct thorough experiments, including a user study to validate the effectiveness of our method. Our code is made available on the project webpage to foster future research from the community.
Figures & tables
Figure 2: Overview – We inject bounding box control for video diffusion models by editing their cross attention maps within the network. However, not all such edits are friendly to video diffusion models as they are not trained with such edits. Thus, when applying these edits, we make sure that this editing process is differentiable ( Section 3.2 ) and adjust the edit parameters in a way such that the network behaves as intended—attention being focused on desired regions ( Section 3.3 ). We show the original bounding boxes in blue and the adjusted bounding boxes in orange . Though the adjustments are minimal and close to the original user input, they create a drastic difference in terms of video generation quality and adherence to the bounding boxes.
Figure 3: Example of attention map editing – We show an example of the generated video frame with the desired user control bounding box, and the associated attention map edits. While effective, Trailblazer Ma et al. (2024b) relies on a replacement operation that is not differentiable with respect to the box parameters. Our method, on the other hand, performs a smooth differentiable edit.
Figure 4: User study interface – We show a sample of the user study interface for prompt ‘the lion is walking’. Users are asked to select their preference over a set of five video generations, provided in a random order. Users are allowed to select multiple choices or no preference. In this example, only the generation results are shown without user control to isolate quality vs. control. For this question, C and E look preferable, whereas the control is located at the bottom-left where the lion is at A, B, and D. C and E completely ignore user control, yet generate a preferred view of the prompt. As we are interested in controlled generation, we consider both answers to the quality preference and the trajectory faithfulness when evaluating the quality of generations.
Figure 5: Qualitative results comparing Peekaboo Jain et al. (2024) and our method (full). – Each row shows two representative frames per method (left to right: Peekaboo, Ours). Our method yields better generation quality and improved alignment with the text prompt.
Figure 6: Qualitative results with T2V-Turbo Li et al. (2025) (top row) and Trailblazer Ma et al. (2024b) (bottom row) backbones. – We display the original user control in blue and our optimized boxes in orange . As shown, ours provides improved rendering quality and adherence to control and the prompt, shown in the top and bottom row. See video results for best viewing.
Model
PickScore ↑
HPSv2 ↑
mIOU ↑
Trailblazer Ma et al. (2024b)
0.244
0.222
0.37
Our boxes + Trailblazer backbone
0.257
0.223
0.36
Our method w/o Box Opt.
0.243
0.221
0.37
Our method (full)
0.257
0.225
0.37
Peekaboo Jain et al. (2024)
0.125
0.189
0.30
Trailblazer Ma et al. (2024b)
0.146
0.222
0.37
Table 1: Quantitative results with human-preference and control metrics – Our full method outperforms baseline and demonstrates consistent preference across different architectures, while achieving competitive control.
Figure 7: User study results – We report the user preference for trajectory faithfulness on the x-axis, and quality of generation (for generations that adhere to control) on the y-axis. The 95% confidence intervals are drawn as green dashed lines. Our method significantly outperforms Trailblazer Ma et al. (2024b) with both backbones.
Figure 8: Qualitative results comparing Freetraj Qiu et al. (2024) and our method on T2V-Turbo Li et al. (2025) . Frames are shown at a common display height (aspect ratio preserved); frame resolutions differ (Freetraj: 320×512 , Ours: 256×320 ). Ours still delivers better generation quality and adherence to control.
Figure 9: Additional qualitative results with Trailblazer Ma et al. (2024b) – We show additional examples of our edits. As shown, ours is more consistent with user intent and of higher quality. The prompt for the top 2 rows is "The firebrat insect is moving" , while the prompt for the bottom 2 rows is "The jellyfish is swimming" . Video results are available in the supplementary local HTML page.
Figure 10: Additional qualitative results with T2V-Turbo Li et al. (2025) backbone – We show additional examples of our edits. Ours show more consistency with user intent and is of higher quality within the box. The prompt for the top 2 rows is "The surgeonfish is swimming" , while the prompt for the bottom 2 rows is "The tiger is walking" . Video results are also available in the supplementary local HTML page.
Figure 11: Failure cases – We show example failure cases when the video model fails to generate the desired output for the prompt the orca is swimming . Here, regardless of the bounding box input, the model is unable to generate the content as it has no knowledge of the target prompt. We hypothesize that more advanced video models, which are being made increasingly available Research (2023) ; Wan et al. (2025) , will alleviate this problem.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Attention map
Attention reshaped
# Edit channels
Edited copy (lower channels)
Layer info
240, 1600, 77
240, 40, 40, 77
120
120, 40, 40, 77
down_blocks.0.0.0
240, 1600, 77
240, 40, 40, 77
120
120, 40, 40, 77
down_blocks.0.1.0
480, 400, 77
480, 20, 20, 77
240
240, 20, 20, 77
down_blocks.1.0.0
480, 400, 77
480, 20, 20, 77
240
240, 20, 20, 77
down_blocks.1.1.0
960, 100, 77
960, 10, 10, 77
480
480, 10, 10, 77
down_blocks.2.0.0
960, 100, 77
960, 10, 10, 77
480
480, 10, 10, 77
down_blocks.2.1.0
Appendix
Table 2: Edited cross attention maps – As in Trailblazer Ma et al. (2024b) , we edit only the lower channels across all layers. Each layer follows the spatial cross-attention naming where down_blocks.0.0.0 can be fully written as down_blocks . 0 .attentions. 0 .transformer_blocks. 0 .attn2 .
Figure 12: Effect of regularization strength/penalty – We show an example frame with original bounding boxes as blue and the adjusted boxes as orange . A higher penalty (1.0) allows less movement, while too loose penalty (0.001) causes generation to stray away from user intent. We set the penalty to 0.1 for all our experiments, which strikes a good balance.
Figure 13: Application to perspective and linear trajectories (top and bottom respectively) – We show example frames of applying our method to T2V-Turbo Li et al. (2025) with a trajectory that linearly interpolates two bounding boxes (start and end). Ours still delivers improvements, both in terms of generation quality within the bounding box and adherence.
α
PickScore ↑
HPSv2 ↑
mIOU ↑
0
0.210
0.230
0.31
0.001
0.211
0.230
0.32
0.01
0.199
0.228
0.36
0.1
0.194
0.225
0.38
1
0.187
0.221
0.37
Appendix
Table 3: Ablation results for deviation penalty term λreg=α×A
Normalization
PickScore ↑
HPSv2 ↑
mIOU ↑
True
0.509
0.225
0.38
False
0.491
0.223
0.38
Appendix
Table 4: Ablation results for smoothing kernel normalization
{\lambda_{\text{\negattn}}}
PickScore ↑
HPSv2 ↑
mIOU ↑
0
0.228
0.219
0.36
1
0.231
0.221
0.39
10
0.256
0.225
0.38
100
0.285
0.228
0.31
Appendix
Table 5: Ablation results for background preservation term {\lambda_{\text{\negattn}}}
λedge
PickScore ↑
HPSv2 ↑
mIOU ↑
0.0001
0.240
0.224
0.40
0.001
0.248
0.226
0.38
0.03
0.243
0.225
0.38
0.1
0.268
0.232
0.36
Appendix
Table 6: Ablation results for edge strength λedge
Figure 14: Additional qualitative results for morphing task (Parrot → Penguin)– We show transition frames, where our optimized boxes provide benefit for better control of the morphing task.
Figure 15: Additional qualitative results with stationary-to-move and U-turn trajectories – We show example frames where our method is applied to complex patterns and preserves difficult motions. Best viewed in motion on the project webpage.
Figure 16: Extreme scenario case with very small boxes and in a zig-zag trajectory – We present an extreme scenario with zig-zag motion using very small boxes, different from the object size. Our method compensates and generates motion that follows user intent. Best viewed in motion on the project webpage.
Diffusion Transformers have recently achieved strong performance in video generation, yet controlling scene geometry under viewpoint changes and camera motion remains challenging. In this work, we revisit the role of positional encoding in video diffusion transformers and show that it provides a useful spatial bias for geometry-aware control. Specifically, if reference tokens are encoded according to their projected locations in the target view, the denoising model is encouraged to retrieve content from position aligned regions of the input video. Building on this observation, we introduce a geometry-aware cross-attention mechanism that enables target video latent tokens to attend to structured context tokens derived from reference images or frames. To establish correspondence between the reference content and the target camera trajectory, we equip the context tokens with a projected positional encoding scheme that combines target-view 2D reprojection with depth-aware disambiguation. At the same time, we preserve the original spatiotemporal positional encoding of the generated video latent, allowing geometric guidance to be injected while maintaining consistency with the video model's native latent structure. The resulting framework provides a simple and effective approach for controllable video generation. It improves spatial controllability in viewpoint-dependent editing tasks, including camera re-trajectory, novel-view video synthesis, and geometry-aware video editing, while preserving the generative prior of the underlying video diffusion model. The code is available at: https://github.com/MTLab/PE-Field.
Yunpeng Bai, Haoxiang Li, Qixing Huang
University of Texas at Austin, USA · Pixocial Technology, USA
Modern Diffusion Transformers for video generation provide limited control over the progression of time and the editing of temporal dynamics. We propose a temporal-control methodology that extends a pretrained DiT with explicit time editing, allowing control over motion speed and temporal structure without redesigning the backbone. Its core implementation augments the pretrained model with a lightweight temporal module, preserving the original generative prior while expanding its controllable dynamic range.
Konstantin Kuklev, Viacheslav Vasilev, Alexander Kunitsyn +2
We present DiT Readout (ReaDiT) Guidance, a lightweight framework for controlling generation with Diffusion Transformer (DiT) models via their internal feature representations. ReaDiT Guidance uses features from a single DiT block to steer the generative process according to spatial targets - like depth, pose, or edge maps - provided at test time. Furthermore, since modern text-to-video models are largely built on DiT backbones, ReaDiT Guidance naturally extends to video generation, enabling camera and motion control. Experimental results demonstrate that our approach achieves competitive or improved results compared to existing feature-based and off-the-shelf adapter-based approaches while requiring fewer parameters.