Making Video Models Adhere to User Intent with Minor Adjustments

Daniel Ajisafe; Eric Hedlin; Helge Rhodin; Kwang Moo Yi

cs.CVMar 20, 2026

Making Video Models Adhere to User Intent with Minor Adjustments

Authors: Daniel Ajisafe, Eric Hedlin, Helge Rhodin, Kwang Moo Yi

Organizations: Department of Computer Science, The University of British Columbia · Faculty of Technology, Bielefeld University

Abstract

With the recent drastic advancements in text-to-video diffusion models, controlling their generations has drawn interest. A popular way for control is through bounding boxes or layouts. However, enforcing adherence to these control inputs is still an open problem. In this work, we show that by slightly adjusting user-provided bounding boxes we can improve both the quality of generations and the adherence to the control inputs. This is achieved by simply optimizing the bounding boxes to better align with the internal attention maps of the video diffusion model while carefully balancing the focus on foreground and background. In a sense, we are modifying the bounding boxes to be at places where the model is familiar with. Surprisingly, we find that even with small modifications, the quality of generations can vary significantly. To do so, we propose a smooth mask to make the bounding box position differentiable and an attention-maximization objective that we use to alter the bounding boxes. We conduct thorough experiments, including a user study to validate the effectiveness of our method. Our code is made available on the project webpage to foster future research from the community.

Figures & tables

Figure 2: Overview – We inject bounding box control for video diffusion models by editing their cross attention maps within the network. However, not all such edits are friendly to video diffusion models as they are not trained with such edits. Thus, when applying these edits, we make sure that this editing process is differentiable ( Section 3.2 ) and adjust the edit parameters in a way such that the network behaves as intended—attention being focused on desired regions ( Section 3.3 ). We show the original bounding boxes in blue and the adjusted bounding boxes in orange . Though the adjustments are minimal and close to the original user input, they create a drastic difference in terms of video generation quality and adherence to the bounding boxes.

Panel of Figure 3 — Figure 3: Example of attention map editing – We show an example of the generated video frame with the desired user control bounding box, and the associated attention map edits. While effective, Trailblazer Ma et al. (2024b) relies on a replacement operation that is not differentiable with respect to the box parameters. Our method, on the other hand, performs a smooth differentiable edit.

Figure 4: User study interface – We show a sample of the user study interface for prompt ‘the lion is walking’. Users are asked to select their preference over a set of five video generations, provided in a random order. Users are allowed to select multiple choices or no preference. In this example, only the generation results are shown without user control to isolate quality vs. control. For this question, C and E look preferable, whereas the control is located at the bottom-left where the lion is at A, B, and D. C and E completely ignore user control, yet generate a preferred view of the prompt. As we are interested in controlled generation, we consider both answers to the quality preference and the trajectory faithfulness when evaluating the quality of generations.

Panel of Figure 5 — Figure 5: Qualitative results comparing Peekaboo Jain et al. (2024) and our method (full). – Each row shows two representative frames per method (left to right: Peekaboo, Ours). Our method yields better generation quality and improved alignment with the text prompt.

Panel of Figure 6 — Figure 6: Qualitative results with T2V-Turbo Li et al. (2025) (top row) and Trailblazer Ma et al. (2024b) (bottom row) backbones. – We display the original user control in blue and our optimized boxes in orange . As shown, ours provides improved rendering quality and adherence to control and the prompt, shown in the top and bottom row. See video results for best viewing.

Model	PickScore $\uparrow$	HPSv2 $\uparrow$	mIOU $\uparrow$
Trailblazer Ma et al. (2024b)	0.244	0.222	0.37
Our boxes + Trailblazer backbone	0.257	0.223	0.36
Our method w/o Box Opt.	0.243	0.221	0.37
Our method (full)	0.257	0.225	0.37
Peekaboo Jain et al. (2024)	0.125	0.189	0.30
Trailblazer Ma et al. (2024b)	0.146	0.222	0.37

Table 1: Quantitative results with human-preference and control metrics – Our full method outperforms baseline and demonstrates consistent preference across different architectures, while achieving competitive control.

Figure 7: User study results – We report the user preference for trajectory faithfulness on the x-axis, and quality of generation (for generations that adhere to control) on the y-axis. The 95% confidence intervals are drawn as green dashed lines. Our method significantly outperforms Trailblazer Ma et al. (2024b) with both backbones.

Panel of Figure 8 — Figure 8: Qualitative results comparing Freetraj Qiu et al. (2024) and our method on T2V-Turbo Li et al. (2025) . Frames are shown at a common display height (aspect ratio preserved); frame resolutions differ (Freetraj: $320{\times}512$ , Ours: $256{\times}320$ ). Ours still delivers better generation quality and adherence to control.

Panel of Figure 9 — Figure 9: Additional qualitative results with Trailblazer Ma et al. (2024b) – We show additional examples of our edits. As shown, ours is more consistent with user intent and of higher quality. The prompt for the top 2 rows is "The firebrat insect is moving" , while the prompt for the bottom 2 rows is "The jellyfish is swimming" . Video results are available in the supplementary local HTML page.

Panel of Figure 10 — Figure 10: Additional qualitative results with T2V-Turbo Li et al. (2025) backbone – We show additional examples of our edits. Ours show more consistency with user intent and is of higher quality within the box. The prompt for the top 2 rows is "The surgeonfish is swimming" , while the prompt for the bottom 2 rows is "The tiger is walking" . Video results are also available in the supplementary local HTML page.

Panel of Figure 11 — Figure 11: Failure cases – We show example failure cases when the video model fails to generate the desired output for the prompt the orca is swimming . Here, regardless of the bounding box input, the model is unable to generate the content as it has no knowledge of the target prompt. We hypothesize that more advanced video models, which are being made increasingly available Research (2023) ; Wan et al. (2025) , will alleviate this problem.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Attention map	Attention reshaped	# Edit channels	Edited copy (lower channels)	Layer info
240, 1600, 77	240, 40, 40, 77	120	120, 40, 40, 77	down_blocks.0.0.0
240, 1600, 77	240, 40, 40, 77	120	120, 40, 40, 77	down_blocks.0.1.0
480, 400, 77	480, 20, 20, 77	240	240, 20, 20, 77	down_blocks.1.0.0
480, 400, 77	480, 20, 20, 77	240	240, 20, 20, 77	down_blocks.1.1.0
960, 100, 77	960, 10, 10, 77	480	480, 10, 10, 77	down_blocks.2.0.0
960, 100, 77	960, 10, 10, 77	480	480, 10, 10, 77	down_blocks.2.1.0

Appendix

Table 2: Edited cross attention maps – As in Trailblazer Ma et al. (2024b) , we edit only the lower channels across all layers. Each layer follows the spatial cross-attention naming where down_blocks.0.0.0 can be fully written as down_blocks . 0 .attentions. 0 .transformer_blocks. 0 .attn2 .

Panel of Figure 12 — Figure 12: Effect of regularization strength/penalty – We show an example frame with original bounding boxes as blue and the adjusted boxes as orange . A higher penalty (1.0) allows less movement, while too loose penalty (0.001) causes generation to stray away from user intent. We set the penalty to 0.1 for all our experiments, which strikes a good balance.

Panel of Figure 13 — Figure 13: Application to perspective and linear trajectories (top and bottom respectively) – We show example frames of applying our method to T2V-Turbo Li et al. (2025) with a trajectory that linearly interpolates two bounding boxes (start and end). Ours still delivers improvements, both in terms of generation quality within the bounding box and adherence.

$\alpha$	PickScore $\uparrow$	HPSv2 $\uparrow$	mIOU $\uparrow$
0	0.210	0.230	0.31
0.001	0.211	0.230	0.32
0.01	0.199	0.228	0.36
0.1	0.194	0.225	0.38
1	0.187	0.221	0.37

Appendix

Table 3: Ablation results for deviation penalty term

{\lambda_{\text{reg}}}{=}\alpha\times\sqrt{A}

Normalization	PickScore $\uparrow$	HPSv2 $\uparrow$	mIOU $\uparrow$
True	0.509	0.225	0.38
False	0.491	0.223	0.38

Appendix

Table 4: Ablation results for smoothing kernel normalization

${\lambda_{\text{$ \neg $attn}}}$	PickScore $\uparrow$	HPSv2 $\uparrow$	mIOU $\uparrow$
0	0.228	0.219	0.36
1	0.231	0.221	0.39
10	0.256	0.225	0.38
100	0.285	0.228	0.31

Appendix

Table 5: Ablation results for background preservation term

{\lambda_{\text{

\neg

attn}}}

$\lambda_{edge}$	PickScore $\uparrow$	HPSv2 $\uparrow$	mIOU $\uparrow$
0.0001	0.240	0.224	0.40
0.001	0.248	0.226	0.38
0.03	0.243	0.225	0.38
0.1	0.268	0.232	0.36

Appendix

Table 6: Ablation results for edge strength

\lambda_{edge}

Panel of Figure 14 — Figure 14: Additional qualitative results for morphing task (Parrot $\rightarrow$ Penguin)– We show transition frames, where our optimized boxes provide benefit for better control of the morphing task.

Panel of Figure 15 — Figure 15: Additional qualitative results with stationary-to-move and U-turn trajectories – We show example frames where our method is applied to complex patterns and preserves difficult motions. Best viewed in motion on the project webpage.

Panel of Figure 16 — Figure 16: Extreme scenario case with very small boxes and in a zig-zag trajectory – We present an extreme scenario with zig-zag motion using very small boxes, different from the object size. Our method compensates and generates motion that follows user intent. Best viewed in motion on the project webpage.

Controllable Video Generation

Controllable video generation steers how videos look, move, and change over time using user-provided instructions and structured controls.

See all 321 papers

PE-Field 4D: Video Generation Models as Canvas
Jul 17, 2026Yunpeng Bai, Haoxiang Li, Qixing HuangCamera-Controlled Video Generation Diffusion Transformer
University of Texas at Austin, USA · Pixocial Technology, USA
Making Time Editable in Video Diffusion Transformers
Jun 8, 2026Konstantin Kuklev, Viacheslav Vasilev, Alexander Kunitsyn +2Video Diffusion Models Diffusion Transformer
Kandinsky Lab.
ReaDiT Guidance: Control for Image and Video Generation using Diffusion Transformer Features
Sep 4, 2026Jay Mahajan, Chang Liu, Rauf Makharov +3Diffusion Model Guidance Controllable Image Generation
University of Illinois Urbana-Champaign · Google

Jul 17, 2026J/K move · Enter open · S save

PE-Field 4D: Video Generation Models as Canvas

Yunpeng Bai, Haoxiang Li, Qixing Huang

University of Texas at Austin, USA · Pixocial Technology, USA

Camera-Controlled Video Generation Diffusion Transformer Dynamic Novel View Synthesis Video Generation Controllable Video Generation

Open paper AskPDF arXiv

Diffusion Transformers have recently achieved strong performance in video generation, yet controlling scene geometry under viewpoint changes and camera motion remains challenging. In this work, we revisit the role of positional encoding in video diffusion transformers and show that it provides a useful spatial bias for geometry-aware control. Specifically, if reference tokens are encoded according to their projected locations in the target view, the denoising model is encouraged to retrieve content from position aligned regions of the input video. Building on this observation, we introduce a geometry-aware cross-attention mechanism that enables target video latent tokens to attend to structured context tokens derived from reference images or frames. To establish correspondence between the reference content and the target camera trajectory, we equip the context tokens with a projected positional encoding scheme that combines target-view 2D reprojection with depth-aware disambiguation. At the same time, we preserve the original spatiotemporal positional encoding of the generated video latent, allowing geometric guidance to be injected while maintaining consistency with the video model's native latent structure. The resulting framework provides a simple and effective approach for controllable video generation. It improves spatial controllability in viewpoint-dependent editing tasks, including camera re-trajectory, novel-view video synthesis, and geometry-aware video editing, while preserving the generative prior of the underlying video diffusion model. The code is available at: https://github.com/MTLab/PE-Field.

Abstract

Controllable Video Generation

Explore similar work

PE-Field 4D: Video Generation Models as Canvas

Making Time Editable in Video Diffusion Transformers

ReaDiT Guidance: Control for Image and Video Generation using Diffusion Transformer Features