Latent Space Jam: Layout Guided Video Synthesis

Asaf Joseph; Moshe Mandel; Ohad Fried

Figure 1. We steer the output (right) of a pre-trained video diffusion model towards a given layout (left) without training. Guiding a video using layout and text.

Figure 2. Overview of our training-free layout-guided video synthesis pipeline. Given a text prompt and spatiotemporal layout constraints, we first perform Layout Generation, where the latents of a frozen text-to-video diffusion model are optimized using a cross-attention-based layout loss. This latent refinement process aligns token-specific attention with the target layout and produces a coarse spatiotemporal guiding video in latent space. Next, after the guide is optionally spatially and/or temporally upsampled, we render the final video: the guiding video is injected at an early timestep into the video diffusion pipeline. This second stage refines appearance and details while preserving the imposed layout and motion, yielding the final high-quality video. A diagram of a training-free layout-guided video synthesis pipeline.

Figure 3. The guide (left) is of low visual quality but adheres to layout, the refined video (right) improves the visual quality and maintains layout adherence. Comparison of two generated outputs of a coarse guide and a refined video. Both show a goat climbing down rocks in a slalom motion.

Figure 4. Outputs of Text-to-Video (T2V) and our method (Ours) across different backbone models (LTX-2, WAN 2.2), together with the corresponding attention maps extracted from an intermediate timestep. Attempting to control layout via textual prompt (underlined) fails, whereas controlling the attention map results in a video that corresponds to the desired layouts (orange/blue bounding boxes). Comparing attention maps and their corresponding outputs.

Figure 5. Comparison of different learning rates during optimization. In each section, guide (top) and refined (bottom) videos are shown. The left column (low learning rate) produces guide videos with weaker adherence to the intended layout (indicated by orange squares). In contrast, the right column (high learning rate) achieves stronger layout adherence, but at the cost of reduced visual quality. Comparison of generated videos of different learning rates, the videos show a person on a bicycle riding from left to right of frame.

	Guide							Refined
	IoU ( $\uparrow$ )	AP50 ( $\uparrow$ )	CD ( $\downarrow$ )	I. ( $\uparrow$ )	I-AP50 ( $\uparrow$ )	FVD ( $\downarrow$ )	ClipSim ( $\uparrow$ )	IoU ( $\uparrow$ )	AP50 ( $\uparrow$ )	CD ( $\downarrow$ )	I. ( $\uparrow$ )	I-AP50 ( $\uparrow$ )	FVD ( $\downarrow$ )	ClipSim ( $\uparrow$ )
	STATIC
PEEKABOO	17.02	8.80	24.74	45.17	45.10	139.00	30.98	19.45	4.37	18.96	58.76	66.71	132.92	29.14
FreeTraj	18.80	12.87	22.05	45.46	44.64	178.25	26.62	19.47	5.71	17.08	53.93	57.56	126.75	30.20
TrailBlazer	20.32	12.40	20.65	46.98	45.97	151.23	30.30	16.95	5.37	19.42	44.65	45.13	126.76	30.26
TrailBlazer (Wan 2.2)	21.76	13.51	18.18	55.10	56.76	148.06	28.73	21.25	13.75	18.89	55.98	56.03	140.14	28.75

Table 1. Quantitative comparison with prior training-free layout-guided methods across datasets. IoU and AP50 measure overlap accuracy. CD, I., and I-AP50 measure trajectory accuracy. FVD and ClipSim measure visual quality. Our framework achieves superior spatial alignment and trajectory consistency on all metrics. Notably, Wan 2.2 excels in guide-stage performance, while LTX-2 provides comparable or stronger refined results.

Method	STATIC			DYNAMIC			MULTI
Method	Prompt ( $\uparrow$ )	Layout ( $\uparrow$ )	VQ ( $\uparrow$ )	Prompt ( $\uparrow$ )	Layout ( $\uparrow$ )	VQ ( $\uparrow$ )	Prompt ( $\uparrow$ )	Layout ( $\uparrow$ )	VQ ( $\uparrow$ )
T2V	8.00 $\pm$ 1.76	6.50 $\pm$ 1.96	7.00 $\pm$ 1.24	9.25 $\pm$ 1.52	0.60 $\pm$ 1.11	7.75 $\pm$ 3.53	8.21 $\pm$ 0.90	3.62 $\pm$ 1.41	7.92 $\pm$ 1.06
PEEKABOO	5.00 $\pm$ 1.26	4.15 $\pm$ 1.13	3.58 $\pm$ 1.15	4.84 $\pm$ 1.34	1.19 $\pm$ 0.49	2.64 $\pm$ 0.92	$N/A$	$N/A$	$N/A$
TrailBlazer	5.38 $\pm$ 1.36	4.15 $\pm$ 1.27	4.25 $\pm$ 1.37	1.60 $\pm$ 0.76	4.15 $\pm$ 1.59	1.48 $\pm$ 0.57	0.83 $\pm$ 0.75	1.39 $\pm$ 0.89	1.54 $\pm$ 0.92
FreeTraj	2.58 $\pm$ 0.95	3.77 $\pm$ 1.31	1.62 $\pm$ 0.55	2.92 $\pm$ 0.95	4.46 $\pm$ 1.37	1.96 $\pm$ 0.69	$N/A$	$N/A$	$N/A$
PEEKABOO [refined]	7.33 $\pm$ 0.86	6.08 $\pm$ 1.14	7.25 $\pm$ 0.96	5.68 $\pm$ 1.49	1.39 $\pm$ 0.91	5.72 $\pm$ 1.06	$N/A$	$N/A$	$N/A$

Table 2. User study results (mean score

\pm

confidence interval with 95% confidence level) across prompt adherence (Prompt), layout adherence (Layout), and visual quality (VQ). T2V excels in visual quality and prompt adherence but shows poor layout adherence. Our method achieves the strongest layout adherence while remaining competitive in prompt alignment and visual quality.

Range	# layers	Guide							Refined
Range	# layers	IoU ( $\uparrow$ )	AP50 ( $\uparrow$ )	CD ( $\downarrow$ )	I. ( $\uparrow$ )	I-AP50 ( $\uparrow$ )	FVD ( $\downarrow$ )	ClipSim ( $\uparrow$ )	IoU ( $\uparrow$ )	AP50 ( $\uparrow$ )	CD ( $\downarrow$ )	I. ( $\uparrow$ )	I-AP50 ( $\uparrow$ )	FVD ( $\downarrow$ )	ClipSim ( $\uparrow$ )
0-10	3	33.82	17.35	8.31	86.15	90.65	173.48	29.69	27.84	12.61	11.18	74.00	78.66	157.01	30.22
0-10	10	35.54	20.34	8.87	87.18	91.77	171.70	29.38	26.79	11.38	12.54	71.70	76.01	151.17	29.60
10-20	3	31.94	19.73	10.52	82.04	89.18	161.22	29.15	24.83	11.61	13.61	69.72	75.84	150.15	30.33
10-20	10	25.49	10.88	14.12	72.42	75.65	156.44	29.74	19.64	6.24	16.66	58.56	62.59	145.02	29.97
20-30	3	16.56	1.91	21.42	51.48	53.81	162.91	29.45	15.70	2.18	21.06	48.33	50.43	150.00	30.02

Table 3. Ablation study on the contribution of different layers to layout adherence. Tested on Wan 2.2. The results indicate that earlier layers perform better.

	Resolution (Guide/Refined)	Inference Time		Guide							Refined
	Resolution (Guide/Refined)	Guide/Refined	Backb.	IoU ( $\uparrow$ )	AP50 ( $\uparrow$ )	CD ( $\downarrow$ )	I. ( $\uparrow$ )	I-AP50 ( $\uparrow$ )	FVD ( $\downarrow$ )	ClipSim ( $\uparrow$ )	IoU ( $\uparrow$ )	AP50 ( $\uparrow$ )	CD ( $\downarrow$ )	I. ( $\uparrow$ )	I-AP50 ( $\uparrow$ )	FVD ( $\downarrow$ )	ClipSim ( $\uparrow$ )
Wan	160 $\times$ 49/320 $\times$ 49	00:36/01:10	01:13	37.01	24.15	7.82	86.00	92.48	174.01	28.94	22.67	10.31	15.23	57.75	62.72	156.44	30.60
	160 $\times$ 49/320 $\times$ 81	00:36/01:59	02:07	37.31	24.32	8.02	87.38	94.18	171.82	28.79	25.63	11.44	12.48	69.80	74.63	148.28	30.43
	160 $\times$ 81/320 $\times$ 81	00:50/01:59	02:07	37.54	24.69	8.63	87.22	91.13	181.94	29.09	24.90	9.42	13.49	69.85	72.49	152.32	30.38
	320 $\times$ 49/320 $\times$ 81	01:55/01:59	02:07	46.74	40.75	5.73	89.49	94.59	185.68	27.48	28.20	12.72	11.30	71.27	75.64	158.87	30.61
	320 $\times$ 81/320 $\times$ 81	03:21/01:59	02:07	48.88	46.40	5.34	90.34	96.32	181.73	26.66	27.62	14.55	10.77	71.41	74.12	163.21	30.35

Table 4. Ablation study of resolution contributions to layout adherence. Resolution is denoted as

spatial

height

\times

temporal

resolution. Inference time of guide and refinement stages are compared to that of the backbone model. Focusing on the refined block (right) of the Wan model (top), the results indicate that it is generally better to generate the refined video at a higher resolution. At the highest resolution (

480\times 81

) it is preferable in terms of trajectory accuracy (CD, I, I-AP50) to generate the respective guide at the lowest resolution (row 6), with negligible decrease in overlap accuracy (IoU, AP50).

Figure 6. Comparison between the video guide generated using an ensemble of Text-to-Image methods (left) and the refined video produced by a pre-trained video model (right). The guide fails to capture the interaction between the boy and the scarf: the hands remain static instead of grasping and moving the scarf, resulting in motion that lacks coordination between the objects. Side-by-side comparison of guide and refined video frames of a boy lifting a scarf over his head.

Figure 7. Comparison of our refined video (Ours) against competing methods. Left: dynamic object layout. While T2V produces high-quality visuals, it does not follow the layout. Trailblazer (TB) and FreeTraj (FT) introduce bicycle artifacts, and Peekaboo (Peek.) shows weaker alignment. Right: multiple-object layout; empty frames indicate unsupported methods. Remaining methods fail to maintain spatial and temporal consistency: the explosion occurs prematurely, and the person does not follow the left-to-right trajectory specified by the blue box. Comparison of outputs of different methods from 2 different datasets: Dynamic and Multi. The first example is of a bicyclist riding from right to left, the second example is of a man walking from left to right, and an explosion happens in the background.

Figure 8. Additional applications of our method. Left: video editing, where the input video is transformed according to the text prompt. Right: stylization, where only selected semantic elements retain color while the rest of the scene remains monochrome. Side-by-side comparison of two applications. Left: a Joker video edited so his hair appears on fire. Right: a film noir stylization where only a woman's red dress remains colored.

Latent Space Jam: Layout Guided Video Synthesis

Abstract

Figures & tables

Controllable Video Generation

Spatially-Grounded Text-to-Video Generation via Inference-Time Gradient-Free Optimization

LooseControlVideo: Directorial Video Control using Spatial Blocking

CoaG: Cylinders on a Grid for Coarse 3D Layout Control in Video Generation

Abstract

Controllable Video Generation

Explore similar work

Spatially-Grounded Text-to-Video Generation via Inference-Time Gradient-Free Optimization

LooseControlVideo: Directorial Video Control using Spatial Blocking

CoaG: Cylinders on a Grid for Coarse 3D Layout Control in Video Generation