ReaDiT Guidance: Control for Image and Video Generation using Diffusion Transformer Features

Jay Mahajan; Chang Liu; Rauf Makharov; Viraj Shah; Alexander Schwing; Svetlana Lazebnik

Method	Image	Video	Params (M)	Data	Implicit Model Conditioning
OminiControl	✓	✗	14.5	Small	✗
ControlNet (Image)	✓	✗	1487	Small	✗
ControlNet (Video)	✗	✓	2500	Large	✗
RG	✓	✗	13.5	Small	✓
DiTFlow	✗	✓ (Motion only)	0	None	✗
Tora	✗	✓ (Motion only)	1100	Large	✗

Table 1: Comparison of DiT Control Approaches. Overview of supported modalities, parameter overhead, training data requirements, and whether the method utilizes Implicit Model Conditioning (i.e., the diffusion model does not process the condition during its forward pass). For ControlNet, parameters for image and video are estimated from SD3-Medium and CogVideoX-5B respectively.

Figure 1: Method Overview. Left: ReaDiT takes a single intermediate feature from the frozen DiT and predicts spatial outputs, which are used to guide image generation by optimizing the latent

z_{t}

. Right: ReaDiT is trained to predict task-specific outputs from DiT features extracted by noising an input image to timestep

t

and passing it through the frozen DiT.

Figure 2: Architecture of ReaDiT. Input features are conditioned on

t

and encoded into multi-resolution feature maps. The feature maps are progressively fused and upsampled to a final feature map, from which a task-specific head produces the spatial output.

Figure 3: Spatial Prediction Results. We compare the prediction of ReaDiT to off-the-shelf models across depth, pose, and edge tasks. ReaDiT produces meaningful predictions from frozen DiT features, which can also be used to monitor spatial properties during the generation process. Additional examples are presented in the supplementary material.

Figure 4: Spatial Image Guidance. Given a target spatial map, the trained ReaDiT guides the generation process to match the specific layout while maintaining image quality and adherence to the text prompt. Each column is generated from a text prompt following a shared format with varying suffixes, e.g . , column1: “A standing [suffix] in a studio”, where key suffixes are annotated in the figure. Full prompts and more results can be found in the supplementary material.

Figure 5: Qualitative Comparison. Guidance of different spatial tasks from ReaDiT compared against RG [ 19 ] . Generation guided by ReaDiT follows the target more faithfully even under complex spatial maps. More examples are provided in the supplementary material.

		Depth		Pose				Edge
		RMSE ( $\downarrow$ )		PCK@ $\mathbf{0.2}$ ( $\uparrow$ )		mAP ( $\uparrow$ )		ODS ( $\uparrow$ )
Method	Params (M)	Gen	Real	Gen	Real	Gen	Real	Gen	Real
RG [ 19 ]	13.5	0.3704	0.4436	0.4881	0.3380	0.2739	0.1598	0.5783	0.5733
ControlNet [ 39 ]	1487	0.2618	0.3016	0.5012	0.4375	0.2946	0.2563	0.4908	0.5489
ReaDiT (Ours)	53	0.2492	0.3374	0.5380	0.4856	0.3709	0.3412	0.6968	0.6644
ControlNet + ReaDiT (Ours)	1540	0.2595	0.2961	0.4977	0.5386	0.3659	0.4067	0.5688	0.6192

Table 2: Guidance Accuracy Comparison between ReaDiT, RG, and ControlNet. We evaluate under two settings: control maps from generated images (Gen) and from real images (Real). Best model score per setting is bolded ; second best is underlined . ReaDiT outperforms RG across all tasks and metrics under both settings. With significantly fewer parameters, ReaDiT achieves comparable or better accuracy than ControlNet. Furthermore, ReaDiT is complementary to ControlNet, and combining the two further improves performance. We also compare against Flux-based OminiControl and implement our method on Flux.

Figure 6: Spatial Video Guidance. Given a target spatial video input, ReaDiT guides the video generation process to match the specified spatial layout while maintaining visual quality and prompt adherence. Additional examples are shown in the supplementary material.

Figure 7: Motion Video Guidance Results. From left to right: target motion inputs, unguided generation and its captured motion, guided generation and its captured motion. ReaDiT successfully guides video generation to follow the target motion input, as confirmed by the closer correspondence between the motion inputs and motion captured. Without guidance, the generated video fails to reproduce the intended motion.

Figure 8: Consistent Dual Guidance. Depth and pose targets are extracted from the same source image. Compared to single-task guidance, dual guidance better captures both the scene geometry and the human pose.

Figure 9: Inconsistent Dual Guidance. Depth and pose targets are extracted from different source images with potentially conflicting spatial layouts. Rows correspond to different depth targets, while columns to different pose targets. Despite the spatial mismatch, ReaDiT produces coherent, high-quality images that follow both the depth configuration and the pose.

		Depth	Pose		Edge
Method	Params (M)	RMSE ( $\downarrow$ )	PCK@ $\mathbf{0.2}$ ( $\uparrow$ )	mAP ( $\uparrow$ )	ODS ( $\uparrow$ )
RG [ 19 ]	13.5	0.4610	0.1892	0.0496	0.5304
ReaDiT (Ours)	53	0.4225	0.3651	0.1555	0.6101

Table 3: Spatial Video Guidance Comparison between ReaDiT and RG. Best model score is bolded . ReaDiT outperforms RG in all tasks and in all metrics demonstrating its superior performance and extendability to video guided generation.

Method	Params (M)	EPE ( $\downarrow$ )
ImageConductor [ 14 ]	417	30.304
Tora [ 42 ]	1100	16.100
DiTFlow [ 23 ]	0	18.576
ReaDiT (Ours)	53	29.644

Table 4: Motion Video Guidance Comparisons. ReaDiT performs competitively against dedicated motion-control baselines. While our approach does not surpass the state of the art in quantitative metrics, it provides a highly versatile, architecture-agnostic alternative that can be seamlessly integrated with existing frameworks.

Base Model	Spatial Task	Batch Size	Learning Rate	Epochs
SD3-Medium [ 5 ]	Depth, Pose, Edge	8	$1\times 10^{-5}$	2
Flux [ 12 ]	Depth, Pose, Edge	4	$5\times 10^{-5}$	2
CogVideoX [ 37 ]	Depth	4	$5\times 10^{-5}$	3
	Pose			3
	Edge			2
	Motion	1	$1\times 10^{-5}$	2

Table 5: Training Hyperparameters for ReaDiT.

Figure 10: U-Net (SDXL) Up-Block Features vs. DiT (SD3) Features from the last 9 Blocks. U-Net features have much more variation across different layers compared to DiT features, which remain largely identical. Thus, while RG [ 19 ] uses the

n

last blocks, our ReaDiT builds upon the information extracted from a single block.

Figure 11: RMSE (

\downarrow

) vs. Block Index for Depth Prediction Using Features from each DiT Block. RMSE measures the accuracy of depth prediction. Results are shown for SD3-Medium (left) and SD3-Large (right). The red dots are the three lowest RMSE values obtained across all DiT blocks. The dotted red line is the RMSE using the last 10 blocks together. Using a single block to predict depth consistently outperforms using the last 10 blocks together.

Figure 12: Additional Spatial Prediction Results. Qualitative comparison of spatial predictions from ReaDiT against RG [ 19 ] and off-the-shelf task-specific models across depth, pose, and edge tasks. ReaDiT consistently outperforms RG in prediction quality, demonstrating its superior ability to leverage DiT features for spatial understanding.

Figure 13: Additional Spatial Image Guidance. We present additional guided generation results given a target spatial map. The results again show that ReaDiT guides the outputs to matches the target spatial layout across diverse scenes and styles while preserving visual quality.

Figure 14: Additional Qualitative Comparison on Image Guidance. Additional group of guidance comparison between ReaDiT and RG [ 19 ] . ReaDiT consistently follows the target spatial map without artifacts observed in RG.

	Depth		Pose				Edge
	RMSE ( $\downarrow$ )		PCK@ $\mathbf{0.2}$ ( $\uparrow$ )		mAP ( $\uparrow$ )		ODS ( $\uparrow$ )
Method	Gen	Real	Gen	Real	Gen	Real	Gen	Real
RG [ 19 ]	0.3704	0.4436	0.4881	0.3380	0.2739	0.1598	0.5783	0.5733
RG + Our Strategy	0.3720	0.4257	0.4846	0.3595	0.2806	0.1937	0.6992	0.6635
ReaDiT (Ours)	0.2492	0.3374	0.5380	0.4856	0.3709	0.3412	0.6968	0.6644

Table 6: Guidance Accuracy Comparison between ReaDiT, RG, and RG with Our Guidance Strategy. We evaluate under two settings: control maps from generated images (Gen) and from real images (Real). Best score per setting is bolded ; second best is underlined . Applying our guidance strategy to RG improves it on most tasks and metrics, but ReaDiT still outperforms both across nearly all settings.

Figure 15: Comparison of Spatial Image Guidance Results under Different Prompt Settings. ReaDiT adheres to the input spatial target while respecting the text prompt, handling cases where the two partially conflict.

	Depth		Pose
	RMSE ( $\downarrow$ )		PCK@ $\mathbf{0.2}$ ( $\uparrow$ )		mAP ( $\uparrow$ )
Guidance	Gen	Real	Gen	Real	Gen	Real
Depth only	0.3234	0.3658	–	–	–	–
Pose only	–	–	0.3709	0.3412	0.5380	0.4856
Depth + Pose	0.3232	0.3534	0.5773	0.3589	0.6712	0.5017

Table 7: Dual Guidance Quantitative Results (Consistent). Comparison of single-task and dual-task guidance accuracy. Dual guidance achieves comparable or better adherence to both targets, indicating that the two guidance signals are complementary.

Table 8: Influence of Guidance Ratio on Dual Guidance. We vary the ratio between depth and pose guidance losses. A balanced ratio yields the best overall performance across both tasks.

Figure 16: Qualitative Comparison for Spatial Video Guidance. Compared to RG [ 19 ] , ReaDiT demonstrates stronger spatial control across both depth and pose conditions, while generating consistent video sequences.

	Generated		Real
Method	CLIP-T ( $\uparrow$ )	LPIPS ( $\uparrow$ )	CLIP-T ( $\uparrow$ )	LPIPS ( $\uparrow$ )
No Guidance	36.14	0.594	32.32	0.647
Guided (Ours)	35.68	0.592	32.18	0.644

Table 9: Influence of Guidance on Diversity and Text Alignment. Pairwise LPIPS and CLIP-T with and without our method, using spatial maps from generated and real images. Scores are nearly identical, indicating that diversity and text alignment are preserved.

Figure 17: Additional Spatial Video Guidance. Additional examples of spatial video guidance across depth, pose, and edge conditions. ReaDiT successfully guides video generation to follow the target spatial video.

Figure 18: Additional Motion Video Guidance Results. Additional examples of motion-guided generation results, showing zoom-in (top) and pan-right (bottom) camera motions. For each example, we present the unguided and guided results alongside their corresponding captured motion. The motion captured in videos guided by ReaDiT consistently follows the input motion, demonstrating the effectiveness of ReaDiT in motion-guided video generation.

Base Model	Task	Steps	LR
SD3-Medium [ 5 ]	Depth, Pose, Edge	0–34	$1.5\!\times\!10^{-3}$
SD3-Medium [ 5 ]	Depth, Pose, Edge	35–60	$1.0\!\times\!10^{-3}$
Flux [ 12 ]	Depth, Pose, Edge	0–34	$3.0\!\times\!10^{-3}$
Flux [ 12 ]	Depth, Pose, Edge	35–60	$1.0\!\times\!10^{-3}$
CogVideoX [ 37 ]	Depth	0–60	$2.0\!\times\!10^{-3}$
	Pose	0–34	$1.0\!\times\!10^{-3}$

Table 10: Inference Guidance Details. Learning rate (LR) schedule relative to the denoising step range.

Figure 19: Examples of Evaluation Images and Prompts. (a) A real MSCOCO image with its VLM-generated caption. (b) A generated image from SD3-Medium with its text prompt.

ReaDiT Guidance: Control for Image and Video Generation using Diffusion Transformer Features

Abstract

Figures & tables

Explore similar work

Making Time Editable in Video Diffusion Transformers

PE-Field 4D: Video Generation Models as Canvas

Controlling Motion Transfer in Diffusion Transformers via Attention Heads