We present DiT Readout (ReaDiT) Guidance, a lightweight framework for controlling generation with Diffusion Transformer (DiT) models via their internal feature representations. ReaDiT Guidance uses features from a single DiT block to steer the generative process according to spatial targets - like depth, pose, or edge maps - provided at test time. Furthermore, since modern text-to-video models are largely built on DiT backbones, ReaDiT Guidance naturally extends to video generation, enabling camera and motion control. Experimental results demonstrate that our approach achieves competitive or improved results compared to existing feature-based and off-the-shelf adapter-based approaches while requiring fewer parameters.
Figures & tables
Method
Image
Video
Params (M)
Data
Implicit Model Conditioning
OminiControl
✓
✗
14.5
Small
✗
ControlNet (Image)
✓
✗
1487
Small
✗
ControlNet (Video)
✗
✓
2500
Large
✗
RG
✓
✗
13.5
Small
✓
DiTFlow
✗
✓ (Motion only)
0
None
✗
Tora
✗
✓ (Motion only)
1100
Large
✗
Table 1: Comparison of DiT Control Approaches. Overview of supported modalities, parameter overhead, training data requirements, and whether the method utilizes Implicit Model Conditioning (i.e., the diffusion model does not process the condition during its forward pass). For ControlNet, parameters for image and video are estimated from SD3-Medium and CogVideoX-5B respectively.
Figure 1: Method Overview. Left: ReaDiT takes a single intermediate feature from the frozen DiT and predicts spatial outputs, which are used to guide image generation by optimizing the latent zt . Right: ReaDiT is trained to predict task-specific outputs from DiT features extracted by noising an input image to timestep t and passing it through the frozen DiT.
Figure 2: Architecture of ReaDiT. Input features are conditioned on t and encoded into multi-resolution feature maps. The feature maps are progressively fused and upsampled to a final feature map, from which a task-specific head produces the spatial output.
Figure 3: Spatial Prediction Results. We compare the prediction of ReaDiT to off-the-shelf models across depth, pose, and edge tasks. ReaDiT produces meaningful predictions from frozen DiT features, which can also be used to monitor spatial properties during the generation process. Additional examples are presented in the supplementary material.
Figure 4: Spatial Image Guidance. Given a target spatial map, the trained ReaDiT guides the generation process to match the specific layout while maintaining image quality and adherence to the text prompt. Each column is generated from a text prompt following a shared format with varying suffixes, e.g . , column1: “A standing [suffix] in a studio”, where key suffixes are annotated in the figure. Full prompts and more results can be found in the supplementary material.
Figure 5: Qualitative Comparison. Guidance of different spatial tasks from ReaDiT compared against RG [ 19 ] . Generation guided by ReaDiT follows the target more faithfully even under complex spatial maps. More examples are provided in the supplementary material.
Depth
Pose
Edge
RMSE ( ↓ )
PCK@ 0.2 ( ↑ )
mAP ( ↑ )
ODS ( ↑ )
Method
Params (M)
Gen
Real
Gen
Real
Gen
Real
Gen
Real
RG [ 19 ]
13.5
0.3704
0.4436
0.4881
0.3380
0.2739
0.1598
0.5783
0.5733
ControlNet [ 39 ]
1487
0.2618
0.3016
0.5012
0.4375
0.2946
0.2563
0.4908
0.5489
ReaDiT (Ours)
53
0.2492
0.3374
0.5380
0.4856
0.3709
0.3412
0.6968
0.6644
ControlNet + ReaDiT (Ours)
1540
0.2595
0.2961
0.4977
0.5386
0.3659
0.4067
0.5688
0.6192
Table 2: Guidance Accuracy Comparison between ReaDiT, RG, and ControlNet. We evaluate under two settings: control maps from generated images (Gen) and from real images (Real). Best model score per setting is bolded ; second best is underlined . ReaDiT outperforms RG across all tasks and metrics under both settings. With significantly fewer parameters, ReaDiT achieves comparable or better accuracy than ControlNet. Furthermore, ReaDiT is complementary to ControlNet, and combining the two further improves performance. We also compare against Flux-based OminiControl and implement our method on Flux.
Figure 6: Spatial Video Guidance. Given a target spatial video input, ReaDiT guides the video generation process to match the specified spatial layout while maintaining visual quality and prompt adherence. Additional examples are shown in the supplementary material.
Figure 7: Motion Video Guidance Results. From left to right: target motion inputs, unguided generation and its captured motion, guided generation and its captured motion. ReaDiT successfully guides video generation to follow the target motion input, as confirmed by the closer correspondence between the motion inputs and motion captured. Without guidance, the generated video fails to reproduce the intended motion.
Figure 8: Consistent Dual Guidance. Depth and pose targets are extracted from the same source image. Compared to single-task guidance, dual guidance better captures both the scene geometry and the human pose.
Figure 9: Inconsistent Dual Guidance. Depth and pose targets are extracted from different source images with potentially conflicting spatial layouts. Rows correspond to different depth targets, while columns to different pose targets. Despite the spatial mismatch, ReaDiT produces coherent, high-quality images that follow both the depth configuration and the pose.
Depth
Pose
Edge
Method
Params (M)
RMSE ( ↓ )
PCK@ 0.2 ( ↑ )
mAP ( ↑ )
ODS ( ↑ )
RG [ 19 ]
13.5
0.4610
0.1892
0.0496
0.5304
ReaDiT (Ours)
53
0.4225
0.3651
0.1555
0.6101
Table 3: Spatial Video Guidance Comparison between ReaDiT and RG. Best model score is bolded . ReaDiT outperforms RG in all tasks and in all metrics demonstrating its superior performance and extendability to video guided generation.
Method
Params (M)
EPE ( ↓ )
ImageConductor [ 14 ]
417
30.304
Tora [ 42 ]
1100
16.100
DiTFlow [ 23 ]
0
18.576
ReaDiT (Ours)
53
29.644
Table 4: Motion Video Guidance Comparisons. ReaDiT performs competitively against dedicated motion-control baselines. While our approach does not surpass the state of the art in quantitative metrics, it provides a highly versatile, architecture-agnostic alternative that can be seamlessly integrated with existing frameworks.
Base Model
Spatial Task
Batch Size
Learning Rate
Epochs
SD3-Medium [ 5 ]
Depth, Pose, Edge
8
1×10−5
2
Flux [ 12 ]
Depth, Pose, Edge
4
5×10−5
2
CogVideoX [ 37 ]
Depth
4
5×10−5
3
Pose
3
Edge
2
Motion
1
1×10−5
2
Table 5: Training Hyperparameters for ReaDiT.
Figure 10: U-Net (SDXL) Up-Block Features vs. DiT (SD3) Features from the last 9 Blocks. U-Net features have much more variation across different layers compared to DiT features, which remain largely identical. Thus, while RG [ 19 ] uses the n last blocks, our ReaDiT builds upon the information extracted from a single block.
Figure 11: RMSE ( ↓ ) vs. Block Index for Depth Prediction Using Features from each DiT Block. RMSE measures the accuracy of depth prediction. Results are shown for SD3-Medium (left) and SD3-Large (right). The red dots are the three lowest RMSE values obtained across all DiT blocks. The dotted red line is the RMSE using the last 10 blocks together. Using a single block to predict depth consistently outperforms using the last 10 blocks together.
Figure 12: Additional Spatial Prediction Results. Qualitative comparison of spatial predictions from ReaDiT against RG [ 19 ] and off-the-shelf task-specific models across depth, pose, and edge tasks. ReaDiT consistently outperforms RG in prediction quality, demonstrating its superior ability to leverage DiT features for spatial understanding.
Figure 13: Additional Spatial Image Guidance. We present additional guided generation results given a target spatial map. The results again show that ReaDiT guides the outputs to matches the target spatial layout across diverse scenes and styles while preserving visual quality.
Figure 14: Additional Qualitative Comparison on Image Guidance. Additional group of guidance comparison between ReaDiT and RG [ 19 ] . ReaDiT consistently follows the target spatial map without artifacts observed in RG.
Depth
Pose
Edge
RMSE ( ↓ )
PCK@ 0.2 ( ↑ )
mAP ( ↑ )
ODS ( ↑ )
Method
Gen
Real
Gen
Real
Gen
Real
Gen
Real
RG [ 19 ]
0.3704
0.4436
0.4881
0.3380
0.2739
0.1598
0.5783
0.5733
RG + Our Strategy
0.3720
0.4257
0.4846
0.3595
0.2806
0.1937
0.6992
0.6635
ReaDiT (Ours)
0.2492
0.3374
0.5380
0.4856
0.3709
0.3412
0.6968
0.6644
Table 6: Guidance Accuracy Comparison between ReaDiT, RG, and RG with Our Guidance Strategy. We evaluate under two settings: control maps from generated images (Gen) and from real images (Real). Best score per setting is bolded ; second best is underlined . Applying our guidance strategy to RG improves it on most tasks and metrics, but ReaDiT still outperforms both across nearly all settings.
Figure 15: Comparison of Spatial Image Guidance Results under Different Prompt Settings. ReaDiT adheres to the input spatial target while respecting the text prompt, handling cases where the two partially conflict.
Depth
Pose
RMSE ( ↓ )
PCK@ 0.2 ( ↑ )
mAP ( ↑ )
Guidance
Gen
Real
Gen
Real
Gen
Real
Depth only
0.3234
0.3658
–
–
–
–
Pose only
–
–
0.3709
0.3412
0.5380
0.4856
Depth + Pose
0.3232
0.3534
0.5773
0.3589
0.6712
0.5017
Table 7: Dual Guidance Quantitative Results (Consistent). Comparison of single-task and dual-task guidance accuracy. Dual guidance achieves comparable or better adherence to both targets, indicating that the two guidance signals are complementary.
Table 8: Influence of Guidance Ratio on Dual Guidance. We vary the ratio between depth and pose guidance losses. A balanced ratio yields the best overall performance across both tasks.
Figure 16: Qualitative Comparison for Spatial Video Guidance. Compared to RG [ 19 ] , ReaDiT demonstrates stronger spatial control across both depth and pose conditions, while generating consistent video sequences.
Generated
Real
Method
CLIP-T ( ↑ )
LPIPS ( ↑ )
CLIP-T ( ↑ )
LPIPS ( ↑ )
No Guidance
36.14
0.594
32.32
0.647
Guided (Ours)
35.68
0.592
32.18
0.644
Table 9: Influence of Guidance on Diversity and Text Alignment. Pairwise LPIPS and CLIP-T with and without our method, using spatial maps from generated and real images. Scores are nearly identical, indicating that diversity and text alignment are preserved.
Figure 17: Additional Spatial Video Guidance. Additional examples of spatial video guidance across depth, pose, and edge conditions. ReaDiT successfully guides video generation to follow the target spatial video.
Figure 18: Additional Motion Video Guidance Results. Additional examples of motion-guided generation results, showing zoom-in (top) and pan-right (bottom) camera motions. For each example, we present the unguided and guided results alongside their corresponding captured motion. The motion captured in videos guided by ReaDiT consistently follows the input motion, demonstrating the effectiveness of ReaDiT in motion-guided video generation.
Base Model
Task
Steps
LR
SD3-Medium [ 5 ]
Depth, Pose, Edge
0–34
1.5×10−3
35–60
1.0×10−3
Flux [ 12 ]
Depth, Pose, Edge
0–34
3.0×10−3
35–60
1.0×10−3
CogVideoX [ 37 ]
Depth
0–60
2.0×10−3
Pose
0–34
1.0×10−3
Table 10: Inference Guidance Details. Learning rate (LR) schedule relative to the denoising step range.
Figure 19: Examples of Evaluation Images and Prompts. (a) A real MSCOCO image with its VLM-generated caption. (b) A generated image from SD3-Medium with its text prompt.
Modern Diffusion Transformers for video generation provide limited control over the progression of time and the editing of temporal dynamics. We propose a temporal-control methodology that extends a pretrained DiT with explicit time editing, allowing control over motion speed and temporal structure without redesigning the backbone. Its core implementation augments the pretrained model with a lightweight temporal module, preserving the original generative prior while expanding its controllable dynamic range.
Konstantin Kuklev, Viacheslav Vasilev, Alexander Kunitsyn +2
Diffusion Transformers have recently achieved strong performance in video generation, yet controlling scene geometry under viewpoint changes and camera motion remains challenging. In this work, we revisit the role of positional encoding in video diffusion transformers and show that it provides a useful spatial bias for geometry-aware control. Specifically, if reference tokens are encoded according to their projected locations in the target view, the denoising model is encouraged to retrieve content from position aligned regions of the input video. Building on this observation, we introduce a geometry-aware cross-attention mechanism that enables target video latent tokens to attend to structured context tokens derived from reference images or frames. To establish correspondence between the reference content and the target camera trajectory, we equip the context tokens with a projected positional encoding scheme that combines target-view 2D reprojection with depth-aware disambiguation. At the same time, we preserve the original spatiotemporal positional encoding of the generated video latent, allowing geometric guidance to be injected while maintaining consistency with the video model's native latent structure. The resulting framework provides a simple and effective approach for controllable video generation. It improves spatial controllability in viewpoint-dependent editing tasks, including camera re-trajectory, novel-view video synthesis, and geometry-aware video editing, while preserving the generative prior of the underlying video diffusion model. The code is available at: https://github.com/MTLab/PE-Field.
Yunpeng Bai, Haoxiang Li, Qixing Huang
University of Texas at Austin, USA · Pixocial Technology, USA
Diffusion Transformers (DiTs) have advanced video generation with high-quality, temporally coherent results. However, extending them to motion transfer, which requires following reference motion while aligning with a target prompt, remains challenging due to limited understanding of motion and structure representations within DiTs. We analyze video DiTs at the attention-head level and identify distinct heads specialized for motion and spatial structure. Based on this insight, we propose a head-aware controllable motion transfer framework that requires no parameter updates. Our method refines motion cues from motion-specialized heads via semantic correspondence guidance and preserves structure through selective feature injection. This head-level control not only enables accurate motion transfer but also provides an interpretable foundation for controllable video generation with DiTs.
Sunyoung Jung, Jiwoo Park, Yoonseok Choi +3
Yonsei University · LG Electronics · University of California, Merced