Multi-label classification has broad applications and depends on powerful representations capable of capturing multiple interactions. We introduce Diff-Feat, a simple yet effective framework that extracts intermediate features from pre-trained diffusion-Transformer models for both images and text, and fuses them for downstream tasks. We observe distinct patterns in optimal feature selection: for vision tasks, the most discriminative intermediate feature along the diffusion process occurs at the middle step and is located at the middle block in Transformer. In contrast, for language tasks, the best feature occurs at the noise-free step and is located at the deepest block. Notably, a consistent phenomenon across diverse datasets: a fixed "Middle Layer" yields the best performance on various downstream classification tasks for images (under DiT-XL/2-256×256 and various U-ViT architectures). We further analyze the underlying mechanism and its scaling behavior across model scales. To efficiently select representations, we devise a heuristic local-search algorithm that pinpoints the locally optimal "image-text"×"block-timestep" pair among a few candidates, avoiding an exhaustive grid search. A simple fusion-linear projection followed by addition-of the selected representations achieves 46.1% mAP on Visual Genome 500 (with 500 categories) and 80.2% mAP on Pix2Cap-COCO (with 80 categories), consistently outperforming strong baseline methods. Code is available at https://github.com/lt-0123/Diff-Feat.
Figures & tables
Fig. 1 : Overview of the Diff-Feat framework.
Fig. 2 : Multi-label classification performance of image diffusion representations across different Transformer blocks and diffusion timesteps on the VG500. Brighter regions indicate higher mAP (left) or OP (right) values, highlighting the optimal selection of features.
Fig. 3 : Multi-label classification performance of text diffusion representations across different Transformer blocks and diffusion timesteps on the VG500. Brighter regions indicate higher mAP (left) or OP (right) values.
Fusion Method
mAP
CP
CR
CF1
OP
OR
OF1
Simple Concat
78.0
85.5
59.9
69.5
91.6
67.8
77.91
Linear Concat
79.0
82.8
68.0
74.2
89.0
74.9
81.3
Cross Attention
76.8
83.6
64.9
72.5
89.9
72.1
80.0
Linear Add
79.6
82.9
68.2
74.4
89.2
75.1
81.6
TABLE I : Ablation study of fusion methods on Pix2Cap-COCO ( timg=50,bimg=12;ttxt=0,btxt=24 ).
Fig. 4 : Multi-label classification performance (mAP) on VG500 using Linear Addition fusion across different image–text representation pairs. Each group denotes an image representation (e.g., (t50-b12) ), and each bar corresponds to a text representation. The best combination in each group is highlighted with white diagonal stripes.
Method
mAP(%)
ResNet-101 ( He et al., 2016 )
30.9
ResNet-SRN ( Zhu et al., 2017 )
33.5
SS-GRL ( Chen et al., 2019a )
36.6
C-Tran ( Lanchantin et al., 2021 )
38.4
DRGN ( Zhou et al., 2024a )
39.8
DATran ( Zhou et al., 2024b )
40.1
TABLE II : Comparison with strong baseline models on VG500.
Method
mAP
CP
CR
CF1
OP
OR
OF1
ResNet-50 + LLaMA3.2 1B (LP)
70.4
73.9
58.1
64.6
82.3
65.9
73.2
CLIP (LP)
76.3
84.4
57.8
67.1
91.8
65.9
76.7
ViLT (LP)
73.0
84.0
56.9
66.6
90.7
64.0
75.1
Diff-Feat (Ours, LP)
79.6
82.9
68.2
74.4
89.2
75.1
81.6
Diff-Feat (Ours, FT)
80.2
83.7
68.0
74.7
89.5
75.0
81.6
TABLE III : Comparison with strong baseline models on Pix2Cap-COCO.
Fig. 5 : Heatmap visualizations of classification performance across diffusion timesteps and Transformer blocks on multiple image benchmarks.
Model Architecture
Optimal Layer
Total Layers
mAP
CIFAR10 (U-ViT-S/2)
10
13
28.24
CelebA 64x64 (U-ViT-S/4)
9
13
27.64
ImageNet 256x256 (U-ViT-L/2)
12
21
59.15
ImageNet 512x512 (U-ViT-L/4)
11
21
54.88
ImageNet 256x256 (U-ViT-H/2)
16
29
64.32
ImageNet 512x512 (U-ViT-H/4)
16
29
61.16
TABLE IV : Optimal Layer Selection Across Different Architectures at Diffusion Timestep t=50 on the MS-COCO dataset.
Fig. 6 : Robustness comparison between coarse- and fine-grained labels along two orthogonal axes of DiT on the MS-COCO dataset.
Fig. 7 : Semantic stability of text embeddings under different diffusion timestep at a fixed block depth b=24 on the VG500.
Fig. 8 : Attention map visualization across diffusion timesteps and Transformer blocks for two representative samples in the MS-COCO. The maps demonstrate how the model gradually focuses on semantically meaningful regions.
Diffusion transformers have demonstrated remarkable generative capabilities, yet the rich perceptual representations computed across their denoising trajectory are discarded once the content is rendered. We present MMDiff, a framework that transforms a frozen diffusion transformer into a multi-modal generative system that jointly produces images alongside any combination of dense perceptual modalities using lightweight decoder heads. Our central finding is that perceptual information is temporally distributed along the denoising trajectory, and that multi-timestep feature fusion with spatially varying aggregation weights is essential, improving semantic segmentation results by up to 28.7% mIoU over single-timestep extraction. We further adopt concept-driven attention extraction for interpretable spatial guidance, and show that frozen diffusion features are competitive with and complementary to state-of-the-art encoders such as DINOv3. By training only lightweight decoder heads on a frozen backbone, we achieve strong performance in semantic segmentation, salient object detection, and depth estimation, and demonstrate that this framework enables effective synthetic data generation at scale.
The platonic representation hypothesis suggests that sufficiently large models converge to a shared representation geometry, even across modalities. Motivated by this, we ask: Can the semantic knowledge of a language model efficiently improve a vision model? As an answer, we introduce TextTeacher, a simple auxiliary objective that injects text embeddings as additional information into image classification training. TextTeacher uses readily available image captions, a pre-trained and frozen text encoder, and a lightweight projection to produce semantic anchors that efficiently guide representations during training while leaving the inference-time model unchanged. On ImageNet with standard ViT backbones, TextTeacher improves accuracy by up to +2.7 percentage points (p.p.) and yields consistent transfer gains (on average +1.0 p.p.) under the same recipe and compute. It outperforms vision knowledge distillation, yielding more accuracy at a constant compute budget or similar accuracy, but 33% faster. Our analysis indicates that TextTeacher acts as a feature-space preconditioner, shaping deeper layers in the first stages of training, and aiding generalization by supplying complementary semantic cues. TextTeacher adds negligible overhead, requires no costly multimodal training of the target model and preserves the simplicity and latency of pure vision models. Project page with code and captions: https://nauen-it.de/publications/text-teacher
Tobias Christian Nauen, Stanislav Frolov, Brian Bernhard Moser +3
RPTU University Kaiserslautern-Landau · German Research Center for Artificial Intelligence (DFKI)
Diffusion Transformers (DiT) trained with flow matching in a VAE latent space have unified visual generation across images and videos. A natural next step toward a single architecture for both generation (visual synthesis) and understanding (text generation) is to apply this framework to language modeling. We propose TextLDM, which transfers the visual latent diffusion recipe to text generation with minimal architectural modification. A Transformer-based VAE maps discrete tokens to continuous latents, enhanced by Representation Alignment (REPA) with a frozen pretrained language model to produce representations effective for conditional denoising. A standard DiT then performs flow matching in this latent space, identical in architecture to its visual counterpart. The central challenge we address is obtaining high-quality continuous text representations: we find that reconstruction fidelity alone is insufficient, and that aligning latent features with a pretrained language model via REPA is critical for downstream generation quality. Trained from scratch on OpenWebText2, TextLDM substantially outperforms prior diffusion language models and matches GPT-2 under the same settings. Our results establish that the visual DiT recipe transfers effectively to language, taking a concrete step toward unified diffusion architectures for multimodal generation and understanding.