Multi-label classification has broad applications and depends on powerful representations capable of capturing multiple interactions. We introduce Diff-Feat, a simple yet effective framework that extracts intermediate features from pre-trained diffusion-Transformer models for both images and text, and fuses them for downstream tasks. We observe distinct patterns in optimal feature selection: for vision tasks, the most discriminative intermediate feature along the diffusion process occurs at the middle step and is located at the middle block in Transformer. In contrast, for language tasks, the best feature occurs at the noise-free step and is located at the deepest block. Notably, a consistent phenomenon across diverse datasets: a fixed "Middle Layer" yields the best performance on various downstream classification tasks for images (under DiT-XL/2-256×256 and various U-ViT architectures). We further analyze the underlying mechanism and its scaling behavior across model scales. To efficiently select representations, we devise a heuristic local-search algorithm that pinpoints the locally optimal "image-text"×"block-timestep" pair among a few candidates, avoiding an exhaustive grid search. A simple fusion-linear projection followed by addition-of the selected representations achieves 46.1% mAP on Visual Genome 500 (with 500 categories) and 80.2% mAP on Pix2Cap-COCO (with 80 categories), consistently outperforming strong baseline methods. Code is available at https://github.com/lt-0123/Diff-Feat.
Figures & tables
Fig. 1 : Overview of the Diff-Feat framework.
Fig. 2 : Multi-label classification performance of image diffusion representations across different Transformer blocks and diffusion timesteps on the VG500. Brighter regions indicate higher mAP (left) or OP (right) values, highlighting the optimal selection of features.
Fig. 3 : Multi-label classification performance of text diffusion representations across different Transformer blocks and diffusion timesteps on the VG500. Brighter regions indicate higher mAP (left) or OP (right) values.
Fusion Method
mAP
CP
CR
CF1
OP
OR
OF1
Simple Concat
78.0
85.5
59.9
69.5
91.6
67.8
77.91
Linear Concat
79.0
82.8
68.0
74.2
89.0
74.9
81.3
Cross Attention
76.8
83.6
64.9
72.5
89.9
72.1
80.0
Linear Add
79.6
82.9
68.2
74.4
89.2
75.1
81.6
TABLE I : Ablation study of fusion methods on Pix2Cap-COCO ( timg=50,bimg=12;ttxt=0,btxt=24 ).
Fig. 4 : Multi-label classification performance (mAP) on VG500 using Linear Addition fusion across different image–text representation pairs. Each group denotes an image representation (e.g., (t50-b12) ), and each bar corresponds to a text representation. The best combination in each group is highlighted with white diagonal stripes.
Method
mAP(%)
ResNet-101 ( He et al., 2016 )
30.9
ResNet-SRN ( Zhu et al., 2017 )
33.5
SS-GRL ( Chen et al., 2019a )
36.6
C-Tran ( Lanchantin et al., 2021 )
38.4
DRGN ( Zhou et al., 2024a )
39.8
DATran ( Zhou et al., 2024b )
40.1
TABLE II : Comparison with strong baseline models on VG500.
Method
mAP
CP
CR
CF1
OP
OR
OF1
ResNet-50 + LLaMA3.2 1B (LP)
70.4
73.9
58.1
64.6
82.3
65.9
73.2
CLIP (LP)
76.3
84.4
57.8
67.1
91.8
65.9
76.7
ViLT (LP)
73.0
84.0
56.9
66.6
90.7
64.0
75.1
Diff-Feat (Ours, LP)
79.6
82.9
68.2
74.4
89.2
75.1
81.6
Diff-Feat (Ours, FT)
80.2
83.7
68.0
74.7
89.5
75.0
81.6
TABLE III : Comparison with strong baseline models on Pix2Cap-COCO.
Fig. 5 : Heatmap visualizations of classification performance across diffusion timesteps and Transformer blocks on multiple image benchmarks.
Model Architecture
Optimal Layer
Total Layers
mAP
CIFAR10 (U-ViT-S/2)
10
13
28.24
CelebA 64x64 (U-ViT-S/4)
9
13
27.64
ImageNet 256x256 (U-ViT-L/2)
12
21
59.15
ImageNet 512x512 (U-ViT-L/4)
11
21
54.88
ImageNet 256x256 (U-ViT-H/2)
16
29
64.32
ImageNet 512x512 (U-ViT-H/4)
16
29
61.16
TABLE IV : Optimal Layer Selection Across Different Architectures at Diffusion Timestep t=50 on the MS-COCO dataset.
Fig. 6 : Robustness comparison between coarse- and fine-grained labels along two orthogonal axes of DiT on the MS-COCO dataset.
Fig. 7 : Semantic stability of text embeddings under different diffusion timestep at a fixed block depth b=24 on the VG500.
Fig. 8 : Attention map visualization across diffusion timesteps and Transformer blocks for two representative samples in the MS-COCO. The maps demonstrate how the model gradually focuses on semantically meaningful regions.