Designing systems that attribute a generated image to its source model has lately become a major area of AI research. Most existing datasets and systems related to image source attribution are targeted towards the detection of architectural signatures of the source models. Distinct from this, focusing on stable diffusion, we introduce a novel task, IntrAA-SD, Intra-Architecture source Attribution for Stable Diffusion, where the source models to be differentiated share the same architectural backbone and differ only due to fine-tuning. The proposed task aligns with the emergence of fine-tuned generators having the same architecture, like Juggernaut XL and RealVis XL, which are fine-tuned versions of Stable Diffusion XL (SDXL). We provide a first-of-its-kind 140K-image benchmark dataset containing images generated from different fine-tunes of SDXL, along with in-domain and out-of-distribution prompts. We also provide a baseline framework for IntrAA-SD, which leverages a style-conditioned Mixture-of-Experts network. It uses a frozen CLIP backbone to capture the image style and gate the MoE, which processes fine-tuned trace features from a high-resolution LoRA-adapted DINOv2 network. The fine-tuned models used, the prompting strategies considered, and the formation details employed in the novel dataset are elaborated. Our baseline intra-architecture source attribution method is evaluated in terms of attribution accuracy on our dataset, and ablation studies are also performed. Code and dataset is available at https://github.com/mallikagarg/IntrAA-SD.
Figures & tables
Figure 1: Sample images from our IntrAA-SD benchmark dataset generated using identical prompts by SDXL fine-tunes represented by M1: SDXL Base, M2: Juggernaut, M3: RealVis, M4: DreamShaper, M5: Animagine, M6: Counterfeit, M7: ZavyChroma, M8: Copax, M9: Proteus, M10: AAM Anime. P1: “ volcanic landscape with lava flow ”, P2: “ electric car charging at a station ”. Despite sharing the same backbone, each fine-tune produces outputs with subtle visual distinctness with some pairs (like M2 & M3) more similar than others. Note: We use M1-M10 to denote the corresponding fine-tuned models throughout the paper.
“ bowl of ramen with egg, nori, chopsticks, steam rising ”
Table 2: In-Domain (ID) prompt categories in our ISDB dataset with example prompts.
#
Category
Example Prompt
Adversarial Strategies (7 categories)
1
Cross-Domain
“ studio portrait, soft lighting, 85mm lens ” → fed to Animagine XL; “ flat anime character, cel shading ” → fed to RealVis XL
2
Void Prompts
“ pure white background, nothing else”; “complete darkness, empty frame ”
3
Nested Representation
“ a photograph of a charcoal sketch pinned to a cork board ”; “ a watercolor painting lying on a wooden desk ”
4
Text-Heavy
“ a handwritten letter on aged parchment, cursive text filling the page ”
5
Hands & Anatomy
“ a pianist playing a grand piano, fingers on keys, concert hall ”
Table 3: Out-of-Distribution (OOD) prompt categories in our ISDB dataset with example prompts.
Figure 2: Our IntrAA-SD Baseline Framework. A frozen CLIP backbone captures the image style (Sec. 4.1 ) to compute gating weights across N experts. A LoRA-adapted DINOv2 backbone extracts the fine-tunes’ related patterns fed into experts (Sec. 4.2 ). The expert outputs are combined via style-conditioned Mixture-of-Experts (Sec. 4.3 ) and concatenated with the style context for the attribution (Sec. 4.4 ).
Source Model
ID
OOD
SDXL Base (M1)
99.42
91.50
Juggernaut XL (M2)
99.79
70.50
RealVis XL (M3)
98.67
82.50
DreamShaper XL (M4)
99.83
99.00
Animagine XL (M5)
99.92
98.00
CounterfeitXL (M6)
98.29
98.00
Table 4: Intra-architecture source attribution accuracies (%) of our IntrAA-SD baseline framework on our ISDB dataset, for the test images generated with In-Domain (ID) and Out-of-Distribution (OOD) prompts.
Components
Variants
Acc %
Gating
A1: Equal weighting (no gating)
98.40
strategy
A2: Structure-based gating (DINOv2 → gate)
98.42
Style concat
A3: No style concatenation (after MoE) before classification
98.21
Style or
A4: Image Style-only (Frozen CLIP + linear head)
87.59
trace feature
A5: Source trace-only (LoRA DINOv2 + linear head)
98.52
All
IntrAA-SD framework
99.03
Table 5: IntrAA-SD Ablation: significance of components. All variants use LoRA r=16, 518px, 4 experts (unless removed).
Table 7: Ablation on the loss function ( Ltotal ) parameter λbal
Figure 3: t-SNE visualization of model trace feature embeddings.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Method
M1
M2
M3
M4
M5
M6
M7
M8
M9
M10
Overall
ID
Spec [ 15 ]
18.13
12.96
11.86
46.45
40.83
36.51
9.31
20.49
32.01
21.32
24.99
UnivFD [ 30 ]
24.46
27.79
30.32
70.52
76.47
29.73
33.85
35.62
56.71
61.41
44.69
DE-FAKE [ 41 ]
62.96
65.37
61.32
89.58
93.91
76.24
62.28
67.97
87.87
86.22
75.37
NPR [ 46 ]
88.27
85.53
86.78
97.97
96.56
94.03
86.51
89.81
94.42
95.08
91.50
CNNDetect [ 48 ]
87.26
83.93
88.96
98.32
98.40
95.38
87.01
90.27
96.34
96.28
92.21
Ours ∗
99.42
99.79
98.67
99.83
99.92
98.29
99.67
97.88
98.58
98.21
99.03
Appendix
Table 1: Intra-architecture source attribution accuracy (%) on our ISDB test set. Models - M1: SDXL Base, M2: Juggernaut, M3: RealVis, M4: DreamShaper, M5: Animagine, M6: Counterfeit, M7: ZavyChroma, M8: Copax, M9: Proteus, M10: AAM Anime. ∗ Ours is the only algorithm (& the first) in the list which is designed for intra-architecture source attribution. All others listed are existing architecture-level source attribution methods.
Figure 1: t-SNE projection of frozen CLIP ViT-L/14 features of input images generated using a suitable shared prompt across all 10 SDXL fine-tunes.
Experts
Photorealistic
Stylized
Anime
M4
M2
M1
M3
avg.
M8
M7
M9
avg.
M6
M10
M5
avg.
Expert 1
35.5
36.7
37.1
37.6
36.7
37.2
36.6
34.9
36.2
33.5
28.1
15.8
25.8
Expert 2
24.9
24.8
23.0
25.2
24.5
26.3
26.8
27.2
26.8
27.2
29.2
27.4
27.9
Expert 3
23.7
21.6
23.6
21.2
22.5
18.8
19.5
19.6
19.3
17.8
17.4
13.3
16.2
Expert 4
15.9
16.9
16.2
16.1
16.3
17.7
17.1
18.3
17.7
21.6
25.3
43.6
30.2
Appendix
Table 2: Mean Mixture of Expert (MoE) gating weights (%) per model, grouped & averaged by style type.
Training Set
Test Set Accuracy%
ISDB-ID
ImageNet
COCO
ImageNet
80.59
98.35
96.47
COCO
76.16
90.31
98.91
ISDB-ID (Ours)
99.03
84.58
85.95
Appendix
Table 3: Cross-Dataset Performance of Our IntrAA-SD baseline
True ↓ / Pred →
M1
M2
M3
M4
M5
M6
M7
M8
M9
M10
M1 (SDXL Base)
923
3
2
0
1
5
11
46
5
4
M2 (Juggernaut)
22
708
149
8
0
14
49
28
16
6
M3 (RealVis)
16
87
816
4
0
12
35
18
9
3
M4 (DreamShaper)
0
5
0
985
0
0
4
0
1
5
M5 (Animagine)
4
0
0
0
984
1
3
0
0
8
M6 (Counterfeit)
1
3
0
1
2
978
5
0
3
7
Appendix
Table 4: Confusion Matrix related to performance of our IntrAA-Diff on ISDB-ODD set. Photorealistic: M1-4 , Stylized: M7-9 , Anime: M5-6, M10 .
Figure 2: Significant changes (Red: increase, Blue: decrease) at patch-level feature norm after LoRA adaptation of DINOv2 for images from four different fine-tuned models.