Structural Self-Teaching for Compositional Generation in Unified Multimodal Models
Organizations: Seoul National University · Samsung Electronics
Abstract
We propose Structural Self-Teaching (SST), a novel framework that enhances compositional image generation in diffusion-based unified multimodal models (UMMs) by leveraging their internal understanding path as a source of structural supervision. The generation path of UMMs often struggles with dense compositional prompts, leading to missing objects, attribute binding failures, and spatial errors. In contrast, their understanding path localizes the same phrases in images that satisfy these prompts. SST converts this phrase-level grounding into training-time structural signals for the generation path. Specifically, we extract phrase-conditioned support masks from internal attention maps derived from the understanding path and employ them through two mechanisms. Phrase-local contrastive alignment uses these masks as pooling regions and aligns each generated phrase region with the same region of the target image. A structural scaffold builds a phrase-level layout from the same masks and modulates an intermediate spatial state of the generation path during training. The scaffold is phased out during training and removed at inference. SST requires no external structural annotations for training and no additional input at test time. Extensive experiments across multiple UMMs demonstrate that SST improves compositional fidelity over LoRA fine-tuning on the same data, with no inference-time cost. Project page is available at https://bellos1203.github.io/sst
Figures & tables
| Method | AM | GM | Category | Atomicity | |||||||||||
| Obj. | Attr. | Count | Pos. | Verb | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | |||
| BLIP-3o 4B | 53.87 | 11.78 | 77.94 | 52.87 | 30.72 | 35.06 | 15.44 | 34.78 | 28.21 | 14.22 | 6.14 | 3.84 | 3.53 | 2.25 | 1.26 |
| + LoRA FT | 54.70 0.18 | 13.00 0.35 | 76.62 | 54.24 | 31.64 | 36.33 | 13.37 | 39.34 | 28.24 | 14.65 | 10.71 | 4.33 | 3.46 | 1.90 | 1.44 |
| + SST (Ours) | 58.29 0.15 | 16.13 0.28 | 81.29 | 54.49 | 37.17 | 41.53 | 17.03 | 47.41 | 33.19 | 16.92 | 14.43 | 5.72 | 4.53 | 3.93 | 2.92 |
| BLIP-3o 8B | 59.35 | 13.50 | 85.08 | 57.79 | 33.46 | 47.13 | 20.34 | 31.63 | 30.90 | 15.59 | 12.30 | 6.79 | 5.61 | 2.67 | 2.52 |
| + LoRA FT | 60.48 0.23 | 14.86 0.48 | 86.31 | 58.00 | 34.84 | 46.94 | 21.22 | 33.05 | 32.77 | 16.25 | 13.87 | 7.49 | 6.92 | 5.23 | 3.27 |
| Method | AM | GM | Object | Attribute | Count | Position | Verb | |
| BLIP-3o 4B | LoRA FT | 54.70 0.18 | 13.00 0.35 | 76.62 0.42 | 54.24 0.68 | 31.64 0.44 | 36.33 0.59 | 13.37 0.66 |
| Phrase-local only | 56.04 0.34 | 15.42 0.41 | 79.88 0.52 | 54.23 0.37 | 35.77 0.66 | 37.98 0.31 | 17.28 0.84 | |
| Scaffold only | 55.24 0.28 | 14.43 0.33 | 77.42 0.34 | 53.94 0.24 | 33.12 0.56 | 38.67 0.44 | 14.42 0.83 | |
| Full SST | 58.29 0.15 | 16.13 0.28 | 81.29 0.25 | 54.49 0.20 | 37.17 0.59 | 41.53 0.42 | 17.03 0.78 | |
| BAGEL | LoRA FT | 73.01 0.31 | 26.47 0.21 | 93.23 0.41 | 68.43 0.49 | 57.39 1.27 | 52.71 0.55 | 25.32 1.09 |
| Phrase-local only | 73.69 0.30 | 27.12 0.39 | 93.47 0.35 | 70.24 0.58 | 58.20 0.41 | 52.78 0.36 | 26.47 0.71 |
| Mask source | AM | GM | Object | Attribute | Count | Position | Verb |
| None (LoRA FT) | 54.56 | 13.35 | 76.22 | 53.60 | 31.40 | 36.29 | 13.60 |
| Grounded-SAM (external) | 59.58 | 16.84 | 83.09 | 55.68 | 35.28 | 40.52 | 18.64 |
| Attention-derived (SST) | 58.34 | 16.05 | 81.43 | 54.34 | 37.32 | 41.57 | 16.29 |
| (SST external) |
| BLIP-3o variants | Show-o2 / BAGEL | |||||||||
| 1 | 5 | 10 | 15 | 0.01 | 0.1 | 0.2 | 0.5 | |||
| BLIP-3o 4B | 14.11 | 15.36 | 16.05 | 14.71 | Show-o2 7B | 15.66 | 16.55 | 16.12 | 15.44 | |
| BLIP-3o 8B | 14.55 | 16.18 | 16.89 | 15.23 | BAGEL | 26.14 | 27.21 | 26.40 | 25.76 | |
| Layer | Position | Overall (GM) |
| No scaffold | 38.27 | 15.48 |
| 0 | 41.57 | 16.05 |
| 3 | 40.79 | 15.86 |
| 7 | 40.05 | 15.45 |
| 11 | 39.46 | 15.52 |
| Canvas | GM |
| No scaffold | 15.48 |
| Uniform | 15.37 |
| Shuffled | 14.69 |
| Full (Ours) | 16.05 |
| BLIP-3o 4B | BAGEL | |||
| Method | LPIPS | DINOv2 | LPIPS | DINOv2 |
| Base | 0.321 [0.316, 0.326] | 0.127 [0.121, 0.133] | 0.586 [0.579, 0.592] | 0.300 [0.293, 0.308] |
| LoRA FT | 0.283 [0.275, 0.291] | 0.098 [0.093, 0.102] | 0.628 [0.618, 0.638] | 0.243 [0.235, 0.251] |
| SST (Ours) | 0.317 [0.308, 0.326] | 0.123 [0.117, 0.128] | 0.626 [0.614, 0.638] | 0.256 [0.249, 0.263] |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Backbone | LoRA | LoRA | Learning rate |
| BLIP-3o 4B | 64 | 16 | |
| BLIP-3o 8B | 64 | 16 | |
| Show-o2 7B | 32 | 16 | |
| BAGEL | 64 | 16 |
| BLIP-3o | Show-o2 | BAGEL | |
| Student feature | (EVA-CLIP) | patchified (VAE latent) | final LLM hidden state |
| Reference feature | EVA-CLIP feature | unified visual representation | SigLIP feature |
| Trainable modules (LoRA) | DiT cross-attention | diffusion head | generation MoE (all LLM layers) |
| Scaffold injection point | first NextDiT block | first diffusion_head_a block | first LLM layer (latent positions) |
| Hyperparameter | Symbol | Value |
| Sink fraction | 0.05 | |
| Non-sink mass threshold | 0.7 | |
| Top- candidates | 8 | |
| Aggregation temperature | 0.07 | |
| Numerical stabilizer |
| Method | CA | SCA | CSA | CC | MC | SS | Ct | Overall |
| BLIP-3o 4B | ||||||||
| Base | 12.50 | 17.50 | 40.00 | 15.00 | 15.00 | 52.50 | 27.50 | 25.71 |
| + LoRA FT | 12.50 | 20.00 | 42.50 | 15.00 | 12.50 | 55.00 | 27.50 | 26.43 |
| + SST (Ours) | 20.00 | 25.00 | 40.00 | 17.50 | 17.50 | 62.50 | 32.50 | 30.71 |
| BLIP-3o 8B | ||||||||
| Base | 15.00 | 22.50 | 45.00 | 10.00 | 25.00 | 62.50 | 35.00 | 30.71 |
| Per-phrase selection | Soft scoring | AIR | Pt |
| 0.321 | 0.603 | ||
| ✓ | 0.371 | 0.645 | |
| ✓ | ✓ | 0.411 | 0.718 |
| AIR | Pt | AM | GM | |
| Temperature ( ) | ||||
| 0.341 | 0.643 | 56.28 | 14.87 | |
| 0.411 | 0.718 | 58.34 | 16.05 | |
| 0.433 | 0.726 | 58.26 | 16.01 | |
| Top- ( ) | ||||
| 0.373 | 0.661 | 55.58 | 14.37 | |
| Backbone | Metric | (SST LoRA FT) | 95% CI | Win rate |
| BLIP-3o 4B | AM | [ , ] | 59.4% | |
| GM | [ , ] | 60.9% | ||
| BAGEL | AM | [ , ] | 56.4% | |
| GM | [ , ] | 53.9% |
| Backbone | Metric | LoRA FT | SST | [95% CI] |
| BLIP-3o 4B | AM (human) | 58.3 | 62.6 | [ , ] |
| AM (Soft-TIFA) | 56.1 | 59.4 | ||
| All-atom rate (human) | 6.3 | 8.8 | [ , ] | |
| GM (Soft-TIFA) | 9.5 | 11.5 | ||
| Agreement | 88.2 | 87.8 | ||
| BAGEL | AM (human) | 70.0 | 76.9 | [ , ] |
Compositional Image Generation
Compositional image generation studies how to combine objects, attributes, spatial relations, and visual references into coherent, controllable images.
See all 94 papers