Can Visual Generation Expand What Unified Multimodal Models Know?
Organizations: Tsinghua University
Abstract
Unified multimodal models (UMMs) receive image-generation supervision that exposes them to visual information not fully documented in text. Can learning to generate images expand what pretrained models know? We introduce Knowledge Transfer Probing (KTP), a controlled protocol that supplies new knowledge through image-generation training and tests its accessibility through text-only question answering. Across seven knowledge categories and three UMMs, we observe direct but uneven transfer from visual supervision to language. Motivated by these observations, we successively introduce Textual Alignment and Interleaved Image--Text Alignment, which connect generation and answering through shared knowledge and interleaved responses, respectively. Guidance on selected entities can improve knowledge access on other entities without QA supervision, with additional gains from interleaved alignment. Further analysis shows that knowledge correctly reproduced in images is often more accessible through text, although substantial gaps remain between the two forms of access. These findings support visual generation as a source of knowledge beyond textual supervision, and motivate joint investigation of training data design and model architecture to make this knowledge more broadly accessible.
Figures & tables
| Model | Metric | color | count | number | shape | size | spatial | text | Avg |
|---|---|---|---|---|---|---|---|---|---|
| Baseline | random guessing | 16.7 | 25.0 | 0.0 | 20.0 | 50.0 | 0.0 | 0.0 | 15.96 |
| BAGEL | textual accessibility | 14.0 | 14.0 | 0.0 | 10.0 | 24.0 | 0.0 | 0.0 | 8.86 |
| visual accessibility | 20.0 | 42.0 | 4.0 | 58.0 | 60.0 | 0.0 | 36.0 | 31.43 | |
| GLM-Omni | textual accessibility | 16.0 | 24.0 | 2.0 | 20.0 | 40.0 | 0.0 | 0.0 | 14.57 |
| visual accessibility | 70.0 | 46.0 | 72.0 | 88.0 | 24.0 | 62.0 | 16.0 | 54.00 | |
| SenseNova-U1 | textual accessibility | 20.0 | 32.0 | 0.0 | 20.0 | 68.0 | 0.0 | 0.0 | 20.00 |
| Model | Training Method | Color | Count | Number | Shape | Size | Spatial | Text | Avg |
|---|---|---|---|---|---|---|---|---|---|
| BAGEL | direct | 14.0 | 14.0 | 0.0 | 10.0 | 24.0 | 0.0 | 0.0 | 8.86 |
| textual alignment | 4.0 | 28.0 | 0.0 | 24.0 | 52.0 | 0.0 | 0.0 | 15.43 | |
| interleaved alignment | 12.0 | 32.0 | 0.0 | 20.0 | 56.0 | 0.0 | 0.0 | 17.14 | |
| GLM-Omni | direct | 16.0 | 24.0 | 2.0 | 20.0 | 40.0 | 0.0 | 0.0 | 14.57 |
| textual alignment | 16.0 | 20.0 | 0.0 | 12.0 | 40.0 | 0.0 | 0.0 | 12.57 | |
| interleaved alignment | 24.0 | 20.0 | 0.0 | 64.0 | 40.0 | 40.0 | 0.0 | 26.86 |
| Model | Inference | color | count | number | shape | size | spatial | text | Avg |
|---|---|---|---|---|---|---|---|---|---|
| BAGEL | Text-only | 12.0 | 32.0 | 0.0 | 20.0 | 56.0 | 0.0 | 0.0 | 17.14 |
| Visual CoT | 16.0 | 20.0 | 4.0 | 56.0 | 48.0 | 0.0 | 0.0 | 20.57 | |
| GLM-Omni | Text-only | 24.0 | 20.0 | 0.0 | 64.0 | 40.0 | 40.0 | 0.0 | 26.86 |
| Visual CoT | 24.0 | 48.0 | 36.0 | 68.0 | 48.0 | 64.0 | 0.0 | 41.14 | |
| SenseNova-U1 | Text-only | 4.0 | 16.0 | 0.0 | 84.0 | 96.0 | 0.0 | 0.0 | 28.57 |
| Visual CoT | 8.0 | 20.0 | 20.0 | 92.0 | 96.0 | 8.0 | 0.0 | 34.86 |
| Model | Training | (%) | (%) | (pp) |
|---|---|---|---|---|
| BAGEL | Textual Alignment | 23.91 | 12.40 | |
| Interleaved Alignment | 32.61 | 11.63 | ||
| GLM-Omni | Textual Alignment | 6.90 | 18.18 | |
| Interleaved Alignment | 31.25 | 23.16 | ||
| SenseNova-U1 | Textual Alignment | 46.39 | 5.13 | |
| Interleaved Alignment | 44.00 | 8.00 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Attribute | Knowledge | Value range | Images |
|---|---|---|---|
| Color | The color of a fictional font | blue, green, black, red, white, yellow | 1,226 |
| Count | The count of a fictional collection | 2, 3, 4, 5 | 1,195 |
| Shape | The shape of a fictional badge | circle, five-pointed star, pentagon, square, triangle | 1,248 |
| Spatial | The stripe arrangement of a fictional flag | — | 1,220 |
| Size | The larger country on a fictional map | — | 1,218 |
| Number | The number of a fictional person’s jersey | 0–99 | 1,244 |
| Model | Image generation | Textual alignment | Interleaved alignment |
|---|---|---|---|
| BAGEL | 0.956 | 0.022 | 0.022 |
| GLM-Omni | 0.990 | 0.005 | 0.005 |
| SenseNova-U1 | 0.851 | 0.063 | 0.086 |
| Hyperparameter | BAGEL | SenseNova-U1 | GLM-Omni |
|---|---|---|---|
| Learning rate | |||
| LR scheduler | constant with linear warm-up | constant | constant with linear warm-up |
| Weight decay | 0 | 0 | 0.1 |
| Gradient clipping norm | 1.0 | 1.0 | 1.0 |
| Warm-up steps | 100 | 0 | 100 |
| Training steps | 4000 | 1500 | 1500 |
Unified Multimodal Models
Unified multimodal models combine multiple data types in shared systems for cross-modal understanding, generation, reasoning, and interaction.
See all 282 papers