Every Measurement, Every Direction, All at Once: Multimodal Flow Matching for Molecules and Spectra
Organizations: Lila Sciences
Abstract
Generative modeling in science is often data-limited, where experiments are expensive. However, each experiment may yield several coupled measurements of the same physical entity. Standard generative models often train only on a single direction, discarding supervision from the rest. We argue that each experiment contains more training signal than this asymmetry allows the model to exploit: random masking enables measurements to serve as either conditioning inputs or prediction targets, yielding multiple supervised denoising tasks from a single experiment. We investigate this premise with MOSAIC, a multimodal flow matching transformer over molecular 3D structure and UV, IR, and Raman spectra. MOSAIC is a single model that is capable of any-to-any generation: generating any subset of modalities from any disjoint subset of the others. On QM9S, MOSAIC achieves 88.8% molecular graph recovery, outperforming the strongest spectra-to-structure baseline by 23.2% in structure generation, while simultaneously outperforming a structure-to-spectra specialist model in spectral generation. Ablations show that the any-to-any objective improves structure generation over structure-only training across model sizes. Furthermore, MOSAIC supports arbitrary subsets of available measurements and enables molecular identification by matching generated spectra to experimental references.
Figures & tables
| Method | Spectral inputs | Generation | Gen. 3D | Gen. spectra | Any-to- any |
|---|---|---|---|---|---|
| DiffMS [ 9 ] | MS/MS | discrete diffusion | — | — | — |
| FlowMS [ 10 ] | MS/MS | discrete flow matching | — | — | — |
| FRIGID [ 11 ] | MS/MS | masked diffusion | — | — | — |
| Alberts et al. [ 12 ] | IR | autoregressive | — | — | — |
| SMEN–SD [ 13 ] | IR | autoregressive | — | — | — |
| SpectraLLM [ 14 ] | UV, IR, Raman, NMR, MS | autoregressive | — | — | — |
| Cond. | Acc@1 | H-RMSD | Comp | Tani MG | Valid | Spec cos | Steps | ms/mol | |||
| UV | IR | Raman | |||||||||
| Autoregressive spectra-to-SMILES | |||||||||||
| Alberts et al. [ 12 ] | IR+form. | — | — | — | — | — | |||||
| SMEN–SD [ 13 ] | UV | — | — | — | — | — | — | ||||
| SMEN–SD [ 13 ] | IR | — | — | — | — | — | — | ||||
| SMEN–SD [ 13 ] | Raman | — | — | — | — | — | — | ||||
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Arm | Acc@1 | RMSD DS | H-RMSD | Comp | Atom | Bond | Valid | Tani MG | Spec cos ( ) | |||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| (%) | (Å) | (Å) | (%) | (%) | (%) | (%) | UV | IR | Raman | |||
| Data scale (fixed 6M DiT) | ||||||||||||
| 1k | Struct | — | — | — | ||||||||
| Multi | ||||||||||||
| 5k | Struct | — | — | — | ||||||||
| Multi | ||||||||||||
| Model | Cosine similarity | R@1 | R@5 | R@10 | MRR | |
|---|---|---|---|---|---|---|
| Median | Mean | |||||
| Random | — | — | ||||
| DetaNet | ||||||
| MOSAIC | ||||||
| H-RMSD | IR cos | Raman cos | UV cos | ||
|---|---|---|---|---|---|
| 0.981 | |||||
| 0.907 | 0.947 | 0.859 |
| Timestep rule | Acc@1 (%) | H-RMSD (Å) |
|---|---|---|
| Independent | ||
| Shared quantile | ||
| Exact shared |
| Measurement | Shared quantile | Exact shared |
|---|---|---|
| Gradient variation, full run | ||
| Gradient variation, first k steps | ||
| Gradient variation, final k steps | ||
| Normalized variation, full run |
| Component | Model | Hidden | Depth | Batch | Steps | GPU-h |
|---|---|---|---|---|---|---|
| Pretraining | ||||||
| Encoder | large encoder | k | ||||
| UAE-3D | structure AE | — | k | |||
| Generative training: data scaling (6M DiT) | ||||||
| 6M multi | k | |||||
| 6M struct-only | k | |||||
| Modality | Samples | Notes |
| UV | Electronic absorption spectrum | |
| IR | Vibrational spectrum | |
| Raman | Vibrational spectrum | |
| Structure | Relaxed 3D geometry, up to atoms over H, C, N, O, F . | |
| Loss weights | Aggregate | Per-target | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Config | Score | LOO | ONE | struct | UV | IR | Raman | |||
| Full | ||||||||||
| No contrastive | ||||||||||
| No distillation | ||||||||||
| Recon only | ||||||||||
| Distillation target | Acc@1 ( ) | H-RMSD (Å) | Composition ( ) |
|---|---|---|---|
| Full-sequence mean | |||
| CLS-only | |||
| Tokenwise | |||
| No distillation |
| Encoder | Targets | Acc@1 | H-RMSD | Comp | Tani MG | Acc@1 |
|---|---|---|---|---|---|---|
| Frozen pretrained | Multi | |||||
| Frozen pretrained | Struct-only | |||||
| Random-init | Multi | |||||
| Random-init | Struct-only | |||||
| Pretrained, fine-tuned | Multi | |||||
| Pretrained, fine-tuned | Struct-only |
Multimodal Generation
Multimodal generation creates coordinated text, images, audio, video, 3D, and other outputs from mixed inputs while preserving meaning and consistency.
See all 258 papers