OMG-Agent: Toward Robust Missing Modality Generation with Decoupled Coarse-to-Fine Agentic Workflows
Organizations: University of Electronic Science and Technology of China
Abstract
Data incompleteness severely impedes the reliability of multimodal systems. Existing reconstruction methods face distinct bottlenecks: conventional parametric/generative models are prone to hallucinations due to over-reliance on internal memory, while retrieval-augmented frameworks struggle with retrieval rigidity. Critically, these end-to-end architectures are fundamentally constrained by Semantic-Detail Entanglement -- a structural conflict between logical reasoning and signal synthesis that compromises fidelity. In this paper, we present Omni-Modality Generation Agent (OMG-Agent), a novel framework that shifts the paradigm from static mapping to a dynamic coarse-to-fine Agentic Workflow. By mimicking a deliberate-then-act cognitive process, OMG-Agent explicitly decouples the task into three synergistic stages: (1) an MLLM-driven Semantic Planner that resolves input ambiguity via Progressive Contextual Reasoning, creating a deterministic structured semantic plan; (2) a non-parametric Evidence Retriever that grounds abstract semantics in external knowledge; and (3) a Retrieval-Injected Executor that utilizes retrieved evidence as flexible feature prompts to overcome rigidity and synthesize high-fidelity details. Extensive experiments on multiple benchmarks demonstrate that OMG-Agent consistently surpasses state-of-the-art methods, maintaining robustness under extreme missingness, e.g., a -point gain on CMU-MOSI at % missing rates.
Figures & tables
| Dataset | CMU-MOSI | |||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Modalities | L V A (✗ ✗ ❍) | L V A (✗ ❍ ✗) | L V A (❍ ✗ ✗) | L V A (✗ ❍ ❍) | L V A (❍ ✗ ❍) | L V A (❍ ❍ ✗) | L V A (❍ ❍ ❍) | |||||||||||||||
| Method | / F1 / | / F1 / | / F1 / | / F1 / | / F1 / | / F1 / | / F1 / | |||||||||||||||
| Recovery | AE | 49.9 | 47.6 | 15.9 | 52.9 | 50.1 | 16.2 | 79.0 | 79.1 | 28.5 | 55.7 | 54.1 | 17.1 | 80.8 | 80.5 | 39.6 | 82.0 | 79.4 | 42.1 | 82.6 | 82.6 | 44.0 |
| CRA | 54.5 | 50.8 | 16.2 | 56.1 | 53.1 | 16.5 | 83.2 | 82.7 | 39.9 | 59.8 | 57.7 | 17.8 | 83.3 | 83.0 | 40.8 | 82.9 | 83.3 | 43.8 | 83.9 | 83.8 | 44.4 | |
| MCTN | 56.1 | 54.5 | 16.5 | 55.0 | 54.4 | 16.3 | 79.1 | 79.2 | 41.0 | 57.5 | 57.4 | 16.8 | 81.0 | 81.0 | 43.2 | 81.1 | 81.2 | 42.1 | 81.4 | 81.5 | 43.4 | |
| MMIN | 55.3 | 51.5 | 15.5 | 57.0 | 54.0 | 15.5 | 83.8 | 83.8 | 41.6 | 60.4 | 58.5 | 19.5 | 84.0 | 84.0 | 42.3 | 83.8 | 83.9 | 42.0 | 84.6 | 84.4 | 44.8 | |
| / F1 / | ||||
|---|---|---|---|---|
| L / V / A | IMDer | CorrKD | PMSM | Ours |
| ✗ ✗ ❍ | 58.8 / 45.4 / 15.5 | 55.5 / 55.8 / 15.7 | 55.3 / 44.2 / 15.5 | 60.5 / 57.2 / 20.9 |
| ✗ ❍ ✗ | 57.8 / 42.3 / 15.5 | 51.1 / 51.4 / 16.2 | 58.9 / 43.4 / 13.5 | 60.2 / 58.4 / 21.1 |
| ❍ ✗ ✗ | 83.2 / 83.0 / 41.8 | 82.5 / 82.3 / 43.6 | 82.3 / 41.9 / 43.6 | 83.8 / 83.8 / 42.5 |
| ✗ ❍ ❍ | 59.3 / 58.9 / 21.4 | 57.9 / 44.7 / 19.5 | 61.1 / 59.1 / 22.7 | 62.0 / 61.8 / 21.9 |
| ❍ ✗ ❍ | 83.2 / 83.0 / 42.1 | 82.6 / 82.4 / 42.9 | 82.3 / 82.2 / 35.3 | 84.4 / 84.3 / 42.9 |
| Variants | CMU-MOSI | CMU-MOSEI |
|---|---|---|
| Ours | 76.4 / 76.3 / 36.0 | 77.4 / 75.0 / 48.5 |
| Instruction-Following Regularization | ||
| w/o | 72.8 / 73.1 / 33.2 | 74.5 / 74.5 / 46.2 |
| w/o | 71.9 / 72.0 / 32.7 | 73.2 / 72.7 / 44.1 |
| w/o Both | 70.3 / 70.1 / 28.4 | 72.4 / 73.0 / 43.3 |
| Task-Aware Optimization | ||
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Module | Parameter | Value |
|---|---|---|
| Global | Optimizer / Initial LR | Adam / |
| Batch Size (MOSI / MOSEI) | 32 / 128 | |
| Planner | Penalty | 0.3 / 0.1 |
| Candidate Set Size | 5 | |
| Schema Format | Triplet [E, A, S] | |
| Retriever | Retrieval Count / Temp | 10 / 0.07 |
| Dataset | Method | Missing Rate (MR) | |||||||
|---|---|---|---|---|---|---|---|---|---|
| 0.0 | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | ||
| CMU-MOSI | CCA | 74.7 / 74.1 / 29.9 | 71.6 / 71.2 / 27.6 | 68.8 / 68.4 / 25.6 | 65.7 / 65.0 / 26.3 | 62.1 / 61.0 / 23.0 | 60.4 / 59.5 / 21.4 | 59.9 / 57.4 / 21.9 | 53.3 / 53.8 / 19.2 |
| AE | 85.9 / 85.2 / 45.8 | 82.3 / 83.1 / 41.9 | 78.4 / 80.3 / 39.8 | 74.0 / 77.3 / 35.7 | 69.8 / 72.1 / 34.2 | 66.6 / 67.9 / 33.3 | 60.2 / 64.3 / 29.0 | 55.6 / 58.7 / 28.5 | |
| DCCA | 75.3 / 75.4 / 30.5 | 72.1 / 72.8 / 28.0 | 69.3 / 69.1 / 25.8 | 65.4 / 65.2 / 25.7 | 62.8 / 62.0 / 24.2 | 60.9 / 59.9 / 21.6 | 58.6 / 57.3 / 21.2 | 57.4 / 56.0 / 20.4 | |
| DCCAE | 77.3 / 77.4 / 31.2 | 74.5 / 74.7 / 28.1 | 71.8 / 71.9 / 27.6 | 67.0 / 66.7 / 25.8 | 63.6 / 62.8 / 24.2 | 62.0 / 61.3 / 23.0 | 59.6 / 58.5 / 20.9 | 58.1 / 57.4 / 20.6 | |
| CRA | 85.7 / 85.0 / 45.1 | 82.6 / 81.9 / 42.2 | 78.5 / 79.2 / 40.2 | 75.1 / 76.2 / 35.6 | 70.2 / 71.4 / 34.4 | 67.4 / 67.8 / 33.7 | 62.4 / 64.7 / 30.3 | 59.4 / 59.2 / 27.1 | |
| Retrieval Strategy | CMU-MOSI | CMU-MOSEI |
|---|---|---|
| Content-Only Retrieval | 74.4 / 72.1 / 32.5 | 74.0 / 73.8 / 46.6 |
| Random-Plan Retrieval | 71.2 / 71.0 / 26.7 | 72.8 / 73.4 / 43.2 |
| Plan-Driven Retrieval (Ours) | 76.4 / 76.3 / 36.0 | 77.4 / 75.0 / 48.5 |
| Variants | CMU-MOSI | CMU-MOSEI |
|---|---|---|
| Ours | 76.4 / 76.3 / 36.0 | 77.4 / 75.0 / 48.5 |
| Concat-Injection | 75.2 / 74.6 / 34.8 | 76.2 / 73.6 / 46.4 |
| Reversed-Injection | 70.2 / 71.0 / 29.6 | 71.5 / 72.2 / 43.3 |
| Single-Stream | 73.2 / 73.2 / 31.8 | 75.3 / 74.5 / 45.8 |
Missing-Modality Learning
Missing-modality learning develops multimodal models that remain useful when input streams are absent, corrupted, or available only during training.
See all 184 papers