Mobile-O: Unified Multimodal Understanding and Generation on Mobile Device
Authors: Abdelrahman Shaker, Ahmed Heakl, Jaseel Muhammad, Ritesh Thawkar, Omkar Thawakar, Senmao Li, Hisham Cholakkal, Ian Reid, Eric P. Xing, Salman Khan, Fahad Shahbaz Khan
Organizations: Mohamed bin Zayed University of Artificial Intelligence · Carnegie Mellon University · Linköping University
Unified multimodal models can both understand and generate visual content within a single architecture. Existing models, however, remain data-hungry and too heavy for deployment on edge devices. We present Mobile-O, a compact vision-language-diffusion model that brings unified multimodal intelligence to a mobile device. Its core module, the Mobile Conditioning Projector (MCP), fuses vision-language features with a diffusion generator using depthwise-separable convolutions and layerwise alignment. This design enables efficient cross-modal conditioning with minimal computational cost. Trained on only a few million samples and post-trained in a novel quadruplet format (generation prompt, image, question, answer), Mobile-O jointly enhances both visual understanding and generation capabilities. Despite its efficiency, Mobile-O attains competitive or superior performance compared to other unified models, achieving 74% on GenEval and outperforming Show-O and JanusFlow by 5% and 11%, while running 6x and 11x faster, respectively. For visual understanding, Mobile-O surpasses them by 15.3% and 5.1% averaged across seven benchmarks. Running in only ~3s per 512x512 image on an iPhone, Mobile-O establishes the first practical framework for real-time unified multimodal understanding and generation on edge devices. We hope Mobile-O will ease future research in real-time unified multimodal intelligence running entirely on-device with no cloud dependency. Our code, models, datasets, and mobile application are publicly available at https://amshaker.github.io/Mobile-O/
Figures & tables
Figure 2: Overview of Mobile-O . Left: The proposed framework consists of an efficient image encoder with a compact autoregressive language model for visual understanding. For image generation, a lightweight linear diffusion transformer (DiT) is employed alongside a simple yet effective VAE-based encoder–decoder. Right: Our novel Mobile Conditioning Projector (MCP) bridges the understanding and generation tasks by directly conditioning the diffusion model on weighted hidden states from the VLM without the need for intermediate query tokens. The projector leverages layer-wise feature fusion, depthwise separable convolutions, and efficient channel attention to produce high-fidelity conditioning signals with minimal cost, enabling seamless deployment on edge devices.
Figure 3: Overview of the proposed unified multimodal post-training pipeline. We jointly optimize multimodal understanding and generation through a multi-task objective using a quadruplet format ( generation prompt, image, question, answer ). Both I2T and T2I losses are computed simultaneously, enabling aligned cross-modal learning where each training sample supports both multimodal understanding and generation.
Type
Model
# Total Params
MMMU ↑
TextVQA ↑
MMVet ↑
SEED ↑
ChartQA ↑
POPE ↑
GQA ↑
Average ↑
Und. Only > 1B
LLaVA-Phi [ 50 ]
3.1 B
-
48.6
28.9
-
-
85.0
-
-
LLaVA-v 1.5 -Phi- 1.5 [ 50 ]
1.6 B
30.7
-
-
-
-
84.1
56.5
-
MobileVLM [ 7 ]
1.7 B
-
41.5
-
-
-
84.5
56.1
-
MobileVLM-V2 [ 8 ]
1.7 B
-
52.1
-
-
-
84.3
59.3
-
LLaVa-OV [ 16 ]
1.6 B
31.4
-
29.1
65.5
61.4
-
-
-
Und. Only ≤ 1B
Smol-VLM-0.5B [ 23 ]
0.6 B
33.7
60.2
-
-
62.8
-
-
-
Table 1: Comparison with recent multimodal understanding models. “Und.” and “Gen.” denote “understanding” and “generation”, respectively. Total Params represent the sum of visual encoder, language model, and diffusion/unet components (when applicable). Compared to unified models with similar size ( ≤ 2B), our Mobile-O-0.5B achieves superior overall performance with a score of 61.9 averaged over seven datasets. Further, Mobile-O-0.5B also outperforms its understanding-only counterpart (FastVLM) by 1.6% in average performance.
Type
Method
# Total Params
Single Obj.
Two Obj.
Counting
Colors
Position
Color Attri.
Overall ↑
Gen. Only > 1B
LlamaGen [ 33 ]
3.8 B
0.71
0.34
0.21
0.58
0.07
0.04
0.32
LDM [ 29 ]
1.5 B
0.92
0.29
0.23
0.70
0.02
0.05
0.37
PixArt- α [ 3 ]
4.9 B
0.98
0.50
0.44
0.80
0.08
0.07
0.48
SDXL [ 27 ]
3.0 B
0.98
0.74
0.39
0.85
0.15
0.23
0.55
SDv 2.1 [ 29 ]
1.9 B
0.98
0.51
0.44
0.85
0.07
0.17
0.50
SANA-0.6B [ 42 ]
2.6 B
0.99
0.77
0.62
0.88
0.21
0.47
0.66
Table 2: Evaluation of text-to-image generation performance on the GenEval benchmark. “Und.” and “Gen.” denote “understanding” and “generation”, respectively. Total Params represent the sum of the visual encoder, language model, and diffusion/unet components (when applicable). Compared to unified models with similar size ( ≤ 2B), our Mobile-O-0.5B achieves superior overall score of 0.74 and outperforms Show-o-Clip-ViT [ 43 ] by 5.0%.
Figure 4: Qualitative comparison of text-to-image generation (left) and visual understanding (right) across unified multimodal models . Each column shows Janus, JanusFlow, Show-O, and Mobile-O (ours) for the same prompts/questions. Mobile-O yields more consistent, detailed, and semantically faithful images with high fidelity and style diversity for image generation. For visual understanding, it delivers more accurate and contextually coherent responses. Additional results are presented in suppl. material. Best viewed zoomed in.
Figure 5: Qualitative image editing results of Mobile-O-0.5B. Given a source image and a textual editing instruction, Mobile-O-0.5B produces the edited output. The model is fine-tuned on only 46k editing samples from ShareGPT4V [ 5 ] .
Model
Und. Acc. (%)
Gen. Acc. (%)
FastVLM-1.5B [ 39 ]
64.8%
–
SANA-1.5B [ 42 ]
–
66%
Mobile-O-1.5B (SFT)
64.8%
75%
Mobile-O-1.5B (Post-train)
66.2%
78%
Table 3: Mobile-O-1.5B : Scaling to FastVLM-1.5B and SANA-1.5B components. Understanding performance is averaged over seven benchmarks (MMMU, TextVQA, SEED-Bench, ChartQA, POPE, GQA, MM-Vet). Generation quality is measured by GenEval overall score. The proposed post-training stage consistently improves both capabilities.
Proj.
# Layers
Fusion
Compress
CA
Acc. (%)
Params (M)
MLP
–
–
–
–
68.5
3.3
MCP
1
Uniform
✓
–
68.4
1.7
MCP
4
Uniform
✓
–
69.6
1.7
MCP
4
Learnable
✓
–
70.0
1.7
MCP
4
Learnable
✓
✓
70.4
2.4
Table 4: Ablation on the Mobile Conditioning Projector (MCP). We study the effect of layer fusion, learnable weighting, and the refinement block.
Method
Und. Acc. (%)
Gen. Acc. (%)
SFT
60.5
73.3
SFT + Post-Train (image-text pairs)
60.6
73.4
SFT + Post-Train (quadruplets)
62.1
74.2
Table 5: Effect of Unified Post-Training. Our post-training data improves both understanding and generation alignment when using joint quadruplets.
Model
Vision Enc. (ms)
TTFT (ms)
Latency (s)
MacBook M2 Pro
Janus
783±244
289±19
201±15614
JanusFlow
1909±466
935±152
24±0.8
Show-O
699±107
797±5
47±0.2
Mobile-O-0.5B (Ours)
56±7
187±31
4±0.5
Jetson Orin Nano
Table 6: Image understanding and generation performance comparison on MacBook M2 Pro, Jetson Orin Nano, and iPhone for Mobile-O-0.5B . Vision Enc. and TTFT denote understanding latency, while Latency indicates image generation latency.
Proj.
# Layers
Fusion
Compress
CA
Accuracy (%)
MCP
1
-
✓
✓
68.7
MCP
2
Learnable
✓
✓
69.8
MCP
4
Learnable
✓
✓
70.4
MCP
8
Learnable
✓
✓
70.2
Table 7: Ablation on the number of layers for the Mobile Conditioning Projector (MCP). We systematically vary the number of VLM layers aggregated by MCP to condition the diffusion process. All configurations use the final design of MCP: learnable fusion, compression , and channel attention (CA).
Figure 6: Mobile-O running natively on iPhone 17 Pro. We demonstrate real-world deployment of Mobile-O ’s unified capabilities on consumer hardware. (a) Text-to-image generation: Given a detailed prompt describing a Bengal tiger. (b) Image-to-text generation: Mobile-O provides detailed visual descriptions, analyzing composition and subject positioning
Stage 1: Cross-Modal Alignment
Stage 2: Supervised Fine-tuning
Stage 3: Unified Post-Training
Data Source
JourneyDB (4M) +
BLIP3o-60K +
BLIP3o-60K +
BLIP3o-Short (5M)
ShareGPT-4o (45K)
ShareGPT-4o (45K)
Total Samples
9M
105K
105K
Format
Prompt-image pairs
Prompt-image pairs
Quadruplet {p,ximg,q,a}
Learning Rate
2e-4
2e-4
1e-4
Batch Size
512
384
128
Table 8: Three-stage training setup for Mobile-O. Stage 1 establishes cross-modal alignment using large-scale image-text pairs. Stage 2 performs targeted fine-tuning to address weaknesses in complex gestures, common objects, and landmarks. Stage 3 introduces unified multimodal post-training with quadruplet samples {p,ximg,q,a} for joint understanding and generation. All experiments were conducted on 8 × A100 GPUs.
Figure 7: Qualitative comparison on dense text understanding and information extraction. We evaluate Mobile-O against other models on a challenging OCR and comprehension task requiring the model to read, parse, and summarize the back cover text of a book. Green text indicates correctly extracted information, while red indicates hallucinations or errors. Mobile-O demonstrates superior performance in accurately extracting key bibliographic details, including the correct title, author, editors, and price information from the densely-packed text on the book cover.
Figure 8: Qualitative comparison with SANA-0.6B on text-to-image generation. We compare Mobile-O (1.6B total parameters) against SANA-0.6B (2.6B total parameters), our generation baseline, on challenging prompts requiring photorealistic rendering, complex lighting, and fine-grained details. Mobile-O demonstrates competitive or superior visual quality across diverse scenarios, including wildlife photography, landscape composition, and portrait rendering. Best viewed zoomed in.
Figure 9: Qualitative comparison of image-to-text across unified models below 2B. Mobile-O is compared against Janus [ 41 ] , JanusFlow [ 22 ] , and Show-O [ 43 ] on diverse visual question answering tasks, including scientific reasoning, OCR, object recognition, and cultural knowledge. Green indicates correct answers, red indicates errors. Mobile-O demonstrates competitive visual understanding, despite its mobile-optimized architecture, correctly answering complex questions that require fine-grained visual analysis and domain knowledge.
Figure 10: Qualitative comparison of text-to-image generation across unified models below 2B. Mobile-O is compared against Janus [ 41 ] , JanusFlow [ 22 ] , and Show-O [ 43 ] on challenging prompts spanning fantasy, photorealism, and scientific visualization. Despite its mobile-optimized architecture, Mobile-O maintains competitive visual quality and prompt adherence. Best viewed zoomed in.
Figure 11: Additional Text-to-Image generation examples of Mobile-O . Best viewed zoomed in.
Detailed Prompts for Image Generation in Fig. 11 — (i,j) denotes the image at row i , column j
Row 1
Portrait Photography
[2pt/2pt] (1,1)
Young woman with freckles, green eyes with detailed iris and catchlight, natural skin texture, flowing red hair catching golden hour sunlight, warm vibrant lighting, gentle expression, shallow depth of field
(1,2)
Elderly Asian man with silver hair and peaceful expression, kind brown eyes, colorful traditional clothing, bright natural window lighting, detailed skin texture
(1,3)
Young man with curly hair and beard, bright turquoise shirt, outdoor lighting
(1,4)
Child with bright blue eyes and blonde hair, curious expression, rosy cheeks, individual eyelashes visible, natural freckles, bright daylight from side creating dimension, colorful playground in blurred background
Row 2
Flowers & Garden Scenes
Table 9: Text-to-image generation prompts used for visualization.
Unifying visual understanding and generation in one model holds immense promise, but remains challenging and expensive due to heavy compute and data demands and conflicts between the visual features needed for these two capabilities. To address these challenges, we present Argus-Unified, a compact, effective and unified multimodal model built with low demand on computation and data. Instead of aligning modalities from scratch, Argus-Unified effectively leverages pretrained vision-language models (VLMs) that provide strong multimodal priors. Specifically, we introduce hybrid visual tokens that preserve continuous tokens for understanding while learning discrete tokens for generation from a frozen unified vision encoder. Our training pipeline includes two stages: the first stage learns a quantizer and image decoder on top of the frozen vision encoder, the second stage trains the LLM initialized from a pretrained VLM for the unified multimodal modeling. Using by far the least amount of data (15.6M) and the lowest cost (~$2,000), we demonstrate that unified multimodal models can be trained economically while achieving strong performance in both understanding and generation. Notably, our model attains state-of-the-art multimodal understanding on GQA, POPE, and VQAv2, and competitive generation quality compared to models with dedicated vision encoders (e.g., Janus, Janus-Pro), all at ~10x lower cost and with ~5x less data. We envision Argus-Unified as a useful baseline that lowers the development barrier for unified models.
Unified multimodal models are envisioned to bridge the gap between understanding and generation. Yet, to achieve competitive performance, state-of-the-art models adopt largely decoupled understanding and generation components. This design, while effective for individual tasks, weakens the connection required for mutual enhancement, leaving the potential synergy empirically uncertain. We propose to explicitly restore this synergy by introducing Understanding-Oriented Post-Training (UNO), a lightweight framework that treats understanding not only as a distinct task, but also a direct supervisory signal to steer generative representations. By incorporating objectives that encode semantic abstraction (captioning) and structural details (visual regression), we enable effective gradient flow from understanding to generation. Extensive experiments on image generation and editing demonstrate that understanding can serve as an effective catalyst for generation.
Zeyu Liu, Zanlin Ni, Yang Yue +5
1Tsinghua University · 2Kolors Team, Kuaishou Technology
Most unified large multimodal models (LMMs) that support both visual understanding and image generation still rely on curated post-training supervision, such as human annotations, preference labels, or external reward models. We ask whether a unified LMM can improve both abilities autonomously using only unlabeled images. We propose a self-evolving training framework with three internal roles: a Proposer that generates visual questions, a Solver that answers and evaluates them, and a Generator that synthesizes images. Training uses only self-derived consistency signals, without human annotations, preference labels, or task-trained external reward/judge models. To stabilize learning, we introduce Solver Token Entropy (STE), a continuous difficulty signal based on token-level prediction uncertainty that remains useful even when sample-level consistency becomes unreliable. For image generation, we design a multi-scale internal evaluation scheme that combines question-answer fidelity scoring with cycle-consistent captioning. This creates a solver-mediated coupling, where better visual understanding enables more reliable generation assessment and stronger internal training signals. The framework preserves the same role decomposition, reward logic, and training schedule across diffusion-based BLIP3o, rectified-flow BAGEL, and autoregressive VARGPT-v1.1 architectures, requiring only each backbone's native prompting and generation interface. Across eight understanding metrics, our method consistently improves over the corresponding base models. On BAGEL, it achieves a +3.5% absolute gain on MMMU and improves GenEval image generation performance from 82% to 85%. Code and models are publicly released.