MoI-Bench: A Benchmark for Image Generation and Editing on Mobile Devices
Organizations: University College London · Nanjing University · The Hong Kong Polytechnic University · East China Normal University
Abstract
Developing on-device image models requires considering practical user needs, task types and device resource constraints. However, existing benchmarks provide limited guidance on identifying capability gaps between on-device models and server-side models. We introduce MoI-Bench, a benchmark grounded in mobile user needs, comprising 64 subtasks and 1150 samples. It defines 16 operations, such as removal and replacement, and eight visual target categories covering content such as portraits and text. These form two orthogonal axes whose applicable combinations define specific tasks, such as removing facial makeup. To better guide model improvement, we further propose MoI-Eval, which introduces task-specific criteria covering instruction following, visual quality and detail preservation. We further construct sample-level evaluation questions and aggregate their scores into overall quality and capability profiles along the operation, target and criteria axes. We evaluate representative on-device and server-side models and measure mobile deployment costs. Results show that this decoupled Operation x Target design provides finer-grained guidance for improving compact models. For example, it highlights the need to prioritize specific capability gaps (e.g., text-replacement) over broad application-level goals (e.g., improving text editing). Unlike conventional benchmarks that rank models using a single score, we introduce a two-dimensional Quality-Deployability capability map to assess models for practical mobile use. The map shows that current on-device models can achieve efficient generation or editing but expanding their task coverage deserves greater attention than further improving deployment efficiency. Further progress requires improving editing quality, particularly identity preservation and text rendering, while broadening task coverage.
Figures & tables
| Benchmark | Size | Task coverage | Max. Refs | MT | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| T2I | Loc. | Glob. | Act. | Pos. | View | Impl. | Multi-obj. | ||||
| Text-to-image benchmarks | |||||||||||
| GenEval Ghosh et al. (2023) | 553 | 0 | |||||||||
| GenAI-Bench Li et al. (2024a) | 1,600 | 0 | |||||||||
| Qwen-Image-Bench Li et al. (2026) | 1,000 | 0 | |||||||||
| General image-editing benchmarks | |||||||||||
| Quality | Deployability | ||||||
|---|---|---|---|---|---|---|---|
| Tier | Overall | Coverage | Each group | Tier | Size (MiB) | E2E (s) | Memory (MiB) |
| Q4 | D4 | ||||||
| Q3 | D3 | ||||||
| Q2 | — | — | D2 | — | — | — | |
| Q1 | — | — | D1 | Unavailable Deployment | |||
| Model | Tasks | Type | GPT-4o | Gemini 3.1 Pro | Q-tier | ||||
|---|---|---|---|---|---|---|---|---|---|
| (IF, VQ, DP) | (IF, VQ, DP) | ||||||||
| Closed-source Models | |||||||||
| Nano Banana 2 Google DeepMind (2026) | Both | – | 0.95 | 1.00 | (0.95, 0.97, 0.97) | 0.93 | 1.00 | (0.94, 0.96, 0.96) | Q4 |
| Seedream 5.0 Pro ByteDance Seed (2026) | Both | – | 0.95 | 1.00 | (0.95, 0.97, 0.98) | 0.93 | 1.00 | (0.93, 0.97, 0.96) | Q4 |
| GPT Image 2 OpenAI (2026) | Both | – | 0.97 | 1.00 | (0.97, 0.98, 0.98) | 0.96 | 1.00 | (0.96, 0.98, 0.96) | Q4 |
| Open-weight Reference Models | |||||||||
| Model | Generation | Local Editing | Global Editing | Understand. Editing | Ref. Editing | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| T2I | Add. | Rem. | Repl. | Adj. | Style | Bkg. | Rest. | View | Sem. | Outp. | Inp. | R1 | B1+R1 | B1+R2 | B1+R3 | |
| Closed-source Models | ||||||||||||||||
| Nano Banana 2 | 0.99 | 0.94 | 0.95 | 0.98 | 0.96 | 0.99 | 0.97 | 0.92 | 0.80 | 0.95 | 0.95 | 0.99 | 0.96 | 0.93 | 0.88 | 0.80 |
| Seedream 5.0 Pro | 0.98 | 0.95 | 0.96 | 0.98 | 0.96 | 0.94 | 0.97 | 0.92 | 0.83 | 0.93 | 0.87 | 1.00 | 0.97 | 0.94 | 0.90 | 0.99 |
| GPT Image 2 | 0.99 | 0.97 | 0.97 | 0.99 | 0.98 | 0.99 | 0.99 | 0.95 | 0.82 | 0.95 | 0.92 | 1.00 | 0.97 | 0.96 | 0.87 | 0.99 |
| Open-weight Reference Models | ||||||||||||||||
| Model | General Objects | Animals | Portraits | Text&Docs. | ||||
|---|---|---|---|---|---|---|---|---|
| Concrete | Abstract | Body&Limbs | Face&Hair | Cloth.&Accs. | Multi-subjects | |||
| Closed-source Models | ||||||||
| Nano Banana 2 | 0.95 | 0.95 | 0.99 | 0.94 | 0.95 | 0.97 | 0.94 | 0.93 |
| Seedream 5.0 Pro | 0.93 | 0.98 | 0.96 | 0.95 | 0.96 | 0.99 | 0.95 | 0.94 |
| GPT Image 2 | 0.95 | 0.97 | 0.98 | 0.96 | 0.96 | 0.99 | 0.97 | 0.96 |
| Open-weight Reference Models | ||||||||
| Model | IF | VQ | DP | |||||
|---|---|---|---|---|---|---|---|---|
| Fidelity | Selection | Integrity | Coherence | Presentation | Context | Structure | Identity | |
| Closed-source Models | ||||||||
| Nano Banana 2 | 0.94 | 0.95 | 0.98 | 0.96 | 0.98 | 0.98 | 0.96 | 0.99 |
| Seedream 5.0 Pro | 0.94 | 0.96 | 0.98 | 0.96 | 0.98 | 0.97 | 0.98 | 0.98 |
| GPT Image 2 | 0.96 | 0.97 | 0.99 | 0.97 | 0.99 | 0.98 | 0.98 | 0.98 |
| Open-weight Reference Models | ||||||||
| Model | Params. | Res. | Steps | iPhone 17 Pro | Xiaomi 15 Pro | D-tier | ||||
|---|---|---|---|---|---|---|---|---|---|---|
| Size (MiB) | E2E (s) | Mem. (MiB) | Size (MiB) | E2E (s) | Mem. (MiB) | |||||
| Janus-Pro-1B † | 2.1B | – | 2975.4 | 58.26 | 1505.4 | 2974.1 | 76.01 | 3253.8 | D2 | |
| Mobile-O-0.5B | 1.7B | 20 | 1718.3 | 2.81 | 426.8 | 1770.2 | 13.30 | 1757.1 | D3 | |
| SANA-Sprint-0.6B | 3.5B | 2 | 2860.9 | 4.49 | 1727.2 | 3069.1 | 25.24 | 3896.7 | D2 | |
| SANA-600M | 3.5B | 18 | 2857.2 | 15.33 | 1775.2 | 3065.4 | 139.68 | 3897.6 | D2 | |
| DreamLite | 2.5B | 4 | 1684.6 | 3.22 | 1305.7 | 1804.5 | 18.19 | 2390.5 | D3 | |
| Model | Precision | iPhone 17 Pro | Xiaomi 15 Pro | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Backbone | Decoder | Size | E2E | Mem. | Status | Size | E2E | Mem. | Status | |
| Janus-Pro-1B | FP16 | W8A16 | 2935.0 | 58.96 | 1502.2 | C | 2934.0 | 77.59 | 3253.8 | C |
| W8A16 | FP16 | 1764.4 | 54.54 | 1500.8 | C | – | – | – | F | |
| W8A16 | W8A16 | 1724.0 | 54.28 | 1503.6 | C | – | – | – | F | |
| Mobile-O-0.5B | FP16 | W8A16 | 1580.1 | 2.84 | 449.2 | C | 1619.8 | 12.49 | 1767.0 | C |
| W8A16 | FP16 | 1155.7 | 2.68 | 423.5 | C | 1209.3 | 8.21 | 3149.2 | C | |
| Model | Cond. | Generation | Decode | Other | Total | ||
|---|---|---|---|---|---|---|---|
| Steps | Branches | Time/step | |||||
| Janus-Pro-1B | 2.214 | 576 | 2 | 0.0971 | 0.083 | 0.042 | 58.256 |
| Mobile-O-0.5B | 0.095 | 20 | 2 | 0.1134 | 0.230 | 0.216 | 2.809 |
| SANA-Sprint-0.6B | 0.281 | 2 | 1 | 0.3920 | 3.154 | 0.270 | 4.488 |
| SANA-600M | 0.298 | 18 | 2 | 0.6378 | 3.035 | 0.515 | 15.328 |
| DreamLite | 0.065 | 4 | 1 | 0.7645 | 0.036 | 0.065 | 3.224 |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Topic | Question and response options |
|---|---|
| Q1a. Current usage | How often do you use AI tools on your phone, such as chat assistants, image generators or photo editors? Options: Daily; Several times a week; About once a week; Less than once a week; Never. |
| Q1b. Image tools | Would you like to generate or edit images using AI tools on your phone? Options: Yes; Maybe; No. |
| Q1c. On-device interest | Would you be interested in image AI that runs directly on your phone without uploading your photos to a server? Options: Yes; Maybe; No. |
| Q2. Desired use cases | Which of the following would you most like to use AI image tools for on your phone? Select up to eight options. Options: (1) Retouch portraits, such as removing blemishes or wrinkles. (2) Remove unwanted people or objects from photos. (3) Restore old, blurry or damaged photos. (4) Edit or translate text in images and documents. (5) Improve or creatively edit pet photos. (6) Make memes or stickers. (7) Create personal avatars or profile pictures. (8) Generate wallpapers. (9) Try different hairstyles, makeup or accessories. (10) Change clothing or preview outfit combinations. (11) Replace backgrounds in personal or product photos. (12) Turn photos into cartoons, paintings or other styles. (13) Create posters, invitations or greeting cards. (14) Improve product photos for online listings. (15) Preview furniture or decorations in a room. (16) Combine people or objects from several photos. |
| Q3. Usability requirements | Suppose you ask an AI tool to remove a small blemish from your cheek in a portrait and leave the rest of the photo unchanged. What would matter to you when judging the result? Select up to eight options. Options: (1) Results are returned quickly. (2) The requested blemish is removed. (3) Photos remain private. (4) Clothing, background and other people remain unchanged. (5) The skin retains natural texture without visible artifacts. (6) The tool requires little storage. (7) The person remains recognisable, with facial features preserved. (8) The tool works offline. (9) Only the intended area is retouched. (10) The overall photo remains clear and visually balanced. (11) The tool does not slow down the phone. (12) The original pose, framing and positions remain unchanged. (13) The retouched area matches the surrounding skin tone and lighting. |
| Group | Operation | Definition |
|---|---|---|
| Generation | T2I | Generate an image from text without an input image. |
| Local editing | Addition | Add specified content while retaining existing, unrelated content. |
| Removal | Delete specified content and complete the exposed region. | |
| Replacement | Replace selected content with a specified alternative. | |
| Adjustment | Change attributes of existing content, such as color, size or pose. | |
| Global editing | Stylization | Apply a visual style across the image. |
| Group | Target | Definition |
|---|---|---|
| General Objects | Concrete Objects | Discrete entities such as vehicles, furniture and everyday objects. |
| Abstract Objects | Visual effects and relations, such as lighting, shadows and reflections. | |
| Animals | Animals | Animals, including pets, their appearance and their accessories. |
| Portraits | Body & Limbs | Human body shape, limb configuration and pose. |
| Face & Hair | Facial features, expressions, makeup, hairstyles and identity. | |
| Clothing & Accessories | Human garments and accessories, such as coats, glasses and jewelry. |
| Group | Criterion | Definition |
|---|---|---|
| IF | Fidelity | Satisfying the requested outcome and explicit constraints. |
| Selection | Applying the change to the correct region or instance. | |
| VQ | Integrity | Well-formed generated or edited content without defects or artifacts. |
| Coherence | Consistent boundaries, lighting and appearance within the output and around edited regions. | |
| Presentation | Overall readability, composition and physical plausibility. | |
| DP | Context | Preserving non-target objects and background details. |
| Criterion | Rubric focus and questions | Scores |
|---|---|---|
| Fidelity | Remove the requested person. (1) Is the leftmost person completely removed from the image without leaving visible remnants? | 0.6 |
| Selection | Edit the correct person only. (1) Did the removal target only the leftmost person (man in dark t-shirt), leaving the adjacent man in the white t-shirt untouched? | 1 |
| Integrity | Reconstruct complete shapes and clean textures. (1) Is the reconstructed background on the far left structurally complete without holes, repeated parts, or malformed shapes? (2) Are the textures in the repaired region on the left free of smearing, ringing, or synthetic artifacts? | 1, 0.6 |
| Coherence | Match the surrounding geometry, colour and lighting. (1) Does the reconstructed region on the left continue the surrounding street and background geometry without a visible seam? (2) Do the color and lighting in the filled area on the left match the neighboring image area? | 1, 0.6 |
| Presentation | Maintain overall clarity and plausible composition. (1) Is the final image clear and usable without awkward crop or weak focus? (2) Is the overall scene physically plausible and balanced in scale and composition? | 1, 1 |
| Context | Preserve the unedited background. (1) Are the buildings in the background unchanged in appearance? (2) Is the street surface outside the removed region preserved in color, texture, and lighting? | 1, 1 |
| Operation | IF | VQ | DP |
|---|---|---|---|
| Text-to-image | 0.60 | 0.40 | — |
| Addition, removal, replacement | 0.45 | 0.25 | 0.30 |
| Adjustment | 0.40 | 0.25 | 0.35 |
| Stylization, background editing | 0.45 | 0.35 | 0.20 |
| Restoration | 0.40 | 0.45 | 0.15 |
| Viewpoint editing | 0.45 | 0.30 | 0.25 |
| Model | Overall quality | Relative change | Subtasks with quality | Q-tier | ||
|---|---|---|---|---|---|---|
| Original | Text-NF4 | |||||
| Ovis-U1-3B | 0.541734 | 0.538307 | Q2 Q2 | |||
| DreamLite | 0.523777 | 0.520798 | Q2 Q2 | |||
| SANA-600M | 0.102904 | 0.101979 | Q1 Q1 | |||
| Model | Operation | Target | Criteria | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Gen. | Local | Global | Underst. | Ref. | General | Animals | Portraits | Text. | IF | VQ | DP | |
| Closed-source Models | ||||||||||||
| Nano Banana 2 | 0.969 | 0.943 | 0.889 | 0.885 | 0.938 | 0.925 | 0.971 | 0.941 | 0.897 | 0.939 | 0.957 | 0.962 |
| GPT Image 2 | 0.981 | 0.968 | 0.927 | 0.935 | 0.967 | 0.954 | 0.988 | 0.967 | 0.934 | 0.962 | 0.981 | 0.964 |
| Seedream 5.0 Pro | 0.966 | 0.948 | 0.864 | 0.907 | 0.952 | 0.919 | 0.954 | 0.951 | 0.903 | 0.934 | 0.969 | 0.959 |
| Open-weight Reference Models | ||||||||||||
| Model | IF | VQ | DP | |||||
|---|---|---|---|---|---|---|---|---|
| Fidelity | Selection | Integrity | Coherence | Presentation | Context | Structure | Identity | |
| Closed-source Models | ||||||||
| Nano Banana 2 | 0.941 | 0.952 | 0.980 | 0.956 | 0.978 | 0.976 | 0.964 | 0.986 |
| GPT Image 2 | 0.963 | 0.969 | 0.989 | 0.975 | 0.987 | 0.976 | 0.979 | 0.984 |
| Seedream 5.0 Pro | 0.945 | 0.961 | 0.983 | 0.959 | 0.978 | 0.970 | 0.979 | 0.984 |
| Open-weight Reference Models | ||||||||
| Setting | Mean | Max. | Spearman | Same tier |
|---|---|---|---|---|
| Default | 0.0000 | 0.0000 | 1.000 | 19/19 |
| Equal weights | 0.0016 | 0.0052 | 1.000 | 19/19 |
| IF | 0.0009 | 0.0038 | 1.000 | 19/19 |
| IF | 0.0011 | 0.0045 | 1.000 | 19/19 |
| VQ | 0.0007 | 0.0029 | 1.000 | 19/19 |
| VQ | 0.0009 | 0.0035 | 1.000 | 19/19 |
| Setting | Unchanged | Tier changes |
| Q-tier: 19 models, GPT-4o | ||
| 15/19 | LongCat-Image-Edit, DeepGen 1.0, Lance-3B and Ovis-U1-3B: | |
| 18/19 | DeepGen 1.0: | |
| 19/19 | None | |
| 17/19 | Nano Banana 2 and Seedream 5.0 Pro: | |
| D-tier: six on-device models, iPhone 17 Pro | ||
Image Editing Evaluation
Image editing evaluation measures whether edits follow instructions while preserving identity, context, and visual quality.
See all 92 papers