MoI-Bench: A Benchmark for Image Generation and Editing on Mobile Devices

Kailai Feng; Haoxiang Rao; Yuxiang Wei; Bo Chen; Junsong Li; Yang Pan; Chengkun Cai; Jagmohan Chauhan

cs.CVOct 27, 2026

MoI-Bench: A Benchmark for Image Generation and Editing on Mobile Devices

Authors: Kailai Feng, Haoxiang Rao, Yuxiang Wei, Bo Chen, Junsong Li, Yang Pan, Chengkun Cai, Jagmohan Chauhan

Organizations: University College London · Nanjing University · The Hong Kong Polytechnic University · East China Normal University

Abstract

Developing on-device image models requires considering practical user needs, task types and device resource constraints. However, existing benchmarks provide limited guidance on identifying capability gaps between on-device models and server-side models. We introduce MoI-Bench, a benchmark grounded in mobile user needs, comprising 64 subtasks and 1150 samples. It defines 16 operations, such as removal and replacement, and eight visual target categories covering content such as portraits and text. These form two orthogonal axes whose applicable combinations define specific tasks, such as removing facial makeup. To better guide model improvement, we further propose MoI-Eval, which introduces task-specific criteria covering instruction following, visual quality and detail preservation. We further construct sample-level evaluation questions and aggregate their scores into overall quality and capability profiles along the operation, target and criteria axes. We evaluate representative on-device and server-side models and measure mobile deployment costs. Results show that this decoupled Operation x Target design provides finer-grained guidance for improving compact models. For example, it highlights the need to prioritize specific capability gaps (e.g., text-replacement) over broad application-level goals (e.g., improving text editing). Unlike conventional benchmarks that rank models using a single score, we introduce a two-dimensional Quality-Deployability capability map to assess models for practical mobile use. The map shows that current on-device models can achieve efficient generation or editing but expanding their task coverage deserves greater attention than further improving deployment efficiency. Further progress requires improving editing quality, particularly identity preservation and text rendering, while broadening task coverage.

Figures & tables

Figure 1: Overview of our benchmark and evaluation framework: (a) tasks grounded in mobile user needs; (b) structured capability evaluation along operation, target and criteria axes; and (c) model improvement guided by the Quality-Deployability map.

Table 1: The Operation

\times

Target Task Matrix of MoI-Bench. Each cell describes the corresponding task. Merged cells indicate tasks shared across target categories; N/A marks combinations excluded from the benchmark.

Figure 2: (a) Survey responses on interest in on-device image AI, everyday use cases and user priorities. (b–d) Sample distributions by operation, target and input configuration. Each sample is assigned to one target category according to the primary visual content to generate or modify.

Text-to-image benchmarks
Benchmark	Size	Task coverage								Max. Refs	MT
Benchmark	Size	T2I	Loc.	Glob.	Act.	Pos.	View	Impl.	Multi-obj.	Max. Refs	MT
GenEval Ghosh et al. (2023)	553	$\checkmark$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	0	$\times$
GenAI-Bench Li et al. (2024a)	1,600	$\checkmark$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	0	$\times$
Qwen-Image-Bench Li et al. (2026)	1,000	$\checkmark$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	0	$\times$
General image-editing benchmarks

Table 2: Task coverage across generation and editing benchmarks. Except for T2I, task columns describe editing tasks and may overlap.

\checkmark

: documented coverage;

\times

: outside the stated scope. Act., Pos., Impl. denote action, object-position or spatial-relation changes, implicit editing respectively; Max. Refs counts reference images excluding the base image and MT denotes multi-turn editing. a ImgEdit-Bench includes training and evaluation prompts.

Quality				Deployability
Tier	Overall $Q$	Coverage $C_{0.6}$	Each group	Tier	Size (MiB)	E2E (s)	Memory (MiB)
Q4	$\geq 0.90$	$\geq 90\%$	$\geq 0.90$	D4	$\leq 1{,}024$	$\leq 2$	$\leq 2{,}048$
Q3	$\geq 0.70$	$\geq 60\%$	$\geq 0.60$	D3	$\leq 2{,}048$	$\leq 30$	$\leq 4{,}096$
Q2	$\geq 0.30$	—	—	D2	—	—	—
Q1	$<0.30$	—	—	D1	Unavailable Deployment

Table 3: Quality and deployability tiers. All listed requirements must be satisfied, with higher tiers taking precedence.

C_{0.6}

denotes the proportion of subtasks with mean quality scores of at least

0.6

. The group requirement applies to each of IF, VQ and DP. Q4 and D4 are the highest tiers. D2–D4 require successful generation of valid outputs on the target device. — indicates no additional constraint.

Closed-source Models
Model	Tasks	Type	GPT-4o			Gemini 3.1 Pro			Q-tier
Model	Tasks	Type	$Q$	$C_{0.6}$	(IF, VQ, DP)	$Q$	$C_{0.6}$	(IF, VQ, DP)	Q-tier
Nano Banana 2 Google DeepMind (2026)	Both	–	0.95	1.00	(0.95, 0.97, 0.97)	0.93	1.00	(0.94, 0.96, 0.96)	Q4
Seedream 5.0 Pro ByteDance Seed (2026)	Both	–	0.95	1.00	(0.95, 0.97, 0.98)	0.93	1.00	(0.93, 0.97, 0.96)	Q4
GPT Image 2 OpenAI (2026)	Both	–	0.97	1.00	(0.97, 0.98, 0.98)	0.96	1.00	(0.96, 0.98, 0.96)	Q4
Open-weight Reference Models

Table 4: Overall quality and task coverage evaluated by GPT-4o and Gemini 3.1 Pro. Tasks indicates support for generation (Gen.), editing (Edit), or both (Both); Both does not imply support for all subtasks. Type denotes the image generation mechanism: diffusion-based (Diff., including flow matching), autoregressive (AR).

C_{0.6}

denotes the proportion of subtasks with mean quality scores of at least

0.6

. Criterion-group scores are reported as (IF, VQ, DP). Both judges yield the same quality tiers under the thresholds in Tab. 3 ; Q4 is the highest tier. Within each model group, models are ordered by ascending quality tier and then GPT-4o overall quality.

Figure 3: Qualitative comparison of representative image generation and editing models on MoI-Bench across the five operation groups. Zoom in for better quality.

Figure 4: Capability profiles of representative models across (a) operations, (b) targets and (c) evaluation criteria, scored by GPT-4o. Higher scores indicate better performance. Radial scales differ across subplots.

Closed-source Models
Model	Generation	Local Editing				Global Editing				Understand. Editing			Ref. Editing
Model	T2I	Add.	Rem.	Repl.	Adj.	Style	Bkg.	Rest.	View	Sem.	Outp.	Inp.	R1	B1+R1	B1+R2	B1+R3
Nano Banana 2	0.99	0.94	0.95	0.98	0.96	0.99	0.97	0.92	0.80	0.95	0.95	0.99	0.96	0.93	0.88	0.80
Seedream 5.0 Pro	0.98	0.95	0.96	0.98	0.96	0.94	0.97	0.92	0.83	0.93	0.87	1.00	0.97	0.94	0.90	0.99
GPT Image 2	0.99	0.97	0.97	0.99	0.98	0.99	0.99	0.95	0.82	0.95	0.92	1.00	0.97	0.96	0.87	0.99
Open-weight Reference Models

Table 5: Operation-level quality evaluated by GPT-4o. Scores average sample quality over the retained samples for each operation. Unsupported tasks contribute zero. Add., Rem., Repl. and Adj. denote addition, removal, replacement and adjustment. Style, Bkg., Rest. and View denote stylization, background modification, restoration and viewpoint changes. Sem., Outp. and Inp. denote semantic editing, outpainting and inpainting. R1 uses one reference image and B1+R1–B1+R3 use one base image and one to three references.

Closed-source Models
Model	General Objects		Animals	Portraits				Text&Docs.
Model	Concrete	Abstract	Animals	Body&Limbs	Face&Hair	Cloth.&Accs.	Multi-subjects	Text&Docs.
Nano Banana 2	0.95	0.95	0.99	0.94	0.95	0.97	0.94	0.93
Seedream 5.0 Pro	0.93	0.98	0.96	0.95	0.96	0.99	0.95	0.94
GPT Image 2	0.95	0.97	0.98	0.96	0.96	0.99	0.97	0.96
Open-weight Reference Models

Table 6: Target-level quality evaluated by GPT-4o across the eight target categories. Scores average sample quality using the sample-level target assignments. Unsupported tasks contribute zero, and confirmed execution errors are excluded. Scores are rounded to two decimal places.

Figure 5: Operation $\times$ Target quality profiles evaluated by GPT-4o. Cells report mean sample quality for each task. N/A denotes excluded combinations, and $\dagger$ marks unsupported tasks scored as zero. Merged cells indicate shared tasks.

Closed-source Models
Model	IF		VQ			DP
Model	Fidelity	Selection	Integrity	Coherence	Presentation	Context	Structure	Identity
Nano Banana 2	0.94	0.95	0.98	0.96	0.98	0.98	0.96	0.99
Seedream 5.0 Pro	0.94	0.96	0.98	0.96	0.98	0.97	0.98	0.98
GPT Image 2	0.96	0.97	0.99	0.97	0.99	0.98	0.98	0.98
Open-weight Reference Models

Table 7: Capability profiles for the eight evaluation criteria, evaluated by GPT-4o. IF, VQ and DP denote instruction following, visual quality and detail preservation. Scores follow the existing criterion-profile aggregation and include zeros for unsupported tasks. Scores are rounded to two decimal places.

Model	Params.	Res.	Steps	iPhone 17 Pro			Xiaomi 15 Pro			D-tier
Model	Params.	Res.	Steps	Size (MiB)	E2E (s)	Mem. (MiB)	Size (MiB)	E2E (s)	Mem. (MiB)	D-tier
Janus-Pro-1B †	2.1B	$384^{2}$	–	2975.4	58.26	1505.4	2974.1	76.01	3253.8	D2
Mobile-O-0.5B	1.7B	$512^{2}$	20	1718.3	2.81	426.8	1770.2	13.30	1757.1	D3
SANA-Sprint-0.6B	3.5B	$1024^{2}$	2	2860.9	4.49	1727.2	3069.1	25.24	3896.7	D2
SANA-600M	3.5B	$1024^{2}$	18	2857.2	15.33	1775.2	3065.4	139.68	3897.6	D2
DreamLite	2.5B	$1024^{2}$	4	1684.6	3.22	1305.7	1804.5	18.19	2390.5	D3

Table 8: Mobile deployment results on the iPhone 17 Pro and Xiaomi 15 Pro. Params. includes all parameterized components required for inference, including encoders and image codecs. Res. and Steps denote output resolution and denoising steps. Size, E2E and Mem. denote model storage, end-to-end inference latency and peak application memory. Memory is measured using physical footprint on iOS and process PSS on Android. † The Android Janus-Pro deployment uses precomputed prompt embeddings; its reported latency excludes prompt preparation.

Model	Precision		iPhone 17 Pro				Xiaomi 15 Pro
Model	Backbone	Decoder	Size	E2E	Mem.	Status	Size	E2E	Mem.	Status
Janus-Pro-1B	FP16	W8A16	2935.0	58.96	1502.2	C	2934.0	77.59	3253.8	C
	W8A16	FP16	1764.4	54.54	1500.8	C	–	–	–	F
	W8A16	W8A16	1724.0	54.28	1503.6	C	–	–	–	F
Mobile-O-0.5B	FP16	W8A16	1580.1	2.84	449.2	C	1619.8	12.49	1767.0	C
	W8A16	FP16	1155.7	2.68	423.5	C	1209.3	8.21	3149.2	C

Table 9: Deployment results under different backbone and decoder precisions on the iPhone 17 Pro and Xiaomi 15 Pro. Size and Mem. are measured in MiB, and E2E latency is measured in seconds. Blank cells indicate results pending verification or measurement; – denotes confirmed execution failure. Status describes the inspected outputs: C denotes recognizable outputs and F denotes severe quality degradation or failed execution.

Model	Cond.	Generation			Decode	Other	Total
Model	Cond.	Steps	Branches	Time/step	Decode	Other	Total
Janus-Pro-1B	2.214	576	2	0.0971	0.083	0.042	58.256
Mobile-O-0.5B	0.095	20	2	0.1134	0.230	0.216	2.809
SANA-Sprint-0.6B	0.281	2	1	0.3920	3.154	0.270	4.488
SANA-600M	0.298	18	2	0.6378	3.035	0.515	15.328
DreamLite	0.065	4	1	0.7645	0.036	0.065	3.224

Table 10: Latency breakdown on the iPhone 17 Pro with FP16 backbones and decoders. Times are in seconds; – denotes unavailable results.

Figure 6: Quality and deployability of representative models. Left: GPT-4o quality versus iOS inference latency, with color indicating task coverage and bubble area indicating total parameters; diamonds denote undisclosed parameter counts. Models without iOS latency measurements are shown separately. Right: quality and deployability tiers.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Topic	Question and response options
Q1a. Current usage	How often do you use AI tools on your phone, such as chat assistants, image generators or photo editors? Options: Daily; Several times a week; About once a week; Less than once a week; Never.
Q1b. Image tools	Would you like to generate or edit images using AI tools on your phone? Options: Yes; Maybe; No.
Q1c. On-device interest	Would you be interested in image AI that runs directly on your phone without uploading your photos to a server? Options: Yes; Maybe; No.
Q2. Desired use cases	Which of the following would you most like to use AI image tools for on your phone? Select up to eight options. Options: (1) Retouch portraits, such as removing blemishes or wrinkles. (2) Remove unwanted people or objects from photos. (3) Restore old, blurry or damaged photos. (4) Edit or translate text in images and documents. (5) Improve or creatively edit pet photos. (6) Make memes or stickers. (7) Create personal avatars or profile pictures. (8) Generate wallpapers. (9) Try different hairstyles, makeup or accessories. (10) Change clothing or preview outfit combinations. (11) Replace backgrounds in personal or product photos. (12) Turn photos into cartoons, paintings or other styles. (13) Create posters, invitations or greeting cards. (14) Improve product photos for online listings. (15) Preview furniture or decorations in a room. (16) Combine people or objects from several photos.
Q3. Usability requirements	Suppose you ask an AI tool to remove a small blemish from your cheek in a portrait and leave the rest of the photo unchanged. What would matter to you when judging the result? Select up to eight options. Options: (1) Results are returned quickly. (2) The requested blemish is removed. (3) Photos remain private. (4) Clothing, background and other people remain unchanged. (5) The skin retains natural texture without visible artifacts. (6) The tool requires little storage. (7) The person remains recognisable, with facial features preserved. (8) The tool works offline. (9) Only the intended area is retouched. (10) The overall photo remains clear and visually balanced. (11) The tool does not slow down the phone. (12) The original pose, framing and positions remain unchanged. (13) The retouched area matches the surrounding skin tone and lighting.

Appendix

Table 11: Questionnaire structure and one of the 25 Q3 scenarios. Q1 contains three single-choice questions. Q2 allows up to eight selections. Each Q3 scenario presents 10–15 requirements and allows up to eight selections.

Group	Operation	Definition
Generation	T2I	Generate an image from text without an input image.
Local editing	Addition	Add specified content while retaining existing, unrelated content.
	Removal	Delete specified content and complete the exposed region.
	Replacement	Replace selected content with a specified alternative.
	Adjustment	Change attributes of existing content, such as color, size or pose.
Global editing	Stylization	Apply a visual style across the image.

Appendix

Table 12: Definitions of the 16 operation units.

Group	Target	Definition
General Objects	Concrete Objects	Discrete entities such as vehicles, furniture and everyday objects.
	Abstract Objects	Visual effects and relations, such as lighting, shadows and reflections.
Animals	Animals	Animals, including pets, their appearance and their accessories.
Portraits	Body & Limbs	Human body shape, limb configuration and pose.
	Face & Hair	Facial features, expressions, makeup, hairstyles and identity.
	Clothing & Accessories	Human garments and accessories, such as coats, glasses and jewelry.

Appendix

Table 13: Definitions of the eight target categories.

Group	Criterion	Definition
IF	Fidelity	Satisfying the requested outcome and explicit constraints.
	Selection	Applying the change to the correct region or instance.
VQ	Integrity	Well-formed generated or edited content without defects or artifacts.
	Coherence	Consistent boundaries, lighting and appearance within the output and around edited regions.
	Presentation	Overall readability, composition and physical plausibility.
DP	Context	Preserving non-target objects and background details.

Appendix

Table 15: Definitions of the eight evaluation criteria.

Criterion	Rubric focus and questions	Scores
Fidelity	Remove the requested person. (1) Is the leftmost person completely removed from the image without leaving visible remnants?	0.6
Selection	Edit the correct person only. (1) Did the removal target only the leftmost person (man in dark t-shirt), leaving the adjacent man in the white t-shirt untouched?	1
Integrity	Reconstruct complete shapes and clean textures. (1) Is the reconstructed background on the far left structurally complete without holes, repeated parts, or malformed shapes? (2) Are the textures in the repaired region on the left free of smearing, ringing, or synthetic artifacts?	1, 0.6
Coherence	Match the surrounding geometry, colour and lighting. (1) Does the reconstructed region on the left continue the surrounding street and background geometry without a visible seam? (2) Do the color and lighting in the filled area on the left match the neighboring image area?	1, 0.6
Presentation	Maintain overall clarity and plausible composition. (1) Is the final image clear and usable without awkward crop or weak focus? (2) Is the overall scene physically plausible and balanced in scale and composition?	1, 1
Context	Preserve the unedited background. (1) Are the buildings in the background unchanged in appearance? (2) Is the street surface outside the removed region preserved in color, texture, and lighting?	1, 1

Appendix

Table 16: A complete sample-grounded question set. Scores refer to one Ovis-U1-3B output evaluated by GPT-4o.

Operation	IF	VQ	DP
Text-to-image	0.60	0.40	—
Addition, removal, replacement	0.45	0.25	0.30
Adjustment	0.40	0.25	0.35
Stylization, background editing	0.45	0.35	0.20
Restoration	0.40	0.45	0.15
Viewpoint editing	0.45	0.30	0.25

Appendix

Table 17: Criterion-group weights by operation. A dash denotes an inapplicable group.

Model	Overall quality $Q$		$\Delta Q$	Relative change	Subtasks with quality $\geq 0.6$	Q-tier
Model	Original	Text-NF4	$\Delta Q$	Relative change	Subtasks with quality $\geq 0.6$	Q-tier
Ovis-U1-3B	0.541734	0.538307	$-0.003427$	$-0.63\%$	$24/64\rightarrow 26/64$	Q2 $\rightarrow$ Q2
DreamLite	0.523777	0.520798	$-0.002979$	$-0.57\%$	$27/64\rightarrow 25/64$	Q2 $\rightarrow$ Q2
SANA-600M	0.102904	0.101979	$-0.000925$	$-0.90\%$	$7/64\rightarrow 7/64$	Q1 $\rightarrow$ Q1

Appendix

Table 18: Effect of 4-bit NF4 text-encoder quantization on quality.

\Delta Q

is the quantized score minus the original score. Coverage and tier transitions are shown from the original to the quantized configuration.

Closed-source Models
Model	Operation					Target				Criteria
Model	Gen.	Local	Global	Underst.	Ref.	General	Animals	Portraits	Text.	IF	VQ	DP
Nano Banana 2	0.969	0.943	0.889	0.885	0.938	0.925	0.971	0.941	0.897	0.939	0.957	0.962
GPT Image 2	0.981	0.968	0.927	0.935	0.967	0.954	0.988	0.967	0.934	0.962	0.981	0.964
Seedream 5.0 Pro	0.966	0.948	0.864	0.907	0.952	0.919	0.954	0.951	0.903	0.934	0.969	0.959
Open-weight Reference Models

Appendix

Table 19: Capability profiles evaluated by Gemini 3.1 Pro ( gemini-3.1-p-priority ). General and Text. denote general objects and text and documents. IF, VQ and DP denote instruction following, visual quality and detail preservation. Target scores use the same sample categories as the GPT-4o results. Scores range from

0

1

, with higher values indicating better performance.

Closed-source Models
Model	IF		VQ			DP
Model	Fidelity	Selection	Integrity	Coherence	Presentation	Context	Structure	Identity
Nano Banana 2	0.941	0.952	0.980	0.956	0.978	0.976	0.964	0.986
GPT Image 2	0.963	0.969	0.989	0.975	0.987	0.976	0.979	0.984
Seedream 5.0 Pro	0.945	0.961	0.983	0.959	0.978	0.970	0.979	0.984
Open-weight Reference Models

Appendix

Table 20: Scores for the eight evaluation criteria, evaluated by GPT-4o ( gpt-4o-2024-11-20 ). IF, VQ and DP denote instruction following, visual quality and detail preservation. Scores range from

0

1

, with higher values indicating better performance.

Figure 7: Additional qualitative comparisons for generation and local addition (examples 1–12).

Figure 8: Additional qualitative comparisons for local edit (examples 13–24).

Figure 9: Additional qualitative comparisons for local edit (examples 25–36).

Figure 10: Additional qualitative comparisons for global edit and understanding edit (examples 37–48).

Figure 11: Additional qualitative comparisons for reference edit (examples 49–60).

Figure 12: Additional qualitative comparisons for reference edit (examples 61–64).

Setting	Mean $\|\Delta Q\|$	Max. $\|\Delta Q\|$	Spearman $\rho$	Same tier
Default	0.0000	0.0000	1.000	19/19
Equal weights	0.0016	0.0052	1.000	19/19
IF $+25\%$	0.0009	0.0038	1.000	19/19
IF $-25\%$	0.0011	0.0045	1.000	19/19
VQ $+25\%$	0.0007	0.0029	1.000	19/19
VQ $-25\%$	0.0009	0.0035	1.000	19/19

Appendix

Table 21: Sensitivity to criterion-group weights under GPT-4o. Changes are relative to the default weights across 19 models. Same tier counts unchanged assignments under the final quality thresholds.

Setting	Unchanged	Tier changes
Q-tier: 19 models, GPT-4o
$\delta=-0.05$	15/19	LongCat-Image-Edit, DeepGen 1.0, Lance-3B and Ovis-U1-3B: $Q2\rightarrow Q3$
$\delta=-0.02$	18/19	DeepGen 1.0: $Q2\rightarrow Q3$
$\delta=+0.02$	19/19	None
$\delta=+0.05$	17/19	Nano Banana 2 and Seedream 5.0 Pro: $Q4\rightarrow Q3$
D-tier: six on-device models, iPhone 17 Pro

Appendix

Table 22: Sensitivity to tier thresholds relative to Tab. 3 . Quality requirements are shifted by

\delta

and deployment limits are scaled by

\alpha

. Unchanged counts are relative to the default tiers.

Image Editing Evaluation

Image editing evaluation measures whether edits follow instructions while preserving identity, context, and visual quality.

See all 92 papers

May 27, 2026

BlazeEdit: Generalist Image Editing on Mobile Devices with Image-to-Image Diffusion Models

The remarkable generation quality of modern diffusion models often comes at the cost of massive parameter counts, which necessitate server-side inference with significant computational costs and potential privacy risks. Consequently, there is growing momentum toward developing efficient on-device alternatives. While recent efforts have optimized text-to-image models for mobile hardware, they remain relatively bulky, typically ranging from 0.5B to 1B parameters. We present BlazeEdit, a highly efficient, generalist image-to-image diffusion model tailored for on-device deployment. By identifying that many practical image editing tasks do not require text-based guidance, we eliminate the text-conditioning components and develop a multi-task architecture that consolidates object removal, outpainting, tone correction, relighting, and sticker generation into a single, compact model of only 195M parameters. BlazeEdit achieves a substantial reduction in download size and memory overhead while maintaining competitive generation quality. It completes a full inference pass in just 290ms on a Pixel 10, delivering a seamless, privacy-preserving, and lightning-fast experience for generalist image editing on the edge.

Fei Deng, Yanwu Xu, Zhipeng Bao +4

Google

Diffusion Model Acceleration Diffusion-Based Image Editing Image Editing Diffusion Models On-Device Inference

Ask PDF

Mar 16, 2026

Omni IIE Bench: Benchmarking the Practical Capabilities of Image Editing Models

While Instruction-based Image Editing (IIE) has achieved significant progress, existing benchmarks pursue task breadth via mixed evaluations. This paradigm obscures a critical failure mode crucial in professional applications: the inconsistent performance of models across tasks of varying semantic scales. To address this gap, we introduce Omni IIE Bench, a high-quality, human-annotated benchmark specifically designed to diagnose the editing consistency of IIE models in practical application scenarios. Omni IIE Bench features an innovative dual-track diagnostic design: (1) Single-turn Consistency, comprising shared-context task pairs of attribute modification and entity replacement; and (2) Multi-turn Coordination, involving continuous dialogue tasks that traverse semantic scales. The benchmark is constructed via an exceptionally rigorous multi-stage human filtering process, incorporating a quality standard enforced by computer vision graduate students and an industry relevance review conducted by professional designers. We perform a comprehensive evaluation of 8 mainstream IIE models using Omni IIE Bench. Our analysis quantifies, for the first time, a prevalent performance gap: nearly all models exhibit a significant performance degradation when transitioning from low-semantic-scale to high-semantic-scale tasks. Omni IIE Bench provides critical diagnostic tools and insights for the development of next-generation, more reliable, and stable IIE models.

Yujia Yang, Yuanxiang Wang, Zhenyu Guan +11

University of Chinese Academy of Sciences · Tencent

Image Editing Image Editing Evaluation Text-Guided Image Editing Benchmark Design

Ask PDF

Mar 30, 2026

DreamLite: A Lightweight On-Device Unified Model for Image Generation and Editing

Diffusion models have made significant progress in both text-to-image (T2I) generation and text-guided image editing. However, these models are typically built with billions of parameters, leading to high latency and increased deployment challenges. While on-device diffusion models improve efficiency, they largely focus on T2I generation and lack support for image editing. In this paper, we propose DreamLite, a compact unified on-device diffusion model (0.39B) that supports both T2I generation and text-guided image editing within a single network. DreamLite is built on a pruned mobile U-Net backbone and unifies conditioning through in-context spatial concatenation in the latent space. It concatenates images horizontally as input, using a (target | blank) configuration for generation tasks and (target | source) for editing tasks. To stabilize the training of this compact model, we introduce a task-progressive joint pretraining strategy that sequentially targets T2I, editing, and joint tasks. After high-quality SFT and reinforcement learning, DreamLite achieves GenEval (0.72) for image generation and ImgEdit (4.11) for image editing, outperforming existing on-device models and remaining competitive with several server-side models. By employing step distillation, we further reduce denoising processing to just 4 steps, enabling our DreamLite could generate or edit a 1024 x 1024 image in less than 1s on a Xiaomi 14 smartphone. To the best of our knowledge, DreamLite is the first unified on-device diffusion model that supports both image generation and image editing.

Kailai Feng, Yuxiang Wei, Bo Chen +5

Intelligent Creation Lab, ByteDance

T2I Diffusion Models Diffusion-Based Image Editing On-Device Inference

Ask PDF

Abstract

Image Editing Evaluation

Explore similar work

BlazeEdit: Generalist Image Editing on Mobile Devices with Image-to-Image Diffusion Models

Omni IIE Bench: Benchmarking the Practical Capabilities of Image Editing Models

DreamLite: A Lightweight On-Device Unified Model for Image Generation and Editing