Generative Artificial Intelligence (GenAI) features such as image editing, object removal, and prompt-guided image transformation are increasingly integrated into mobile applications. However, deploying Large Vision Models (LVMs) for such tasks on resource-constrained devices remains challenging due to their high memory and compute requirements. While Low-Rank Adapters (LoRAs) enable parameter-efficient task adaptation, existing Mobile deployment pipelines typically compile separate model binaries for each LoRA + a copy of the foundation model, resulting in redundant storage and increased runtime overhead. In this work, we present a unified framework for enabling multi-task GenAI inference on edge devices using a single shared model. Our key idea is to treat LoRA weights as runtime inputs rather than embedding them into the compiled model graph, allowing dynamic task switching at runtime without recompilation. Then, to support efficient on-device execution, we introduce QUAD (Quantization with Unified Adaptive Distillation), a quantizationaware training strategy that aligns multiple LoRA adapters under a shared quantization profile. We implement the proposed system with a lightweight runtime stack compatible with mobile NPUs and evaluate it across multiple chipsets. Experimental results demonstrate up to 6x and 4x reduction in memory footprint and latency improvements, respectively, while maintaining high visual quality across multiple GenAI tasks.
Figures & tables
Figure 1: Schematic for a simplified version of compilation and deployment of LVM and depicting the problem definition
Figure 2: Pictorial representation of proposed approach depicting a) modification of logic to ensure LoRA as input in the architecture, b) offline construction of a single frozen graph that will serve multiple tasks and c) on-device runtime scenario with the proposed approach
Figure 3: Schematic for a simplified version of compilation and deployment of LVM and depicting the problem definition
Figure 4: Flowsheet of QUAD framework
Figure 5: Software stack for facilitating one-for-all LVMs on embedded devices such as smartphones. In this schematic, we show the stack for on-device deployment of two use-cases with a single LVM as described in Section 4
Usecase
Metric
FP32 vs INT8
Prompt
simd
0.9428
guided
simimage
0.881
Image
Structure loss
0.045
Transformation
Custom clip score
0.008
Object
FID
5.5287
Removal
LPIP
0.12
Table 1: Accuracy analysis of quantized model. The prompt guided image transformation use-case is considered as base LoRA and the effect of quantization is studied in its case. In case of Object removal, QUAD is applied and thus the effect of enforcing the shared quantization parameters is presented.
Figure 6: Accuracy comparison in terms of Image Quality between the FP32 model on server and Quantized model deployed on device for the prompt guided Image Transformation use-case
Figure 7: Accuracy comparison in terms of Image Quality between the FP32 model on server(x86) and Quantized model deployed on device (GS25) for the object removal use-case
Chipset
Qualcomm (GS25)
LSI (GS25)
Mediatek (Tab S11)
Scenario
Prompt Guided Image Transformation
Object Removal
Prompt Guided Image Transformation
Object Removal
Prompt Guided Image Transformation
Object Removal
Sampler
OLSS
LCM
OLSS
ED
OLSS
ED
Steps
8
4
8
4
8
4
VAE Encoder execute (ms)
449
458
427
425
639
643
Unet execute (ms)
250
249
409
175
509
254
VAE Decoder execute (ms)
839
836
896
898
1090
1082
Table 2: On-Device Key Performance Indicators (memory and latency) of 2 use-cases with our proposed QUAD framework and inference stack with multi-LoRA as input and a SD 1.5 based 1.1B model as Foundation Model.
Scenario
Text-to-Image
Sketch-to-Image
Sticker Generation
Portrait Studio
VAE Encoder execute (ms)
NA
68
NA
NA
Unet execute (ms)
48
48
48
51
VAE Decoder execute (ms)
150
152
152
149
End-to-End (ms)
1052
1527
1080
1874
Shared FM ROM (MB)
844
LoRA ROM (MB)
NA
77
77
77
Table 3: On-Device Key Performance Indicators (memory and latency) of 4 use-cases with our proposed QUAD framework and inference stack with multi-LoRA as input and a SD 1.5 based 0.7B parameter model as Foundation model(OLSS sampler-8 steps)
Scenario
FID
LPIPS
PSNR
SSIM
0:100
12.2308
0.1083
32.712
0.9808
20:80
12.45
0.1086
32.6766
0.9806
40:60
13.0494
0.1086
32.6819
0.9806
60:40
12.7991
0.109
32.5978
0.9803
80:20
14.2763
0.1125
31.4065
0.9777
90:10
26.6183
0.1372
28.8822
0.9628
Table 4: Ablation analysis. Comparison of accuracy measured on-device (GS 25) in prompt guided Image Transformation use-case when quantized with INT8 precision and mixed bit precision. Proposed model is quantized with W8A16. Each scenario x:y implies x% params in W8A8 and y% params in in W8A16
Scenario
Init Time (ms)
Execute Time (ms)
LoRA ROM size (MB)
FID
SSIM
PSNR
0:100
1156
469
181
25.884
0.8064
25.28
10:90
1144
450
158
27.419
0.7951
25.32
20:80
1101
438
138
27.106
0.7951
25.23
30:70
1097
405
138
27.108
0.7941
25.18
Table 5: Ablation analysis. Comparison of accuracy and on-device KPIs measured on Tab S11 in Object removal use-case when quantized with INT8 precision and mixed bit precision. Proposed model is quantized with W8A16. Each scenario x:y implies x% params in W8A8 and y% params in in W8A16.
Deploying large language models (LLMs) on smartphones poses significant engineering challenges due to stringent constraints on memory, latency, and runtime flexibility. In this work, we present a hardware-aware framework for efficient on-device inference of a LLaMA-based multilingual foundation model supporting multiple use cases on Samsung Galaxy S24 and S25 devices with SM8650 and SM8750 Qualcomm chipsets respectively. Our approach integrates application-specific LoRAs as runtime inputs to a single frozen inference graph, enabling dynamic task switching without recompilation or memory overhead. We further introduce a multi-stream decoding mechanism that concurrently generates stylistic variations - such as formal, polite, or jovial responses - within a single forward pass, reducing latency by up to 6x. To accelerate token generation, we apply Dynamic Self-Speculative Decoding (DS2D), a tree-based strategy that predicts future tokens without requiring a draft model, yielding up to 2.3x speedup in decode time. Combined with quantization to INT4 and architecture-level optimizations, our system achieves 4-6x overall improvements in memory and latency while maintaining accuracy across 9 languages and 8 tasks. These results demonstrate practical feasibility of deploying multi-use-case LLMs on edge devices, advancing the commercial viability of Generative AI in mobile platforms.
Deploying vision-language models (VLMs) on mobile devices is challenging due to their significant memory and compute requirements. We present a framework for quantizing VLMs for efficient inference on resource-constrained hardware. Our approach combines a quantization pipeline that uses the model itself to generate training data and does not require access to the training setup, with a novel 2.7-bit-per-parameter format supporting efficient execution on Arm CPUs. We validate our approach by compressing the Llama 3.2 11B Vision Instruct model to 3.7 GB with 8-bit activations, preserving strong performance on a set of standard visual question answering tasks.
Diffusion Transformers (DiT) have established a new state-of-the-art in high-fidelity image synthesis; however, their massive computational complexity and memory requirements hinder local deployment on resource-constrained edge devices. In this paper, we introduce EdgeDiT, a family of hardware-efficient generative transformers specifically engineered for mobile Neural Processing Units (NPUs), such as the Qualcomm Hexagon and Apple Neural Engine (ANE). By leveraging a hardware-aware optimization framework, we systematically identify and prune structural redundancies within the DiT backbone that are particularly taxing for mobile data-flows. Our approach yields a series of lightweight models that achieve a 20-30% reduction in parameters, a 36-46% decrease in FLOPs, and a 1.65-fold reduction in on-device latency without sacrificing the scaling advantages or the expressive capacity of the original transformer architecture. Extensive benchmarking demonstrates that EdgeDiT offers a superior Pareto-optimal trade-off between Frechet Inception Distance (FID) and inference latency compared to both optimized mobile U-Nets and vanilla DiT variants. By enabling responsive, private, and offline generative AI directly on-device, EdgeDiT provides a scalable blueprint for transitioning large-scale foundation models from high-end GPUs to the palm of the user.