PersonalQ: Select, Quantize, and Serve Personalized Diffusion Models for Efficient Inference

Qirui Wang; Qi Guo; Yiding Sun; Junkai Yang; Dongxu Zhang; Shanmin Pang; Qing Guo

cs.AIMar 24, 2026

PersonalQ: Select, Quantize, and Serve Personalized Diffusion Models for Efficient Inference

Authors: Qirui Wang, Qi Guo, Yiding Sun, Junkai Yang, Dongxu Zhang, Shanmin Pang, Qing Guo

Organizations: School of Software Engineering, Xi’an Jiaotong University, China · Nankai University, China

Abstract

Personalized text-to-image generation lets users fine-tune diffusion models into repositories of concept-specific checkpoints, but serving these repositories efficiently is difficult for two reasons: natural-language requests are often ambiguous and can be misrouted to visually similar checkpoints, and standard post-training quantization can distort the fragile representations that encode personalized concepts. We present PersonalQ, a unified framework that connects checkpoint selection and quantization through a shared signal -- the checkpoint's trigger token. Check-in performs intent-aligned selection by combining intent-aware hybrid retrieval with LLM-based reranking over checkpoint context and asks a brief clarification question only when multiple intents remain plausible; it then rewrites the prompt by inserting the selected checkpoint's canonical trigger. Complementing this, Trigger-Aware Quantization (TAQ) applies trigger-aware mixed precision in cross-attention, preserving trigger-conditioned key/value rows (and their attention weights) while aggressively quantizing the remaining pathways for memory-efficient inference. Experiments show that PersonalQ improves intent alignment over retrieval and reranking baselines, while TAQ consistently offers a stronger compression-quality trade-off than prior diffusion PTQ methods, enabling scalable serving of personalized checkpoints without sacrificing fidelity.

Figures & tables

Fig. 1: Problem and overview. (a) Natural-language prompts must be routed to the correct personalized checkpoint; existing methods often misroute queries. (b) Standard post-training quantization degrades personalized concepts.

Fig. 2: Trigger token (e.g., <sks> ) is vulnerable under quantization. Following the token-specific sensitivity test in Sec. III-B , we quantize only the cross-attention Key/Value rows of the trigger token while keeping all other rows in full precision. Under 4-bit quantization, the trigger token exhibits much larger degradation than ordinary words, motivating our TAQ method.

Fig. 3: PersonalQ. Input: prompt $p$ and a personalized checkpoint repository. Output: image $\hat{\mathbf{y}}$ from low-bit inference. (a) Check-in selects a checkpoint $c^{\star}$ using prompt semantics and metadata, then rewrites $p$ into $p^{\prime}$ by inserting the checkpoint’s trigger (e.g., bear $\rightarrow$ <bear-v4> ). (b) TAQ quantizes cross-attention with trigger-aware mixed precision: it keeps trigger-conditioned K/V rows (and their attention pathways) in higher precision and quantizes the remaining activations for efficient inference with preserved personalization.

Checkpoint Model	Quantization Method	Bit-width	BOPs	MS-COCO		PartiPrompts
Checkpoint Model	Quantization Method	(W/A)	(T)	FID $\downarrow$	CLIP $\uparrow$	FID $\downarrow$	CLIP $\uparrow$
Stable Diffusion-v1-5	Full Precision	32/32	893	10.96	0.315	9.77	0.336
	Q-Diffusion	8/8	56	27.16	0.261	23.44	0.267
	TFMQ-DM	8/8	56	24.34	0.279	21.66	0.299
	DGQ	8/8	56	15.24	0.291	13.26	0.317
	TAQ	8/8	56	11.03	0.297	10.49	0.327

TABLE I: Quantitative Comparison. Results of different quantization methods for personalized diffusion checkpoints.

Panel of Figure 5 — Fig. 4: Qualitative comparison. (a) SDXL-Turbo with W8A8 (8-bit weights / 8-bit activations). (b) Stable Diffusion v1.5 with W8A8 (8-bit weights / 8-bit activations); Each column corresponds to a different personalized checkpoint.

Selection Method	Intent Score $\uparrow$	Human Pref. $p\uparrow$
Random	2.14 $\pm$ 0.82	0.891
Reranker	3.21 $\pm$ 0.76	0.857
Stylus	3.68 $\pm$ 0.69	0.821
Check-in	4.42 $\pm$ 0.51	1.000

TABLE II: Checkpoint selection. Intent Score : mean

\pm

std LLM-judge rating (1–5; temp=0) over subject/style/temporal/context. Human Preference (

p

): fraction of pairwise votes where Check-in is preferred over the method (

p=\Pr(\text{\emph{Check-in}}\succ\text{method})

; higher is better). We set

p=1.00

for Check-in .

Retr.	Rsn.	Intent Score $\uparrow$	FID $\downarrow$	CLIP $\uparrow$
$\times$	$\times$	3.21	11.74	0.289
$\times$	✓	3.99	11.61	0.291
✓	$\times$	4.19	11.23	0.293
✓	✓	4.42	11.03	0.297

TABLE III: Effect of Check-in components. Retr. : intent-aware retrieval; Rsn. : personalized reasoning. Clarification is enabled on-demand in all variants (held constant).

Bits(W/A)	Quantizer	Separate trigger tokens	FID $\downarrow$	CLIP $\uparrow$
8/8	Linear	$\times$	15.83	0.292
8/8	Linear	✓	11.04	0.298
8/8	Logarithmic	$\times$	17.62	0.287
8/8	Logarithmic	✓	13.67	0.294
8/4	Linear	$\times$	54.12	0.245
8/4	Linear	✓	44.53	0.262

TABLE IV: Effect of TAQ and trigger-token separation.

Bits(W/A)	Check-in	TAQ	Intent score $\uparrow$	FID $\downarrow$	CLIP $\uparrow$
8/8	$\times$	$\times$	3.22	15.22	0.286
	✓	$\times$	4.39	14.81	0.294
	$\times$	✓	3.21	11.74	0.289
	✓	✓	4.42	11.03	0.297
8/4	$\times$	$\times$	3.21	54.12	0.245
	✓	$\times$	4.40	51.31	0.253

TABLE V: Effect of component synergy.

Backbone	Intent Score	Retrieve	Reason	Clarify	Generation (W8A8)	End-to-End	Multi-turn
	Performance	Inference time (s)
GPT-4o (API)	4.42	1.3	20.31	11.17	12.31	45.09	2.1
Gemini 2.5 Flash (API)	4.38	1.3	16.54	8.28	12.31	38.43	2.3
Qwen2.5-VL-72B (API)	4.35	1.3	25.89	10.35	12.31	49.85	2.5

TABLE VI: MLLM latency ablation. Average number of user turns per request (including the initial prompt);

>1

implies clarification.

Diffusion Model Serving

Diffusion model serving optimizes the scheduling, execution, and resource use of diffusion models to meet latency and throughput goals.

See all 16 papers

ZIPP:Zero-shot Image Personalization from Personas
Jun 7, 2026Harini SI, Somesh Singh, Yaman Kumar Singla +2T2I Generation Persona Prompting
Adobe Media and Data Science Research (MDSR) · SUNY at Buffalo · IIIT-Delhi
Naïve PAINE: Lightweight Text-to-Image Generation Improvement with Prompt Evaluation
Mar 12, 2026Joong Ho Kim, Nicholas Thai, Souhardya Saha Dip +2T2I Generation T2I Generation Evaluation
LSU ATHENA Lab, Baton Rouge, LA, U.S.A. · LSU Vision Lab, Baton Rouge, LA, U.S.A.
Anchor to Expand: Semantic Anchoring for Personalized Text-to-Image Diffusion Models
Nov 27, 2025Seoyun Yang, Gihoon Kim, Taesup KimSubject-Driven Image Generation T2I Generation
Graduate School of Data Science, Seoul National University

Jun 7, 2026J/K move · Enter open · S save

ZIPP:Zero-shot Image Personalization from Personas

Harini SI, Somesh Singh, Yaman Kumar Singla, David Doermann+1

Adobe Media and Data Science Research (MDSR) · SUNY at Buffalo · IIIT-Delhi

T2I Generation Persona Prompting Persona-Conditioned Generation

Open paper AskPDF arXiv

Text-to-image diffusion models are increasingly deployed in open-ended creative contexts, yet their outputs remain impersonal, optimized for aggregate aesthetics rather than individual taste. Human preferences are pluralistic: one user favoring muted, nostalgic portraits may prefer vibrant street photography, while another gravitates toward dreamy film aesthetics. Existing methods require dense interaction histories or per-user fine-tuning, failing in cold-start settings and collapsing context-dependent preferences into a static representation. We introduce zero-shot image personalization from personas (ZIPP), which conditions image generation on natural-language personas (concise descriptors of a user's identity and aesthetic sensibilities) without any user-specific data or weight updates. ZIPP uses an LLM to rewrite prompts from the perspective of a given persona, steering diffusion models toward personalized outputs. To mine personas at scale, we train an inductive Graph Attention Network over a 22M-user Reddit interaction graph with dual contrastive objectives aligning graph structure with visual behavior, then verbalize learned representations into natural-language personas via an MLLM. We introduce ZIPBench, the first zero-shot personalization benchmark with 1.5K users, graph-mined personas, and 40K generated images. Across four benchmarks and 14 LLMs spanning five model families, persona conditioning yields consistent gains (13-20%), with frontier models benefiting most. In the few-shot setting, ZIPP matches or exceeds fine-tuned baselines trained on 100+ examples per user. ZIPP achieves the lowest preference distributional divergence (CMMD 0.16 vs. 0.55), and IPF-normalized demographic evaluation shows it substantially reduces subpopulation bias present in existing methods. Human evaluation confirms a 79% win rate over generic generation and 58-65% over all fine-tuned baselines.

Abstract

Diffusion Model Serving

Explore similar work

ZIPP:Zero-shot Image Personalization from Personas

Naïve PAINE: Lightweight Text-to-Image Generation Improvement with Prompt Evaluation

Anchor to Expand: Semantic Anchoring for Personalized Text-to-Image Diffusion Models