Co-occurrence Patterns of LoRA Adapters in Production Diffusion Model Inference Services

Tao Zhang; Bin Liao; Tao Zhou; Yanping Liu

cs.DCSep 20, 2026

Co-occurrence Patterns of LoRA Adapters in Production Diffusion Model Inference Services

Authors: Tao Zhang, Bin Liao, Tao Zhou, Yanping Liu

Organizations: School of Information Engineering, Guizhou University of Traditional Chinese Medicine, Guiyang, 550025, China · School of Big Data and Statistics, Guizhou University of Finance and Economics, Guiyang, 550025, China · School of Mathematics and Statistics, Guizhou University of Finance and Economics, Guiyang, 550025, China · School of Mathematics and Statistics, Beijing Technology and Business University, Beijing, 100048, China

Abstract

Low-rank adaptation (LoRA) has become a key technology for serving large-scale personalized large language models and diffusion models in the cloud. However, the co-occurrence patterns, resource contention relationships, and evolutionary regularities of adapters under production inference workloads have not been systematically or quantitatively studied. Based on GenTD26, Alibaba's production diffusion model inference dataset, this paper adopts a graph-theoretic framework to construct an adapter co-occurrence network and conducts a characterization from both static structure and dynamic evolution. Our main findings are as follows. (1) The co-occurrence network is extremely sparse, and adapter usage frequency follows a significant heavy-tailed distribution. (2) Introducing the first adapter incurs a 66.1% execution-latency overhead, with diminishing marginal costs afterwards. (3) Co-occurrence relationships are driven by base models: in 90.6% of multi-adapter requests, all adapters share the same dominant base model; 66.2% of significant co-occurrence edges connect same-model adapter pairs; and in 85.8% of multi-adapter requests, all adapter pairs form significant co-occurrence edges. (4) The adapter ecosystem exhibits a core-periphery bipolar structure, with a weekly Jaccard similarity of 0.696 at the model level and a churn rate of 54.5% for the top-10 hottest models within a 12-hour window. Based on these findings, we propose a preloading strategy built on top-k co-occurrence statistics; offline experiments show that it covers 81.0% of test-set co-occurrence pairs at k=3, and sensitivity analyses across frequency thresholds and time windows verify the robustness of the conclusions. These results provide a data-driven basis for cache preloading, adaptive scheduling, and GPU memory management in LoRA inference services.

Figures & tables

Attribute	Value
Provider	Alibaba commercial image generation
Duration	November 15 – December 8, 2024 (24 days)
Number of requests	26,823 ( lora_request_trace )
Number of processing records	68,195 ( data_trace_processed )
GPU sampling points	157,417 (duty cycle)
Containers	143

Table 1: Summary of the GenTD26 dataset

Figure 1: LoRA adapter frequency distribution. (a) Top-30 adapters by occurrence, annotated with the Gini coefficient (0.749) and the top-5 share (19.7%), with colors distinguishing frequency tiers (dark blue: ranks 1–5; red: 6–10; light blue: 11–30); (b) log–log frequency–rank plot with the rank–frequency regression fit line ( $\alpha=1.30$ , a descriptive measure; strict power-law fitting is given in the main text).

Number of LoRAs	Number of requests	Mean (s)	Standard deviation (s)	Degradation
0	52,993	23.6	13.4	(baseline)
1	14,112	39.2	25.1	+66.1%
2	754	40.4	21.2	+71.2%
3	277	44.7	16.0	+89.4%
4	43	40.8	12.6	+72.9%
5	10	38.8	11.1	+64.4%

Table 2: Execution time vs. number of LoRA adapters

Figure 2: LoRA count distribution and latency impact. (a) Distribution of the LoRA count per request (77.7% none, 20.7% single, 1.6% multiple); (b) mean execution time with percentage degradation over the no-LoRA baseline (23.6 s).

Figure 3: LoRA loading latency. (a) Histogram of 27,415 loading operations (mean = 4.40 s, median = 4.38 s, $P_{90}=6.14$ s); (b) CDF of loading latency versus execution-time baselines (no-LoRA 23.6 s, single-LoRA 39.2 s); a typical loading operation accounts for only 18.6% of the no-LoRA execution time.

Figure 4: Evolution of the co-occurrence network with the frequency threshold $\tau_{\min}$ . (a)–(e) Network topology at $\tau_{\min}=1,5,10,20,50$ : node size is proportional to adapter frequency, color represents degree centrality (with a unified mapping across panels), gray dots are isolated adapters that only satisfy the frequency threshold, and the red box marks the default threshold 10; (f) network-scale metrics (number of nodes, number of edges, number of connected nodes, number of non-trivial connected components) versus $\tau_{\min}$ on a log scale. The co-occurrence edge threshold is fixed at $w_{\min}=3$ .

Figure 5: Connected core of the co-occurrence network (degree $>0$ ; 51 nodes, 68 edges, edge weight $\geq 5$ ). Node size is proportional to adapter frequency, and color denotes the dominant base model of each adapter (gray = other models). The two largest connected components contain 15 and 8 adapters, respectively.

Figure 6: Bipartite graph of base models and LoRA adapters (top 8 models with their top 10 adapters each; 70 edges). Squares = base models (size proportional to the number of processing-trace records; together accounting for 80.2% of records); circles = adapters (size proportional to frequency; color = dominant base model; black border = cross-model adapter).

$\tau_{\min}$	$\|V\|$	$\|E\|$	Density (%)	Communities	Non-trivial comps.	Same-model over-expr.
5	463	83	0.08	412	13	2.7 $\times$
10	274	68	0.18	233	10	2.6 $\times$
20	154	49	0.42	125	8	2.8 $\times$
50	63	23	1.18	49	5	3.5 $\times$

Table 3: Sensitivity of network characteristics to the frequency threshold

\tau_{\min}

(

w_{\min}=3

)

$\tau_{\min}$	$\|V\|$	$\|E\|$	Density (%)	Communities	Non-trivial comps.	Largest comp.
5	463	83	0.08	412	13	16
10	274	68	0.18	233	10	15
15	195	52	0.27	163	9	13
20	154	49	0.42	125	8	13
30	108	31	0.54	90	5	12
50	63	23	1.18	49	5	11

Table 4: Complete sensitivity analysis across the frequency threshold

\tau_{\min}

(

w_{\min}=3

)

Figure 7: Temporal evolution. (a) Daily LoRA request share (request trace, 24 days; mean 14.7%; no LoRA requests on November 16–17); (b) daily active base model count (mean 34.2; adjacent-day model Jaccard mean 0.497, weekly model Jaccard mean 0.696); (c) distribution of adapter cross-base-model spread ( $\leq 2$ models: 786; 3–17: 83; $\geq 18$ : 5).

Metric	SoCC’25	This paper
Unique adapters	705	874
Proportion of requests carrying LoRA adapters	21.2%	22.3%
Multi-adapter requests	—	1.6%
Rank–frequency exponent $\alpha$	—	1.30
Adapter frequency Gini coefficient	—	0.749
Latency degradation (1 adapter)	69%	66.1%

Table 5: Comparison of LoRA characterization findings

Diffusion Model Serving

Diffusion model serving optimizes the scheduling, execution, and resource use of diffusion models to meet latency and throughput goals.

See all 18 papers

May 7, 2026

Rethinking Adapter Placement: A Dominant Adaptation Module Perspective

Low-rank adaptation (LoRA) is a widely used parameter-efficient fine-tuning method that places trainable low-rank adapters into frozen pre-trained models. Recent studies show that using fewer LoRA adapters may still maintain or even improve performance, but existing methods still distribute adapters broadly, leaving where to place a limited number of adapters to maximize performance largely open. To investigate this, we introduce PAGE (Projected Adapter Gradient Energy), a gradient-based sensitivity probe that estimates the initial trainable gradient energy available to each candidate LoRA adapter. Surprisingly, we find that PAGE is highly concentrated on a single shallow FFN down-projection across two model families and four downstream tasks. We term this module the dominant adaptation module and show that its layer index is architecture-dependent but task-stable. Motivated by this finding, we propose DomLoRA, a placement method that places a single adapter at the dominant adaptation module. With only 0.7% of vanilla LoRA's trainable parameters, DomLoRA outperforms it on average across downstream tasks, including instruction following, mathematical reasoning, coding, and multi-turn conversation. This method also matches or improves other LoRA variants and reduces training time by up to 2.74

\times

compared with broad placement, supporting the dominant adaptation module perspective as a practical placement guideline.

Suoxin Zhang, Run He, Di Fang +3

South China University of Technology, China · Zhejiang University, China

LLM Fine-Tuning Adapter Tuning Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Ask PDF

Aug 4, 2025

PLoRA: Efficient Concurrent LoRA Training for Large Language Models

Low-Rank Adaptation (LoRA) has gained popularity as a fine-tuning approach for Large Language Models (LLMs) due to its low resource requirements and good performance. While numerous studies have investigated ways to improve LoRA serving efficiency by serving multiple LoRAs concurrently, existing methods assume that a wide range of LoRA adapters are available for serving. In our work, we conduct extensive empirical studies to show that current LoRA training paradigms do not efficiently utilize hardware resources and incur high overhead to obtain a performant LoRA adapter. Leveraging these insights, we propose PLoRA, which automatically orchestrates concurrent LoRA fine-tuning jobs under given hardware and model constraints and develops performant kernels to improve training efficiency. Across a range of LLMs and LoRA configurations, PLoRA improves training throughput by up to 12.8x and reduces the overall fine-tuning makespan by up to 7.52x compared to existing approaches.

Minghao Yan, Zhuang Wang, Zhen Jia +2

University of Wisconsin-Madison · Amazon Web Services

LLM Fine-Tuning Efficient Language Model Training Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Ask PDF

Jun 26, 2026

When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning

Low-Rank Adaptation (LoRA) has become the standard tool for parameter-efficient fine-tuning of large pretrained models. When applied sequentially across tasks in Continual Learning (CL), the standard assumption is that each new task requires a dedicated low-rank adapter. In this work, we challenge this assumption empirically and structurally. We show that task-specific LoRA adapters in CL exhibit significant low-rank redundancy: the subspaces spanned by adapters trained on different tasks substantially overlap, and in many cases earlier adapters can faithfully represent later tasks. Building on this observation, we propose LiteLoRA, a plug-and-play gating mechanism that learns at train time whether to recruit a new adapter or reuse existing low-rank representations. Our method reduces the number of active adapters by 20-70% while matching or exceeding state-of-the-art performance on standard CL benchmarks, revealing that structural redundancy is pervasive and that selective learning is sufficient to achieve stability without sacrificing plasticity.

Tanguy Dieudonné, Giulia Lanzillotta, Enis Simsar +2

Department of Computer Science, ETH Zurich, Zurich, Switzerland.

Continual Learning Low-Rank Adaptation Incremental Learning Parameter-Efficient Fine-Tuning Continual LoRA Adapter-Based Continual Learning

Ask PDF

Abstract

Diffusion Model Serving

Explore similar work

Rethinking Adapter Placement: A Dominant Adaptation Module Perspective

PLoRA: Efficient Concurrent LoRA Training for Large Language Models

When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning