Serving and fine-tuning a large language model on the same GPUs can turn the idle capacity between inference bursts into training progress, and low-rank adaptation (LoRA) makes this co-location cheap because both workloads read a single frozen copy of the base weights. The difficulty lies in the fine-tuning backward pass, which has no counterpart in inference. Current co-serving systems schedule backward work per serving iteration: each iteration carries a fixed backward budget, and requests that arrive in the meantime wait for it to finish, so fine-tuning dictates inference latency whenever load spikes. We observe that the backward pass of a LoRA microbatch decomposes into per-layer tasks over the same weights that the inference forward pass visits, which allows backward work to be scheduled at layer rather than iteration granularity. Building on this observation, LayerLoom attaches same-layer backward tasks to the inference forward pass and, at each layer, runs only as many of them as the latency slack of in-flight requests permits. To keep a ready task at every layer, LayerLoom maintains a wavefront of fine-tuning microbatches staggered across layers, and it bounds the wavefront's activation memory by offloading microbatch state to host memory and prefetching it ahead of the layer where the scheduler plans to use it. A fused kernel serves the paired forward and backward GEMMs from a single read of each base weight. Because all backward work of an iteration finishes within its forward pass, new requests join the next iteration as in continuous batching. On an RTX 4090 GPU running Llama-3.1-8B under a bursty production trace, LayerLoom raises joint SLO attainment by 13.5-21.8 points over prior works with fine-tuning throughput that is comparable or up to 12% higher, and it reaches the same training losses.
Figures & tables
Figure 1: FlexLLM co-serving on an A800 GPU. (a)–(c) Joint SLO attainment, TTFT, and TPOT under the low-, median-, and high-burstiness traces. (d) Fine-tuning throughput relative to the uncapped default (line), and TTFT SLO attainment (bars), when the fine-tuning microbatch is capped at various sizes, under the high burstiness trace. "Def" refers to FlexLLM’s uncapped default microbatches, with a median of 3308 tokens.
Figure 2: One co-serving iteration of a four-layer model, with a request arriving at the same time in every panel. Fl and Bl are the forward and backward of layer l , and superscripts in (e) denote different microbatches.
Figure 3: Cross-microbatch backward wavefront. Each backward microbatch (i.e., BWD-I to BWD-V) follows reverse layer order (gray dashed arrows), while their next ready tasks (blue) collectively follow the forward traversal.
Figure 4: A row-wise weight partition completes forward locally but distributes backward across CTAs.
Figure 5: LayerLoom system architecture.
Figure 6: Joint SLO attainment and fine-tuning throughput at the Mid (a, b) and High (c, d) loads.
Figure 7: Inference latency. Dotted lines mark the SLO targets.
Figure 8: Saved fine-tuning state at the Mid load in GPU and pinned host memory. OOM marks configurations that run out of GPU memory.
Figure 9: Joint SLO attainment (bars) and fine-tuning throughput (line) versus the wavefront limit M at the Mid load, from single 300-s runs.
Mid load
High load
Policy
SLO (%)
FT (tok/s)
SLO (%)
FT (tok/s)
Greedy (ours)
88.7
511
81.9
478
Lookahead
86.9
463
77.3
437
Split
78.7
485
79.8
466
Table 1: Scheduling policies of LayerLoom with 48 backward microbatches , averaged over five runs.
Figure 10: Shared-weight down_proj kernel. (a) Speedup over two unfused kernels on the same mainloop. (b) down_proj GPU time per iteration inside LayerLoom at the Mid load.
Figure 11: Mean training loss per optimizer update (a) without inference load, where the two curves overlap, and (b) under the Mid load, with two runs per system.
LLM serving systems are provisioned for peak load to meet strict latency targets, leaving substantial GPU compute idle whenever traffic falls below peak. We present DeltaServe, a host-agnostic co-serving design that converts this idle inference capacity into LoRA fine-tuning throughput while preserving inference service-level objectives (SLOs). DeltaServe integrates with existing inference engines through a compact hook interface that requires only multi-LoRA batching support. It exploits the shared execution structure of inference prefill and LoRA fine-tuning forward passes, and uses an SLO-aware scheduler to admit and execute fine-tuning only when sufficient inference headroom is available. The scheduler is driven by a CUDA-graph-aware latency model calibrated offline and refined online. We integrate DeltaServe with vLLM, SGLang, and S-LoRA. On a production trace from Company X, DeltaServe on vLLM delivers 2.9x higher fine-tuning throughput than LLMStation at 100% inference SLO compliance, versus 85% for LLMStation. It also achieves 39% higher fine-tuning throughput than a baseline running vLLM+torchtune, using no additional hardware and maintaining full SLO compliance.
Low-Rank Adaptation (LoRA) has gained popularity as a fine-tuning approach for Large Language Models (LLMs) due to its low resource requirements and good performance. While numerous studies have investigated ways to improve LoRA serving efficiency by serving multiple LoRAs concurrently, existing methods assume that a wide range of LoRA adapters are available for serving. In our work, we conduct extensive empirical studies to show that current LoRA training paradigms do not efficiently utilize hardware resources and incur high overhead to obtain a performant LoRA adapter. Leveraging these insights, we propose PLoRA, which automatically orchestrates concurrent LoRA fine-tuning jobs under given hardware and model constraints and develops performant kernels to improve training efficiency. Across a range of LLMs and LoRA configurations, PLoRA improves training throughput by up to 12.8x and reduces the overall fine-tuning makespan by up to 7.52x compared to existing approaches.
Minghao Yan, Zhuang Wang, Zhen Jia +2
University of Wisconsin-Madison · Amazon Web Services
Low-Rank Adaptation (LoRA) has become the dominant parameter-efficient fine-tuning method for large language models, yet standard practice applies LoRA adapters uniformly to all transformer layers regardless of their relevance to the downstream task. We introduce Aletheia, a gradient-guided layer selection method that identifies the most task-relevant layers via a lightweight gradient probe and applies LoRA adapters only to those layers with asymmetric rank allocation. Across 81 experiment rows covering 14 successful models from 8 architecture families (0.5B-72B parameters, including dense and Mixture-of-Experts architectures), with one additional documented failed Pythia/GPT-NeoX attempt in Campaign 2, Aletheia achieves a 15-28% training speedup (mean 23.1%, p < 0.001) with bounded extra forgetting and broadly matched downstream behavior on the evaluated MMLU, GSM8K, and HumanEval benchmark pack. Across the tested families and scales, Campaign 1 shows a 100% per-model speed win rate and Campaign 2 shows broadly preserved downstream behavior within a bounded-degradation framing. Together these results support a practical model-economics claim: intelligent layer selection can make LoRA fine-tuning materially more efficient without introducing major downstream damage on the evaluated set.
Abdulmalek Saket
Royal Fenice Kft / ALETHEIA PROTOCOL research, Budapest, Hungary