ALTO: Adaptive LoRA Tuning and Orchestration for Heterogeneous LoRA Training Workloads

Jingwei Zuo; Xinze Feng; Zien Liu; Kaijian Wang; Fanjiang Ye; Ye Cao; Zhuang Wang; Yuke Wang

cs.LGApr 7, 2026

ALTO: Adaptive LoRA Tuning and Orchestration for Heterogeneous LoRA Training Workloads

Authors: Jingwei Zuo, Xinze Feng, Zien Liu, Kaijian Wang, Fanjiang Ye, Ye Cao, Zhuang Wang, Yuke Wang

Organizations: Rice University · Independent Researcher

Abstract

Low-Rank Adaptation (LoRA) is now the dominant method for parameter-efficient fine-tuning of large language models, but achieving a high-quality adapter often requires systematic hyperparameter tuning because LoRA performance is highly sensitive to configuration choices. In practice, this leads to many concurrent LoRA jobs, often spanning heterogeneous tasks in multi-tenant environments. Existing systems largely handle these jobs independently, which both wastes computation on weak candidates and leaves GPUs underutilized. We present ALTO (Adaptive LoRA Tuning and Orchestration), a co-designed training system that accelerates LoRA hyperparameter tuning while enabling efficient cluster sharing across heterogeneous tasks. The central insight behind ALTO is that when multiple tuning jobs run concurrently over a shared frozen backbone, they expose optimization opportunities that single-job designs cannot exploit. Building on this, ALTO monitors loss trajectories to terminate unpromising configurations early, uses fused grouped GEMM together with a new rank-local adapter parallelism to co-locate surviving adapters and reclaim freed GPU capacity, and combines intra-task and inter-task scheduling to improve multi-task placement by leveraging the predictable duration of LoRA jobs. Extensive evaluation shows that ALTO achieves up to $13.8\times$ speedup over state-of-the-art without sacrificing adapter quality.

Figures & tables

Figure 1: Different hyperparameters yield significantly varying downstream task performance in LoRA training. (a) Best validation loss distributions across 165 hyperparameter configs (varying batch size, learning rate, and LoRA rank) for Llama-3.1-8B. (b) GSM8K test accuracy of the best checkpoint per config shows a maximum of 73.9% difference. (c) DPO Reward accuracy of 60 different configurations on UltraFeedback dataset shows a maximum of 26.7% difference.

Figure 3: LoRA fine-tuning prefer small batch size training. SFT validation loss for Llama-3.1-8B on (a) GSM8K, (b) Tulu-3, and (c) OpenThoughts3; DPO reward accuracy for Qwen2.5-32B on (d) UltraFeedback across five learning rates. Performance peaks at small batch sizes (

\leq

16) and degrades beyond 32 for these settings.

Figure 4: GPU memory and average SM utilization when training a single LoRA adapter with small batch sizes. A substantial portion of GPU resources remains idle, motivating the need for batched multi-adapter training.

Figure 5: (a) Shortest-Job-First serve fails to fully utilize the GPU resources when multiple training tasks come. (b) Makespan-aware inter-task scheduling can fully minimize the overall training makespan.

Figure 6: The illustration of the loss curves of three typical redundant training patterns: (a) Overfitting, (b) Diverging, and (c) Underperforming.

Figure 7: Rank correlation between validation loss at the end of the warmup phase and at the end of training. The relative ordering of well-behaved configurations is largely preserved, and the eventually best configuration is always preserved in top 25% after the warmup, validating the effectiveness of warmup-based exiting.

Figure 8: Two executor modes in ALTO ’s batched execution engine. (a) Multi-GPU executor with adapter parallelism: base model weights are sharded across ranks and synchronized via all-gather, while each rank trains a disjoint set of LoRA jobs locally via grouped GEMM, therefore there is no adapter gradients cross rank boundaries. (b) Single-GPU executor : the full base model and multiple LoRA jobs reside on a single device, trained concurrently through grouped GEMM batching.

Variable	Domain	Description
$s_{i}$	$\mathbb{R}_{\geq 0}$	Start time of task $i$
$x_{ig}$	$\{0,1\}$	1 iff task $i$ occupies GPU $g$
$y_{ij}$	$\{0,1\}$	1 iff task $i$ precedes $j$ ( $i{<}j$ )
$C_{\max}$	$\mathbb{R}_{\geq 0}$	Makespan

Table 1: Constraint programming decision variables for inter-task scheduling.

Figure 9: End-to-end training speedup of ALTO across single-GPU and multi-GPU configurations. From left to right: Llama-3.1-8B and Qwen2.5-7B on a single H100 GPU, Qwen2.5-32B on 2

\times

H100 and Llama-3.1-70B on 4

\times

H100. Each configuration trains 60 (single-GPU) or 64 (multi-GPU) heterogeneous LoRA adapters with varied ranks, batch sizes, and learning rates across three datasets. ALTO achieves up to 9.5

\times

speedup on single-GPU and up to 13.8

\times

speedup on multi-GPU settings, consistently outperforming all baselines. Speedup is evaluated relative to LoRAFusion.

Figure 10: Model quality of the best configuration found by ALTO on single-GPU 7B–8B models, compared against expert-recommended hyperparameters from Unsloth and Tinker. (a) GSM8K test accuracy (higher is better). Completion loss on the evaluation split of (b) Tulu-3 and (c) OpenThoughts3 (lower is better). For ALTO , each bar shows the best result found across batched runs with per-adapter batch sizes in

\{1,2,4,8\}

Figure 11: DPO training on UltraFeedback. Bars show end-to-end speedup over sequential training; diamond markers show best preference accuracy across all per-adapter batch sizes. ALTO with early exit achieves

4.7\times

speedup while preserving the same preference accuracy (76.2%) as Batched-LoRA without early exit.

Figure 12: Evalution of ALTO components on 8-GPU training makespan. B = Batched LoRA, S = Scheduler, EE = Early Exit. The full system (B+S+EE) achieves a

5.2\times

reduction in makespan compared to batching alone (B), with early exit contributing the largest individual gain.

Figure 13: Adapter Parallelism (AP) microbenchmark on 4

\times

H100 GPUs: speedup relative to FSDP across per-adapter batch sizes, with 8 adapters and sequence length 256. AP (Ours) outperforms FSDP, TP, mLoRA, and LoRAFusion at all batch sizes, peaking at

4.7\times

at per-adapter batch size 2.

Per-adapter	PyTorch	Sequential	Fused	Speedup over
BS	(s)	(s)	(s)	PyTorch	Sequential
1	754.1	2016.6	394.6	$1.91\times$	$5.1\times$
2	514.4	1098.4	296.0	$1.74\times$	$3.7\times$
4	328.5	599.2	240.8	$1.36\times$	$2.5\times$

Table 2: Kernel microbenchmark on Llama-3.2-1B, GSM8K, 32 adapters with ranks 16, 32, 64 equally mixed. PyTorch : per-adapter LoRA run sequentially, backbone model passed in a batch. Sequential : each adapter trained alone. Fused : ALTO ’s grouped Triton kernel.

Figure 14: Single-GPU ablation on Llama-3.1-8B / GSM8K across per-adapter batch sizes. Left: GSM8K evaluation accuracy (strict parsing). Gray dots show individual adapter accuracies from the full sweep without selection. Colored markers compare the best accuracy found by Multi-LoRA batching alone vs. Multi-LoRA with early exit. Right: Best validation loss. Early exit preserves or improves quality by concentrating resources on promising configurations.

Figure 15: Training samples saved by each early-exit pattern across seven model–dataset combinations. Stacked bars decompose savings by detector: underperformance filtering dominates in SFT (

{\sim}66\%

), while overfitting and divergence contribute more in DPO. Green diamonds show the ratio of best validation loss w/ vs. w/o early exit; values at or near 1.0 confirm that early termination preserves model quality.

Figure 16: Sensitivity of early exit predictions to warmup percentage. Left: Spearman rank correlation between loss at warmup and final validation performance. Middle: Fraction of true top-25% configurations captured by early predictions. Right: Whether the best configuration is included in the predicted top-25%. The dashed line marks 5%, our default warmup threshold, where all metrics stabilize across models and datasets.

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 753 papers

Aug 4, 2025

PLoRA: Efficient Concurrent LoRA Training for Large Language Models

Low-Rank Adaptation (LoRA) has gained popularity as a fine-tuning approach for Large Language Models (LLMs) due to its low resource requirements and good performance. While numerous studies have investigated ways to improve LoRA serving efficiency by serving multiple LoRAs concurrently, existing methods assume that a wide range of LoRA adapters are available for serving. In our work, we conduct extensive empirical studies to show that current LoRA training paradigms do not efficiently utilize hardware resources and incur high overhead to obtain a performant LoRA adapter. Leveraging these insights, we propose PLoRA, which automatically orchestrates concurrent LoRA fine-tuning jobs under given hardware and model constraints and develops performant kernels to improve training efficiency. Across a range of LLMs and LoRA configurations, PLoRA improves training throughput by up to 12.8x and reduces the overall fine-tuning makespan by up to 7.52x compared to existing approaches.

Minghao Yan, Zhuang Wang, Zhen Jia +2

University of Wisconsin-Madison · Amazon Web Services

LLM Fine-Tuning Efficient Language Model Training Low-Rank Adaptation Parameter-Efficient Fine-Tuning

Ask PDF

Apr 4, 2026

Aletheia: Gradient-Guided Layer Selection for Efficient LoRA Fine-Tuning Across Architectures

Low-Rank Adaptation (LoRA) has become the dominant parameter-efficient fine-tuning method for large language models, yet standard practice applies LoRA adapters uniformly to all transformer layers regardless of their relevance to the downstream task. We introduce Aletheia, a gradient-guided layer selection method that identifies the most task-relevant layers via a lightweight gradient probe and applies LoRA adapters only to those layers with asymmetric rank allocation. Across 81 experiment rows covering 14 successful models from 8 architecture families (0.5B-72B parameters, including dense and Mixture-of-Experts architectures), with one additional documented failed Pythia/GPT-NeoX attempt in Campaign 2, Aletheia achieves a 15-28% training speedup (mean 23.1%, p < 0.001) with bounded extra forgetting and broadly matched downstream behavior on the evaluated MMLU, GSM8K, and HumanEval benchmark pack. Across the tested families and scales, Campaign 1 shows a 100% per-model speed win rate and Campaign 2 shows broadly preserved downstream behavior within a bounded-degradation framing. Together these results support a practical model-economics claim: intelligent layer selection can make LoRA fine-tuning materially more efficient without introducing major downstream damage on the evaluated set.

Abdulmalek Saket

Royal Fenice Kft / ALETHEIA PROTOCOL research, Budapest, Hungary

LLM Fine-Tuning Efficient Language Model Training Parameter-Efficient Fine-Tuning Low-Rank Adaptation

Ask PDF

Apr 20, 2026

TLoRA: Task-aware Low Rank Adaptation of Large Language Models

Low-Rank Adaptation (LoRA) has become a widely adopted parameter-efficient fine-tuning method for large language models, with its effectiveness largely influenced by the allocation of ranks and scaling factors, as well as initialization. Existing LoRA variants typically address only one of these factors, often at the cost of increased training complexity or reduced practical efficiency. In this work, we present Task-aware Low-Rank Adaptation (TLoRA), a unified framework that jointly optimizes initialization and resource allocation at the outset of training. TLoRA introduces a data-driven initialization strategy that aligns the LoRA

A

matrix with task-relevant subspaces by performing singular value decomposition on the product of pre-trained weights and input activation covariance. After this, the

A

matrix is frozen, and only the

B

matrix is trained. Furthermore, TLoRA employs a sensitivity-based importance metric to adaptively allocate ranks and scaling factors across layers under a fixed parameter budget. We conduct extensive experiments that demonstrate TLoRA consistently performs excellently across various tasks, including natural language understanding, commonsense reasoning, math reasoning, code generation, and chat generation, while significantly reducing the number of trainable parameters.

Weicheng Lin, Yi Zhang, Jiawei Dang +1

College of Computer Science and Software Engineering, Shenzhen University, China

LLM Fine-Tuning Low-Rank Adaptation Adaptive Rank Allocation Parameter-Efficient Fine-Tuning

Ask PDF

Abstract

LLM Fine-Tuning

Explore similar work

PLoRA: Efficient Concurrent LoRA Training for Large Language Models

Aletheia: Gradient-Guided Layer Selection for Efficient LoRA Fine-Tuning Across Architectures

TLoRA: Task-aware Low Rank Adaptation of Large Language Models