Bi-Phase Training: Learning Efficiently From Scratch

Abel Gurung; Joseph Campbell

cs.LGOct 27, 2026

Bi-Phase Training: Learning Efficiently From Scratch

Authors: Abel Gurung, Joseph Campbell

Organizations: Purdue University

Abstract

Pre-training foundational models is computationally expensive as models scale. In this work, we introduce Bi-Phase Training (BPT), a parameter-efficient pre-training method that preserves the high-rank learning dynamics observed in full-parameter optimization with substantially reduced trainable parameters. BPT parameterizes each weight update as a composition of constrained high-rank left/right transformations via trainable diagonal matrices and low-rank adapters whose updates are periodically merged into the frozen base weights, resulting in a compounding expansion of accessible directions during training. Across language-model pre-training from scratch (Qwen2.5 models from 100M to 1.5B parameters and OLMo-2-1B) on FineWeb-Edu (10B and 100B tokens), BPT achieves validation perplexity close to full-parameter baselines while using far fewer trainable parameters; e.g., at the 1.5B scale, BPT uses 66% fewer trainable parameters with only a small validation-perplexity gap. Additionally, a comprehensive evaluation across diverse downstream tasks shows that BPT preserves much of the full-parameterized model's downstream performance.

Figures & tables

Figure 1: Evaluation log-perplexity vs. log trainable parameters (non-embedding) across model sizes. BPT approaches the fully-parameterized baseline while using far fewer trainable parameters; on the 1.5B model it achieves comparable eval loss with 66% fewer trainable parameters

Figure 2: Update-rank trend across batch sizes and methods. Left: rank change after the first update step versus batch size, showing scaling effects for SGD and consistently high-rank updates for Adam-full and BPT. Middle: rank at the end of training (

t=T

) versus batch size, where full-parameter methods and BPT reach near-full rank even from small batches, while ReLoRA remains rank-limited. Right: rank evolution over normalized training steps for batch size 16, comparing Adam-full, SGD-full, BPT, and ReLoRA.

Figure 3: Overview of parameter-efficient training techniques. Blue represents non-trainable parameters, and green represents trainable parameters. $\mathrm{UV}$ is the product of two low-rank matrices of rank- $r$ , and $\mathrm{A}$ and $\mathrm{B}$ are diagonal matrices.

Figure 4: Eval loss over training steps. Left: OLMo-2-1B. Right: Qwen-2.5-1.5B. BPT tracks the Full baseline closely and outperforms ReLoRA over time, despite using substantially fewer trainable parameters.

Model	Method	Trainable	Wiki	LAMB	LAMB	Hella	WinoG	PIQA	BoolQ	SciQ	OBQA	ARC-E	ARC-C	Avg	SWDE
		Params	ppl $\downarrow$	ppl $\downarrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc $\uparrow$		acc $\uparrow$
10 Billion Tokens
	Full	1.5B	2.0	47.9	30.9	33.1	51.3	66.9	59.9	81.1	23.6	59.2	23.0	47.7	17.7
Qwen-2.5-1.5B	ReLoRA	0.5B	5.0	42.8	32.2	33.1	50.2	66.5	60.6	82.6	23.2	58.8	24.5	48.0	15.0
	BPT	0.5B	4.8	34.6	34.6	34.7	52.1	66.8	57.2	81.2	23.0	61.9	27.3	48.8	15.3
	Full	1.5B	1.9	48.2	30.6	32.8	51.1	66.6	57.9	82.8	23.2	58.9	23.5	47.5	14.4

Table 1: Performance across Language Modeling, Commonsense, QA, and Extraction Tasks: lower is better for perplexity; higher is better for accuracy.

Figure 5: Qwen-2.5-1.5B: Validation Loss vs. Training Steps (Full vs. BPT) for 100B tokens

Figure 6: BPT ablations and memory summary. Left/Middle: rank and merge-frequency ablations on the 100M model trained on 10B tokens. Higher rank for the low-rank matrices leads to lower validation loss. Less frequent merges also lead to lower validation loss. Right: Peak GPU memory for Qwen-2.5-1.5B. BPT saves 5.8 GB to train a 1.5B model.

Method	Trainable	Eval
	Param	Loss $\downarrow$
HyperAdapt	137M	5.486
VeRA	137M	4.640
LoRA	206M	3.051
DoRA	206M	3.102
ReLoRA	206M	3.038

Table 2: Comparison of PEFT Methods: Trainable Parameter and Eval Loss. Models are pre-trained from scratch on 10 billion tokens.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Models	Layers	Heads (Q / KV)	Hidden Size	Intermediate Size	Training Tokens
100M	8	8/ 2	512	2048	10B
0.5B	24	14 / 2	896	4864	10B
1.5B	36	16 / 2	2048	11008	10B and 100B

Appendix

Table 3: Model Architecture Detail and Total Training Tokens

Model	Full	BPT	Total	Non-embedding
	Trainable	Trainable	Reduction (%)	Reduction (%)
Small	108M	89M	17.8	63.5
Medium	0.5B	0.2B	58.2	80.4
Large	1.5B	0.5B	65.9	77.4

Appendix

Table 4: Trainable parameter counts by method and model size

Method	Models	100M	0.5B	1.5B
	Optimizer	AdamW
	Warmup Steps	2000
	Decay Steps	10000,8000,8000
	Max Grad Norm	1.0
	Max Seq. Len	512
	Batch Size	320,512,1024

Appendix

Table 5: The hyperparameters used for pre-training Qwen-2.5 Models.

Method	Models		1B
	Optimizer	AdamW
	Warmup Steps	2000
	Decay Steps	8000
	Max Grad Norm	1.0
	Max Seq. Len	512
	Batch Size	960

Appendix

Table 6: The hyperparameters used for pre-training OLMo-2-1B.

Method	Models	1.5B
	Optimizer	AdamW
	Warmup Steps	2000
	Decay Steps	24,000
	Max Grad Norm	1.0
	Max Seq. Len	512
	Batch Size	1024

Appendix

Table 7: The hyperparameters used for pre-training Qwen-2.5 1.5B on 100 billion tokens.

Efficient Language Model Training

Efficient language model training studies how to reduce the data, compute, memory, time, energy, and cost needed to build capable language models.

See all 165 papers

Hierarchical Low-Rank Reparameterization for Parameter-Efficient Fine-Tuning
Oct 27, 2026Rong Huang, Yi Xiong, Yuan-Yuan Cheng +1LLM Fine-Tuning Parameter-Efficient Fine-Tuning
University of Science and Technology of China, Hefei, Anhui 230026, China
HRM-Text: Efficient Pretraining Beyond Scaling
May 20, 2026Guan Wang, Changling Liu, Chenyu Wang +6Language Model Pretraining Recurrent Neural Networks
1Sapient Intelligence · 2MIT
DLR: Zero-Inference-Cost Latent Residuals for Low-Rank Pre-Training
Jun 27, 2026Dong Wang, Wenwu Tang, Yun Cheng +1Language Model Pretraining LLM Compression
Graz University of Technology, Austria · Swiss Data Science Center, Switzerland

Oct 27, 2026J/K move · Enter open · S save

Hierarchical Low-Rank Reparameterization for Parameter-Efficient Fine-Tuning

Rong Huang, Yi Xiong, Yuan-Yuan Cheng, Xiao-Ming Fu

University of Science and Technology of China, Hefei, Anhui 230026, China

LLM Fine-Tuning Parameter-Efficient Fine-Tuning Low-Rank Adaptation Low-Rank Compression

Open paper AskPDF arXiv

Parameter-efficient fine-tuning (PEFT) methods adapt large models by introducing a small number of trainable parameters. However, we find that these parameters still contain redundancy and can be further compressed while maintaining strong task performance. To this end, we propose a novel hierarchical low-rank reparameterization method. Its core idea is to reorganize the trainable parameters of PEFT methods into an approximately square matrix and then express it via a low-rank decomposition, which can be applied recursively. This forms a multi-level structure with factors compressed level by level, drastically reducing the number of trainable parameters. In our experiments, we combine this reparameterization with LoRA, DoRA, MoSA, and RaSA. We evaluate commonsense reasoning on LLaMA-3.1-8B and Qwen2.5-7B, with additional GSM8K math reasoning experiments on LLaMA-3.1-8B. Under comparable trainable-parameter budgets, our method combined with LoRA or DoRA achieves better performance. Furthermore,when the parameter count of the original methods is compressed by approximately an order of magnitude, our method can still maintain competitive task performance and even surpass the corresponding baselines in average accuracy across the eight commonsense reasoning benchmarks.

Abstract

Efficient Language Model Training

Explore similar work

Hierarchical Low-Rank Reparameterization for Parameter-Efficient Fine-Tuning

HRM-Text: Efficient Pretraining Beyond Scaling

DLR: Zero-Inference-Cost Latent Residuals for Low-Rank Pre-Training