Quantization-Aware LoRA

LoRA: Low-Rank Adaptation

Follow this topic Ask about this topic

Momentum

2 papers in the last four weeks, with none the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 10

All topics

Analysis of Quantized and Efficiently Adapted Protein Language Models
Sep 30, 2026Ilan Yaniv Zeisler, Sebastian Clancy, Pouriya Bayat +8Protein Language Models Quantization-Aware LoRA
Leslie Dan Faculty of Pharmacy, University of Toronto, Toronto, ON, Canada. · Department of Electrical and Computer Engineering, University of Toronto, Toronto, Ontario, Canada. · Department of Computer Science, University of Toronto, Toronto, Ontario, Canada. +2
Optimizing the Phi-2 Small Language Model for Real-time Chatbot Applications Using Parameter-Efficient Fine-Tuning (PEFT) with QLoRA Quantization
Sep 27, 2026PhanTan Khanh Nguyen, Ashfaq Ali Shafin, Khandaker Mamun AhmedOn-Device Language Model Inference Small Language Models
Beacom College of Computer and Cyber Sciences, Dakota State University · Mathematics and Computer Science, Augustana College
From Financial Sentiment Classification to Return Predictability: A QLoRA Benchmark of Large Language Models
Aug 4, 2026Fusheng LuoLLM Evaluation Quantitative Finance
Johns Hopkins University Baltimore, Maryland, USA
The Parts Are Greater Than the Sum: Automated Task Sequencing for Efficient Training of Multi-Policy LLMs
Jul 31, 2026Jiajia Tang, Sizhe Yuen, Francisco Gomez Medina +2Continual Learning for LLMs Low-Rank Adaptation
The Alan Turing Institute, London, UK · King’s College London, London, UK · University of Southampton, Southampton, UK
Encoding Invisible Causation for Bridge Diagnostic Agents: Triple-Guided Retrieval-Augmented Fine-Tuning with QLoRA
Jul 23, 2026Takato YasunoCausal Attribution Low-Rank Adaptation
TerraMARS: A Domain-Adapted Small-Language-Model Pipeline for Mars Terraforming Literature
Jun 18, 2026Jyotsna Singh, Ash Black, Jeff Larsen +1Small Language Models Quantization-Aware LoRA
College of Information Science, University of Arizona, Tucson, AZ, USA · Biosphere 2, University of Arizona, Tucson, AZ, USA · Department of Ecology and Evolutionary Biology, University of Arizona, Tucson, AZ, USA +1
Small LLMs for Biomedical Claim Verification: Cost-Effective Fine-Tuning, Structural Dataset Shortcuts, and Cross-Domain Generalization
Jun 11, 2026Gaurav KumarClaim Verification LLM Fine-Tuning
Moveworks AI · University of California San Diego
LinguIUTics at PsyDefDetect: Iterative Imbalance-Aware Fine-tuning of Qwen3-8B for Psychological Defense Mechanism Classification
May 30, 2026Shefayat E Shams Adib, Ahmed Alfey Sani, Md Hasibur Rahman Alif +1Class-Imbalanced Learning Quantization-Aware LoRA
Department of Computer Science and Engineering, Islamic University of Technology, Dhaka, Bangladesh
Signs Beat Floats: Low-Rank Double-Binary Adaptation for On-Device Fine-Tuning
May 22, 2026Yoshihiko Fujisawa, Yuma Ichikawa, Yudai Fujimoto +2LLM Quantization Memory-Efficient Fine-Tuning
Fujitsu Limited · Institute of Science Tokyo · RIKEN Center for AIP +1
On-the-Fly Adaptation to Quantization: Configuration-Aware LoRA for Efficient Fine-Tuning of Quantized LLMs
Sep 22, 2025Rongguang Ye, Ming Tang, Edith C. H. NgaiMixed-Precision Quantization LLM Quantization
Southern University of Science and Technology · The University of Hong Kong

Sep 30, 2026, cs.LGJ/K move · Enter open · S save

Analysis of Quantized and Efficiently Adapted Protein Language Models

Ilan Yaniv Zeisler, Sebastian Clancy, Pouriya Bayat, Saaim Raad+7

Leslie Dan Faculty of Pharmacy, University of Toronto, Toronto, ON, Canada. · Department of Electrical and Computer Engineering, University of Toronto, Toronto, Ontario, Canada. · Department of Computer Science, University of Toronto, Toronto, Ontario, Canada.+2

Protein Language Models Quantization-Aware LoRA Molecular Generation Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Background: Protein language models (PLMs) are increasingly used for sequence generation and property prediction, but their size makes fine-tuning and deployment expensive. The effects of quantization and parameter efficient fine-tuning on performance, representations and generation remain insufficiently characterized. Results: We evaluated 4-bit quantization and low-rank adapter fine-tuning (QLoRA) across ESM-2, ESMC, ProtBERT, ProtT5, Ankh, Ankh3 and Profluent-E1. Across protein prediction tasks, many model-task pairs retained more than 90% of full fine-tuning performance. Peak GPU memory savings approached 90% for the largest models, although performance and efficiency varied by model, dataset and training configuration. QLoRA often preserved early-layer representations while inducing task-specific adaptations in middle and late layers, resembling full fine-tuning with smaller representational changes. Training speed and power effects were more varied. For unconditional generation with ProLLaMA, ProtGPT2, ProGen2, ProteinGLM and ESM3, 4-bit quantization largely preserved predicted structural and sequence-level properties, but token-level analysis revealed model-dependent shifts in autoregressive output distributions. Conclusion: QLoRA and 4-bit quantization reduce PLM computational requirements, particularly GPU memory usage. Our results support QLoRA as a first-pass strategy for memory limited adaptation, reserving full fine-tuning for challenging tasks, unstable architectures or low validation recovery. For generative PLMs, sequence-level and structural metrics should be complemented with distributional analysis, since downstream predictions alone may miss quantization-induced shifts. These approaches can broaden access to large-scale protein modelling while requiring model- and task-specific validation.

Momentum

Related topics

Latest papers 10