Consumer Level GPUs

Consumer-level GPUs are increasingly being explored for running and training large language models (LLMs), driven by the need to democratize access to this powerful technology. Research focuses on optimizing LLM inference and fine-tuning on these less powerful but more widely available devices, employing techniques like memory-efficient quantization (e.g., 2-bit quantization), optimized inference engines that leverage the sparsity of neuron activations, and efficient communication strategies for distributed training across multiple consumer GPUs. This work is significant because it potentially lowers the barrier to entry for LLM research and deployment, enabling broader participation and fostering innovation beyond resource-rich environments.

Papers

December 9, 2024

SparseAccelerate: Efficient Long-Context Inference for Mid-Range GPUs
James Vo
Long Context Long Context Task Long Context LLM Consumer Level GPUs

November 15, 2024

TEESlice: Protecting Sensitive Neural Network Models in Trusted Execution Environments When Attackers have Pre-Trained Models
Ding Li, Ziqi Zhang, Mengyu Yao, Yifeng Cai, Yao Guo, Xiangqun Chen
Pre Trained Model DNN Model Pseudo Victim AttackER Sensitivity Aware Trusted Execution Environment 3D Graph Test Environment Private Feature Consumer Level GPUs

October 9, 2024

InAttention: Linear Context Scaling for Transformers
Joseph Eisner
Transformer Megatron Decepticons Transformer Model Long Context Decoder Only Transformer GPU Memory Consumer Level GPUs

December 16, 2023

PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU
Yixin Song, Zeyu Mi, Haotong Xie, Haibo Chen
Large Language Model Large Language Model Inference Sparse Activation Inference System Hybrid Inference Consumer Level GPUs

September 28, 2023

ModuLoRA: Finetuning 2-Bit LLMs on Consumer GPUs by Integrating with Modular Quantizers
Junjie Yin, Jiahao Dong, Yingheng Wang, Christopher De Sa, Volodymyr Kuleshov
Large Language Model Low Rank Adaptation Memory Efficient Fine Tuning Consumer Level GPUs

September 3, 2023

FusionAI: Decentralized Training and Deploying LLMs with Massive Consumer-Level GPUs
Zhenheng Tang, Yuxin Wang, Xin He, Longteng Zhang, Xinglin Pan, Qiang Wang, Rongfei Zeng, Kaiyong Zhao, Shaohuai Shi, Bingsheng He, Xiaowen Chu
Single GPU Decentralized Training GPU Memory LLM Deployment Consumer Level GPUs

November 16, 2021

CGX: Adaptive System Support for Communication-Efficient Deep Learning
Ilia Markov, Hamidreza Ramezanikebrya, Dan Alistarh
Communication Efficient Resource Management Multi GPU Adaptive System GPU Training Consumer Level GPUs