Memory Reduction

Memory reduction in neural network training and inference is a critical research area aiming to enable the development and deployment of larger, more complex models on resource-constrained devices. Current efforts focus on optimizing model architectures (e.g., transformers, convolutional neural networks) through techniques like sparse training, low-rank approximations, and efficient operator ordering, as well as employing quantization strategies for both weights and activations. These advancements are crucial for expanding the accessibility and scalability of deep learning, impacting fields ranging from natural language processing and computer vision to federated learning and edge computing.

Papers

October 21, 2024

Understanding and Alleviating Memory Consumption in RLHF for LLMs
Jin Zhou, Hanmei Yang, Steven (Jiaxun) Tang, Mingcan Xiang, Hui Guan, Tongping Liu
Large Language Model Reinforcement Learning Fine Tuning Human Understanding Reinforcement Learning From Human Feedback Memory Management Memory Reduction

July 21, 2024

Token-Picker: Accelerating Attention in Text Generation with Minimized Memory Transfer via Probability Estimation
Junyoung Park, Myeonggu Kang, Yunki Han, Yanggon Kim, Jaekang Shin, Lee-Sup Kim
Text Generation Human Attention Probability Estimation Chip Memory Memory Reduction Token Selection

May 24, 2024

Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks
Jialin Zhao, Yingtao Zhang, Xinghang Li, Huaping Liu, Carlo Vittorio Cannistraci
Scientific Inference Low Rank Spectral Learning Hyperbolic Neural Network Memory Efficient Training Memory Reduction

May 8, 2024

When Foresight Pruning Meets Zeroth-Order Optimization: Efficient Federated Learning for Low-Memory Devices
Pengyu Zhang, Yingjie Liu, Yingbo Zhou, Xiao Du, Xian Wei, Ting Wang, Mingsong Chen
Federated Learning Zeroth Order Resource Constrained Device Backpropagation Free Progressive Pruning Artificial Intelligence of Thing Memory Reduction Space Pruning

March 14, 2024

Achieving Pareto Optimality using Efficient Parameter Reduction for DNNs in Resource-Constrained Edge Environment
Atah Nuh Mih, Alireza Rahimi, Asfia Kawnine, Francis Palma, Monica Wachowicz, Rickey Dubay, Hung Cao
Deep Neural Network DNN Framework Pareto Optimal Device Training Resource Constrained Edge Memory Reduction Parameter Reduction

March 6, 2024

GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, Yuandong Tian
Natural Gradient Low Rank Adaptation Projection Bias Memory Efficient Memory Reduction

January 21, 2024

LR-CNN: Lightweight Row-centric Convolutional Neural Network Training for Memory Reduction
Zhigang Wang, Hangyu Yang, Ning Wang, Chuanfei Xu, Jie Nie, Zhiqiang Wei, Yu Gu, Ge Yu
Convolutional Neural Network Multi Layer Convolution Layer Dataflow Architecture Memory Reduction

October 30, 2023

ROAM: memory-efficient large DNN training via optimized operator ordering and memory layout
Huiyao Shu, Ang Wang, Ziji Shi, Hanyu Zhao, Yong Li, Lu Lu
Memory Efficient Training Memory Efficiency Class Enhanced Attentive Response Memory Reduction Memory Architecture Tensor Layout

June 7, 2023

Optimizing ViViT Training: Time and Memory Reduction for Action Recognition
Shreyank N Gowda, Anurag Arnab, Jonathan Huang
Action Recognition Transformer Based Model Time Matter Temporal Transformer Spatial Transformer Memory Reduction Video Vision Transformer

April 3, 2023

RPTQ: Reorder-based Post-training Quantization for Large Language Models
Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang, Yuzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, Bingzhe Wu
Post Training Quantization Large Scale Language Model Multiplier Free Quantization Product Quantization Memory Reduction

February 28, 2023

GradMA: A Gradient-Memory-based Accelerated Federated Learning with Alleviated Catastrophic Forgetting
Kangyang Luo, Xiang Li, Yunshi Lan, Ming Gao
Continual LEArning Federated Learning Catastrophic Forgetting Memory Reduction

November 29, 2022

Neural Transducer Training: Reduced Memory Consumption with Sample-wise Computation
Stefan Braun, Erik McDermott, Roger Hsiao
Automatic Speech Recognition Neural Transducer Memory Efficient Training Memory Reduction Sparse Fourier

October 19, 2022

Tempo: Accelerating Transformer-Based Model Training through Memory Footprint Reduction
Muralidhar Andoorveedu, Zhanda Zhu, Bojian Zheng, Gennady Pekhimenko
Deep Learning Model Transformer Based Model Efficient Training Transformer Training Training Throughput Tempo Estimation Memory Reduction

February 1, 2022

Few-Bit Backward: Quantized Gradients of Activation Functions for Memory Footprint Reduction
Georgii Novikov, Daniel Bershatsky, Julia Gusak, Alex Shonenkov, Denis Dimitrov, Ivan Oseledets
Back Propagation Memory Reduction Neural Network Training Bit Flip

Memory Reduction

Papers

Understanding and Alleviating Memory Consumption in RLHF for LLMs

Token-Picker: Accelerating Attention in Text Generation with Minimized Memory Transfer via Probability Estimation

Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks

When Foresight Pruning Meets Zeroth-Order Optimization: Efficient Federated Learning for Low-Memory Devices

Achieving Pareto Optimality using Efficient Parameter Reduction for DNNs in Resource-Constrained Edge Environment

GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

LR-CNN: Lightweight Row-centric Convolutional Neural Network Training for Memory Reduction

ROAM: memory-efficient large DNN training via optimized operator ordering and memory layout

Optimizing ViViT Training: Time and Memory Reduction for Action Recognition

RPTQ: Reorder-based Post-training Quantization for Large Language Models

GradMA: A Gradient-Memory-based Accelerated Federated Learning with Alleviated Catastrophic Forgetting

Neural Transducer Training: Reduced Memory Consumption with Sample-wise Computation

Tempo: Accelerating Transformer-Based Model Training through Memory Footprint Reduction

Few-Bit Backward: Quantized Gradients of Activation Functions for Memory Footprint Reduction