Memory Capacity

Memory capacity, the ability of a system to store and retrieve information, is a central research area in both neuroscience and artificial intelligence. Current research focuses on understanding and improving memory in large language models (LLMs) and neural networks, exploring techniques like hierarchical memory management, efficient embedding methods, and dynamic neuron selection to overcome limitations in storage and retrieval. These efforts aim to enhance the performance of AI systems in tasks requiring long-term memory and improve the efficiency of training and inference, with implications for applications ranging from personalized recommendations to visual reasoning. Furthermore, investigations into the relationship between memory capacity and network architecture, particularly in recurrent and deep neural networks, are revealing fundamental insights into the nature of computation and representation in these systems.

Papers

October 28, 2024

NeuZip: Memory-Efficient Training and Inference with Dynamic Compression of Neural Networks
Yongchang Hao, Yanshuai Cao, Lili Mou
Neural Network Scientific Inference Memory Efficient Training Memory Capacity Lossless Performance

October 9, 2024

Retrieval-Augmented Decision Transformer: External Memory for In-context RL
Thomas Schmied, Fabian Paischer, Vihang Patil, Markus Hofmarcher, Razvan Pascanu, Sepp Hochreiter
Reinforcement Learning Context Learning Memory Capacity Retrieval Enhanced Transformer

September 16, 2024

Schrodinger's Memory: Large Language Models
Wei Wang, Qing Li
Memory Trace Memory Mechanism Memory Capacity Memorization Capability

June 5, 2024

FragRel: Exploiting Fragment-level Relations in the External Memory of Large Language Models
Xihang Yue, Linchao Zhu, Yi Yang
Unstructured Text Hierarchical Memory Memory Capacity Aware Fragment

May 22, 2024

Interpolation with deep neural networks with non-polynomial activations: necessary and sufficient numbers of neurons
Liam Madden
Neural Network Polynomial Activation Three Layer Neural Network Memory Capacity Input Output Pair N$ Neuron

February 6, 2024

ProactivePIM: Accelerating Weight-Sharing Embedding Layer with PIM for Scalable Recommendation System
Youngsuk Kim, Junghwan Lim, Hyuk-Jae Lee, Chae Eun Rhee
Low Rate ACCELERATION Hash Code Memory Capacity Near Memory 1T1C DRAM Cell Memory Hierarchy Heterogeneous Memory

December 8, 2023

Towards On-device Learning on the Edge: Ways to Select Neurons to Update under a Budget Constraint
Aël Quélennec, Enzo Tartaglione, Pavlo Mozharovskyi, Van-Tam Nguyen
Extreme Edge Single Neuron Level Efficient Learning New Way Device Learning Budget Constraint Device Use Case Memory Capacity Neuron Selection

September 12, 2023

Neural Network Layer Matrix Decomposition reveals Latent Manifold Encoding and Memory Capacity
Ng Shyh-Chang, A-Li Luo, Bo Qiu
Universal Approximation Hopfield Network Latent Manifold Learning Decomposition Memory Capacity Optimal Encoding

August 3, 2023

Memory capacity of two layer neural networks with smooth activations
Liam Madden, Christos Thrampoulidis
Neural Network Multi Layer Memory Capacity Input Dimension

May 2, 2023

Memory of recurrent networks: Do we compute it right?
Giovanni Ballarin, Lyudmila Grigoryeva, Juan-Pablo Ortega
Recurrent Neural Network Memory Trace Recurrent Network Echo State Network Controllability Condition Memory Capacity

April 30, 2023

Working Memory Capacity of ChatGPT: An Empirical Study
Dongyu Gong, Xingchen Wan, Dingmin Wang
Large Language Model ChatGPT Generated Conversation Empirical Study Working Memory Spatial Prediction Memory Capacity

April 19, 2023

MTrainS: Improving DLRM training efficiency using heterogeneous memories
Hiwot Tadese Kassa, Paul Johnson, Jason Akers, Mrinmoy Ghosh, Andrew Tulloch, Dheevatsa Mudigere, Jongsoo Park, Xing Liu, Ronald Dreslinski, Ehsan K. Ardestani
Content Addressable Memory Deep Learning Recommendation Model Memory Capacity Heterogeneous Memory

April 14, 2023

The role of object-centric representations, guided attention, and external memory on generalizing visual relations
Guillermo Puebla, Jeffrey S. Bowers
Deep Neural Network Human Attention Integral Role Visual Reasoning Object Centric Representation Visual Relation Abstract Visual Reasoning Memory Capacity

January 31, 2023

The Power of External Memory in Increasing Predictive Model Capacity
Cenk Baykal, Dylan J Cutler, Nishanth Dikkala, Nikhil Ghosh, Rina Panigrahy, Xin Wang
Deep Network Real Power Softmax Function Input Representation Model Capacity Memory Capacity Binary Token

May 6, 2022

Dynamically writing coupled memories using a reinforcement learning agent, meeting physical bounds
Théo Jules, Laura Michel, Adèle Douin, Frédéric Lechenault
Memory Trace Reinforcement Learning Agent General Bound Personal Memory Memory Capacity Adiabatic Quantum

March 2, 2022

Continual Learning of Multi-modal Dynamics with External Memory
Abdullah Akgül, Gozde Unal, Melih Kandemir
Continual LEArning Continual Learning Cross Modal Episodic Memory Memory Capacity Neural Memory

December 5, 2021

Boosting Mobile CNN Inference through Semantic Memory
Yun Li, Chen Zhang, Shihao Han, Li Lyna Zhang, Baoqun Yin, Yunxin Liu, Mengwei Xu
Device Deep Semantic Memory Hierarchical Memory Memory Capacity Mobile CNN