Memory-Efficient Optimization

Follow this topic Ask about this topic

Memory-efficient optimization studies how to fit model training and inference within limited GPU or device memory without giving up useful performance. For training, approaches include compressed or selectively allocated optimizer states, zeroth-order and low-dimensional updates, and fusing gradient computation with updates to avoid storing intermediates. For inference, methods compress, evict, or offload attention KV caches and coordinate memory across GPU, CPU, and storage; they are commonly evaluated on long-context and reasoning benchmarks as well as throughput and memory use. The main tension is that aggressive memory savings can hurt accuracy or add runtime complexity, so allocation often needs to adapt to the model, workload, and hardware.

Latest papers

67 papers1 paper in the last four weeks, down 75% on the four weeks before. 0.0% of all new papers.

AF-Muon: An AdamW-Free Muon Optimizer for Tied-Embedding Models
Oct 1, 2026Arash Lagzian, Paniz Halvachi, Junming Zhang +2Memory-Efficient Optimization Muon Optimizer
National University of Singapore, Singapore · Independent Researcher · LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China
Cache-Aware Joint Router Adaptation for Memory-Efficient MoE Inference
Sep 4, 2026Zhenhe Wu, Yaping Jin, Qinghua Xing +6Mixture of Experts Memory-Efficient Optimization
Huawei Technologies · Beihang University · Tianjin University +2
GrowPage: On-Demand KV Budgeting for Efficient LLM Reasoning Serving
Sep 3, 2026Qiankun Ma, Yanjiang Zhou, Zinan Xiong +5LLM Inference KV Caching
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Pengcheng Laboratory · University of Chinese Academy of Sciences
Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning
Sep 3, 2026Heng Wang, Jielin Qiu, Wenting Zhao +8KV Caching Memory-Efficient Optimization
Salesforce AI Research · University of Illinois Urbana-Champaign
LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference
Sep 2, 2026Renyuan Liu, Yuyang Leng, Kaiyan Liu +7LLM Inference Efficiency On-Device Language Model Inference
Richard · Global Technology Applied Research, JPMorganChase · University of Kansas +1
Blog: Survey of Optimizers
Aug 28, 2026Ruoran XuNeural Network Optimization Memory-Efficient Optimization
Beyond Dense Adam States: Adaptive Log-Space Quantization for Memory-Efficient Optimizers
Aug 23, 2026Yan WangDeep Learning Optimization Memory-Efficient Optimization
Independent Researcher
SubZero+: Memory-Efficient Adaptive Zeroth-Order LLM Fine-Tuning in Random Subspaces
Aug 16, 2026Ziming Yu, Shuyao Xiao, Xingyu Zhao +6Memory-Efficient Fine-Tuning Zeroth-Order Optimization
Beijing Normal University · Singapore Management University · Xiaomi Inc. +2
Batch Size or Negatives? A Selection Rule for Memory-Constrained Recommender Training
Aug 11, 2026Artyom Sabitov, Daniil Volkov, Alexey ZaytsevMemory-Efficient Optimization Recommender Systems
Applied AI Institute, Moscow, Russia Risk department
Memory-Efficient Activation Checkpointing with Sliding Window and Hirschberg's Algorithm for 0/1 Knapsack Solving in PyTorch
Aug 9, 2026Jędrzej MaczanDynamic Programming Memory-Efficient Optimization
Cohere Labs Community Poland
ZeroLock: Concurrent Memory-Efficient LLM Training via Modular Update Decoupling
Aug 8, 2026Wentao Dai, Xuanran Li, Yuxiang Zhang +2On-Device Training Memory-Efficient Fine-Tuning
Department of Computer Science and Engineering, Southern University of Science and Technology, China · Department of Mathematics, Southern University of Science and Technology, China · School of Computing, Montclair State University, New Jersey, USA
MESH: Memory-Efficient Sinkhorn Optimization for Mixture-of-Experts Training
Aug 5, 2026Masato FujitakeNeural Network Optimization Memory-Efficient Optimization
FA Research, Fast Accounting Co., Ltd.
TaskPress: Query-Agnostic KV Cache Compression via Task-Guided Pruning
Aug 4, 2026Wonpyo Park, Seung-won HwangLLM Pruning KV Caching
Seoul National University · Google
AnchorKV: Anchor-Residual KV Cache Compression
Aug 3, 2026Malik Khalaf, Yara Shamshoum, Nitzan Hodos +2KV Caching Long-Context Language Model Inference
Department of Computer Science, Technion - Israel Institute of Technology
RestoreKV: Recovering Full-Cache Behavior Under Aggressive Query-Agnostic KV Cache Eviction
Aug 2, 2026Changwoo Baek, Seungjun Shin, Kyeongbo KongKV Caching Memory-Efficient Optimization
Pusan National University · Sookmyung Women’s University
HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decoding
Jul 24, 2026Chao Fang, Jun Yin, Man Shi +1KV Caching Memory-Efficient Optimization
ESAT-MICAS, KU Leuven, Leuven, Belgium · UC Berkley, CA, USA
MoA-Structured Decode Attention DNF Derivation, KV-Cache Accumulation, GQA/MQA, and OpenACC Kernel
Jul 21, 2026Lenore Mulin, Gaetan HainsAttention Mechanisms Efficient Transformer Inference
University at Albany, SUNY · LACL, Universit´e Paris-Est Cr´eteil
Where Should Optimizer State Live? Tiered State Allocation for Memory-Efficient Mixture-of-Experts Training
Jul 21, 2026Nuemaan MalikDeep Learning Optimization Mixture-of-Experts Language Models
Independent Researcher
Stronger Memory-Query Tradeoffs for Convex Optimization: The Limitations of Subquadratic Memory
Jul 21, 2026Michael Menart, Aleksandar Nikolov, Ohad ShamirQuery Complexity Oracle Complexity
Department of Computer Science, University of Toronto, Vector Institute · Department of Computer Science, University of Toronto · Weizmann Institute of Science and University of Toronto +1
PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization
Jul 17, 2026Yuchen Yang, Yifan Zhao, Anisha Dasgupta +1LLM Quantization Mixture-of-Experts Language Models
University of Illinois Urbana-Champaign
Adaptive Filtering of the KV Cache: Diagnosing and Correcting Structural-Role Bias in LLM Inference
Jul 14, 2026Soumil MandalKV Caching Long-Context Language Model Inference
ServiceNow
Full-Pipeline Inference Optimization for MiMo-V2.5 Series: Pushing Hybrid SWA Efficiency to the Limit
Jul 14, 2026Xiaomi MiMo Team, Anqi Liu, Aoxin Ma +28Efficient Multimodal Inference KV Caching
MiMo Team, Xiaomi
DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression
Jul 7, 2026Anna Cordoba, Adam Puente Tercero, Nerea Angulo Hijo +4KV Caching Long-Context Language Model Inference
Instituto de Investigación en Visión Artificial
FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference
Jul 7, 2026Anna Córdoba, Adam Puente Tercero, Nerea Angulo Hijo +4KV Caching Long-Context Language Model Inference
Instituto de Investigación en Visión Artificial
Ember: A Kilobyte-State Optimizer for Token Interfaces
Jul 1, 2026Kathan ShahLanguage Model Pretraining LLM Fine-Tuning
Preprint.
FORGE: Fused On-Register Gradient Elimination for Memory-Efficient LLM Training
Jun 22, 2026Dikshant Kukreja, Kritarth Prasad, Avinash Anand +6Automatic Differentiation Backpropagation
Puch AI · Singapore Institute of Technology · Nanyang Technological University +2
Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers
Jun 20, 2026Xin GaoTransformer Inference KV Caching
Department of Mathematics and Statistics York University Toronto, M3J 1P3
SILAGE: Memory-Efficient, Full-Gradient-Free Nonconvex Optimization for Nested Finite Sums
Jun 14, 2026Igor Sokolov, Laurent Condat, Peter RichtárikMemory-Efficient Optimization Nonconvex Stochastic Optimization
*Center of Excellence for Generative AI, King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia. · KAUST
Gefen: Optimized Stochastic Optimizer
Jun 11, 2026Nadav Benedek, Tomer Koren, Ohad FriedDeep Learning Optimization Memory-Efficient Optimization
Reichman University · Tel Aviv University, Google Research
ReasonAlloc: Hierarchical Decoding-Time KV Cache Budget Allocation for Reasoning Models
Jun 9, 2026Wenhao Liu, Hao Shi, Yunhe Li +7KV Caching Memory-Efficient Optimization
1Tsinghua University · 2City University of Hong Kong · 3Peking University +2

Oct 1, 2026J/K move · Enter open · S save

AF-Muon: An AdamW-Free Muon Optimizer for Tied-Embedding Models

Arash Lagzian, Paniz Halvachi, Junming Zhang, Zhouhan Lin+1

National University of Singapore, Singapore · Independent Researcher · LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China

Memory-Efficient Optimization Muon Optimizer LLM Training

Open paper AskPDF arXiv

Muon improves large-scale training by applying a spectral-norm steepest-descent update to matrix parameters, but practical models also contain parameter blocks that do not fit dense-matrix geometry. One important case is the tied vocabulary table, which appears in language models and other token generators and can receive multiple structurally different gradient sources, from sparse input lookups to dense output-classifier updates. In the reference recipe these blocks are handed to an auxiliary AdamW optimizer, which restores second-moment state and updates the aliased table as a generic tensor. We propose AF-Muon, an AdamW-free extension of Muon that keeps the Muon matrix update for hidden weight matrices while using a support-aware finite-cap linear minimization oracle for tied vocabulary tables and an RMS-normalized update for one-dimensional auxiliary parameters. AF-Muon therefore trains every parameter class with a single first-moment buffer and no second-moment state, saving around 20% optimizer-state memory relative to Hybrid Muon in our benchmark. Across nine tied-token settings - decoder-only language models from 124M to 1B parameters, a fully shared T5-style encoder-decoder, and ImageGPT-style image-token, protein, and sparse-MoE variants, spanning text, image, and protein-sequence data - AF-Muon improves mean validation loss and perplexity over both Hybrid Muon and a SCION-style Sign endpoint. Long-horizon runs and hyperparameter sensitivity studies confirm the gain is robust, and identical-momentum diagnostics attribute it to the finite cap, which preserves more within-row magnitude than Sign while bounding the coordinate concentration of row-RMS. These results identify tied vocabulary tables as a distinct optimizer geometry and yield a robust AdamW-free Muon variant across models, modalities, and architectures, with about 1% step-time overhead in matched training.