LLM Inference Acceleration

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

108 papers in the last four weeks, up 120% on the four weeks before. 1.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 677

All topics

FD-RAG: Federated Dual-System Retrieval-Augmented Generation
May 22, 2026Tianhao Gao, Kai Yang, Yiyang LiRetrieval-Augmented Generation Memory-Augmented Language Models
School of Computer Science and Technology, Tongji University
GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs
May 21, 2026Jianing Deng, Song Wang, Dongwei Wang +4Mixed-Precision Quantization LLM Quantization
University of Pittsburgh · University of Central Florida · University of Arizona +1
Asymmetric Virtual Memory Paging for Hybrid Mamba-Transformer Inference
May 21, 2026An Xuan NguyenKV Caching Memory-Efficient Optimization
Codepawl Ho Chi Minh City, Vietnam
Meta-Soft: Leveraging Composable Meta-Tokens for Context-Preserving KV Cache Compression
May 21, 2026Wei Luo, Yi Huang, Songchen Ma +3KV Caching KV-Cache Eviction
Guangdong Institute of Intelligence Science and Technology · University of Macau · Chinese University of Hong Kong, Shenzhen +1
Skill Weaving: Efficient LLM Improvement via Modular Skillpacks
May 21, 2026Zhuo Li, Guodong Du, Zesheng Shi +5LLM Compression LLM Inference Acceleration
Harbin Institute of Technology, Shenzhen, China · The Hong Kong Polytechnic University · Huawei Technologies Co., Ltd. +1
SpecHop: Continuous Speculation for Accelerating Multi-Hop Retrieval Agents
May 21, 2026Mehrdad Saberi, Keivan Rezaei, Soheil FeiziMulti-Hop QA Tool-Augmented Language Model Agents
University of Maryland, College Park
EntmaxKV: Support-Aware Decoding for Entmax Attention
May 20, 2026Gonçalo Duarte, Miguel Couceiro, Marcos V. TrevisoSelf-Attention KV Caching
Instituto Superior Técnico, Universidade de Lisboa. · ELLIS Unit Lisbon. · INESC-ID. +1
Llamas on the Web: Memory-Efficient, Performance-Portable, and Multi-Precision LLM Inference with WebGPU
May 20, 2026Reese Levine, Rithik Sharma, Nikhil Jain +5GPU Acceleration LLM Inference
UC Santa Cruz · Microsoft Research
Mix-Quant: Quantized Prefilling, Precise Decoding for Agentic LLMs
May 19, 2026Haiquan Lu, Zigeng Chen, Gongfan Fang +2LLM Quantization Long-Context Language Model Inference
National University of Singapore
Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decoding
May 19, 2026Yuhao Shen, Tianyu Liu, Xinyi Hu +9Speculative Decoding LLM Inference Acceleration
1Zhejiang University · 2Qwen Applications Business Group of Alibaba · University of Science and Technology of China
FlexDraft: Flexible Speculative Decoding via Attention Tuning and Bonus-Guided Calibration
May 19, 2026Yaojie Zhang, Jianuo Huang, Junlong Ke +5Speculative Decoding LLM Inference Acceleration
EPIC Lab, SJTU · School of Software Engineering, HUST · UESTC +3
OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
May 19, 2026Zunhai Su, Rui Yang, Chao Zhang +11KV Caching Rotation-Based Quantization
Tsinghua University · Meituan LongCat Team · The University of Hong Kong +3
MMoA: An AI-Agent framework with recurrence for Memoried Mixure-of-Agent
May 18, 2026Rui ChuLLM Inference Acceleration Multi-Agent Systems
Post-Trained MoE Can Skip Half Experts via Self-Distillation
May 18, 2026Xingtai Lv, Li Sheng, Kaiyan Zhang +12Mixture of Experts Mixture-of-Experts Inference
Tsinghua University · Shanghai AI Lab · WeChat AI +1
Prune, Update and Trim: Robust Structured Pruning for Large Language Models
May 18, 2026Diego Coello de Portugal Mecke, Tom Hanika, Lars Schmidt-ThiemeLLM Pruning LLM Inference Acceleration
ISMLL & DARC VWFS · University of Hildesheim · Hildesheim, Germany +1
Context Memorization for Efficient Long Context Generation
May 18, 2026Yasuyuki Okoshi, Hao Mark Chen, Guanxi Lu +3Memory-Augmented Language Models Long-Context Language Model Inference
Institute of Science Tokyo, Japan · Imperial College London, UK
KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
May 18, 2026Jian Lin, Jiazhi Mi, Zicong Hong +5KV-Cache Offloading KV Caching
Hong Kong University of Science and Technology, China · Xi’an Jiaotong University, China
Predictive Prefetching for Retrieval-Augmented Generation
May 18, 2026Wuyang Zhang, Shichao PeiRetrieval-Augmented Generation LLM Inference Acceleration
Department of Computer Science, University of Massachusetts Boston, Boston, United States of America.
CoX-MoE: Coalesced Expert Execution for High-Throughput MoE Inference with AMX-Enabled CPU-GPU Co-Execution
May 18, 2026Muyoung Son, Yi Chen, Seungjae Yoo +2Expert Offloading GPU Acceleration
KAIST · Daejeon, Republic of Korea
OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization
May 18, 2026Zhongzhu Zhou, Donglin Zhuang, Jisen Li +4Rotation-Based Quantization LLM Inference Acceleration
Together AI · University of Sydney · University of Illinois Urbana-Champaign
VeriCache: Turning Lossy KV Cache into Lossless LLM Inference
May 17, 2026Jiayi Yao, Samuel Shen, Kuntai Du +7KV-Cache Offloading KV Caching
University of Chicago · Tensormesh Inc. · Samsung Semiconductor +1
Delay-Adaptive Speculation Control for Low-Latency Edge-Cloud LLM Inference
May 17, 2026Kangkang Sun, Jianhua Li, Xiuzhen Chen +2Sequential Decision Making Optimal Stopping
Shanghai Key Laboratory of Integrated Administration Technologies for Information Security, School of Computer Science, Shanghai Jiao Tong University, Shanghai 200240, China · School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen, Guangdong, China
TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks
May 16, 2026Hanzhang Shen, Haoran Wu, Yiren Zhao +1Mixed-Precision Quantization LLM Inference Acceleration
University of Cambridge · Imperial College London
Charon: A Unified and Fine-Grained Simulator for Large-Scale LLM Training and Inference
May 16, 2026Mengtian Yang, Zhekun Zhang, Mingheng Wu +3LLM Inference LLM Inference Acceleration
ByteDance Seed · University of Texas at Austin
Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps
May 16, 2026Yanke Zhou, Yiduo Li, Hanlin Tang +6Self-Attention Long-Context Language Model Inference
Nanjing University · Alibaba Group
CompactAttention: Accelerating Chunked Prefill with Block-Union KV Selection
May 16, 2026Jiwon Song, Dongwon Jo, Beomseok Kang +1Self-Attention KV-Cache Management
Seoul National University
Lever: Speculative LLM Inference on Smartphones
May 16, 2026Tuowei Wang, Fengzu Li, Yanfan Sun +2On-Device Language Model Inference LLM Inference
Tsinghua University · Beijing, China · Beihang University +1
Towards Generalization of Block Attention via Automatic Segmentation and Block Distillation
May 15, 2026Shuaiyi Li, Zhisong Zhang, Yan Wang +5Long-Context Modeling LLM Inference Acceleration
The Chinese University of Hong Kong · City University of Hong Kong · Tencent +2
IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compression
May 15, 2026Ali Abbasi, Chayne Thrash, Haoran Qin +2LLM Compression Low-Rank Matrix Decomposition
Vanderbilt University · University of California, Davis
STS: Efficient Sparse Attention with Speculative Token Sparsity
May 15, 2026Ceyu Xu, Jiangnan Yu, Yongji Wu +1Self-Attention Long-Context Language Model Inference
The Hong Kong University of Science and Technology · UC Berkeley

May 22, 2026, cs.IRJ/K move · Enter open · S save

FD-RAG: Federated Dual-System Retrieval-Augmented Generation

Tianhao Gao, Kai Yang, Yiyang Li

School of Computer Science and Technology, Tongji University

Retrieval-Augmented Generation Memory-Augmented Language Models LLM Inference Acceleration Edge ML Hypergraph Learning Federated Learning

Open paper AskPDF arXiv

Retrieval-augmented generation (RAG) has emerged as a paradigm for grounding large language models in external knowledge, yet most existing RAG systems assume centralized knowledge access and ample computation. These assumptions break down in edge environments, where knowledge is fragmented across devices, raw data cannot be shared, and repeated LLM calls are prohibitively expensive. We propose FD-RAG, a federated dual-system RAG framework that decouples lightweight memory access from on-demand LLM reasoning for decentralized deployment. Specifically, FD-RAG learns semantic-aware adaptive hypergraphs over local corpora and distills them into compact QA memories. At inference time, it answers well-covered queries via direct memory matching and invokes LLM-based reasoning only when necessary, while tracing retrieved memories to hypergraph-grounded evidence. To mitigate cross-device knowledge fragmentation, FD-RAG aggregates anonymized memories across devices without exposing raw documents. Experiments on QA benchmarks show that FD-RAG improves accuracy by up to 7.8% while reducing latency by 8.4

\times

compared with strong local and federated baselines. We also provide theoretical analysis establishing an

\mathcal{O}(1/ε^{2})

convergence rate for the proposed hypergraph learning, supporting its tractable deployment in edge settings.

Momentum

Related topics

Latest papers 677