Parallel LLM Inference

LLM: Large Language Model

Latest papers

22 papers3 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

Workload-Static Optical Circuit Switching For Scalable LLM Inference Decode
Nov 3, 2026Maximilian Thürmer, Nicolas Scotto di Perto, Eloise Stein +6LLM Inference Acceleration Parallel LLM Inference
Salience Labs Ltd, 100 Park Drive, Oxfordshire, OX14 4RY, United Kingdom · NVIDIA Corporation, Santa Clara, CA, USA · Hardware and Artificial Intelligence Lab, Heidelberg University, Germany
Multi-layer Indexer: Overlapping Selection and Attention in DeepSeek Sparse Attention
Nov 2, 2026Serdarcan Dilbaz, Arjun Krishna, Kiran Kamble +1Long-Context Language Model Inference LLM Inference Acceleration
Writer AI Research, Writer, Inc.
BasisKV: Value Latents for Communication and Key Routing
Nov 2, 2026Lixun Zhang, Zishan Shao, Kangning Cui +3KV Cache Selection KV-Cache Management
Duke University · Wake Forest University
Compiling Autoregressive LLM Control into Policy-Exact Parallel Inference for Hierarchical RAG
Oct 22, 2026Zehao Lin, Qianfeng Shen, Xixuan Hao +8Language Model-Based Control Efficient Language Model Inference
MemTensor (Shanghai) Technology Co., Ltd. · China Haisum Engineering Co., Ltd. · The Hong Kong University of Science and Technology (Guangzhou) +1
FlashMap: Flow Map Speculative Decoding
Oct 17, 2026Jeff Shen, Zhejian Zhou, Ryan Lee +2Speculative Decoding LLM Inference Acceleration
Polymathic AI · Princeton University · USC Information Sciences Institute +2
Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off
Oct 14, 2026Javad Mirzaei, Jeebak MitraParallel LLM Inference LLM Inference
Office of CTO Dell Technologies Inc. Canada
TriFleetRCA: On-Premise LLM Root Cause Analysis for Kubernetes
Sep 20, 2026Rohit Patel, Susil Kumar Mohanty, Jeenal ChaudharyLLM Inference Parallel LLM Inference
Department of Computer Science and Engineering Indian Institute of Technology Jodhpur, Jodhpur, India
Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling
Sep 16, 2026Mobina Kashaniyan, Ali JannesariLLM Inference Efficiency LLM Inference
Department of Computer Science Iowa State University, USA
Self-Orchestrating Language Models: Leveraging Semantic Dependence for Efficient Inference
Sep 13, 2026Tian JinAutoregressive Decoding Efficient Language Model Reasoning
Department of Electrical Engineering and Computer Science, MASSACHUSETTS INSTITUTE OF TECHNOLOGY
NeuroPrefetcher: Storage-Aware Sparse LLM Inference via Delta Prefetching
Aug 23, 2026Nobel Dhar, Md Romyull Islam, Xuechen Zhang +4Parallel LLM Inference LLM Inference
Kennesaw State University Marietta, USA · Memory Solutions Lab, Samsung Semiconductor, Inc. San Jose, California, USA
Celty: SpMspV GPU Kernel and SIMT Co-Design for Efficient Dual-Sparse LLM Inference
Aug 2, 2026Ruokai Yin, Priyadarshini PandaGPU Kernel Optimization LLM Inference
Yale University New Haven, USA · University of Southern California Los Angeles, USA
When Does Delegation Beat Majority? A Delegation-Based Aggregator for Multi-Sample LLM Inference
Jun 6, 2026Yasushi Sakai, Allen Song, Kent LarsonLLM Inference Parallel LLM Inference
1MIT Media Lab, Cambridge, MA
Speculative Decoding Across Languages
May 28, 2026Nirajan Paudel, Michael Ginn, Luc De Nardi +1Multilingual Language Models Speculative Decoding
University of Colorado
Memory-Bound but Not Bandwidth-Limited: The Physical AI Inference Gap in Batch-1 LLM Decode
May 28, 2026Josef ChenGPU Kernel Optimization GPU Acceleration
KAIKAKU
BlockBatch: Multi-Scale Consensus Decoding for Efficient Diffusion Language Model Inference
May 28, 2026Xiaoyou Wu, Cheng-Jhih Shih, Binfei Ji +2Blockwise Diffusion Language Models Diffusion Language Model Inference
Georgia Institute of Technology
PSD: Pushing the Pareto Frontier of Diffusion LLMs via Parallel Speculative Decoding
May 15, 2026Shengyin Sun, Yiming Li, Renxi Liu +7Parallel Speculative Decoding Masked Diffusion Models
Department of Computer Science City University of Hong Kong · Huawei Technologies · City University of Hong Kong
DeInfer: Efficient Parallel Inferencing for Decomposed Large Language Models
Apr 20, 2026You-Liang Huang, Xinhao Huang, Chengxi Liao +1LLM Inference Parallel LLM Inference
Boston University · Boston, Massachusetts, USA · HKUST (Guangzhou) +2
Characterizing Performance-Energy Trade-offs of Large Language Models in Multi-Request Workflows
Mar 12, 2026Md. Monzurul Amin Ifath, Israat HaqueParallel LLM Inference LLM Inference
Dalhousie University, Halifax, NS, Canada
Reject, Resample, Repeat: Understanding Parallel Reasoning in Language Model Inference
Mar 9, 2026Noah Golowich, Fan Chen, Dhruv Rohatgi +4Parallel Reasoning in Language Models LLM Inference
Microsoft Research · MIT · NYU
Parallelization Strategies for Dense LLM Deployment: Navigating Through Application-Specific Tradeoffs and Bottlenecks
Mar 5, 2026Burak Topcu, Musa Oguzhan Cim, Poovaiah Palangappa +2Parallel LLM Inference LLM Inference
The Pennsylvania State University, State College, Pennsylvania, USA · Advanced Micro Devices, San Jose, California, USA · Advanced Micro Devices, Portland, Oregon, USA
MineDraft: A Framework for Batch Parallel Speculative Decoding
Feb 24, 2026Zhenwei Tang, Arun Verma, Zijian Zhou +4Speculative Decoding LLM Inference Acceleration
College of Computing and Data Science, Nanyang Technological University, Republic of Singapore · Singapore-MIT Alliance for Research and Technology Centre · Department of Computer Science, National University of Singapore, Republic of Singapore +1
Mixture-of-Models: Distributed Inference over Heterogeneous LLMs as Model-Level Mixture-of-Experts
Jan 23, 2026Tims Pecerskis, Aivars SmirnovsMixture-of-Experts Inference Language Model Ensembles
Peeramid Labs · The AI Futures Collective

Nov 3, 2026J/K move · Enter open · S save

Workload-Static Optical Circuit Switching For Scalable LLM Inference Decode

Maximilian Thürmer, Nicolas Scotto di Perto, Eloise Stein, Natanijel Vasic+5

Salience Labs Ltd, 100 Park Drive, Oxfordshire, OX14 4RY, United Kingdom · NVIDIA Corporation, Santa Clara, CA, USA · Hardware and Artificial Intelligence Lab, Heidelberg University, Germany

LLM Inference Acceleration Parallel LLM Inference LLM Inference Scheduling Energy-Aware LLM Inference

Open paper AskPDF arXiv

LLM inference decode executes fine-grained collectives whose latency directly limits per-user interactivity, yet scale-up fabrics are still built from electronic packet switches designed for training. We observe that, for a given model deployment and parallelization, the communication structure of decode is fixed, and propose workload-static optical circuit switching (OCS): circuits are configured once per deployment, co-designed with model parallelism and with topology-aware collective schedules that turn spare accelerator ports into fewer communication phases or higher bandwidth. We evaluate this design with GXNET, a graph-based workload emulator coupled to analytical and flit-level network models grounded in hardware measurements. In a 576-accelerator scale-up domain serving 5T-parameter MoE models with up to 5M tokens of context, all-OCS fabrics improve interactivity by up to 65% and energy efficiency by up to 119% over 4:1 oversubscribed packet-switched fabrics.