Inference Workloads

Follow this topic Ask about this topic

Momentum

4 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 47

All topics

FlexiFlow: Bandit-based Model Switching in ML Workflows
Oct 5, 2026Abhilash Jindal, Todd Nief, Bhanu Prakash Vangala +6Inference Workloads Heterogeneous Modeling Tools
IIT Delhi India · University of Chicago USA · University of Missouri USA +1
CIPHER-MoE: Balancing Efficiency and Routing Fidelity in Trillion-Scale MoE Training
Oct 5, 2026Jing Li, Jian Meng, Yingmeng Gao +8Inference Workloads
Tongji University · Cornell University · Harbin Institute of Technology, Shenzhen +1
The Output-Space Hypothesis: Enumerative Equivalence Checking for Tensor Programs
Sep 17, 2026Paul Biberstein, Joseph Devietti, Mayur NaikCuda Bug
University of Pennsylvania Philadelphia, PA, USA
COMPASS-ABS: Reducing Fragmentation in Shared GPU Clusters for Deep Learning Training Workloads
Sep 16, 2026Yukai Zhou, Hongfan WuInference Workloads Fragmentation
SeqMoE: Toward Full-Load Performance via Predictive and Graph-Compatible MoE Offloading
Sep 14, 2026Zihan Wang, Yuqi Wang, Lei Gong +5Mixture-Of-Expert Inference Offloading
University of Science and Technology of China Hefei, China · Suzhou Institute for Advanced Research, University of Science and Technology of China Suzhou, China
Memory Compression for High-Fanout Agent Sandboxes
Sep 12, 2026Mengming Li, Ceyu XU, Qijun Zhang +4Sandbox Task-Aware Compression
HKUST
Network-Aware Forecasting on Wireless Access Points
Sep 2, 2026Niloo Bahadori, Swadhin Pradhan, Peiman AminiWifi Inference Workloads
Cisco Systems
Prof-K: Probabilistic One-Pass Filtering for Efficient Top-k Selection
Aug 12, 2026Tadeusz Dziarmaga, Witold Sikora, Łukasz Struski +2Top-K Inference Workloads
Jagiellonian University
EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inference
Aug 8, 2026Yize Wu, Ke Gao, Ling Li +1Load Balancing Mixture-Of-Expert Inference
Intelligent Software Research Center, Institute of Software, CAS, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
ML-for-ML
Aug 6, 2026Yutong Zhao, Noga H. Rotman, Gianni Antichi +1Inference Workloads Bringing Network Coding
UCL and BUPT · United Kingdom · UCL +3
Meganeura: Portable GPU Training and Inference through Vulkan and Metal
Aug 3, 2026Dzmitry MalyshauPytorch Inference Workloads
Independent Researcher
On the Limits of Machine-Learned Ranking for Modern Microarchitectural Policies
Aug 2, 2026Yanxin Zhang, Shayne Wadle, Yuxuan Xiong +3Hardware Efficiency Mle-Bench Lite
NVIDIA · University of Wisconsin–Madison
The SpiNNaker2 chip: a many-core platform for flexible and scalable brain-inspired computing
Jul 27, 2026Stefan Scholze, Johannes Partzsch, Sebastian Höppner +27Neuromorphic Computing Chip Design
Chair of Highly-Parallel VLSI-Systems and Neuro-Microelectronics, Technische Universität Dresden, Germany · Centre for Tactile Internet with Human-in-the-loop (CeTI) · Advanced Processor Technologies Group, Department of Computer Science, University of Manchester, UK +2
Relaxed activation analysis of dataflow networks - A clock calculus for machine learning and real-time scheduling
Jul 23, 2026William Gaudelier, Albert Cohen, Dumitru Potop ButucaruTaxonomy-Driven Dataflow Model Clocks
Inria, Paris, France · Google DeepMind, Paris, France
BRIM: Workload-Balanced Dual-Sided Bit-Serial Sparse Inference Accelerator
Jul 20, 2026Varun Manjunath, Ruokai Yin, Donghyun Lee +2Inference Workloads Neural Network Inference
University of Southern California Los Angeles, USA · Yale University New Haven, USA
A Blueprint for Equilibrium-Based Differentiable Continuous-Variable Thermodynamic Computing
Jul 17, 2026Owen Lockwood, Jérémy Béjanin, Joost Bus +4Thermodynamics Inference Workloads
Extropic Corporation, San Francisco, California 94111, USA · 2Noumenal Labs Inc, Dallas, Texas 75229, USA · 1Extropic Corporation, San Francisco, California 94111, USA +1
Quota Marketplace: Dynamic Pricing for Efficient Allocation of ML Training Resources
Jul 9, 2026Balasubramanian Sivan, Renato Paes Leme, Mihai Tiuca +6Schedulers Dynamic Pricing
Google · Drexel University
Optimizing ML Workload Partitioning between CPUs and CIM Accelerators for Heterogeneous Computing
Jul 6, 2026Joel Klein, Rebecca Pelke, Roberto Laudani +2Inference Workloads In-Memory Computing
Institute for Communication Technologies and Embedded Systems, RWTH Aachen University, Germany
WattGPU: Predicting Inference Power and Latency on Unseen GPUs and LLMs
Jul 2, 2026Mauricio Fadel Argerich, Jonathan Fürst, Marta Patiño-MartínezGraphics Processing Unit Architectures LLM Inference Optimization
Universidad Politécnica de Madrid, C. de los Ciruelos, 28660 Boadilla del Monte, Madrid, Spain · Zurich University of Applied Sciences, Gertrudstrasse 15, 8400 Winterthur, Switzerland
DMuon: Efficient Distributed Muon Training with Near-Adam Overhead
Jun 25, 2026Vincent Chen, Starrick Liu, Regis Cheng +8Muon Inference Workloads
X SQUARE ROBOT TEAM
Load Testing for Machine Learning Model Serving Systems at Scale
Jun 20, 2026Amr S. Abdelfattah, Nakul Tirumalai, Indu Mohanan +4Large Language Model Serving Inference Workloads
Meta Inc., Menlo Park, USA
Recency/Frequency Adaptive KV Caching for Large Language Model Serving
Jun 19, 2026Yang Shen, Meghana Madhyastha, Robert Underwood +2Key-Value Cache LLM Inference Optimization
Department of Computer Science, Johns Hopkins University, Baltimore, USA
Agentic Framework for Deep Learning workload migration via In-Context Learning
Jun 14, 2026Qiyue Liang, Steven Ingram, George Vanica +4Pytorch Agentic Framework
Google, Sunnyvale, CA · 2Google, Kirkland, WA · 1Google, Sunnyvale, CA +2
FMplex: Model Virtualization for Serving Extensible Foundation Models
Jun 8, 2026Hetvi Shastri, Pragya Sharma, Walid A. Hanafy +3Wireless Foundation Models Large Language Model Serving
University of Massachusetts Amherst · University of California Los Angeles
Resource-aware Computation-Communication Overlap for multi-GPU ML Workloads
Jun 8, 2026Minyu Cui, Miquel PericasCpu-Gpu Hybrid Designs Inference Workloads
Chalmers University of Technology and University of Gothenburg, Gothenburg, Sweden
Breaking the Ice: Analyzing Cold Start Latency in vLLM
Jun 5, 2026Huzaifa Shaaban Kabakibo, Animesh Trivedi, Lin WangLLM Inference Optimization Large Language Models(Llms
Paderborn University, Paderborn, Germany · IBM Research Europe, Zurich, Switzerland
Terastal: Layer-Variant-based Scheduling for Real-Time Multi-DNN Workloads on Heterogeneous Accelerators
Jun 5, 2026Sing-Yao Wu, Fengshuo Song, Eli BozorgzadehInference Workloads Real-Time Systems
Department of Computer Science, University of California, Irvine, CA, USA
Uncertainty-Aware End-to-End Co-Design of Neural Network Processors: From Training and Mapping to Fabrication
Jun 3, 2026Yuyang Du, Yujun Huang, Gioele ZardiniCo-Design Neural Processing Units
School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Laboratory for Information & Decision Systems, Massachusetts Institute of Technology, Cambridge, MA 02139
EvalStop: Using World Feedback to Detect and Correct Reward Overoptimization in Multi-Tenant RLHF Platforms
Jun 2, 2026Guilin Zhang, Chuanyi Sun, Kai Zhao +3Reinforcement Learning From Human Feedback Schedulers
Workday AI Research · The George Washington University, Washington, DC, USA.
NetKV: Network-Aware Decode Instance Selection for Disaggregated LLM Inference
Jun 2, 2026Mubarak Adetunji OjewaleKv-Cache Management Time-To-First-Token
Cloud Competency Centre, National College of Ireland
ViBE: Co-Optimizing Workload Skew and Hardware Variability for MoE Serving
May 30, 2026Seokjin Go, Marko Scrbak, Ephrem Wu +2Mixture-Of-Expert Inference Inference Workloads
Georgia Institute of Technology Atlanta, GA, USA · Advanced Micro Devices, Inc. Austin, TX, USA · Advanced Micro Devices, Inc. Santa Clara, CA, USA +1
Access Sets Matter: Budgeting Expert Reads for Scalable Weight-Space Model Merging
May 28, 2026Yuanyi Wang, Yanggan Gu, Su Lu +5Merge Inference Workloads
The Hong Kong Polytechnic University, PolyU · Hong Kong Polytechnic University Daya Bay Technology and Innovation Research Institute.
Inference-Native Zeroth-Order Optimization for LLMs
May 27, 2026Zelin Li, Caiwen DingLarge Language Model Fine-Tuning Model Fine-Tuning
University of Minnesota
Accelerating Divisible Load Processing Through Machine Learning: A Practical Framework for Large-Scale Workloads
May 22, 2026Bharadwaj VeeravalliInference Workloads Workload
Department of Electrical and Computer Engineering, National University of Singapore, 4 Engineering Drive 3, Singapore
MLCommons Chakra: Advancing Performance Benchmarking and Co-design using Standardized Execution Traces
May 11, 2026Srinivas Sridharan, Theodor-Adrian Badea, Andy Balogh +26Inference Workloads Co-Design
1NVIDIA · 2Keysight · 3AMD +7
BatchWeave: A Consistent Object-Store-Native Data Plane for Large Foundation Model Training
May 11, 2026Ting Sun, Junjie Zhang, Xiao Yan +7Inference Workloads Wireless Foundation Models
Lionrock AI Lab, China Merchants Group, Hong Kong, China · Wuhan University, Wuhan, China · The Chinese University of Hong Kong, Shenzhen, China
Direction-Preserving Number Representations
May 8, 2026Bardia Zadeh, George A. ConstantinidesFull-Precision Performance Vectors
Department of Electrical and Electronic Engineering Imperial College London London, United Kingdom
DBLP: Phase-Aware Bounded-Loss Transport for Burst-Resilient Distributed ML Training
May 3, 2026Zechen Ma, Zixi Qu, Jinyan Yi +2Inference Workloads Transport
Department of Computer Science, University of Toronto, Toronto, Canada
MoE-Prefill: Zero Redundancy Overheads in MoE Prefill Serving
May 3, 2026Zhaoyuan Su, Olatunji Ruwase, Karthik Ganesan +5Mixture-Of-Expert Inference Prefill
University of Virginia · 2Snowflake AI Research · 3Harvard University
Strait: Perceiving Priority and Interference in ML Inference Serving
Apr 30, 2026Haidong Zhao, Nikolaos GeorgantasInference Workloads Deadlines
Inria & Sorbonne University · Paris, France · Inria
Focus Session: Hardware and Software Techniques for Accelerating Multimodal Foundation Models
Apr 23, 2026Muhammad Shafique, Abdul Basit, Muhammad Abdullah Hanif +3Multimodal Foundation Model Inference Workloads
eBRAIN Lab, New York University (NYU) Abu Dhabi, Abu Dhabi, United Arab Emirates
One-Block Transformer (1BT) for EEG-Based Cognitive Workload Assessment
Apr 21, 2026Stefanos Gkikas, Christian Arzate Cruz, Thomas Kassiotis +3Inference Workloads Cognitive Load
Honda Research Institute Japan · Wako City, Japan · Department of Electronic Engineering +6
A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMs
Feb 23, 2026Zijie Liu, Jie Peng, Jinhao Duan +7Load Balancing LLM Inference Optimization
University of North Carolina at Chapel Hill · University of Minnesota Twin Cities · North Carolina State University +2
VTC: DNN Compilation with Virtual Tensors for Data Movement Elimination
Feb 11, 2026Muyan Hu, Ahan Gupta, Jiachen Yuan +7Inference Workloads Neural Network Inference
University of Illinois Urbana-Champaign · Georgia Institute of Technology · Microsoft
DCC: Data-Centric Compilation of Machine Learning Kernels for Processing-In-Memory Architectures
Nov 19, 2025Peiming Yang, Sankeerth Durvasula, Ivan Fernandez +4Inference Workloads Code Optimization
University of Toronto & Vector Institute · Barcelona Supercomputing Center · ETH Zürich +2
FuseSampleAgg: One-Pass Neighborhood Estimation for Budgeted Knowledge-Graph Refresh and Validation
Nov 17, 2025Aleksandar Stanković, Haoran Du, Xinming WangSubgraphs Knowledge Graph Embeddings
University of Novi Sad, Novi Sad, Serbia · Shanghai Key Lab of Intelligent Information Processing,2026 College of Computer Science and Artificial Intelligence, Fudan University, China · Institute of Automation, Chinese Academy of Sciences, China
Scalable Synthesis of distributed LLM workloads through Symbolic Tensor Graphs
Nov 13, 2025Changhai Man, Joongun Park, Hanjiang Wu +3Inference Workloads Large Language Model Workflows
Georgia Institute of Technology · NVIDIA Inc.

Oct 5, 2026, cs.LGJ/K move · Enter open · S save

FlexiFlow: Bandit-based Model Switching in ML Workflows

Abhilash Jindal, Todd Nief, Bhanu Prakash Vangala, Shankaradithyaa V+5

IIT Delhi India · University of Chicago USA · University of Missouri USA+1

Inference Workloads Heterogeneous Modeling Tools

Open paper AskPDF arXiv

Model optimizations help improve inference performance and accuracy of ML workflows. However, relying on a single model to perform inference across all data batches often fails to maximize accuracy and thus overall performance. In many cases, alternate models could perform better on specific subsets of data where a primary model underperforms. Our experiments with real ML workflows indeed show that switching models improves workflow accuracy by up to 23%. Yet, current systems lack the ability to adaptively switch between models based on performance, forcing users to manually test models in sequence. We present FlexiFlow, a dataflow system that dynamically switches between alternate models when the current model exhibits low accuracy. FlexiFlow learns to rank models using a novel multi-armed bandit approach that accounts for model runtimes, probability of passing user-defined assertions, and the computational structure of the ML workflow. We show that the standard Thompson sampling approach is insufficient for switching models in ML workflows. In contrast, our proposed approaches are effective and scales to complex real-world ML workflows. Experiments show that switching models at runtime while reusing intermediate results provides higher accuracy, but also 48% efficiency gain compared to sequential workflow runs.

Momentum

Related topics

Latest papers 47