Split Inference

Follow this topic Ask about this topic

Split inference studies how to partition a neural network so devices such as phones, microcontrollers, edge servers, and cloud GPUs can collaborate on a prediction. Common approaches split at layer or finer granularity, then schedule computation and activation transfers to balance device memory, network cost, and latency across CNN and vision workloads as well as large language models. Because intermediate activations can reveal inputs, another major line of work measures reconstruction attacks and uses feature obfuscation, differential privacy, or trusted execution to protect them. The central tension is to adapt partitions and resource sharing to changing workloads and networks without sacrificing accuracy, speed, or privacy.

Latest papers

13 papers3 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

Consommé: Conditioned Token Merging for Split Inference
Oct 23, 2026Man Ngo, Jan S. RellermeyerSplit Inference Efficient ViTs
Leibniz University Hannover Hannover, Germany
Exposing AI Inference as a Telco Network API over the 5G User Plane
Oct 10, 2026Rodrigo Moreira, Larissa Ferreira Rodrigues Moreira, Joberto S. B. Martins +2Edge Inference Split Inference
Federal University of Viçosa (UFV), Minas Gerais, Brazil · University of Salvador (UNIFACS), Salvador, Brazil · University of São Paulo (USP), São Paulo, Brazil +1
DySCo: Dynamic Sharding for Collaborative Edge-Cloud LLM Inference with Depth-Synchronized Batching
Oct 6, 2026Jingpo Xu, Paul Joe Maliakel, Ivona Brandic +1LLM Inference Edge-Cloud Collaborative Inference
University of Amsterdam Amsterdam, Netherlands · TU Wien Vienna, Austria
Crossflow: Prefill-Decode Elasticity for Agentic LLM Serving
Sep 22, 2026Yi Xu, Ehsan K. Ardestani, Wenyin Fu +7Disaggregated LLM Serving LLM Inference Scheduling
Meta Platforms
SpliTEE: Fast and Private LLM Inference by Coupling GPU-Assisted Trusted Execution Environments with Differential Privacy
Sep 14, 2026Shashie Dilhara Batan Arachchige, Robin Carpentier, Hassan Jameel Asghar +1Privacy-Preserving Language Models Split Inference
School of Computing, Macquarie University
Position Matters: Feature Inversion Attacks in ViT Split Inference with Token Reduction and Shuffling
Sep 1, 2026Stefano Leggio, Giulio Rossolini, Alessandro BiondiVision Transformer Split Inference
Dept. of Excellence in Robotics & AI, Scuola Superiore Sant’Anna Pisa, Italy
A Semantic Communication Approach to Fiducial Marker Processing in 5G-Enabled Edge SLAM
Aug 10, 2026Boris Radovanovic, Vukan Ninkovic, Katarina Vidojevic +2Edge Inference Semantic Communication
Faculty of Technical Sciences, University of Novi Sad, Serbia · The Institute for Artificial Intelligence Research and Development of Serbia, Serbia
Privacy from Symmetry: Orthogonally Equivariant Transformers for LLM Inference
Jun 15, 2026Alexander Yukhimchuk, Andrey Shulga, Mladen Kolar +1Privacy-Preserving Language Models LLM Inference
MBZUAI · University of Southern California
What Does the Server See? Understanding Privacy Leakage from Large Language Models in Split Inference
May 22, 2026Mingyuan Fan, Yu Liu, Fuyi Wang +1Privacy-Preserving Language Models Adversarial Attacks
East China Normal University Shanghai, China · RMIT University Melbourne, Australia
Keyed Nonlinear Transform: Lightweight Privacy-Enhancing Feature Sharing for Medical Image Analysis
May 13, 2026Haebom Lee, Gyeongjung KimMedical Image Analysis Split Inference
OOLU Soft Co., Ltd.
Split CNN Inference on Networked Microcontrollers
May 10, 2026Junyu Lu, Shashwath Suresh, Hao Liu +2Efficient Neural Network Inference Edge Inference
Delft University of Technology, Delft, The Netherlands
DPU or GPU for Accelerating Neural Networks Inference -- Why not both? Split CNN Inference
Apr 30, 2026Ali Emre Oztas, Mahir Demir, James Garside +1AI Accelerator Inference GPU Acceleration
The University of Manchester
Efficient Cross-View Localization in 6G Space-Air-Ground Integrated Network
Mar 12, 2026Min Hao, Yanbing Xu, Maoqiang Wu +6Cross-View Geo-Localization Split Inference
School of Electronic Science and Engineering, South China Normal University, Foshan, China

Oct 23, 2026J/K move · Enter open · S save

Consommé: Conditioned Token Merging for Split Inference

Man Ngo, Jan S. Rellermeyer

Leibniz University Hannover Hannover, Germany

Split Inference Efficient ViTs Visual Token Compression Visual Token Merging

Open paper AskPDF arXiv

Split inference partitions a Vision Transformer between a resource-constrained edge device and a server. However, transmitting intermediate tokens across the communication channel is costly. Deploying fine-tuned backbones to every endpoint becomes expensive as well. We study token reduction for split inference under a frozen backbone, where only a small adapter is shipped. We observe that the CLS token plays a dual role: it acts as the global summary the receiver needs for classification, and it serves as a signal on which patches to keep. We propose Consommé, a method utilizing both characteristics. There, an information bottleneck compresses the CLS token into a compact latent which is transmitted in place of the full CLS token and conditions a learned patch gate merging patches into exactly

k

tokens via soft assignment. It reaches any budget in a single step, thus escaping the per-layer cap of bipartite matching (BPM). On DeiT-S/T, it leads all methods at aggressive token budgets and remains operable at shallow splits where BPM cannot operate, while matching fine-tuned methods at gentle budgets at a fraction of their deployment cost.