Overview

Transformer inference covers the work needed to run trained models, from processing a prompt (prefill) to generating tokens one at a time (decoding). Common approaches reduce attention and KV-cache costs through sparsity, alternative attention mechanisms, compression, or cache reuse; others use quantization, parallel computation, adaptive depth, and specialized GPU, FPGA, or edge hardware. Researchers evaluate these methods on language-model quality, retrieval and reasoning benchmarks, latency, memory use, energy, and sometimes privacy guarantees. The central tension is reducing computation and memory without changing outputs or degrading task performance, while inconsistent hardware and evaluation assumptions make results difficult to compare.

Latest papers

CardsList

63 papers6 papers in the last four weeks, down 45% on the four weeks before. 0.0% of all new papers.

  1. Implementing Binary Transformers with Approximate Memories with Permutation Invariance Enabled Selective (PIES) Protection

    Nov 17, 2026Zhen Gao, Xiao Yin, Kexin Liang +3Neural Network Fault ToleranceApproximate Computing

  2. The Transformer Paradigm: Dynamic Processing and a New Principle of Inference

    Nov 13, 2026Marco Giunti, Fabrizia Giulia GaravagliaTransformer InferenceDynamic Neural Networks

  3. MOAI: Module-Optimizing Architecture for non-interactive secure transformer Inference

    Nov 10, 2026Linru Zhang, Xiangning Wang, Jun Jie Sim +5Privacy-Preserving Language ModelsPrivacy-Preserving ML

  4. Transformer-FSQ: Discrete Latents for Extreme KV-Cache Compression

    Nov 6, 2026Andras Szecsenyi, Thomas L. Lee, Vaisakh Shaj +2KV-Cache CompressionKV-Cache Quantization

  5. Atik: A Shared-Mesh Transformer Accelerator for RISC-V

    Nov 6, 2026Ahmed Zeer, Erkan UsluNeural Network Hardware AccelerationTransformer Inference

  6. INTRA: Interleaved Non-contiguous Token spaRse Attention

    Oct 25, 2026Yuxuan Xia, Wenbo Lu, Shaoyi Zheng +1Sparse AttentionTransformer Inference

  7. TRACE: Transformer Residual Activation Codebook Encoding for Communication-efficient Distributed Edge Inference

    Oct 15, 2026Muhammad Azlan Qazi, Alexandros Iosifidis, Qi ZhangEfficient Transformer InferenceTransformer Inference

  8. When Does Depth Help? Understanding Layerwise Inference in Tabular Foundation Models

    Oct 14, 2026Wenwen Gong, Haotian Gu, Zhen Yang +6Tabular Foundation ModelsTransformer Inference

  9. Not All Dimensions Are Equal: Outlier-Aware Attention Linearization with Grouped Quadratic Kernels

    Oct 9, 2026Weiqi Sun, Qianyue Wu, Wenjia Wang +1Attention ApproximationEfficient Attention

  10. Stochastic Rounding in Low-Precision Transformer Inference: A Variable-Precision Emulation Study of a Small GPT-2

    Oct 1, 2026Yohan Chatelain, Pablo de Oliveira CastroTransformer InferenceStochastic Rounding

  11. The Geometry of Inference in Transformer Residual Streams

    Sep 29, 2026Timur Mudarisov, Mikhail Burtsev, Radu StateTransformer InterpretabilityTransformer Inference

  12. The Extender: A Log-Structured Transformer

    Sep 26, 2026Jakob ErikssonTransformerTransformer Inference

  13. DegreeSpar: Structured Degree Sparsity for Efficient Secure Transformer Inference

    Sep 26, 2026Yifei Cai, Zhuoran Li, Xiaozuo Shen +2Transformer InferencePrivacy-Preserving ML

  14. Programs-of-Layers in LLMs through the Lens of Cortical Areas

    Sep 25, 2026Justus Westerhoff, Stephan Olbrich, Hatem Oraby +3Transformer InferenceAdaptive Computation

  15. A Smaller Transformer in Your Transformer

    Sep 17, 2026Dhananjay Tomar, Marius Aasan, Andreas Kleppe +1Transformer InferenceVision Transformer

  16. Recent Developments in Transformer Inference Deployment on FPGA Platforms: A Survey

    Sep 1, 2026Arjan Blankestijn, Uraz Odyurt, Amirreza YousefzadehEfficient Transformer InferenceTransformer Inference

  17. CacheBridge: Efficient Cross-Model KV Transfer for Multi-Model LLM Serving

    Sep 1, 2026Xingyu Qu, Siyuan Lu, Zhiyu Chen +2Transformer InferenceKV Caching

  18. The Transformer Revolution, Part 1: Dynamic Processing through Output-Weight Interconnections

    Aug 4, 2026Marco Giunti, Fabrizia Giulia GaravagliaTransformer InferencePrompt Sensitivity

  19. At-the-Roofline Sparse Tensor Contractions on Vector Processors for Transformer Inference

    Jul 28, 2026Bowen Wang, Chi Zhang, Diyou Shen +3Efficient Neural Network InferenceTransformer Inference

  20. Agree on the Model, Verify the Inference: GKR Protocols for HND-Based Transformer Inference

    Jul 23, 2026Xiaolong Liang, Juanjuan Li, Rui Qin +1Neural Network VerificationTransformer Inference

  21. MoA-Structured Decode Attention DNF Derivation, KV-Cache Accumulation, GQA/MQA, and OpenACC Kernel

    Jul 21, 2026Lenore Mulin, Gaetan HainsAttention MechanismsEfficient Transformer Inference

  22. On the Diverse Dynamical Behaviors Arising in Deep Linear Transformers

    Jul 20, 2026Sixu Li, Thomas Jacob Maranzatto, Jan Peszek +5Dynamical SystemsKuramoto Model

  23. Layer-Parallel Inference Reduces Encrypted Nonlinear Depth in Transformers

    Jul 6, 2026Ligong Han, Kai Xu, Hao Wang +3Efficient Transformer InferenceTransformer Inference