Efficient Transformer Inference

Follow this topic Ask about this topic

The field aims to make Transformer serving faster and cheaper, especially for autoregressive decoding and long contexts where attention, KV caches, memory movement, and hardware utilization are bottlenecks. Approaches include linear, sparse, windowed, or phase-aware attention; cache compression and sharing; quantization, pruning, and early exits; and optimized kernels, compilers, accelerators, and distributed or encrypted execution. Many methods adapt computation by token, layer, or workload, while others co-design model structure with hardware such as GPUs, CPUs, NPUs, and FPGAs. The central tension is turning theoretical savings into measured end-to-end speed and energy gains without unacceptable quality loss, since results depend on the model, workload, and hardware.

Latest papers

122 papers15 papers in the last four weeks, up 7% on the four weeks before. 0.1% of all new papers.

Pianist Transformer: Towards Expressive Piano Performance Rendering via Scalable Self-Supervised Pre-Training
Dec 2, 2025Hong-Jie You, Jie-Jing Shao, Xiao-Wen Yang +3Efficient Transformer Inference Self-Supervised Pre-Training
State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China · School of Artificial Intelligence, Nanjing University, Nanjing 210023, China · School of Intelligence Science and Technology, Nanjing University, Suzhou 215163, China
Controllably Efficient Language Models
Nov 7, 2025Jatin Prakash, Aahlad Puli, Rajesh RanganathEfficient Transformer Inference Long-Context Language Modeling
New York University

Dec 2, 2025J/K move · Enter open · S save

Pianist Transformer: Towards Expressive Piano Performance Rendering via Scalable Self-Supervised Pre-Training

Hong-Jie You, Jie-Jing Shao, Xiao-Wen Yang, Lin-Han Jia+2

State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing 210023, China · School of Artificial Intelligence, Nanjing University, Nanjing 210023, China · School of Intelligence Science and Technology, Nanjing University, Suzhou 215163, China

Efficient Transformer Inference Self-Supervised Pre-Training Symbolic Music Generation

Open paper AskPDF arXiv

Existing methods for expressive music performance rendering, a conditional generation task that aims to generate a human-like performance from a symbolic score, rely on supervised learning over small labeled datasets, which limits scaling of both data volume and model size, despite the availability of vast unlabeled music, as in vision and language. To address this gap, we introduce Pianist Transformer, with three key contributions: 1) introducing large-scale self-supervised learning into expressive piano performance rendering through a unified Musical Instrument Digital Interface (MIDI) representation, enabling pre-training on 10B tokens of unlabeled MIDI data; 2) an efficient asymmetric Transformer with note-level compression, substantially improving training efficiency, memory usage, and inference speed for long-context music modeling; 3) a state-of-the-art rendering model with an editable workflow, achieving strong objective and subjective results and enabling integration into real-world music production workflows. Overall, Pianist Transformer outlines a scalable path toward human-like performance synthesis in the music domain. Code, audio samples, and model checkpoints are available on our project page: https://yhj137.github.io/pianist-transformer-demo/.

Latest papers

Pianist Transformer: Towards Expressive Piano Performance Rendering via Scalable Self-Supervised Pre-Training

Controllably Efficient Language Models