Efficient Language Model Inference

Overview

The field studies how to serve language models with less computation, memory, time, and energy while preserving useful outputs. Approaches include quantization and sparsity, faster decoding such as speculative and parallel generation, KV-cache and context compression, and smaller or specialized models routed to tasks they can handle. For reasoning models, early stopping and adaptive compute allocation aim to spend more tokens on difficult questions and fewer on easy ones. The central challenge is that savings depend on the model, workload, hardware, and serving setup, and reducing token counts or latency does not always reduce total cost or energy without harming accuracy or reliability.

Latest papers

CardsList

354 papers39 papers in the last four weeks, down 44% on the four weeks before. 0.3% of all new papers.

  1. Late Entry in Looped Language Models

    Nov 18, 2026Ziqi Jin, Jiaxi Li, Weiwen Xu +1Efficient Language Model InferenceLayer Skipping in Language Model Decoding

  2. Modeling LLM Reasoning Dynamics with Structural Representations

    Nov 18, 2026Ilia Karpov, Andrey Atamanyuk, Dmitry Ilvovsky +4Structured Reasoning in Language ModelsLLM Reasoning

  3. FACT: Fixed-Budget Allocation and Cross-Step Training for Speculative Decoding

    Nov 17, 2026Hadi Fares, Mayssa Hajj Hassan, Ali Salman +1Speculative DecodingEfficient Language Model Inference

  4. BugGen 2.0: Rethinking the Roles of LLMs for Scalable RTL Bug Synthesis

    Nov 17, 2026Minh Luu, Michael Quinn, Aakash Tyagi +2Synthetic Benchmark GenerationLLM Code Generation

  5. Weight-Space Programming Doesn't Need Transformers

    Nov 17, 2026Hyunjong Ok, Jaeho LeeWeight-Space ProgrammingAssociative Memory

  6. LoopSkim: Lossless Multi-Depth Speculative Decoding for Looped Language Models

    Nov 15, 2026Muhan Xiao, Shijian LiSpeculative DecodingEfficient Language Model Inference

  7. RE-PRESENT: Evidence Triangulation across Alternative Candidate-Space Views for Efficient LLM Ensemble Inference

    Nov 10, 2026Juming Xiong, Weixin Liu, Kevin Guo +7Language Model EnsemblesMulti-LLM Collaboration

  8. Think or Remember: The Exchange Rate of Agentic Inference

    Nov 7, 2026Juan Wisznia, Santiago Luis Groba Alonso, Matías Gómez Taddeo +3Cost-Aware InferenceEfficient Language Model Inference

  9. TrialJev: Open-Weight Single Pass Eligibility Scoring for Patient Trial Screening

    Nov 6, 2026Omar Ibrahim, Grant Hamblin, Benjamin Haibe-KainsMedical LLMsEfficient Language Model Inference

  10. Towards Foundation Decision Models: Drex

    Nov 5, 2026Nischay Dhankhar, Alisher Amantay, Zhanibek Datbayev +6LLM Decision-MakingEfficient Language Model Inference

  11. Extending Reasoning in Typed Decision Models via Looping

    Nov 5, 2026David Berghaus, Arber Zela, Behzad Shomali +4Efficient Language Model ReasoningAdaptive Reasoning in Language Models

  12. ACSeL: Adaptive Configuration Search for Hardware-Constrained LLM Inference

    Nov 5, 2026Bokun Wang, Hung-Yueh Chiang, Diana MarculescuLLM InferenceEfficient Language Model Inference

  13. Shared Context Candidate Scoring with Jointly Adapted Readout Networks

    Nov 3, 2026Jingyang Du, Yiben Fu, Yang ShenMultiple-Choice Question AnsweringLLM Fine-Tuning

  14. Say Less, Mean Better: Training-Free Verbosity Control for Language Models

    Nov 3, 2026Mahmoud Alyosify, Mirna Imbabi, Eman Elkhamisi +1Efficient Language Model Inference

  15. Library of Layers (LoL): Adaptively Ordering Transformer Layers at Runtime

    Nov 2, 2026Abdellah Rahmani, Sean McLeish, Ang Li +3TransformerAdaptive Reasoning in Language Models

  16. Local or Escalate? Post-Training Expert Routing for Real-Time Duplex Speech Models

    Nov 1, 2026Ruoyu He, Jisen Li, Anthony Ji +5LLM RoutingFull-Duplex Spoken Dialogue Systems

  17. A Tool in the Forward Pass: Frozen Tape Modules as Layers of a Small Language Model

    Oct 31, 2026Michael Granitzer, Saber Zerhoudi, Mohammad Al-Maamari +2LLM Tool UseEfficient Language Model Inference