Inference Time

Inference time, the time taken for a model to process an input and produce an output, is a critical factor in the performance and scalability of large language models (LLMs) and other deep learning systems. Current research focuses on optimizing inference efficiency through techniques like adaptive sampling, architecture search for efficient inference-time techniques, and model compression methods, aiming to reduce computational costs without sacrificing accuracy. These advancements are crucial for deploying LLMs in resource-constrained environments and improving the responsiveness of AI applications, impacting both the efficiency of AI systems and their accessibility to a wider range of users.

Papers

January 2, 2025

HoneypotNet: Backdoor Attacks Against Model Extraction
Yixu Wang, Tianle Gu, Yan Teng, Yingchun Wang, Xingjun Ma
Backdoor Attack Inference Time Model Extraction Attack Model Extraction Victim Model Honeypot Interface

December 30, 2024

Efficiently Serving LLM Reasoning Programs with Certaindex
Yichao Fu, Junda Chen, Siqi Zhu, Zheyu Fu, Zhongdongming Dai, Aurick Qiao, Hao Zhang
Complex Reasoning Task LLM Reasoning Inference Time Reasoning Question

December 18, 2024

Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models
Yinlam Chow, Guy Tennenholtz, Izzeddin Gur, Vincent Zhuang, Bo Dai, Sridhar Thiagarajan, Craig Boutilier, Rishabh Agarwal, Aviral Kumar, Aleksandra Faust
Full Model Imitation Learning Large Language Inference Time Inference Task Inference Strategy Best of N

December 12, 2024

Dipper: Diversity in Prompts for Producing Large Language Model Ensembles in Reasoning tasks
Gregory Kang Ruey Lau, Wenyang Hu, Diwen Liu, Jizhuo Chen, See-Kiong Ng, Bryan Kian Hsiang Low
Large Language Model Complex Prompt Diverse Ensemble Reasoning Task Diversity Awareness Inference Time Batch Inference

December 11, 2024

HARP: A challenging human-annotated math reasoning benchmark
Albert S. Yue, Lovish Madaan, Ted Moskovitz, DJ Strouse, Aaditya K. Singh
Language Model Mathematical Reasoning Reasoning Benchmark Inference Time

December 4, 2024

Scaling Inference-Time Search with Vision Value Model for Improved Visual Comprehension
Wang Xiyao, Yang Zhengyuan, Li Linjie, Lu Hongjin, Xu Yuancheng, Lin Chung-Ching Lin, Lin Kevin, Huang Furong, Wang Lijuan
Vision Language Model Vision Model Large Relevance Improvement Visual Understanding Inference Time Multimodal Benchmark

November 27, 2024

Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignment
Soumya Suvra Ghosal, Souradip Chakraborty, Vaibhav Singh, Tianrui Guan, Mengdi Wang, Ahmad Beirami, Furong Huang, Alvaro Velasquez, Dinesh Manocha, Amrit Singh Bedi
Jailbreak Attack Multimodal LLM Human SAFETY Inference Time Temporal Alignment Safety Alignment Jailbreak Evaluation

November 15, 2024

An exploration of the effect of quantisation on energy consumption and inference time of StarCoder2
Pepijn de Reus, Ana Oprescu, Jelle Zuidema
Large Language Model Mixed Effect Linear Compression Energy Consumption Inference Time Heterogeneous Quantization

October 29, 2024

Scaling LLM Inference with Optimized Sample Compute Allocation
Kexun Zhang, Shang Zhou, Danqing Wang, William Yang Wang, Lei Li
Language Model LLM Inference Inference Time Inference System

October 28, 2024

Flaming-hot Initiation with Regular Execution Sampling for Large Language Models
Weizhe Chen, Zhicheng Zhang, Guanlin Liu, Renjie Zheng, Wenlei Shi, Chen Dun, Zheng Wu, Xing Jin, Lin Yan
Large Language Model Language Model Reasoning Task Inference Time Parallel Sampling Program Sampling

October 26, 2024

Fast Best-of-N Decoding via Speculative Rejection
Hanshi Sun, Momin Haider, Ruiqi Zhang, Huitao Yang, Jiahao Qiu, Ming Yin, Mengdi Wang, Peter Bartlett, Andrea Zanette
Inference Time Temporal Alignment Alignment Approach Risky Prompt Rejection Alignment Algorithm Best of N

October 18, 2024

MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time
Mozhi Zhang, Pengyu Wang, Chenkun Tan, Mianqiu Huang, Dong Zhang, Yaqian Zhou, Xipeng Qiu
Language Model Direct Preference Optimization Inference Time Large Language Model Alignment Alignment Dataset Interest Preference

October 13, 2024

Improving 3D Few-Shot Segmentation with Inference-Time Pseudo-Labeling
Mohammad Mozafari, Hosein Hasani, Reza Vahidimajd, Mohamadreza Fereydooni, Mahdieh Soleymani Baghshah
Segmentation Mask Shot Segmentation Inference Time Shot 3D Final Segmentation

October 9, 2024

ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time
Yi Ding, Bolian Li, Ruqi Zhang
Vision Language Model Adversarial Input Inference Time Multimodal AI Temporal Alignment Safety Alignment Modal Attack Estimated Time of Arrival

October 3, 2024

Adaptive Inference-Time Compute: LLMs Can Predict if They Can Do Better, Even Mid-Generation
Rohin Manvi, Anikait Singh, Stefano Ermon
Large Language Model Inference Time Adversarial Reward Best of N

September 23, 2024

Archon: An Architecture Search Framework for Inference-Time Techniques
Jon Saad-Falcon, Adrian Gamarra Lafuente, Shlok Natarajan, Nahum Maru, Hristo Todorov, E. Kelly Buchanan, Mayee Chen, Neel Guha, Christopher Ré, Azalia Mirhoseini
Reasoning Benchmark Inference Time Architecture Search

September 21, 2024

Multiple-Exit Tuning: Towards Inference-Efficient Adaptation for Vision Transformer
Zheng Liu, Jinchao Zhu, Nannan Li, Gao Huang
Vision Transformer Pre Trained Vision Transformer Inference Time Parameter Efficient Transfer

August 20, 2024

Security Assessment of Hierarchical Federated Deep Learning
D Alqattan, R Sun, H Liang, G Nicosia, V Snasel, R Ranjan, V Ojha
Adversarial Attack Adversarial Training Inference Time Hierarchical Federated Learning Training Time Attack Security Evaluation

August 7, 2024

NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time
Yilong Chen, Guoxia Wang, Junyuan Shang, Shiyao Cui, Zhenyu Zhang, Tingwen Liu, Shuohuan Wang, Yu Sun, Dianhai Yu, Hua Wu
Large Language Model Long Context General Text Inference Time KV Cache Salt Detection Text Task

August 6, 2024

Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
Charlie Snell, Jaehoon Lee, Kelvin Xu, Aviral Kumar
Inference Time Model Scaling

Inference Time

Papers

HoneypotNet: Backdoor Attacks Against Model Extraction

Efficiently Serving LLM Reasoning Programs with Certaindex

Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models

Dipper: Diversity in Prompts for Producing Large Language Model Ensembles in Reasoning tasks

HARP: A challenging human-annotated math reasoning benchmark

Scaling Inference-Time Search with Vision Value Model for Improved Visual Comprehension

Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignment

An exploration of the effect of quantisation on energy consumption and inference time of StarCoder2

Scaling LLM Inference with Optimized Sample Compute Allocation

Flaming-hot Initiation with Regular Execution Sampling for Large Language Models

Fast Best-of-N Decoding via Speculative Rejection

MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time

Improving 3D Few-Shot Segmentation with Inference-Time Pseudo-Labeling

ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time

Adaptive Inference-Time Compute: LLMs Can Predict if They Can Do Better, Even Mid-Generation

Archon: An Architecture Search Framework for Inference-Time Techniques

Multiple-Exit Tuning: Towards Inference-Efficient Adaptation for Vision Transformer

Security Assessment of Hierarchical Federated Deep Learning

NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time

Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters