Inference Time

Inference time, the time taken for a model to process an input and produce an output, is a critical factor in the performance and scalability of large language models (LLMs) and other deep learning systems. Current research focuses on optimizing inference efficiency through techniques like adaptive sampling, architecture search for efficient inference-time techniques, and model compression methods, aiming to reduce computational costs without sacrificing accuracy. These advancements are crucial for deploying LLMs in resource-constrained environments and improving the responsiveness of AI applications, impacting both the efficiency of AI systems and their accessibility to a wider range of users.

Papers

October 26, 2023

Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time
Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, Beidi Chen
Large Language Model LLM Inference Inference Latency Inference Time Dense Model Contextual Sparsity Deja Vu

October 24, 2023

Integrating Language Models into Direct Speech Translation: An Inference-Time Solution to Control Gender Inflection
Dennis Fucci, Marco Gaido, Sara Papi, Mauro Cettolo, Matteo Negri, Luisa Bentivogli
Machine Translation Speech Translation Gender Information Inference Time Tuned Lm Direct Speech to Speech Translation

September 26, 2023

Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decoding
Jiacheng Liu, Andrew Cohen, Ramakanth Pasunuru, Yejin Choi, Hannaneh Hajishirzi, Asli Celikyilmaz
Language Model Proximal Policy Optimization Monte Carlo Tree Search Inference Time Valuation Model State of the Art Reinforcement Hybrid Text

August 3, 2023

Quantification of Predictive Uncertainty via Inference-Time Sampling
Katarína Tóthová, Ľubor Ladický, Daniel Thul, Marc Pollefeys, Ender Konukoglu
Multi Modal Uncertainty Estimation Predictive Uncertainty Inference Time Ambiguous Data

August 2, 2023

Isolation and Induction: Training Robust Deep Neural Networks against Model Stealing Attacks
Jun Guo, Aishan Liu, Xingyu Zheng, Siyuan Liang, Yisong Xiao, Yichao Wu, Xianglong Liu
Robust Deep Inference Time Model Stealing Attack Social Isolation Automatic Induction Defense Model

July 5, 2023

NMS Threshold matters for Ego4D Moment Queries -- 2nd place solution to the Ego4D Moment Queries Challenge 2023
Lin Sui, Fangzhou Mu, Yin Li
Temporal Action Localization Place Solution Inference Time Canonical Threshold Ego4D Natural Language Query

June 13, 2023

Domain Information Control at Inference Time for Acoustic Scene Classification
Shahed Masoudian, Khaled Koutini, Markus Schedl, Gerhard Widmer, Navid Rekabsaz
Domain Shift Inference Time Acoustic Scene Classification Domain Information

May 27, 2023

Data Minimization at Inference Time
Cuong Tran, Ferdinando Fioretto
Scientific Inference Sensitive Data Inference Time Input Feature Data Minimization Information Accuracy

January 10, 2023

Structured Case-based Reasoning for Inference-time Adaptation of Text-to-SQL parsers
Abhijeet Awasthi, Soumen Chakrabarti, Sunita Sarawagi
Semantic Parsing Text to SQL Inference Time Case Based Reasoning Text to SQL Parser

December 25, 2022

Quality at the Tail of Machine Learning Inference
Zhengxin Yang, Wanling Gao, Chunjie Luo, Lei Wang, Fei Tang, Xu Wen, Jianfeng Zhan
Quality Issue Inference Time Exact Inference Machine Learning Inference Real Time Inference Thin Tail

December 8, 2022

Vicious Classifiers: Assessing Inference-time Data Reconstruction Risk in Edge Computing
Mohammad Malekzadeh, Deniz Gunduz
Scientific Inference Generative Adversarial Inference Time Edge Inference Data Reconstruction Attack Data Reconstruction Target Inference

November 28, 2022

Object Permanence in Object Detection Leveraging Temporal Priors at Inference Time
Michael Fürst, Priyash Bhugra, René Schuster, Didier Stricker
Particle Filter Inference Time Shi Detector Temporal Prior Object Permanence Two Stage Detector

October 10, 2022

Unified Detoxifying and Debiasing in Language Generation via Inference-time Adaptive Optimization
Zonghan Yang, Xiaoyuan Yi, Peng Li, Yang Liu, Xing Xie
Pre Trained Language Model Language Generation Self Debiasing Inference Time Text Distribution Fine Grained Detoxification Ethical NaTural Language

August 1, 2022

On the Evaluation of User Privacy in Deep Neural Networks using Timing Side Channel
Shubhi Shukla, Manaar Alam, Sarani Bhattacharya, Debdeep Mukhopadhyay, Pabitra Mitra
Deep Neural Network Global Evaluation Membership Inference Attack Inference Time Side Channel User Privacy Recent Deep Learning

June 30, 2022

ZeroC: A Neuro-Symbolic Model for Zero-shot Concept Recognition and Acquisition at Inference Time
Tailin Wu, Megan Tjandrasuwita, Zhengxuan Wu, Xuelin Yang, Kevin Liu, Rok Sosič, Jure Leskovec
Zero Shot Visual Concept Inference Time Channel Acquisition Concept Based Model Neuro Symbolic Architecture

June 1, 2022

IDANI: Inference-time Domain Adaptation via Neuron-level Interventions
Omer Antverg, Eyal Ben-David, Yonatan Belinkov
Domain Adaptation Large Pre Trained Model Unseen Domain Inference Time Unseen Data Neuron Level

May 27, 2022

Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval
Pascal Notin, Mafalda Dias, Jonathan Frazer, Javier Marchena-Hurtado, Aidan Gomez, Debora S. Marks, Yarin Gal
Protein Sequence Inference Time Autoregressive Transformer Automatic Alignment Multiple Sequence Alignment Protein Stability

March 16, 2022

Memorizing Transformers
Yuhuai Wu, Markus N. Rabe, DeLesley Hutchins, Christian Szegedy
Language Model Memory Trace Internal Representation Inference Time Differentiable Activation Function

March 10, 2022

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, Ludwig Schmidt
Fine Tuning Large Pre Trained Model Fine Tuned Model Balancing Weight Inference Time Model Soup

December 15, 2021

Mask-combine Decoding and Classification Approach for Punctuation Prediction with real-time Inference Constraints
Christoph Minixhofer, Ondřej Klejch, Peter Bell
Classification Algorithm Inference Time Explicit in Document Tagging Punctuation Restoration Real Time Inference Multiple Prediction Mask Decoder

Inference Time

Papers

Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time

Integrating Language Models into Direct Speech Translation: An Inference-Time Solution to Control Gender Inflection

Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decoding

Quantification of Predictive Uncertainty via Inference-Time Sampling

Isolation and Induction: Training Robust Deep Neural Networks against Model Stealing Attacks

NMS Threshold matters for Ego4D Moment Queries -- 2nd place solution to the Ego4D Moment Queries Challenge 2023

Domain Information Control at Inference Time for Acoustic Scene Classification

Data Minimization at Inference Time

Structured Case-based Reasoning for Inference-time Adaptation of Text-to-SQL parsers

Quality at the Tail of Machine Learning Inference

Vicious Classifiers: Assessing Inference-time Data Reconstruction Risk in Edge Computing

Object Permanence in Object Detection Leveraging Temporal Priors at Inference Time

Unified Detoxifying and Debiasing in Language Generation via Inference-time Adaptive Optimization

On the Evaluation of User Privacy in Deep Neural Networks using Timing Side Channel

ZeroC: A Neuro-Symbolic Model for Zero-shot Concept Recognition and Acquisition at Inference Time

IDANI: Inference-time Domain Adaptation via Neuron-level Interventions

Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval

Memorizing Transformers

Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

Mask-combine Decoding and Classification Approach for Punctuation Prediction with real-time Inference Constraints