Selective Prediction

Selective prediction focuses on enhancing the reliability of machine learning models by enabling them to abstain from making predictions when their confidence is low or when the input data is deemed unreliable. Current research emphasizes developing robust confidence estimation methods, often employing techniques like Monte Carlo dropout or KL divergence, and exploring how model architecture, training data, and post-processing techniques influence prediction reliability across various tasks (e.g., classification, regression, and question answering). This field is crucial for deploying machine learning models in high-stakes applications where errors are costly, improving overall system trustworthiness and reducing reliance on human intervention for error correction.

Papers

October 31, 2024

Joint Training for Selective Prediction
Zhaohui Li, Rebecca J. Passonneau
Natural Language Processing Joint Learning Model Confidence Joint Training Selective Prediction

September 27, 2024

The Craft of Selective Prediction: Towards Reliable Case Outcome Classification -- An Empirical Study on European Court of Human Rights Cases
T.Y.S.S. Santosh, Irtiza Chowdhury, Shanshan Xu, Matthias Grabmair
Empirical Study Confidence Estimation Human Right Prediction Confidence Selective Prediction Confidence Measure Legal NLP

September 11, 2024

A Practical Theory of Generalization in Selectivity Learning
Peizhi Wu, Haoshu Xu, Ryan Marcus, Zachary G. Ives
Strong Generalization Generalization Error Bound Selective Engagement Query Learning Selective Prediction Selective Learning

July 27, 2024

Inference-Time Selective Debiasing
Gleb Kuzmin, Nemeesh Yadav, Ivan Smirnov, Timothy Baldwin, Artem Shelmanov
Debiasing Method Data Debiasing Selective Prediction Selective Inference

April 16, 2024

Consistency and Uncertainty: Identifying Unreliable Responses From Black-Box Vision-Language Models for Selective Visual Question Answering
Zaid Khan, Yun Fu
Vision Language Model High Uncertainty Anticipation Strong Consistency Selective Prediction Reliable Prediction Inconsistent Response

February 23, 2024

Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoning
Tejas Srinivasan, Jack Hessel, Tanmay Gupta, Bill Yuchen Lin, Yejin Choi, Jesse Thomason, Khyathi Raghavi Chandu
Vision Language Model Vision Language Selective Engagement Low Confidence Abstention Ability Selective Prediction Vision Language Reasoning

February 16, 2024

Selective Prediction for Semantic Segmentation using Post-Hoc Confidence Estimation and Its Performance under Distribution Shift
Bruno Laboissiere Camargos Borges, Bruno Machado Pacheco, Danilo Silva
Semantic Segmentation Pre Trained Model System Performance Distribution Shift Confidence Aware Selective Prediction

November 15, 2023

Model Agnostic Explainable Selective Regression via Uncertainty Estimation
Andrea Pugnana, Carlos Mougan, Dan Saattrup Nielsen
Uncertainty Estimation Selective Prediction Agnostic Model Selective Regression

October 18, 2023

Adaptation with Self-Evaluation to Improve Selective Prediction in LLMs
Jiefeng Chen, Jinsung Yoon, Sayna Ebrahimi, Sercan O Arik, Tomas Pfister, Somesh Jha
Large Language Model Language Understanding Adaptation Concern Self Feedback Selective Prediction

September 28, 2023

Selective Nonparametric Regression via Testing
Fedor Noskov, Alexander Fishkov, Maxim Panov
Novel Regression Level Test Nonparametric Regression Selective Prediction Selective Regression

June 14, 2023

Improving Selective Visual Question Answering by Learning from Your Peers
Corentin Dancette, Spencer Whitehead, Rishabh Maheshwary, Ramakrishna Vedantam, Stefan Scherer, Xinlei Chen, Matthieu Cord, Marcus Rohrbach
Visual Question Answering Peer Agent Selective Prediction

May 28, 2023

Training Private Models That Know What They Don't Know
Stephan Rabanser, Anvith Thudi, Abhradeep Thakurta, Krishnamurthy Dvijotham, Nicolas Papernot
Differential Privacy Selective Prediction Non Private Differential Privacy Constraint Private Learning Algorithm

May 2, 2023

Post-Abstention: Towards Reliably Re-Attempting the Abstained Instances in QA
Neeraj Varshney, Chitta Baral
Natural Language Processing Yes No Question Abstention Ability Unreliable Prediction Selective Prediction

April 7, 2023

ASPEST: Bridging the Gap Between Active Learning and Selective Prediction
Jiefeng Chen, Jinsung Yoon, Sayna Ebrahimi, Sercan Arik, Somesh Jha, Tomas Pfister
Machine Learning Active Learning Selective Prediction New Learning

February 23, 2023

What Can We Learn From The Selective Prediction And Uncertainty Estimation Performance Of 523 Imagenet Classifiers
Ido Galil, Mohammed Dabbah, Ran El-Yaniv
Uncertainty Estimation Uncertainty Estimation Method Selective Engagement Selective Prediction Better Uncertainty Estimation ImageNet Classifier Uncertainty Estimation Performance

September 30, 2022

Out-of-Distribution Detection and Selective Generation for Conditional Language Models
Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanan, Peter J. Liu
Distribution Detection Generative Language Model Abstractive Summarization Generation Task Selective Prediction Conditional Language Model

September 16, 2022

On the Relation between Sensitivity and Accuracy in In-context Learning
Yanda Chen, Chen Zhao, Zhou Yu, Kathleen McKeown, He He
Context Learning Network Sensitivity Relational Information Label Bias Selective Prediction Private Prediction Entropy Function

June 17, 2022

Towards Better Selective Classification
Leo Feng, Mohamed Osama Ahmed, Hossein Hajimirsadeghi, Amir Abdi
Entropy Regularized Learning Generalizable Selective Classification Selective Prediction

March 1, 2022

Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settings
Neeraj Varshney, Swaroop Mishra, Chitta Baral
NLP Task Adversarial Setting Unknown Out of Domain Selective Prediction NLI Datasets

December 13, 2021

Role of Human-AI Interaction in Selective Prediction
Elizabeth Bondi, Raphael Koster, Hannah Sheahan, Martin Chadwick, Yoram Bachrach, Taylan Cemgil, Ulrich Paquet, Krishnamurthy Dvijotham
Integral Role Human AI Interaction Selective Prediction