Human Uncertainty

Human uncertainty, inherent in perception and judgment, significantly impacts the reliability and effectiveness of AI systems, particularly in evaluating model performance and human-in-the-loop applications. Current research focuses on quantifying and incorporating this uncertainty, employing methods like Bayesian neural networks and hierarchical reinforcement learning to improve model robustness and decision-making under uncertainty. Addressing human uncertainty is crucial for developing trustworthy AI systems, improving the accuracy of automated evaluations, and enhancing the safety and reliability of AI in safety-critical domains.

Papers

October 3, 2024

Beyond correlation: The impact of human uncertainty in measuring the effectiveness of automatic evaluation and LLM-as-a-judge
Aparna Elangovan, Jongwoo Ko, Lei Xu, Mahsa Elyasi, Ling Liu, Sravan Bodapati, Dan Roth
Total Correlation Automatic Evaluation LLM a a Judge Label Uncertainty Human Uncertainty

June 12, 2024

Hierarchical Reinforcement Learning for Swarm Confrontation with High Uncertainty
Qizhen Wu, Kexin Liu, Lei Chen, Jinhu Lü
Deep Reinforcement Learning Hierarchical Reinforcement Learning Swarm Robotics Pursuit Evasion Game Swarm Confrontation Human Uncertainty

April 24, 2024

Automatic AI controller that can drive with confidence: steering vehicle with uncertainty knowledge
Neha Kumari, Sumit Kumar. Sneha Priya, Ayush Kumar, Akash Fogla
Uncertainty Quantification Bayesian Neural Network Cyber Physical System Vehicle Plan High Confidence Prediction Confidence Intelligent Controller Human Uncertainty

March 22, 2023

Human Uncertainty in Concept-Based AI Systems
Katherine M. Collins, Matthew Barker, Mateo Espinosa Zarlenga, Naveen Raman, Umang Bhatt, Mateja Jamnik, Ilia Sucholutsky, Adrian Weller, Krishnamurthy Dvijotham
Human AI Interaction Interpretable Concept Human Uncertainty

December 13, 2022

Multi-Target Decision Making under Conditions of Severe Uncertainty
Christoph Jansen, Georg Schollmeyer, Thomas Augustin
Decision Making Pareto Optimal Adverse Condition Multi Criterion Decision Decision Theory Probabilistic Information Human Uncertainty

May 30, 2022

Going Beyond One-Hot Encoding in Classification: Can Human Uncertainty Improve Model Performance?
Christoph Koller, Göran Kauermann, Xiao Xiang Zhu
Deep Learning Classification Code Model Performance Label Distribution Label Uncertainty Labeling Technique One Hot Encoding Human Uncertainty

Human Uncertainty

Papers

Beyond correlation: The impact of human uncertainty in measuring the effectiveness of automatic evaluation and LLM-as-a-judge

Hierarchical Reinforcement Learning for Swarm Confrontation with High Uncertainty

Automatic AI controller that can drive with confidence: steering vehicle with uncertainty knowledge

Human Uncertainty in Concept-Based AI Systems

Multi-Target Decision Making under Conditions of Severe Uncertainty

Going Beyond One-Hot Encoding in Classification: Can Human Uncertainty Improve Model Performance?