Diverse Test

Diverse test methodologies for deep learning models and other complex systems aim to identify weaknesses and ensure robustness by evaluating performance across a wide range of inputs. Current research focuses on developing efficient techniques for generating diverse and informative test sets, including leveraging large language models to synthesize test cases and employing bio-inspired search algorithms to explore the input space effectively. These advancements are crucial for improving the reliability and safety of AI systems, particularly in safety-critical applications like autonomous driving and medical diagnostics, where thorough testing is paramount. The ultimate goal is to move beyond simple coverage metrics towards more effective methods that prioritize the discovery of actual failures.

Papers

October 8, 2024

Generalizing to any diverse distribution: uniformity, gentle finetuning and rebalancing
Andreas Loukas, Karolis Martinkus, Ed Wagstaff, Kyunghyun Cho
Domain Adaptation Strong Generalization Uniformity Metric Dynamic Rebalancing Distribution Classifier Diverse Test

June 12, 2024

DLLens: Testing Deep Learning Libraries via LLM-aided Synthesis
Meiziniu Li, Dongze Li, Jianmeng Liu, Jialun Cao, Yongqiang Tian, Shing-Chi Cheung
Differential Testing Diverse Test Deep Learning Library Testing

April 10, 2024

Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competition
Kehua Feng, Keyan Ding, Kede Ma, Zhihua Wang, Qiang Zhang, Huajun Chen
Large Language Model Human Evaluation Optimal Discrepancy Unbiased Evaluation Diverse Test

March 16, 2024

Enhancing Out-of-Distribution Detection with Multitesting-based Layer-wise Feature Fusion
Jiawei Li, Sitong Li, Shanshan Wang, Yicheng Zeng, Falong Tan, Chuanlong Xie
Distribution Detection Feature Fusion Distribution Sample Ambiguous Input Diverse Test

May 25, 2023

Rethinking Diversity in Deep Neural Network Testing
Zi Wang, Jihye Choi, Ke Wang, Somesh Jha
Deep Neural Network Diversity Enhancement Diversity Metric Deep Learning Testing Diverse Test DNN Testing

May 17, 2022

Supervised Learning for Coverage-Directed Test Selection in Simulation-Based Verification
Nyasha Masamba, Kerstin Eder, Tim Blackmore
Supervised Learning Coverage Directed Test Diverse Test Based Verification

March 22, 2022

Machine Learning Testing in an ADAS Case Study Using Simulation-Integrated Bio-Inspired Search-Based Testing
Mahshid Helali Moghadam, Markus Borg, Mehrdad Saadatmand, Seyed Jalaleddin Mousavirad, Markus Bohlin, Björn Lisper
Machine Learning Cyber Physical System Bio Inspired Search Based Diverse Test Ada Case Study

December 20, 2021

Black-Box Testing of Deep Neural Networks Through Test Case Diversity
Zohreh Aghababaeyan, Manel Abdellatif, Lionel Briand, Ramesh S, Mojtaba Bagherzadeh
Deep Neural Network Diversity Metric Box Testing Neuron Coverage Diverse Test DNN Testing

Diverse Test

Papers

Generalizing to any diverse distribution: uniformity, gentle finetuning and rebalancing

DLLens: Testing Deep Learning Libraries via LLM-aided Synthesis

Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competition

Enhancing Out-of-Distribution Detection with Multitesting-based Layer-wise Feature Fusion

Rethinking Diversity in Deep Neural Network Testing

Supervised Learning for Coverage-Directed Test Selection in Simulation-Based Verification

Machine Learning Testing in an ADAS Case Study Using Simulation-Integrated Bio-Inspired Search-Based Testing

Black-Box Testing of Deep Neural Networks Through Test Case Diversity