Evaluation Data

Evaluation data is crucial for assessing the performance and fairness of machine learning models, particularly large language models (LLMs). Current research emphasizes developing robust and reliable evaluation datasets that address issues like data contamination, small sample sizes in subgroup analyses, and the need for temporally consistent benchmarks across diverse tasks (e.g., text-to-image generation, grammatical error correction). This involves creating new datasets, refining existing evaluation metrics, and employing techniques like structured regression to improve accuracy, especially for underrepresented groups. Improved evaluation methodologies are vital for advancing model development and ensuring responsible AI deployment across various applications.

Papers

November 1, 2024

Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations
Evan Miller
Language Model Global Evaluation Performance Evaluation Model Evaluation Ensemble Prediction Statistical Approach Evaluation Data

June 5, 2024

HelloFresh: LLM Evaluations on Streams of Real-World Human Editorial Actions across X Community Notes and Wikipedia edits
Tim Franzmeyer, Aleksandar Shtedritski, Samuel Albanie, Philip Torr, João F. Henriques, Jakob N. Foerster
New Benchmark Human Editing Wikipedia Article LLM Benchmark LLM Evaluation Evaluation Data

January 26, 2024

A structured regression approach for evaluating model performance across intersectional subgroups
Christine Herlihy, Kimberly Truong, Alexandra Chouldechova, Miroslav Dudik
Model Performance AI Fairness Evaluation Datasets Structured Regression Evaluation Data Intersectional Group

August 6, 2023

Spanish Pre-trained BERT Model and Evaluation Data
José Cañete, Gabriel Chaperon, Rodrigo Fuentes, Jou-Hui Ho, Hojin Kang, Jorge Pérez
Language Model Multilingual Corpus Pre Trained BERT Spanish Language Model Spanish Dictionary Evaluation Data

January 9, 2023

PatentsView-Evaluation: Evaluation Datasets and Tools to Advance Research on Inventor Name Disambiguation
Olivier Binette, Sarvo Madhavan, Jack Butler, Beth Anne Card, Emily Melluso, Christina Jones
DH Research Easy Tool Patent Text Evaluation Datasets Name Disambiguation Evaluation Data

December 15, 2022

TeTIm-Eval: a novel curated evaluation data set for comparing text-to-image models
Federico A. Galatolo, Mario G. C. A. Cimino, Edoardo Cogotti
Text to Image Model Human Evaluation Long Form Novel Novel Evaluation G Eval CLIP Score Evaluation Data

April 30, 2022

A New Evaluation Method: Evaluation Data and Metrics for Chinese Grammar Error Correction
Nankai Lin, Nankai Lin, Xiaotian Lin, Ziyu Yang, Shengyi Jiang
Sentence Level Metric Library Novel Evaluation Word Segmentation Chinese Grammatical Error Correction Chinese Word Segmentation Evaluation Data

March 7, 2022

ILDAE: Instance-Level Difficulty Analysis of Evaluation Data
Neeraj Varshney, Swaroop Mishra, Chitta Baral
Evaluation Method Evaluation Datasets Instance Hardness Evaluation Data

Evaluation Data

Papers

Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations

HelloFresh: LLM Evaluations on Streams of Real-World Human Editorial Actions across X Community Notes and Wikipedia edits

A structured regression approach for evaluating model performance across intersectional subgroups

Spanish Pre-trained BERT Model and Evaluation Data

PatentsView-Evaluation: Evaluation Datasets and Tools to Advance Research on Inventor Name Disambiguation

TeTIm-Eval: a novel curated evaluation data set for comparing text-to-image models

A New Evaluation Method: Evaluation Data and Metrics for Chinese Grammar Error Correction

ILDAE: Instance-Level Difficulty Analysis of Evaluation Data