Generative Evaluation

Generative evaluation focuses on assessing the quality and capabilities of generative models, particularly large language models (LLMs) and image generation models, by analyzing their outputs. Current research emphasizes developing robust evaluation metrics that go beyond simple comparisons to human-generated content, addressing issues like bias in automated metrics and the need for methods sensitive to nuances in generated text and images (e.g., mode collapse, pragmatic understanding, and compositional fidelity). These efforts are crucial for improving the reliability and trustworthiness of generative models across diverse applications, ranging from educational tools to medical image synthesis, by providing more accurate and insightful assessments of model performance.

Papers

October 29, 2024

SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Types
Yutao Mou, Shikun Zhang, Wei Ye
Large Language Model MT Bench Open Source LLM Novel Benchmark Discriminative Task Diverse Task Generative Evaluation

May 14, 2024

Is the Pope Catholic? Yes, the Pope is Catholic. Generative Evaluation of Non-Literal Intent Resolution in LLMs
Akhila Yerukola, Saujas Vaduguru, Daniel Fried, Maarten Sap
Intent Prediction Communicative Intent Generative Evaluation Effective Pragmatic Interlocutor

November 28, 2023

Dendrogram distance: an evaluation metric for generative networks using hierarchical clustering
Gustavo Sutter Carvalho, Moacir Antonelli Ponti
Evaluation Metric Hierarchical Clustering Generative Network Novel Metric Real Datasets Generative Evaluation Dendrogram Construction

November 16, 2023

LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scores
Yiqi Liu, Nafise Sadat Moosavi, Chenghua Lin
Medical LLM Evaluation Metric Modern Language Model LLM Based Metric Generative Evaluation

August 7, 2023

Automated Distractor and Feedback Generation for Math Multiple-choice Questions via In-context Learning
Hunter McNichols, Wanyong Feng, Jaewook Lee, Alexander Scarlatos, Digory Smith, Simon Woodhead, Andrew Lan
Context Learning Multiple Choice Question High Quality Distractors Feedback Generation Generative Evaluation

June 7, 2023

Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models
George Stein, Jesse C. Cresswell, Rasa Hosseinzadeh, Yi Sui, Brendan Leigh Ross, Valentin Villecroze, Zhaoyan Liu, Anthony L. Caterini, J. Eric T. Taylor, Gabriel Loaiza-Ganem
Diffusion Model Generative Model Supervised ImageNet Generative Evaluation

March 2, 2023

Counterfactual Edits for Generative Evaluation
Maria Lymperaiou, Giorgos Filandrianos, Konstantinos Thomas, Giorgos Stamou
Generative Model Story Visualization Scene Synthesis Generative Evaluation Counterfactual Editing

January 30, 2023

Curvature Filtrations for Graph Generative Model Evaluation
Joshua Southern, Jeremy Wayland, Michael Bronstein, Bastian Rieck
Generative Model Topological Data Analysis Hand Crafted Descriptor Graph Curvature Generative Evaluation Cellular Sublevel Filtration

December 12, 2022

T5Score: Discriminative Fine-tuning of Generative Evaluation Metrics
Yiwei Qin, Weizhe Yuan, Graham Neubig, Pengfei Liu
Distance Metric Raw Text Generative Evaluation Embedding Based Metric

September 28, 2022

medigan: a Python library of pretrained generative models for medical image synthesis
Richard Osuala, Grzegorz Skorupko, Noussair Lazrak, Lidia Garrucho, Eloy García, Smriti Joshi, Socayna Jouide, Michael Rutherford, Fred Prior, Kaisar Kushibar, Oliver Diaz, Karim Lekadir
Generative Model Generative Adversarial Network Python Library Medical Image Synthesis Generative Evaluation

Generative Evaluation

Papers

SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Types

Is the Pope Catholic? Yes, the Pope is Catholic. Generative Evaluation of Non-Literal Intent Resolution in LLMs

Dendrogram distance: an evaluation metric for generative networks using hierarchical clustering

LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scores

Automated Distractor and Feedback Generation for Math Multiple-choice Questions via In-context Learning

Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models

Counterfactual Edits for Generative Evaluation

Curvature Filtrations for Graph Generative Model Evaluation

T5Score: Discriminative Fine-tuning of Generative Evaluation Metrics

medigan: a Python library of pretrained generative models for medical image synthesis