Automated Evaluation

Follow this topic Ask about this topic

Momentum

48 papers in the last four weeks, up 12% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 321

All topics

FAGER: Factually Grounded Evaluation and Refinement of Text-to-Image Models
May 18, 2026Youngsun Lim, Cusuh Ham, Pin-Yu Chen +1Factual Consistency Evaluation Automated Evaluation
Boston University · Adobe · IBM Research
SKG-Eval: Stateful Evaluation of Multi-Turn Dialogue via Incremental Semantic Knowledge Graphs
May 15, 2026Avijit Shil, Suman SamuiLLM Evaluation Factual Consistency Evaluation
Maulana Abul Kalam Azad University of Technology, West Bengal, India · National Institute of Technology Durgapur, West Bengal, India
LEAP: Trajectory-Level Evaluation of LLMs in Iterative Scientific Design
May 14, 2026Marilyn Zhang, Tianfeng Chen, Fabián Barzuna +2LLM Evaluation AI Agent Benchmarks
Editor's Choice: Evaluating Abstract Intent in Image Editing through Atomic Entity Analysis
May 14, 2026Mor Ventura, Roy Hirsch, Yonatan Bitton +2Image Editing Evaluation Text-Guided Image Editing
1Technion – Israel Institute of Technology · 2Google Research · *Work done during an internship at Google Research.
In-IDE Toolkit for Developers of AI-Based Features
May 14, 2026Yaroslav Sokolov, Yury Khudyakov, Lenar Sharipov +3AI Agent Monitoring Automated Evaluation
JetBrains Berlin, Germany · JetBrains Munich, Germany · JetBrains Amsterdam, Netherlands +1
Deepchecks: Evaluating Retrieval-Augmented Generation (RAG)
May 14, 2026Assaf Gerner, Netta Madvil, Nadav Barak +11LLM Evaluation RAG Evaluation
Deepchecks, Ramat Gan, Israel · Ben-Gurion University, Beer Sheva, Israel
Creativity Bias: How Machine Evaluation Struggles with Creativity in Literary Translations
May 13, 2026Kyo Gerrits, Rik van Noord, Ana Guerberof ArenasCreativity Assessment LLM-as-a-Judge
Centre for Language and Cognition, University of Groningen
Skill-Aligned Annotation for Reliable Evaluation in Text-to-Image Generation
May 13, 2026Abdelrahman Eldesokey, Merey Ramazanova, Ahmad Sait +4Human-in-the-Loop Evaluation Automated Evaluation
King Abdullah University of Science and Technology (KAUST) · Thuwal, Saudi Arabia
A Comparative Study of Controlled Text Generation Systems Using Level-Playing-Field Evaluation Principles
May 12, 2026Michela Lorandi, Anya BelzControllable Text Generation Automated Evaluation
Dublin City University, Dublin, Ireland
DiffScore: Text Evaluation Beyond Autoregressive Likelihood
May 12, 2026Wen Lai, Yingli Shen, Dingnan Jin +4LLM Evaluation Language Model Generation Evaluation
1Ant Group · 2Tsinghua University · 3Technical University of Munich
An Empirical Study of Automating Agent Evaluation
May 12, 2026Kang Zhou, Sangmin Woo, Haibo Ding +15Agent Evaluation AI Agent Benchmarks
AWS AI Labs
VISOR: A Vision-Language Model-based Test Oracle for Testing Robots
May 11, 2026Prasun Saurabh, Pablo Valle, Aitor Arrieta +2Uncertainty Estimation for VLA Models Automated Software Testing
Simula Research Laboratory and Oslo Metropolitan University · Mondragon University · National Institute of Informatics
Text-to-CAD Evaluation with CADTests
May 8, 2026Dimitrios Mallis, Marco Wang, Ahmet Serdar Karadeniz +3Text-to-CAD Generation Automated Evaluation
SnT, University of Luxembourg · University of Trento · Fondazione Bruno Kessler
PolySQL: Scaling Text-to-SQL Evaluation Across SQL Dialects via Automated Backend Isomorphism
May 8, 2026Yotam Perlitz, Elad Venezian, Corentin Royer +2Benchmark Design Text-to-SQL
1IBM Research AI · 2ETH Zürich
Multi-Dimensional Evaluation of LLMs for Grammatical Error Correction
May 8, 2026Adnan Labib, Qiao Wang, Yixuan Huang +1LLM Evaluation Grammatical Error Correction
King’s College London, London, United Kingdom · Hosei University, Tokyo, Japan · Waseda University, Tokyo, Japan +1
SAGE: Hierarchical LLM-Based Literary Evaluation through Ontology-Grounded Interpretive Dimensions
May 8, 2026Tianyu Wang, Nianjun ZhouComputational Literary Studies LLM Evaluation
Mercy University, Math & Computer Science Department · Dobbs Ferry, NY, USA · IBM T.J. Watson Research Center +1
Beyond Single Ground Truth: Reference Monism as Epistemic Injustice in ASR Evaluation
May 8, 2026Anna Seo Gyeong Choi, Maria Teleki, James Caverlee +3ASR Evaluation Automatic Speech Recognition
Department of Information Science, Cornell University · Department of Computer Science, Texas A&M University · Rev AI +2
DynT2I-Eval: A Dynamic Evaluation Framework for Text-to-Image Models
May 7, 2026Juntong Wang, Jiarui Wang, Huiyu Duan +3Learning to Rank Benchmark Design
Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University, Shanghai, China
DoGMaTiQ: Automated Generation of Question-and-Answer Nuggets for Report Evaluation
May 6, 2026Bryan Li, William Walden, Yu Hou +6Language Model Generation Evaluation RAG Evaluation
Google Inc. · University of Pennsylvania · Johns Hopkins University +3
FlowEval: Reference-based Evaluation of Generated User Interfaces
May 5, 2026Jason Wu, Priyan Vaithilingam, Eldon Schoop +2Web Application Generation Automated Evaluation
Purdue University · Apple
A Paradigm for Interpreting Metrics and Identifying Critical Errors in Automatic Speech Recognition
May 5, 2026Thibault Bañeras-Roux, Mickael Rouvier, Jane Wottawa +1ASR Evaluation Automatic Speech Recognition
Nantes University, LS2N, France
Runtime Evaluation of Procedural Content Generation in an Endless Runner Game Using Autonomous Agents
May 3, 2026Rishabh KarGenerative Modeling Procedural Content Generation
Department of Informatics King’s College London London, United Kingdom
Claw-Eval-Live: A Live Agent Benchmark for Evolving Real-World Workflows
Apr 30, 2026Chenxin Li, Zhengyang Tang, Mingxin Huang +8LLM Agent Evaluation AI Agent Benchmarks
1The Chinese University of Hong Kong · 2The Chinese University of Hong Kong, Shenzhen · 3South China University of Technology +4
Agent-Agnostic Evaluation of SQL Accuracy in Production Text-to-SQL Systems
Apr 30, 2026Taslim Jamal Arif, Kuldeep SinghText-to-SQL Automated Evaluation
Measurement Risk in Supervised Financial NLP: Rubric and Metric Sensitivity on JF-ICR
Apr 30, 2026Sidi Chang, Peiying Zhu, Yuxiao Chen +1LLM Evaluation Rubric-Based Evaluation
Blossom AI Labs, Tokyo, Japan · Blossom AI Labs, San Francisco, USA
SciHorizon-DataEVA: An Agentic System for AI-Readiness Evaluation of Heterogeneous Scientific Data
Apr 29, 2026Dianyu Liu, Chuan Qin, Xi Chen +6AI for Science Automated Evaluation
Computer Network Information Center, Chinese Academy of Sciences Beijing, China
Training Computer Use Agents to Assess the Usability of Graphical User Interfaces
Apr 28, 2026Alice Gao, Weixi Tong, Rishab Vempati +4Computer-Use Agents GUI Agents
University of Washington Seattle, WA, USA · Purdue University West Lafayette, IN, USA
LLM-ReSum: A Framework for LLM Reflective Summarization through Self-Evaluation
Apr 28, 2026Huyen Nguyen, Haoxuan Zhang, Yang Zhang +2LLM Evaluation LLM Self-Refinement
Cigna Group Evernorth Health Services Austin, Texas, USA · dept. of Information Science University of North Texas Denton, Texas, USA · dept. of Data Science University of North Texas Denton, Texas, USA
SciEval: A Benchmark for Automatic Evaluation of K-12 Science Instructional Materials
Apr 28, 2026Zhaohui Li, Peng He, Zhiyuan Chen +4LLM Evaluation AI in Education
Department of Computer Science and Engineering, University at Buffalo, USA · Department of Teaching and Learning, Washington State University, USA · College of Chemistry, Beijing Normal University, China +1
The Structured Output Benchmark: A Multi-Source Benchmark for Evaluating Structured Output Quality in Large Language Models
Apr 28, 2026Abhinav Kumar Singh, Harsha Vardhan Khurdula, Yoeven D Khemlani +1LLM Evaluation LLM Information Extraction
1New Delhi, India · JigsawStack, Inc. 2San Francisco, CA, USA · 3Durgapur, WB, India

May 18, 2026, cs.CVJ/K move · Enter open · S save

FAGER: Factually Grounded Evaluation and Refinement of Text-to-Image Models

Youngsun Lim, Cusuh Ham, Pin-Yu Chen, Deepti Ghadiyaram

Boston University · Adobe · IBM Research

Factual Consistency Evaluation Automated Evaluation T2I Generation Evaluation

Open paper AskPDF arXiv

Existing text-to-image (T2I) evaluation metrics mainly assess whether generated images align with information explicitly stated in the prompt, but often fail to capture factual requirements that are implicit, externally grounded, or identity-defining. As a result, they are not well suited for evaluating factual correctness in prompts involving scientific knowledge, historical facts, products, or culture-specific concepts. We propose FActually Grounded Evaluation and Refinement (FAGER), an agentic framework that evaluates whether generated images correctly reflect visually verifiable facts grounded in or implied by the prompt, while also providing actionable feedback for improvement. FAGER first constructs a structured factual rubric by combining LLM-based fact proposal with reference-guided visual fact extraction and verification, then converts the rubric into question-answer pairs for VLM-based evaluation. To validate FAGER as a factuality metric, we introduce a Factual A/B test, which measures whether a metric prefers factual reference images over corresponding generated images. Across five datasets spanning science, history, products, culture, and knowledge-intensive concepts, FAGER consistently outperforms prior metrics on this test. We further show that FAGER can be used to refine T2I outputs in a fully training-free manner, yielding substantial factuality gains across datasets.

Momentum

Related topics

Latest papers 321