Generative AI Evaluation

Follow this topic Ask about this topic

Momentum

29 papers in the last four weeks, up 142% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 211

All topics

When Reasoning Supervision Hurts: TTCW-Based Long-Form Literary Review Generation
May 19, 2026Jinlong Liu, Mohammed Bahja, Mark LeeSupervised Fine-Tuning Creativity Assessment
School of Computer Science, University of Birmingham, United Kingdom
How Far Are We From True Auto-Research?
May 18, 2026Zhengxin Zhang, Ning Wang, Sainyam Galhotra +1LLM Agent Evaluation AI Agent Benchmarks
Cornell University
QQJ: Quantifying Qualitative Judgment for Scalable and Human-Aligned Evaluation of Generative AI
May 17, 2026Marjan Veysi, Pirooz Shamsinejadbabaki, Mohammad Zare +1LLM-as-a-Judge Human-in-the-Loop Evaluation
AI Lab, Arioobarzan Engineering Team, Shiraz, Iran · Department of Computer Engineering and Information Technology, Shiraz University of Technology, Shiraz, Iran · Department of Informatics, Bioengineering, Robotics and Systems Engineering, University of Genoa, Genoa, Italy
UniPPTBench: A Unified Benchmark for Presentation Generation Across Diverse Input Settings
May 17, 2026Bo Zhao, Maosheng Pang, Chen Zhang +3Multimodal Generation Evidence-Grounded Generation
PromptDecipher: Supporting AI Tutor Authoring Through Editable Simulated Interactions
May 15, 2026Miina Koyama, Ruiwei Xiao, John StamperIntelligent Tutoring Systems LLM Prompting
Carnegie Mellon University Pittsburgh, Pennsylvania, USA
Defining Cultural Capabilities for AI Evaluation: A Taxonomy Grounded in Intercultural Communication Theory
May 15, 2026Isar Nejadgholi, Masoud Kianpour, Krishnapriya Vishnubhotla +1Cultural Alignment Generative AI Evaluation
National Research Council, Canada · Toronto Metropolitan University, Canada · Mila, Quebec AI Institute, Canada
Unsteady Metrics and Benchmarking Cultures of AI Model Builders
May 13, 2026Stefan Baack, Christo Buschek, Maty BohacekLLM Evaluation Benchmark Design
Independent Researcher · Mozilla Fellowship · Stanford University
Measuring Google AI Overviews: Activation, Source Quality, Claim Fidelity, and Publisher Impact
May 13, 2026Haofei Xu, Umar Iqbal, Jacob M. MontgomeryCitation Verification Generative Retrieval
Washington University in St. Louis
AI-Generated Slides: Are They Good? Can Students Tell?
May 13, 2026Juho Leinonen, Lisa Zhang, Arto HellasGenerative AI in Education Generative AI Evaluation
Aalto University · Espoo, Finland · University of Toronto Mississauga +1
A Comparative Study of Controlled Text Generation Systems Using Level-Playing-Field Evaluation Principles
May 12, 2026Michela Lorandi, Anya BelzControllable Text Generation Automated Evaluation
Dublin City University, Dublin, Ireland
Beyond Text Prompts: Visual-to-Visual Generation as A Unified Paradigm
May 12, 2026Yaofang Liu, Kangning Cui, Meng Chu +7Vision-Language Models Conditional Image Generation
City University of Hong Kong · City University of Hong Kong (Dongguan) · The Hong Kong University of Science and Technology +4
Read, Grep, and Synthesize: Diagnosing Cross-Domain Seed Exposure for LLM Research Ideation
May 12, 2026Yunju Choi, Min SongResearch Idea Generation Large Language Model-Guided Scientific Discovery
1Yonsei University, Seoul, Republic of Korea · 2OnomaAI, Seoul, Republic of Korea.
Measuring Five-Nines Reliability: Sample-Efficient LLM Evaluation in Saturated Benchmarks
May 11, 2026Eungyeup Kim, Chenchen Gu, Vashisth Tiwari +1LLM Evaluation LLM Reliability
Carnegie Mellon University
Grounded Satirical Generation with RAG
May 11, 2026Oona Itkonen, Yuxin Su, Linyao Du +1Grounded Text Generation Text Generation
University of Helsinki
Useful for Exploration, Risky for Precision: Evaluating AI Tools in Academic Research
May 11, 2026Anthea Dathe, Kiran Hoffmann, Aline MangoldAI-Assisted Scientific Research Scientific Reproducibility
Department Speculative Transformation Dresden University of Technology Dresden, 01062
Active Testing of Large Language Models via Approximate Neyman Allocation
May 11, 2026Zeli Liu, Jiancheng Zhang, Cong Liu +1LLM Evaluation Language Model Generation Evaluation
University of California, Riverside
Measuring What Matters: Benchmarking Generative, Multimodal, and Agentic AI in Healthcare
May 8, 2026Prasanna Desikan, Harshit Rajgarhia, Shivali Dalmia +1Healthcare Benchmark Design
Centific AI Research Redmond, WA, USA
Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors
May 8, 2026Shuhaib Mehri, Philippe Laban, Sumuk Shashidhar +4Human Behavior Simulation User Simulation
University of Illinois Urbana-Champaign · Microsoft Research · University of California, Berkeley
Benchmarked Yet Not Measured -- Generative AI Should be Evaluated Against Real-World Utility
May 7, 2026Ishani Mondal, Shweta BhardwajHuman-Centered AI Generative AI Evaluation
University of Maryland, College Park, USA
Uneven Evolution of Cognition Across Generations of Generative AI Models
May 7, 2026Isaac Galatzer-Levy, Daniel McDuff, Xin Liu +1Cognitive Modeling Multimodal Model Evaluation
1Google DeepMind · 2Google Research · University of Washington
MIND: Monge Inception Distance for Generative Models Evaluation
May 7, 2026Quentin Berthet, Yu-Han Wu, Clement Crepy +3Image Generation Evaluation Sliced Wasserstein Distance
Google DeepMind · LPSM, Sorbonne Université
Ex Ante Evaluation of AI-Induced Idea Diversity Collapse
May 7, 2026Nafis Saami Azad, Raiyan Abdul BatenLLM Evaluation Computational Creativity
Bellini College of Artificial Intelligence, Cybersecurity, and Computing University of South Florida Tampa, FL–33620, USA
SEQUOR: A Multi-Turn Benchmark for Realistic Constraint Following
May 7, 2026Beatriz Canaverde, Duarte M. Alves, José Pombal +2Instruction Following Generative AI Evaluation
Instituto de Telecomunicac¸˜oes · Instituto Superior T´ecnico, Universidade de Lisboa · Sword Health +2
Optimal Transport Audio Distance with Learned Riemannian Ground Metrics
May 7, 2026Wonwoo JeongSpeech Generation Evaluation Metric Learning
Sogang University
Logical Consistency as a Bridge: Improving LLM Hallucination Detection via Label Constraint Modeling between Responses and Self-Judgments
May 5, 2026Hao Mi, Qiang Sheng, Shaofei Wang +7Hallucination in Language Models LLM Hallucination Detection
Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences
Scale-Aware Adversarial Analysis: A Diagnostic for Generative AI in Multiscale Complex Systems
May 1, 2026Mengke Zhao, Guang-Xing Li, Duo Xu +1Generative Modeling Generative AI Evaluation
School of Astronomy and Space Science, Nanjing University, 163 Xianlin Avenue, Nanjing 210023, Jiangsu, People’s Republic of China · Key Laboratory of Modern Astronomy and Astrophysics (Nanjing University), Ministry of Education, Nanjing 210023, Jiangsu, People’s Republic of China · South-Western Institute for Astronomy Research, Yunnan University, Kunming 650091, People’s Republic of China +1
Visual Generation in the New Era: An Evolution from Atomic Mapping to Agentic World Modeling
Apr 30, 2026Keming Wu, Zuhao Yang, Kaichen Zhang +24World Models Generative Modeling
1Tsinghua University · 12LMMs-Lab · 2Nanyang Technological University +9
From LLM-Driven Trading Card Generation to Procedural Relatedness: A Pokémon Case Study
Apr 30, 2026Johannes Pfau, Panagiotis VrettisGenerative Modeling Procedural Content Generation
Utrecht University · Utrecht, Netherlands
The False Resonance: A Critical Examination of Emotion Embedding Similarity for Speech Generation Evaluation
Apr 29, 2026Yun-Shao Tsai, Yi-Cheng Lin, Huang-Cheng Chou +5TTS Synthesis Speech Generation Evaluation
Graduate Institute of Communication Engineering, National Taiwan University, Taiwan · Ming Hsieh Department of Electrical and Computer Engineering, University of Southern California, USA · Gilbert AI Lab, USA +2
Calibrated Surprise: An Information-Theoretic Account of Creative Quality
Apr 29, 2026Bo Zou, Chao XuCreativity Assessment Computational Creativity

May 19, 2026, cs.CLJ/K move · Enter open · S save

When Reasoning Supervision Hurts: TTCW-Based Long-Form Literary Review Generation

Jinlong Liu, Mohammed Bahja, Mark Lee

School of Computer Science, University of Birmingham, United Kingdom

Supervised Fine-Tuning Creativity Assessment LLM Evaluation Rubric-Guided Generation Text Generation Generative AI Evaluation

Open paper AskPDF arXiv

Automatic evaluation of long-form literary writing remains challenging, as generic LLM-as-Judge approaches may not fully capture creativity-related dimensions such as originality and flexibility. Although the Torrance Test of Creative Writing (TTCW) provides a structured creativity framework, and prior work has demonstrated reference-based TTCW evaluation at the pairwise level, no large-scale dataset exists for long-form TTCW-based literary review generation. We address this gap by constructing a dataset of 263,911 long-form stories, each annotated with scalar scores and meta-synthesised review comments across 14 TTCW-based dimensions. Using this dataset, we fine-tune Qwen3 models at two scales, 4B and 8B, under two conditions: with and without reasoning content. Results show that non-reasoning fine-tuning achieves stronger and more stable performance, with the best setting reaching an evaluation score of 0.6820. Further analysis shows that reasoning-supervised models are more prone to parse failures, often continuing with irrelevant or repetitive reasoning-style text rather than completing the required 14-metric review report. These results suggest that, for fixed-format rubric-based review generation, reasoning supervision is not straightforwardly beneficial, and precise metric-aligned scoring remains challenging even after task-specific fine-tuning.

Momentum

Related topics

Latest papers 211