Language Model Generation Evaluation

Ask about this topic

Momentum

30 papers in the last four weeks, up 100% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 206

All topics
CardsList

  1. Document-Level Text Simplification in Estonian Using Large Language Models

    Oct 7, 2026Meeri-Ly Muru, Eduard BarbuLow-Resource Language ProcessingLanguage Model Generation Evaluation

    National Library of Estonia · Institute of Computer Science, University of Tartu Tartu, Estonia

  2. I would rather quit NLP than read another paper like this: The rise of antithesis in NLP papers

    Oct 7, 2026Olga Zamaraeva, Adrián Gude, Roi Santos-Ríos +1Language Model Generation EvaluationLanguage Model Post-Training

    Universidade da Coruña, CITIC

  3. What pass@k Cannot Measure: Evaluating Diversity and Capability Retention after Post-Training

    Oct 5, 2026Subham Rath, Raj Dandekar, Rajat Dandekar +1LLM EvaluationLanguage Model Generation Evaluation

    Accenture Strategy & Consulting · Vizuara AI Labs (vizuara.com) · Vizuara AI Labs (vizuara.ai)

  4. SOL: Measuring Gaps between Text Distributions by Double Sliced Wasserstein Metrics

    Oct 5, 2026Gregor Kornhardt, Moritz Piening, Jannis Chemseddine +1Non-Autoregressive Text GenerationLanguage Model Generation Evaluation

    Institute of Mathematics, TU Berlin, Berlin, Germany · Department of Computer Science, University of Göttingen, Göttingen, Germany

  5. Keyword Harnesses Fail Open: A Cheap Diagnostic Ladder for Tool-Use Claims in Small Language Models

    Oct 1, 2026Juan S. SantillanaSupervised Fine-TuningTool-Use Evaluation

    Independent Researcher

  6. Generalization Is Stability, Not Accuracy: Multi-Axis Evaluation of LLMs

    Oct 1, 2026Nagham Omar, Mahmoud Jabarin, Maya Rozenshtein +3Prompt SensitivityLLM Evaluation

    Technion – Israel Institute of Technology

  7. Beyond Leaderboards: Tokenomics of Agentic Small Language Model Ensembles

    Oct 1, 2026Alexei N. Skurikhin, Emily M. Taylor, Nathan A. DeBardelebenLLM Inference EfficiencyLanguage Model Generation Evaluation

    Los Alamos National Laboratory Los Alamos, New Mexico, USA

  8. LLM-as-a-Judge for Low-Resource Languages: Adapting Ragas and Comparative Ranking for Romanian

    Sep 30, 2026Claudiu Creanga, Liviu P. DinuLLM EvaluationLLM-as-a-Judge

    Interdisciplinary School of Doctoral Studies, University of Bucharest, Romania · HLT Research Center, University of Bucharest, Romania · Faculty of Mathematics and Computer Science, University of Bucharest, Romania

  9. Mubric: Mutation Testing-Guided Rubric Generation for LLM Evaluation

    Sep 29, 2026Jiayuxuan Yang, Jie M. Zhang, Yiling Lou +1Language Model Generation EvaluationRubric-Based Evaluation

    Tsinghua University · King’s College London · University of Illinois Urbana-Champaign

  10. Dating the Model: Hidden Dates in System Prompts Affect LLM Evaluation

    Sep 29, 2026Mario Sanz-Guerrero, Minh Duc Bui, Manuel Mager +1LLM EvaluationLanguage Model Generation Evaluation

    Johannes Gutenberg University Mainz, Germany · Universidad Iberoamericana, Mexico · University of Colorado Boulder, USA

  11. Generating Edit-Inducing Questions for AI Research Manuscripts

    Sep 29, 2026Sebastian Joseph, Zichao Wang, Jennifer Healey +3Text GenerationLanguage Model Generation Evaluation

    The University of Texas at Austin · Adobe Research

  12. Coherence-Aware Distributional Evaluation of Open-Ended Text Generation

    Sep 28, 2026Jinnuo Liu, Junhao Zhu, Weifeng Jiang +2Language Model Generation EvaluationGenerative AI Evaluation

    New York University · Center for Data Science, NYU Shanghai · Georgia Institute of Technology

  13. Simple Diffusion Language Models Are More Effective Few-Step Generators Than Reported

    Sep 27, 2026Hasan Amin, Ming Yin, Rajiv KhannaMasked Diffusion ModelsLanguage Model Generation Evaluation

    Purdue University

  14. SlopBench: How Well Can We Rank Language Models by Slop? A Multi-Domain Benchmark of Repetitive AI Writing

    Sep 27, 2026Dhruv Roongta, Harsha Gaddipati, Anh Tuan HuynhLLM EvaluationLanguage Model Generation Evaluation

    Slashy, San Francisco, California, USA

  15. MISHAP-Bench: A Hallucination Benchmark for Large Audio-Language Models

    Sep 27, 2026Zhi Wen Soi, Giulio Segalini, Jian-Jia Chen +1Audio-Language Model EvaluationAudio-Language Model Hallucination Detection

    University of Neuchâtel · TU Dortmund University · RWTH Aachen University +1

  16. Self-Designed Evaluators and Warm Memory for Long-Horizon Agents

    Sep 27, 2026Saeid Asgari, Emre Kiciman, Leonardo de Oliveira Nunes +1Language Model Generation EvaluationLong-Horizon Agent Evaluation

    Microsoft

  17. Knowing Is Not Choosing: What Explicit Verification Adds Beyond Generative Preference

    Sep 27, 2026Yilong Li, Chengpo Yan, Aayan Arish +1LLM EvaluationLLM Answer Verification

    Department of Computer Science University of Wisconsin-Madison Madison, WI 53703, USA

  18. Style, Not Self: Surface Cues Explain Zero-Shot Code Attribution by Large Language Models

    Sep 24, 2026Ehsan Barkhordar, Surendrabikram ThapaLLM EvaluationLanguage Model Generation Evaluation

    Koç University · Virginia Tech

  19. Two Emojis of Difference: What Multilingual Affective Generation Benchmarks Actually Measure

    Sep 24, 2026Fardeen Sadab, Adib SakhawatMultilingual Language Model EvaluationLanguage Model Generation Evaluation

    Department of Computer Science and Engineering Islamic University of Technology, Dhaka, Bangladesh

  20. Likelihood Ranking doesn't Scale Like Prompting in LLMs

    Sep 24, 2026Alessandro Bondielli, Lucia Passaro, Davide Bacciu +1LLM EvaluationLanguage Model Generation Evaluation

    CoLingLab, Department of Philology, Literature and Linguistics, University of Pisa · Department of Computer Science, University of Pisa

  21. Reasoning Instructions Can Break Answer Decoding in Vision--Language Models

    Sep 24, 2026Zeyan Li, Siyuan Qiu, Jianfeng XuVLM EvaluationLanguage Model Generation Evaluation

    Shanghai Jiao Tong University

  22. Reference-Based Analysis of Coherence and Diversity in Open-Ended Text Generation

    Sep 23, 2026Esteban Garcés AriasOpen-Ended GenerationLanguage Model Generation Evaluation

    Department of Statistics, LMU Munich Munich Center for Machine Learning (MCML)

  23. Speculative Evaluation of Stochastic LLMs

    Sep 23, 2026Qianli Shen, Xiang Li, Ruomeng Ding +3LLM EvaluationLanguage Model Generation Evaluation

    Alibaba Group · National University of Singapore · University of North Carolina at Chapel Hill

  24. Recognized but Not Produced: A Generation Benchmark for Culturally Specific Kinship Terms

    Sep 22, 2026Sahil Pardasani, Madhusudan SinghMultilingual Language Model EvaluationLanguage Model Generation Evaluation

    Blockchain Data Intelligence Lab The Pennsylvania State University University Park, PA, USA

  25. JEV-as-a-Judge: Accept When Confident, Escalate When Unsure

    Sep 22, 2026Yubo Li, Yidi Miao, Ramayya Krishnan +1LLM-as-a-JudgeLanguage Model Generation Evaluation

    Carnegie Mellon University

  26. A Semiotics-Aware Framework for Evaluating Fidelity and Coverage in Natural Language Generation

    Sep 22, 2026Lorenzo Zangari, Davide PiccaLLM EvaluationLanguage Model Generation Evaluation

    University of Lausanne Lausanne, Switzerland

  27. Efficient Cost-Aware LLM Evaluation via Bayesian Bandit Gittins Indices

    Sep 22, 2026Qian Xie, Yueli He, Nairen CaoMulti-Armed BanditsLLM Evaluation

    Cornell University · Columbia University · New York University +1

  28. Made in Hungary: Comments on the performance of generative language models

    Sep 16, 2026Mátyás Osváth, Enikő Héja, Noémi Ligeti-NagyLLM EvaluationLanguage Model Generation Evaluation

    Institute for Language Technologies and Applied Linguistics ELTE Research Centre for Linguistics Budapest, Hungary

  29. A Calibrated Instrument for Measuring How Inference Optimizations Affect Output Quality

    Sep 16, 2026Jerry KaplanLLM QuantizationLLM Evaluation

    Adjunct Lecturer, Computer Science and Masters in International Policy, Stanford University

  30. EviScope: Paired Counterfactual Evidence Diagnostics for Faithful and Efficient Grounded Language Models

    Sep 15, 2026Suryadeep Singh DeswalCounterfactual EvaluationLLM Grounding

    Indian Institute of Technology Roorkee Roorkee, India

  31. ImpossibleRubrics: Stress-Testing Generated Rubrics as Reward Signals

    Sep 15, 2026Bowen Qin, Yi Xie, Yesheng Liu +1Reward HackingLanguage Model Generation Evaluation

    National University of Singapore · Peking University · Institute of Automation, Chinese Academy of Sciences

  32. YallaMorph: A Benchmark for Evaluating Arabic Morphological Generation in Large Language Models

    Sep 9, 2026Mahmoud Reda, Salam Khalifa, Reham Marzouk +1Multilingual Language Model EvaluationArabic NLP

    New York University Abu Dhabi · Stony Brook University · Mohamed bin Zayed University of Artificial Intelligence

  33. API Benchmark Scores Do Not Reliably Transfer to Chatbot Interfaces

    Sep 8, 2026Jennifer Wang, Joachim Baumann, Daniel E. Ho +1LLM EvaluationLanguage Model Generation Evaluation

    Stanford University

  34. Beyond Fluent Generation: A CPU Reliability Benchmark for MCP-Style Tool Calling in Sub-2B Small Language Models for Edge Deployment

    Sep 7, 2026Abrar Shahriar Qurat-Ul-Ain MastoiTool-Augmented Language Model AgentsOn-Device Language Model Inference

    University of the West of England, Bristol,BS16 1QY United Kingdom

  35. Beyond Scores: Understanding LLM-as-a-Judge Mechanisms in Summarization Evaluation

    Sep 1, 2026Himil Vasava, Ming JiangLLM-as-a-JudgeLanguage Model Generation Evaluation

    University of Wisconsin-Madison

  36. How Correct Is Your Answer? A Semantic Correctness Framework for Open QA Evaluation

    Sep 1, 2026Elitsa Yotkova, Violeta Kastreva, Petar Velkov +4LLM Answer VerificationLanguage Model Generation Evaluation

    Sofia University “St. Kliment Ohridski” · ETH Zürich · University of Zurich +1

  37. Does task decomposition improve automatic NLG evaluation?

    Sep 1, 2026Sebastian Steindl, Nikos Voskarides, Alberto Gasparin +1LLM EvaluationLLM-as-a-Judge

    Amazon, Barcelona, Spain · Amazon, Berlin, Germany

  38. Hints Help But Do They Teach? Evaluating Skills Transfer in Code Generation

    Sep 1, 2026Will BadrLanguage Model Generation EvaluationLLM Prompting

  39. Value Over Language Model: Detecting Original Contribution in Writing

    Sep 1, 2026Vibhhu Sharma, Thorsten Joachims, Sarah DeanLLM EvaluationAI-Generated Text Detection

    Cornell University

  40. CLIN: an Objective Framework for Evaluating Creativity in Short Persian Literary Text

    Aug 31, 2026Mohammad Reza Modarres, Armin Tourajmehr, Yadollah Yaghoobzadeh +1Creativity AssessmentLLM Evaluation

    School of Electrical and Computer Engineering, College of Engineering, University of Tehran, Tehran, Iran · Tehran Institute for Advanced Studies, Khatam University, Iran · Cardiff University, United Kingdom

  41. Small Language Models as Judges for Rubric-Based Reinforcement Learning

    Aug 30, 2026Fengyu Xie, Yilun Zhao, Bingsen Chen +2LLM-as-a-JudgeLanguage Model Generation Evaluation

    New York University · Yale University

  42. Sleight of Word Benchmark: Can Language Models Notice If Their Own Output Was Tampered With?

    Aug 30, 2026Alberto CetoliLanguage Model Generation EvaluationLanguage Model Self-Assessment

    Cleo

  43. IndicDetect: Evaluating Cross-Lingual LLM-Generated Text Detection for Hindi, Telugu, and Tamil

    Aug 30, 2026Bhaskar Ganesh Devalla, Junchao Wu, Nilesh Dokuparthi +4AI-Generated Text DetectionLanguage Model Generation Evaluation

    NLP2CT Lab, Faculty of Information Science and Computing, University of Macau · Department of Applied Computer Science, SRH University, Germany · Populus Group, Pittsburgh, United States of America

  44. GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation

    Aug 30, 2026Yifan Chen, Haitao Li, Qingyao Ai +4LLM-as-a-JudgeLanguage Model Generation Evaluation

    Department of Computer Science and Technology, Tsinghua University · National University of Singapore

  45. Novels generated by language models show compressed formal variation

    Aug 12, 2026Mehdy Sedaghat Payam, Justin QuinnLLM EvaluationLong-Form Document Generation

    University of Maryland · University of West Bohemia · Charles University

  46. Who Thinks Best Depends on How Long You Let Them: Budget-Dependent Rankings in LLM Evaluation

    Aug 12, 2026Rodrigo Guedes de Souza, Alison R. PanissonLLM EvaluationLanguage Model Generation Evaluation

    Federal University of Santa Catarina (UFSC) Brazil

  47. EVIL-Detect for NLPCC 2026 Shared Task 6: LLM-Generated Text Detection

    Aug 11, 2026Hongrui Bao, Hangyu Rong, Zhuoshang Wang +2AI-Generated Text DetectionLanguage Model Generation Evaluation

    Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China

  48. Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibility

    Aug 10, 2026Siyang Wu, Yibo Jiang, Bryon AragamPrompt SensitivityLLM Evaluation

    Data Science Institute University of Chicago Chicago, USA · Department of Computer Science University of Chicago Chicago, USA · Booth School of Business University of Chicago Chicago, USA

  49. TCS-BENCH: Benchmarking State-of-the-Art Generative AI Theoretical Computer Science Research Ability

    Aug 10, 2026Vincent Cohen-Addad, Dimitris Paparas, Ernest van Wijland +13LLM EvaluationAutomated Theorem Proving

    †Google 1 · ‡CNRS, IRIF, Université Paris-Cité · §Princeton University, Department of Computer Science +2

  50. AI Evaluation Should Measure Verification Cost, Not Correctness Alone

    Aug 9, 2026Viviana Crescitelli, Generoso Immediato, Fabio Persia +1Language Model Generation EvaluationGenerative AI Evaluation

    Hitachi, Ltd., Tokyo, Japan · Hitachi Rail, Naples, Italy · Department of Information Engineering, Computer Science and Mathematics, University of L’Aquila, Italy

  51. DS@GT ARC at Touché: Large Language Models for Retrieval-Augmented Debate

    Aug 8, 2026Anthony Miyaguchi, Conor JohnstonLLM EvaluationLLM-as-a-Judge

    Georgia Institute of Technology, North Ave NW, Atlanta, GA 30332, USA

  52. Can Language Models Imagine Without Seeing? Ekphrasis: Measuring Visual Creative Ideation in Text-Only LLMs

    Aug 7, 2026Hongyu Luo, He Wang, Huihao Jing +6Creativity AssessmentLLM Evaluation

    The Hong Kong University of Science and Technology, Hong Kong SAR, China

  53. DevIntent: How Much Does LLM-Generated Code Violate Developer Intent?

    Aug 7, 2026Susana Haing, Natan Vidra, Spurthi SettyLanguage Model Generation EvaluationCode Generation Evaluation

    Anote AI

  54. Where Models Converge and Humans Diverge: A Coverage Framework for Distributional Pluralism in Open-Ended Generation

    Aug 6, 2026Zini Yang, Emily Wenger, Richard SoOpen-Ended GenerationLLM Evaluation

    Department of Computer Science, Duke University, Durham, NC, USA · Department of Electrical and Computer Engineering, Duke University, Durham, NC, USA · Department of English, Duke University, Durham, NC, USA

  55. What We Observe as LLM Behavior Can Be a Side-effect of Inference Backend

    Aug 5, 2026Shahed Masoudian, Passant Shafaei, Monorama Swain +1LLM EvaluationLanguage Model Generation Evaluation

    Johannes Kepler University Linz

  56. Kathleen Writes: Autoregressive Generation and Data Scaling Without Attention

    Aug 5, 2026George FountzoulasLanguage Model Generation EvaluationAutoregressive Language Modeling

    Department of Computer Engineering & Informatics · Frederick University, Nicosia, Cyprus

  57. EuroExec: Frontier Language Models Fall Short of Expert Judgment on European Executive Decision Tasks

    Aug 5, 2026Pau Arnal, Khaled Denfir, Danylo Smahliuk +2Open-Ended GenerationLLM Evaluation

    Sovrano AI

  58. Scoring Rules! Statistical and Strategic Alignment for Text Evaluation Metrics

    Aug 2, 2026Shengwei Xu, Yuxuan Lu, Yifan Wu +2Language Model Generation EvaluationAutomated Evaluation

    University of Michigan · Peking University · Microsoft Research +1

  59. CraftAlign: Feature-Grounded Evaluation and Revision Guidance for AI Stories

    Aug 2, 2026Yang Yang, Boyun Xu, Shaofeng Liang +5AI-Generated Text DetectionLanguage Model Generation Evaluation

    The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511400, China · Institute of Deep Perception Technology, JITRI, Wuxi 214000, China

  60. ModelEquivBench: Certifying Multi-Relational Evaluation of LLM-Generated Optimization Models

    Jul 31, 2026Penglin Zhu, Jungang XuLLM EvaluationPairwise Comparison

    School of Computer Science and Technology, University of the Chinese Academy of Sciences