Language Model Generation Evaluation

Follow this topic Ask about this topic

Momentum

30 papers in the last four weeks, up 100% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 209

All topics

Dimension-Level Intent Fidelity Evaluation for Large Language Models: Evidence from Structured Prompt Ablation
May 14, 2026GAng PengMultilingual Language Model Evaluation LLM Evaluation
Huizhou Lateni AI Technology Co., Ltd., Huizhou, China · Huizhou University, Huizhou, China
NodeSynth: Socially Aligned Synthetic Data for AI Evaluation
May 14, 2026Qazi Mamunur Rashid, Xuan Yang, Zhengzhe Yang +5Language Model Generation Evaluation Language Model Safety Evaluation
Google Research · Google · Google Deepmind
RTLC -- Research, Teach-to-Learn, Critique: A three-stage prompting paradigm inspired by the Feynman Learning Technique that lifts LLM-as-judge accuracy on JudgeBench with no fine-tuning
May 13, 2026Andrea MorandiLLM Evaluation Pairwise Comparison
Cisco Systems, Inc.
Predicting Disagreement with Human Raters in LLM-as-a-Judge Difficulty Assessment without Using Generation-Time Probability Signals
May 12, 2026Yo EharaAnnotator Disagreement LLM-as-a-Judge
Tokyo Gakugei University
DiffScore: Text Evaluation Beyond Autoregressive Likelihood
May 12, 2026Wen Lai, Yingli Shen, Dingnan Jin +4LLM Evaluation Language Model Generation Evaluation
1Ant Group · 2Tsinghua University · 3Technical University of Munich
VERDI: Single-Call Confidence Estimation for Verification-Based LLM Judges via Decomposed Inference
May 11, 2026Jasmine Qi, Danylo Dantsev, Muyang SunPost-Hoc Calibration LLM-as-a-Judge
Indeed Inc.
LLARS: Enabling Domain Expert & Developer Collaboration for LLM Prompting, Generation and Evaluation
May 11, 2026Philipp Steigerwald, Mara Stieler, Jennifer Burghardt +2Language Model Generation Evaluation LLM Prompting
Technische Hochschule Nürnberg Georg Simon Ohm, Nürnberg, Germany · Faculty of Computer Science, Centre for Artificial Intelligence (KIZ) · Faculty of Social Sciences, Institute for E-Counselling
StereoTales: A Multilingual Framework for Open-Ended Stereotype Discovery in LLMs
May 11, 2026Pierre Le Jeune, Étienne Duchesne, Weixuan Xiao +4Open-Ended Generation Multilingual Language Model Evaluation
Giskard AI · École Normale Supérieure, INSERM, Paris
Active Testing of Large Language Models via Approximate Neyman Allocation
May 11, 2026Zeli Liu, Jiancheng Zhang, Cong Liu +1LLM Evaluation Language Model Generation Evaluation
University of California, Riverside
NARRA-Gym for Evaluating Interactive Narrative Agents
May 8, 2026Yue Huang, Yuchen Ma, Jiayi Ye +14Language Model Generation Evaluation Interactive Storytelling
University of Notre Dame · 2LMU Munich · 11Munich Center for Machine Learning +8
Delulu: A Verified Multi-Lingual Benchmark for Code Hallucination Detection in Fill-in-the-Middle Tasks
May 7, 2026Mahdi Erfanian, Nelson Daniel Troncoso, Aashna Garg +4Language Model Error Detection Language Model Generation Evaluation
University of Illinois Chicago · Microsoft
ReasonLab: A Controlled and Auditable Evaluation of Prompting Techniques for Multiple-Choice QA
May 7, 2026Inder Preet, Shuxin Lin, Dhaval PatelLLM Evaluation Language Model Generation Evaluation
IBM, Dublin · IBM, New York
Beyond Fixed Benchmarks and Worst-Case Attacks: Dynamic Boundary Evaluation for Language Models
May 7, 2026Haoxiang Wang, Da Yu, Huishuai ZhangLLM Evaluation Language Model Generation Evaluation
Peking University · Google Research · Wangxuan Institute of Computer Technology, Peking University State Key Laboratory of General Artificial Intelligence
Automatically Finding and Validating Unexpected Side-Effects of Interventions on Language Models
May 6, 2026Quintin Pope, Ajay Hayagreeve Balaji, Jacques Thibodeau +1Language Model Generation Evaluation LLM Auditing
Oregon State University · Independent Researcher
StoryAlign: Evaluating and Training Reward Models for Story Generation
May 6, 2026Haotian Xia, Hao Peng, Yunjia Qi +4Reward Modeling Language Model Generation Evaluation
Department of Computer Science and Technology, Tsinghua University
RaguTeam at SemEval-2026 Task 8: Meno and Friends in a Judge-Orchestrated LLM Ensemble for Faithful Multi-Turn Response Generation
May 6, 2026Ivan Bondarenko, Roman Derunets, Oleg Sedukhin +3Language Model Generation Evaluation RAG Evaluation
Novosibirsk State University
DoGMaTiQ: Automated Generation of Question-and-Answer Nuggets for Report Evaluation
May 6, 2026Bryan Li, William Walden, Yu Hou +6Language Model Generation Evaluation RAG Evaluation
Google Inc. · University of Pennsylvania · Johns Hopkins University +3
EvoLM: Self-Evolving Language Models through Co-Evolved Discriminative Rubrics
May 5, 2026Shuyue Stella Li, Rui Xin, Teng Xiao +8RL for Language Models LLM Evaluation
University of Washington · 2Allen Institute for AI · University of Pennsylvania
Beating the Style Detector: Three Hours of Agentic Research on the AI-Text Arms Race
May 4, 2026Andreas Maier, Moritz Zaiss, Siming BayerAI-Generated Text Detection Language Model Generation Evaluation
Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany · Institute of Neuroradiology, University Hospital Erlangen, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany
A multilingual hallucination benchmark: MultiWikiQHalluA
May 4, 2026Freja Thoresen, Dan Saattrup SmartMultilingual Language Models Multilingual Language Model Evaluation
Alexandra Institute · Rued Langgaards Vej 7, 2300 København
Prosa: Rubric-Based Evaluation of LLMs on Real User Chats in Brazilian Portuguese
May 2, 2026Roseval Malaquias Junior, Giovana Kerche Bonás, Thales Sales Almeida +6LLM Evaluation LLM-as-a-Judge
Maritaca AI · Jusbrasil
The Partial Testimony of Logs: Evaluation of Language Model Generation under Confounded Model Choice
May 2, 2026Jikai Jin, Vasilis SyrgkanisLLM Evaluation Off-Policy Evaluation
Optimization before Evaluation: Evaluation with Unoptimised Prompts Can be Misleading
Apr 30, 2026Nicholas Sadjoli, Tim Siefken, Atin Ghosh +2LLM Evaluation Language Model Generation Evaluation
SAP · Stanford University
From Coarse to Fine: Benchmarking and Reward Modeling for Writing-Centric Generation Tasks
Apr 30, 2026Qingyu Ren, Tianjun Pan, Xingzhou Chen +1Reward Modeling RL for Language Models
Shanghai Key Laboratory of Data Science, College of Computer Science and Artificial Intelligence, Fudan University · 2Shanghai Artificial Intelligence Laboratory
Instruction Complexity Induces Positional Collapse in Adversarial LLM Evaluation
Apr 29, 2026Jon-Paul CacioliLLM Evaluation Language Model Generation Evaluation
Independent Researcher, Melbourne, Australia
Theory-Grounded Evaluation Exposes the Authorship Gap in LLM Personalization
Apr 29, 2026Yash Ganpat SawantLLM Evaluation LLM Personalization
Independent AI Researcher
The Surprising Universality of LLM Outputs: A Real-Time Verification Primitive
Apr 28, 2026Alex Bogdan, Adrian de Valois-FranklinLLM Evaluation Language Model Generation Evaluation
Evolutionairy AI Toronto, Canada
MGTEVAL: An Interactive Platform for Systemtic Evaluation of Machine-Generated Text Detectors
Apr 28, 2026Yuanfan Li, Qi Zhou, Chengzhengxu Li +5AI-Generated Text Detection Language Model Generation Evaluation
Faculty of Electronic and Information Engineering, Xi’an Jiaotong University · Queen Mary University of London
LongSumEval: Question-Answering Based Evaluation and Feedback-Driven Refinement for Long Document Summarization
Apr 28, 2026Huyen Nguyen, Haoxuan Zhang, Yang Zhang +2Language Model Generation Evaluation Factual Consistency Evaluation
Cigna Group Evernorth Health Services Austin, Texas, USA · dept. of Information Science University of North Texas Denton, Texas, USA · dept. of Data Science University of North Texas Denton, Texas, USA
JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems
Apr 26, 2026Rohith Reddy Bellibatlu, Edward Raff, Wenbin ZhangLLM Evaluation LLM-as-a-Judge
Florida International University, Miami, FL · CrowdStrike, New York, USA · University of Maryland Baltimore County, Baltimore, USA

May 14, 2026, cs.CLJ/K move · Enter open · S save

Dimension-Level Intent Fidelity Evaluation for Large Language Models: Evidence from Structured Prompt Ablation

GAng Peng

Huizhou Lateni AI Technology Co., Ltd., Huizhou, China · Huizhou University, Huizhou, China

Multilingual Language Model Evaluation LLM Evaluation Language Model Generation Evaluation Instruction Following

Open paper AskPDF arXiv

Holistic evaluation scores capture overall output quality but do not distinguish whether a model reproduced the structural form of a user's request from whether it preserved the user's specific intent. We propose a dimension-level intent fidelity evaluation framework, applied here through a structured prompt ablation study across 2,880 outputs spanning three languages, three task domains, and six LLMs, that separately measures structural recovery and intent fidelity for each semantic dimension. This framework reveals a systematic structural-fidelity split: among Chinese-language outputs with complete paired scores, 25.7% received perfect holistic alignment scores (GA=5) while exhibiting measurable dimensional intent deficits; among English-language outputs, this proportion rose to 58.6%. Human evaluation confirmed that these split-zone outputs represent genuine quality deficits and that dimensional fidelity scores track human judgements more reliably than holistic scores do. A public-private decomposition of 2,520 ablation cells characterises when models successfully compensate for missing intent and when they fail, while proxy annotation distinguishes prior inferability from default recoverability. A weight-perturbation experiment shows that moderate misalignment is typically absorbed, whereas severe dimensional inversion is consistently harmful. These findings demonstrate that dimension-level intent fidelity evaluation is a necessary complement to holistic assessment when evaluating LLM outputs for user-specific tasks.

Momentum

Related topics

Latest papers 209