LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,643

All topics

DS@GT ARC at Touché: Large Language Models for Retrieval-Augmented Debate
Aug 8, 2026Anthony Miyaguchi, Conor JohnstonLLM Evaluation LLM-as-a-Judge
Georgia Institute of Technology, North Ave NW, Atlanta, GA 30332, USA
CORDA: A Benchmark for Hierarchical Harm-Centric Moral Reasoning in Large Language Models
Aug 8, 2026Siddarth Singh, Victoria Williams, Simon Rosen +6LLM Evaluation Moral Reasoning in Language Models
University of the Witwatersrand, Johannesburg, South Africa
Evaluator Ensembles Under Reward Hacking: Covariance Geometry and Finite-Search Guarantees
Aug 8, 2026Fariya Afrin, Ibne Farabi ShihabReward Hacking LLM Evaluation
Department of Computer Science, Kalinga Institute of Industrial Technology · Department of Computer Science, Iowa State University
Thinking Hard, Not Smart: Reasoning Models Fail to Ration Test-Time Compute Across Questions
Aug 8, 2026Chenrui Fan, Yize Cheng, Ming Li +3LLM Evaluation Inference-Time Compute Allocation
University of Maryland, College Park · MBZUAI, UAE
Counterfactual Benchmarking and Training for Factuality Consistency and Order-Robust Grounded Reasoning in LLMs over Heterogeneous Knowledge
Aug 8, 2026Shibo Chu, Yuze Liu, Tiehua Zhang +4LLM Evaluation LLM Grounding
Tongji University, Shanghai, China · Swinburne University of Technology, Melbourne, Australia · Wuhan University of Technology, Wuhan, China
Who Verifies the Benchmark? Decentralizing Trust in Large Language Model Evaluation
Aug 7, 2026Sahil Pardasani, Madhusudan SinghLLM Evaluation LLM-as-a-Judge
Blockchain Data Intelligence Lab The Pennsylvania State University, 201 Old Main, University Park, 16802, PA, United States of America.
GeoBenchLLM: A Comprehensive Benchmark for Evaluating LLMs on Geo-Related Tasks
Aug 7, 2026Rodrigo Ferreira Rodrigues, Karim Radouane, Jose G Moreno +1Spatial Reasoning Benchmarks LLM Evaluation
University of Toulouse Toulouse, France
Zero Gap Is Not Restoration: Stratified Per-Question Probability Evaluation and Step-wise Mitigation of Benchmark Contamination
Aug 7, 2026Ruijie Hou, Yueyang Jiao, Zhao Wang +1LLM Evaluation Memorization in Generative Models
Zhejiang University
Artificial Intelligence Can Match Domain Experts in Evidence Extraction and Critical Appraisal of Microbial Oncogenesis Research Publications
Aug 7, 2026Kaela Kokkas, Hairong Wang, Richard Klein +11LLM Evaluation Oncology
Department of Clinical Microbiology and Infectious Diseases, Faculty of Health Sciences, University of the Witwatersrand, Johannesburg, South Africa · School of Computer Science and Applied Mathematics, University of the Witwatersrand, Johannesburg, South Africa · Wits Machine Intelligence and Neural Discovery (MIND) Institute, University of the Witwatersrand, Johannesburg, South Africa +13
Recipes for Creativity: Iterative Generation and Evaluation in Large Language Models
Aug 7, 2026Rens Anderson, Tessa Verhoef, Amirhossein ZohrehvandCreativity Assessment LLM Evaluation
LIACS Leiden University Einsteinweg 55 2333 CC Leiden The Netherlands
Benchmarking and Reasoning Distillation of Large Language Models for Feedback Controller Design in Complex Dynamical Systems
Aug 7, 2026Zhongchao Zhou, Yixuan Xie, Wenwei Yu +5LLM Evaluation Language Model-Based Control
Matsuo-Iwasawa Laboratory, Department of Technology Management for Innovation, School of Engineering, The University of Tokyo, Tokyo, Japan · Department of Statistics, University of Illinois Urbana-Champaign, Champaign, IL, USA · Center for Frontier Medical Engineering, Chiba University, Chiba, Japan +1
Can Language Models Imagine Without Seeing? Ekphrasis: Measuring Visual Creative Ideation in Text-Only LLMs
Aug 7, 2026Hongyu Luo, He Wang, Huihao Jing +6Creativity Assessment LLM Evaluation
The Hong Kong University of Science and Technology, Hong Kong SAR, China
Critical Acclaim Orientation in Large Language Models: Evidence from Film Preference Elicitation
Aug 7, 2026Jonghyun Jee, Aaron ShawLLM Evaluation Cultural Bias in Language Models
Department of Communication Studies, Northwestern University
Blind to the Pivotal Vote: Aggregate Independence Metrics Miss Where Verification Actually Helps
Aug 7, 2026Yang ShuLLM Evaluation LLM-as-a-Judge
Zhejiang University, Hangzhou, China
LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers
Aug 7, 2026Tao Feng, Fangxu Yu, Haozhen Zhang +9LLM Evaluation LLM Routing
University of Illinois Urbana-Champaign · University of Maryland, College Park · 3Nanyang Technological University +2
TeXFix-Bench: An Empirically Grounded Multi-Format Benchmark for LLM-Based Document Source Repair
Aug 7, 2026Prajwal S. VenkateshmurthyLLM Evaluation
Independent Researcher San Jose, CA, USA
TradeVerse: A Longitudinal Benchmark of Political Negotiation in International Trade
Aug 6, 2026Debodeep Banerjee, Amitangshu DasguptaLLM Evaluation Automated Negotiation
DI, University of Pisa · RazorPay · DISI, University of Trento
Don't `Well, Actually' Me Unless You Know What You're Talking About: Weak Presupposition Verification Degrades General QA Performance
Aug 6, 2026Shenran Wang, Vered Shwartz, Hila GonenLLM Evaluation LLM Answer Verification
Department of Computer Science, University of British Columbia · Vector Institute · CIFAR AI Chair +1
The Bitter Lesson of Tool Calling
Aug 6, 2026Ishan Patel, Sahil Sen, Elias Lumer +1LLM Evaluation Tool-Augmented Language Model Agents
Commercial Technology and Innovation Office PricewaterhouseCoopers, U.S.A
Benchmarking and Enhancing LLMs for Rule-Intensive Review of National Standard Documents
Aug 6, 2026Tao Wang, Qihao Yang, Rongjiao Liang +4LLM Evaluation
School of Computer Science, South China Normal University, Guangzhou, China · School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China · China National Institute of Standardization, Beijing, China
What Current AI Benchmarks Leave Unmeasured: Modality, Search, Citations, and Implications (for Safety Evaluations)
Aug 6, 2026Ro Encarnación, Tina Behzad, Emma Lurie +1LLM Safety Benchmarks LLM Evaluation
University of Pennsylvania · Stony Brook University
LangChoiceBench: Measuring and Explaining Programming-Language Choice in LLMs
Aug 6, 2026Lukas Twist, Twm Stone, Helen Yannakoudakis +1LLM Evaluation Language Model Bias Evaluation
King’s College London, London, UK · MATS Research
EpiBench: Can LLMs Understand Epitopes for Antibody Drug Discovery?
Aug 6, 2026Zirui Wang, Jiaqi Wang, Qinghan Wang +4LLM Evaluation Drug Discovery
1Valhalla Technology · 2Zhejiang University
Mitigating Scoring Bias in LLM-as-a-Judge via Random Number Generation
Aug 6, 2026Yuma Asato, Kiyoaki Shirai, Natthawut KertkeidkachornLLM Evaluation LLM-as-a-Judge
Japan Advanced Institute of Science and Technology
Human-Like Anaphor Resolution in Large Language Models
Aug 6, 2026Keane Zhang, Varshini Chinta, Raj Sanjay Shah +1Coreference Resolution LLM Evaluation
Georgia Institute of Technology
Where Models Converge and Humans Diverge: A Coverage Framework for Distributional Pluralism in Open-Ended Generation
Aug 6, 2026Zini Yang, Emily Wenger, Richard SoOpen-Ended Generation LLM Evaluation
Department of Computer Science, Duke University, Durham, NC, USA · Department of Electrical and Computer Engineering, Duke University, Durham, NC, USA · Department of English, Duke University, Durham, NC, USA
Can Open-Weight LLMs Produce Kernel-Verified Coq Proofs? A Pilot Study
Aug 5, 2026Ahmed Ryan, Md Erfan, Akond Ashfaque Ur Rahman +1LLM Evaluation Automated Theorem Proving
University of Alabama, Tuscaloosa, Alabama, USA · Auburn University, Auburn, Alabama, USA
Evaluating and Improving Pedagogical Fit in LLM-Based AI Tutors with the Pedagogical Suitability Index
Aug 5, 2026Benjamin Barlog, Hudson Craig, Zedong PengLLM Evaluation LLM Alignment
Department of Computer Science University of Montana Missoula, MT, USA
Evidence Lock Before Commitment: A Frozen Interface Degrades LLM-as-Judge Evaluation
Aug 5, 2026Divyansh SinghHuman Preference Evaluation LLM Evaluation
University of Florida
Sharding Prevents LLM Oversight Failures and Adversarial Exploitation
Aug 5, 2026Victor Akinwande, J. Zico Kolter, Aran NayebiScalable Oversight LLM Evaluation
School of Computer Science 2Neuroscience Institute Carnegie Mellon University

Aug 8, 2026, cs.IRJ/K move · Enter open · S save

DS@GT ARC at Touché: Large Language Models for Retrieval-Augmented Debate

Anthony Miyaguchi, Conor Johnston

Georgia Institute of Technology, North Ave NW, Atlanta, GA 30332, USA

LLM Evaluation LLM-as-a-Judge Language Model Generation Evaluation Pragmatic Reasoning in Language Models Multi-Turn Dialogue Evaluation RAG Evaluation

Open paper AskPDF arXiv

We extend the DS@GT ARC working-note submission to the Touché 2025 Retrieval-Augmented Debate task. The task has two subtasks: generating the next utterance in a simulated debate, and evaluating debate responses according to the Gricean maxims of Quantity, Quality, Relation, and Manner. The DS@GT ARC submission consisted of six leading LLMs from three providers through a retrieval-augmented prompting pipeline. We summarize the results from the working paper and explore whether multi-LLM evaluator agreement is a reliable proxy for official evaluation performance. The analysis shows that frontier LLM systems are strong response generators, and as evaluators they agree strongly within model families. However this consensus does not reliably track the official evaluation target, with the largest gap on the Quality maxim. The accompanying source code for this paper is located at https://github.com/dsgt-arc/touche-2025-rad and https://github.com/dsgt-arc/touche-2025-rad-analysis.

Momentum

Related topics

Latest papers 1,643