LLM Evaluation

LLM: Large Language Model

The field studies how to measure language-model capabilities, reliability, and risks, and whether those measurements support sound comparisons and deployment decisions. Approaches include curated and synthetic benchmarks, psychometric and adaptive testing, execution-based checks, robustness tests, and human or LLM-based judges. Researchers also examine calibration, uncertainty, contamination, prompt sensitivity, and whether scores reflect the intended capability across languages and domains. A central tension is that efficient aggregate scores can hide important errors and measurement biases, while more realistic, human-grounded evaluations are costly and difficult to standardize.

Latest papers

2,432 papers286 papers in the last four weeks, up 16% on the four weeks before. 1.7% of all new papers.

Detoxify: A framework for abusive text transformation using LLMs
Jul 14, 2025Rohitash Chandra, Jiyong Choi, Jayesh SonawaneLLM Evaluation Text Detoxification
Transitional Artificial Intelligence Research Group, School of Mathematics and Statistics, UNSW, Sydney, Australia · Centre for Artificial Intelligence and Innovation, Pingla Institute, Sydney, Australia · Centre for Artificial Intelligence and Innovation, Pingla
LLM Bidders Preserve the Mechanism-Level Orderings of Human Bidders
Jul 12, 2025Anand Shah, Kehang Zhu, Yanchen Jiang +4LLM Evaluation Human Behavior Simulation
MIT & NBER
LLM-Based Social Simulations Require a Boundary
Jun 24, 2025Zengqing Wu, Run Peng, Takayuki Ito +2LLM Evaluation Human Behavior Simulation
University of Osaka · University of Michigan · Kyoto University +1
Measuring Intent Comprehension in LLMs
Jun 19, 2025Nadav Kunievsky, James A. EvansPrompt Sensitivity LLM Evaluation
Knowlesdge Lab, University of Chicago, Chicago, Illinois, USA · Knowledge Lab, Data Science Institute, Department of Sociology, University of Chicago, Chicago, Illinois, USA.
LLMs are Brittle to Simple Code Transformations: Introducing CETBench - A Benchmark for Code-Equivalence Checking
Jun 4, 2025Neeva Oza, Ishaan Govil, Parul Gupta +3LLM Evaluation Language Model Robustness
Indian Institute of Technology Delhi · IBM Research AI
C-VARC: A Large-Scale Chinese Value Rule Corpus for Value Alignment of Large Language Models
Jun 2, 2025Ping Wu, Guobin Shen, Dongcheng Zhao +6Value Alignment LLM Alignment
BrainCog Lab, Institute of Automation, Chinese Academy of Sciences, Beijing, China · The School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Beijing Key Laboratory of Safe AI and Superalignment, Beijing, China +2
Sarc7: Evaluating Sarcasm Detection and Generation with Seven Types and Emotion-Informed Techniques
May 31, 2025Raina Gao, Alyssa Jeong, Lang Xiong +4LLM Evaluation Sarcasm Detection
Algoverse AI Research
Benchmarks Are Not Atomic: Composition-Aware LLM Evaluation using BenchHub
May 31, 2025Eunsu Kim, Haneul Yoo, Guijin Son +3LLM Evaluation
KAIST · Yonsei University · OnelineAI +1
SocialMaze: A Benchmark for Evaluating and Enhancing Social Reasoning in Large Language Models in Complex Social Environments
May 29, 2025Zixiang Xu, Yanbo Wang, Yue Huang +13LLM Evaluation AI Agent Benchmarks
Mohamed bin Zayed University of Artificial Intelligence · University of Southern California · University of Notre Dame +3
PBEBench: A Multi-Step Programming by Examples Reasoning Benchmark inspired by Historical Linguistics
May 29, 2025Atharva Naik, Prakam, Yash Mathur +6Program Synthesis Inductive Inference
Carnegie Mellon University · Independent Researcher · Ohio State University
Assessing and Refining ChatGPT's Performance in Identifying Targeting and Inappropriate Language: A Comparative Study
May 27, 2025Baran Barbarestani, Isa Maks, Piek VossenContent Moderation Toxicity Detection
Vrije Universiteit Amsterdam, Computational Linguistics and Text Mining Lab
UrduFactCheck: An Agentic Fact-Checking Framework for Urdu with Evidence Boosting and Benchmarking
May 21, 2025Sarfraz Ahmad, Hasan Iqbal, Momina Ahsan +6LLM Evaluation Factual Consistency Evaluation
MBZUAI
R3: Robust Rubric-Agnostic Reward Models
May 19, 2025David Anugraha, Zilu Tang, Lester James V. Miranda +5Reward Modeling LLM Evaluation
Department of Computer Science, Stanford University · Department of Computer Science, Boston University · Allen Institute for AI +4
Enriching Patent Claim Generation with European Patent Dataset
May 18, 2025Lekang Jiang, Chengzu Li, Stephan GoetzText Generation LLM Evaluation
University of Cambridge
Can LLM-based Financial Investing Strategies Outperform the Market in Long Run?
May 11, 2025Weixian Waylon Li, Hyeonjun Kim, Mihai Cucuringu +1LLM Evaluation Quantitative Finance
AIAI, School of Informatics The University of Edinburgh Edinburgh, United Kingdom · Global Finance Research Center Sungkyunkwan University Seoul, Republic of Korea · Dept. of Mathematics; Dept. of Statistics & OMI University of California, Los Angeles; University of Oxford United States; United Kingdom
The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs
Apr 24, 2025Piotr Nawrot, Robert Li, Renjie Huang +3Transformer Inference LLM Evaluation
University of Edinburgh · Cohere · Meta
Evaluating LLMs on Chinese Topic Constructions: A Research Proposal Inspired by Tian et al. (2024)
Apr 21, 2025Xiaodong YangLLM Evaluation Language Model Probing
College of Foreign Languages Zhejiang University of Technology 288, Liuhe Rd., Xihu District, Hangzhou, Zhejiang, 310023 China
DMind Benchmark: Toward a Holistic Assessment of LLM Capabilities across the Web3 Domain
Apr 18, 2025Enhao Huang, Pengyu Sun, Shuxun Wang +13LLM Evaluation Software Vulnerability Detection
Zhejiang University Hangzhou, China · The State Key Laboratory of Blockchain and Data Security · DMind.ai Singapore +1
SAGE: A Search-AuGmented Evaluation of Large Language Models on Free-Form QA
Apr 10, 2025Sher Badshah, Ali Emami, Hassan SajjadLLM Evaluation Language Model Generation Evaluation
Dalhousie University · Emory University
Physics: Benchmarking Foundation Models on University-Level Physics Problem Solving
Mar 26, 2025Kaiyue Feng, Yilun Zhao, Yixin Liu +4LLM Evaluation Physical Reasoning
New York University · Yale University · Notre Dame University
Scaling Evaluation-time Compute with Reasoning Models as Evaluators
Mar 25, 2025Seungone Kim, Ian Wu, Jinu Lee +8LLM Evaluation Language Model Generation Evaluation
CMU · UIUC2 · KAIST AI3 +3
No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding
Mar 7, 2025Michael Krumdick, Charles Lovering, Varshini Reddy +2LLM Evaluation LLM-as-a-Judge
Kensho Technologies · MIT
AnnoCaseLaw: Legal Annotation Benchmark of Civil Negligence
Feb 28, 2025Rilton Franzone, Magnus Sesodia, Alina Petrova +7Legal NLP LLM-Assisted Annotation
University of Oxford · Thomson Reuters Labs · Vienna University of Technology +1
Large Language Models for Cryptocurrency Transaction Analysis: A Bitcoin Case Study
Jan 30, 2025Yuchen Lei, Yuexin Xiang, Rafael Dowsley +5LLM Evaluation LLMs for Cybersecurity
School of Cyber Science and Engineering, Wuhan University, Wuhan 430072, China · Faculty of Information Technology, Monash University, Clayton, VIC 3800, Australia · CSIRO’s Data61, Eveleigh, NSW 2015, Australia +2
Humanity's Last Exam
Jan 24, 2025Long Phan, Alice Gatti, Ziwen Han +1178LLM Evaluation Language Model Reasoning Evaluation
Michael Pokorny · Quinn · Tony +1
Compound-QA: A Benchmark for Evaluating LLMs on Compound Questions
Nov 15, 2024Yutao Hou, Yajing Luo, Zhiwen Ruan +4Multi-Hop QA LLM Evaluation
Shanghai University of Finance and Economics · Southern University of Science and Technology · University of Edinburgh +1
Growing a Tail: Increasing Output Diversity in Large Language Models
Nov 5, 2024Michal Shur-Ofry, Bar Horowitz-Amsalem, Adir Rahamim +1LLM Evaluation LLM Prompting
Law Faculty, Hebrew University of Jerusalem; Jerusalem, Israel. · Faculty of Computer Science, Technion – Israel Institute of Technology; Haifa, Israel.
Evaluation of OpenAI o1: Opportunities and Challenges of AGI
Sep 27, 2024Tianyang Zhong, Zhengliang Liu, Yi Pan +74LLM Evaluation Code Generation
Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation
Sep 1, 2024Jasper Dekoninck, Maximilian Baader, Martin VechevLLM Evaluation
Department of Computer Science, ETH Zurich, Switzerland
VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models
Jul 16, 2024Haodong Duan, Xinyu Fang, Junming Yang +41VLM Evaluation LLM Evaluation

Jul 14, 2025J/K move · Enter open · S save

Detoxify: A framework for abusive text transformation using LLMs

Rohitash Chandra, Jiyong Choi, Jayesh Sonawane

Transitional Artificial Intelligence Research Group, School of Mathematics and Statistics, UNSW, Sydney, Australia · Centre for Artificial Intelligence and Innovation, Pingla Institute, Sydney, Australia · Centre for Artificial Intelligence and Innovation, Pingla

LLM Evaluation Text Detoxification

Open paper AskPDF arXiv

Although Large Language Models (LLMs) have demonstrated significant advancements in natural language processing tasks, their effectiveness in the classification and transformation of abusive text into non-abusive versions remains an area for exploration. In this study, we present Detoxify: a framework that employs LLMs to transform abusive text (tweets and reviews) containing hate speech and profanity into non-abusive text while retaining the original intent. We evaluate the performance of four state-of-the-art LLMs, such as Gemini, GPT-4o, DeekSeek and Groq, on their ability to identify abusive text. We aim to transform and obtain a text that is clean of abusive and inappropriate content, but maintains a similar level of sentiment and semantics, i.e. the transformed text needs to maintain its message. Afterwards, we evaluate the raw and transformed datasets with sentiment analysis and semantic analysis. Our results show Groq provides vastly different results when compared with other LLMs. We have identified similarities between GPT-4o and DeepSeek. Groq stood out as the most distinct, as it often restructured sentences with excessive positive phrasing, with the original context lost or altered.