LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,631

All topics

The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societies
Date pendingJiaxu Zhou, Jen-tse Huang, Xuhui Zhou +5LLM Evaluation LLM Auditing
CUHK · JHU · CMU +3
Evidence for Limited Metacognition in LLMs
Date pendingChristopher AckermanLLM Evaluation LLM Uncertainty Estimation
KoSimpleQA: A Korean Factuality Benchmark with an Analysis of Reasoning LLMs
Date pendingDonghyeon Ko, Kyubyung Chae, Yeguk Jin +6Multilingual Language Model Evaluation LLM Evaluation
Naver Cloud · Graduate School of Data Science, Seoul National University
Measuring Pragmatic Influence in Large Language Model Instructions
Date pendingYilin Geng, Omri Abend, Eduard Hovy +1LLM Evaluation LLM Prompting
University of Melbourne · Hebrew University of Jerusalem
Evaluating LLM-Simulated Conversations in Modeling Inconsistent and Uncollaborative Behaviors in Human Social Interaction
Date pendingRyo Kamoi, Ameya Godbole, Binglin Zhou +5LLM Evaluation Human Behavior Simulation
Penn State University · University of Southern California · Microsoft Corporation
Inverse Turing Bench: Evaluating Language Models as Judges of Human vs. AI Dialogue
Date pendingWilliam Hager, Ishika Rathi, Masum Hasan +1LLM Evaluation AI-Generated Text Detection
University of Rochester · Independent Researcher · Stony Brook University
Efficient Adaptation of LLMs for Hate Speech Detection in Low-Resource Languages: A Comparative Study on Roman Urdu
Date pendingToneema Zubair, Muhammad Junaid Asif, Faisal Kamiran +2LLM Evaluation Hate Speech Detection
Department of Computer Sciences Information Technology University Lahore (54000), Pakistan · Artificial Intelligence Technology Centre National Centre for Physics Islamabad (44000), Pakistan
Ordinary, Reasonable Chatbots: Do AI Models Track Human Legal Judgments?
Date pendingNirav Patel, Emily Wenger, Christopher BuccafuscoLLM Evaluation Demographic Bias in Language Models
Duke University, Departments of Computer Science & Electrical and Computer Engineering · Duke University Law School
LatentMD: Benchmarking Markdown Boundary Failures in LLM-Generated Text
Date pendingSungjune Lee, Myungjoo KangLLM Evaluation Structured Output Generation
Seoul National University
Accuracy is Not Enough: A Divergence-Based Approach to Evaluate Fidelity Loss in Quantized LLMs
Date pendingShahzeb Qamar, Lorenz Sparrenberg, Christian Bauckhage +5LLM Quantization LLM Evaluation
Fraunhofer IAIS, Germany · PhenoRob Cluster of Excellence, University of Bonn, Germany · Lamarr Institute for Machine Learning and Artificial Intelligence, Germany +4
Limitations of Automated Simulatability: LLM Simulators Can Bypass Explanations
Date pendingAntonin Poché, Fanny Jourdan, Nils Feldhus +6LLM Evaluation Automated Evaluation
IRT Saint Exupéry · IRIT, Université de Toulouse · University of Groningen +7

Date pending, cs.CLJ/K move · Enter open · S save

The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societies

Jiaxu Zhou, Jen-tse Huang, Xuhui Zhou, Man Ho Lam+4

CUHK · JHU · CMU+3

LLM Evaluation LLM Auditing Human Behavior Simulation Agent-Based Modeling Large Language Model-Based Social Simulation

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used to simulate human collective behavior, yet claims that such simulations are human-like remain largely untested. We conducted a systematic audit (pre-registered on OSF) of LLM-based social simulations across four databases (Scopus, IEEE Xplore, ACM Digital Library, and arXiv). Across 576 studies reported in 350 recent papers, we applied six methodological evaluations: agent Profile, Interaction, Memory, Minimal-Control, Unawareness, and Realism (PIMMUR). Coding every study against pre-specified rules, we revealed that PIM were met more often than MUR. Frontier LLMs correctly identified the underlying social experiment in 65.2% of cases, and 50.6% of prompts imposed constraints that pre-determined the outcome. These compliance rates are upper bounds, because incomplete methodological reporting (for example, unreleased prompts) limits the available evidence. Reproducing five representative experiments (e.g., opinion dynamics), we found that reported collective phenomena often vanish or reverse once PIMMUR principles are enforced, indicating that many "emergent" behaviors are methodological artifacts rather than genuine social dynamics. Current LLM simulations may therefore capture model-specific biases rather than universal features of human social behavior, raising concerns about their use as scientific proxies for human society.

Momentum

Related topics

Latest papers 1,631