LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,631

All topics

ASCIIBench: Evaluating Language-Model-Based Understanding of Visually-Oriented Text
Dec 2, 2025Kerry Luo, Michael Fu, Joshua Peguero +6VLM Evaluation LLM Evaluation
Algoverse AI Research
A Comparative Study of Student Perspectives on Technical Writing Feedback Quality: Evaluating LLMs, SLMs, and Humans in Computer Science Topics
Dec 1, 2025Suqing Liu, Runlong Ye, Christopher Eaton +2LLM Evaluation Computer Science Education
McMaster University, Hamilton, Ontario, Canada · Department of Computer Science, University of Toronto, Toronto, Ontario, Canada · Research Institute for the Study of University Pedagogy, University of Toronto Mississauga, Mississauga, Ontario, Canada +1
ORCA: Open-ended Response Correctness Assessment for Audio Question Answering
Nov 28, 2025Šimon Sedláček, Sara Barahona, Bolaji Yusuf +9Audio-Language Model Evaluation LLM Evaluation
AUDIAS, Universidad Autónoma de Madrid, Spain · University of Buenos Aires, Argentina · Tufts University, USA +1
Large language models replicate and predict human cooperation across experiments in game theory
Nov 6, 2025Andrea Cera Palatsi, Samuel Martin-Gutierrez, Ana S. Cardenal +1LLM Evaluation Game Theory
Department for Computational Social Sciences and Humanities, Barcelona Supercomputing Center · Grupo de Sistemas Complejos, Universidad Polit´ecnica de Madrid · School of Law and Political Science, Universitat Oberta de Catalunya +1
Sequential Bayesian Evaluation of Large Language Model Behavior
Nov 4, 2025Saatvik Kher, Shang Wu, Rachel Longjohn +2LLM Evaluation Language Model Generation Evaluation
Department of Computer Science, University of California, Irvine · Department of Statistics, University of California, Irvine
Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities
Nov 4, 2025Amanda Bertsch, Adithya Pratapa, Teruko Mitamura +2LLM Evaluation Long-Context Language Model Evaluation
Carnegie Mellon University, Language Technologies Institute
A Quantitative Study of Sustained Focus in Large Language Models via Repetitive Deterministic Prediction Tasks
Nov 2, 2025Wanda Hou, Leon Zhou, Hong-Ye Hu +3LLM Evaluation LLM Reliability
EdenCode Inc. · Stevenson School · Harvard University +2
Idea2Plan: Exploring AI-Powered Research Planning
Oct 28, 2025Jin Huang, Silviu Cucerzan, Sujay Kumar Jauhar +1LLM Evaluation LLM-as-a-Judge
University of Michigan · Microsoft Research
Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)
Oct 27, 2025Liwei Jiang, Yuanjun Chai, Margaret Li +7Open-Ended Generation LLM Evaluation
University of Washington · Allen Institute for Artificial Intelligence · Carnegie Mellon University +3
Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks
Oct 26, 2025Peiyu Li, Xiuxiu Tang, Si Chen +4LLM Evaluation Psychometric Validation
University of Notre Dame, Notre Dame, USA
User Perceptions vs. Proxy LLM Judges: Privacy and Helpfulness in LLM Responses to Privacy-Sensitive Scenarios
Oct 23, 2025Xiaoyuan Wu, Roshni Kaushik, Wenkai Li +2LLM Evaluation LLM-as-a-Judge
Fujitsu Research of America, Inc. · Carnegie Mellon University
CreativityPrism: A Cross-Domain Evaluation Framework for Large Language Model Creativity
Oct 23, 2025Zhaoyi Joey Hou, Bowei Alvin Zhang, Yining Lu +9Creativity Assessment LLM Evaluation
University of Pittsburgh · Johns Hopkins University · University of Notre Dame +4
The Chameleon Nature of LLMs: Quantifying Multi-Turn Stance Instability in Search-Enabled Language Models
Oct 19, 2025Shivam Ratnakar, Sanjay RaghavendraRetrieval-Augmented Generation LLM Evaluation
University of Southern California
BuildArena: A Physics-Aligned Interactive Benchmark of LLMs for Engineering Construction
Oct 18, 2025Tian Xia, Tianrun Gao, Wenhao Deng +4LLM Evaluation LLM Grounding
Westlake University, China · Uniforce AI, China
LiveOIBench: Can Large Language Models Outperform Human Contestants in Informatics Olympiads?
Oct 10, 2025Kaijian Zou, Aaron Xiong, Yunxiang Zhang +6LLM Evaluation Competitive Programming
University of Michigan, Ann Arbor.
StatEval: A Comprehensive Benchmark for Large Language Models in Statistics
Oct 10, 2025Yuchen Lu, Run Yang, Yichen Zhang +6LLM Evaluation Reasoning Evaluation
1Shanghai University of Finance and Economics
Large Language Model Selection with Limited Annotations
Oct 10, 2025Yavuz Durmazkeser, Patrik Okanovic, Andreas Kirsch +2Model Selection LLM Evaluation
TU Delft · ETH Zurich
oMeBench: Towards Robust Benchmarking of LLMs in Organic Mechanism Elucidation and Reasoning
Oct 9, 2025Ruiling Xu, Yifan ZhangLLM Evaluation Benchmark Design
University of Illinois Urbana-Champaign
Psychological Imagination Networks Show Cross-Population Centrality and Clustering Alignment in Humans That Large Language Models Fail to Replicate
Oct 5, 2025Saurabh Ranjan, Brian OdegaardLLM Evaluation
University of Florida
When Guessing is Rewarded: Rethinking Language Model Evaluation with Distributional Uncertainty Scoring
Oct 5, 2025Thomas F BurnsLLM Evaluation Language Model Generation Evaluation
Aleph Alpha Research
What and Whose Knowledge? Measuring Epistemic Diversity in Large Language Models
Oct 5, 2025Dustin Wright, Sarah Masud, Jared Moore +5LLM Evaluation Cultural Bias in Language Models
University of Copenhagen · Stanford University · University of Colorado Boulder +1
Uncovering the Computational Ingredients of Human-Like Representations in LLMs
Oct 1, 2025Zach Studdiford, Timothy T. Rogers, Kushin Mukherjee +1LLM Evaluation LLM Fine-Tuning
University of Wisconsin–Madison · Stanford University
Rethinking Reward Models for Multi-Domain Test-Time Scaling
Oct 1, 2025Dong Bok Lee, Seanie Lee, Sangwoo Park +12Reward Modeling LLM Evaluation
KAIST · TH Nürnberg · Microsoft Research Asia
Predicting Effects, Missing Distributions: Evaluating LLMs as Human Behavior Simulators in Operations Management
Sep 30, 2025Runze Zhang, Xiaowei Zhang, Mingyang ZhaoLLM Evaluation Human Behavior Simulation
Department of Industrial Engineering and Decision Analytics, Hong Kong University of Science and Technology, Clear Water Bay, Hong Kong
When LLMs Benchmark Themselves: Deconstructing Self-Bias in Automated Evaluation
Sep 30, 2025Wenda Xu, Sweta Agrawal, Vilém Zouhar +2LLM Evaluation Language Model Generation Evaluation
Google · ETH Zurich
IMProofBench: Benchmarking AI on Research-Level Mathematical Proof Generation
Sep 30, 2025Johannes Schmitt, Gergely Bérczi, Jasper Dekoninck +57Mathematical Reasoning Benchmarks LLM Evaluation
Predicting LLM Reasoning Performance with Small Proxy Model
Sep 25, 2025Woosung Koh, Juyoung Suk, Sungjun Han +2Language Model Pretraining LLM Evaluation
Trillion Labs · KAIST AI · ▽Trillion Labs, ♠KAIST AI
Polarity Detection of Sustainable Development Goals in News Text
Sep 24, 2025Andrea Cadeddu, Alessandro Chessa, Vincenzo De Leo +5LLM Evaluation Sentiment Analysis
Linkalab s.r.l., Viale Elmas, 142, Cagliari, 09122, Italy · Department of Mathematics and Computer Science, University of Cagliari, via Ospedale2026 72, Cagliari, 09124, Italy · Knowledge Media Institute, The Open University, Walton Hall, Kents Hill, Milton Keynes, MK76AA, United Kingdom
Clotho: Measuring Task-Specific Pre-Generation Test Adequacy for LLM Inputs
Sep 22, 2025Juyeon Yoon, Somin Kim, Robert Feldt +1LLM Evaluation LLM Reliability
Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea · Chalmers University of Technology, Gothenburg, Sweden
LLMs and their Limited Theory of Mind: Evaluating Mental State Annotations in Situated Dialogue
Sep 2, 2025Katharine Kowalyshyn, Matthias ScheutzLLM Evaluation Theory of Mind
Department of Computer Science Tufts University

Dec 2, 2025, cs.LGJ/K move · Enter open · S save

ASCIIBench: Evaluating Language-Model-Based Understanding of Visually-Oriented Text

Kerry Luo, Michael Fu, Joshua Peguero, Husnain Malik+5

Algoverse AI Research

VLM Evaluation LLM Evaluation

Open paper AskPDF arXiv

Large language models (LLMs) have demonstrated several emergent behaviors with scale, including reasoning and fluency in long-form text generation. However, they continue to struggle with tasks requiring precise spatial and positional reasoning. ASCII art, a symbolic medium where characters encode structure and form, provides a unique probe of this limitation. We introduce ASCIIBench, a novel benchmark for evaluating both the generation and classification of ASCII-text images. ASCIIBench consists of a filtered dataset of 5,315 class-labeled ASCII images and is, to our knowledge, the first publicly available benchmark of its kind. Alongside the dataset, we release weights for a fine-tuned CLIP model adapted to capture ASCII structure, enabling the evaluation of LLM-generated ASCII art. Our analysis shows that cosine similarity over CLIP embeddings fails to separate most ASCII categories, yielding chance-level performance even for low-variance classes. In contrast, classes with high internal mean similarity exhibit clear discriminability, revealing that the bottleneck lies in representation rather than generational variance. These findings position ASCII art as a stress test for multimodal representations and motivate the development of new embedding methods or evaluation metrics tailored to symbolic visual modalities. All resources are available at https://github.com/ASCIIBench/ASCIIBench.

Momentum

Related topics

Latest papers 1,631