LLM Evaluation

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

194 papers in the last four weeks, up 87% on the four weeks before. 1.9% of all new papers.

Jul 13Week of Sep 28

Latest papers 1,643

All topics

Paraphrase-Induced Output-Mode Collapse: When LLMs Break Character Under Semantically Equivalent Inputs
May 6, 2026Aofan Liu, Jingxiang MengPrompt Sensitivity LLM Evaluation
1Peking University. · University of Chicago.
Benchmarking LLMs on the Massive Sound Embedding Benchmark (MSEB)
May 6, 2026Cyril Allauzen, Tom Bagby, Georg Heigold +2Audio-Language Model Evaluation LLM Evaluation
Google USA & Germany
CANDI: Contextual Alignment for Niche Domains Question Answering
May 6, 2026Megha Chakraborty, Darssan L. Eswaramoorthi, Het Riteshkumar Shah +5LLM Evaluation LLM Grounding
University of South Carolina · IIIT-Delhi · Indian Institute of Technology Madras
How Does Thinking Mode Change LLM Moral Judgments? A Controlled Instant-vs-Thinking Comparison Across Five Frontier Models
May 6, 2026Sai Sourabh MadurLLM Evaluation Moral Reasoning in Language Models
Independent researcher (ex-Meta)
Deployment-Relevant Alignment Cannot Be Inferred from Model-Level Evaluation Alone
May 6, 2026Varad Vishwarupe, Nigel Shadbolt, Marina Jirotka +1LLM Evaluation LLM Alignment
Department of Computer Science, University of Oxford
From Memorization to Creation: Evaluating the Cognitive Depth of LLM-Generated Educational Questions
May 6, 2026Xiaolong Wang, Zhe Zhao, Song Lai +5LLM Evaluation LLM Prompting
Squirrel Ai Learning Shanghai, China · City University of Hong Kong Hong Kong, China · Zhejiang Normal University Squirrel Ai Learning Jinhua, China +2
Evaluating Advanced Prompting on Gemini Flash for Multi-Hop Biomedical QA
May 5, 2026Ahmed Bajaber, Mohammed AlliheediMulti-Hop QA LLM Evaluation
1Saudi Med AI Lab (SMAIL) 2Prince Sultan University, Riyadh, Saudi Arabia · 2Prince Sultan University, Riyadh, Saudi Arabia · 3Al-Baha University, Al Baha, Saudi Arabia
Towards Self-Referential Analytic Assessment: A Profile-Based Approach to L2 Writing Evaluation with LLMs
May 5, 2026Stefano Bannò, Kate Knill, Mark GalesLLM Evaluation Automated Essay Scoring
ALTA Institute, Department of Engineering, University of Cambridge (UK)
ReplaySCM: A Benchmark for Executable Causal Mechanism Induction from Interventions
May 5, 2026Serafim BatzoglouLLM Evaluation Structural Causal Models
Not All That Is Fluent Is Factual: Investigating Hallucinations of Large Language Models in Academic Writing
May 5, 2026Humam Khan, Md Tabrez Nafis, Shahab Saquib Sohail +2LLM Evaluation LLM Prompting
Department of Computer Science and Engineering, SEST Jamia Hamdard New Delhi, India · Energy, Resources and Industry WSP Calgary, Canada
Frontier Lag: A Bibliometric Audit of Capability Misrepresentation in Academic AI Evaluation
May 5, 2026David Gringras, Misha SalahshoorLLM Evaluation LLM Auditing
Harvard University · AISST, Cambridge, MA
The Counterexample Game: Iterated Conceptual Analysis and Repair in Language Models
May 5, 2026Daniel Drucker, Kyle MahowaldLLM Self-Correction LLM Evaluation
Department of Philosophy The University of Texas at Austin · Department of Linguistics The University of Texas at Austin
EvoLM: Self-Evolving Language Models through Co-Evolved Discriminative Rubrics
May 5, 2026Shuyue Stella Li, Rui Xin, Teng Xiao +8RL for Language Models LLM Evaluation
University of Washington · 2Allen Institute for AI · University of Pennsylvania
MCJudgeBench: A Benchmark for Constraint-Level Judge Evaluation in Multi-Constraint Instruction Following
May 5, 2026Jaeyun Lee, Junyoung Koh, Zeynel Tok +2LLM Evaluation LLM-as-a-Judge
University of Oxford · Yonsei University
TriBench-Ko: Evaluating LLM Risks in Judicial Workflows
May 5, 2026Haesung Lee, Gyubin Choi, Eun-Ju Lee +5LLM Evaluation Language Model Safety Evaluation
Center for Trustworthy AI, Seoul National University · Graduate School of Data Science, Seoul National University · Responsible AI Team, KT +1
OracleProto: A Reproducible Framework for Benchmarking LLM Native Forecasting via Knowledge Cutoff and Temporal Masking
May 5, 2026Yiding Ma, Chengyun Ruan, Kaibo Huang +2LLM Evaluation Forecasting Benchmarks
Beijing University of Posts and Telecommunications
PatRe: A Full-Stage Office Action and Rebuttal Generation Benchmark for Patent Examination
May 5, 2026Qiyao Wang, Xinyi Chen, Longze Chen +4LLM Evaluation Legal Document Analysis
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Dalian University of Technology +2
Revisiting Graph-Tokenizing Large Language Models: A Systematic Evaluation of Graph Token Understanding
May 5, 2026Zhongjian Zhang, Yue Yu, Mengmei Zhang +3LLM Evaluation LLM Reasoning with Graphs
Beijing University of Posts and Telecommunications · China Telecom Bestpay · Beihang University
Revisiting the Travel Planning Capabilities of Large Language Models
May 5, 2026Bo-Wen Zhang, Jin Ye, Peng-Yu Hua +4LLM Self-Correction LLM Evaluation
State Key Laboratory of Novel Software Technology, Nanjing University, China · School of Intelligence Science and Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China
Evaluating Prompting and Execution-Based Methods for Deterministic Computation in LLMs
May 4, 2026Hongkun YuLLM Evaluation LLM Prompting
Virginia Tech Blacksburg, VA, USA
Benchmarking Local Language Models for Social Robots using Edge Devices
May 4, 2026Dorian Lamouille, Matevž B. Zorec, Farnaz Baksh +1LLM Evaluation On-Device Language Model Inference
University of Tartu Tartu, Estonia · Institute of Computer Science University of Tartu Tartu, Estonia · Institute of Technology University of Tartu Tartu, Estonia
Evaluating Reasoning Models for Queries with Presuppositions
May 4, 2026Rose Sathyanathan, Kinshuk Vasisht, Danish PruthiLLM Evaluation Language Model Robustness
Indian Institute of Science Bengaluru, KA, India
SemEval-2026 Task 7: Everyday Knowledge Across Diverse Languages and Cultures
May 4, 2026Nedjma Ousidhoum, Junho Myung, Carla Perez-Almendros +27Multilingual Language Model Evaluation LLM Evaluation
Cardiff University · KAIST · MBZUAI +13
HalluScan: A Systematic Benchmark for Detecting and Mitigating Hallucinations in Instruction-Following LLMs
May 4, 2026Ahmed CherifLLM Evaluation LLM Hallucination Mitigation
1Sofrecom Tunisia, Orange Innovation, Tunis, 1053, Tunisia.
When Correct Isn't Usable: Improving Structured Output Reliability in Small Language Models
May 4, 2026Cosimo Galeone, Minsu Park, Giuseppe Ettorre +1LLM Evaluation LLM Prompting
Alomana, Grottaglie, Italy
Complexity Horizons of Compressed Models in Analog Circuit Analysis
May 4, 2026Pacome Simon MbonimpaLLM Evaluation LLM Compression
Carnegie Mellon University-Africa
Submodular Benchmark Selection
May 4, 2026Alexander SmolaSubmodular Optimization LLM Evaluation
Boson AI 4677 Old Ironsides Dr Santa Clara CA 95054
Retrieval and Multi-Hop Reasoning in 1M-Token Context Windows: Evaluating LLMs on Classical Chinese Text
May 4, 2026Eric H. C. ChowLong-Context Retrieval In-Context Retrieval
School of Humanities, The University of Hong Kong
CLaC at SemEval-2026 Task 6: Response Clarity Detection in Political Discourse
May 4, 2026Nawar Turk, Lucas Miquet-Westphal, Leila KosseimLLM Evaluation Language Model Ensembles
Computational Linguistics at Concordia (CLaC) Lab · Dept. of Computer Science and Software Engineering · Concordia University, Montréal, Québec, Canada
So Many Opinions, So Many LLMs: Comparing Large Language Models to Traditional Machine Learning for Open- Ended Survey Analysis
May 4, 2026Abdullah Akinde, Mariam Akinde, Rasheedat Emiola +1LLM Evaluation LLM Interpretability
Austin Peay State University, United States

May 6, 2026, cs.CLJ/K move · Enter open · S save

Paraphrase-Induced Output-Mode Collapse: When LLMs Break Character Under Semantically Equivalent Inputs

Aofan Liu, Jingxiang Meng

1Peking University. · University of Chicago.

Prompt Sensitivity LLM Evaluation LLM Reliability Structured Output Generation Language Model Robustness

Open paper AskPDF arXiv

When the substantive content of a request is rewritten, do large language models still answer in the format the original task asked for? We find that they often do not, even at temperature zero. On a 150-query evaluation over five compact 2025-era LLMs and four task types, we observe a systematic failure mode we call prompt-variant output-mode collapse: when a closed-form prompt asks for a bare label or a single choice token, content-preserving prompt variants can push the model into conversational prose, the requested format dissolves, and exact-match evaluation pipelines silently misjudge the result. To make this measurable, we release PARACONSIST, a 900-prompt benchmark of 150 base queries with five lexical, syntactic, and semantic-expansion prompt variants each, and a Semantic Consistency Score that decomposes prompt-variant robustness into answer consistency, sentence-BERT semantic similarity, and length stability. Under a whole-word answer-set match, only ~22% of closed-form variant responses preserve the ground-truth label inside their output, while ~78% drift away from the answer space entirely. In our pool, the dominant predictor of collapse is task structure rather than model identity, with model differentiation jointly carried by answer consistency and length stability. Robustness audits should therefore track response-mode preservation as a first-class reliability target alongside answer accuracy.

Momentum

Related topics

Latest papers 1,643