LLM Reliability

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

78 papers in the last four weeks, up 105% on the four weeks before. 0.8% of all new papers.

Jul 13Week of Sep 28

Latest papers 507

All topics

PRIME: Evaluating Prompt Resolution Under Incompatible Instructions in LLMs
Jun 21, 2026Tehreem Javed, Shumaim Fatimah, Masooma Bakhtiari +2LLM Evaluation Language Model Generation Evaluation
School of Electrical Engineering and Computer Science (SEECS), National University of Sciences and Technology (NUST), Islamabad, Pakistan
BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories
Jun 21, 2026Shreyas KCMultilingual Language Model Evaluation LLM-as-a-Judge
Denoising Iterative Self-Correction: Structured Verification Loops for Reliable LLM Reasoning
Jun 19, 2026Shen Yin, David Ken, Joel StremmelLLM Self-Correction LLM Self-Refinement
Thomson Reuters Labs
Evaluating LLMs for Real-World Web Vulnerability Detection
Jun 19, 2026Sebastian Neef, Luca Jungnickel, Antonio Benjamin Buchholz +2Software Vulnerability Detection LLM Reliability
Technische Universität Berlin, Einsteinufer 17, 10587 Berlin, Germany · Freie Universität Berlin, Kaiserswerther Str. 16-18, 14195 Berlin, Germany
Peeking Inside LLMs: Leveraging Internal Artifacts of LLMs for Enhancing Reliability in Legal Classification
Jun 18, 2026Sudipta Santra, Debtanu Datta, Saptarshi GhoshLegal NLP Language Model Error Detection
Indian Institute of Technology Kharagpur, Kharagpur, 721302, West Bengal, India
Can LLMs Reason About Brand Ownership? An Empirical Study of Domain Attribution Intelligence
Jun 18, 2026Fathima Mashood, Mohamed NabeelLLM Evaluation LLM Reliability
Confidence Calibration for Multimodal LLMs: An Empirical Study through Medical VQA
Jun 18, 2026Yuetian Du, Yucheng Wang, Ming Kong +4Multimodal QA Language Model Calibration
College of Computer Science and Technology, Zhejiang University · School of Computer Science and Technology, Xidian University · Zhihui Medical Technology (Shanghai) Co., Ltd., Shanghai, China +1
A Systematic Evaluation of Black-Box Uncertainty Estimation Methods for Large Language Models
Jun 18, 2026Jiayi Wang, Xu-Yao ZhangLLM Evaluation Uncertainty Quantification
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, Beijing 100049, China · State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias
Jun 17, 2026Justin D. Norman, Michael U. Rivera, D. Alex HughesInter-Rater Reliability LLM Evaluation
UC Berkeley School of Information 102 South Hall Drive, Berkeley CA, 94708
Possible or Definite? A Benchmark for Evaluating Diagnostic Uncertainty Preservation in Clinical Text
Jun 16, 2026Hongbo Du, Zixin Lu, Jiaming QuLLM Reliability Clinical Language Model Evaluation
Trine University · University of Michigan · Amazon
Security and Privacy Prompts in the Wild: What Users Ask LLMs and How LLMs Respond
Jun 16, 2026Hobin Kim, Xiaoyuan Wu, Omer Akgul +2LLM Auditing LLM Reliability
Carnegie Mellon University · RSAC Labs
From Drift to Coherence: Stabilizing Beliefs in LLMs
Jun 16, 2026SongEun Kim, Seungyoo Lee, Edwin Fong +2LLM Fine-Tuning Bayesian Inference
Department of Statistics, Seoul National University · Korea Advanced Institute of Science & Technology · The University of Hong Kong +1
Prompt Perturbation for Reliable LLM Evaluation over Comparison Graphs
Jun 16, 2026Dong Huang, Jianbo Sun, Pengkun YangLLM Evaluation Pairwise Comparison
Department of Statistics and Data Science, Tsinghua University
Understanding LLMs in Title-Abstract Screening: From Disagreements to Recommendations
Jun 16, 2026Mika Mäntylä, Patricia Matsubara, Katia Romero Felizardo +5LLM Evaluation Evidence Selection
University of Helsinki, Finland · UFMS, Brazil · UTFPR – Federal University of Technology - Paraná, Brazil +3
Learning to Refine Hidden States for Reliable LLM Reasoning
Jun 16, 2026Chia-Hsuan Hsu, Jui-Ming YaoRL for Language Model Reasoning Efficient Language Model Reasoning
The Consistency Dilemma in LLMs: Generator-Evaluator Agreement and Vulnerability to Mistakes
Jun 16, 2026Marina Mancoridis, Zoë HitzigLLM Evaluation Reasoning Consistency in Language Models
Massachusetts Institute of Technology · Harvard Society of Fellows
Quantifying Consistency in LLM Logical Reasoning via Structural Uncertainty
Jun 15, 2026Baishali Chaudhury, Mengdie Flora Wang, Hyunji Hayley Park +3Reasoning Consistency in Language Models LLM Reliability
AWS Generative AI Innovation Center
Islamic Large Language Models: From Knowledge Acquisition to Trustworthy and Hallucination-Resistant AI
Jun 15, 2026Mohammed Amine MouhoubQuestion Answering LLM Reliability
Paris Dauphine University
Using AI in engineering education: a balancing act, driven by clear purpose
Jun 15, 2026Olya KudinaAI in Education Generative AI in Education
TU Delft, The Netherlands · Optentia Unit, North-West University, South Africa
Know Your Limits : On the Faithfulness of LLMs as Solvers and Autoformalizers in Legal Reasoning
Jun 15, 2026Olivia Peiyu Wang, Sanna Wong-Toropainen, Daneshvar Amrollahi +4LLM Reliability Autoformalization
UC Santa Cruz · Univ. Helsinki · CodeX, Stanford +4
Who Flips? Self- and Cross-Model Counterarguments Reveal Answer Instability in LLMs
Jun 14, 2026Nafiseh Nikeghbal, Amir Hossein Kargaran, Shaghayegh Kolli +1LLM Evaluation LLM Reliability
1Technical University of Munich · Munich Center for Machine Learning · 1Technical University of Munich 2 Munich Center for Machine Learning +2
Intelligence Is Not the Bottleneck: Validating an LLM First-Pass Manuscript Score Against Peer-Review Outcomes
Jun 14, 2026Costa GeorgantasLLM Evaluation LLM Reliability
The Truth Stays in the Family: Enhancing Contextual Grounding via Inherited Truthful Heads in Model Lineages
Jun 14, 2026Miso Choi, Seonga Choi, Mincheol Kwon +3LLM Grounding LLM Interpretability
Korea University, Seoul, Republic of Korea · Kakao Mobility Corp., Seongnam, Republic of Korea
Re-feeding Is Not Replaying: Measuring Replay Noise in Counterfactual Token-Credit Estimation
Jun 14, 2026Nils MattesonCounterfactual Evaluation Token-Level Credit Assignment
Northeastern University
LLM-Assisted Stance Detection in Scientific Discourse: A Test Case in Bayesian Cognitive Science
Jun 14, 2026Eyup Engin Kucuk, Tarik Kelestemur, Ömer Dağlar TanrikuluStance Detection LLM-Assisted Annotation
University of New Hampshire · Department of Psychology · Independent Researcher
If These Walls Could Talk: Critical Play with Large Language Models in Museums
Jun 14, 2026Anders Sundnes LøvlieLarge Language Model-Based Role-Play Simulation LLM Reliability
Metric Match: A Subset Selection Approach to Evaluating LLM Judge Reliability
Jun 12, 2026Alyssa Unell, Natalie Dullerud, Naomi Boneh +4LLM-as-a-Judge Language Model Generation Evaluation
Department of Computer Science Stanford University · Department of Medicine Stanford University
CoRA: Confidence-Rationale Alignment for Reliable Chain-of-Thought Reasoning
Jun 12, 2026Juming Xiong, Weixin Liu, Kevin Guo +9CoT Reasoning RL for Language Model Reasoning
Vanderbilt University · Vanderbilt University Medical Center · Intuit AI Research
Beyond Perplexity: UTF-8 Validity in Byte-aware Language Models
Jun 12, 2026Sangwhan Moon, Daisuke Oba, Youmi Ma +2Byte-Level Language Model LLM Reliability
Google LLC / Mountain View, CA, USA · Institute of Science Tokyo / Tokyo, Japan · Mohamed bin Zayed University of Artificial Intelligence / Abu Dhabi, United Arab Emirates
Operadic consistency: a label-free signal for compositional reasoning failures in LLMs
Jun 11, 2026Nathaniel Bottman, Yinhong Liu, Kyle RichardsonCompositional Reasoning Reasoning Consistency in Language Models
Incubilate · University of Cambridge · Allen Institute for Artificial Intelligence

Jun 21, 2026, cs.AIJ/K move · Enter open · S save

PRIME: Evaluating Prompt Resolution Under Incompatible Instructions in LLMs

Tehreem Javed, Shumaim Fatimah, Masooma Bakhtiari, Gibrail Islam+1

School of Electrical Engineering and Computer Science (SEECS), National University of Sciences and Technology (NUST), Islamabad, Pakistan

LLM Evaluation Language Model Generation Evaluation LLM Prompting LLM Reliability

Open paper AskPDF arXiv

Large language models (LLMs) often encounter conflicting prompts, although current instruction following benchmarks assess those meta-instructions in isolation, limiting the insights about how models process conflicting instructions. We introduce a framework \textit{PRIME}(\textit{Prompt Resolution under Incompatible Meta-Instructions Evaluation}) to analyze behavior of LLMs when provided with conflicting instructions. \textit{PRIME} purposefully produces calibrated conflicts across response length, output format, and reasoning; classifying model responses with a deterministic behavioral taxonomy. We are evaluating five instruction tuned open weight LLMs in two distinct settings, balanced and naturally distributed. The conclusion we reach upon analysis is that conflict type is more significant in affecting behavior than model scale, and various failure modes across different categories of conflict. Our findings emphasize the value of developing conflict awareness and suggest ability of LLM to follow instructions cannot be assessed through isolated constraints alone.

Momentum

Related topics

Latest papers 507