LLM Reliability

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

78 papers in the last four weeks, up 105% on the four weeks before. 0.8% of all new papers.

Jul 13Week of Sep 28

Latest papers 507

All topics

Can LLM-as-a-Judge Reliably Verify Rubrics in Agentic Scenarios?
Jun 29, 2026Yangda Peng, Yunjia Qi, Haotian Xia +8LLM-as-a-Judge AI Agent Benchmarks
Department of Computer Science and Technology, Tsinghua University · Tencent Hunyuan
Toward Secure and Reliable PDDL Formalization of Large Language Models with Planner-in-the-Loop Feedback
Jun 29, 2026Jiamei Jiang, Jiajing Zhang, Feifei Mo +2LLM Planning LLM Reliability
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · School of Industry-education Integration, University of Chinese Academy of Sciences, Beijing, China
Citation Discipline in Spec-Driven Development: A Cross-Model Empirical Study of Output Determinism and Automated Hallucination Detection in LLM-Generated Code
Jun 28, 2026Subham PandaCode Generation LLM Reliability
Reliability-Prioritized Fine-Grained Generation in Multimodal Large
Jun 28, 2026Xiaomeng Fan, Wei Wu, Yuwei Wu +9Direct Preference Optimization Multimodal Large Language Models
Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science & Technology, Beijing Institute of Technology · Guangdong Laboratory of Machine Perception and Intelligent Computing, Shenzhen MSU-BIT University · Department of Electrical and Computer System Engineering, Monash University
Diagnosing and Repairing Factual Errors in RAG under Budget Constraints
Jun 28, 2026Soroush Hashemifar, Havva Alizadeh Noughabi, Fattane Zarrinkalam +1Retrieval-Augmented Generation LLM Reliability
College of Engineering, University of Guelph, Guelph, ON, Canada · Cyber Science Lab, School of Computer Science, University of Guelph, Guelph, ON, Canada
Travel-Oriented Reasoning Large Language Model via Domain-Specific Knowledge Graphs
Jun 28, 2026Vignesh Ram Nithin Kappagantula, Shayan Hassantabar, Samuel Simpson +1LLM Grounding LLM Reasoning with Graphs
Expedia Group Seattle, Washington, USA
AB-RAG: Adaptive Budgeted Retrieval-Augmented Generation for Reliable Question Answering
Jun 27, 2026Ansh KamthanRetrieval-Augmented Generation LLM Answer Verification
Department of Artificial Intelligence and Machine Learning Manipal University Jaipur Jaipur, India
When Can Conformal Risk Control Certify LLM Outputs? Bounds, Impossibility, and Adaptation for Structured Generation
Jun 27, 2026Varun KotteConformal Prediction LLM Reliability
Independent Researcher
The strength of clinical evidence is recoverable from language model representations but not from their stated grades
Jun 27, 2026Soroosh Tayebi ArastehLLM Interpretability Language Model Calibration
Lab for AI in Medicine, RWTH Aachen University, Aachen, Germany · Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen · Department of Urology, Stanford University, Stanford, CA, USA +1
When More Sampling Hurts: The Modal Ceiling and Correlation Ceiling of Test-Time Scaling
Jun 27, 2026Yong Yi Bay, Kathleen A. YearickLLM Evaluation Self-Consistency Decoding
University of Illinois at Urbana-Champaign
Correct codes for the wrong reasons? validating LLMs as measurement instruments for theoretical constructs
Jun 26, 2026Manuel PitaLLM Evaluation Psychometric Validation
Artificial Intelligence, Social Interaction and Complexity Laboratory CICANT, Universidade Lusófona 376, Campo Grande, 1700-097 Lisbon, Portugal
Towards Reliable and Robust LLM Planning: Symbolic Feedback-Driven Iterative Self-Refinement Framework
Jun 26, 2026Jiajing Zhang, Jiamei Jiang, Chenyang Zhang +3LLM Self-Refinement LLM Planning
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · School of Industry-education Integration, University of Chinese Academy of Sciences, Beijing, China
When are likely answers right? On Sequence Probability and Correctness in LLMs
Jun 25, 2026Johannes Zenn, Jonas GeipingLLM Evaluation Language Model Decoding
Max Planck Institute for Intelligent Systems · ELLIS Institute Tübingen · AI Center Tübingen +2
LLM-based Models for Detecting Emerging Topics in Service Feedback
Jun 25, 2026Mahsa Tavakoli, Ruth Bankey, Cristián BravoHuman-in-the-Loop AI LLM Reliability
Department of Statistical and Actuarial Sciences, The University of Western Ontario, London, Ontario, Canada. · Revenue Agency.
Can Large Language Models Reliably Code Qualitative Humanitarian Data? A Benchmark Study Against Human Expert Adjudication
Jun 25, 2026Jerome Marston, Tino Kreutzer, Salomé Garnier +3LLM Evaluation Human-in-the-Loop Annotation
Kobo Inc · McGill University · Harvard University
Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix It
Jun 24, 2026Yupu Hao, Zhuoran Jin, Huanxuan Liao +2Supervised Fine-Tuning LLM Reliability
1The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artiﬁcial Intelligence, University of Chinese Academy of Sciences, Beijing, China
Necessary but Not Sufficient: Temperature Control and Reproducibility in LLM-as-Judge Safety Evaluations
Jun 24, 2026Hiroki TambaLLM Evaluation LLM-as-a-Judge
Tamba Research Academy
Cliff Tokens: Analyzing Failure Trigger Tokens in LLM Mathematical Reasoning
Jun 24, 2026Jaeyong Ko, Jinu Lee, Pilsung Kang +1Numerical Reasoning in Language Models LLM Reliability
Seoul National University · Boston University
Reclaim Evaluation: A Lossy Memory Is Worse Than an Empty One
Jun 24, 2026Alex KwonLLM Evaluation Memory-Augmented Language Models
Independent Researcher
LLM-Based Scientific Peer Review: Methods, Benchmarks, and Reliability Challenges
Jun 23, 2026Thi Huyen Nguyen, Zahra AhmadiScholarly Peer Review LLM Security
L3S Research Center, Leibniz University Hannover Hannover, Germany · Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School Lower Saxony Center for AI and Causal Methods in Medicine (CAIMed) Hannover, Germany
Grad Detect: Gradient-Based Hallucination Detection in LLMs
Jun 23, 2026Anand Kamat, Daniel Blake, Brent M. WernessLLM Reliability LLM Hallucination Detection
Amazon
Beyond Logprobs: A Multi-Signal Confidence Engine for LLM-Based Document Field Extraction
Jun 23, 2026Nitesh KumarSelective Prediction Document Information Extraction
1Perfios Software Solutions Pvt. Ltd.
Age of LLM: A Strategic 1v1 Benchmark for Reasoning, Diplomacy and Reliability of Large Language Models under Fog of War
Jun 23, 2026Arnaud RicciStrategic Reasoning Risks in Language Models AI Agent Benchmarks
Independent researcher, Switzerland.
On the Stability of Prompt Ranking in Large Language Model Evaluation
Jun 23, 2026Shaoshuai Du, Penghao Liang, Yixian Shen +3LLM Evaluation LLM Prompting
University of Amsterdam, Amsterdam, Netherlands · Northeastern University, Boston, MA, USA · University of California San Diego, La Jolla, CA, USA +1
Pigeonholing: how bad prompts hurt models, causing collapse and mistakes
Jun 23, 2026Hyunji Nam, Keertana Chidambaram, Dorottya Demszky +1LLM Reliability In-Context Learning
Stanford University · University of Washington
VeryTrace: Verifying Reasoning Traces through Compilable Formalism and Structured Verification
Jun 23, 2026Ninghan Zhong, Ahmet Ege Tanriverdi, Kaan Kale +1Formal Verification Structured Reasoning
School of Electrical and Computer Engineering, Georgia Institute of Technology, USA · Department of Electrical and Computer Engineering, Bogazici University, Turkey
Maestro Order: A Model-Agnostic Orchestration Harness
Jun 22, 2026Hidayet AksuLLM Agent Orchestration LLM Reliability
Can LLMs Reliably Self-Report Adversarial Prefills, and How?
Jun 22, 2026Quang Minh Nguyen, Uzair Ahmed, Taegyoon KimLanguage Model Safety Evaluation Adversarial Attacks on LLMs
KAIST
The Model as One Rater Among Several: Measuring Political Positions in Data-Sparse Regions with a Language-Model Panel
Jun 22, 2026Tarek GaraInter-Rater Reliability LLM-Assisted Annotation
*Independent researcher.
When Preferences Fail to Become Incentives: A Utility-Behavior Gap in Large Language Models
Jun 22, 2026Yujun Zhou, Christopher M. AckermanLLM Evaluation LLM Alignment

Jun 29, 2026, cs.CLJ/K move · Enter open · S save

Can LLM-as-a-Judge Reliably Verify Rubrics in Agentic Scenarios?

Yangda Peng, Yunjia Qi, Haotian Xia, Guanzhong He+7

Department of Computer Science and Technology, Tsinghua University · Tencent Hunyuan

LLM-as-a-Judge AI Agent Benchmarks LLM Reliability Rubric-Based Evaluation

Open paper AskPDF arXiv

Rubric-based scoring has become a widely used paradigm in model evaluation, typically with LLM-as-a-Judge (LaaJ) for rubric scoring. However, the reliability of LaaJ for rubric scoring remains underexplored. This concern is especially pronounced in agentic scenarios, where long, complex outputs further challenge reliable scoring. To address this, we conduct a systematic meta-evaluation of LaaJ reliability for rubric verification. We introduce RuVerBench, the first benchmark for assessing LaaJ reliability in rubric verification for agentic scenarios. RuVerBench covers two prevalent agentic domains, deep research and agentic coding, with 2,458 instances, each containing a model-generated output, a rubric, and a human-annotated label indicating whether the output satisfies the rubric. Using RuVerBench, we evaluate numerous frontier LLMs and find that even the most advanced models achieve strong performance but still exhibit substantial noise. We further analyze the impact of key LaaJ strategies, including prompt design, batching, and majority voting, on rubric verification. We find that weaker models are more sensitive to prompt variations, batched verification presents a trade-off between accuracy and efficiency, and majority voting yields effective but diminishing returns. We have released our dataset and code to facilitate future research: https://github.com/THU-KEG/RuVerBench.

Momentum

Related topics

Latest papers 507