LLM-as-a-Judge

Follow this field Ask about this field

Recent momentum

emerging

0 papers in the last 28 days · 0.0% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this field, kept on the site without email delivery.

Period ending 2026-09-21

23 new papers

A weekly snapshot of new work published in LLM-as-a-Judge.

Period ending 2026-09-14

16 new papers

A weekly snapshot of new work published in LLM-as-a-Judge.

Period ending 2026-09-07

36 new papers

A weekly snapshot of new work published in LLM-as-a-Judge.

Inside this field

Focused directions

Focused topic

Large Language Model Evaluation

Within LLM-as-a-Judge

244 papersExplore

Focused topic

Llm-As-A-Judge

Within LLM-as-a-Judge

174 papersExplore

Focused topic

Benchmarking Large Language Models

Within LLM-as-a-Judge

156 papersExplore

Focused topic

Large Language Model Judges

Within LLM-as-a-Judge

104 papersExplore

Focused topic

Judges

Within LLM-as-a-Judge

81 papersExplore

679 papers

Latest in LLM-as-a-Judge

Browse all topics →

Deepchecks: Evaluating Retrieval-Augmented Generation (RAG)
May 14, 2026Assaf Gerner, Netta Madvil, Nadav Barak +11Large Language Model Evaluation Code Quality
Improving Reproducibility in Evaluation through Multi-Level Annotator Modeling
May 13, 2026Deepak Pandita, Flip Korn, Chris Welty +1Reproducibility Large Language Model Evaluation
RTLC -- Research, Teach-to-Learn, Critique: A three-stage prompting paradigm inspired by the Feynman Learning Technique that lifts LLM-as-judge accuracy on JudgeBench with no fine-tuning
May 13, 2026Andrea MorandiLlm-As-A-Judge Judges
Creativity Bias: How Machine Evaluation Struggles with Creativity in Literary Translations
May 13, 2026Kyo Gerrits, Rik van Noord, Ana Guerberof ArenasLiterature Creativity
Is Agentic AI Ready for Real-World Hardware Engineering? A Deep Dive with Phoenix-bench
May 13, 2026Qingyun Zou, Feng Yu, Hongshi Tan +2Electronic Design Automation Benchmarking Large Language Models
Discovery of Hidden Miscalibration Regimes
May 13, 2026Katarzyna Kobalczyk, Mihaela van der SchaarMulticalibration and Calibration Confidence Calibration
Robust Checkpoint Selection for Multimodal LLMs via Agentic Evaluation and Stability-Aware Ranking
May 13, 2026Qinwu Xu, Zhuoheng Li, Jessie SalasLarge Language Model Evaluation Multimodal Evaluation
VERA-MH: Validation of Ethical and Responsible AI in Mental Health
May 13, 2026Luca Belli, Kate H. Bentley, Josh Gieringer +6Mental Health Support Chatbots
PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimization
May 13, 2026Huihao Jing, Wenbin Hu, Shaojin Chen +5Benchmarking Large Language Models Code Optimization
Faithful or Fabricated? A Causal Framework for Rationalization Bias in LLM Judges
May 13, 2026Riya Tapwal, Abhishek Kumar, Carsten MapleLarge Language Model Judges Rationality
Just Ask for a Table: A Thirty-Token User Prompt Defeats Sponsored Recommendations in Twelve LLMs
May 12, 2026Andreas Maier, Jeta Sopa, Gozde Gul Sahin +2Large Language Model Evaluation Recommendation
Visual Aesthetic Benchmark: Can Frontier Models Judge Beauty?
May 12, 2026Yichen Feng, Yuetai Li, Chunjiang Liu +14Aesthetics Evaluation Benchmarks
Predicting Disagreement with Human Raters in LLM-as-a-Judge Difficulty Assessment without Using Generation-Time Probability Signals
May 12, 2026Yo EharaLlm-As-A-Judge Mean Opinion Scores
Iterative Audit Convergence in LLM-Managed Multi-Agent Systems: A Case Study in Prompt-Engineering Quality Assurance
May 12, 2026Elias CalboreanuMulti-Agent Large Language Model Algorithm Auditing
MolDeTox: Evaluating Language Model's Stepwise Fragment Editing for Molecular Detoxification
May 12, 2026Jueon Park, Wonjune Jang, Jiwoo Lee +2Toxicity Detoxification
SAGE: Scalable Automated Robustness Augmentation for LLM Knowledge Evaluation
May 12, 2026Xiaoyuan Li, Yuzhe Wang, Moxin Li +6Large Language Model Evaluation Verifier
VERDI: Single-Call Confidence Estimation for Verification-Based LLM Judges via Decomposed Inference
May 11, 2026Jasmine Qi, Danylo Dantsev, Muyang SunLarge Language Model Judges Confidence Estimation
Measuring Five-Nines Reliability: Sample-Efficient LLM Evaluation in Saturated Benchmarks
May 11, 2026Eungyeup Kim, Chenchen Gu, Vashisth Tiwari +1Large Language Model Reliability Large Language Model Evaluation
Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge
May 11, 2026Wenbo Zhang, Lijinghua Zhang, Liner Xiang +1Large Language Model Reasoning Judges
Auditing Multimodal LLM Raters: Central Tendency Bias in Clinical Ordinal Scoring
May 11, 2026Jiaqing Zhang, Sandeep Elluri, Bhanu Cherukuvada +7Mean Opinion Scores Llm-As-A-Judge
Navigating the Sea of LLM Evaluation: Investigating Bias in Toxicity Benchmarks
May 11, 2026Regina Gugg, Selina Niederländer, Andreas Stöckl +1Large Language Model Evaluation Toxicity
Valid Best-Model Identification for LLM Evaluation via Low-Rank Factorization
May 11, 2026Elad Tolochinsky, Yaniv Tenzer, Yaniv RomanoLarge Language Model Evaluation Model Selection
IndustryBench: Probing the Industrial Knowledge Boundaries of LLMs
May 11, 2026Songlin Bai, Xintong Wang, Linlin Yu +12Large Language Model Evaluation First-Price Auctions
Calibrate, Don't Curate: Label-Efficient Estimation from Noisy LLM Judges
May 10, 2026Yanran LiJudges Multicalibration and Calibration
CHAINTRIX: A multi-pipeline LLM-augmented framework for automated smart-contract security auditing
May 10, 2026Gabriela Dobrita, Simona-Vasilica Oprea, Adela BaraSmart Contracts Large Language Model Safety
Two Ways to De-Bias an LLM-as-a-Judge: A Continuous-Score Comparison of Hierarchical Bayesian Calibration and Neural-ODE Score Transport
May 9, 2026Andrea MorandiLlm-As-A-Judge Neural Ordinary Differential Equations
Forge: Quality-Aware Reinforcement Learning for NP-Hard Optimization in LLMs
May 9, 2026Xiaozhe Li, Xinyu Fang, Shengyuan Ding +5Reasoning Benchmark Reinforcement Learning With Verifiable Reward
Coordinates of Capability: A Unified MTMM-Geometric Framework for LLM Evaluation
May 8, 2026Adib Sakhawat, Tahsin Islam, Takia Farhin +3Large Language Model Evaluation Evaluation Metrics
Magis-Bench: Evaluating LLMs on Magistrate-Level Legal Tasks
May 8, 2026Ramon Pires, Thales Sales Almeida, Celio Larcher Junior +6Legal Domain Llm-As-A-Judge
Alignment as Jurisprudence
May 8, 2026Nicholas CaputoInterpretive Openness Judges
Augmenting Human Evaluation with LLM Judges: How Many Human Reviews Do You Need?
May 8, 2026Jane Paik KimLarge Language Model Evaluation Large Language Model Judges
Benchmarking EngGPT2-16B-A3B against Comparable Italian and International Open-source LLMs
May 8, 2026Andrea Sassella, Andrea Chizzola, Tommaso Bianchi +2Benchmarking Large Language Models Italian Case Studies
LLMSYS-HPOBench: Hyperparameter Optimization Benchmark Suite for Real-World LLM Systems
May 8, 2026Siyu Wu, Yulong Ye, Zezhen Xiang +3Benchmarking Large Language Models Hyperparameter Optimization
Multi-Dimensional Evaluation of LLMs for Grammatical Error Correction
May 8, 2026Adnan Labib, Qiao Wang, Yixuan Huang +1Grammatical Error Correction Large Language Model Evaluation
Efficient Data Selection for Multimodal Models via Incremental Optimization Utility
May 8, 2026Jinhao Jing, Qiannian Zhao, Chao Huang +1Llm-As-A-Judge
SAGE: Hierarchical LLM-Based Literary Evaluation through Ontology-Grounded Interpretive Dimensions
May 8, 2026Tianyu Wang, Nianjun ZhouLiterature Large Language Model Evaluation
NSMQ Riddles: A Benchmark of Scientific and Mathematical Riddles for Quizzing Large Language Models
May 8, 2026George Boateng, Naafi Ibrahim, Samuel John +8Benchmarking Large Language Models Reasoning Benchmark
An Interpretable and Scalable Framework for Evaluating Large Language Models
May 7, 2026Xinhao Qu, Qiang Heng, Hao Zeng +1Large Language Model Evaluation Item Response Theory
Bias and Uncertainty in LLM-as-a-Judge Estimation
May 7, 2026James FiedlerLlm-As-A-Judge Raw Judge Outputs
PrefixGuard: From LLM-Agent Traces to Online Failure-Warning Monitors
May 7, 2026Xinmiao Huang, Jinwei Hu, Rajarshi Roy +3Action Traces Large Language Model Judges
Measuring Evaluation-Context Divergence in Open-Weight LLMs: A Paired-Prompt Protocol with Pilot Evidence of Alignment-Pipeline-Specific Heterogeneity
May 7, 2026Florian A. D. Burnat, Brittany I. DavidsonLarge Language Model Evaluation Heterogeneity
Towards Annotation-Free Validation of MLLMs: A Vision-Language Logical Consistency Metric
May 7, 2026Ying Gu, Mei Chee Leong, Hui Li Tan +3Large Language Model Evaluation Textvqa
Beyond Accuracy: Policy Invariance as a Reliability Test for LLM Safety Judges
May 7, 2026Shihao Weng, Yang Feng, Xiaofei XieLlm-As-A-Judge Judges
Towards Reliable LLM Evaluation: Correcting the Winner's Curse in Adaptive Benchmarking
May 7, 2026Yang Xu, Jiefu Zhang, Haixiang Sun +3Large Language Model Evaluation Tuning
Evaluation Awareness in Language Models Has Limited Effect on Behaviour
May 7, 2026Amelie Knecht, Lucas Florin, Thilo HagendorffLarge Language Model Evaluation Moral Reasoning
Revealing Modular Gradient Noise Imbalance in LLMs: Calibrating Adam via Signal-to-Noise Ratio
May 7, 2026Ziqing Wen, Zhouyang Liu, Jiahuan Wang +4Adam Benchmarking Large Language Models
Multi-Dimensional Behavioral Evaluation of Agentic Stock Prediction Systems Using Large Language Model Judges with Closed-Loop Reinforcement Learning Feedback
May 7, 2026Mohammad Al Ridhawi, Mahtab Haj Ali, Hussein Al OsmanAgentic Evaluations Agentic Systems
DataDignity: Training Data Attribution for Large Language Models
May 7, 2026Xiaomin Li, Andrzej Banburski-Fahey, Jaron LanierLarge Language Model Evaluation Training Data Attribution
MRI-Eval: A Tiered Benchmark for Evaluating LLM Performance on MRI Physics and GE Scanner Operations Knowledge
May 6, 2026Perry E. RadauDiagnostic Benchmark Large Language Model Evaluation
Misaligned by Reward: Socially Undesirable Preferences in LLMs
May 6, 2026Gayane Ghazaryan, Esra DönmezLarge Language Model Alignment Large Language Model Bias
KernelBenchX: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernels
May 6, 2026Han Wang, Jintao Zhang, Kai Jiang +3Graphics Processing Unit Kernels Benchmarking Large Language Models
BenCSSmark: Making the Social Sciences Count in LLM Research
May 6, 2026Arnault Chatelain, Étienne Ollion, Qianwen Guan +7Benchmarking Large Language Models Computational Social Science
Benchmarking LLMs on the Massive Sound Embedding Benchmark (MSEB)
May 6, 2026Cyril Allauzen, Tom Bagby, Georg Heigold +2Large Audio Language Models Audio Latent
RaguTeam at SemEval-2026 Task 8: Meno and Friends in a Judge-Orchestrated LLM Ensemble for Faithful Multi-Turn Response Generation
May 6, 2026Ivan Bondarenko, Roman Derunets, Oleg Sedukhin +3Large Language Model Judges Ensemble
Probing Structural Mathematical Reasoning in Language Models with Algebraic Trapdoors
May 5, 2026Igor RivinMathematical Reasoning Computable
Frontier Lag: A Bibliometric Audit of Capability Misrepresentation in Academic AI Evaluation
May 5, 2026David Gringras, Misha SalahshoorArtificial Intelligence Evaluation Large Language Model Evaluation
EvoLM: Self-Evolving Language Models through Co-Evolved Discriminative Rubrics
May 5, 2026Shuyue Stella Li, Rui Xin, Teng Xiao +8Large Language Model Evaluation Rubrics
Measuring AI Reasoning: A Guide for Researchers
May 4, 2026Munachiso Samuel Nwadike, Zangir Iklassov, Kareem Ali +2Reasoning Traces Large Language Model Evaluation
MolViBench: Evaluating LLMs on Molecular Vibe Coding
May 4, 2026Jiatong Li, Yuxuan Ren, Weida Wang +4Vast Chemical Space Code Generation
Submodular Benchmark Selection
May 4, 2026Alexander SmolaSubmodular Maximization Benchmarking Large Language Models

May 14, 2026 · cs.AIJ/K move · Enter open · S save

Deepchecks: Evaluating Retrieval-Augmented Generation (RAG)

Assaf Gerner, Netta Madvil, Nadav Barak, Alex Zaikman+10

Deepchecks, Ramat Gan, Israel · Ben-Gurion University, Beer Sheva, Israel

Large Language Models (LLMs) augmented with Retrieval-Augmented Generation (RAG) techniques are revolutionizing applications across multiple domains, such as healthcare, finance, and customer service. Despite their potential, evaluating RAG systems remains a complex challenge due to the stochastic nature of generated outputs and the intricate interplay between retrieval and generation components. This paper introduces Deepchecks, a comprehensive framework tailored for evaluating RAG applications. Deepchecks' evaluation framework addresses RAG applications evaluation through a multi-faceted approach, root cause analysis and production monitoring. By ensuring alignment with application-specific requirements, Deepchecks framework provides a robust foundation for assessing reliability, relevance, and user satisfaction in RAG systems.

Large Language Model Evaluation Code Quality

Open paper Ask about it PDF