Overview

This field examines how automated judges can favor certain answers for reasons unrelated to quality, including response order, writing style, verbosity, model identity, language, and demographic or cultural cues. Researchers use controlled counterfactual tests and benchmarks such as MT-Bench, RewardBench, and multilingual or domain-specific evaluation sets to measure when judgments change and whether they match human ratings. Mitigation approaches include randomizing presentation, adding rubrics and reference answers, calibrating scores with human comparisons, combining judges robustly, and training models to separate relevant evidence from distracting cues. A central challenge is that repeatable judgments and agreement among models do not guarantee validity, while interventions that reduce bias can also erase real quality differences or legitimate differences in human preferences.

Latest papers

CardsList

128 papers11 papers in the last four weeks, down 56% on the four weeks before. 0.1% of all new papers.

  1. Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judges

    Feb 14, 2026Ruomeng Ding, Yifei Pang, He Sun +3LLM AlignmentLLM-as-a-Judge

  2. FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge

    Feb 6, 2026Bo Yang, Lanfei Feng, Yunkui Chen +3LLM EvaluationLLM-as-a-Judge

  3. Agents-as-Peers: Bias-Controlled LLM Evaluation as Multi-Agent Peer Assessment

    Feb 1, 2026Yanki Margalit, Nurit Cohen-Inger, Erni Avram +3LLM EvaluationAI Agent Evaluation

  4. Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations

    Jan 30, 2026Dani Roytburg, Matthew Bozoukov, Matthew Nguyen +3LLM-as-a-JudgeLanguage Model Generation Evaluation

  5. Breaking the Mirror: Activation-Based Mitigation of Self-Preference in LLM Evaluators

    Sep 3, 2025Dani Roytburg, Matthew Bozoukov, Matthew Nguyen +3LLM-as-a-JudgeLanguage Model Steering