Language Model Generation Evaluation

Follow this topic Ask about this topic

Momentum

30 papers in the last four weeks, up 100% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 209

All topics

Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelines
Apr 25, 2026Sadman Kabir SoumikDebiased ML LLM-as-a-Judge
Independent Researcher
BLAST: Benchmarking LLMs with ASP-based Structured Testing
Apr 24, 2026Manuel Alejandro Borroto Santana, Erica Coppolillo, Francesco Calimeri +3Logic Programming LLM Evaluation
University of Calabria, Rende, Italy · ICAR-CNR, Rende, Italy · Artificial Intelligence LAB, DeMaCS, University of Calabria, Rende, Italy +1
Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models
Apr 23, 2026Mohammed Safi Ur Rahman Khan, Sanjay Suryanarayanan, Tushar Anand +1VLM Evaluation VLM Robustness
♠Nilekani Centre at AI4Bharat ♢Indian Institute of Technology Madras · ♠Nilekani Centre at AI4Bharat · ♣BITS Pilani, Hyderabad
HoWToBench: Holistic Evaluation for LLM's Capability in Human-level Writing using Tree of Writing
Apr 21, 2026Andrew Zhuoer Feng, Cunxiang Wang, Yu Luo +7LLM Evaluation LLM-as-a-Judge
Department of Computer Science and Technology, Tsinghua University
Beyond One Output: Visualizing and Comparing Distributions of Language Model Generations
Apr 20, 2026Emily Reif, Claire Yang, Jared Hwang +3Open-Ended Generation Data Visualization
University of Washington, Seattle, WA, USA · Allen Institute for AI
Benchmarking System Dynamics AI Assistants: Cloud Versus Local LLMs on CLD Extraction and Discussion
Apr 20, 2026Terry LeitchLLM Quantization Dynamical Systems
Ruxton AI
VIBE: Voice-Induced open-ended Bias Evaluation for Large Audio-Language Models via Real-World Speech
Apr 19, 2026Yi-Cheng Lin, Yusuke Hirota, Sung-Feng Huang +1Audio-Language Model Evaluation Gender Bias in Language Models
Graduate Institute of Communication Engineering National Taiwan University Taipei, Taiwan · NVIDIA Research NVIDIA Taipei, Taiwan · AI Center of Research Excellence National Taiwan University Taipei, Taiwan
Configuration Over Selection: Hyperparameter Sensitivity Exceeds Model Differences in Open-Source LLMs for RTL Generation
Apr 18, 2026Minghao Shao, Zeng Wang, Weimin Fu +5LLM Evaluation Language Model Generation Evaluation
NYU Tandon School of Engineering, USA · Kansas State University, USA · NYU Abu Dhabi, UAE
Spotlights and Blindspots: Evaluating Machine-Generated Text Detection
Apr 17, 2026Kevin Stowe, Kailash PatilAI-Generated Text Detection Language Model Generation Evaluation
Pindrop
Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations
Apr 16, 2026Manan Gupta, Dhruv KumarLLM Evaluation LLM-as-a-Judge
1BITS Pilani, Pilani Campus, India
IUQ: Interrogative Uncertainty Quantification for Long-Form Large Language Model Generation
Apr 16, 2026Haozhi Fan, Jinhao Duan, Kaidi XuUncertainty Quantification Language Model Generation Evaluation
University of Pennsylvania · 2UNC Chapel Hill · 3City University of Hong Kong
Filtered Reasoning Score: Evaluating Reasoning Quality on a Model's Most-Confident Traces
Apr 13, 2026Manas Pathak, Xingyao Chen, Shuozhe Li +2LLM Evaluation Language Model Generation Evaluation
University of Texas at Austin
Polyglot Teachers: Evaluating Language Models for Multilingual Synthetic Data Generation
Apr 13, 2026Lester James V. Miranda, Ivan Vulić, Anna KorhonenSupervised Fine-Tuning Multilingual Language Models
Language Technology Lab, University of Cambridge
Persona Matters: Effects of Activation Steering on Short Answer Generation and Scoring
Apr 8, 2026Yongchao Wu, Aron HenrikssonOpen-Ended Generation Language Model Steering
Stockholm University, NOD-huset, Borgarfjordsgatan 12, 16455 Stockholm, Sweden
ACES: Who Tests the Tests? Leave-One-Out AUC Consistency for Code Generation
Apr 5, 2026Hui Sun, Yun-Ji Zhang, Zheng Xie +4Language Model Generation Evaluation Code Generation Evaluation
National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China
When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Models
Mar 27, 2026Juan Gabriel Kostelec, Qinghai GuoLLM Inference Efficiency LLM Evaluation
Huawei Switzerland · ACS Lab, Huawei Technologies
SpreadsheetArena: Decomposing Preference in LLM Generation of Spreadsheet Workbooks
Feb 16, 2026Srivatsa Kundurthy, Clara Na, Michael Handley +5LLM Evaluation Language Model Generation Evaluation
Longitude Labs Inc., New York, NY · Cornell University, Ithaca, NY · Carnegie Mellon University, Pittsburgh, PA +1
CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation
Feb 9, 2026Jitian Zhao, Changho Shin, Tzu-Heng Huang +2LLM Evaluation LLM-as-a-Judge
Department of Statistics, University of Wisconsin–Madison, Madison, WI, USA · Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI, USA.
Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations
Jan 30, 2026Dani Roytburg, Matthew Bozoukov, Matthew Nguyen +3LLM-as-a-Judge Language Model Generation Evaluation
Department of Machine Learning, Carnegie Mellon University, Pittsburgh, PA, USA · Apart Research, San Francisco, California, USA · Department of Computer Science and Engineering, University of California San Diego, La Jolla, CA, USA +2
Quantifying the Effect of Test Set Contamination on Generative Evaluations
Jan 7, 2026Rylan Schaeffer, Joshua Kazdan, Baber Abbasi +8Language Model Generation Evaluation Benchmark Contamination
Stanford Computer Science · Stanford Statistics · EleutherAI +3
Sequential Bayesian Evaluation of Large Language Model Behavior
Nov 4, 2025Saatvik Kher, Shang Wu, Rachel Longjohn +2LLM Evaluation Language Model Generation Evaluation
Department of Computer Science, University of California, Irvine · Department of Statistics, University of California, Irvine
Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)
Oct 27, 2025Liwei Jiang, Yuanjun Chai, Margaret Li +7Open-Ended Generation LLM Evaluation
University of Washington · Allen Institute for Artificial Intelligence · Carnegie Mellon University +3
When Guessing is Rewarded: Rethinking Language Model Evaluation with Distributional Uncertainty Scoring
Oct 5, 2025Thomas F BurnsLLM Evaluation Language Model Generation Evaluation
Aleph Alpha Research
When LLMs Benchmark Themselves: Deconstructing Self-Bias in Automated Evaluation
Sep 30, 2025Wenda Xu, Sweta Agrawal, Vilém Zouhar +2LLM Evaluation Language Model Generation Evaluation
Google · ETH Zurich
Towards AI-Assisted Research Writing: Benchmarking LLMs for AI/ML Introduction Generation
Aug 19, 2025Krishna Garg, Firoz Shaik, Sambaran Bandyopadhyay +1LLM Evaluation Language Model Generation Evaluation
University of Illinois Chicago · Adobe Research
Evaluating Style-Personalized Text Generation: Challenges and Directions
Aug 8, 2025Anubhav Jangra, Bahareh Sarrafzadeh, Silviu Cucerzan +2LLM-as-a-Judge Language Model Generation Evaluation
Columbia University, USA · Microsoft, USA · The University of York, UK
R3: Robust Rubric-Agnostic Reward Models
May 19, 2025David Anugraha, Zilu Tang, Lester James V. Miranda +5Reward Modeling LLM Evaluation
Department of Computer Science, Stanford University · Department of Computer Science, Boston University · Allen Institute for AI +4
SAGE: A Search-AuGmented Evaluation of Large Language Models on Free-Form QA
Apr 10, 2025Sher Badshah, Ali Emami, Hassan SajjadLLM Evaluation Language Model Generation Evaluation
Dalhousie University · Emory University
Scaling Evaluation-time Compute with Reasoning Models as Evaluators
Mar 25, 2025Seungone Kim, Ian Wu, Jinu Lee +8LLM Evaluation Language Model Generation Evaluation
CMU · UIUC2 · KAIST AI3 +3

Apr 25, 2026, cs.AIJ/K move · Enter open · S save

Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelines

Sadman Kabir Soumik

Independent Researcher

Debiased ML LLM-as-a-Judge Language Model Generation Evaluation Language Model Bias Evaluation Bias in LLM-as-a-Judge

Open paper AskPDF arXiv

LLM-as-a-Judge has become the dominant paradigm for evaluating language model outputs, yet LLM judges exhibit systematic biases that compromise evaluation reliability. We present a comprehensive empirical study comparing nine debiasing strategies across five judge models from four provider families (Google, Anthropic, OpenAI, Meta), three benchmarks (MT-Bench n=400, LLMBar n=200, custom n=375), and four bias types. Our headline practical finding is that a mid-tier model with the right debiasing can outperform frontier judges at a fraction of the cost: Gemini 2.5 Flash with the Combined Budget strategy reaches the highest agreement of any configuration we tested (71.0%, kappa=0.549) at ~

0.001 per evaluation, about 15x cheaper than the best frontier setup (Claude Sonnet 4, 69.5%, ~

0.015). Other key findings: (1) Style bias is the dominant bias (0.10-0.76 across models, favoring markdown over plain prose), far exceeding position bias (<=0.04), yet is rarely studied. (2) Verbosity bias is heterogeneous when measured length-aware: Pro, Flash, and Llama prefer longer answers (+0.24 to +0.44), Claude prefers concise (-0.12), and GPT-4o is neutral (-0.04); on truncation controls all models correctly prefer the complete response (0.88-1.00 accuracy). (3) Debiasing helps multiple models: Claude S8 (+11.5 pp), Flash S8 (+7.5 pp), and Claude S5 (+7.3 pp) survive Holm-Bonferroni correction, with Flash S1 (+4.7 pp) and Llama S8 (+4.5 pp) also significant. We release our evaluation framework, the 375-pair controlled dataset, and per-instance cached results for all nine strategies.

Momentum

Related topics

Latest papers 209