cs.LGMay 23, 2025

But what is your honest answer? Aiding LLM-judges with honest alternatives using steering vectors

Authors: Leon Eshuijs, Archie Chaudhury, Alan McBeth, Ethan Nguyen

Organizations: Vrije Universiteit Amsterdam · Independent · University of North Carolina at Charlotte

Abstract

LLM-as-a-judge is widely used as a scalable substitute for human evaluation, yet judges are least reliable on subtle dishonesty such as sycophancy and manipulation, where a judge scoring a response in isolation lacks a reference point. We introduce Judge Using Safety-Steered Alternatives (JUSSA), a framework that reduces this evaluator uncertainty using the assessed model's own internal representations. An honesty-promoting steering vector, optimized from a single training example, generates a contrastive alternative against which each response is judged. On a new manipulation dataset with human-validated response pairs, JUSSA improves AUROC for both GPT-4.1 (0.893→\rightarrow0.946) and Claude Haiku (0.859→\rightarrow0.929) judges. The gains concentrate almost entirely on the pairs a judge would otherwise misrank. They reverse once a judge has saturated a task and for the weakest judge we test, so contrastive evaluation helps most when a judge is capable enough to use the reference but has not yet saturated the task. A layer-wise analysis further locates steering effectiveness in the middle layers, where model representations of honest and dishonest prompt processing diverge. Our work demonstrates that steering vectors can serve as tools for evaluation rather than for improving model outputs at inference, opening a new direction for white-box auditing.

Figures & tables

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Context Over Content: Exposing Evaluation Faking in Automated Judges

    Apr 16, 2026Manan Gupta, Inderjeet Nair, Lu Wang +1LLM-as-a-JudgeLanguage Model Safety Evaluation

  2. FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge

    Feb 6, 2026Bo Yang, Lanfei Feng, Yunkui Chen +3LLM EvaluationLLM-as-a-Judge

  3. Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees

    Aug 18, 2026Sher Badshah, Ali Emami, Hassan SajjadLLM-as-a-JudgeSelective Prediction