StableJudge: A Lightweight Adversarial Defense Framework for LLM-as-a-Judge

Anning Hu; Yangyu Li; Yuqi Huang; Shengchao Hu; Junchi Yan

cs.CROct 12, 2026

StableJudge: A Lightweight Adversarial Defense Framework for LLM-as-a-Judge

Authors: Anning Hu, Yangyu Li, Yuqi Huang, Shengchao Hu, Junchi Yan

Organizations: Shanghai Jiao Tong University

Abstract

Large language models (LLMs) are increasingly used as judges to assign scalar scores that drive downstream decision-making, including tasks such as paper review and recommendation. However, LLM-as-a-Judge pipelines are vulnerable to score-hijacking prompt injection embedded in evaluated content, which corrupts scoring signals and distorts induced rankings. Existing defenses often enforce robustness by modifying the judge model itself, making safety dependent on a specific model's scale and training and requiring repeated effort when the judge is updated or replaced. We propose StableJudge, a lightweight defense framework that externalizes prompt-injection mitigation as a reusable pre-scoring capability. StableJudge centers on a compact 0.13B Sentry that localizes injected spans and performs targeted sanitization prior to evaluation, and is trained using adversarially evolving attacks to improve robustness beyond a fixed prompt set. To stabilize measurement, StableJudge derives deterministic, fine-grained scalar scores from the judge's output distribution. Across held-out scientific abstracts and out-of-distribution attack variants, StableJudge improves score recovery under attack while largely preserving general-purpose performance, demonstrating that robust LLM-based judging can be achieved by decoupling safety from judge scale via reusable defense capabilities. Code and data will be released.

Figures & tables

Figure 1: Overview of StableJudge for LLM-as-a-Judge. (A) Illustrative score-hijacking example. (B) Mutual red–blue training: localization misses guide new attacks, and effective attacks train Sentry. (C) Deployment: the same frozen Sentry sanitizes inputs across judge backends, aiming to recover each unchanged judge’s clean-input score.

Figure 2: StableJudge Sentry. The tagger fuses semantic and pattern features to predict

O/B/I

labels; only candidate spans with verifier probability

v_{j}\geq\tau=0.5

are sanitized. Selected windows and character-offset mapping extend the same pipeline to long inputs.

Figure 3: Mutual red–blue evolution. (a) Score-impact filtering selects effective attacks to train the tagger and verifier; the updated tagger’s miss report guides the next generation round. (b) Examples of evolved prompt-injection payloads.

Model	Params	Char IoU $\uparrow$	Damage $\downarrow$	Char IoU $\uparrow$	Damage $\downarrow$	Clean FPR $\downarrow$
		Seen Attacks		Unseen (Evolved) Attacks
Gemini-3-Flash	API	96.96%	12.75%	97.81%	13.75%	0.00%
GPT-5.2	API	98.11%	15.25%	98.63%	15.50%	0.00%
Claude-4.6-Sonnet	API	73.43%	10.50%	68.65%	10.25%	0.00%
Doubao-Seed-1.6	API	92.68%	10.75%	95.93%	13.00%	0.00%
Qwen3-0.6B (thinking)	0.6B	6.09%	2.75%	5.85%	1.75%	6.18%

Table 1: Prompt injection detection and localization on seen and unseen (evolved, held-out) attack distributions. We report mean character-level span IoU (

\uparrow

) between predicted removal spans and gold injected spans, attack-split damage rate (

\downarrow

), and clean false-positive rate (FPR,

\downarrow

Panel of Figure 4 — Figure 4: Adversarial evolution and score recovery. Left: a qualitative projection of payload formulations across rounds. Right: the primary Qwen3-8B judge’s clean, attacked, and defended scores on the same 298 effective attacks; residuals quantify recovery toward its clean-input scores.

Round	Perfect	Detected $\uparrow$	Missed $\downarrow$	Damage $\downarrow$
	Attack Removal
Round 0	0.00%	1.00%	99.00%	58.50%
Round 1	98.25%	99.50%	0.50%	8.50%
Round 2	98.00%	99.50%	0.50%	2.25%
Round 3	99.25%	99.50%	0.50%	2.25%
Round 4	99.50%	99.50%	0.50%	1.00%

Table 2: Evolution of Sentry performance across red–blue adversarial training rounds.

	Seen Attacks		Unseen Attacks
System	Recall $\uparrow$	Integrity $\uparrow$	Recall $\uparrow$	Integrity $\uparrow$
Tagger	97.75%	99.73%	99.50%	99.73%
Sentry	96.50%	100.00%	100.00%	100.00%

Table 3: Sentry pipeline ablation: tagger-only versus full localization and verification.

Table 8

Dimension	Dataset	Metric	Qwen3-8B (Base)	StableJudge (Ours)	$\Delta$
Knowledge	GPQA (Diamond)	Accuracy $\uparrow$	28.79%	28.79%	+ 0.00%
Reasoning	GSM8K (5-shot)	Exact Match $\uparrow$	88.25%	86.81%	-1.44%
Instruction	IFEval	Inst. Strict Acc $\uparrow$	48.32%	49.04%	+0.72%
Instruction	IFEval	Prompt Strict Acc $\uparrow$	33.64%	35.12%	+1.48%
General	MMLU Average	Accuracy $\uparrow$	74.72%	74.98%	+0.26%
STEM	MMLU-STEM	Accuracy $\uparrow$	75.48%	76.40%	+0.92%

Table 6: General capability retention of the StableJudge scoring agent relative to base Qwen3-8B across knowledge, reasoning, and instruction-following benchmarks.

Payload selection	All	Head	Middle	Tail	Char. coverage	Mean scans
Cue-matched	98.89%	100.00%	96.67%	100.00%	98.17%	3.03

Table 7: Full-paper localization with cue-matched payloads. All/Head/Middle/Tail report detection rates; character coverage is mean gold-span recall. Mean scans counts inspected regions per attacked paper.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Judge / cohort	$N$	$\|\Delta_{\mathrm{atk}}\|$	$\Delta_{\mathrm{rec}}$	$\|\Delta_{\mathrm{rec}}\|$	Success
Same fixed set of unseen inputs
Qwen3-8B (adapted)	400	1.040	$-0.416$	0.422	60.75%
Qwen2.5-7B-Instruct	400	0.880	$-0.039$	0.055	98.25%
Llama-3.1-8B-Instruct	400	0.390	$-0.125$	0.131	97.25%
Effective attacks, selected separately for each judge
Qwen3-8B (adapted)	298	1.103	$-0.350$	0.359	69.46%

Appendix

Table 8: Score recovery using a frozen Sentry.

|\Delta_{\mathrm{atk}}|

\Delta_{\mathrm{rec}}

, and

|\Delta_{\mathrm{rec}}|

are sample means. Success means

|\Delta_{\mathrm{rec}}|\leq 0.407007

Slice	Clean / attacked	Clean false positives	Detection	Mean removal	Perfect removal
All	805 / 200	0.99%	99.50%	99.24%	97.50%
Instruction/list-heavy	309 / 78	1.62%	100.00%	99.93%	98.72%
Code	165 / 42	0.00%	97.62%	96.51%	90.48%
Recipe	65 / 14	0.00%	100.00%	100.00%	100.00%

Appendix

Table 9: Carrier-domain localization and preservation of clean text. False-positive rate is the fraction of clean inputs edited.

Tagger encoder	Perfect removal	Clean false positives
DistilBERT	98.80%	0.00%
all-MiniLM-L6-v2	96.80%	1.61%

Appendix

Table 10: Controls on 250 cross-run attacks and 372 clean inputs. Perfect removal is at least 95% payload-token recall.

Explore similar work

FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge
Feb 6, 2026Bo Yang, Lanfei Feng, Yunkui Chen +3LLM Evaluation LLM-as-a-Judge
College of Computer Science, Zhejiang University, Hangzhou, China · School of Software Technology, Zhejiang University, Hangzhou, China
How Reliable Is Your Jailbreak Judge? Calibration and Adversarial Robustness of Automated ASR Scoring
Jun 24, 2026Yang GaoLLM Evaluation LLM-as-a-Judge
Veyon Solutions
On the Adversarial Robustness of Multimodal LLM Judges
Jun 14, 2026Zihan Wang, Guansong Pang, Zelin Liu +3LLM-as-a-Judge Adversarial Robustness
School of Computer Science and Engineering, Beihang University, Beijing, China · School of Computing and Information Systems, Singapore Management University, Singapore · State Key Laboratory of Virtual Reality Technology and System, Beihang University, Beijing, China +1

Feb 6, 2026J/K move · Enter open · S save

FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge

Bo Yang, Lanfei Feng, Yunkui Chen, Yu Zhang+2

College of Computer Science, Zhejiang University, Hangzhou, China · School of Software Technology, Zhejiang University, Hangzhou, China

LLM Evaluation LLM-as-a-Judge Language Model Bias Evaluation Bias in LLM-as-a-Judge

Open paper AskPDF arXiv

Existing LLM-as-a-Judge systems suffer from three fundamental limitations: limited adaptivity to task- and domain-specific evaluation criteria, systematic biases driven by non-semantic cues such as position, length, format, and model provenance, and evaluation inconsistency that leads to contradictory judgments across different evaluation modes (e.g., pointwise versus pairwise). To address these issues, we propose FairJudge, an adaptive, debiased, and consistent LLM-as-a-Judge. Unlike prior approaches that treat the judge as a static evaluator, FairJudge models judging behavior itself as a learnable and regularized policy. From a data-centric perspective, we construct a high-information-density judging dataset that explicitly injects supervision signals aligned with evaluation behavior. Building on this dataset, we adopt a curriculum-style SFT-DPO-GRPO training paradigm that progressively aligns rubric adherence, bias mitigation, and cross-mode consistency, while avoiding catastrophic forgetting. Experimental results on multiple internal and public benchmarks show that FairJudge consistently improves agreement and F1, reduces non-semantic biases, and outperforms substantially larger instruction-tuned LLMs. All resources will be publicly released after acceptance to facilitate future research.

Abstract

Explore similar work

FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge

How Reliable Is Your Jailbreak Judge? Calibration and Adversarial Robustness of Automated ASR Scoring

On the Adversarial Robustness of Multimodal LLM Judges