cs.CRMar 31, 2026
SaveSecurity in LLM-as-a-Judge: A Comprehensive SoK
Organizations: Department of Electrical, Computer and Biomedical Engineering, University of Pavia, Pavia, Italy · Department of Computer Applications, Cochin University of Science and Technology, Kerala, India
Abstract
LLM-as-a-Judge (LaaJ) is a novel paradigm in which powerful language models are used to assess the quality, safety, or correctness of generated outputs. While this paradigm has significantly improved the scalability and efficiency of evaluation processes, it also introduces novel security risks and reliability concerns that remain largely unexplored. In particular, LLM-based judges can become both targets of adversarial manipulation and instruments through which attacks are conducted, potentially compromising the trustworthiness of evaluation pipelines. In this paper, we present the first Systematization of Knowledge (SoK) focusing on the security aspects of LLM-as-a-Judge systems. We perform a comprehensive literature review across major academic databases, analyzing 863 works and selecting 45 relevant studies published between 2020 and 2026. Based on this study, we propose a taxonomy that organizes recent research according to the role played by LLM-as-a-Judge in the security landscape, distinguishing between attacks targeting LaaJ systems, attacks performed through LaaJ, defenses leveraging LaaJ for security purposes, and applications where LaaJ is used as an evaluation strategy in security-related domains. We further provide a comparative analysis of existing approaches, highlighting current limitations, emerging threats, and open research challenges. Our findings reveal significant vulnerabilities in LLM-based evaluation frameworks, as well as promising directions for improving their robustness and reliability. Finally, we outline key research opportunities that can guide the development of more secure and trustworthy LLM-as-a-Judge systems.
Figures & tables
| Symbol | Description |
|---|---|
| AA | Adversarial Attack |
| CTI | Cyber Threat Intelligence |
| DL | Deep Learning |
| FL | Federated Learning |
| GPT | Generative Pre-trained Transformer |
| ICL | In-Context Learning |
Table 1: Summary of the acronyms used in the paper
| Paper | Year | Timeline | # Papers | Definition | Evaluation | Security Aspects | Domain/Focus |
|---|---|---|---|---|---|---|---|
| Gu et al. [ 14 ] | 2024 | 2023-2025 | 220 | ● | ● | ○ | NLP, Social Intelligence, Multi-Modal Evaluation, and others |
| Xu et al. [ 57 ] | 2024 | 2020-2024 | 180 | ○ | ◐ | ◐(Security aspects related to system trustworthiness) | Benchmark Data Contamination (BDC) |
| Li et al. [ 23 ] | 2024 | 2022-2025 | 270 | ● | ● | ◐(Briefly mentioned as evaluation outputs) | Evaluation, Alignment, Retrieval, and Reasoning |
| Li et al. [ 26 ] | 2024 | 2023-2024 | 300 | ● | ● | ◐(Adversarial Attack briefly mentioned) | NLP, Multi-Modal Evaluation, Information retrieval, and others |
| Cao et al. [ 7 ] | 2025 | 2023-2025 | 370 | ◐ | ● | ○ | - |
| He et al. [ 15 ] | 2025 | 2023-2025 | 42 | ● | ● | ◐(Security aspects related to system trustworthiness) | Software Engineering |
Table 2: Survey papers related to our work
| Ref. | Year | Scope | Taxonomy | Context |
| Raina et al. [ 38 ] | 2024 | Adversarial robustness of LLMs used as assessors | Attack | Assessment and benchmarking using LLMs |
| Shi et al. [ 44 ] | 2024 | Prompt injection attack that manipulates LaaJ systems to select an attacker-chosen response | Attack | Decision-making systems using LaaJ |
| Ye et al. [ 58 ] | 2024 | Evaluate reliability of LaaJ | Evaluation | - |
| Wei et al. [ 55 ] | 2025 | Introduces Emoji Attack that exploits token segmentation bias to trick LaaJ | Attack | - |
| Abeyratne et al. [ 1 ] | 2025 | Propose an unsupervised ensemble method to evaluate LLM-generated responses | Evaluation | Law |
| Balog et al. [ 3 ] | 2025 | Empirical study that investigates biases in LaaJ within information retrieval pipelines | Evaluation | Information Retrieval |
Table 3: Summary of the main purpose of all the analysed papers
| Ref | Year | Role of the Judge | Attack Category | Attack Type | Judge Model(s) | Eval Metrics | Main Results | Benchmark |
|---|---|---|---|---|---|---|---|---|
| Raina et al. [ 38 ] | 2024 | Zero-shot assessor for text quality | Universal adversarial attack | Short concatenated phrases (2–5 tokens) | FlanT5-xl (surrogate), Mistral-7B, Llama2-7B, GPT-3.5 | ASR, score inflation, Perplexity-based detection | Inflated scores on all inputs; ASR up to 70%; perplexity detection modestly effective | SummEval, TopicalChat (comparative & absolute) |
| Shi et al. [ 44 ] | 2024 | Selects best response among candidates (search, RLAIF, tool selection) | Prompt-injection (optimization) | Gradient-based injected sequence (JudgeDeceiver) | Open-source LLMs (e.g., Llama-3-8B) & proprietary models (implicit) | Attack Success Rate (ASR), detection-rate | Very high ASR; known-answer & perplexity defenses miss 70% of attacks | LLM-Bar, MT-Bench, tool-selection benchmarks |
| Tong et al. [ 50 ] | 2025 | Scores candidate models (point-wise & pair-wise) | Backdoor poisoning | Web poisoning, malicious annotator, weight poisoning | Mistral-7B-Instruct-v0.2, Qwen1.5-7B-Chat, LlaMA-3-8B-Instruct | Score inflation, ASR | 1% poisoned data inflates scores up to 3 (1.4 4.6/5); 20% inflation under web poisoning; toxic-prompt mis-classification 89% | No explicit benchmark; real-world leader-board style evaluation |
| Maloyan et al. [ 31 ] | 2025 | Evaluates text quality, code correctness, argument strength | Prompt-injection | Basic Injection, Contextual Misdirection, Adaptive Search-Based Attack, Universal-Prompt-Injection | Gemma-3-27B-Instruct, Gemma-3-4B-Instruct, Llama-3.2-3B-Instruct, GPT-4, Claude-3-Opus | Attack Success Rate (ASR) | Max ASR 73.8% (Gemma-3-4B-Instruct, ASA) and 67.7% (Gemma-3-27B-Instruct, CM) | Five models four tasks (human-preference, search arena, MT, code review) |
| Wei et al. [ 55 ] | 2024 | Safety-risk moderation of generated content | Token-segmentation bias / jailbreak | Emoji insertion via in-context learning | Llama Guard, ShieldLM, WildGuard, GPT-3.5/4, Gemini, Claude, DeepSeek, o3-mini | Unsafe-prediction ratio ( indicates success) | Reduces detection up to 12% overall; ShieldLM drops from 71.9% to 3.5% | Ten state-of-the-art judges |
| Zhao et al. [ 64 ] | 2025 | Reward model for RLVR (verification) | “Master-key” attack | Single-token triggers (e.g., “Thought process:”) | GPT-4o, Claude-4, GPT-o1, Qwen-2.5-72B, LLaMA-3-70B | False-Positive Rate (FPR) | FPR up to 80% across 10 keys & 5 benchmarks | VerifyBench / VerifyBench-Hard |
Table 4: Summary of Threat Landscape for LLM-as-a-Judge
| Ref. | Year | System Name | Focus / Goal | Method | Datasets | Metrics | Results |
|---|---|---|---|---|---|---|---|
| Li et al. [ 24 ] | 2025 | J-Detector | Auditing and detecting automated judgments | Lightweight neural framework analyzing correlations between candidate outputs and judgment scores; uses linguistic and LLM-derived features | Multiple datasets of LLM-generated and human judgments | Accuracy, F1-score and AUC | High detectability ( accuracy, F1 & AUC ) shows LLM judgments have identifiable patterns, reveal biases, and are clearly separable from human judgments. |
| Shen et al. [ 43 ] | 2025 | GPTracker | Monitoring misuse in LLM ecosystems | Automated crawling and interaction with GPT agents; combines LLM-based scoring with manual verification | Collected GPT instances over 8 months; analyzed thousands of conversation flows | Accuracy, Precision, Recall, and F1-score | Identified widespread misuse across the GPT ecosystem, with 2,051 GPTs violating OpenAI’s terms of service based on LLM scoring and human review, achieving precision in detection. |
| Mao et al. [ 32 ] | 2025 | LLMVulExp | Explainable security analysis with LLMs | Vulnerability detection with LLMs using instruction tuning (LoRA), Chain-of-Thought reasoning, and key code extraction | Standard vulnerability datasets | Precision, Recall, and F1-score | LLMVulExp improves vulnerability detection with fine-tuned LLMs, achieving high precision, recall, F1 scores ( 90%) and delivering detailed, actionable explanations that outperform baseline LLM methods. |
Table 5: Comparison of LLM-as-a-Judge Mitigation Strategies
| Ref. | Year | Role | Application Domain | Framework Name | Benchmark / Task | Metrics | Main Results |
| Ye et al. [ 58 ] | 2024 | T | - | CALM | Multi-task LLM evaluation datasets | Robustness rate, Consistency rate | Identified 12 bias types; SOTA LLM judges exhibit systematic biases. |
| Abayratne et al. [ 1 ] | 2025 | I | General Purpose | AlignLLM | Answer Evaluation | WILRAlign | WILRAlign (ensemble of judges) has the highest correlation to ground truth compared to single judges (125 % more than the mean). Question-space alignment is higher. |
| Balog et al. [ 3 ] | 2025 | I | General Purpose | - | Evaluation of automated re-ranking methods | NDCG, Cohen’s k, Kendall’s | LaaJs are biased towards LLM-based rankers. LaaJs have higher leniency than human judges. LaaJs have low sensitivity to subtle performance differences. |
| Belcastro et al. [ 4 ] | 2025 | I | AD | KLAGE | Network monitoring | Detection accuracy, Explanation quality | Multi-hop KG + LLM improves detection and explainability. |
| Blefari et al. [ 5 ] | 2025 | I | AD | CyberRAG | SQLi, XSS, SSTI | Accuracy, BERTScore | Modular RAG pipeline; 94% classification accuracy, robust explanations. |
| Cantini et al. [ 6 ] | 2025 | I | Toxicity Detection | CLEAR-Bias | Detecting demographic bias in LLMs. | Robustness (stereotype rejection), Fairness (lack of polarization). | Bias resilience is uneven. Some small models outperform larger ones in safety. Jailbreak attacks using low-resource languages or refusal suppression are effective. Fine-tuning may increase bias. |
Table 6: Summary of LaaJ works for Security Evaluation. Role of Judge (I: Instrument, T: Target); Application Domain (AD: Attack Detection, CR-SE: Code review/ Software engineering)
Figure 6: LaaJ positions itself as a middle-ground between the expensive but flexible human annotation and cheaper but more rigid methods of evaluation.
| Dimension | Dominant Paradigm | Alternative Paradigm |
|---|---|---|
| Model access | Black box (text as output) [ 48 ] [ 8 ] [ 3 ] [ 56 ] [ 62 ] [ 47 ] [ 65 ] [ 25 ] [ 6 ] [ 21 ] [ 42 ] | White box (logits as output) Mentioned in [ 3 ] |
| Scoring method | Absolute/pointwise (assigns independent scores) [ 48 ] [ 51 ] [ 8 ] , [ 3 ] , [ 56 ] [ 62 ] [ 47 ] [ 62 ] [ 47 ] [ 25 ] [ 6 ] [ 1 ] [ 21 ] [ 42 ] | Relative / pairwise or listwise (ranks items) [ 8 ] [ 3 ] [ 65 ] |
| Deployment | Cloud (runs remotely) [ 51 ] [ 8 ] [ 3 ] [ 65 ] [ 42 ] | Local (runs on premise) [ 3 ] [ 56 ] [ 62 ] , [ 47 ] , [ 1 ] |
| Subject of Evaluation | First-order (judge a model) [ 48 ] [ 51 ] [ 8 ] [ 3 ] [ 56 ] [ 62 ] [ 47 ] [ 65 ] [ 25 ] [ 6 ] [ 1 ] [ 21 ] [ 42 ] | Higher-order (judge a judge) Used in [ 6 ] (as meta-prompt) |
| Rubric design | Ensembling (aggregate multiple criteria) [ 48 ] [ 56 ] [ 62 ] [ 47 ] [ 25 ] [ 6 ] [ 1 ] [ 21 ] | Single rubric (one criterion) [ 51 ] (but averages several runs), [ 65 ] |
| Knowledge source | Internal (using model weights, or for logical consistency) [ 48 ] , [ 51 ] (attack, no attack), [ 8 ] , [ 3 ] , [ 56 ] , [ 62 ] , [ 47 ] , [ 65 ] , [ 6 ] (but uses builtin LLM world knowledge), [ 21 ] , [ 42 ] | External (injecting domain-specific knowledge) [ 8 ] , [ 56 ] , [ 62 ] , [ 47 ] , [ 25 ] (LaaJ has access to golden answer), [ 1 ] |
Table 7: Some high-level binary dimensions that can be used to broadly classify LaaJ systems. Each dimension is orthogonal to the others.
Figure 7: Yearly Distribution of Publications
Figure 8: Number of Publications by Venue
More in this topic
LLM-as-a-Judge
Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.
See all 555 papers