aiXamine: Unified Black-Box Evaluation of Cross-Dimensional Trade-offs in LLM Safety, Security, and Privacy

Fatih Deniz; Yazan Boshmaf; Dorde Popovic; Issa Khalil

cs.CRAug 20, 2026

aiXamine: Unified Black-Box Evaluation of Cross-Dimensional Trade-offs in LLM Safety, Security, and Privacy

Authors: Fatih Deniz, Yazan Boshmaf, Dorde Popovic, Issa Khalil

Organizations: Qatar Computing Research Institute (QCRI), HBKU, Doha, Qatar

Abstract

The critical failure modes in deployed large language models (LLMs) are cross-dimensional: a model can score 99.3 in safety alignment while refusing one in three benign queries, or improve across every capability metric while losing 21 points in privacy. Existing evaluation frameworks that assess safety, security, and privacy independently cannot detect these patterns. We introduce aiXamine, a unified black-box platform that evaluates LLM trustworthiness across safety, security, and privacy as interdependent properties. aiXamine orchestrates 46 tests across nine services through an automated red-teaming pipeline, producing hierarchical risk profiles, from prompt-level diagnostics to cross-service trade-off analytics, that enable reproducible comparison of proprietary and open-weight systems under identical conditions. Applying aiXamine to over 120 LLMs through more than 5,000 test runs, we conduct the largest joint safety, security, and privacy study to date and uncover three cross-dimensional phenomena invisible to single-axis evaluation. First, safety enforcement incurs a quantifiable safety tax: stronger alignment systematically increases over-refusal, forcing providers to choose between protection and utility. Second, privacy is near-orthogonal to other trustworthiness dimensions and not captured by standard alignment. Third, we identify and formally characterize distillation-induced robustness collapse: off-policy distillation without on-policy correction causes entropy collapse, catastrophically destroying robustness (56.9 $\to$ 2.6) on the same base architecture. These findings, compounded by diminishing returns from scale and category-dependent safety behaviors, demonstrate that trustworthiness is inherently multi-dimensional: progress along one axis does not guarantee, and can actively undermine, progress along others, yet current alignment methods treat it as a single objective.

Figures & tables

Figure 1: aiXamine ’s evaluation pipeline. Benchmark prompts and their adversarial and OOD variants are dispatched under standardized inference parameters; responses are scored by deterministic rules or multi-judge ensembles; and normalized scores are aggregated at the service, category, and subcategory levels.

Figure 2: Hierarchical structure of the SSP evaluation framework. The three dimensions form complementary levels of trustworthiness: Safety & Reliability for behavioral integrity, Security & Robustness for adversarial resilience, and Privacy & Fairness for societal trust. Each level aggregates multiple services, each comprising test suites that evaluate core competencies in model safety, security, and privacy.

Model	Adv. Robust.	Code Sec.	Fair. & Bias	Halluc.	Jail. Robust.	M&D Priv.	OOD Robust.	Over Ref.	Safety Align.	Overall Score
	Services
GPT-5	71.6	79.7	68.8	83.9	90.2	69.4	89.5	87.9	95.6	81.8
Gemini 3	77.2	74.3	76.7	82.7	70.3	65.1	91.6	94.4	93.2	80.6
Gemma4-31B	75.2	75.3	79.0	75.7	73.6	69.6	89.1	85.3	97.4	80.0
Claude-4.5	61.6	70.3	62.7	80.8	98.5	78.7	88.1	82.3	97.0	80.0
Gemma4-26B-A4B	70.1	76.4	77.5	73.2	72.7	70.8	89.5	82.3	97.3	78.9

Table 1: SSP evaluation scores for the top 20 models across nine services. No model dominates all dimensions: models exhibit distinct strength-weakness profiles, with trade-offs between safety, robustness, privacy, and utility. Extended results are in Appendix 0.E.1 ; the complete leaderboard is available through the aiXamine portal.

Figure 3: Cross-service correlation and dependency analysis.

Figure 4: Safety vs. Utility Tension Analysis. Model performance for eight key services ( $y$ -axis) against the Over Refusal ( $x$ -axis) score. A higher Over Refusal score means the model refuses fewer benign prompts. The dashed lines indicate the mean performance on each axis, dividing the models into four quadrants. The top-right quadrant represents the ideal state of high performance and high utility. The plots visually demonstrate the safety tax , where models with the highest safety (e.g., Claude-4.5, GPT-120B) have the lowest utility. Conversely, services like Model & Data Privacy show that high performance and high utility can be achieved simultaneously.

Figure 5: The performance impact of model size for open-weight families (Qwen, Gemma, Llama) that share the same post-training within each family. The plot shows a general upward trend with scale but highlights diminishing returns beyond $\sim$ 4B parameters.

Figure 6: The impact of different distillation methods on (a) Llama3.1-8B and (b) Qwen2.5-14B base models. The radar plots highlight the dramatic and divergent effects of R1 distillation in Adversarial and OOD Robustness, while the IDA method has a more nuanced, and in the case of Qwen, positive impact.

Figure 7: Per-sample cosine similarity between base and distilled model representations on 50 AdvGLUE-SST2 adversarial prompts. R1 distillation produces near-orthogonal representations, while IDA distillation maintains high alignment with the base model.

Family	Old Model	New Model	$\boldsymbol{\Delta}$ Overall	$\boldsymbol{\Delta}$ Privacy
Claude	Claude-3.5	Claude-4.5	$+4.88$	$-2.09$
ChatGPT	GPT-4o	GPT-5	$+3.84$	$-21.73$
Grok	Grok 3	Grok 4	$+2.62$	$+1.04$
Gemini	Gemini 2.0	Gemini 2.5	$-0.80$	$-0.21$
Qwen	Qwen2.5-14B	Qwen3-14B	$+3.26$	$-6.77$
Llama	Llama3.2-3B	Llama4-17B-16E	$+1.20$	$-13.53$

Table 2: Per-family comparison across model generations, showing changes in overall and privacy scores. Some comparisons span architectural changes (e.g., Llama3.2

\to

Llama4), yet privacy regression recurs in most families.

Figure 8: Generalization across five safety categories in the WildGuard test for (a) Grok and (b) GPT model families.

Service	Avg Std (%)	Avg Gap (%)
Adversarial Robustness	13.7	35.8
Model & Data Privacy	16.1	23.9
Fairness & Bias	7.2	18.4
Code Security	9.4	17.3
Hallucination	11.6	28.1
Safety Alignment	5.6	16.8

Table 3: Service-level generalization statistics across test categories. Avg Std: mean standard deviation of a model’s category-level scores within each test, averaged across models. Avg Gap: mean difference between a model’s best and worst category score within each test. Higher values indicate more fragmented, category-dependent behavior.

Panel of Table 4 — Table 4: Coverage of major frameworks and toolkits across aiXamine’s nine services, grouped by the three evaluation dimensions. denotes explicit coverage; indicates partial or implicit coverage; means no coverage.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Uniform		Healthcare		Code Asst.		Cust. Chat.		Cont. Mod.		Research
	Score	Rank	Score	Rank	Score	Rank	Score	Rank	Score	Rank	Score	Rank
GPT-5	81.8	1	81.2	2	82.2	1	82.1	2	86.0	1	77.9	3
Gemini 3	80.6	2	79.4	4	78.3	6	83.5	1	83.2	3	78.2	2
Gemma4-31B	80.0	3	79.3	5	77.9	7	82.1	3	82.9	5	77.4	4
Claude-4.5	80.0	4	81.6	1	79.8	2	79.8	6	85.4	2	77.2	5
Gemma4-26B-A4B	78.9	5	78.3	6	76.9	9	80.7	4	81.4	8	76.3	7

Appendix

Table 6: Weighted scores and rankings of the top-15 models (by Uniform score), reranked 1–15 within each stakeholder profile. Shaded cells mark rank improvements

\geq

5 relative to Uniform; red cells mark drops

\geq

	Services
Model	Adv. Robust.	Code Sec.	Fair. & Bias	Halluc.	Jail. Robust.	M&D Priv.	OOD Robust.	Over Ref.	Safety Align.	Overall Score
GPT-5	71.6	79.7	68.8	83.9	90.2	69.4	89.5	87.9	95.6	81.8
Gemini 3	77.2	74.3	76.7	82.7	70.3	65.1	91.6	94.4	93.2	80.6
Claude-4.5	61.6	70.3	62.7	80.8	98.5	78.7	88.1	82.3	97.0	80.0
GPT-4o	62.6	73.1	67.5	77.5	51.4	91.1	86.9	94.4	97.5	78.0
DeepSeek-R1	67.2	77.1	61.6	74.9	51.8	84.7	86.2	91.5	97.4	76.9

Appendix

Table 7: Extended aiXamine Leaderboard with all model sizes from the selected families, grouped by (1) API-based, (2) open-weight, and (3) distilled models. The complete leaderboard is available through the aiXamine portal.

Model	SimpleQA	TriviaQA	TruthfulQA	SelfCheckGPT	HaluEval	FaithEval	Vectara	Overall Score
GPT-5	72.55	95.43	83.80	99.58	74.57	64.34	97.11	83.91
Gemini 3	58.60	95.70	92.65	85.36	80.23	77.49	88.57	82.66
Claude-4.5	48.08	92.17	88.24	97.91	70.97	72.08	96.03	80.78
GPT-4o	36.92	95.31	85.91	89.12	70.07	67.75	97.59	77.52
Claude-3.5	53.70	91.29	76.96	95.40	59.06	52.76	95.43	74.94
DeepSeek-R1	30.93	100.00	80.64	86.19	75.04	58.35	92.90	74.86

Appendix

Table 8: aiXamine Hallucination Leaderboard

Model	CyberSecEval 3					SecCodePLT				Overall Score
Model	C	C++	Java	Php	Python	Inst	Auto	Norm	Aug	Overall Score
GPT-5	66.30	83.78	84.50	88.89	70.80	82.41	81.18	72.05	91.54	79.72
DeepSeek-R1	91.63	94.79	85.59	88.58	92.17	63.59	60.08	53.99	69.68	77.11
Gemini 3	64.02	77.22	84.93	87.65	70.37	72.72	72.24	61.12	83.84	74.27
GPT-4o	92.73	94.21	86.03	90.43	91.74	54.28	52.76	44.30	62.74	73.07
Claude-3.5	63.22	82.43	84.93	90.43	70.66	66.63	66.54	57.22	75.95	72.17

Appendix

Table 9: aiXamine Code Security Leaderboard

Model	AdvGlue					AdvGlue++					Overall Score
Model	MNLI	QNLI	QQP	RTE	SST2	MNLI	QNLI	QQP	RTE	SST2	Overall Score
Gemini 3	86.78	81.76	83.33	90.12	85.14	69.68	74.94	60.81	52.76	86.22	77.15
GPT-5	73.55	80.41	79.49	90.12	72.30	47.09	82.89	55.26	52.80	82.06	71.60
Grok 4	65.29	79.73	80.77	96.30	79.05	31.97	77.28	54.63	54.99	83.41	70.34
Grok 4.1	61.16	80.41	76.92	92.59	79.05	35.79	73.99	51.19	48.52	83.01	68.26
DeepSeek-R1	74.38	79.05	75.64	91.36	63.51	60.91	59.81	45.52	43.10	78.54	67.18

Appendix

Table 12: aiXamine Adversarial Robustness Leaderboard

Model	PAIR	Jailbroken	Cipher	Overall Score
Claude-4.5	96.00	99.52	99.88	98.46
GPT-5	75.00	98.82	96.80	90.21
Grok 4.1	38.00	99.10	97.12	78.07
Grok 4	23.00	96.77	99.72	73.17
Claude-3.5	41.00	99.07	71.55	70.54
Gemini 3	20.00	94.90	96.15	70.35

Appendix

Table 13: aiXamine Jailbreak Robustness Leaderboard

Model	Word Level		Bible		Romantic		Shakespeare		Tweet		Overall Score
Model	Aug	ShaW	p=0	p=0.6	p=0	p=0.6	p=0	p=0.6	p=0	p=0.6	Overall Score
Gemini 3	97.02	95.64	89.11	86.58	89.91	88.99	92.66	88.42	93.92	93.69	91.59
Grok 4	95.30	93.58	87.84	84.40	88.30	87.39	91.74	86.93	91.86	92.32	89.97
Grok 4.1	94.95	93.58	87.96	84.29	87.84	87.39	91.17	86.70	92.09	91.74	89.77
Grok 3	94.61	92.89	87.50	84.29	87.73	87.96	90.60	85.78	92.66	91.86	89.59
GPT-5	94.50	92.32	87.84	82.91	86.93	87.39	91.86	87.16	93.12	90.60	89.46

Appendix

Table 14: aiXamine OOD Robustness Leaderboard

Model	Adult					Gendercare			Preference		BBQ	Overall Score
Model	Sex	Race	Edu	Hours	Type	M	F	N	Lifestyle	Ideology		Overall Score
Gemini 3	90.11	92.21	71.56	92.52	58.04	74.07	74.01	73.87	4.88	86.08	92.18	76.72
GPT-5	92.22	92.45	72.92	91.52	52.82	82.09	73.36	77.16	0.00	34.81	92.41	68.78
GPT-4o	87.09	94.22	72.54	92.75	44.67	75.78	69.71	72.13	7.32	32.91	93.55	67.53
Claude-4.5	91.31	94.88	65.75	96.93	41.67	71.72	68.52	71.36	0.00	25.95	83.36	62.71
DeepSeek-R1	93.90	95.07	65.04	96.17	39.42	73.06	70.82	65.83	0.00	5.06	93.09	61.59

Appendix

Table 16: aiXamine Fairness & Bias Leaderboard

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Jun 24, 2026

RAS: Measuring LLM Safety Through Refusal Alignment

Safety evaluation of large language models (LLMs) is commonly performed by querying models with unsafe or jailbreak prompts and judging whether their outputs violate a safety policy. Although useful, output-level evaluation is expensive, sensitive to judge choice, and easily tied to fixed question banks. We propose SafeVec, a white-box evaluation procedure that measures safety from internal representations rather than generated answers. SafeVec first extracts layer-wise refusal directions from a safety-aligned reference model, then selects stable layer windows where safe and unsafe behaviors are separable, and finally scores a target model by measuring whether its hidden states align with these refusal directions under unsafe and jailbreak prompts. The resulting metric, RAS (Refusal Alignment Score), maps representation-level refusal alignment to a calibrated 0-100 safety score. Across Llama, Gemma, and Qwen model families, RAS separates aligned models from uncensored and abliterated variants, tracks output-level attack success rate, and is substantially faster than judge-based evaluation. These results suggest that refusal alignment provides a compact and efficient signal for white-box LLM safety evaluation.

Chang-Chieh Huang, Yan-Lun Chen, Chia-Mu Yu +1

National Yang Ming Chiao Tung University · Hon Hai Research Institute

LLM Safety Alignment LLM Safety LLM Safety Evaluation Refusal Behavior in Language Models LLM Safety Monitoring

Ask PDF

Jun 29, 2026

EvalSafetyGap: A Hybrid Survey and Conceptual Framework for LLM Evaluation-Safety Failures

LLM evaluation and AI safety face a shared measurement problem: benchmark scores, reward-model signals, and reported safety metrics can improve while the latent properties they are meant to represent remain difficult to verify. This paper combines a hybrid survey - a systematic search paired with narrative synthesis and separately tracked grey evidence - with a conceptual framework and a structured ten-model audit. The synthesis spans eight evidence streams: benchmark validity, dynamic evaluation, LLM-as-judge reliability, safety evaluation, jailbreak/refusal robustness, reward hacking, mechanistic interpretability, and governance/auditability, covering 2018-2026 evaluation-safety measurement work. We introduce EvalSafetyGap as an organizing hypothesis for comparing evaluation-side and alignment-side proxy failures under optimization pressure, using Goodhart's Law together with two constructs we develop here - an Instability Decomposition and an Alignment Trilemma - as tools for generating testable comparisons. The audit shows how conclusions shift when capability, behavioral safety, and governance are measured separately. In this sample (

n = 10

), the association between capability and sustained adversarial robustness is statistically indeterminate using the displayed Table 3 inputs (Pearson

r = +0.232

p = 0.520

), and the apparent open-closed safety gap is modest, driven mainly by governance and disclosure rather than behavioral robustness, and sensitive to how a single borderline model is classified; attempt-budget results are protocol dependent. Because the public evidence uses heterogeneous protocols, the audit is diagnostic rather than rank-generating. The contribution is a shared vocabulary and evidence map to support dynamic evaluation, transparent source reporting, multi-attempt safety measurement, and auditable alignment practice.

Buğra Alperen Uluırmak, Rifat Kurban

Erciyes University, Türkiye. · Abdullah Gül University, Türkiye.

AI Alignment LLM Safety Alignment LLM Safety Evaluation Benchmark Validity

Ask PDF

Aug 12, 2026

Making Your LLMs More Objective: Stabilizing LLM Safety Behavior Across Traits with Trait-Invariant Safety Tuning

Aligned large language models (LLMs) are expected to exhibit safety behavior based on the content of the user request: they should refuse unsafe requests and comply with safe ones. However, we show that the same request can elicit substantially different safety decisions under different traits assigned in the system prompt, a failure mode we call trait-induced safety variation. To measure this failure, we introduce refusal-based metrics: Trait-Induced Deviation measures dataset-level deviation from the no-trait baseline, while Trait-Induced Flip Rate measures whether the same request receives different safety decisions across traits. We then provide a representation-level analysis of the mechanism behind trait-induced safety shifts and find that traits perturb the model's safety representations within a low-dimensional subspace. To achieve trait-invariant safety, where safety behavior remains stable across traits, we introduce Trait-Invariant Safety Tuning (TIST), a simple yet effective self-distillation framework that aligns an LLM's trait-conditioned behavior with its no-trait behavior. Guided by our analysis, we further propose Trait-Subspace Neutralization (TraSN), an instantiation of TIST, which enforces invariance only within the identified trait subspace. Experiments show that TraSN improves trait-invariant safety and strengthens harmful-request safety while preserving general capability. Our results highlight traits as an important factor in LLM safety and robust model behavior.

Lang Cao

University of Illinois Urbana-Champaign

LLM Alignment Language Model Safety Evaluation LLM Safety Alignment Language Model Robustness Safety-Preserving Fine-Tuning of Language Models

Ask PDF

Abstract

LLM Evaluation

Explore similar work

RAS: Measuring LLM Safety Through Refusal Alignment

EvalSafetyGap: A Hybrid Survey and Conceptual Framework for LLM Evaluation-Safety Failures

Making Your LLMs More Objective: Stabilizing LLM Safety Behavior Across Traits with Trait-Invariant Safety Tuning