Evaluating Behavioral Context for Interpretable IAM Policy Risk Scoring in Cloud Environments

Yassin Elsharkawy

cs.CROct 5, 2026

Evaluating Behavioral Context for Interpretable IAM Policy Risk Scoring in Cloud Environments

Authors: Yassin Elsharkawy

Organizations: Dept. of Computers and Artificial Intelligence Capital University Cairo, Egypt

Abstract

IAM policy analysis typically emphasizes the authorization capabilities encoded in a policy, but security analyst review priority may also depend on the behavioral and environmental context surrounding a policy event. This paper evaluates whether contextual information provides measurable incremental value for interpretable IAM policy risk prioritization beyond policy and effective-authorization information. AWS is used as the experimental cloud provider because its IAM and audit-telemetry ecosystem enables controlled evaluation using AWS IAM Context Bench, a benchmark containing 534 real AWS experimental observations across policy, environment, and behavioral scenarios, including matched cases where policy and environment remain fixed while behavioral context changes. Three Explainable Boosting Machine models are evaluated under the same leakage-controlled grouped cross-validation protocol: a policy-centric baseline, a policy-plus-environment model, and a full-context model incorporating CloudTrail telemetry. The full-context model substantially reduces analyst-priority prediction error relative to the policy-centric baseline and closely tracks the reference priority ordering. In matched same-policy context pairs, the policy-centric model remains invariant, whereas the full-context model separates benign and suspicious behavioral conditions with high directional accuracy. The results also show improved concentration of high-priority cases at the top of simulated analyst review queues. These findings indicate that behavioral and environmental context can provide useful incremental information for analyst-oriented IAM risk prioritization while preserving an interpretable additive model structure. The formulation is applicable beyond AWS conceptually, although cross-provider validation remains future work.

Figures & tables

Fig. 1: End-to-end evaluation workflow: IAM policy events and infrastructure states are captured with CloudTrail telemetry, split via grouped cross-validation, and scored by an additive EBM to produce analyst-priority rankings.

Fig. 2: Progressive feature tiers: $M_{0}$ (policy authorization), $M_{1}$ (environment topology), and $M_{2}$ (CloudTrail behavioral telemetry).

Dimension	Count	Description / Experimental Property
Total Samples	534	Controlled real-AWS experimental runs
Type-A Pairs	208	416 samples; matched policy and env.
Type-B Pairs	39	78 samples; matched policy, differing env.
Unpaired Samples	40	Edge-case structural policy variants
Scenario Families	13	S01–S13 privilege escalation / exfil.
Policy Lineages	96	Grouping units for 5-fold CV

TABLE I: AWS-IAM-ContextBench Benchmark Composition

Model	Tier	Feat.	Summary of Included Feature Signals
$M_{0}$	Policy	15	Syntax, wildcards, statements, effective auth
$M_{1}$	Env.	28	$M_{0}$ + Admin roles, KMS/Secrets, boundaries
$M_{2}$	Context	42	$M_{1}$ + CloudTrail errors, user-agent, MFA

TABLE II: Model Definitions and Feature Tiers

Metric	$M_{0}$ (Pol)	$M_{1}$ (Env)	$M_{2}$ (Ctx)	$\Delta(M_{2}-M_{0})$
MAE $\downarrow$	4.8859	4.5272	1.2702	-3.6157
RMSE $\downarrow$	6.0335	5.2609	1.7286	-4.3048
Spearman $\rho$ $\uparrow$	0.4034	0.5666	0.9726	+0.5692
Kendall $\tau$ $\uparrow$	0.2835	0.4072	0.8640	+0.5805
NDCG@10 $\uparrow$	0.8446	0.8714	0.9787	+0.1342
NDCG@25 $\uparrow$	0.9166	0.9321	0.9874	+0.0708

TABLE III: Experiment 1: Incremental Context Ablation Results (Out-of-Fold Cross-Validation,

N=534

)

Fig. 3: Out-of-fold performance comparison of $M_{0}$ , $M_{1}$ , and $M_{2}$ . Error bars represent 95% cluster-bootstrap confidence intervals.

Fig. 4: Matched Type-A context pair separation ( $N=207$ ). $M_{0}$ predictions remain invariant ( $\Delta=0.0000$ ), whereas $M_{2}$ separates suspicious contexts from benign baselines (+6.1049 mean gap).

Evaluation Metric	$M_{0}$ (Policy)	$M_{2}$ (Full Ctx)
Mean Context Priority Gap ( $\Delta_{ij}$ )	0.0000	+6.1049
Median Context Priority Gap	0.0000	+6.3517
Mean Absolute Invariance Gap	0.0000	6.1049
Pairwise Direction Accuracy	0.0%	96.62%
Pairwise Ranking Accuracy	0.0%	89.37%
Context Gap MAE (vs. Oracle)	6.6657	1.1051

TABLE IV: Experiment 2: Matched Type-A Context Pair Evaluation (

N=207

Pairs)

$K$	Metric	$M_{0}$	$M_{1}$	$M_{2}$	Lift vs. $M_{0}$ (%)
10	NDCG@10	0.8176	0.8743	0.9787	+19.7%
10	Precision@10	0.9000	0.7000	1.0000	+11.1%
10	Recall@10	0.3000	0.2333	0.3333	+11.1%
25	NDCG@25	0.9027	0.9336	0.9874	+9.4%
25	Precision@25	0.8000	0.8000	0.9600	+20.0%
25	Recall@25	0.6667	0.6667	0.8000	+20.0%

TABLE V: Experiment 3: Analyst Review Queue Triage Performance Across Inspection Budgets (

K

)

Fig. 5: Local additive EBM explanation for matched Type-A pair p_s01_002 . Anomalous CloudTrail telemetry elevates priority by $+7.79$ points while policy and environment contributions remain identical.

Explore similar work

AI Native Asset Intelligence
May 9, 2026Gal Engelberg, Leon Goldberg, Konstantin Koutsyi +3
Sola Security, Tel Aviv, Israel
Contextualization of Third-Party Cloud Security Findings
Oct 6, 2026Leon Goldberg, Gal EngelbergSecurity Evaluation Severity
Sola Security, Tel Aviv, Israel
Trace2Policy: From Expert Behavior Traces to Self-Evolving Decision Agents
Jun 9, 2026Junli Zha, Jinbo Wang, Chao Zhou +1Large Language Model Policy Optimization Expert Demonstrations
SF Express

May 9, 2026, cs.CRJ/K move · Enter open · S save

AI Native Asset Intelligence

Gal Engelberg, Leon Goldberg, Konstantin Koutsyi, Boris Plotnikov+2

Sola Security, Tel Aviv, Israel

Open paper AskPDF arXiv

Modern security environments generate fragmented signals across cloud resources, identities, configurations, and third-party security tools. Although AI-native security assistants improve access to this data, they remain largely reactive: users must ask the right questions and interpret disconnected findings. This does not scale in enterprise environments, where signal importance depends on exposure, exploitability, dependencies, and business context. Repeated AI queries may therefore produce unstable prioritization without a structured basis for comparing assets. This paper introduces AI-native asset intelligence, a framework that transforms heterogeneous security data into a structured intelligence layer for consistent, contextual, and proactive asset-level reasoning. The framework combines a modeling layer, representing assets, identities, relationships, controls, attack vectors, and blast-radius patterns, with a scoring layer that converts fragmented signals into a normalized measure of asset importance. The scoring system separates intrinsic exposure, based on misconfigurations and attack-vector evidence, from contextual importance, based on anomaly, blast radius, business criticality, and data criticality. AI contextualization refines severity and business/data classifications, while deterministic aggregation preserves consistency. We evaluate the scoring system on a production snapshot with 131,625 resources across 15 vendors and 178 asset types. Sensitivity analyses and ablations show that severity mappings control finding sensitivity, AI severity adjustment refines prioritization, attack-vector scoring responds to rare exploitability evidence, and contextual modulation selectively modifies exposed resources based on business or data importance. The results support AI-native asset intelligence as a foundation for stable prioritization and proactive security-posture reasoning.

Abstract

Explore similar work

AI Native Asset Intelligence

Contextualization of Third-Party Cloud Security Findings

Trace2Policy: From Expert Behavior Traces to Self-Evolving Decision Agents