Automated Analysis of Global AI Safety Initiatives: A Taxonomy-Driven LLM Approach

Takayuki Semitsu; Naoto Kiribuchi; Kengo Zenitani

cs.AIApr 4, 2026

Automated Analysis of Global AI Safety Initiatives: A Taxonomy-Driven LLM Approach

Authors: Takayuki Semitsu, Naoto Kiribuchi, Kengo Zenitani

Organizations: Japan AI Safety Institute, Tokyo, Japan

Abstract

We present an automated crosswalk framework that compares an AI safety policy document pair under a shared taxonomy of activities. Using the activity categories defined in Activity Map on AI Safety as fixed aspects, the system extracts and maps relevant activities, then produces for each aspect a short summary for each document, a brief comparison, and a similarity score. We assess the stability and validity of LLM-based crosswalk analysis across public policy documents. Using five large language models, we perform crosswalks on ten publicly available documents and visualize mean similarity scores with a heatmap. The results show that model choice substantially affects the crosswalk outcomes, and that some document pairs yield high disagreements across models. A human evaluation by three experts on two document pairs shows high inter-annotator agreement, while model scores still differ from human judgments. These findings support comparative inspection of policy documents.

Figures & tables

Figure 1: Overview of the crosswalk for policy documents. AI safety policy documents issued by different institutions are compared using a shared taxonomy. For each aspect, the output includes a summary of each document, a comparison, and a similarity score.

p	Activity	One-sentence description
p1	Content Authentication and Provenance	Develop and deploy trustworthy authentication/provenance mechanisms (e.g., watermarking) so users can identify AI-generated content when technically possible.
p2	Addressing Societal Risks	Address risks in high-impact domains (e.g., critical infrastructure, CBRNE), dual-use, and autonomous agents.
p3	AI Safety Evaluation and Red-Teaming	Conduct safety evaluations and red-teaming to identify, assess, and mitigate risks of AI systems.
p4	Responsible Information Sharing	Engage in responsible information sharing and incident reporting across organizations to reduce vulnerabilities and misuse of advanced AI systems.
p5	Enabling and Fostering AI Safety Science	Promote R&D on safety evaluation techniques to support institution design grounded in scientific knowledge.
p6	Ensuring Security Throughout the AI Lifecycle	Invest in and implement robust security management across the AI lifecycle, including physical, cyber, and insider-threat controls.

Table 1: AMAIS activity categories (p1, …, p15) used as the shared crosswalk coordinate system.

Extracted activity item (English)	Supporting excerpt	AMAIS aspect
Hiring technical experts and partnering with researchers across government, academia, and industry (p. 4).	“we have hired technical experts from top industry and academic labs, … We have built partnerships with leading AI labs, research organisations, academia, and segments of the UK government”	$p12$ (Human Capital Investment and Education)
Distilling research findings into best practices, standards, and protocols for AI safety and security (p. 4).	“International protocols: Working with key partners across government, we distil key research findings into best practices, standards, and protocols for AI safety and security and cohere model developers, deployers, and international actors around them.”	$p7$ (Advocating for Policy and Governance Frameworks)

Table 2: Two example activity items extracted from the UK-AISI document “Our Research Agenda” ( UK-AISI, 2025 ) .

Label	Title	Entity
$A$	Our Research Agenda	UK-AISI,2025
$B$	ASEAN Guide on AI Governance and Ethics	ASEAN,2024
$C$	CAISI Research Program at CIFAR 2025 Year in Review: Building Safe AI for Canadians	CIFAR,2026
$D$	Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence	The White House (US),2023
$E$	America’s Action Plan	The White House (US),2025
$F$	The Singapore Consensus on Global AI Safety Research Priorities	Bengio et al.,2025

Table 3: Document set used in the case study and the label assignment adopted in this paper.

Method	Model ID	Model name
a	mistral.mistral-large-3-675b-instruct	Mistral Large 3
b	qwen.qwen3-next-80b-a3b	Qwen3 Next 80B A3B
c	google.gemma-3-27b-it	Gemma 3 27B
d	openai.gpt-oss-120b-1_0	gpt-oss-120b
e	nvidia.nemotron-nano-3-30b	Nemotron Nano 3 30B

Table 4: List of LLMs used in the experiments. All other settings (e.g., prompts and temperature) are fixed and identical across experiments.

Figure 2: Heatmap of similarity scores where rows correspond to AI-safety activity items and columns correspond to Document $D_{2}$ (Document $D_{1}$ is fixed to UK-AISI). Each value is averaged over results from five models.

Figure 3: Heatmap of the standard deviation of similarity scores where rows correspond to AI-safety activity items and columns correspond to Document $D_{2}$ (Document $D_{1}$ is fixed to UK-AISI). Lower values indicate closer agreement among the results produced by five AI models.

Figure 4: Heatmap comparing crosswalk results produced by pairs of AI models (out of five). Rows correspond to one model and columns to another. Each cell reports the mean absolute difference between the two models’ crosswalk results, averaged over 9 document pairs (Document 1 fixed; Document 2 varying across nine documents) and 15 activity items.

ID	Activity item	Score 1	Score 2	Score 3	Std. dev.	Median
1	Content Authentication and Provenance	0	1	2	1.000	1
2	Addressing Societal Risks	3	2	2	0.577	2
3	AI Safety Evaluation and Red-Teaming	4	3	3	0.577	3
4	Responsible Information Sharing	4	4	4	0.000	4
5	Enabling and Fostering AI Safety Science	2	3	2	0.577	2
6	Ensuring Security Throughout the AI Lifecycle	3	3	3	0.000	3

Table 5: Human similarity scores for the crosswalk between Document

A

and Document

D

. Std. dev. is computed across the three annotator scores; values are rounded to three decimals.

ID	Activity item	Score 1	Score 2	Score 3	Std. dev.	Median
1	Content Authentication and Provenance	0	2	2	1.155	2
2	Addressing Societal Risks	3	3	3	0.000	3
3	AI Safety Evaluation and Red-Teaming	5	4	3	1.000	4
4	Responsible Information Sharing	4	3	3	0.577	3
5	Enabling and Fostering AI Safety Science	2	3	2	0.577	2
6	Ensuring Security Throughout the AI Lifecycle	0	2	3	1.528	2

Table 6: Human similarity scores for the crosswalk between Document

A

and Document

E

. Std. dev. is computed across the three annotator scores; values are rounded to three decimals.

Figure 5: Mean absolute difference between human annotator scores and LLM-produced crosswalk similarity scores, averaged over the 15 AMAIS activity items and the two document pairs of $(A,D)$ and $(A,E)$ .

Figure 6: Mean absolute difference between human annotator scores and LLM-produced crosswalk similarity scores, averaged over annotators for the two document pairs of $(A,D)$ and $(A,E)$ .

AI Safety

Artificial intelligence safety studies how to prevent harmful, unreliable, or uncontrollable behavior and manage risks across AI systems and their deployment.

See all 134 papers

aiXamine: Unified Black-Box Evaluation of Cross-Dimensional Trade-offs in LLM Safety, Security, and Privacy
Aug 20, 2026Fatih Deniz, Yazan Boshmaf, Dorde Popovic +1LLM Evaluation LLM Safety Evaluation
Qatar Computing Research Institute (QCRI), HBKU, Doha, Qatar
Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agents
Apr 11, 2026Miles Q. Li, Benjamin C. M. Fung, Boyang Li +2AI Agent Safety Benchmarks AI Agent Evaluation
School of Information Studies, McGill University, Canada · Department of Computer Science, Kean University, USA · College of Technological Innovation, Zayed University, UAE
Taxonomy-Driven Analysis of Open-Source AI Risk Mitigation Tools
Aug 7, 2026Afreen Alam, Evgenija Popchanovska, Ana Gjorgjevikj +4AI Risk Management LLM Security
Department of Administrative Sciences, Metropolitan College, Boston University, Boston, MA 02215, USA · Faculty of Computer Science and Engineering, Ss. Cyril and Methodius University, Skopje 1000, North Macedonia · Department of Computer Science, Metropolitan College, Boston University, Boston, MA 02215, USA

Aug 20, 2026J/K move · Enter open · S save

aiXamine: Unified Black-Box Evaluation of Cross-Dimensional Trade-offs in LLM Safety, Security, and Privacy

Fatih Deniz, Yazan Boshmaf, Dorde Popovic, Issa Khalil

Qatar Computing Research Institute (QCRI), HBKU, Doha, Qatar

LLM Evaluation LLM Safety Evaluation LLM Security LLM Safety Benchmarks Over-Refusal

Open paper AskPDF arXiv

The critical failure modes in deployed large language models (LLMs) are cross-dimensional: a model can score 99.3 in safety alignment while refusing one in three benign queries, or improve across every capability metric while losing 21 points in privacy. Existing evaluation frameworks that assess safety, security, and privacy independently cannot detect these patterns. We introduce aiXamine, a unified black-box platform that evaluates LLM trustworthiness across safety, security, and privacy as interdependent properties. aiXamine orchestrates 46 tests across nine services through an automated red-teaming pipeline, producing hierarchical risk profiles, from prompt-level diagnostics to cross-service trade-off analytics, that enable reproducible comparison of proprietary and open-weight systems under identical conditions. Applying aiXamine to over 120 LLMs through more than 5,000 test runs, we conduct the largest joint safety, security, and privacy study to date and uncover three cross-dimensional phenomena invisible to single-axis evaluation. First, safety enforcement incurs a quantifiable safety tax: stronger alignment systematically increases over-refusal, forcing providers to choose between protection and utility. Second, privacy is near-orthogonal to other trustworthiness dimensions and not captured by standard alignment. Third, we identify and formally characterize distillation-induced robustness collapse: off-policy distillation without on-policy correction causes entropy collapse, catastrophically destroying robustness (56.9

\to

2.6) on the same base architecture. These findings, compounded by diminishing returns from scale and category-dependent safety behaviors, demonstrate that trustworthiness is inherently multi-dimensional: progress along one axis does not guarantee, and can actively undermine, progress along others, yet current alignment methods treat it as a single objective.

Abstract

AI Safety

Explore similar work

aiXamine: Unified Black-Box Evaluation of Cross-Dimensional Trade-offs in LLM Safety, Security, and Privacy

Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agents

Taxonomy-Driven Analysis of Open-Source AI Risk Mitigation Tools