CESBench: Benchmarking Large Language Models on Cryptographic Engineering Security for IoT Devices

Wenquan Zhou; An Wang; Jing Liang; Peien Feng; Jingqi Zhang; Yaoling Ding; Liehuang Zhu

cs.CRSep 18, 2026

CESBench: Benchmarking Large Language Models on Cryptographic Engineering Security for IoT Devices

Authors: Wenquan Zhou, An Wang, Jing Liang, Peien Feng, Jingqi Zhang, Yaoling Ding, Liehuang Zhu

Organizations: School of Cyberspace Science and Technology, Beijing Institute of Technology, Beijing 100081, China · State Key Laboratory of Cryptography and Digital Economy Security, Shandong University, Qingdao 266237, China · Xuteli School, Beijing Institute of Technology, Beijing 100081, China

Abstract

For Internet of Things (IoT) devices, a secure algorithm alone is not enough: an attacker with physical access can attack the implementation directly, and its flaws are hard to fix once deployed. Large language models (LLMs) are now used to build and analyze such implementations. LLM benchmarks exist for cryptography and general cybersecurity, but none covers cryptographic engineering. In this paper, we present CESBench, 380 expert-written items across six sub-domains of cryptographic engineering security for IoT devices: side-channel, fault injection, implementation, countermeasures, evaluation, and integration. Four task types target different competences: 209 multiple-choice items test recall, 67 judgment items require a security verdict and its justification, 63 scenario items require an engineering diagnosis, and 41 code tasks are graded by 572 test cases. To validate the benchmark, 11 open-weight and proprietary LLMs answer every item. Multiple-choice and code responses are scored automatically, and judgment and scenario responses by an LLM judge, whose scores are checked against a second judge from another model family and human re-scoring. Composite scores range from 54.4% to 83.6%. The top score on each task type is 98.6% for multiple choice, 95.1% for code, and 88.4% for scenario diagnosis, but only 58.8% for judgment. Across models, 88.5% of verdicts are correct, yet their justifications earn only 53.4% of the rubric marks. Multiple choice is near its ceiling for the strongest models and most code tasks are solved, whereas justifying a security verdict remains the weakest competence. The benchmark, prompts, and per-item results are public.

Figures & tables

	Scope		Task type			Scoring
Benchmark	Crypto	Physical	Judgment	Scenario	Code	Validated
AICrypto [ 8 ]	$\checkmark$	$\times$	$\checkmark$	$\times$	$\checkmark$	$\checkmark$
CREBench [ 12 ]	$\checkmark$	$\times$	$\times$	$\checkmark$	$\checkmark$	$\times$
CryptoQA [ 10 ]	$\checkmark$	$\times$	$\times$	$\times$	$\times$	$\times$
HardSecBench [ 13 ]	$\checkmark$	$\checkmark$	$\times$	$\times$	$\checkmark$	$\times$
HarmChip [ 14 ]	$\checkmark$	$\checkmark$	$\times$	$\times$	$\times$	$\times$

TABLE I: Coverage of the Three Gaps by Existing LLM Benchmarks

Fig. 1: Composition of CESBench.

Sub-domain	Multiple choice	Judgment	Scenario	Code	Total
D1 Side-channel	42	17	15	6	80
D2 Fault injection	30	8	8	5	51
D3 Implementation	44	15	14	10	83
D4 Countermeasures	33	11	10	8	62
D5 Evaluation	30	8	8	5	51
D6 Integration	30	8	8	7	53

TABLE II: Question Distribution of CESBench

Sub-domain	Requirement (source clauses)
D1	Keep secrets from leaking through timing, power, and electromagnetic emissions (SESIP 3.4.2; PSA Certified Level 3).
D2	Counter perturbation and tampering by a physical attacker (SESIP 3.4.2; PSA Certified Level 3).
D3	Provide cryptographic operations, key generation, and random numbers (SESIP 3.5; NIST IR 8259A Data Protection).
D4	Detect or prevent physical attacks, and erase residual secrets (SESIP 3.4.2 and 3.6.4).
D5	Be evaluated at a defined assurance level (SESIP1–SESIP5; PSA Certified levels).
D6	Protect software integrity, update securely, store keys, and identify the device (ETSI EN 303 645 5.3, 5.4, and 5.7; ISO/IEC 27402 5.2.7; NIST IR 8259A).

TABLE III: Requirements in IoT Security Standards That Each Sub-Domain Addresses

Fig. 2: One item per task type.

Sub-domain	L0	L1	L2	Total
D1 Side-channel	0	59	21	80
D2 Fault injection	0	35	16	51
D3 Implementation	32	29	22	83
D4 Countermeasures	1	41	20	62
D5 Evaluation	4	35	12	51
D6 Integration	21	18	14	53

TABLE IV: Device-Condition Labels by Sub-Domain

Model	Vendor	Weights	Params (active)	Released
DeepSeek-V3	DeepSeek	open	671B (37B)	Dec 2024
GLM-4-32B-0414	Zhipu AI	open	32B (32B)	Apr 2025
Llama-4-Maverick	Meta	open	400B (17B)	Apr 2025
Hunyuan-A13B	Tencent	open	80B (13B)	Jun 2025
Ling-flash-2.0	Ant Group	open	100B (6.1B)	Sep 2025
MiniMax-M2.5	MiniMax	open	230B (10B)	Feb 2026

TABLE V: Evaluated Models and Judges

		Task type				Sub-domain
Model	Composite	T1	T2	T3	T4	D1	D2	D3	D4	D5	D6
GLM-5.2	83.6	98.6	58.8	84.2	92.7	84.1	81.7	85.4	85.3	71.8	90.7
Kimi-K2.6	82.5	97.1	58.0	82.2	92.7	83.5	80.6	75.6	85.5	80.5	89.7
Gemini-3.7-Flash	81.4	98.6	54.9	81.9	90.2	85.5	79.9	78.5	83.3	79.7	81.5
GPT-5.6-Luna	80.8	97.6	54.5	88.4	82.9	85.0	76.3	77.9	81.6	77.5	84.3
DeepSeek-V4-Pro	80.2	97.1	47.0	81.4	95.1	78.9	78.6	78.5	79.5	77.3	87.6

TABLE VI: Scores in Percent by Task Type and by Sub-Domain

Fig. 3: Scores by sub-domain and task type for every model.

Fig. 4: Judgment items lost to a wrong verdict or a weak justification, per model.

		Mean score (%)			Share (%)
Kind	$n$	Leading five	Other six	All	Fully met	Not met
Mechanism	83	87.3	68.7	77.2	65.4	11.1
Assessment	24	91.2	62.2	75.4	59.5	8.7
Design	83	83.6	55.7	68.4	55.6	18.8
Standard	28	79.3	57.1	67.2	51.9	17.5
Diagnosis	53	75.1	54.6	63.9	50.6	22.8

TABLE VII: Scores on Scenario Rubric Dimensions by Kind of Answer

Fig. 5: Failed code tasks per model by grader class.

Level	$n$	Score	$n$	Score	$n$	Score	$n$	Score
	T1		T2		T3		T4
L0 general	33	93.4	8	65.1	8	80.2	9	88.9
L1 device-oriented	129	93.9	44	46.5	22	73.1	22	77.7
L2 device-constrained	47	91.9	15	39.9	33	66.6	10	54.5
All items	209	93.3	67	47.2	63	70.6	41	74.5

TABLE VIII: Items and Pooled Scores by Device-Condition Level

Check	Task	$n$	ICC(2,1)	$\rho$	Bias
Repeat pass	T2	44	0.99	0.98	$-$ 0.1
Repeat pass	T3	56	0.97	0.97	$+$ 0.5
Second judge	T2	44	0.95	0.93	$+$ 1.3
Second judge	T3	56	0.83	0.91	$+$ 2.3
Human rater	T2	77	0.87	0.86	$-$ 2.3
Human rater	T3	174	0.78	0.77	$+$ 3.6

TABLE IX: Judge Reliability

Cybersecurity

Cybersecurity develops methods to protect digital and cyber-physical systems from attacks, vulnerabilities, data loss, and misuse.

See all 355 papers

Apr 22, 2026

CyberCertBench: Evaluating LLMs in Cybersecurity Certification Knowledge

The rapid evolution and use of Large Language Models (LLMs) in professional workflows require an evaluation of their domain-specific knowledge against industry standards. We introduceCyberCertBench, a new suite of Multiple Choice Question Answering (MCQA) benchmarks derived from industry recognized certifications. CyberCertBench evaluates LLM domain knowledgeagainst the professional standards of Information Technology cybersecurity and more specializedareas such as Operational Technology and related cybersecurity standards. Concurrently, we propose and validate a novel Proposer-Verifier framework, a methodology to generate interpretable,natural language explanations for model performance. Our evaluation shows that frontier modelsachieve human expert level in general networking and IT security knowledge. However, theiraccuracy declines in questions that require vendor-specific nuances or knowledge in formalstandards, like, e.g., IEC 62443. Analysis of model scaling trend and release date demonstratesremarkable gains in parameter efficiency, while recent larger models show diminishing returns.Code and evaluation scripts are available at: https://github.com/GKeppler/CyberCertBench.

Gustav Keppler, Ghada Elbez, Veit Hagenmeyer

Institute for Automation and Applied Informatics (IAI), Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany

LLM Evaluation Cybersecurity LLMs for Cybersecurity

Ask PDF

Mar 11, 2026

TOSSS: a CVE-based Software Security Benchmark for Large Language Models

With their increasing capabilities, Large Language Models (LLMs) are now used across many industries. They have become useful tools for software engineers and support a wide range of development tasks. As LLMs are increasingly used in software development workflows, a critical question arises: are LLMs good at software security? At the same time, organizations worldwide invest heavily in cybersecurity to reduce exposure to disruptive attacks. The integration of LLMs into software engineering workflows may introduce new vulnerabilities and weaken existing security efforts. We introduce TOSSS (Two-Option Secure Snippet Selection), a benchmark that measures the ability of LLMs to choose between secure and vulnerable code snippets. Existing security benchmarks for LLMs cover only a limited range of vulnerabilities. In contrast, TOSSS relies on the CVE database and provides an extensible framework that can integrate newly disclosed vulnerabilities over time. Our benchmark gives each model a security score between 0 and 1 based on its behavior; a score of 1 indicates that the model always selects the secure snippet, while a score of 0 indicates that it always selects the vulnerable one. We evaluate 14 widely used open-source and closed-source models on C/C++ and Java code and observe scores ranging from 0.48 to 0.89. LLM providers already publish many benchmark scores for their models, and TOSSS could become a complementary security-focused score to include in these reports.

Marc Damie, Murat Bilgehan Ertan, Domenico Essoussi +3

Modat, The Hague, The Netherlands

LLM Safety Benchmarks Software Security LLM Security Software Vulnerability Detection

Ask PDF

Apr 4, 2026

CREBench: Evaluating Large Language Models in Cryptographic Binary Reverse Engineering

Reverse engineering (RE) is central to software security, particularly for cryptographic programs that handle sensitive data and are highly prone to vulnerabilities. It supports critical tasks such as vulnerability discovery and malware analysis. Despite its importance, RE remains labor-intensive and requires substantial expertise, making large language models (LLMs) a potential solution for automating the process. However, their capabilities for RE remain systematically underexplored. To address this gap, we study the cryptographic binary RE capabilities of LLMs and introduce CREBench, a benchmark comprising 432 challenges built from 48 standard cryptographic algorithms, 3 insecure crypto key usage scenarios, and 3 difficulty levels. Each challenge follows a Capture-the-Flag (CTF) RE challenge, requiring the model to analyze the underlying cryptographic logic and recover the correct input. We design an evaluation framework comprising four sub-tasks, from algorithm identification to correct flag recovery. We evaluate eight frontier LLMs on CREBench. GPT-5.4, the best-performing model, achieves 64.03 out of 100 and recovers the flag in 59% of challenges. We also establish a strong human expert baseline of 92.19 points, showing that humans maintain an advantage in cryptographic RE tasks. Our code and dataset are available at https://github.com/wangyu-ovo/CREBench.

Baicheng Chen, Yu Wang, Ziheng Zhou +4

Shanghai Qi Zhi Institute · The Chinese University of Hong Kong, Shenzhen · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China +4

Software Reverse Engineering LLMs for Cybersecurity LLM Evaluation

Ask PDF

Abstract

Cybersecurity

Explore similar work

CyberCertBench: Evaluating LLMs in Cybersecurity Certification Knowledge

TOSSS: a CVE-based Software Security Benchmark for Large Language Models

CREBench: Evaluating Large Language Models in Cryptographic Binary Reverse Engineering