CESBench: Benchmarking Large Language Models on Cryptographic Engineering Security for IoT Devices

Wenquan Zhou; An Wang; Jing Liang; Peien Feng; Jingqi Zhang; Yaoling Ding; Liehuang Zhu

cs.CRSep 18, 2026

CESBench: Benchmarking Large Language Models on Cryptographic Engineering Security for IoT Devices

Authors: Wenquan Zhou, An Wang, Jing Liang, Peien Feng, Jingqi Zhang, Yaoling Ding, Liehuang Zhu

Organizations: School of Cyberspace Science and Technology, Beijing Institute of Technology, Beijing 100081, China · State Key Laboratory of Cryptography and Digital Economy Security, Shandong University, Qingdao 266237, China · Xuteli School, Beijing Institute of Technology, Beijing 100081, China

Abstract

For Internet of Things (IoT) devices, a secure algorithm alone is not enough: an attacker with physical access can attack the implementation directly, and its flaws are hard to fix once deployed. Large language models (LLMs) are now used to build and analyze such implementations. LLM benchmarks exist for cryptography and general cybersecurity, but none covers cryptographic engineering. In this paper, we present CESBench, 380 expert-written items across six sub-domains of cryptographic engineering security for IoT devices: side-channel, fault injection, implementation, countermeasures, evaluation, and integration. Four task types target different competences: 209 multiple-choice items test recall, 67 judgment items require a security verdict and its justification, 63 scenario items require an engineering diagnosis, and 41 code tasks are graded by 572 test cases. To validate the benchmark, 11 open-weight and proprietary LLMs answer every item. Multiple-choice and code responses are scored automatically, and judgment and scenario responses by an LLM judge, whose scores are checked against a second judge from another model family and human re-scoring. Composite scores range from 54.4% to 83.6%. The top score on each task type is 98.6% for multiple choice, 95.1% for code, and 88.4% for scenario diagnosis, but only 58.8% for judgment. Across models, 88.5% of verdicts are correct, yet their justifications earn only 53.4% of the rubric marks. Multiple choice is near its ceiling for the strongest models and most code tasks are solved, whereas justifying a security verdict remains the weakest competence. The benchmark, prompts, and per-item results are public.

Figures & tables

	Scope		Task type			Scoring
Benchmark	Crypto	Physical	Judgment	Scenario	Code	Validated
AICrypto [ 8 ]	$\checkmark$	$\times$	$\checkmark$	$\times$	$\checkmark$	$\checkmark$
CREBench [ 12 ]	$\checkmark$	$\times$	$\times$	$\checkmark$	$\checkmark$	$\times$
CryptoQA [ 10 ]	$\checkmark$	$\times$	$\times$	$\times$	$\times$	$\times$
HardSecBench [ 13 ]	$\checkmark$	$\checkmark$	$\times$	$\times$	$\checkmark$	$\times$
HarmChip [ 14 ]	$\checkmark$	$\checkmark$	$\times$	$\times$	$\times$	$\times$

TABLE I: Coverage of the Three Gaps by Existing LLM Benchmarks

Fig. 1: Composition of CESBench.

Sub-domain	Multiple choice	Judgment	Scenario	Code	Total
D1 Side-channel	42	17	15	6	80
D2 Fault injection	30	8	8	5	51
D3 Implementation	44	15	14	10	83
D4 Countermeasures	33	11	10	8	62
D5 Evaluation	30	8	8	5	51
D6 Integration	30	8	8	7	53

TABLE II: Question Distribution of CESBench

Sub-domain	Requirement (source clauses)
D1	Keep secrets from leaking through timing, power, and electromagnetic emissions (SESIP 3.4.2; PSA Certified Level 3).
D2	Counter perturbation and tampering by a physical attacker (SESIP 3.4.2; PSA Certified Level 3).
D3	Provide cryptographic operations, key generation, and random numbers (SESIP 3.5; NIST IR 8259A Data Protection).
D4	Detect or prevent physical attacks, and erase residual secrets (SESIP 3.4.2 and 3.6.4).
D5	Be evaluated at a defined assurance level (SESIP1–SESIP5; PSA Certified levels).
D6	Protect software integrity, update securely, store keys, and identify the device (ETSI EN 303 645 5.3, 5.4, and 5.7; ISO/IEC 27402 5.2.7; NIST IR 8259A).

TABLE III: Requirements in IoT Security Standards That Each Sub-Domain Addresses

Fig. 2: One item per task type.

Sub-domain	L0	L1	L2	Total
D1 Side-channel	0	59	21	80
D2 Fault injection	0	35	16	51
D3 Implementation	32	29	22	83
D4 Countermeasures	1	41	20	62
D5 Evaluation	4	35	12	51
D6 Integration	21	18	14	53

TABLE IV: Device-Condition Labels by Sub-Domain

Model	Vendor	Weights	Params (active)	Released
DeepSeek-V3	DeepSeek	open	671B (37B)	Dec 2024
GLM-4-32B-0414	Zhipu AI	open	32B (32B)	Apr 2025
Llama-4-Maverick	Meta	open	400B (17B)	Apr 2025
Hunyuan-A13B	Tencent	open	80B (13B)	Jun 2025
Ling-flash-2.0	Ant Group	open	100B (6.1B)	Sep 2025
MiniMax-M2.5	MiniMax	open	230B (10B)	Feb 2026

TABLE V: Evaluated Models and Judges

		Task type				Sub-domain
Model	Composite	T1	T2	T3	T4	D1	D2	D3	D4	D5	D6
GLM-5.2	83.6	98.6	58.8	84.2	92.7	84.1	81.7	85.4	85.3	71.8	90.7
Kimi-K2.6	82.5	97.1	58.0	82.2	92.7	83.5	80.6	75.6	85.5	80.5	89.7
Gemini-3.7-Flash	81.4	98.6	54.9	81.9	90.2	85.5	79.9	78.5	83.3	79.7	81.5
GPT-5.6-Luna	80.8	97.6	54.5	88.4	82.9	85.0	76.3	77.9	81.6	77.5	84.3
DeepSeek-V4-Pro	80.2	97.1	47.0	81.4	95.1	78.9	78.6	78.5	79.5	77.3	87.6

TABLE VI: Scores in Percent by Task Type and by Sub-Domain

Fig. 3: Scores by sub-domain and task type for every model.

Fig. 4: Judgment items lost to a wrong verdict or a weak justification, per model.

		Mean score (%)			Share (%)
Kind	$n$	Leading five	Other six	All	Fully met	Not met
Mechanism	83	87.3	68.7	77.2	65.4	11.1
Assessment	24	91.2	62.2	75.4	59.5	8.7
Design	83	83.6	55.7	68.4	55.6	18.8
Standard	28	79.3	57.1	67.2	51.9	17.5
Diagnosis	53	75.1	54.6	63.9	50.6	22.8

TABLE VII: Scores on Scenario Rubric Dimensions by Kind of Answer

Fig. 5: Failed code tasks per model by grader class.

Level	$n$	Score	$n$	Score	$n$	Score	$n$	Score
	T1		T2		T3		T4
L0 general	33	93.4	8	65.1	8	80.2	9	88.9
L1 device-oriented	129	93.9	44	46.5	22	73.1	22	77.7
L2 device-constrained	47	91.9	15	39.9	33	66.6	10	54.5
All items	209	93.3	67	47.2	63	70.6	41	74.5

TABLE VIII: Items and Pooled Scores by Device-Condition Level

Check	Task	$n$	ICC(2,1)	$\rho$	Bias
Repeat pass	T2	44	0.99	0.98	$-$ 0.1
Repeat pass	T3	56	0.97	0.97	$+$ 0.5
Second judge	T2	44	0.95	0.93	$+$ 1.3
Second judge	T3	56	0.83	0.91	$+$ 2.3
Human rater	T2	77	0.87	0.86	$-$ 2.3
Human rater	T3	174	0.78	0.77	$+$ 3.6

TABLE IX: Judge Reliability

Cybersecurity

Cybersecurity develops methods to protect digital and cyber-physical systems from attacks, vulnerabilities, data loss, and misuse.

See all 355 papers

CyberCertBench: Evaluating LLMs in Cybersecurity Certification Knowledge
Apr 22, 2026Gustav Keppler, Ghada Elbez, Veit HagenmeyerLLM Evaluation Cybersecurity
Institute for Automation and Applied Informatics (IAI), Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany
TOSSS: a CVE-based Software Security Benchmark for Large Language Models
Mar 11, 2026Marc Damie, Murat Bilgehan Ertan, Domenico Essoussi +3LLM Safety Benchmarks Software Security
Modat, The Hague, The Netherlands
CREBench: Evaluating Large Language Models in Cryptographic Binary Reverse Engineering
Apr 4, 2026Baicheng Chen, Yu Wang, Ziheng Zhou +4Software Reverse Engineering LLMs for Cybersecurity
Shanghai Qi Zhi Institute · The Chinese University of Hong Kong, Shenzhen · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China +4

Apr 22, 2026J/K move · Enter open · S save

CyberCertBench: Evaluating LLMs in Cybersecurity Certification Knowledge

Gustav Keppler, Ghada Elbez, Veit Hagenmeyer

Institute for Automation and Applied Informatics (IAI), Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany

LLM Evaluation Cybersecurity LLMs for Cybersecurity

Open paper AskPDF arXiv

The rapid evolution and use of Large Language Models (LLMs) in professional workflows require an evaluation of their domain-specific knowledge against industry standards. We introduceCyberCertBench, a new suite of Multiple Choice Question Answering (MCQA) benchmarks derived from industry recognized certifications. CyberCertBench evaluates LLM domain knowledgeagainst the professional standards of Information Technology cybersecurity and more specializedareas such as Operational Technology and related cybersecurity standards. Concurrently, we propose and validate a novel Proposer-Verifier framework, a methodology to generate interpretable,natural language explanations for model performance. Our evaluation shows that frontier modelsachieve human expert level in general networking and IT security knowledge. However, theiraccuracy declines in questions that require vendor-specific nuances or knowledge in formalstandards, like, e.g., IEC 62443. Analysis of model scaling trend and release date demonstratesremarkable gains in parameter efficiency, while recent larger models show diminishing returns.Code and evaluation scripts are available at: https://github.com/GKeppler/CyberCertBench.

Abstract

Cybersecurity

Explore similar work

CyberCertBench: Evaluating LLMs in Cybersecurity Certification Knowledge

TOSSS: a CVE-based Software Security Benchmark for Large Language Models

CREBench: Evaluating Large Language Models in Cryptographic Binary Reverse Engineering