Why Static Skill Inconsistencies Do Not Equate to Runtime Risks: A Behavioral Analysis of LLM Agent Execution

Fengrui Fan; Huahao Wang; Jingyi Fang; Yutao Hu; Yueming Wu; Deqing Zou

cs.AIOct 13, 2026

Why Static Skill Inconsistencies Do Not Equate to Runtime Risks: A Behavioral Analysis of LLM Agent Execution

Authors: Fengrui Fan, Huahao Wang, Jingyi Fang, Yutao Hu, Yueming Wu, Deqing Zou

Organizations: Huazhong University of Science and technology, China

Abstract

LLM agent frameworks heavily rely on external skill assets to standardize tool usage, encode operational workflows, and supply executable runtime resources. Despite their widespread adoption, multi-component skill packages frequently suffer from cross-artifact inconsistencies, where documentation, scripts, and configuration files specify conflicting behavioral constraints. Although static auditing techniques can uncover such contractual discrepancies, static defect volume cannot reliably reflect real-world agent performance degradation. This gap leaves practitioners without rigorous evidence to assess the practical risk of flawed agent skills. This paper presents a systematic empirical study to demystify the runtime implications of skill inconsistencies. We conduct large-scale auditing over 575 community and benchmark skills, yielding 1,182 manually validated contract inconsistencies. We further build a controlled evaluation pipeline with 49 executable agent tasks and three representative LLMs across NoSkill, OriginalSkill, and FixedSkill experimental settings. Through fine-grained behavioral analysis of 441 paired execution trajectories, we quantify the actual performance impacts of resolved defects.

Figures & tables

Figure 2. A real-world Skill. ① marks an Inter Document inconsistency, while ② and ③ mark Document Code inconsistencies.

Figure 3. The Study pipeline of the paper.

Source	Software Engineering	Media & Content Production	Finance & Economics	Cybersecurity	Office & White Collar	Industrial & Physical Systems	Natural Science	Mathematics & Operations Research
SkillBench ( $n{=}87$ )	16 18.4%	5 5.7%	9 10.3%	7 8.0%	14 16.1%	14 16.1%	14 16.1%	8 9.2%
Github ( $n{=}488$ )	395 80.9%	36 7.4%	18 3.7%	5 1.0%	24 4.9%	2 0.4%	6 1.2%	2 0.4%

Table 1. Category distribution of the SkillBench and Github datasets.

Figure 4. Taxonomy of skill - level inconsistencies.

Model	Active / total params	SWE-bench Verified	Ours ( NoSkill )
DeepSeek-V4-Flash ( DeepSeek-AI et al., 2026 ; Continuum Labs, 2026 ; Hanks, 2026 )	13B / 284B	79.0	0.689
Claude Haiku 4.5 ( Anthropic, 2025a )	—	73.3	0.534
Qwen3.6-35B-A3B ( Qwen Team, 2026 ; oMLX, 2026 )	3B / 35B	73.4	0.498

Table 2. Models and baseline performance.

Code	Definition
Exposed	Agent reads the skill, either from skills/ or through the Skill tool.
Unused	Agent never reads the skill and the inconsistency lies outside execution.
Bypass	Agent is exposed but does not use the inconsistent element. No changed file is executed and no shipped-only form is authored.
Follow	The inconsistent behaviour reaches the artifact. A tied verifier check fails in OriginalSkill and passes in FixedSkill .
Recover	The inconsistent behaviour is triggered, but the tied checks still pass.
BothFail	The tied check fails in both trials and the fix is not decisive.

Table 3. Trajectory coding scheme.

Panel of Figure 5 — Figure 5. Distribution of confirmed contract inconsistency counts per skill. The y-axis shows the proportion of skills in each bin of per-skill counts.

Dataset	Category	#Skills	$\geq$ 1 Inc.	Total Inc.	Mean	Max	Inc./kLine
GitHub Skills	Doc-only	340	202 (59.4%)	421	1.24	7	6.17
	w/ artifacts	148	127 (85.8%)	601	4.06	18	0.98
	All	488	329 (67.4%)	1,022	2.09	18	1.50
SkillBench Skills	Doc-only	41	25 (61.0%)	59	1.44	8	2.05
	w/ artifacts	46	31 (67.4%)	101	2.20	18	0.91
	All	87	56 (64.4%)	160	1.84	18	1.14

Table 4. Inconsistent findings by corpus and skill structure.

Artifact boundary	Inconsistency type	GitHub Skills	SkillBench Skills
Document–Code	over_promise	250 (24.5%)	73 (45.6%)
	semantically_conflicting	93 (9.1%)	16 (10.0%)
	under_promise	1 (0.1%)	1 (0.6%)
	subtotal	344 (33.7%)	90 (56.2%)
Inter Document	shared_concept_definition_divergence	162 (15.9%)	27 (16.9%)
	cross_document_workflow_conflict	81 (7.9%)	2 (1.2%)

Table 5. Distribution of contract inconsistencies by taxonomy.

Corpus	Metric	Software Engineering	Media	Office	Finance	Natural Science	Cybersecurity	Industrial	Math/Formal	All
GitHub Skills	$n$	395	36	24	18	6	5	2	2	488
GitHub Skills	Rate	66%	75%	71%	83%	50%	80%	100%	0%	67%
SkillBench Skills	$n$	16	5	14	9	14	7	14	8	87
SkillBench Skills	Rate	69%	60%	50%	56%	79%	71%	57%	75%	64%

Table 6. Inconsistency prevalence across domains.

Model	NoSkill	Original	Fixed	$\Delta$	Tasks $\uparrow$ / $=$ / $\downarrow$
Claude Haiku 4.5	0.534	0.741	0.877	$+0.136$	30 / 10 / 9
Qwen3.6-35B	0.498	0.707	0.792	$+0.085$	26 / 9 / 14
DeepSeek-Flash	0.689	0.870	0.893	$+0.023$	13 / 30 / 6

Table 7. Effectiveness across skill conditions on GitHub tasks.

Model	Condition	Tokens (med.)	Calls	Turns	Time (s)	Trunc. (%)	Tokens/point
Claude Haiku 4.5	NoSkill	472k	19	21	128	30	1.22M
	Original	472k	16	19	124	24	0.83M
	Fixed	435k	16	21	110	20	0.69M
Qwen3.6-35B	NoSkill	335k	18	24	580	19	0.96M
	Original	362k	15	21	508	15	0.71M
	Fixed	344k	14	21	468	18	0.63M

Table 8. Efficiency across skill conditions on GitHub tasks.

Model	$n$	Exposed	Rate	Path only	Tool only	Both
Claude Haiku 4.5	147	144	98%	75	21	48
Qwen3.6-35B	147	137	93%	120	1	16
DeepSeek-V4-Flash	147	147	100%	119	1	27
All	441	428	97.1%	314	23	91

Table 9. Skill exposure by model and access route.

Model	Recover $\rightarrow$ success	Follow $\rightarrow$ failure	BothFail	Used, no effect	Bypass	Unused	All
Haiku	40	44	40	9	11	3	147
Qwen	33	35	43	11	15	10	147
DeepSeek	82	18	20	19	8	0	147
All	155	97	103	39	34	13	441

Table 10. Response distribution by model.

Figure 6. Response composition by inconsistency type and model. BothFail denotes the control class.

Response	$n$ (fires)	$\Delta$ score	net $\Delta$ token	Footprint turns	Footprint (s)
Follow $\rightarrow$ failure	97 (87)	$+0.366$	$+34$ k	14.4	104
Recover $\rightarrow$ success	155 (155)	$-0.038$	$-57$ k	12.3	60
Bypass	34 (4)	$+0.097$	$+2$ k	3.8	40
BothFail (control)	103 (87)	$+0.030$	$+7$ k	19.6	169
Used, no effect	39 (3)	$-0.047$	$-44$ k	11.3	24
Unused	13 (3)	$+0.127$	$+4$ k	16.0	88

Table 11. Outcome by response class.

Fix effect	$n$	Share	Mean $\Delta$ score
Failure count reduced (partial fix)	38	37%	$+0.352$
Failure count unchanged	37	36%	$+0.011$
Failure count increased (adverse)	28	27%	$-0.380$
Total	103	100%

Table 12. Decomposition of the BothFail control class.

LLM Agent Reliability

Large language model (LLM) agent reliability concerns dependable, safe, and verifiable behavior across tools, multi-step tasks, and changing environments.

See all 843 papers

Runtime Skill Audit: Targeted Runtime Probing for Agent Skill Security
Jun 10, 2026Tu Lan, Chaowei XiaoAI Agent Auditing LLM Agent Security
Johns Hopkins University
SkillSentry: Reliable Skill Execution for LLM Agents via Runtime Assurance
Aug 10, 2026You Lu, Xinyu Huang, Bihuan Chen +1LLM Agent Skill Learning LLM Agents
College of Computer Science and Artificial Intelligence, China
SkillConsist: Detecting Inconsistencies in Agent Skills via Bidirectional Graph Alignment
Aug 7, 2026Chaofan Meng, Yuhang Zheng, Yingnan Zhou +1Agent Evaluation Agent Skill Security Auditing
College of Cryptology and Cyber Science, Nankai University

Jun 10, 2026J/K move · Enter open · S save

Runtime Skill Audit: Targeted Runtime Probing for Agent Skill Security

Tu Lan, Chaowei Xiao

Johns Hopkins University

AI Agent Auditing LLM Agent Security Agent Skill Security Auditing

Open paper AskPDF arXiv

Agent skills let LLM agents reuse instructions, resources, tools, and workflows, but they also create a new place for malicious behavior to hide. A skill may look benign in its documentation or code while becoming harmful only when it is invoked with particular user requests, local assets, persistent state, or multi-step tool interactions. This makes purely static vetting brittle. We present Runtime Skill Audit (RSA), a dynamic analysis method that audits skills by asking what the skill-mediated agent actually does under targeted runtime conditions. Instead of testing every skill with the same generic tasks, RSA profiles risk-relevant interfaces, prepares the execution context needed to exercise them, and assigns security labels from the resulting trace evidence. We instantiate RSA on OpenClaw and evaluate it on 100 skills against representative static baselines. RSA achieves 90.0% accuracy with an 88.0% true positive rate and an 8.0% false positive rate, improving accuracy by 13.0 percentage points over the best static baseline. Under self-evolving attacks, static detectors collapse after one or two rounds, while RSA continues to detect 19--20 out of 20 malicious skills across rounds.

Abstract

LLM Agent Reliability

Explore similar work

Runtime Skill Audit: Targeted Runtime Probing for Agent Skill Security

SkillSentry: Reliable Skill Execution for LLM Agents via Runtime Assurance

SkillConsist: Detecting Inconsistencies in Agent Skills via Bidirectional Graph Alignment