Authors: Mohammad Wali Ur Rahman, Martin Manuel Lopez, Lamia Tasnim Mim, Carter Farthing, Julius Battle, Kathryn Buckley, Salim Hariri
Organizations: Department of Electrical and Computer Engineering from the University of Arizona, Tucson, AZ 85721, USA. · Avirtek, Inc., Tucson, AZ 85712, USA. · Joint Interoperability Test Command (JITC), United States Department of Defense, AZ 85613, USA.
For digital infrastructure to be safe, compatible, and standards-aligned, automated communication protocol compliance verification is crucial. Nevertheless, current rule-based systems are becoming less and less effective since they are unable to identify subtle or intricate non-compliance, which attackers frequently use to establish covert communication channels in IPv6 traffic. In order to automate IPv6 compliance verification, this paper presents the Artificial Intelligence Driven Compliance Checker Engine (AICCE), a novel generative system that combines dual-architecture reasoning and retrieval-augmented generation (RAG). Specification segments pertinent to each query can be efficiently retrieved thanks to the semantic encoding of protocol standards into a high-dimensional vector space. Based on this framework, AICCE offers two complementary pipelines: (i) Explainability Mode, which uses parallel LLM agents to render decisions and settle disputes through organized discussions to improve interpretability and robustness, and (ii) Script Execution Mode, which converts clauses into Python rules that can be executed quickly for dataset-wide verification. With the debate mechanism enhancing decision reliability in complicated scenarios and the script-based pipeline lowering per-sample latency, AICCE achieves accuracy and F1-scores of up to 99% when tested on IPv6 packet samples across sixteen cutting-edge generative models. By offering a scalable, auditable, and generalizable mechanism for identifying both routine and covert non-compliance in dynamic communication environments, our results show that AICCE overcomes the blind spots of conventional rule-based compliance checking systems.
TABLE I : Qualitative Comparison of Network Protocol Validation Methods Across Key Criteria.
Fig. 1 : Basic Architecture Diagram of a) Llama, b) GPT and c) Mixtral 8x7b
Fig. 2 : Visualization of ANN search with the Hierarchical Navigable Small World (HNSW) algorithm.
Fig. 3 : AICCE Framework in Explainability Mode
Fig. 4 : AICCE Framework in Script Execution Mode
Aspect
Architecture A
Architecture B
Primary Focus
Interpretability, coverage
Speed, automation
Multi-agent Usage
per section & per packet
per section
Debate Mechanism
Yes
Yes
Granularity
Section-level judgments
Rule-level execution
Output
Rationale + verdict
Rule matrix + verdict
Exec Speed
∼2 s / packet
∼0.8 s / packet
TABLE II : Comparison of Framework Architectures
Violation Type
Difficulty
Occurrences
IPv6 Version Violation
Trivial
52
Length Field Violation
Moderate
54
IPv6 Address Violation
Trivial
54
Hop Limit Violation
Moderate
52
TCP/UDP/ICMPv6 Overlap
Challenging
49
Flow Label Violation
Moderate
54
TABLE III : Violation Types by Detection Difficulty
Provider
Model(s)
API / Access Layer
OpenAI
GPT-4o
OpenAI API [ 37 ]
o1
GPT-4-Turbo
GPT-4o-mini
GPT-3.5-Turbo
Meta
Llama 3.3-70B
Groq API [ 38 ]
TABLE IV : LLM providers, models, and access layers used in experiments
Fig. 5 : Model accuracy comparison under ablation vs. debate-enabled settings for Architecture A. Debate consistently improves accuracy across models, particularly for weaker baselines.
Fig. 6 : F1-score comparison under ablation vs. debate-enabled settings for Architecture A. Debate reduces false positives and stabilizes precision–recall balance.
Fig. 7 : Decision latency comparison under ablation vs. debate-enabled settings for Architecture A. As expected, enabling debate increases runtime, but accuracy gains justify the overhead.
Fig. 8 : Number of debate phases initiated per model in Architecture A. Stronger CoT models required fewer debates, while weaker baselines triggered debates more frequently.
Fig. 9 : Performance of models in Script Execution Mode (Architecture B). Subfigures show (a) accuracy, (b) F1-score, (c) latency, and (d) number of retries.
System
Accuracy
Precision
Recall
F1-score
Latency
Explainability
Err. Recovery
Logistic Regression ∗
0.93
0.9452
0.69
0.7977
0.00035
No
No
Random Forest ∗
0.938
0.8557
0.83
0.8426
0.00086
No
No
Extra Trees ∗
0.928
0.8333
0.80
0.8163
0.00043
No
No
XGBoost ∗
0.56
0.2987
0.89
0.4472
0.00016
No
No
LightGBM ∗
0.50
0.2656
0.85
0.4048
0.00020
No
No
Support Vector Machine ∗
0.914
0.9831
0.58
0.7296
0.00029
No
No
TABLE V : Performance Comparison of Different Compliance Checker Systems
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Method
Model
Acc
Prec
Rec
F1
Latency
Debates /Retries
Offers Explainability
Error Recovery
Classical ML ∗
Random Forest
0.938
0.8557
0.83
0.8426
0.00086
N/A
No
No
MLP Neural Net
0.93
0.8095
0.85
0.8293
0.00233
N/A
No
No
Logistic Regression
0.93
0.9452
0.69
0.7977
0.00035
N/A
No
No
Extra Trees
0.928
0.8333
0.80
0.8163
0.00043
N/A
No
No
SVM
0.914
0.9831
0.58
0.7296
0.00029
N/A
No
No
KNN
0.888
0.9583
0.46
0.6216
0.00001
N/A
No
No
Appendix
TABLE VI : Comprehensive comparison across AICCE configurations and baselines.
Verifying building compliance requires validating thousands of rules against large Building Information Modeling (BIM) designs, which is laborious, capital-intensive, and unscalable. Existing Automated Compliance Checkers (ACCs) are often difficult to generalize across different scenarios, as they are typically developed for highly specific rule sets and use cases. In addition, many ACCs are proprietary, meaning the underlying verification code is not released to end users, so users cannot verify whether their regulatory intent can be accurately captured. We introduce ARCHER (Agentic Rule and Compliance Harness for Executable Regulations), a test-driven, deterministically orchestrated multi-agent program-synthesis harness that generates auditable verification code from regulatory Codes of Practice, enabling transparent, adaptable, and scalable compliance checking. To characterize what makes agentic synthesis work, we evaluate a taxonomy of six harnesses of increasing agentic sophistication across four backbone models, spanning realistic data-governance tiers (from frontier third-party APIs to a fully on-premise open-weights model) on a novel dataset derived from real-world compliance scenarios. ARCHER's deterministic multi-agent orchestration achieves the highest accuracy for every backbone, improving mean union accuracy by 82% over a naive single-pass prompting baseline. Our cost-accuracy analysis further shows that using the ARCHER harness, a self-hosted open-weights model can reach 97.8% of frontier-API accuracy at a quarter of the cost, making data-sovereign compliance checking practical.
As corporate AI adoption continues to grow, enterprise-grade LLM agents are being deployed into sensitive contexts such as hiring, healthcare, and finance. In these contexts, compliance with rules specified in an agent's system context is a first-order legal concern. Currently, no evaluation framework systematically measures which LLM models tend to violate compliance rules, especially under pressure from a persistent user, a hurried manager, or circumstances where violation is convenient or attractive. We introduce PACT (Pressure-Applied Compliance Testing), a benchmark for rule-following under pressure in AI agents assisting employees in daily tasks across twelve regulated enterprise domains and forty-eight scenarios, each set in a realistic multi-turn conversation. Each benchmark item pairs a standing rule against a rule-violating shortcut, and applies a battery of pressures across different wordings and system-prompt modes. We construct PACT component by component under strict LLM-as-judge auditing to ensure samples are unambiguous, ungameable, and realistic enough to avoid eliciting evaluation-aware behavior. We use PACT to profile LLM compliance across six complementary metrics that create a holistic picture of an AI assistant's robustness under pressure and throughout multi-turn conversations, its transparency, and ability to correctly discern where a rule applies. We aggregate this profile into PACTScore, a reliability-weighted compliance rate over all items and modes. Our results across 22 common LLM models spanning multiple providers and sizes show substantial variability in compliance across models and metric dimensions. Even the strongest assistants mis-apply a rule on 6 to 10% of items, and ordinary user pressure raises the violation rate by 65% on average. PACT highlights compliance risks in LLM assistants, motivating guardrails and careful model selection.
Mika Okamoto, Ansel Kaplan Erol
Georgia Institute of Technology · Decagon AI · Baseten
Current AI-assisted engineering workflows lack a built-in mechanism to maintain task-level verification and regulatory traceability at machine-speed delivery. Agile V addresses this gap by embedding independent verification and audit artifact generation into each task cycle. The framework merges Agile iteration with V-Model verification into a continuous Infinity Loop, deploying specialized AI agents for requirements, design, build, test, and compliance, governed by mandatory human approval gates. We evaluate three hypotheses: (H1) audit-ready artifacts emerge as a by-product of development, (H2) 100% requirement-level verification is achievable with independent test generation, and (H3) verified increments can be delivered with single-digit human interactions per cycle. A feasibility case study on a Hardware-in-the-Loop system (about 500 LOC, 8 requirements, 54 tests) supports all three hypotheses: audit-ready documentation was generated automatically (H1), 100% requirement-level pass rate was achieved (H2), and only 6 prompts per cycle were required (H3), yielding an estimated 10-50x cost reduction versus a COCOMO II baseline (sensitivity range from pessimistic to optimistic assumptions). We invite independent replication to validate generalizability.