Security teams face a challenge: the volume of newly disclosed Common Vulnerabilities and Exposures (CVEs) far exceeds the capacity to manually develop detection mechanisms. In 2025, the National Vulnerability Database published over 48,000 new vulnerabilities, motivating the need for automation. We present RuleForge, an AWS internal system that automatically generates detection rules--JSON-based patterns that identify malicious HTTP requests exploiting specific vulnerabilities--from structured Nuclei templates describing CVE details. Nuclei templates provide standardized, YAML-based vulnerability descriptions that serve as the structured input for our rule generation process. This paper focuses on RuleForge's architecture and operational deployment for CVE-related threat detection, with particular emphasis on our novel LLM-as-a-judge (Large Language Model as judge) confidence validation system and systematic feedback integration mechanism. This validation approach evaluates candidate rules across two dimensions--sensitivity (avoiding false negatives) and specificity (avoiding false positives)--achieving AUROC of 0.75 and reducing false positives by 67% compared to synthetic-test-only validation in production. Our 5x5 generation strategy (five parallel candidates with up to five refinement attempts each) combined with continuous feedback loops enables systematic quality improvement. We also present extensions enabling rule generation from unstructured data sources and demonstrate a proof-of-concept agentic workflow for multi-event-type detection. Our lessons learned highlight critical considerations for applying LLMs to cybersecurity tasks, including overconfidence mitigation and the importance of domain expertise in both prompt design and quality review of generated rules through human-in-the-loop validation.
Figure 2. 5×5 Generation Strategy showing five parallel rule candidates with confidence scores and iterative refinement process. The system generates multiple candidates simultaneously and selects the best performer based on validation results.
Figure 3. Systemic Feedback Loop mechanism showing how validation failures generate specific feedback that is integrated into subsequent rule generation attempts. Examples include feedback about missed patterns, overly broad rules, and specificity issues.
Figure 4. CVE Rule Selection Process Flow. The system processes CVEs without existing rules through three parallel analysis components: content-based keyword matching, CISA KEV status verification, and cybersecurity news feed presence checking. These inputs feed into a weighted scoring algorithm that ranks CVEs by priority, with the highest-scoring vulnerabilities selected first for automated rule generation.
Figure 5. Candidate Rule Validation Pipeline. Each generated rule passes sequentially through four stages: (1) Synthetic Testing, (2) LLM-as-a-Judge Confidence Scoring, (3) IP Validation against production web traffic, and (4) Human Review. Approved rules proceed to Production Deployment. Failures at any stage are routed to Feedback Integration, which provides stage-specific reasoning back to the Rule Generation Engine for iterative refinement.
Figure 6. Reliability diagram showing predicted confidence versus actual accuracy for the LLM-as-a-judge confidence scoring system. Points above the diagonal line indicate underconfidence, while points below indicate overconfidence. The system achieves AUROC of 0.75 and ECE of 0.17 on 100 test examples.
Figure 7. Rules passing and failing IP validation across confidence score thresholds. Higher thresholds preserve most passing rules while reducing failures.
Figure 8. CVE Rule Production Rate comparing GenAI Automated versus Human generation in the last four months of 2025, showing productivity improvements with GenAI automation. Absolute qualitative metrics omitted on purpose, see note in the main text.
Figure 9. Comparison of LLM-as-a-judge performance using (a) negative phrasing (asking about problems with the rule) versus (b) positive phrasing (asking about rule correctness). Negative phrasing achieves better AUROC and lower ECE, i.e. improves calibration and discriminative power.
Figure 10. Performance comparison of generic prompts: (a) generic confidence questions showing poor calibration and (b) generic false positive/negative questions demonstrating worse AUROC than specifically worded prompts.