cs.CROct 29, 2026

RiskyPy: Toward Model-Calibrated Prompt Generation for Joint Security and Functionality Evaluation of LLM-Generated Code

Authors: Zacharie Chenail-Larcher, Daniele Lunghi, Melissa Tessa, Quentin Stiévenart, Jacques Klein, Tegawendé F. Bissyandé, Florent Avellaneda, Naouel Moha

Organizations: École de technologie supérieure, Montréal, Québec, Canada · University of Luxembourg, Luxembourg, Luxembourg · Université du Québec à Montréal, Montréal, Québec, Canada

Abstract

Evaluating approaches that improve the security of LLM code generation requires datasets that jointly assess security and functionality and remain adapted to the capabilities of the models being studied. However, existing benchmarks are costly to produce and are generally not calibrated to a chosen range of model capabilities. We introduce RiskyPy, an automated pipeline that generates security-oriented code-generation datasets calibrated to model proficiency using functionality and vulnerability canaries. We instantiate RiskyPy for Python using domains covering 18 CWEs from the 2025 CWE Top 25, producing 101 prompts. Experiments on Qwen3-4B and Seed-Coder-8B show that the generated tasks remain functionally accessible, with 57-72% of prompts yielding functional code, while only 5-10% of generations are free of Bandit findings, leaving room for joint evaluation of security and functionality of security improvement approaches.

Figures & tables

Explore similar work

CardsList
  1. Poster: Rethinking Security in LLM Code Generation through Real-World Risk Scenarios

    Jul 25, 2026Lixun Ma, Ruolong Ma, Bei Wang +4LLM Safety BenchmarksLLM Security

  2. Secure and Functional Code Generation with Self-Evaluation

    Oct 24, 2026Md Afif Al Mamun, Gias UddinSecure Code GenerationRetrieval-Augmented Code Generation

  3. SecCodeBench-V2 Technical Report

    Feb 17, 2026Longfei Chen, Ji Zhao, Lanxiao Cui +24Secure Code GenerationCode Generation Benchmarks