cs.SEApr 22, 2025

OpenClassGen: A Large-Scale Corpus of Real-World Python Classes for LLM Research

Authors: Musfiqur Rahman, SayedHassan Khatoonabadi, Emad Shihab

Organizations: Concordia University Montréal, Canada

Abstract

Existing class-level code generation datasets are either synthetic (ClassEval: 100 classes) or insufficient in scale for modern training needs (RealClassEval: 400 classes), hindering robust evaluation and empirical analysis. We present OpenClassGen, a large-scale corpus of 324,843 Python classes extracted from 2,970 engineered open-source projects. Each entry pairs a human-written class with its corresponding skeleton, which comprises class and method signatures with associated docstrings, and is enriched with 27 static code metrics covering complexity, coupling, cohesion, and inheritance properties. Unlike prior benchmarks that require repository-level context resolution, OpenClassGen provides self-contained class skeletons that serve as complete generation specifications. We demonstrate the corpus's utility by evaluating three LLMs (GPT-o4-mini, Claude-4-Sonnet, Qwen-3-Coder) on a curated, executable subset of 300 classes, enriched with test suites achieving 58% branch coverage. Results show strong semantic similarity (CodeBERTScore-F3: 0.89) but moderate functional correctness (pass rate: 0.33), with substantial variance across models. This variance, along with diverse class characteristics, confirms that OpenClassGen enables meaningful differentiation of LLM capabilities. The dataset supports diverse use cases, including fine-tuning, retrieval-augmented generation, difficulty modelling, and failure mode analysis. The complete dataset and curation scripts are publicly available at https://zenodo.org/records/18409150.

Figures & tables

Explore similar work

CardsList
  1. ClassEval-Pro: A Cross-Domain Benchmark for Class-Level Code Generation

    Apr 29, 2026Yeheng Chen, Chaoxiang Xie, Yuling Shi +4Code GenerationSynthetic Benchmark Generation

  2. Beyond Synthetic Benchmarks: Evaluating LLM Performance on Real-World Class-Level Code Generation

    Oct 30, 2025Musfiqur Rahman, SayedHassan Khatoonabadi, Emad ShihabCode GenerationLLM Code Generation

  3. CodeAlchemy: Synthetic Code Rewriting at Scale

    Jun 8, 2026Ankit Gupta, Aditya Prasad, Rameswar PandaSynthetic Data PretrainingSynthetic Data Generation