OpenClassGen: A Large-Scale Corpus of Real-World Python Classes for LLM Research
Organizations: Concordia University Montréal, Canada
Abstract
Existing class-level code generation datasets are either synthetic (ClassEval: 100 classes) or insufficient in scale for modern training needs (RealClassEval: 400 classes), hindering robust evaluation and empirical analysis. We present OpenClassGen, a large-scale corpus of 324,843 Python classes extracted from 2,970 engineered open-source projects. Each entry pairs a human-written class with its corresponding skeleton, which comprises class and method signatures with associated docstrings, and is enriched with 27 static code metrics covering complexity, coupling, cohesion, and inheritance properties. Unlike prior benchmarks that require repository-level context resolution, OpenClassGen provides self-contained class skeletons that serve as complete generation specifications. We demonstrate the corpus's utility by evaluating three LLMs (GPT-o4-mini, Claude-4-Sonnet, Qwen-3-Coder) on a curated, executable subset of 300 classes, enriched with test suites achieving 58% branch coverage. Results show strong semantic similarity (CodeBERTScore-F3: 0.89) but moderate functional correctness (pass rate: 0.33), with substantial variance across models. This variance, along with diverse class characteristics, confirms that OpenClassGen enables meaningful differentiation of LLM capabilities. The dataset supports diverse use cases, including fine-tuning, retrieval-augmented generation, difficulty modelling, and failure mode analysis. The complete dataset and curation scripts are publicly available at https://zenodo.org/records/18409150.
Figures & tables
| Dataset | Classes | Projects | Real-World |
| ClassEval ( Du et al., 2024 ) | 100 | — | ✗ |
| RealClassEval ( Rahman et al., 2025a ) | 400 | 12 | ✓ |
| JavaBench ( Cao et al., 2024 ) | 106 | 4 | ✗ |
| RepoClassBench ( Deshpande et al., 2024 ) | 130 | 26 | ✓ |
| OpenClassGen | 324,843 | 2,970 | ✓ |
| Field | Description |
|---|---|
| id | Unique index for each class. |
| repository_name | Name of the GitHub repository. |
| file_path | Path to the file containing the class. |
| class_name | Name of the class. |
| human_written_class | Complete class implementation with docstrings. |
| class_skeleton | Extracted skeleton with method bodies replaced by pass . |
| Property | Avg. | Std. | 25% | Med. | 75% |
|---|---|---|---|---|---|
| Method Count | 3.52 | 7.36 | 0.00 | 1.00 | 4.00 |
| Total Lines | 80.20 | 253.92 | 8.00 | 24.00 | 72.00 |
| Code Lines | 50.19 | 160.74 | 6.00 | 15.00 | 44.00 |
| Comment Lines | 20.86 | 102.85 | 0.00 | 3.00 | 15.00 |
| Blank Lines | 10.93 | 33.50 | 1.00 | 3.00 | 10.00 |
| Comment-to-Code Ratio | 0.53 | 1.71 | 0.00 | 0.22 | 0.63 |
| Characteristic | ClassEval | RealClassEval | OpenClassGen |
|---|---|---|---|
| Classes | 100 | 400 | 324,843 |
| Documentation | Complete | Variable | Variable |
| Structural metrics | ✗ | ✗ | 27 metrics |
| Domain diversity | Limited | Moderate | High |
| Syntactic | Semantic | Structural | Functional | ||
|---|---|---|---|---|---|
| Model | ROUGE-L | BLEU | CodeBERTScore-F3 | TSED | Pass Rate |
| GPT-o4-mini | 0.60 | 0.46 | 0.88 | 0.82 | 0.29 |
| Claude-4-Sonnet | 0.68 | 0.50 | 0.90 | 0.84 | 0.36 |
| Qwen-3-Coder | 0.67 | 0.49 | 0.90 | 0.84 | 0.35 |
| Average | 0.65 | 0.48 | 0.89 | 0.83 | 0.33 |
| Benchmark | Limitation | How OpenClassGen Addresses It |
|---|---|---|
| ClassEval | Synthetic data with complete documentation do not reflect real-world variability; 100 samples are insufficient for training or robust evaluation. | 324,843 classes from production repositories with natural documentation variability (19.7% fully documented, 44.4% partial, 36.0% undocumented). |
| RealClassEval | 400 samples limit statistical power; no structural metrics for difficulty analysis. | Three orders of magnitude larger scale enables robust statistical analysis; 27 static metrics per class support difficulty modelling and stratified evaluation. |
| JavaBench | Project-level granularity conflates class generation with dependency resolution; Java-only. | Class-level granularity isolates the generation task from dependency management; self-contained skeletons serve as complete specifications. |
| RepoClassBench | Repository context requirements limit scalability; small sample size. | No repository context required; skeletons are self-contained, enabling evaluation of 324,843 classes without dependency resolution infrastructure. |
Code Generation
Code generation studies methods for turning natural-language, visual, or formal specifications into executable programs and code edits.
See all 344 papers