Beyond Synthetic Benchmarks: Evaluating LLM Performance on Real-World Class-Level Code Generation
Organizations: Concordia University, Montréal, QC, Canada
Abstract
Large language models (LLMs) have advanced rapidly from well-studied function-level code generation to emerging repository-level approaches, yet this progression reveals a critical gap: class-level code generation, which is the fundamental building block connecting individual functions to complete repositories, remains systematically underexplored. Understanding class-level capabilities is a prerequisite to repository-level success, as current approaches cannot distinguish whether failures stem from basic implementation deficiencies or sophisticated context management issues. We introduce RealClassEval, a benchmark derived from real-world open-source repositories, comprising 400 classes divided into seen and unseen partitions based on model training cutoffs. Our systematic evaluation across seven state-of-the-art LLMs reveals fundamental limitations masked by synthetic benchmarks: while LLMs achieve 84 to 89% correctness on synthetic tasks, they attain only 25 to 34% on real-world classes, with minimal distinction between seen and unseen codebases. Notably, this performance range converges with concurrent work using rich natural language descriptions and repository context, suggesting that specification complexity cannot overcome core implementation bottlenecks. Documentation provides marginal improvements (1 to 3%), while retrieval augmentation yields greater gains (4 to7%), specifically with incomplete specifications. Error analysis reveals AttributeError, TypeError, and AssertionError dominate 84% of failures, with distinct patterns between synthetic (logic-focused) and real-world (object-oriented) scenarios. These findings establish the diagnostic foundation necessary for advancing both fundamental LLM capabilities and sophisticated repository-level approaches, while providing actionable guidance for production deployment strategies.
Figures & tables
| Model | Release Date | Knowledge Cutoff | Parameters | Key Characteristics |
| Qwen 2.5 Coder | September 2024 | June 2024 | 32B | Code-specific, optimized for code generation and reasoning |
| GPT-4.1 | April 2025 | June 2024 | 1.8T (estimated) | General-purpose, advanced reasoning and instruction-following |
| GPT-5 | August 2025 | October 2024 | 2.5T (estimated) | General-purpose, state-of-the-art reasoning and context handling |
| GPT-OSS | August 2025 | June 2024 | 20B | General-purpose (open), strong code capabilities |
| Codestral | January 2025 (Latest) | Unknown | 100B | Code-specific, specialized for code, lightweight and efficient |
| DeepSeek-V3 | August 2025 | July 2024 | 126B | General-purpose, focused on long-context code understanding |
| Comparison | Mean Diff. Range (95% CI) | Effect Size Range | # Models Significant / 7 |
| ClassEval vs. Pre-Cutoff | 46.93–61.24% | Medium–Large | 7 / 7 |
| ClassEval vs. Post-Cutoff | 43.96–57.05% | Medium–Large | 7 / 7 |
| Pre-Cutoff vs. Post-Cutoff | -9.05–4.57% | Negligible | 1 / 7 (GPT-OSS) |
| Comparison | Mean Diff. Range (95% CI) | Effect Size Range (Cramér’s V) | # Models Significant / 7 |
| ClassEval vs. Pre-Cutoff | 51.75–62.17% | Medium–Large | 7 / 7 |
| ClassEval vs. Post-Cutoff | 54.20–58.28% | Large | 7 / 7 |
| Pre-Cutoff vs. Post-Cutoff | -3.89–5.62% | Negligible | 0 / 7 (all FDR-adj. ) |
| Comparison | Mean % Diff Range | Improved | Worsened | Unchanged | Significant (after FDR Correction) | Effect Size |
| Full vs. Partial | -0.15% to -2.72% | 97 | 147 | 2,371 | 2/28 | Negligible |
| Full vs. No | -0.44% to -3.13% | 113 | 235 | 2,267 | 0/28 | Negligible |
| Combined | -0.15% to -3.13% | 210 | 382 | 4,638 | 2/56 | Negligible |
| Model | Dataset | Full vs. Partial (%) | Full vs. No (%) | Average Benefit (%) |
| Codestral | Pre-Cutoff | 2.72 | 3.13 | 2.93 |
| Post-Cutoff | 1.01 | 0.80 | 0.90 | |
| Deepseek-V3 | Pre-Cutoff | 0.93 | 1.59 | 1.26 |
| Post-Cutoff | 1.51 | 1.32 | 1.42 | |
| GPT-4.1 | Pre-Cutoff | 0.51 | 2.11 | 1.31 |
| Post-Cutoff | 1.02 | 0.90 | 0.96 |
| Dataset | Classes Helped | Classes Hurt | Benefit Ratio | Net Benefit (%) |
| Pre-Cutoff | 210 (8.9%) | 133 (5.6%) | 1.58:1 | 3.3% |
| Post-Cutoff | 172 (7.1%) | 80 (3.3%) | 2.14:1 | 3.8% |
| Docstring Condition | Mean Percentage Point Improvement | Improved | Worsened | Significant (p FDR ¡0.05) |
| Full | 1.55 | 81 | 51 | 0/7 |
| Partial | 4.25 | 125 | 55 | 5/7 |
| No | 3.58 | 141 | 77 | 0/7 |
| Overall | 3.11 | 347 | 183 | 5/21 |
| Docstring | Model | Mean Pass | FDR-Adjusted | Improved/ |
| Type | Rate % Diff. | Worsened | ||
| Full | Codestral | 1.79 | 0.536 | 11/6 |
| Deepseek-V3 | 3.07 | 0.536 | 16/9 | |
| GPT-4.1 | 0.84 | 0.617 | 6/4 | |
| GPT-5 | -0.77 | 0.798 | 12/13 | |
| GPT-OSS | 2.37 | 0.536 | 17/8 |
| Error Type | Count | Percentage (%) |
| AttributeError | 11,270 | 43.84 |
| TypeError | 5,566 | 21.65 |
| AssertionError | 4,759 | 18.51 |
| Other | 2,679 | 10.42 |
| ValueError | 702 | 2.73 |
| NameError | 339 | 1.32 |
| Model | Comparison | Effect Size | ||
| Codestral | ClassEval vs Pre-Cutoff | 5.46e-144 | 1.04e-143 | Large |
| ClassEval vs Post-Cutoff | 3.10e-116 | 4.65e-116 | Large | |
| Pre-Cutoff vs Post-Cutoff | 1.01e-28 | 1.32e-28 | Small | |
| Deepseek-V3 | ClassEval vs Pre-Cutoff | 5.91e-164 | 2.07e-163 | Large |
| ClassEval vs Post-Cutoff | 1.75e-142 | 3.07e-142 | Large | |
| Pre-Cutoff vs Post-Cutoff | 3.23e-30 | 4.52e-30 | Small |
| Error Type | ClassEval | Pre-Cutoff | Post-Cutoff |
| AssertionError | 71.80 | 5.34 | 17.57 |
| AttributeError | 2.14 | 49.41 | 45.67 |
| TypeError | 2.33 | 22.14 | 23.78 |
| NameError | 14.44 | 0.12 | 0.27 |
| Other | 0.63 | 18.29 | 9.16 |
| ValueError | 5.30 | 2.81 | 2.53 |
| Error Type | Non-RAG | RAG | Absolute | Non-RAG | RAG | Percentage Point |
| Count | Count | Difference | % | % | Difference | |
| Reductions | ||||||
| AttributeError | 639 | 579 | 45.97 | 48.05 | +2.08 | |
| Other | 129 | 76 | 9.28 | 6.31 | ||
| AssertionError | 233 | 190 | 16.76 | 15.77 | ||
| ValueError | 46 | 28 | 3.31 | 2.32 | ||
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Filtering Criterion | Pre-Cutoff | Post-Cutoff |
| Initial Pool | ||
| Filtered on Project Size and Activity | ||
| Filtered on License Declaration | ||
| Filtered on Code Quality Indicators |
| Model | Comparison | Mean Diff (95% CI) | FDR-Adjusted for Mann-Whitney U Tests | Effect Size |
| Codestral | ClassEval vs. Pre-Cutoff | 46.93 [39.46, 54.93] | 1.0233e-10 | Medium |
| ClassEval vs. Post-Cutoff | 51.50 [44.36, 58.63] | 1.4623e-16 | Large | |
| Pre-Cutoff vs. Post-Cutoff | 4.57 [-4.32, 13.74] | 0.4623 | Negligible | |
| Deepseek-V3 | ClassEval vs. Pre-Cutoff | 50.50 [42.82, 57.75] | 1.9786e-11 | Medium |
| ClassEval vs. Post-Cutoff | 49.38 [42.82, 56.03] | 3.5676e-14 | Large | |
| Pre-Cutoff vs. Post-Cutoff | -1.12 [-9.49, 7.59] | 0.6604 | Negligible |
| Model | Comparison | Mean Diff. (95% CI) | FDR-Adjusted | Effect Size |
| Codestral | ClassEval vs. Pre-Cutoff | 51.75 [47.28, 56.83] | 1.9032e-106 | Medium |
| ClassEval vs. Post-Cutoff | 57.37 [53.22, 61.24] | 1.7756e-168 | Large | |
| Pre-Cutoff vs. Post-Cutoff | 5.62 [-0.32, 11.33] | 0.1034 | Negligible | |
| Deepseek-V3 | ClassEval vs. Pre-Cutoff | 54.44 [49.81, 59.06] | 4.7305e-126 | Medium |
| ClassEval vs. Post-Cutoff | 55.99 [52.14, 59.88] | 7.6134e-174 | Large | |
| Pre-Cutoff vs. Post-Cutoff | 1.55 [-3.73, 7.40] | 0.7654 | Negligible |
Code Generation
Code generation studies methods for turning natural-language, visual, or formal specifications into executable programs and code edits.
See all 344 papers