Large language models (LLMs) have advanced rapidly from well-studied function-level code generation to emerging repository-level approaches, yet this progression reveals a critical gap: class-level code generation, which is the fundamental building block connecting individual functions to complete repositories, remains systematically underexplored. Understanding class-level capabilities is a prerequisite to repository-level success, as current approaches cannot distinguish whether failures stem from basic implementation deficiencies or sophisticated context management issues. We introduce RealClassEval, a benchmark derived from real-world open-source repositories, comprising 400 classes divided into seen and unseen partitions based on model training cutoffs. Our systematic evaluation across seven state-of-the-art LLMs reveals fundamental limitations masked by synthetic benchmarks: while LLMs achieve 84 to 89% correctness on synthetic tasks, they attain only 25 to 34% on real-world classes, with minimal distinction between seen and unseen codebases. Notably, this performance range converges with concurrent work using rich natural language descriptions and repository context, suggesting that specification complexity cannot overcome core implementation bottlenecks. Documentation provides marginal improvements (1 to 3%), while retrieval augmentation yields greater gains (4 to7%), specifically with incomplete specifications. Error analysis reveals AttributeError, TypeError, and AssertionError dominate 84% of failures, with distinct patterns between synthetic (logic-focused) and real-world (object-oriented) scenarios. These findings establish the diagnostic foundation necessary for advancing both fundamental LLM capabilities and sophisticated repository-level approaches, while providing actionable guidance for production deployment strategies.
Figures & tables
Figure 1: Workflow for RealClassEval dataset curation.
Model
Release Date
Knowledge Cutoff
Parameters
Key Characteristics
Qwen 2.5 Coder
September 2024
June 2024
32B
Code-specific, optimized for code generation and reasoning
GPT-4.1
April 2025
June 2024
1.8T (estimated)
General-purpose, advanced reasoning and instruction-following
GPT-5
August 2025
October 2024
2.5T (estimated)
General-purpose, state-of-the-art reasoning and context handling
GPT-OSS
August 2025
June 2024
20B
General-purpose (open), strong code capabilities
Codestral
January 2025 (Latest)
Unknown
100B
Code-specific, specialized for code, lightweight and efficient
DeepSeek-V3
August 2025
July 2024
126B
General-purpose, focused on long-context code understanding
Table 1: Characteristics of the evaluated LLMs.
Figure 2: Performance of different LLMs on ClassEval, Pre-Cutoff, and Post-Cutoff data.
Comparison
Mean Diff. Range (95% CI)
Effect Size Range
# Models Significant / 7
ClassEval vs. Pre-Cutoff
46.93–61.24%
Medium–Large
7 / 7
ClassEval vs. Post-Cutoff
43.96–57.05%
Medium–Large
7 / 7
Pre-Cutoff vs. Post-Cutoff
-9.05–4.57%
Negligible
1 / 7 (GPT-OSS)
Table 2: Per-Class Pass Rate Analysis, condensed across all seven models (full per-model breakdown in Table 14 , Appendix A.2 ).
Comparison
Mean Diff. Range (95% CI)
Effect Size Range (Cramér’s V)
# Models Significant / 7
ClassEval vs. Pre-Cutoff
51.75–62.17%
Medium–Large
7 / 7
ClassEval vs. Post-Cutoff
54.20–58.28%
Large
7 / 7
Pre-Cutoff vs. Post-Cutoff
-3.89–5.62%
Negligible
0 / 7 (all FDR-adj. p>0.10 )
Table 3: Pooled Chi-Square Analysis, condensed across all seven models (full per-model breakdown in Table 15 , Appendix A.2 ).
Figure 3: Violin plot of per-class pass rate distributions across docstring conditions (full, partial, no) for Pre-Cutoff and Post-Cutoff datasets, aggregated across all models. Violin plots use kernel density estimation to show data distribution; the y-axis extends beyond [0,1] due to KDE smoothing ( Silverman, 2018 ) , though all actual pass rate values are bounded between 0 and 1.
Comparison
Mean % Diff Range
Improved
Worsened
Unchanged
Significant (after FDR Correction)
Effect Size
Full vs. Partial
-0.15% to -2.72%
97
147
2,371
2/28
Negligible
Full vs. No
-0.44% to -3.13%
113
235
2,267
0/28
Negligible
Combined
-0.15% to -3.13%
210
382
4,638
2/56
Negligible
Table 4: Summary of docstring ablation results across all models and datasets with respect to the number of test cases. Negative differences indicate full docstrings cause more test cases to pass compared to the alternative condition.
Model
Dataset
Full vs. Partial (%)
Full vs. No (%)
Average Benefit (%)
Codestral
Pre-Cutoff
2.72
3.13
2.93
Post-Cutoff
1.01
0.80
0.90
Deepseek-V3
Pre-Cutoff
0.93
1.59
1.26
Post-Cutoff
1.51
1.32
1.42
GPT-4.1
Pre-Cutoff
0.51
2.11
1.31
Post-Cutoff
1.02
0.90
0.96
Table 5: Model-specific performance improvements with full docstrings. Percentages represent pass rate increases when using full documentation relative to partial or no docstrings. Bold entries achieved statistical significance after FDR correction (p FDR <0.05) .
Dataset
Classes Helped
Classes Hurt
Benefit Ratio
Net Benefit (%)
Pre-Cutoff
210 (8.9%)
133 (5.6%)
1.58:1
3.3%
Post-Cutoff
172 (7.1%)
80 (3.3%)
2.14:1
3.8%
Table 6: Temporal analysis of docstring impact. Benefit ratio is the ratio of classes that improved to classes that worsened with full docstrings. The Wilcoxon test comparing Pre-Cutoff vs. Post-Cutoff benefit ratios across models was non-significant.
Docstring Condition
Mean Percentage Point Improvement
Improved
Worsened
Significant (p FDR ¡0.05)
Full
1.55
81
51
0/7
Partial
4.25
125
55
5/7
No
3.58
141
77
0/7
Overall
3.11
347
183
5/21
Table 7: Summary of RAG effects across docstring conditions on Post-Cutoff dataset. Positive mean differences indicate RAG outperforms non-RAG. Per-type FDR correction applied separately to each condition (7 tests per condition).
Docstring
Model
Mean Pass
FDR-Adjusted p
Improved/
Type
Rate % Diff.
Worsened
Full
Codestral
1.79
0.536
11/6
Deepseek-V3
3.07
0.536
16/9
GPT-4.1
0.84
0.617
6/4
GPT-5
-0.77
0.798
12/13
GPT-OSS
2.37
0.536
17/8
Table 8: Model-specific RAG effects organized by docstring condition. A positive difference means RAG performed better, and vice versa. Bold entries indicate statistical significance.
Error Type
Count
Percentage (%)
AttributeError
11,270
43.84
TypeError
5,566
21.65
AssertionError
4,759
18.51
Other
2,679
10.42
ValueError
702
2.73
NameError
339
1.32
Table 9: Overall error type distribution across all conditions.
Model
Comparison
praw
pFDR
Effect Size
Codestral
ClassEval vs Pre-Cutoff
5.46e-144
1.04e-143
Large
ClassEval vs Post-Cutoff
3.10e-116
4.65e-116
Large
Pre-Cutoff vs Post-Cutoff
1.01e-28
1.32e-28
Small
Deepseek-V3
ClassEval vs Pre-Cutoff
5.91e-164
2.07e-163
Large
ClassEval vs Post-Cutoff
1.75e-142
3.07e-142
Large
Pre-Cutoff vs Post-Cutoff
3.23e-30
4.52e-30
Small
Table 10: Dataset-wise error distribution comparison with global FDR correction (21 tests). All comparisons are significant after correction.
Figure 4: Distribution of Error Types Across Datasets.
Error Type
ClassEval
Pre-Cutoff
Post-Cutoff
AssertionError
71.80
5.34
17.57
AttributeError
2.14
49.41
45.67
TypeError
2.33
22.14
23.78
NameError
14.44
0.12
0.27
Other
0.63
18.29
9.16
ValueError
5.30
2.81
2.53
Table 11: Error type distributions by dataset (%). ClassEval shows AssertionError dominance, while real-world datasets show AttributeError and TypeError prominence.
Figure 5: Error Type Changes with RAG Implementation.
Error Type
Non-RAG
RAG
Absolute
Non-RAG
RAG
Percentage Point
Count
Count
Difference
%
%
Difference
Reductions
AttributeError
639
579
−60
45.97
48.05
+2.08
Other
129
76
−53
9.28
6.31
−2.97
AssertionError
233
190
−43
16.76
15.77
−0.99
ValueError
46
28
−18
3.31
2.32
−0.99
Table 12: RAG error substitution patterns for significant models with partial docstrings (aggregate across Deepseek-V3, GPT-5, Llama 4 Maverick). Negative differences indicate reductions, positive differences indicate increases.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Filtering Criterion
Pre-Cutoff
Post-Cutoff
Initial Pool
13,590
91,048
Filtered on Project Size and Activity
5,348
13,197
Filtered on License Declaration
4,021
4,292
Filtered on Code Quality Indicators
2,970
1,735
Appendix
Table 13: Repository filtering funnel for Pre-Cutoff and Post-Cutoff datasets.