Code generation models are actively being used in professional and educational settings. However, the competence of large language models (LLMs) to generate algorithmically diverse code remains largely unknown. We propose the DivAlgo benchmark, a large-scale dataset constituting algorithmically diverse solutions to programming problems. We also introduce a lightweight similarity metric to enable large-scale evaluation. Our experiments across 24 modern code generation models found that uncontrolled model-generated code is significantly less diverse than human solutions (capturing ~31% of human diversity), and even the best diverse code generation strategy could achieve only ~70% of human diversity. Moreover, while it was possible to steer the models to generate solutions based on human strategies, they were unable to reproduce the diversity of the human solutions. These findings indicate the limitations of code generation models in capturing the algorithmic diversity humans exhibit, and that overexposure to model-generated code can narrow programmers' algorithmic vocabulary.
Figures & tables
Figure 1: Illustrative example from our algorithmic diversity benchmark’s test split demonstrating two algorithmically distinct solutions for the same problem. Solution A solves the problem using the simulation paradigm with an array data structure, whereas Solution B adopts a bitmasking paradigm with the binary lifting technique using a 2D matrix data structure. Both solutions have different time and space complexity. The bar graphs show the code similarity of the two solutions, and the labels show the LLM judgments. The ticks in the bar graph indicate the similarity thresholds, below which the solution pair is considered algorithmically diverse in the respective metric space.
Human solutions
Ontology-based algorithmic diversity
Split
instances
C++ sol.
total
median
strong
medium
weak
identical
train
5,623
11,246
148,242
25
2,429
1,463
1,229
502
val
1,000
2,000
25,915
24
419
257
231
93
test
1,000
2,000
26,145
25
428
280
212
80
all
7,623
15,246
200,302
25
3,276
2,000
1,672
675
Table 1: Dataset statistics of the algorithmic diversity benchmark.
Figure 2: Algorithmic diversity ( VendiCodeX-SFT ) of 50 uncontrolled LLM-generated solutions against human-written solutions for the test set problems (detailed results in Table 4 Appendix D ).
Figure 3: Algorithmic diversity ( VendiCodeX-SFT ) of three diverse text generation strategies for three LLMs. (Left) temperature variation, verbalized sampling, and iterative constrained generation on the test set problems and the corresponding scores for human solutions for matched problems. (Right) increase in algorithmic diversity of human solutions and the best-performing LLM, Gemma-4-31b , for 412 problems where at least 25 correct solutions are available for both human and LLM solutions. T=0.8 is denoted as i.i.d. , independent and identically distributed samples (same as the results in Figure 2 ). We report the human solutions corresponding to each model’s success rate to enable a fair comparison across model generations. For the detailed results, refer to Table 5 in Appendix D .
Figure 4: (left) LLM-generated solution’s preference for using one of the two reference approaches vs. the popularity of the same approach in human-written solutions. Alignments for model and human preferences were obtained using the cosine similarities obtained from the CodeX-SFT embedding model. (right) Algorithmic diversity of ensemble of LLM-generated programs ρ at the matched k∗ to the human reference solutions along with the most diverse LLMs for i.i.d. setting and the diverse code generation settings. ρall denotes all 24 LLMs used in this work, ρ3 denotes the three open-source models used for diverse code generation experiments (Figure 3 left).
Model
n
i.i.d.
verbalized
iterative
steer (A + B)
human
Strong — targets differ in paradigm (274 problems)
Gemma-4-31b
245
28.2 [22,33]
50.2 [44,57]
60.8 [54,67]
46.6 [41,52]
70.0 [66,75]
Gemma-4-E4B
179
34.1 [27,41]
31.3 [25,38]
44.1 [37,51]
48.3 [42,55]
70.8 [65,76]
Phi-4
121
24.8 [17,33]
29.8 [22,38]
24.8 [17,32]
26.9 [20,34]
67.7 [61,74]
Medium — targets differ in technique (104 problems)
Gemma-4-31b
87
21.8 [14,31]
32.2 [23,43]
37.9 [28,48]
40.9 [32,50]
59.0 [50,68]
Table 2: Algorithmic coverage ( AlgoCov ) of LLM-generated solutions and corresponding human-written solutions over the two human solutions in the final dataset (referred as A and B). We sampled 10 solutions for each strategy; limiting to 274 strong , 104 medium and 106 weak instances where the ontology-labels are mutually exclusive for the respective classification dimension.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Algorithmic similarity metric agreements on 100 uniformly random solution pairs. (Left) Rank-based Spearman’s ρ and (right) ROC-AUC against the majority of metrics; for LLM-judges, the majority of the other 5 LLMs is used, for CodeXEmbed and PDG all six LLM-judges are used.
Figure 6: Leave-one-out agreement between six proprietary LLMs for the three-level ontology labeling task.
Figure 7: Evaluation of the algorithmic similarity measurement metrics on 700 instances, evaluated against the aggregated majority ground truth from six proprietary models. The confidence interval is obtained via percentile bootstrap over the solution pairs (repeated 3,000 times).
Type-1
Type-2
Type-3
Type-4
T3 − T4
embedding metrics (our implementation)
CodeX-SFT
0.93
0.85
0.87
0.65
+0.23
CodeXEmbed
0.99
0.97
0.97
0.89
+0.08
jina-SFT
0.95
0.95
0.92
0.83
+0.09
jina-embeddings-v2-base-code
0.93
0.94
0.92
0.78
+0.14
embedding metrics, as reported in Zhang and Lu (2026)
Appendix
Table 3: Measuring algorithmic overlap of our trained evaluation metrics on the hand-curated solution pairs across four types of solution pairs in Python ( n=4,3,6 and 17 pairs for each type respectively).
Figure 8: Distribution of ontology labels in the two C++ solutions per programming problem in the algorithmic diversity dataset.
Success Statistics
#tokens
self-div
VendiCodeX-SFT ( k∗ matched / all solutions)
Model
Pass rate
Pass@N
Pass ≥2 @N
med
max
trunc (%)
PDG
CodeX-SFT
k∗/kLLM
VSLLM
VShum
Gemini-3.8-flash
80.9
954
945
–
–
0.7
0.046
0.175
24.7 / 42.8
4.18 / 4.84
13.29 / 14.25
Gemma-4-31b
76.8
927
913
1,073
8,192
5.1
0.049
0.173
24.7 / 42.0
4.14 / 4.75
13.15 / 14.23
Gemma-4-26B
74.0
929
917
2,447
16,384
7.2
0.058
0.171
24.3 / 40.3
3.50 / 3.96
13.01 / 14.26
Gemma-4-12b
64.1
881
858
1,576
16,384
10.0
0.054
0.178
23.3 / 37.3
3.71 / 4.13
12.24 / 14.16
Gemini-flash-lite
53.6
817
782
–
–
0.0
0.050
0.187
21.9 / 34.2
4.32 / 4.75
11.29 / 13.80
Appendix
Table 4: Algorithmic diversity results for unconditioned zero-shot generations on the 1,000 test set programming problems. We present three success statistics: pass rate - percentage of generated solutions that pass all test cases, Pass@N - number of instances with at least one correct generation, and Pass ≥2 @N - number of instances with at least two correct solutions to compute within problem diversity. We also provide the generation statistics, including the median ( med ) and maximum ( max ) number of output tokens, as well as the percentage of solutions that were truncated ( trunc ). For per-problem population diversity, we present the self-diversity ( self-div ) of generated solutions as the average distance of all correct solutions using two distance measurements: program dependence graphs ( PDG ) and the trained embedding model CodeX-SFT (Section B ). We also provide the VendiCodeX-SFT (Eq. 1 ) scores on two settings, matched and all solutions . Matched Vendi score subsamples k∗=min(kLLM,khum) number of solutions from the LLM-generated solutions and the human references over 200 draws without replacement, whereas the all solutions Vendi score is computed at the entire pool size of each solution population. We also provide the human reference diversity scores for all 1,000 test instances.
Success Statistics
#tokens
self-div
VendiCodeX-SFT ( k∗ matched / all solutions)
Model
Sampling
Pass rate
Pass ≥2 @N
k
med
max
trunc (%)
PDG
CodeX-SFT
k∗/kLLM
VSLLM
VShum
Phi-4
i.i.d.
31.4
618
25.3
356
2,048
0.09
0.066
0.169
18.5 / 25.3
3.22 / 3.33
8.79 / 13.18
temp 0.2
33.5
538
31.1
347
2,048
0.08
0.032
0.087
20.6 / 31.1
1.85 / 1.89
9.56 / 12.95
temp 1.2
27.6
587
23.4
380
2,048
1.76
0.079
0.197
17.7 / 23.4
3.73 / 3.85
8.23 / 12.96
iterative
24.0
604
19.8
370
2,048
0.02
0.072
0.219
16.7 / 19.8
4.16 / 4.31
8.01 / 13.06
verbalized
22.7
561
20.2
1,807
13,000
0.01
0.067
0.191
16.8 / 20.2
3.60 / 3.71
8.07 / 13.00
Appendix
Table 5: Algorithmic diversity of diversity conditioned generation approaches on the 1,000 test set programming problems. We present the results for three open-weight models over three strategies: temperature variation with 50 generations ( i.i.d. denotes the standard T=0.8 setting), verbalized sampling (10 independent calls per problem asking to generate five solutions per call), and iterative generation (10 independent chains per problem with five iterative constrained generations per generation chain). All conditions are evaluated on the same 50 generated solutions, and the statistics and diversity measurements are the same as described in Table 4 .
Figure 9: Illustrative example of strong algorithmic diversity from the train split. The two solutions have different programming paradigms. The bar graphs on the bottom left provide the code similarity of different similarity metrics, with the ticks indicating a threshold calibrated on the 10th percentile of randomly selected 150 train instances. Similarity below the threshold is considered algorithmically diverse as per the metric. We also provide the pairwise LLM judgments from the second filtration stage on the bottom right.
Figure 10: Illustrative example of medium algorithmic diversity from the train split. The two solutions have the same programming paradigm, but different techniques. The bar graphs at the bottom left show code similarity across different similarity metrics, with the ticks indicating a threshold calibrated to the 10th percentile of randomly selected 150 training instances. Similarity below the threshold is considered algorithmically diverse as per the metric. We also provide the pairwise LLM judgments from the second filtration stage on the bottom right.
Figure 11: Illustrative example of weak algorithmic diversity from the train split. The two solutions use the same programming paradigm and techniques but different data structures. The bar graphs at the bottom left show code similarity across different similarity metrics, with the ticks indicating a threshold calibrated to the 10th percentile of randomly selected 150 training instances. Similarity below the threshold is considered algorithmically diverse as per the metric. We also provide the pairwise LLM judgments from the second filtration stage on the bottom right.
Figure 12: Illustrative example of identical algorithmic diversity from the train split. The two solutions share the same programming paradigm, techniques, and data structures, and pass through all the filtration steps. The bar graphs at the bottom left show code similarity across different similarity metrics, with the ticks indicating a threshold calibrated to the 10th percentile of randomly selected 150 training instances. Similarity below the threshold is considered algorithmically diverse as per the metric. We also provide the pairwise LLM judgments from the second filtration stage on the bottom right.
Figure 13: The LLM-judge prompt used to obtain the 4-way ordinal classification of algorithmic similarity between two programs for the same problem.
Figure 14: The prompt used to obtain the ontology labels for the two representative solutions in the final dataset.
Figure 15: The prompt used to generate i.i.d. zero-shot solutions for different temperature values.
Figure 16: The prompt used to generate the solutions using the verbalized sampling approach Zhang et al. (2025) . We limit to 5 outputs per generation and re-run it 10 times to obtain 50 solutions for direct comparison with the i.i.d. setting.
Figure 17: The prompt used to iteratively generate diverse solutions. We limit the iterations to 5 generations and re-run it 10 times to obtain 50 solutions for direct comparison with the i.i.d. setting.
Figure 18: The prompts used to generate the steer (A+B) solutions that are instructed to adhere to a particular algorithmic strategy.
High pass rates on established programming benchmarks such as HumanEval and LiveCodeBench do not always show whether a model can reason about algorithms. Many fixed benchmarks eventually become part of the public training ecosystem through released problem statements, editorials, and generated solutions, allowing later models to improve partly by exposure rather than by stronger algorithmic ability. We introduce ALGOBENCH, a framework that automatically builds novel algorithmic problems from known competitive-programming problems through structured constraint-shifting transformations. Each accepted ALGOBENCH variant is traceable to a source problem, but must make the original reference algorithm fail. Beyond pass@k, we introduce complexity-aware metrics -- including OPTT, OPTS, TRAPRATE, GAPT, and CONSENS -- to test whether a solution is not only functionally correct but also asymptotically suitable for the generated problem. Experiments across multiple LLMs and prompting strategies show that performance drops sharply on ALGOBENCH variants, retrieval can increase reuse of the old algorithm, and many correct-looking solutions fail to meet the required complexity. Error analysis shows that failures are mainly algorithmic rather than implementation-level, suggesting that ALGOBENCH evaluates adaptation beyond functional correctness.
Xinyuan Song, Zekun Cai, Liang Zhao
Emory University, Atlanta, GA, USA · The University of Tokyo, Tokyo, Japan · LocationMind, Tokyo, Japan
Large language models (LLMs) have demonstrated strong performance in code generation, where success depends on both recalling relevant algorithmic knowledge and reasoning about how to apply it. However, existing LLM pipelines are opaque, with no explicit separation between these two components. We argue that for well-known algorithms whose canonical implementations are widely accessible in pretraining corpora, code generation is better measured as parametric code retrieval: reproducing a named algorithm from internalised knowledge rather than synthesizing a novel one. We introduce AlgoREval, a benchmark of 599 problems spanning classical 77 algorithms across 14 domains, 7 programming languages, and 4 graph-input representations to evaluate this capability in isolation, and assess 15 models (7B--34B parameters) in a zero-shot setting. We find substantial variation in retrieval accuracy across languages and input representations, even for widely documented algorithms and show that prompt augmentation with retrieved code snippets or structured algorithmic hints improve accuracy on complex algorithms, while SFT achieves broader language gains and GRPO achieves larger per-language gains on specific languages. Together, our results establish parametric code retrieval as a distinct, measurable capability and caution against deploying AI-generated algorithmic code without systematic validation.\footnote{Code and dataset are available at https://github.com/Nickil21/AlgoREval
Nickil Maveli, Antonio Vergari, Shay B. Cohen
School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh, EH8 9AB
Large language models (LLMs) now generate substantial production code, often for tasks with multiple valid algorithmic solutions. Incidental prompt cues, meaning contextual words or metadata outside the task specification, can steer which algorithm the model selects, even when all outputs pass the same tests. Prompt sensitivity is well studied as a tool to improve output quality. Here, output policy means algorithm choice under fixed correctness. We define algorithm steering as cue-induced shifts in algorithm-family distributions and run 46,535 controlled experiments across 11 tasks, 19 cue types (18 channels plus a memoization semantic-vs-surface ablation that preserves meaning while changing typography and punctuation), and 15 model configurations. We find large, systematic shifts in algorithm-family distributions (up to 100 pp), largely consistent with cue semantics, including in applied tasks such as rate limiting. Direct algorithm naming is the most reliable mitigation we tested. Accidental context therefore creates an "invisible lottery" over performance, security, and maintainability.