DivAlgo: A systematic evaluation of algorithmic diversity for code generation models

Anubhav Jangra; Smaranda Muresan

cs.SEOct 13, 2026

DivAlgo: A systematic evaluation of algorithmic diversity for code generation models

Authors: Anubhav Jangra, Smaranda Muresan

Organizations: Department of Computer Science Columbia University, USA · Barnard College Columbia University, USA

Abstract

Code generation models are actively being used in professional and educational settings. However, the competence of large language models (LLMs) to generate algorithmically diverse code remains largely unknown. We propose the DivAlgo benchmark, a large-scale dataset constituting algorithmically diverse solutions to programming problems. We also introduce a lightweight similarity metric to enable large-scale evaluation. Our experiments across 24 modern code generation models found that uncontrolled model-generated code is significantly less diverse than human solutions (capturing ~31% of human diversity), and even the best diverse code generation strategy could achieve only ~70% of human diversity. Moreover, while it was possible to steer the models to generate solutions based on human strategies, they were unable to reproduce the diversity of the human solutions. These findings indicate the limitations of code generation models in capturing the algorithmic diversity humans exhibit, and that overexposure to model-generated code can narrow programmers' algorithmic vocabulary.

Figures & tables

Figure 1: Illustrative example from our algorithmic diversity benchmark’s test split demonstrating two algorithmically distinct solutions for the same problem. Solution A solves the problem using the simulation paradigm with an array data structure, whereas Solution B adopts a bitmasking paradigm with the binary lifting technique using a 2D matrix data structure. Both solutions have different time and space complexity. The bar graphs show the code similarity of the two solutions, and the labels show the LLM judgments. The ticks in the bar graph indicate the similarity thresholds, below which the solution pair is considered algorithmically diverse in the respective metric space.

			Human solutions		Ontology-based algorithmic diversity
Split	instances	C++ sol.	total	median	strong	medium	weak	identical
train	5,623	11,246	148,242	25	2,429	1,463	1,229	502
val	1,000	2,000	25,915	24	419	257	231	93
test	1,000	2,000	26,145	25	428	280	212	80
all	7,623	15,246	200,302	25	3,276	2,000	1,672	675

Table 1: Dataset statistics of the algorithmic diversity benchmark.

Figure 2: Algorithmic diversity (

\mathrm{Vendi}_{\texttt{CodeX-SFT}}

) of 50 uncontrolled LLM-generated solutions against human-written solutions for the test set problems (detailed results in Table 4 Appendix D ).

Panel of Figure 3 — Figure 3: Algorithmic diversity ( $\mathrm{Vendi}_{\texttt{CodeX-SFT}}$ ) of three diverse text generation strategies for three LLMs. (Left) temperature variation, verbalized sampling, and iterative constrained generation on the test set problems and the corresponding scores for human solutions for matched problems. (Right) increase in algorithmic diversity of human solutions and the best-performing LLM, Gemma-4-31b , for 412 problems where at least 25 correct solutions are available for both human and LLM solutions. T=0.8 is denoted as i.i.d. , independent and identically distributed samples (same as the results in Figure 2 ). We report the human solutions corresponding to each model’s success rate to enable a fair comparison across model generations. For the detailed results, refer to Table 5 in Appendix D .

Panel of Figure 4 — Figure 4: (left) LLM-generated solution’s preference for using one of the two reference approaches vs. the popularity of the same approach in human-written solutions. Alignments for model and human preferences were obtained using the cosine similarities obtained from the CodeX-SFT embedding model. (right) Algorithmic diversity of ensemble of LLM-generated programs $\rho$ at the matched $k^{*}$ to the human reference solutions along with the most diverse LLMs for i.i.d. setting and the diverse code generation settings. $\rho_{all}$ denotes all 24 LLMs used in this work, $\rho_{3}$ denotes the three open-source models used for diverse code generation experiments (Figure 3 left).

Strong — targets differ in paradigm (274 problems)
Model	$n$	i.i.d.	verbalized	iterative	steer (A $+$ B)	human
Gemma-4-31b	245	28.2 [22,33]	50.2 [44,57]	60.8 [54,67]	46.6 [41,52]	70.0 [66,75]
Gemma-4-E4B	179	34.1 [27,41]	31.3 [25,38]	44.1 [37,51]	48.3 [42,55]	70.8 [65,76]
Phi-4	121	24.8 [17,33]	29.8 [22,38]	24.8 [17,32]	26.9 [20,34]	67.7 [61,74]
Medium — targets differ in technique (104 problems)
Gemma-4-31b	87	21.8 [14,31]	32.2 [23,43]	37.9 [28,48]	40.9 [32,50]	59.0 [50,68]

Table 2: Algorithmic coverage ( AlgoCov ) of LLM-generated solutions and corresponding human-written solutions over the two human solutions in the final dataset (referred as A and B). We sampled 10 solutions for each strategy; limiting to 274 strong , 104 medium and 106 weak instances where the ontology-labels are mutually exclusive for the respective classification dimension.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Algorithmic similarity metric agreements on 100 uniformly random solution pairs. (Left) Rank-based Spearman’s $\rho$ and (right) ROC-AUC against the majority of metrics; for LLM-judges, the majority of the other 5 LLMs is used, for CodeXEmbed and PDG all six LLM-judges are used.

Figure 6: Leave-one-out agreement between six proprietary LLMs for the three-level ontology labeling task.

Figure 7: Evaluation of the algorithmic similarity measurement metrics on 700 instances, evaluated against the aggregated majority ground truth from six proprietary models. The confidence interval is obtained via percentile bootstrap over the solution pairs (repeated 3,000 times).

embedding metrics (our implementation)
	Type-1	Type-2	Type-3	Type-4	T3 $-$ T4
CodeX-SFT	0.93	0.85	0.87	0.65	+0.23
CodeXEmbed	0.99	0.97	0.97	0.89	+0.08
jina-SFT	0.95	0.95	0.92	0.83	+0.09
jina-embeddings-v2-base-code	0.93	0.94	0.92	0.78	+0.14
embedding metrics, as reported in Zhang and Lu (2026)

Appendix

Table 3: Measuring algorithmic overlap of our trained evaluation metrics on the hand-curated solution pairs across four types of solution pairs in Python (

n=4,3,6

and

17

pairs for each type respectively).

Figure 8: Distribution of ontology labels in the two C++ solutions per programming problem in the algorithmic diversity dataset.

Model	Pass rate	Pass@N	Pass ≥2 @N	med	max	trunc (%)	PDG	CodeX-SFT	$k^{*}/k_{\mathrm{LLM}}$	$\mathrm{VS}^{\mathrm{LLM}}$	$\mathrm{VS}^{\mathrm{hum}}$
	Success Statistics			#tokens			self-div		$\mathrm{Vendi}_{\texttt{CodeX-SFT}}$ ( $k^{*}$ matched / all solutions)
Gemini-3.8-flash	80.9	954	945	–	–	0.7	0.046	0.175	24.7 / 42.8	4.18 / 4.84	13.29 / 14.25
Gemma-4-31b	76.8	927	913	1,073	8,192	5.1	0.049	0.173	24.7 / 42.0	4.14 / 4.75	13.15 / 14.23
Gemma-4-26B	74.0	929	917	2,447	16,384	7.2	0.058	0.171	24.3 / 40.3	3.50 / 3.96	13.01 / 14.26
Gemma-4-12b	64.1	881	858	1,576	16,384	10.0	0.054	0.178	23.3 / 37.3	3.71 / 4.13	12.24 / 14.16
Gemini-flash-lite	53.6	817	782	–	–	0.0	0.050	0.187	21.9 / 34.2	4.32 / 4.75	11.29 / 13.80

Appendix

Table 4: Algorithmic diversity results for unconditioned zero-shot generations on the 1,000 test set programming problems. We present three success statistics: pass rate - percentage of generated solutions that pass all test cases, Pass@N - number of instances with at least one correct generation, and Pass ≥2 @N - number of instances with at least two correct solutions to compute within problem diversity. We also provide the generation statistics, including the median ( med ) and maximum ( max ) number of output tokens, as well as the percentage of solutions that were truncated ( trunc ). For per-problem population diversity, we present the self-diversity ( self-div ) of generated solutions as the average distance of all correct solutions using two distance measurements: program dependence graphs ( PDG ) and the trained embedding model CodeX-SFT (Section B ). We also provide the

\mathrm{Vendi}_{\texttt{CodeX-SFT}}

(Eq. 1 ) scores on two settings, matched and all solutions . Matched

\mathrm{Vendi}

score subsamples

k^{*}=\text{min}(k_{\text{LLM}},k_{\text{hum}})

number of solutions from the LLM-generated solutions and the human references over 200 draws without replacement, whereas the all solutions

\mathrm{Vendi}

score is computed at the entire pool size of each solution population. We also provide the human reference diversity scores for all 1,000 test instances.

		Success Statistics			#tokens			self-div		$\mathrm{Vendi}_{\texttt{CodeX-SFT}}$ ( $k^{*}$ matched / all solutions)
Model	Sampling	Pass rate	Pass ≥2 @N	k	med	max	trunc (%)	PDG	CodeX-SFT	$k^{*}/k_{\mathrm{LLM}}$	$\mathrm{VS}^{\mathrm{LLM}}$	$\mathrm{VS}^{\mathrm{hum}}$
Phi-4	i.i.d.	31.4	618	25.3	356	2,048	0.09	0.066	0.169	18.5 / 25.3	3.22 / 3.33	8.79 / 13.18
	temp 0.2	33.5	538	31.1	347	2,048	0.08	0.032	0.087	20.6 / 31.1	1.85 / 1.89	9.56 / 12.95
	temp 1.2	27.6	587	23.4	380	2,048	1.76	0.079	0.197	17.7 / 23.4	3.73 / 3.85	8.23 / 12.96
	iterative	24.0	604	19.8	370	2,048	0.02	0.072	0.219	16.7 / 19.8	4.16 / 4.31	8.01 / 13.06
	verbalized	22.7	561	20.2	1,807	13,000	0.01	0.067	0.191	16.8 / 20.2	3.60 / 3.71	8.07 / 13.00

Appendix

Table 5: Algorithmic diversity of diversity conditioned generation approaches on the 1,000 test set programming problems. We present the results for three open-weight models over three strategies: temperature variation with 50 generations ( i.i.d. denotes the standard T=0.8 setting), verbalized sampling (10 independent calls per problem asking to generate five solutions per call), and iterative generation (10 independent chains per problem with five iterative constrained generations per generation chain). All conditions are evaluated on the same 50 generated solutions, and the statistics and diversity measurements are the same as described in Table 4 .

Figure 9: Illustrative example of strong algorithmic diversity from the train split. The two solutions have different programming paradigms. The bar graphs on the bottom left provide the code similarity of different similarity metrics, with the ticks indicating a threshold calibrated on the 10th percentile of randomly selected 150 train instances. Similarity below the threshold is considered algorithmically diverse as per the metric. We also provide the pairwise LLM judgments from the second filtration stage on the bottom right.

Figure 10: Illustrative example of medium algorithmic diversity from the train split. The two solutions have the same programming paradigm, but different techniques. The bar graphs at the bottom left show code similarity across different similarity metrics, with the ticks indicating a threshold calibrated to the 10th percentile of randomly selected 150 training instances. Similarity below the threshold is considered algorithmically diverse as per the metric. We also provide the pairwise LLM judgments from the second filtration stage on the bottom right.

Figure 11: Illustrative example of weak algorithmic diversity from the train split. The two solutions use the same programming paradigm and techniques but different data structures. The bar graphs at the bottom left show code similarity across different similarity metrics, with the ticks indicating a threshold calibrated to the 10th percentile of randomly selected 150 training instances. Similarity below the threshold is considered algorithmically diverse as per the metric. We also provide the pairwise LLM judgments from the second filtration stage on the bottom right.

Figure 12: Illustrative example of identical algorithmic diversity from the train split. The two solutions share the same programming paradigm, techniques, and data structures, and pass through all the filtration steps. The bar graphs at the bottom left show code similarity across different similarity metrics, with the ticks indicating a threshold calibrated to the 10th percentile of randomly selected 150 training instances. Similarity below the threshold is considered algorithmically diverse as per the metric. We also provide the pairwise LLM judgments from the second filtration stage on the bottom right.

Figure 13: The LLM-judge prompt used to obtain the 4-way ordinal classification of algorithmic similarity between two programs for the same problem.

Figure 14: The prompt used to obtain the ontology labels for the two representative solutions in the final dataset.

Figure 15: The prompt used to generate i.i.d. zero-shot solutions for different temperature values.

Figure 16: The prompt used to generate the solutions using the verbalized sampling approach Zhang et al. (2025) . We limit to 5 outputs per generation and re-run it 10 times to obtain 50 solutions for direct comparison with the i.i.d. setting.

Figure 17: The prompt used to iteratively generate diverse solutions. We limit the iterations to 5 generations and re-run it 10 times to obtain 50 solutions for direct comparison with the i.i.d. setting.

Figure 18: The prompts used to generate the steer (A+B) solutions that are instructed to adhere to a particular algorithmic strategy.

Code Language Models

Code language models are AI models that generate, explain, edit, and reason about source code across programming tasks.

See all 115 papers

AlgoBench: Benchmarking Algorithmic Adaptation in Code Generation
Jun 30, 2026Xinyuan Song, Zekun Cai, Liang ZhaoSynthetic Benchmark Generation Code Generation Evaluation
Emory University, Atlanta, GA, USA · The University of Tokyo, Tokyo, Japan · LocationMind, Tokyo, Japan
Are you Synthesizing or Recalling? Evaluating LLMs on Algorithmic Code Retrieval
Oct 1, 2026Nickil Maveli, Antonio Vergari, Shay B. CohenCode Generation Evaluation Code Generation Benchmarks
School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh, EH8 9AB
The Invisible Lottery: How Subtle Cues Steer Algorithm Choice in LLM Code Generation
Jun 2, 2026Akanksha Narula, Mofasshara Binte Rafique, Laurent BindschaedlerPrompt Sensitivity LLM Prompting
Max Planck Institute for Software Systems (MPI-SWS), Saarbrücken, Germany · Independent Researcher, Switzerland.

Jun 30, 2026J/K move · Enter open · S save

AlgoBench: Benchmarking Algorithmic Adaptation in Code Generation

Xinyuan Song, Zekun Cai, Liang Zhao

Emory University, Atlanta, GA, USA · The University of Tokyo, Tokyo, Japan · LocationMind, Tokyo, Japan

Synthetic Benchmark Generation Code Generation Evaluation Algorithmic Reasoning Code Generation Benchmarks Algorithmic Generalization

Open paper AskPDF arXiv

High pass rates on established programming benchmarks such as HumanEval and LiveCodeBench do not always show whether a model can reason about algorithms. Many fixed benchmarks eventually become part of the public training ecosystem through released problem statements, editorials, and generated solutions, allowing later models to improve partly by exposure rather than by stronger algorithmic ability. We introduce ALGOBENCH, a framework that automatically builds novel algorithmic problems from known competitive-programming problems through structured constraint-shifting transformations. Each accepted ALGOBENCH variant is traceable to a source problem, but must make the original reference algorithm fail. Beyond pass@

k

, we introduce complexity-aware metrics -- including OPTT, OPTS, TRAPRATE, GAPT, and CONSENS -- to test whether a solution is not only functionally correct but also asymptotically suitable for the generated problem. Experiments across multiple LLMs and prompting strategies show that performance drops sharply on ALGOBENCH variants, retrieval can increase reuse of the old algorithm, and many correct-looking solutions fail to meet the required complexity. Error analysis shows that failures are mainly algorithmic rather than implementation-level, suggesting that ALGOBENCH evaluates adaptation beyond functional correctness.

Abstract

Code Language Models

Explore similar work

AlgoBench: Benchmarking Algorithmic Adaptation in Code Generation

Are you Synthesizing or Recalling? Evaluating LLMs on Algorithmic Code Retrieval

The Invisible Lottery: How Subtle Cues Steer Algorithm Choice in LLM Code Generation