Large language models (LLMs) have demonstrated strong performance in code generation, where success depends on both recalling relevant algorithmic knowledge and reasoning about how to apply it. However, existing LLM pipelines are opaque, with no explicit separation between these two components. We argue that for well-known algorithms whose canonical implementations are widely accessible in pretraining corpora, code generation is better measured as parametric code retrieval: reproducing a named algorithm from internalised knowledge rather than synthesizing a novel one. We introduce AlgoREval, a benchmark of 599 problems spanning classical 77 algorithms across 14 domains, 7 programming languages, and 4 graph-input representations to evaluate this capability in isolation, and assess 15 models (7B--34B parameters) in a zero-shot setting. We find substantial variation in retrieval accuracy across languages and input representations, even for widely documented algorithms and show that prompt augmentation with retrieved code snippets or structured algorithmic hints improve accuracy on complex algorithms, while SFT achieves broader language gains and GRPO achieves larger per-language gains on specific languages. Together, our results establish parametric code retrieval as a distinct, measurable capability and caution against deploying AI-generated algorithmic code without systematic validation.\footnote{Code and dataset are available at https://github.com/Nickil21/AlgoREval
Figures & tables
Figure 1: Ten LLMs implement Prim’s MST algorithm, split into a fast tier (4 models) and a slow tier (6 models) by wall-clock scaling. We time two independently-written, correctness-verified reference implementations. Since both tiers land close to a real, legitimate pattern rather than one looking like an unexplained outlier, the evidence here is symmetric and consistent with both tiers reflecting retrieval of different existing implementations, not one retrieving and the other synthesizing from scratch.
Figure 2: Parametric code retrieval evaluation pipeline. LLMs generate solutions zero-shot, drawing solely on algorithmic knowledge encoded in their pretraining weights. The four panels show four models (three open-weight, one closed API) on the same Bridge Finding problem, each in a different language: Yi-Coder-9B (Python) uses self.time in a plain function with no class, causing a runtime error ; CodeLlama-34B (Rust) passes graph by value where a reference is required, producing a compile error ; gpt-4o-mini (Java, closed API) finds the correct bridges but omits the sort step, yielding wrong output order ; QwQ-32B (JavaScript) normalises, sorts, and passes.
Domain
Problem Types
Entries
Graph algorithms
24
228
Image processing
10
70
Data structures
5
35
Computational geometry
4
28
Sorting algorithms
4
28
Dynamic programming
4
28
Table 1: Dataset breakdown by domain. Each problem type is instantiated across 7 languages. The graph-algorithm problems additionally include up to 4 input-representation variants.
Category
Model
Weights
Params
Pass@1 (%)
Code
Codestral-22B
Open
22
52.6
DeepSeek-Coder-33B
Open
33
46.4
Yi-Coder-9B
Open
9
46.2
CodeGemma-7B
Open
7
25.9
StarCoder2-15B
Open
15
13.7
CodeLlama-34B
Open
34
12.9
Table 2: Zero-shot Pass@1 (%) grouped by category and ranked within each group by Pass@1 . Closed API models (GPT-4.1-mini, GPT-4o-mini) have undisclosed sizes.
Figure 3: Pass@1 heatmap by model and programming language. Rows are grouped by model category (Code, General, Reasoning) and ordered by mean pass rate within each group.
Figure 4: Pass@1 by programming language, broken down by input representation and aggregated across all model predictions. Each group compares edge-list, adjacency-list, adjacency-matrix, and incidence-matrix inputs within a language.
Model
Base
+Hint
Δ (H)
+Snip
Δ (S)
Better
StarCoder2-15B
13.7
12.0
-1.7
8.7
-5.0
H
DeepSeek-R1-Distill-Llama-8B
12.9
11.2
-1.7
12.4
-0.5
S
CodeGemma-7B
25.9
26.0
+0.2
24.5
-1.3
H
CodeLlama-34B
12.9
11.7
-1.2
11.7
-1.2
tie
Llama-3.1-8B
31.4
27.7
-3.7
29.7
-1.7
S
Qwen2.5-7B
37.4
37.6
+0.2
37.1
-0.3
H
Table 3: Pass rates (%) under three conditions across 13 models on all 599 benchmark examples. Base : zero-shot baseline. +Hint : GPT-4o-mini-generated pseudocode. +Snip : language-matched GitHub snippet. Better indicates which augmentation condition outperforms the other per model.
Pass@1 (%) on held-out test split
Method
Overall
[1pt] C
[1pt] C ++
[1pt] Go
[1pt] Java
[1pt] JS
[1pt] Python
[1pt] Rust
Base
90.1
81.0
85.7
100.0
95.2
85.7
96.2
85.7
SFT
94.1
85.7
95.2
100.0
100.0
95.2
96.2
85.7
GRPO
92.1
85.7
76.2
100.0
95.2
100.0
96.2
90.5
Δ SFT
+ 3.9
+ 4.8
+ 9.5
0.0
+ 4.8
+ 9.5
0.0
0.0
Δ GRPO
+ 2.0
+ 4.8
− 9.5
0.0
0.0
+ 14.3
0.0
+ 4.8
Table 4: Pass@1 (%) on the held-out test split (152 problems, 22 algorithms unseen during training) for the base model gpt-oss-20b, SFT, and GRPO. Δ rows show absolute change vs. base in percentage points. SFT leads overall ( +3.9 pp) with consistent gains across C, C ++ , Java, and JavaScript; GRPO achieves the largest JavaScript gain ( +14.3 pp) and improves C and Rust, but regresses on C ++ ( −9.5 pp).
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Domain
Algorithm
Backtracking
N-Queens Problem
Permutations Generator
Sudoku Solver
Compression
Huffman Coding
Run-Length Encoding
Computational Geometry
Closest Pair of Points
Appendix
Table 5: All 77 algorithms in AlgoREval , grouped by domain (14 domains total).
Model
Model ID
Params
Organization
License
CodeGemma-7B
google/codegemma-7b-it
7B
Google
Gemma
CodeLlama-34B
codellama/CodeLlama-34b-Instruct-hf
34B
Meta
Llama 2
Codestral-22B
mistralai/Codestral-22B-v0.1
22B
Mistral AI
MNPL
DeepSeek-Coder-33B
deepseek-ai/deepseek-coder-33b-instruct
33B
DeepSeek
DeepSeek
DeepSeek-R1-Llama-8B
deepseek-ai/DeepSeek-R1-Distill-Llama-8B
8B
DeepSeek
MIT
DeepSeek-R1-Qwen-14B
deepseek-ai/DeepSeek-R1-Distill-Qwen-14B
14B
DeepSeek
MIT
Appendix
Table 6: All 15 models evaluated in AlgoREval . Open-weight models were run locally via vLLM. Closed models were accessed through their respective APIs. Parameter counts for closed API models are not publicly disclosed.
Parameter
Open-weight (vLLM)
Closed API
Temperature
0 (greedy)
0.7
Max output tokens
32,768
8,192
Top- p
1.0
—
Decoding
Greedy
Sampling
Thinking budget
32,768 (thinking models)
N/A
Execution timeout
30 s
30 s
Appendix
Table 7: Generation hyperparameters for the zero-shot evaluation. Open-weight models include all 13 locally-served models. Closed API covers GPT-4o-mini and GPT-4.1-mini. The thinking-model token floor applies to QwQ-32B, DeepSeek-R1-Distill-Llama-8B, and DeepSeek-R1-Distill-Qwen-14B.
Parameter
+Hint
+Snippet
Models evaluated
13 vLLM (open-weight)
13 vLLM (open-weight)
Temperature
0 (greedy)
0 (greedy)
Max output tokens
32,768
32,768
Top- p
1.0
1.0
Execution timeout
30 s
30 s
Context source
GPT-4o-mini (temp. 0)
GitHub Search API
Appendix
Table 8: Prompt-augmentation parameters. Generation settings are identical to the zero-shot baseline. Only the prompt contents differ between conditions.
Parameter
GRPO
SFT
Base model
gpt-oss-20B
gpt-oss-20B
LoRA rank r
16
16
LoRA α
32
32
LoRA dropout
0.05
0.05
Trainable params
≈ 0.5%
≈ 0.5%
Training algorithms
56 (447 examples)
56 (367 positive examples)
Appendix
Table 9: Fine-tuning hyperparameters for GRPO and SFT. Both conditions use the same LoRA configuration and train/test split. SFT trains only on examples where the base model already achieves r=1.0 (rejection sampling), isolating the contribution of policy-gradient credit assignment from the effect of training on correct solutions alone.
Figure 5: Prompt structure used for all evaluations. The system message is fixed across all problems and languages. The user message is generated per problem; it names the algorithm and target language, specifies the required interface, and embeds an executable test case whose output is compared against the ground-truth reference output.
Figure 6: Complete user message for Hopcroft–Karp (Python) under the +Hint condition. The green-shaded block is the only addition relative to the zero-shot baseline; the same language-agnostic hint is used for all seven language variants of the algorithm. Generated offline by gpt-4o-mini-2024-07-18 at temperature =0 .
Figure 7: Complete user message for Hopcroft–Karp (Python) under the +Snippet condition. The orange-shaded block is the only addition relative to the zero-shot baseline. The snippet (D. Eppstein, 2002) was retrieved from GitHub via the GitHub Search API; RosettaCode and GeeksforGeeks results were excluded to avoid overlap with the benchmark’s source material. Files are truncated to at most 80 lines centred on the most keyword-relevant region, hence the snippet ends mid-function. Each language variant receives its own language-matched snippet.
Figure 8: AlgoSim novel rate by model and programming language.
Figure 9: Outcome breakdown per model as a fraction of all 599 problems, sorted by pass rate. Stronger models fail almost exclusively on wrong output, while weaker models are dominated by compile errors, reflecting a gradient from syntactic to semantic retrieval failure.
Figure 10: Pass rate vs. consensus recall. GT pass rate (x-axis) is the fraction of problems where a model’s output matches the ground-truth expected value. Consensus recall (y-axis) is the fraction of a model’s correct answers that also match the plurality output across all models.
Figure 11: Algorithm difficulty heatmap for Python.
Figure 12: Algorithm difficulty heatmap for C++.
Figure 13: Algorithm difficulty heatmap for Java.
Figure 14: Algorithm difficulty heatmap for C.
Figure 15: Algorithm difficulty heatmap for JavaScript.
Figure 16: Algorithm difficulty heatmap for Rust.
Figure 17: Algorithm difficulty heatmap for Go.
Figure 18: Dense regime ( p=0.3 fixed, E∼n2 ): wall time vs. graph size n , one panel per algorithm and one line per model. Solid markers indicate 100% correctness at that n over 8 samples, and hollow markers (colored red-to-green) indicate a partial correct rate, and an ‘x’ indicates every sample crashed or timed out. The dashed gray curve in each panel is the textbook Big-O reference (fit at the algorithm’s expected exponent).
Figure 19: Sparse regime ( p(n)=3ln(n)/n , E∼nlogn ), for the 7 algorithms whose textbook complexity is E -dependent. This regime separates correct O(ElogV) implementations (fitted k≈1 ) from hidden O(V2) ones (fitted k≈2 ), which the dense regime alone cannot cleanly distinguish.
Cond.
Name given
Spec given
(1) Name+Format
correct
none (signature only)
(2) Name+Spec
correct
correct
(3) Spec Only
none
correct
(4) Wrong Name
wrong (real)
correct
(5) Fake Name
fabricated
correct
(6) Wrong Spec (moderate)
correct
one decision rule swapped
Appendix
Table 10: The 6 conditions applied to the algorithms. All conditions are scored by execution against the ground truth.
Figure 20: (a) Mean execution-correctness across six novel algorithms, by condition, for gpt-oss-20B and Qwen2.5-7B-Instruct (20 samples/cell). gpt-oss-20B stays flat (85–92%) regardless of which name accompanies a correct spec, then collapses once the spec is wrong; Qwen2.5-7B-Instruct shows the same shape at a lower level. (b) The same conditions per algorithm (6 algorithms × 2 models). Blue-outlined cells: K-Core Decomposition and Prüfer Sequence Encoding score lower with the correct name than without it, and gpt-oss-20B’s Transitive Reduction scores 75% under moderate corruption versus 0–10% elsewhere in that column.
Large language models (LLMs) now generate substantial production code, often for tasks with multiple valid algorithmic solutions. Incidental prompt cues, meaning contextual words or metadata outside the task specification, can steer which algorithm the model selects, even when all outputs pass the same tests. Prompt sensitivity is well studied as a tool to improve output quality. Here, output policy means algorithm choice under fixed correctness. We define algorithm steering as cue-induced shifts in algorithm-family distributions and run 46,535 controlled experiments across 11 tasks, 19 cue types (18 channels plus a memoization semantic-vs-surface ablation that preserves meaning while changing typography and punctuation), and 15 model configurations. We find large, systematic shifts in algorithm-family distributions (up to 100 pp), largely consistent with cue semantics, including in applied tasks such as rate limiting. Direct algorithm naming is the most reliable mitigation we tested. Accidental context therefore creates an "invisible lottery" over performance, security, and maintainability.
High pass rates on established programming benchmarks such as HumanEval and LiveCodeBench do not always show whether a model can reason about algorithms. Many fixed benchmarks eventually become part of the public training ecosystem through released problem statements, editorials, and generated solutions, allowing later models to improve partly by exposure rather than by stronger algorithmic ability. We introduce ALGOBENCH, a framework that automatically builds novel algorithmic problems from known competitive-programming problems through structured constraint-shifting transformations. Each accepted ALGOBENCH variant is traceable to a source problem, but must make the original reference algorithm fail. Beyond pass@k, we introduce complexity-aware metrics -- including OPTT, OPTS, TRAPRATE, GAPT, and CONSENS -- to test whether a solution is not only functionally correct but also asymptotically suitable for the generated problem. Experiments across multiple LLMs and prompting strategies show that performance drops sharply on ALGOBENCH variants, retrieval can increase reuse of the old algorithm, and many correct-looking solutions fail to meet the required complexity. Error analysis shows that failures are mainly algorithmic rather than implementation-level, suggesting that ALGOBENCH evaluates adaptation beyond functional correctness.
Xinyuan Song, Zekun Cai, Liang Zhao
Emory University, Atlanta, GA, USA · The University of Tokyo, Tokyo, Japan · LocationMind, Tokyo, Japan
This study empirically evaluates whether cost-efficient Large Language Models (LLMs) can be trusted to generate enterprise code to a written specification. Three models (Gemini Flash 3, GPT-5.4 mini and Claude Haiku 4.5) were asked to solve 992 algorithmic problems as Java Spring Boot service methods conforming to a mandated signature and data-transfer-object specification, crossing four model and agentic coding tool combinations with two prompt variants to yield eight configurations, with iteration forbidden and hardcoded answers explicitly prohibited. Eight problem statements were withheld to probe how models respond to missing input. The 7,593 resulting methods were classified by an eight-class outcome taxonomy describing what each does about producing an answer, then deployed and executed, giving 7,936 measured requests joined to that classification. Structural conformance approached ceiling, yet 38.4% of methods do not compute the value they returned and only 12.9% of returned answers were correct. Conditioning on outcome class shows that response reliability and correctness are inversely related, whereas genuinely computing methods answered least often and were correct 19.3%. Limitations include single generation runs per configuration, partial harness coverage, single-pass timing, syntactic classification, and probable corpus contamination.
Chandimal Adikari, Nandika Herath
Faculty of Engineering and Information Sciences University of Wollongong Wollongong, Australia · Faculty of Engineering, Computing and Science Western Sydney University Parramatta, Australia