Planning has been identified as an important factor for improved code generation with Large Language Models (LLMs). However, the interaction between planning capability and coding capability remains largely underexplored, especially when the two capabilities are from different LLMs. In this paper, we conduct the first comprehensive empirical study of planning-coding interaction in LLM-based code generation, where planning and coding are assigned to different LLMs (i.e., planning-coding separation). Specifically, we explore different planner-coder combinations to investigate the effectiveness for coders to utilize planning information from various planners and the impact of further training of coders on planner-coder combinations. Our results demonstrate that almost all planner-coder combinations are as competitive as using the same LLM for planning and coding, and the combination of a stronger planner and a stronger coder typically achieves improved effectiveness. Furthermore, using a large model as the planner paired with a much smaller model as the coder can sometimes be as competitive as using the large model as both the planner and the coder, and further training of the coder can typically boost various planner-coder combinations. These findings provide empirical evidence that planning and coding can be separated to a considerable extent in LLM-based code generation, which may offer a more flexible way to construct code-generation systems.
Figures & tables
Figure 1. Planning prompt. Planning prompt.
Coder
Direct
Planner
3B–4B
Qwen3-4B Gemma-3-4B-it Ministral-3-3B-Instruct
Qwen3-4B
24.57
24.95 23.43 (-6.1%) 21.52 (-13.7%)
Gemma-3-4B-it
18.86
19.62 (+14.4%) 17.14 17.71 (+3.3%)
Ministral-3-3B-Instruct
23.43
26.48 (+36.3%) 24.19 (+24.5%) 19.43
8B
Qwen3-8B Ministral-3-8B-Instruct
Qwen3-8B
24.57
26.29 23.62 (-10.1%)
Table 1. Planner–coder pairing results on LiveCodeBench (Pass@1). Parentheses show the relative change from the same-model pair for each coder. Bold marks the best plan-assisted result for each coder; gray marks the highest-scoring pair in each model group.
Coder
Direct
Planner
3B–4B
Qwen3-4B Gemma-3-4B-it Ministral-3-3B-Instruct
Qwen3-4B
24.88
22.58 18.59 (-17.7%) 17.82 (-21.1%)
Gemma-3-4B-it
21.20
22.43 (+23.7%) 18.13 16.59 (-8.5%)
Ministral-3-3B-Instruct
24.88
23.96 (+30.0%) 19.97 (+8.3%) 18.43
8B
Qwen3-8B Ministral-3-8B-Instruct
Qwen3-8B
24.88
27.04 23.20 (-14.2%)
Table 2. Planner–coder pairing results on Code2Bench (Pass@1). Parentheses show the relative change from the same-model pair for each coder. Bold marks the best plan-assisted result for each coder; gray marks the highest-scoring pair in each model group.
Figure 2. Planner performance across model scales with three small coders. Q3, Gem, and Min abbreviate Qwen3, Gemma-3-it, and Ministral-3-Instruct-2512; DS Flash/Pro abbreviate DeepSeek-V4.1-Flash/V4-Pro, and GPT abbreviates GPT-5.6-Sol. Six panels compare planner models with fixed Qwen3-4B, Gemma-3-4B-it, and Ministral-3-3B-Instruct-2512 coders, arranged in three rows. The left column shows LiveCodeBench and the right column shows Code2Bench. The horizontal axis lists planners and the vertical axis shows Pass@1 percentages. Lines connect Qwen3, Gemma-3, and Ministral-3 planners of different sizes. Separate points show Qwen3.5-27B, DeepSeek-V4.1-Flash, DeepSeek-V4-Pro, and GPT-5.6-Sol. Dashed horizontal lines mark direct generation by each coder. Increasing planner size generally improves small-coder performance, with exceptions among Qwen3 pairings.
Figure 3. Planner performance across model scales with three larger coders. Q3, Gem, and Min abbreviate Qwen3, Gemma-3-it, and Ministral-3-Instruct-2512; DS Flash/Pro abbreviate DeepSeek-V4.1-Flash/V4-Pro, and GPT abbreviates GPT-5.6-Sol. Six panels compare planner models with fixed Qwen3-32B, Gemma-3-27B-it, and Ministral-3-14B-Instruct-2512 coders, arranged in three rows. The left column shows LiveCodeBench and the right column shows Code2Bench. The horizontal axis lists planners and the vertical axis shows Pass@1 percentages. Lines connect Qwen3, Gemma-3, and Ministral-3 planners of different sizes. Separate points show Qwen3.5-27B, DeepSeek-V4.1-Flash, DeepSeek-V4-Pro, and GPT-5.6-Sol. Dashed horizontal lines mark direct generation by each coder. Within-family scaling trends vary by coder; Qwen3.5 and the advanced planners exceed the direct baselines in all six panels.
Coder
Direct
Planner
Qwen3.5-27B
DeepSeek-V4.1-Flash
DeepSeek-V4-Pro
GPT-5.6-Sol
Qwen3-4B
24.57
37.71 (+53.5%)
40.95 (+66.7%)
42.67 (+73.6%)
50.10 (+103.9%)
Qwen3-8B
24.57
39.05 (+58.9%)
42.29 (+72.1%)
43.24 (+76.0%)
51.24 (+108.5%)
Qwen3-14B
28.00
40.57 (+44.9%)
44.38 (+58.5%)
46.86 (+67.3%)
55.24 (+97.3%)
Qwen3-32B
27.43
40.19 (+46.5%)
44.57 (+62.5%)
42.10 (+53.5%)
53.52 (+95.1%)
Gemma-3-4B-it
18.86
25.14 (+33.3%)
27.24 (+44.4%)
28.00 (+48.5%)
26.10 (+38.4%)
Table 3. Coder performance across model scales with four planners on LiveCodeBench (Pass@1). Parentheses show the relative change from Direct for each coder. Bold marks the best planner for each coder; gray marks the best coder for each planner within each group.
Coder
Direct
Planner
Qwen3.5-27B
DeepSeek-V4.1-Flash
DeepSeek-V4-Pro
GPT-5.6-Sol
Qwen3-4B
24.88
30.57 (+22.8%)
32.10 (+29.0%)
32.87 (+32.1%)
34.56 (+38.9%)
Qwen3-8B
24.88
31.80 (+27.8%)
32.41 (+30.2%)
34.25 (+37.7%)
34.87 (+40.1%)
Qwen3-14B
32.26
33.64 (+4.3%)
33.95 (+5.2%)
35.02 (+8.6%)
37.94 (+17.6%)
Qwen3-32B
29.49
33.18 (+12.5%)
34.10 (+15.6%)
35.48 (+20.3%)
37.94 (+28.6%)
Gemma-3-4B-it
21.20
27.50 (+29.7%)
28.42 (+34.1%)
30.41 (+43.5%)
29.80 (+40.6%)
Table 4. Coder performance across model scales with four planners on Code2Bench (Pass@1). Parentheses show the relative change from Direct for each coder. Bold marks the best planner for each coder; gray marks the best coder for each planner within each group.
Coder
Setting
Direct
Planner
Qwen3.5-27B
DeepSeek-V4.1-Flash
DeepSeek-V4-Pro
GPT-5.6-Sol
Qwen2.5-Coder-1.5B
Base
10.86
23.05
24.95
25.90
28.00
SFT
13.71
24.19
24.95
27.43
28.95
Relative Change (%)
+26.3
+5.0
+0.0
+5.9
+3.4
Qwen2.5-Coder-3B
Base
14.29
26.67
29.14
32.95
34.48
SFT
20.00
29.14
32.19
34.48
37.33
Table 5. Base and SFT coder performance on LiveCodeBench (Pass@1). Relative Change is computed from Base to SFT. Bold marks the higher Base/SFT score for each condition.
Coder
Setting
Direct
Planner
Qwen3.5-27B
DeepSeek-V4.1-Flash
DeepSeek-V4-Pro
GPT-5.6-Sol
Qwen2.5-Coder-1.5B
Base
15.21
25.19
25.65
25.96
23.35
SFT
21.20
27.65
29.19
31.18
30.26
Relative Change (%)
+39.4
+9.8
+13.8
+20.1
+29.6
Qwen2.5-Coder-3B
Base
20.28
28.11
27.96
29.95
30.57
SFT
25.81
29.49
31.49
31.64
34.10
Table 6. Base and SFT coder performance on Code2Bench (Pass@1). Relative Change (%) is computed from Base to SFT. Bold marks the higher Base/SFT score for each condition.
Figure 4. Plans and generated code from Qwen3-8B and Ministral-3-8B-Instruct for LiveCodeBench problem 3714, comparing direct generation with four planner-coder pairings. The problem, input constraints, two generated plans, and six code-generation outcomes: both direct generations fail, and only Qwen3-8B planning paired with Ministral-3-8B-Instruct coding passes.
Large Language Models (LLMs) often generate natural-language comments while writing code, and these comments become part of the context used to generate the code that follows. However, it remains unclear which properties of comments affect code-generation performance. We study this question through observational analyses and controlled interventions. On LiveCodeBench, neither comment frequency nor broad comment intent reliably predicts pass@1. We then prefill weaker recipient models with comment blocks written by stronger source models, allowing us to separate comment surface form from the solution content they convey. Comments from source solutions that pass the tests raise recipient pass@1 by 17.2% on average. In contrast, comments describing failed solutions provide no reliable gain, while comments written for a different problem reduce pass@1 by 20.8%. Finally, across a wide range of models and prompt variants, most recipient models show no significant recovery of the external-comment gain, and the best case recovers only 24%. These results show that comments help code generation not merely because they are comments, but because they can provide correct solution content that prompting cannot reliably elicit.
Dangfeng Pan, Zhensu Sun, Cenyuan Zhang +2
Monash University, Australia · Singapore Management University, Singapore
The use of LLMs for code generation has naturally extended to code testing and evaluation. As codebases grow in size and complexity, so does the need for automated test generation. Current approaches for LLM-based test generation rely on strategies that maximize immediate coverage gain, a greedy approach that plateaus on code where reaching deep branches requires setup steps that individually yield zero new coverage. Drawing on principles of Bayesian exploration, we treat the program's branch structure as an unknown environment, and an evolving coverage map as a proxy probabilistic posterior representing what the LLM has discovered so far. Our method, CovQValue, feeds the coverage map back to the LLM, generates diverse candidate plans in parallel, and selects the most informative plan by LLM-estimated Q-values, seeking actions that balance immediate branch discovery with future reachability. Our method outperforms greedy selection on TestGenEval Lite, achieving 51-77% higher branch coverage across three popular LLMs and winning on 77-84% of targets. In addition, we build a benchmark for iterative test generation, RepoExploreBench, where they achieve 40-74%. These results show the potential of curiosity-driven planning methods for LLM-based exploration, enabling more effective discovery of program behavior through sequential interaction
Alfonso Amayuelas, Firas Laakom, Piotr Piękos +5
University of California, Santa Barbara · King Abdullah University of Science and Technology
Code generation systems make each LLM call with a model, a prompt, and decoding settings. However, existing optimization methods usually tune only part of these choices or use one fixed configuration for all tasks: global optimizers search one configuration for all tasks, routers choose only a model, and prompt optimizers keep the model and decoding settings fixed. This leaves their joint, group-specific interactions unclear. We therefore examine how these choices interact and observe that prompts and decoding settings interact, tuning effects vary by model, and the best configuration varies by task difficulty. Guided by these observations, we introduce COMPAS (Code-generation Optimization over Models, Prompts, And Decoding Settings), a difficulty-aware method that learns group-specific quality-cost fronts through low-cost model selection and joint prompt-decoding search, then routes each test task to its matching front online without further search. Under a matched search budget on LiveCodeBench, COMPAS improves pass@1 from 45.9% for the best baseline to 52.8% while reducing cost from 36.57to4.92. This also transfers to repository-level code generation on SWE-bench, resolving 76.0% of tasks versus 70.0% for the best baseline. Code and the reproducibility artifact are available at https://github.com/gjz78910/COMPAS.
Jingzhi Gong, Jie M. Zhang, Gunel Jahangirova +3
King’s College London, University of London · National University of Singapore · University College London, University of London