Inference-Time Code Selection via Symbolic Equivalence Partitioning

David Cho; Yifan Wang; Fanping Sui; Ananth Grama

cs.LGApr 7, 2026

Inference-Time Code Selection via Symbolic Equivalence Partitioning

Authors: David Cho, Yifan Wang, Fanping Sui, Ananth Grama

Organizations: Department of Computer Science, Purdue University, West Lafayette, IN, USA · Texas Instruments, Dallas, TX, USA

Abstract

Sampling multiple candidate programs at inference time is an effective way to improve LLM code generation. However, its benefit depends on reliably selecting a correct solution from the generated pool. We observe that this selection problem has a distinctive semantic structure: correct solutions, despite differences in syntax, implementation, or algorithmic strategy, often converge to the same functional behavior over valid inputs. At the same time, consensus alone is not sufficient for correctness, because models can also produce correlated wrong solutions that implement the same mistaken behavior. We propose Symbolic Equivalence Partitioning (SEP), an inference-time selection framework that first uses problem-provided public examples as lightweight validity signals. SEP then uses symbolic execution to partition the remaining candidate programs into bounded functional equivalence classes and selects from the dominant equivalence class. Across HumanEval+ and LiveCodeBench, SEP consistently improves selection accuracy without auxiliary test generation, learned verifiers, or additional LLM inference. At $N=10$ , SEP improves average accuracy from 0.754 to 0.826 on HumanEval+ and from 0.565 to 0.647 on LiveCodeBench, showing that symbolic functional agreement is an effective signal for inference-time code selection.

Figures & tables

Figure 1: Overview of Symbolic Equivalence Partitioning (SEP).

Figure 2: Illustration of SEP separating f1 and f2 in the walkthrough example. The symbolic engine searches for a valid input satisfying the extracted constraints on which the two programs produce different outputs, and returns a concrete counterexample when such an input is found.

Panel of Figure 3 — Figure 3: Scaling with candidate pool size. Average accuracy across 7 models as the number of sampled candidates increases from $N=5$ to $N=20$ on HumanEval+ and LiveCodeBench. We compare Pass@1, five baselines, and our method.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Temperature	Top- $p$
Phi-4-mini-instruct	0.7	0.8
Phi-4-reasoning	0.8	0.95
Qwen3-4B	0.6	0.95
Qwen3-4B-Instruct-2507	0.7	0.8
Qwen3-14B	0.6	0.95
Llama-3.1-8B-Instruct	0.6	0.9

Appendix

Table 7: Sampling parameters used for candidate generation.

Model	Temp	$N=5$	$N=10$	$N=20$
Llama-3.1-8B-Instruct	0.6	0.853	0.852	0.847
Phi-4-mini-instruct	0.7	0.844	0.857	0.859
Phi-4-reasoning	0.8	0.876	0.877	0.879
gpt-oss-20b	0.6	0.824	0.833	0.830
Qwen3-14B	0.6	0.954	0.951	0.954
Qwen3-4B-Instruct-2507	0.7	0.963	0.966	0.971

Appendix

Table 8: Pairwise equivalence accuracy summary on HumanEval+ (higher is better).

Model	Temp	$N=5$	$N=10$	$N=20$
gpt-oss-20b	0.6	0.831	0.817	0.813
Llama-3.1-8B-Instruct	0.6	0.901	0.866	0.846
Phi-4-mini-instruct	0.7	0.886	0.886	0.884
Phi-4-reasoning	0.8	0.834	0.828	0.818
Qwen3-14B	0.6	0.874	0.874	0.876
Qwen3-4B	0.6	0.888	0.880	0.884

Appendix

Table 9: Pairwise equivalence accuracy summary on LiveCodeBench (higher is better).

Model	Eligible	Flagged	Mostly-wrong	Wrong rep.	Both
Llama-3.1-8B-Instruct	349	9.17% (32)	6.59% (23)	4.30% (15)	1.72% (6)
Phi-4-mini-instruct	387	5.43% (21)	4.91% (19)	1.81% (7)	1.29% (5)
Phi-4-reasoning	441	2.49% (11)	2.49% (11)	0.23% (1)	0.23% (1)
Qwen3-4B-Instruct-2507	416	3.12% (13)	3.12% (13)	0.48% (2)	0.48% (2)
Qwen3-4B	400	3.00% (12)	2.75% (11)	1.25% (5)	1.00% (4)
Qwen3-14B	431	1.86% (8)	1.86% (8)	0.00% (0)	0.00% (0)

Appendix

Table 10: HumanEval+ partition-level failure patterns by model. Results are aggregated over

N\in\{5,10,20\}

. Percentages are computed over consensus-eligible instances, with counts shown in parentheses. The two failure patterns are not mutually exclusive.

Model	Eligible	Flagged	Mostly-wrong	Wrong rep.	Both
Llama-3.1-8B-Instruct	277	3.25% (9)	1.44% (4)	2.17% (6)	0.36% (1)
Phi-4-mini-instruct	328	9.15% (30)	6.10% (20)	5.18% (17)	2.13% (7)
Phi-4-reasoning	1,164	6.10% (71)	3.35% (39)	4.47% (52)	1.72% (20)
Qwen3-4B-Instruct-2507	824	6.43% (53)	4.98% (41)	3.64% (30)	2.18% (18)
Qwen3-4B	1,078	4.73% (51)	2.78% (30)	3.71% (40)	1.76% (19)
Qwen3-14B	1,191	7.14% (85)	4.28% (51)	4.70% (56)	1.85% (22)

Appendix

Table 11: LiveCodeBench partition-level failure patterns by model. Results are aggregated over

N\in\{5,10,20\}

. Percentages are computed over consensus-eligible instances, with counts shown in parentheses. The two failure patterns are not mutually exclusive.

Model	Rep. Acc.		Pairwise Acc.		Symbolic Work (s)
	w/ CI	w/o CI	w/ CI	w/o CI	w/ CI	w/o CI
Qwen3-14B	0.854	0.870	0.874	0.709	47236	57491
Qwen3-4B	0.795	0.790	0.880	0.718	41572	49902
Qwen3-4B-Instruct-2507	0.621	0.621	0.913	0.812	27848	35090
gpt-oss-20b	0.881	0.886	0.817	0.614	51421	62275
Llama-3.1-8B-Instruct	0.256	0.251	0.866	0.803	6960	7600

Appendix

Table 12: Per-model constraint injection ablation on LiveCodeBench (

N=10

). CI improves pairwise equivalence accuracy and reduces symbolic work, while representative accuracy changes only slightly.

Model	Rep. Acc.	Mostly-wrong	Eligible
Qwen3-14B	0.878 / 0.854	3 / 5	144
Qwen3-4B-Inst-2507	0.829 / 0.817	4 / 7	140
Qwen3-4B	0.799 / 0.756	5 / 11	136
Llama-3.1-8B-Inst	0.689 / 0.622	8 / 11	119
Phi-4-mini-inst	0.793 / 0.695	7 / 16	131
Phi-4-reasoning	0.902 / 0.860	2 / 4	146

Appendix

Table 13: Public example filter ablation on HumanEval+ at

N=10

. Each paired entry is reported as with filtering / without filtering.

Model	Rep. Acc.	Mostly-wrong
Qwen3-14B	0.854 / 0.838	15 / 19
Qwen3-4B-Inst-2507	0.621 / 0.566	16 / 36
Qwen3-4B	0.795 / 0.769	7 / 12
Llama-3.1-8B-Inst	0.256 / 0.178	0 / 16
Phi-4-mini-inst	0.256 / 0.203	7 / 20
Phi-4-reasoning	0.863 / 0.840	14 / 19

Appendix

Table 14: Public example filter ablation on LiveCodeBench at

N=10

. Each paired entry is reported as with filtering / without filtering.

Model	$N=5$	$N=10$	$N=20$
Qwen3-14B	612.44 s	1425.86 s	3000.88 s
Qwen3-4B	589.46 s	1357.30 s	2853.48 s
Qwen3-4B-Instruct-2507	734.04 s	1659.71 s	3511.05 s
Llama-3.1-8B-Instruct	482.95 s	1189.17 s	2589.06 s
Phi-4-mini-instruct	665.81 s	1558.50 s	2891.10 s
Phi-4-reasoning	322.60 s	776.52 s	1807.67 s

Appendix

Table 15: SEP runtime on HumanEval+ aggregated across the full benchmark evaluation.

Model	$N=5$	$N=10$	$N=20$
Qwen3-14B	1634.37 s	3769.23 s	7822.59 s
Qwen3-4B	1436.68 s	3317.21 s	6433.39 s
Qwen3-4B-Instruct-2507	973.82 s	2222.15 s	4315.31 s
Llama-3.1-8B-Instruct	335.52 s	761.26 s	1719.09 s
Phi-4-mini-instruct	333.98 s	858.72 s	1852.35 s
Phi-4-reasoning	1479.12 s	3470.91 s	6519.91 s

Appendix

Table 16: SEP runtime on LiveCodeBench aggregated across the full benchmark evaluation.

Model	Test-generation elapsed
gpt-oss-20b	00:50:12
Llama-3.1-8B-Instruct	00:14:13
Phi-4-reasoning	00:22:15
Phi-4-mini-instruct	00:20:59
Qwen3-14B	10:52:58
Qwen3-4B	08:48:38

Appendix

Table 17: CodeT auxiliary test-generation time on HumanEval+ aggregated across the full benchmark evaluation (format: HH:MM:SS). Candidate execution time is not included.

Model	Test-generation elapsed
gpt-oss-20b	20:41:55
Llama-3.1-8B-Instruct	04:11:43
Phi-4-reasoning	16:13:01
Phi-4-mini-instruct	05:44:47
Qwen3-14B	15:10:39
Qwen3-4B	10:22:01

Appendix

Table 18: CodeT auxiliary test-generation time on LiveCodeBench aggregated across the full benchmark evaluation (format: HH:MM:SS). Candidate execution time is not included.

Problem Title
Minimum Average of Smallest and Largest Elements
Apply Operations to Make String Empty
Lexicographically Smallest String After Substring Operation
Apply Operations to Make Two Strings Equal
Find Maximum Removals from Source String
Maximum Number of Groups with Increasing Length

Appendix

Table 19: Excluded LiveCodeBench problem titles.

LLM Code Generation

Large language model code generation studies how models translate instructions and context into executable programs and how to make them reliable.

See all 309 papers

Semantic Voting: Execution-Grounded Consensus for LLM Code Generation
May 9, 2026Shan Jiang, Zijian Yi, Chenguang ZhuExecution-Guided Code Generation Code Generation Evaluation
The University of Texas at Austin
EffiPair: Improving the Efficiency of LLM-generated Code with Differential Execution Feedback
Apr 6, 2026Samira Hajizadeh, Suman JanaInference-Time Optimization Code Optimization
Department of Electrical Engineering, Columbia University, New York, NY, USA · Department of Computer Science, Columbia University, New York, NY, USA
P $^{3}$ : Joint Program-and-Proof Planning for Verified Code Generation
Aug 10, 2026Zenan Li, Ziran Yang, Peiyang Song +2Code Generation Formal Verification
1Apodex · 2Princeton University · 3Caltech +1

May 9, 2026J/K move · Enter open · S save

Semantic Voting: Execution-Grounded Consensus for LLM Code Generation

Shan Jiang, Zijian Yi, Chenguang Zhu

The University of Texas at Austin

Execution-Guided Code Generation Code Generation Evaluation LLM Code Generation

Open paper AskPDF arXiv

LLM code-generation pipelines often sample multiple candidates and select one final answer without access to a complete oracle. Existing pipelines mix textual voting, ranking, and execution-based agreement, but the relative contribution of each component remains unclear. We study 18 configurations across different models, thinking levels, and benchmarks, comparing output-pattern majority voting, weighted voting, MBR-Exec, and SemanticVote - a method that clusters candidates by execution fingerprints on LLM-generated inputs. Three findings emerge. (1) The best execution-based selector exceeds output-pattern majority voting by 19-52 percentage points on every configuration, with every execution-based selector exceeding it by at least 18 points. (2) Once candidates are executed on diverse inputs, aggregation rule has limited effect: SemanticVote, weighted voting, and MBR-Exec are statistically indistinguishable across all 18 configurations. The largest factor is input quality: sketch-based input generation consistently outperforms direct LLM generation by 0.6-2.1 pp and random fuzzing by up to 11.3 pp. (3) Thinking level interacts differently with selection families: deeper thinking improves majority voting by 12 pp but execution-based methods stay flat or degrade as candidate diversity falls. These results frame inference-time code selection as a signal-quality problem rather than an aggregation-rule problem: when oracles are unavailable, the behavioral evidence matters more than the aggregation rule.

Abstract

LLM Code Generation

Explore similar work

Semantic Voting: Execution-Grounded Consensus for LLM Code Generation

EffiPair: Improving the Efficiency of LLM-generated Code with Differential Execution Feedback

P3^{3}3: Joint Program-and-Proof Planning for Verified Code Generation

P $^{3}$ : Joint Program-and-Proof Planning for Verified Code Generation