DuET: Dual Execution for Test Output Prediction with Generated Code and Pseudocode

Hojae Han; Jaejin Kim; Seung-won Hwang; Yu Jin Kim; Moontae Lee

cs.SEApr 13, 2026

DuET: Dual Execution for Test Output Prediction with Generated Code and Pseudocode

Authors: Hojae Han, Jaejin Kim, Seung-won Hwang, Yu Jin Kim, Moontae Lee

Organizations: ETRI · Seoul National University · SNU-LG AI Research Center · LG AI Research

Abstract

This work addresses test output prediction, a key challenge in test case generation. To improve the reliability of predicted outputs by LLMs, prior approaches generate code first to ground predictions. One grounding strategy is direct execution of generated code, but even minor errors can cause failures. To address this, we introduce LLM-based pseudocode execution, which grounds prediction on more error-resilient pseudocode and simulates execution via LLM reasoning. We further propose DuET, a dual-execution framework that combines both approaches by functional majority voting. Our analysis shows the two approaches are complementary in overcoming the limitations of direct execution suffering from code errors, and pseudocode reasoning from hallucination. On LiveCodeBench, DuET achieves the state-of-the-art performance, improving Pass@1 by 13.6 pp.

Figures & tables

Panel of Figure 1 — Figure 1: Comparison of test output prediction methods: (a) TestChain Li and Yuan (2024) grounds generated code and executes it directly. (b) LLM-based Pseudocode Execution grounds generated pseudocode and simulates its execution via an LLM. (c) DuET combines both code and pseudocode grounding, executing them via direct and LLM-based paths, respectively.

Panel of Figure 2 — Figure 2: (a) The problem involves complex functional logic that is accurately captured in the generated pseudocode but mistranslated into executable code (i.e., implementation errors) . (b) Both the generated code and pseudocode are correct, but simulating the pseudocode (e.g., involving deeply nested loops) introduces reasoning challenges that lead to LLM prediction errors (i.e., execution hallucinations) .

Figure 3: Overview of our dual-execution framework for test output prediction. Given a problem description and input, we perform two separate executions in parallel. In the left path, we generate pseudocode and use an LLM to reason over it to predict the output (i.e., LLM-based pseudocode execution ). In the right path, we generate pseudocode independently, translate it into executable code, and run it on the input to obtain the output (i.e., direct code execution ). Multiple outputs from both paths are aggregated via path-weighted functional majority voting to determine the final prediction.

Method	Pass@ $1$
Llama-3-8B-Instruct †	$27.2$
Mixtral-8x22B-Instruct †	$45.6$
Mistral-Large †	$48.6$
Gemini-Pro-1.5-May †	$48.6$
Dracarys-Llama-3.1-70B-Instruct †	$49.1$
Dracarys2-Llama-3.1-70B-Instruct †	$52.1$

Table 1: Test output prediction results on LiveCodeBench (May 1, 2023–Apr 1, 2024). Red-colored models have a knowledge cutoff after this date, introducing the possibility of data contamination. Daggered ( † ) results are sourced from the leader board. 7 7 7 https://livecodebench.github.io/leaderboard.html 8 8 8 We treat unbiased Pass@

k

as random selection (see Appendix K.3 ).

Method	Ground.	Exec.	Pass@ $1$
GPT-4-Turbo-2024-04-09	-	-	$72.368\,421\,052\,631\,5$
TestChain	C	D	$74.342\,105\,26$
DuET	C & P	D & L	$81.578\,947\,37$
- direct code exec.	P	L	$69.078\,947\,37$
Llama-3.1-8B-Inst	-	-	$34.013\,157\,89$
TestChain	C	D	$46.271\,929\,82$

Table 2: Pass@

1

comparison on the LiveCodeBench test output prediction benchmark (Jan 1–Apr 1, 2024). Functional majority voting is applied to all baselines by default. Each method is labeled by its grounding type, code (C) or pseudocode (P), and execution type, direct (D) or LLM-based (L).

Method	Grounding	Execution	Pass@ $1$	Pass@ $2$	Pass@ $5$	Pass@ $10$
No Filtering	-	-	$33.9$	50.9	64.8	73.3
CodeT	-	-	54.2	62.8	71.3	74.3
TestChain	Code	Direct	48.6	54.9	61.4	67.6
DuET	Code & Pseudocode	Direct & LLM-based	57.4	65.2	71.8	75.1
- direct code exec.	Pseudocode	LLM-based	55.9	64.5	71.8	75.1

Table 3: End-to-end code generation performance of Llama-3.1-8B-Instruct under different test output prediction methods, evaluated on LiveCodeBench-Easy (Jan 1, 2024–Feb 1, 2025).

Figure 4: Illustrative example of TestChain’s zero-advantage problem in end-to-end code generation, where test input ( Input: (2, 3) ) is generated by CodeT .

Figure 5: Pass@

1

scores of execution methods on LiveCodeBench test output prediction (Jan 1–Apr 1, 2024) by GPT-4-Turbo-2024-04-09 , across problem subsets based on the generated code correctness.

Grounding	Direct Exec.	LLM-based Exec.
Code	TestChain	LLM-based Code Exec.
Pseudocode	-	LLM-based Pseudo. Exec.

Table 4: Design space taxonomy, categorized by execution target (code vs. pseudocode) and an execution mode (direct vs. LLM-based). DuET , marked as a pink diagonal box, chooses the best combination of direct code execution and LLM-based pseudocode execution .

Figure 6: Pass@1 across bins of generated code execution trace lengths on LiveCodeBench test output prediction (Jan 1–Apr 1, 2024) by GPT-4-Turbo-2024-04-09 . Each execution step corresponds to a single line of code executed during code execution.

Method	Ground.	Exec.	Pass@ $1$
no grounding	-	-	$74.245\,852\,187\,028\,6$
direct code exec. (TestChain)	C	D	$75.452\,488\,687\,782\,8$
LLM-based code exec.	C	L	$72.247\,360\,482\,654\,61$
LLM-based pseudocode exec.	P	L	$74.057\,315\,233\,785\,81$
direct & LLM-based code exec.	C	D & L	$77.375\,565\,610\,859\,74$
LLM-based code & pseudo. exec.	C & P	L	$73.642\,533\,936\,651\,59$

Table 5: Test output prediction results of GPT-4-Turbo-2024-04-09 across grounding combinations and execution methods on LiveCodeBench (May 1, 2023–Apr 1, 2024). Functional majority voting is applied across methods. We highlight representative baselines and our methods using distinct colors: TestChain , LLM-based pseudocode execution , and DuET . Variants that use LLM-based code execution (alone or in combination) are marked in blue. Each method is labeled by its grounding type, code (C) or pseudocode (P), and execution type, direct (D) or LLM-based (L).

Method	Ground.	Exec.	Pass@ $1$
No Filtering	-	-	$24.04$
CodeT	-	-	$22.61$
TestChain	C	D	$24.04$
DuET	C & P	D & L	$25.34$
- direct code exec.	P	L	$24.04$

Table 6: End-to-end code generation performance of QwQ-32B under different test output prediction methods, evaluated on BigCodeBench-Hard.

Figure 7: End-to-end code generation performance of Llama-3.1-8B-Instruct evaluated on DevEval (See Appendix F for the full results).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Pseudocode	LLM-based	Pass@1
Granularity	Execution	Pass@1
Low	Step-by-step (default)	31.1
Low	3-step	31.3
Mid (default)	Step-by-step (default)	34.4
Mid (default)	3-step	32.5
High	Step-by-step (default)	25.1

Appendix

Table 7: Test output prediction performance comparison of Llama-3.1-8B-Instruct with LLM-based pseudocode execution over prompt design choices on LiveCodeBench (May 1, 2023–Apr 1, 2024). Rows vary the granularity of the generated pseudocode (low, mid, high), while columns compare two LLM-based execution prompting strategies: a step-by-step simulation and a condensed 3-step reasoning format. Default settings are indicated in parentheses. Functional majority voting is applied with 10 independent samples per output.

Model	Grounding	Execution	Pass@ $1$
Model	Grounding	Execution	All	Easy	Medium	Hard
Llama-3.1-8B-Instruct	-	-	$34.013\,157\,89$	$35.285\,714\,29$	$40.909\,090\,91$	$23.684\,210\,53$
TestChain	Code	Direct	$46.271\,929\,82$	$55.238\,095\,24$	$51.515\,151\,52$	$23.684\,210\,53$
DuET	Code & Pseudocode	Direct & LLM-based	$53.289\,473\,68$	$61.428\,571\,43$	$62.5$	$27.631\,578\,95$
- direct code exec.	Pseudocode	LLM-based	$34.385\,964\,91$	$35.142\,857\,14$	$27.272\,727\,27$	$41.228\,070\,18$

Appendix

Table 8: Test output prediction results on LiveCodeBench (Jan 1–Apr 1, 2024), where difficulty levels of test inputs are estimated using the correctness rate of code generated by the best performing model ( GPT-4-Turbo-2024-04-09 ). Functional majority voting is applied to all baselines by default.

Model	Grounding	Execution	Pass@ $1$
Model	Grounding	Execution	All	Easy	Medium	Hard
Llama-3.1-8B-Instruct	-	-	$34.013\,157\,89$	$39$	$35.25$	$18.181\,818\,18$
TestChain	Code	Direct	$46.271\,929\,82$	$58$	$39.583\,333\,33$	$43.939\,393\,94$
DuET	Code & Pseudocode	Direct & LLM-based	$53.289\,473\,68$	$64$	$51.25$	$36.363\,636\,36$
- direct code exec.	Pseudocode	LLM-based	$34.385\,964\,91$	$46.4$	$28.833\,333\,33$	$27.272\,727\,27$

Appendix

Table 9: Pass@

1

performance on the LiveCodeBench test output prediction benchmark (Jan 1–Apr 1, 2024), where difficulty levels follow the original LiveCodeBench setting based on the problem’s difficulty. Functional majority voting is applied to all baselines by default.

Method	Grounding	Execution	Pass@ $1$
Method	Grounding	Execution	HumanEval	HumanEval+
No Filtering	-	-	$79.2$	69.9
CodeT	-	-	84.7	75.0
TestChain	Code	Direct	83.0	73.6
DuET	Code & Pseudocode	Direct & LLM-based	85.1	75.4
- direct code exec.	Pseudocode	LLM-based	84.8	75.2

Appendix

Table 10: End-to-end code generation performance of Llama-3.1-70B-Instruct under different test output prediction methods, evaluated on HumanEval and HumanEval+.

Figure 8: End-to-end code generation performance of Llama-3.1-8B-Instruct evaluated on DevEval.

Method	Ground.	Exec.	Pass@ $1$
GPT-4-Turbo-2024-04-09	-	-	$72.368\,421\,052\,631\,5$
TestChain	C	D	$74.342\,105\,26$
DuET	C & P	D & L	$81.578\,947\,37$
- direct code exec.	P	L	$69.078\,947\,37$
Llama-3.1-8B-Inst	-	-	$34.013\,157\,89$
TestChain	C	D	$46.271\,929\,82$

Appendix

Table 11: Pass@

1

Model	DuET Voting Method	Pass@ $1$
GPT-4-Turbo-2024-04-09	functional majority voting	$81.6$
GPT-4-Turbo-2024-04-09	+ path-weighted	$81.6$
Llama-3.1-8B-Inst	functional majority voting	$53.3$
Llama-3.1-8B-Inst	+ path-weighted	$53.3$
Llama-3.1-70B-Inst	functional majority voting	$77.631\,578\,947\,368\,4$
Llama-3.1-70B-Inst	+ path-weighted	$78.3$

Appendix

Table 12: The effect of path-weighted FMV on the LiveCodeBench test output prediction benchmark (Jan 1–Apr 1, 2024). Red-colored models have the possibility of data contamination.

Model Name	Approximate
Model Name	Cutoff Date
GPT-4-Turbo-2024-04-09	2023-04-30
Llama-3.1-8B-Instruct	2023-12-31
Llama-3.1-70B-Instruct	2023-12-31
Mistral-Large	2023-01-01
EXAONE-3.5-7.8B-Instruct	2024-11-30

Appendix

Table 13: Cutoff dates for each LLM.

Execution-Guided Code Generation

Execution-guided code generation uses program runs, tests, and tool feedback to improve generated code and assess its correctness.

See all 67 papers

Confidence-Gated Transductive Test Generation for Code Reranking
Sep 11, 2026Sungjae Lee, Youngsik Yoon, Seockbean Song +3LLM Reranking Automated Test Generation
Department of Computer Science and Engineering, POSTECH, South Korea · Graduate School of Artificial Intelligence, POSTECH, South Korea · Microsoft Research Asia, China
SolidCoder: Bridging the Mental-Reality Gap in LLM Code Generation through Concrete Execution
Apr 20, 2026Woojin Lee, Jin-Xia HuangProperty-Based Testing Execution-Guided Code Generation
Electronics and Telecommunications Research Institute, Republic of Korea
Planning to Explore: Curiosity-Driven Planning for LLM Test Generation
Apr 6, 2026Alfonso Amayuelas, Firas Laakom, Piotr Piękos +5Automated Test Generation LLM Planning
University of California, Santa Barbara · King Abdullah University of Science and Technology

Sep 11, 2026J/K move · Enter open · S save

Confidence-Gated Transductive Test Generation for Code Reranking

Sungjae Lee, Youngsik Yoon, Seockbean Song, Siwei Wang+2

Department of Computer Science and Engineering, POSTECH, South Korea · Graduate School of Artificial Intelligence, POSTECH, South Korea · Microsoft Research Asia, China

LLM Reranking Automated Test Generation

Open paper AskPDF arXiv

Test case synthesis is crucial for evaluating and ranking programs generated by large language models (LLMs). However, constructing high-quality test cases remains challenging because reliable expected outputs are often difficult to obtain. We propose Confidence-Gated Transductive Test Generation (CoTT), which first uses an efficient inductive procedure and invokes transductive generation only when inductive confidence is low. This adaptive design improves output reliability while allocating extra computation only when needed. On code reranking benchmarks, CoTT outperforms prior baselines across the reported metrics while reducing cost relative to applying transductive generation to every input. These results show that confidence-based allocation of test-time computation provides a favorable efficiency-effectiveness trade-off with a single efficient LLM.

Abstract

Execution-Guided Code Generation

Explore similar work

Confidence-Gated Transductive Test Generation for Code Reranking

SolidCoder: Bridging the Mental-Reality Gap in LLM Code Generation through Concrete Execution

Planning to Explore: Curiosity-Driven Planning for LLM Test Generation