ExeBench-FHD: Addressing Fidelity Issues in Decompiler Evaluation

Akarsh Ashok; Dinil Mon Divakaran; Vivek Balachandran

cs.SEOct 20, 2026

ExeBench-FHD: Addressing Fidelity Issues in Decompiler Evaluation

Authors: Akarsh Ashok, Dinil Mon Divakaran, Vivek Balachandran

Organizations: Singapore Institute of Technology · A*STAR Institute of Advanced Intelligence and Computing

Abstract

ExeBench is one of the most widely used benchmarks for evaluating LLM-based decompilers. We analyze ExeBench and identify nine issues that affect how faithfully a test measures a decompiler. Three of them prevent LLM decompilers from recovering C code from ExeBench's function. These are ambiguous parameter order when integer and floating-point types are mixed, unused parameters that optimized assembly may drop, and constants stored outside the function's assembly code. Six more affect test validity: non-void functions without a return, placeholder types, static or inline functions removed by the optimizer, functions named main, trivial bodies and truncated definitions. We present ExeBench-FHD, a backward-compatible revision of ExeBench that adds 16 new fields: call-site assembly, data sections, test wrappers, linked-binary disassembly and five warning flags. In ExeBench, 38.9% of functions have at least one of the five flags; dropping records with any of the four warning flags (all except unused_param, which marks functions that the new wrappers make evaluable) keeps 63.9% of the corpus. In an evaluation at -O2 on four architectures, the new fields raise GPT 6 Sol's re-execution rate by up to 26.5%, 19.1% and 21.0% on functions that read data sections, mix parameter types and have unused parameters, respectively.

Figures & tables

Split	Functions
train_not_compilable	715,700
train_real_compilable	885,074
train_synth_compilable	4,159,373
train_real_simple_io	42,769
train_synth_simple_io	640,116
train_synth_rich_io	107,250

Table 1: The ten ExeBench splits and their number of functions.

ID	Issue	Corpus %	test_real %
P1	Mixed int/float parameters	1.46	11.48
P2	Unused parameter	7.56	7.40
P3	Reads a data section ( -O2 )	16.18	16.31
P4	Non-void without return	1.73	0.19
P5	Dummy UDT dependency	9.27	0.52
P6	static / inline definition	36.31	28.73

Table 2: Prevalence of the nine issues over all 6,564,549 functions and over test_real (2,134). P3: non-empty data section at -O2 on at least one architecture. P6: static or inline modifier.

Arch.	int	float
x86	stack	stack
x86-64	rdi,rsi,rdx,rcx,r8,r9	xmm0 - xmm7
ARM32	r0 - r3	s0 - s15
ARM64	x0 - x7	s0 - s7

Table 3: Registers for int and float arguments (ARM32: hard-float ABI).

Field	Type	Issue
func_call	dict	P1
unused_param	bool	P2
opt_real_exe_wrapper opt_angha_exe_wrapper	str	P2
data_section	dict	P3
no_return	bool	P4
dummy_udt	bool	P5

Table 4: New fields introduced by ExeBench-FHD .

Figure 1: Overview of the ExeBench-FHD construction pipeline.

Split	x86	x86-64	ARM32	ARM64
test_real	9.4	15.8	2.5	5.9
test_synth	27.6	28.3	26.9	27.6
valid_real	10.2	16.5	3.0	6.5
valid_synth	25.9	26.5	25.2	25.8
train_real_compilable	16.9	22.2	18.9	20.0
train_synth_compilable	16.4	17.4	16.6	17.0

Table 5: Records with a non-empty data section at -O2 (%). All ten splits in Table 23 .

Subset	x86-64	x86	ARM64	ARM32
Non-empty data sec. (P3)	+19.5	+18.1	+26.5	+10.0
Mixed param. types (P1)	+17.0	+9.8	+19.1	+12.5
Unused param. (P2)	+17.6	+18.0	+17.1	+21.0

Table 6: Gain in re-execution rate (percentage) for GPT 6 Sol when func_call and data_section are added to the assembly ( test_real , -O2 ).

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Field	Origin	Issue
asm	ExeBench	—
real_iospec , angha_iospec	ExeBench	—
real_io_pairs , angha_io_pairs	ExeBench	—
func_call	New	P1
func_call_errors	New	P1 (diag.)
data_section	New	P3

Appendix

Table 7: Fields holding nested JavaScript Object Notation (JSON) objects (11). diag. : diagnostic field that records why the main field is empty. func_call , data_section and their error fields are indexed by architecture (x86, x86-64, ARM32, ARM64); objdump_asm and its error field are indexed by architecture and then by optimization level ( O0 – O3 , Os ). An error entry is null exactly when that build succeeded.

Field	Origin	Issue
path , ref , fname , doc	ExeBench	—
func_def , func_head , func_head_types	ExeBench	—
real_deps , angha_deps	ExeBench	—
real_error , angha_error	ExeBench	—
real_io_error , angha_io_error	ExeBench	—
real_exe_wrapper , angha_exe_wrapper	ExeBench	—

Appendix

Table 8: String fields that may be empty (null) (20).

Field	Origin	Issue
real_io_pairs_are_trivial	ExeBench	—
angha_io_pairs_are_trivial	ExeBench	—
signature (list of strings)	ExeBench	—
unused_param	New	P2
no_return	New	P4
dummy_udt	New	P5

Appendix

Table 9: Boolean fields and the single list-valued field (8).

Split	Mixed types	Mismatch
test_real	11.5	3.8
test_synth	3.3	0.1
valid_real	11.1	4.2
valid_synth	3.2	0.2
train_real_compilable	2.1	1.9
train_synth_compilable	1.4	0.9

Appendix

Table 10: Mixed parameter types (%). Mismatch : signature lists a different number of types than the definition.

Split	no_ret.	dummy	trivial	trunc.	unused
test_real	0.2	0.5	27.4	0.0	7.4
test_synth	0.3	6.2	31.7	0.0	5.9
valid_real	0.2	0.4	25.4	0.0	7.3
valid_synth	0.3	6.0	33.0	0.0	6.2
train_real_compilable	1.5	4.1	36.1	0.07	6.2
train_synth_compilable	0.7	11.2	24.7	0.03	6.9

Appendix

Table 11: Percentage of records with each warning flag set, per split. Columns are no_return , dummy_udt , trivial_function , truncated_funcs , unused_param .

Split	Records	Disassembled	$\geq$ 1 of 20 (%)	All 20 (%)
test_real	2,134	2,108	98.8	85.8
test_synth	5,000	2,269	45.4	44.8
valid_real	2,133	2,107	98.8	86.2
valid_synth	5,000	2,225	44.5	43.8
train_real_compilable	885,074	534,111	60.3	49.4
train_synth_compilable	4,159,373	1,384,908	33.3	31.4

Appendix

Table 12: Coverage of objdump_asm per split. “

\geq

1 of 20” is the share of records disassembled for at least one of the 20 (architecture, optimization level) combinations (x86, x86-64, ARM32, ARM64

\times

-O0 – -O3 , -Os ); “All 20” is the share that succeeded on every combination.

Split	no_ret.	dummy	trivial	trunc.	unused
test_real	0.2 / 0.2	0.5 / 0.5	27.4 / 27.5	0.0 / 0.0	7.4 / 7.3
test_synth	0.3 / 0.4	6.2 / 3.5	31.7 / 46.3	0.0 / 0.0	5.9 / 6.9
valid_real	0.2 / 0.2	0.4 / 0.4	25.4 / 25.3	0.0 / 0.0	7.3 / 7.3
valid_synth	0.3 / 0.2	6.0 / 4.0	33.0 / 48.4	0.0 / 0.0	6.2 / 8.2
train_real_compilable	1.5 / 1.6	4.1 / 1.9	36.1 / 39.3	0.07 / 0.03	6.2 / 7.9
train_synth_compilable	0.7 / 1.0	11.2 / 5.1	24.7 / 35.6	0.03 / 0.02	6.9 / 8.0

Appendix

Table 13: Percentage of records with each warning flag set, as all records / disassembled records . The second value is over the records counted in the Disassembled column of Table 12 . A record can carry several flags.

Flag pair	Records	%
trivial & unused_param	229,487	3.496
dummy_udt & trivial	160,783	2.449
dummy_udt & unused_param	112,514	1.714
no_return & trivial	44,305	0.675
no_return & unused_param	41,105	0.626
trivial & truncated	9,044	0.138

Appendix

Table 14: Pairwise flag co-occurrence over all 6,564,549 records. Percentages are of the full corpus, not conditional on either flag.

Split	Raw	Dropped	Kept (%)
test_real	2,134	592	72.26
test_synth	5,000	1,847	63.06
valid_real	2,133	547	74.36
valid_synth	5,000	1,902	61.96

Appendix

Table 15: Evaluation-split size after dropping records with dummy_udt or trivial_function set.

Panel of Figure 2 — Figure 2: Percentage of records not classified as trivial when the triviality rule is “body has at most $k$ non-blank lines”, for $k\in\{1,2,3,5,10\}$ , on four representative splits. Percentages are over records whose body could be located by matching braces: 99.74% of the corpus, and 98.3% of train_not_compilable , which holds most truncated definitions. At $k=2$ each curve is within 0.6% of $100\%-$ the trivial_function rate in Table 11 . The gap is the multi-line single- return bodies described in Section C.3 .

Split	Raw	After filter (%)
test_real	2,134	72.21
test_synth	5,000	62.90
valid_real	2,133	74.31
valid_synth	5,000	61.72
train_real_compilable	885,074	60.73
train_synth_compilable	4,159,373	66.06

Appendix

Table 16: Filtering funnel per split. After filter is the percentage of the split’s raw count surviving the drop of all four warning flags ( dummy_udt , truncated_funcs , no_return , trivial_function ); unused_param is not dropped because the either-form wrappers already make those functions evaluable.

Split	Raw	$-$ dummy_udt (%)	All four flags (%)	func_call elig. (%)	data_section elig. (%)	IO-executable (%)
test_real	2,134	99.48	72.21	92.74	98.78	100.00
test_synth	5,000	93.78	62.90	37.82	44.58	100.00
valid_real	2,133	99.58	74.31	92.36	98.78	100.00
valid_synth	5,000	93.96	61.72	36.82	43.94	100.00
train_real_compilable	885,074	95.90	60.73	44.29	58.47	16.11
train_synth_compilable	4,159,373	88.78	66.06	20.22	32.69	17.94

Appendix

Table 17: Filtering funnel and per-field eligibility ( elig. : the field is available for the record).

-

dummy_udt keeps records without dummy_udt . All four flags additionally drops truncated_funcs , no_return and trivial_function , and equals the After filter column of Table 16 . IO examples are present for almost every record, because of how the splits were built, on the evaluation and IO-focused training splits, but not on the two general *_compilable splits or train_not_compilable .

Arch	Test form		Re-compile (%)		Re-execute (%)
		$n$	asm	+ctx	asm	+ctx	$\Delta$
x86-64	original	153	30.7	77.1	29.4	75.2	+45.8
	ExeBench-FHD	153	82.4	83.0	63.4	81.0	+17.6
x86	original	144	31.2	64.6	29.2	63.2	+34.0
	ExeBench-FHD	144	84.0	84.7	63.9	81.9	+18.0
ARM64	original	152	31.6	72.4	30.9	69.7	+38.8

Appendix

Table 18: Functions with a parameter the -O2 code never uses, scored against the original wrapper ( real_exe_wrapper , all parameters) and against ExeBench-FHD ’s wrapper ( opt_real_exe_wrapper ), which also accepts the used-only call and counts a prediction as correct if it passes either form. With context the model recovers all original parameters from func_call . The gain is significant on every architecture (

p<10^{-3}

, exact paired sign test).

Arch		Re-compile (%)		Re-execute (%)
	$n$	asm	+ctx	asm	+ctx	$\Delta$
x86-64	1857	88.0	89.8	80.5	84.1	+3.6
x86	1657	86.9	86.8	76.0	78.8	+2.8
ARM64	1773	86.5	87.9	73.8	76.8	+3.0
ARM32	1752	79.9	81.4	73.6	75.1	+1.5
Functions evaluable on all four architectures

Appendix

Table 19: GPT 6 Sol decompilation of test_real at -O2 : assembly only (asm) vs. assembly plus data_section and func_call (+ctx). Functions with an unused parameter are excluded (see Table 18 ). Top: every function that can be prompted and evaluated on that architecture; bottom: the 1,605 functions evaluable on all four, so rates are comparable across architectures.

n

: number of functions;

\Delta

: re-execution gain from the context.

Arch		Re-compile (%)		Re-execute (%)
	$n$	asm	+ctx	asm	+ctx	$\Delta$
x86-64	287	82.9	87.8	56.1	75.6	+19.5
x86	177	83.6	85.3	42.9	61.0	+18.1
ARM64	117	85.5	88.9	47.0	73.5	+26.5
ARM32	50	72.0	74.0	46.0	56.0	+10.0

Appendix

Table 20: Functions that read bytes outside .text (non-empty data_section ), e.g. floating-point constants or lookup tables in .rodata . Same setup as Table 19 . ARM32 keeps most constants in literal pools inside .text , and ARM64 builds many constants directly in instructions, hence the smaller

n

Arch		Re-compile (%)		Re-execute (%)
	$n$	asm	+ctx	asm	+ctx	$\Delta$
x86-64	199	62.8	75.9	52.8	69.8	+17.0
x86	195	88.7	91.3	65.6	75.4	+9.8
ARM64	199	62.3	81.9	44.7	63.8	+19.1
ARM32	199	59.8	73.4	51.3	63.8	+12.5

Appendix

Table 21: Functions whose signature mixes floating-point and integer/pointer parameters, where the function’s own assembly does not show the parameter order but the call in func_call does. 32-bit x86 passes all arguments on the stack in order, so the function’s assembly already shows it and the gain is smallest there.

Arch	Compile fail (%)	Link fail (%)	OK (%)
x86	15.69	53.62	30.69
x86-64	11.86	55.04	33.10
ARM32	12.26	54.94	32.80
ARM64	12.25	54.97	32.78

Appendix

Table 22: Build outcome of the data_section field over all 6,564,549 records, by architecture. Compiler timeouts (60 s limit) affect fewer than 15 records per architecture and are omitted.

	Has data (%)				Build fails (%)
Split	x86	x86-64	ARM32	ARM64	x86	x86-64	ARM32	ARM64
test_real	9.4	15.8	2.5	5.9	9.6	0.6	4.5	4.5
test_synth	27.6	28.3	26.9	27.6	55.1	54.6	54.7	54.7
valid_real	10.2	16.5	3.0	6.5	9.4	1.1	5.5	5.5
valid_synth	25.9	26.5	25.2	25.8	55.8	55.4	55.4	55.4
train_real_compilable	16.9	22.2	18.9	20.0	48.9	39.8	41.4	41.4

Appendix

Table 23: data_section outcome per split and architecture at -O2 . Has data is the share of records with at least one non-empty data section. Build fails is the share where compiling or linking failed, as opposed to succeeding with no data to report. The remainder built successfully and reference no data outside .text .

Split	All 4	2–3 of 4	1 of 4	Failed on all	Skipped
test_real	80.6	10.7	1.4	7.1	0.1
test_synth	1.1	36.7	0.1	53.6	8.6
valid_real	80.2	10.6	1.6	7.6	0.0
valid_synth	1.0	35.8	0.0	53.3	9.9
train_real_compilable	33.7	9.9	0.6	39.5	16.2
train_synth_compilable	3.4	16.8	0.1	62.9	16.8

Appendix

Table 24: Outcome of building func_call on the four target architectures, as a percentage of each split. Skipped records are never attempted (unsupported parameter type, or a function named main ). Failed on all records are attempted on all four architectures and fail on each. Rows sum to 100 up to rounding.

Largest instance (lines)	Records
301–1,000	45,969
1,001–5,000	4,135
5,001–10,000	220
10,001–50,000	113
50,001–100,000	28
100,001–500,000	10

Appendix

Table 25: Records with at least one objdump_asm instance over 300 instruction lines, bucketed by the record’s largest instance over all architectures and optimization levels. The one unusual record whose objdump_asm is omitted ( Section G.1 ) is not counted.

Benchmark Design

Benchmark design creates tasks, datasets, metrics, and protocols that make AI capabilities measurable and comparisons credible.

See all 394 papers

When LLM Decompilers Recompile More and Preserve Less
Sep 4, 2026Chang Liu, Edward Raff, Kristopher MicinskiSoftware Reverse Engineering Differential Testing
Syracuse University · CrowdStrike
CoDe-R: Refining Decompiler Output with LLMs via Rationale Guidance and Adaptive Inference
Apr 14, 2026Qiang Zhang, Zhongnian LiAdaptive Inference Software Reverse Engineering
School of Computer Science and Technology / School of Artificial Intelligence, China University of Mining and Technology, Xuzhou, China · Mine Digitization Engineering Research Center of the Ministry of Education, China University of Mining and Technology, Xuzhou, China
REFORGE: A Method for Benchmarking LLMs' Reverse Engineering Capabilities in Decompiled Binary Function Naming
Jul 7, 2026Nicolas Koller, Andreas u. SchmidtLLM Evaluation Software Reverse Engineering
Systrion AG Flughafenstraße 52 22335 Hamburg, Germany · Wilhelm Büchner Hochschule Hilpertstraße 31 64295 Darmstadt, Germany

Sep 4, 2026J/K move · Enter open · S save

When LLM Decompilers Recompile More and Preserve Less

Chang Liu, Edward Raff, Kristopher Micinski

Syracuse University · CrowdStrike

Software Reverse Engineering Differential Testing Automated Software Testing

Open paper AskPDF arXiv

Decompilation recovers high-level source from compiled machine code and serves as a foundation for security tasks such as vulnerability detection and malware analysis. Traditional decompilers like Ghidra and Hex-Rays expose whatever they cannot resolve as visible placeholders and often emit pseudocode that will not compile or execute; LLM-based decompilers produce clean, idiomatic C and are now judged almost entirely by recompilability and re-executability: whether the output builds and passes its shipped input/output tests. We show that these metrics can reward the wrong path: a function may recompile and pass every shipped test yet diverge on other legitimate inputs, and a disclosed vulnerability may disappear from the recompiled code with no visible trace of the crash. Neither failure is caught by existing suites. To address this gap, we propose Decompile-Diverge, a behavioral comparison oracle not relying on fixed or hand-crafted tests: for each function it synthesizes a driver, grows a fuzzing corpus from the reference, and reruns the decompiled code on the same inputs to detect changes in the function's behavior. Across eight systems in nine configurations on established LLM decompilation corpora, candidates that pass every shipped test still diverge from the original on our input corpus: 4.9% overall, and as many as 13% for a single system. On 300 real GitHub library functions and 287 CVE-grounded functions, recompilability and behavioral agreement can come apart: the strongest refinement LLM lifts Ghidra's build rate from 75% to 90%, while its Matched rate falls from 74% to 62%; on disclosed vulnerabilities, up to one tenth exhibit Crash Absence in its output. Source-level analysis traces this divergence to introduced fields, types, callees, and guards that replace the visible unknowns traditional tools leave behind.

Abstract

Benchmark Design

Explore similar work

When LLM Decompilers Recompile More and Preserve Less

CoDe-R: Refining Decompiler Output with LLMs via Rationale Guidance and Adaptive Inference

REFORGE: A Method for Benchmarking LLMs' Reverse Engineering Capabilities in Decompiled Binary Function Naming