EXHIB: A Benchmark for Realistic and Diverse Evaluation of Function Similarity in the Wild

Yiming Fan; Jun Yeon Won; Ding Zhu; Melih Sirlanci; Mahdi Khalili; Zhiqiang Lin; Carter Yagemann

cs.CRApr 2, 2026

EXHIB: A Benchmark for Realistic and Diverse Evaluation of Function Similarity in the Wild

Authors: Yiming Fan, Jun Yeon Won, Ding Zhu, Melih Sirlanci, Mahdi Khalili, Zhiqiang Lin, Carter Yagemann

Organizations: The Ohio State University

Abstract

Binary Function Similarity Detection (BFSD) is a core problem in software security, supporting tasks such as vulnerability analysis, malware classification, and patch provenance. In the past few decades, numerous models and tools have been developed for this application; however, due to the lack of a comprehensive universal benchmark in this field, researchers have struggled to compare different models effectively. Existing datasets are limited in scope, often focusing on a narrow set of transformations or types of binaries, and fail to reflect the full diversity of real-world applications. We introduce EXHIB, a benchmark comprising five realistic datasets collected from the wild, each highlighting a distinct aspect of the BFSD problem space. We evaluate 9 representative models spanning multiple BFSD paradigms on EXHIB and observe performance degradations of up to 30% on firmware and semantic datasets compared to standard settings, revealing substantial generalization gaps. Our results show that robustness to low- and mid-level binary variations does not generalize to high-level semantic differences, underscoring a critical blind spot in current BFSD evaluation practices.

Figures & tables

Fig. 1 : Definition of the BFSD Problem

Fig. 2 : Systematized Evolution Graph of the Three Main Trends of the BFSD Problem Solutions

Panel of Table 3 — TABLE I : A Comprehensive Comparison of the Datasets Used in Binary Function Similarity Studies.

Task \Variables	A	B	C	V	O	Ob	S
XA	✓	✗	✗	✗	✗	✗	✗
XB	✗	✓	✗	✗	✗	✗	✗
XA+XB	✓	✓	✗	✗	✗	✗	✗
XC	✗	✗	✓	✓	✓	✗	✗
XC+XB	✗	✓	✓	✓	✓	✗	✗
XV	✗	✗	✗	✓	✗	✗	✗

TABLE II : Variables Used in Different Tasks.

Dataset	Standard	Firmware	Malware	Obfuscated	Semantic
Libraries	181	159	61	125	75
Binaries	16,889	6,086	2,905	4,775	5,415
Functions	1,601,164	19,794	6,225	139,045	4,165
-Training	1,249,747	11,302	3,669	92,972	2,322
-Validation	41,893	795	1,118	16,823	592
-Testing	309,524	7,697	1,438	29,250	1,251

TABLE III : Overall Dataset Size and Splits.

Model Name	Standard						Firmware		Malware			Obfuscated	Semantic
	XA	XC	XC+XB	XV	XO	XM	XV	XO	XC	XV	XO	XOb	XS
FCatalog [ 57 ]	0.44	0.81	0.62	0.99	0.94	0.66	0.99	0.90	0.83	0.99	0.89	0.78	0.70
FunctionSimSearch [ 14 ]	0.72	0.66	0.65	0.81	0.79	0.72	0.74	0.61	0.80	0.97	0.90	0.64	0.68
Asm2Vec [ 10 ]	0.44	0.71	0.62	0.81	0.77	0.70	0.62	0.57	0.92	0.99	0.91	0.67	0.53
Massarelli et al. [ 37 ]	0.87	0.80	0.78	0.95	0.90	0.88	0.91	0.80	0.90	0.91	0.93	0.92	0.73
Gemini [ 58 ]	0.96	0.80	0.79	0.98	0.91	0.90	0.96	0.85	0.89	0.99	0.95	0.84	0.83

TABLE IV : An Overview of the AUC Comparison on All Datasets.

Model Name	AUC								MRR10				Recall@1				Testing Time (s)
	XA	XB	XA+XB	XC	XC+XB	XV	XO	XM	XA+XB	XC	XC+XB	XM	XA+XB	XC	XC+XB	XM	Feat	Inf	Tot 100
FCatalog [ 57 ]	0.44	0.79	0.44	0.81	0.62	0.99	0.94	0.66	0.06	0.36	0.16	0.36	0.02	0.24	0.07	0.30	11840	0	3.83
FunctionSimSearch [ 14 ]	0.72	0.83	0.72	0.66	0.65	0.81	0.79	0.72	0.26	0.20	0.15	0.32	0.19	0.15	0.10	0.26	8204	6526	4.76
Asm2Vec [ 10 ]	0.44	0.66	0.44	0.71	0.62	0.81	0.77	0.70	0.01	0.18	0.02	0.15	0.00	0.10	0.00	0.11	2191	1382	1.15
Massarelli et al. [ 37 ]	0.87	0.93	0.87	0.80	0.78	0.95	0.90	0.88	0.29	0.25	0.16	0.41	0.16	0.17	0.08	0.31	299	5404	1.84
Gemini [ 58 ]	0.96	0.98	0.96	0.80	0.79	0.98	0.91	0.90	0.69	0.31	0.27	0.56	0.58	0.24	0.19	0.47	2657	1004	1.18

TABLE V : Inter-Model Comparison of Standard Dataset.

Panel of Figure 8 — Fig. 3 : Comparison of the recall at different K values for the Standard Dataset.

Panel of Figure 9 — Fig. 4 : Recall graphs of Obfuscation and Semantic Dataset

Model Name	AUC			Testing Time (s)
	XV	XO	XV+XO	Feat	Inf	Tot 100
FCatalog [ 57 ]	0.99	0.90	0.91	257	0	3.34
FunctionSimSearch [ 14 ]	0.74	0.61	0.63	106	28	0.37
Asm2Vec [ 10 ]	0.62	0.57	0.57	11	11	0.29
Massarelli et al. [ 37 ]	0.91	0.80	0.81	3	330	4.33
Gemini [ 58 ]	0.96	0.85	0.87	20	105	1.62

TABLE VI : Inter-Model Comparison of Firmware Dataset.

Model Name	AUC			Testing Time (s)
	XC	XO	XV	Feat	Inf	Tot 100
FCatalog [ 57 ]	0.83	0.89	0.99	352	0	24.50
FunctionSimSearch [ 14 ]	0.8	0.9	0.97	214	39	17.61
Asm2Vec [ 10 ]	0.92	0.91	0.99	17	9	1.81
Massarelli et al. [ 37 ]	0.9	0.91	0.93	2	246	17.26
Gemini [ 58 ]	0.89	0.95	0.99	31	37	4.73

TABLE VII : Inter-Model Comparison of Malware Dataset.

Model Name	XOb			Testing Time (s)
	AUC	MRR@10	Recall@1	Feat	Inf	Tot 100
FCatalog [ 57 ]	0.78	0.38	0.34	1,600	0	6.33
FunctionSimSearch [ 14 ]	0.64	0.17	0.11	1,032	805	7.26
Asm2Vec [ 10 ]	0.67	0.19	0.16	266	297	2.23
Massarelli et al. [ 37 ]	0.92	0.43	0.26	70	1,630	6.72
Gemini [ 58 ]	0.84	0.25	0.16	3,024	204	12.76

TABLE VIII : Inter-Model Comparison of Obfuscation Dataset.

Model Name	XS			Testing Time (s)
	AUC	Recall@1	MRR@10	Feat	Inf	Tot 100
FCatalog [ 57 ]	0.70	0.27	0.28	627	0	50.12
FunctionSimSearch [ 14 ]	0.68	0.18	0.16	657	9	53.24
Asm2Vec [ 10 ]	0.53	0.17	0.11	5	5	0.80
Massarelli et al. [ 37 ]	0.73	0.14	0.11	2	728	58.35
Gemini [ 58 ]	0.83	0.33	0.30	3	138	11.27

TABLE IX : Inter-Model Comparison of Semantic Dataset.

Panel of Figure 14 — Fig. 5 : A Comparison of the Similarity Score Distributions.

Panel of Figure 15 — Fig. 6 : Case Study Problem and Functions.

Panel of Figure 16 — Fig. 7 : Comparison of the SOGs of the Two Functions.

Model Name		Tigress Obfuscations	LLVM Obfuscations
FCatalog [ 57 ]		1.00	0.81
FunctionSimSearch [ 14 ]
	G	0.98	0.56
	G + M	1.00	0.64
	G + M + I	1.00	0.65
	w (G + M + I)	0.67	0.64

TABLE X : AUC Comparison Within Obfuscation Dataset.

Explore similar work

REBENCH: A Procedural, Fair-by-Construction Benchmark for LLMs on Stripped-Binary Types and Names (Extended Version)
Apr 30, 2026Jun Yeon Won, Xin Jin, Shiqing Ma +1Reverse Engineering Large Language Model Benchmarks
Ohio State University Columbus, OH, United States · University of Massachusetts Amherst Amherst, MA, United States
VICBench: A Multi-Language Benchmark for Code Vulnerability Detection
Aug 12, 2026Jin Lu, Xuening Han, Yang Zhong +4Common Vulnerabilities And Exposure Benchmark Datasets
Amazon Web Services · University of Pittsburgh · Purdue University
ASSEMBLAGE-DEEPHISTORY: A Cross-Build Binary Dataset with Temporal Coverage
May 20, 2026Chang Liu, Noah Fleischmann, Nicolò Altamura +3Bits History
Syracuse University, Syracuse, USA · Booz Allen Hamilton, McLean, USA · Independent Researcher, Italy +1

Apr 30, 2026, cs.CRJ/K move · Enter open · S save

REBENCH: A Procedural, Fair-by-Construction Benchmark for LLMs on Stripped-Binary Types and Names (Extended Version)

Jun Yeon Won, Xin Jin, Shiqing Ma, Zhiqiang Lin

Ohio State University Columbus, OH, United States · University of Massachusetts Amherst Amherst, MA, United States

Reverse Engineering Large Language Model Benchmarks Benchmark Datasets Bits Procedural Extended Version

Open paper AskPDF arXiv

Large Language Models (LLMs) have achieved remarkable progress in recent years, driving their adoption across a wide range of domains, including computer security. In reverse engineering, LLMs are increasingly applied to critical tasks such as function and variable name recovery and type inference. However, despite the rapid growth of research in this area, progress has been hindered by the absence of a standardized dataset. Existing studies rely on disparate datasets, preprocessing pipelines, and evaluation metrics, making fair comparisons between approaches difficult and obscuring a clear understanding of LLM capabilities in binary analysis. To address these challenges, we present REBench, a comprehensive benchmark dataset for evaluating LLMs on binary reverse engineering tasks. REBench consolidates a superset of existing datasets, comprising hundreds of millions of lines of source code and a diverse collection of binaries spanning multiple architectures and optimization levels. REBench adopts a knowledge-base-driven methodology that stores byte-level stack information to generate ground truth, ensuring that task difficulty is preserved while maintaining universal applicability. This design enables fair evaluation across tasks while avoiding simplifications that could bias results. As a use case, we apply REBench to measure the reverse engineering performance of LLMs and the result demonstrates difficulties in complex tasks.

Abstract

Explore similar work

REBENCH: A Procedural, Fair-by-Construction Benchmark for LLMs on Stripped-Binary Types and Names (Extended Version)

VICBench: A Multi-Language Benchmark for Code Vulnerability Detection

ASSEMBLAGE-DEEPHISTORY: A Cross-Build Binary Dataset with Temporal Coverage