ISA-Bench: A Benchmark for Computational Reasoning Across Instruction Set Architectures

Aditya Pola; Arkaprava Majumdar; Vineeth N. Balasubramanian

cs.AISep 19, 2026

ISA-Bench: A Benchmark for Computational Reasoning Across Instruction Set Architectures

Authors: Aditya Pola, Arkaprava Majumdar, Vineeth N. Balasubramanian

Organizations: IIT Hyderabad · Microsoft Research, India

Abstract

Large language model code generation benchmarks primarily evaluate well-resourced languages like Python and Java, where models benefit from abundant training data. They provide limited evidence about reasoning in unfamiliar computational models: deriving arithmetic from a single subtract instruction, coordinating parallel programs across communicating nodes, or wiring logic gates into circuits. We present ISA-Bench, a benchmark of programming games with constrained instruction sets. For each game we provide a full execution stack (parser, VM, and verifier), enabling automated evaluation with structured feedback for iterative refinement. Reasoning models achieve higher average solve rates than code-specialized and general-purpose models, but unfamiliar syntax remains a major source of failure. Models solve more tasks with iterative feedback, though the gains vary substantially across architectures. We introduce a reasoning--execution gap (REG) analysis that reveals a recurring disconnect between identifying a plausible computational strategy and expressing it as a correct program in the target ISA. Code is open-sourced.

Figures & tables

Figure 1: Overview of the five ISA-Bench environments. Each panel shows the computational model, an example task or code snippet, and the task range for one game. The environments progress from a single-instruction machine ( SIC-1 , where even addition requires a multi-step derivation from subtract-and-branch) through sequential architectures with increasing structure ( HRM ’s accumulator with conveyor I/O; TIS-100 ’s parallel node grid with blocking inter-node communication) to declarative design ( MHRD ’s compositional circuit wiring from a single NAND primitive) and spatial output generation ( BOX-256 ’s memory-mapped pixel display). This progression tests computational primitives: algebraic derivation, sequential planning, parallel coordination, structural composition, and spatial reasoning.

Figure 2: Evaluation pipeline for each task. The model receives an ISA specification and task description, then generates a program that passes through three stages: parsing (syntax validation), execution (running on the game’s VM), and verification (comparing outputs to test cases). Each stage can produce a distinct failure type. On failure, structured feedback is returned for up to four retries (five attempts in total).

Benchmark	Tasks	Lang.	What it tests
HumanEval	164	Python	Function completion, pass@ $k$
MBPP	974	Python	Entry-level synthesis
CodeContests	13k	C++/Py	Algorithmic reasoning
SWE-bench	2.3k	Python	Repository issue resolution
LiveCodeBench	Live	Python	Contamination-free, self-repair
ISA-Bench	92	5 ISAs	Unfamiliar ISA synthesis, reasoning–execution gap

Table 1: Comparison with code-generation benchmarks. Existing work predominantly targets mainstream programming languages. ISA-Bench evaluates synthesis in unfamiliar instruction sets.

Panel of Figure 3 — Figure 3: Tasks solved per game for 14 models, in the same order as Table 4 . Dark bars show solve@1; light extensions show additional solves obtained through retries, so the full stacked bar shows solve@5.

Panel of Figure 4 — Figure 4: First-attempt outcome distributions by game, aggregated across 14 models.

Score	Plan correctness	Implementation correctness
0	Incorrect or irrelevant reasoning; the model does not identify a viable computational strategy	Non-executable output; parser failure, reasoning text leaked into code block, or wrong output format
1	Partial reasoning that touches relevant concepts but misses the core algorithmic structure	Code that parses but is structurally broken; wrong opcodes, missing control flow, or fundamentally flawed logic
2	Mostly correct high-level strategy with incomplete or flawed low-level details	Near-correct implementation with localized bugs such as off-by-one errors, wrong addresses, or missing edge cases
3	Correct task decomposition and computational strategy for the target ISA	Correct or nearly correct execution; all or almost all test cases pass

Table 2: Scoring rubric for the reasoning–execution gap (REG) analysis. For each task, GPT-5.5 independently scores the model’s computational plan and its implementation under this fixed rubric. Solved tasks receive 3/3 automatically; unsolved tasks are judged from the model’s final attempt including its reasoning trace, generated code, and execution feedback.

Environment	Plan (P)	Impl (I)	Gap (G)
BOX-256	1.44	0.48	0.96
SIC-1	1.71	0.77	0.94
HRM	2.07	1.26	0.81
TIS-100	1.97	1.13	0.84
MHRD	2.60	2.54	0.06

Table 3: Mean plan correctness (P), implementation correctness (I), and reasoning–execution gap (

G=P-I

) averaged across all 14 models. Higher G indicates models plan correctly but fail during implementation. Per-model scores are in Appendix Table 5 .

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Target pixel patterns for all 10 BOX-256 tasks. Each grid is $16\times 16$ pixels. The program must produce the exact pattern by writing color values to shared code/display memory.

Model	T	HRM	SIC-1	TIS	MHRD	BOX	Total
		(36)	(31)	(7)	(8)	(10)	(92)
Gemma-4 31B	T	18/32	2/13	1/3	2/8	0/0	23/56
Qwen3 32B	T	7/16	1/4	1/2	4/7	0/0	13/29
Qwen3 14B	T	8/13	2/2	1/2	3/7	0/0	14/24
Phi-4-Reasoning	T	0/20	0/1	0/1	1/2	0/0	1/24
DeepSeek-R1 32B	T	8/13	0/2	1/1	3/7	0/0	12/23

Appendix

Table 4: Per-model solve counts reported as solve@1 / solve@5, corresponding respectively to first-attempt performance and performance after up to five iterative attempts with execution feedback. Task counts per environment are shown in parentheses. Model type abbreviations: T = thinking/reasoning model, C = coder-specialized model, G = general-purpose model.

	BOX-256			HRM			MHRD			SIC-1			TIS-100
Model	P	I	G	P	I	G	P	I	G	P	I	G	P	I	G
Gemma-4 31B	1.90	0.90	1.00	2.72	2.72	0.00	3.00	3.00	0.00	1.68	1.31	0.37	2.57	1.86	0.71
Qwen3-32B	1.10	0.30	0.80	1.89	1.39	0.50	3.00	3.00	0.00	1.32	0.68	0.65	1.57	1.00	0.57
Qwen3-14B	1.00	0.30	0.70	1.83	1.36	0.47	3.00	3.00	0.00	1.26	0.71	0.55	1.57	1.29	0.29
Qwen3-Coder 30B	1.90	0.70	1.20	2.31	1.50	0.81	3.00	3.00	0.00	2.16	1.42	0.74	2.29	1.57	0.71
DeepSeek-R1 32B	1.00	0.60	0.40	1.83	1.42	0.42	2.75	2.75	0.00	1.16	0.65	0.52	1.57	1.14	0.43

Appendix

Table 5: Per-model plan correctness (P), implementation correctness (I), and reasoning–execution gap (G = P

-

I) across all 14 models and five environments. Mean row corresponds to Table 3 in the main text.

Model	Metric	HRM (36)	SIC-1 (31)	TIS-100 (7)	MHRD (8)
Gemma-4 31B	solve@1	$17.7\pm 2.1$	$0.3\pm 0.6$	$1.3\pm 0.6$	$2.3\pm 0.6$
	solve@5	$32.0\pm 1.0$	$2.7\pm 2.1$	$2.7\pm 0.6$	$7.7\pm 0.6$
Devstral-Small-2	solve@1	$5.7\pm 2.1$	$1.3\pm 0.6$	$0.7\pm 0.6$	$3.7\pm 1.5$
	solve@5	$11.7\pm 2.1$	$1.7\pm 0.6$	$1.0\pm 0.0$	$6.0\pm 1.0$
Qwen2.5-Coder 32B	solve@1	$5.3\pm 1.2$	$1.0\pm 0.0$	$0.3\pm 0.6$	$4.7\pm 0.6$
	solve@5	$7.0\pm 1.0$	$2.0\pm 1.0$	$1.0\pm 0.0$	$5.3\pm 0.6$

Appendix

Table 6: Solved-task counts across three runs, reported as mean

\pm

sample standard deviation. Task counts per environment are shown in parentheses.

Code Generation Benchmarks

Code generation benchmarks measure how well AI models produce code across programming tasks, languages, and real-world settings.

See all 154 papers

PBEBench: A Multi-Step Programming by Examples Reasoning Benchmark inspired by Historical Linguistics
May 29, 2025Atharva Naik, Prakam, Yash Mathur +6Program Synthesis Inductive Inference
Carnegie Mellon University · Independent Researcher · Ohio State University
Playing Psychic: Using Thought Trees to Predict Reasoning Models Accuracy on Coding Tasks
Apr 18, 2026Jiaxin Fang, Runyuan He, Sahil Bhatia +2Structured Reasoning Code Generation Evaluation
University of California, Berkeley
CodeGolf Bench: A Multi-Language Benchmark for Evaluating Concise Code Generation Capabilities of Large Language Models
May 28, 2026Vedant PadwalLanguage Model Generation Evaluation Code Generation
Independent

May 29, 2025J/K move · Enter open · S save

PBEBench: A Multi-Step Programming by Examples Reasoning Benchmark inspired by Historical Linguistics

Atharva Naik, Prakam, Yash Mathur, Darsh Agrawal+5

Carnegie Mellon University · Independent Researcher · Ohio State University

Program Synthesis Inductive Inference LLM Evaluation Logical Reasoning Synthetic Benchmark Generation LLM Reasoning+1

Open paper AskPDF arXiv

Although many benchmarks evaluate the reasoning abilities of Large Language Models (LLMs) within domains such as mathematics, coding, or data wrangling, few abstract away from domain specifics to examine reasoning as a capability in and of itself. We contribute a novel type of benchmark evaluating the inductive reasoning capabilities of LLMs that is inspired by the forward reconstruction task from historical linguistics but is formulated in an extremely simple, general way (in the form of Programming by Examples). The task involves generating a cascade of simple string rewrite programs to transform a given list of input strings into a list of desired output strings. We present a fully automated pipeline that programmatically generates problems of this type with controllable difficulty, enabling scalable evaluation of reasoning models while avoiding contamination. Using this approach, we construct two benchmarks: PBEBench-Lite, which efficiently stratifies models of varying capabilities, and PBEBench, which requires models to induce programs similar in complexity to those constructed by historical linguists. Our experiments reveal a substantial performance gap between models that leverage test-time compute or LCoT (long chain-of-thought) reasoning and those that do not. Moreover, although recent models show promise, the solve rate for both of them drops below 5% for hard instances of the PBEBench dataset (ground truth cascade lengths of 20 and 30, respectively), falling well short of realistic historical linguistics requirements even with computationally expensive, popular scaling techniques from the PBE and reasoning literature. Additionally, we also study the effectiveness of different scaling strategies and the impact of various hyperparameters on the difficulty of the generated data using gpt-oss-120b, the best-performing open-source model.

Abstract

Code Generation Benchmarks

Explore similar work

PBEBench: A Multi-Step Programming by Examples Reasoning Benchmark inspired by Historical Linguistics

Playing Psychic: Using Thought Trees to Predict Reasoning Models Accuracy on Coding Tasks

CodeGolf Bench: A Multi-Language Benchmark for Evaluating Concise Code Generation Capabilities of Large Language Models