Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios

Ruida Hu; Xinchen Wang; Chao Peng; Cuiyun Gao; David Lo

cs.SEApr 8, 2026

Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios

Authors: Ruida Hu, Xinchen Wang, Chao Peng, Cuiyun Gao, David Lo

Organizations: Harbin Institute of Technology, Shenzhen, China · Independent Researcher, China · Singapore Management University, Singapore

Abstract

The evolution of Large Language Models (LLMs) has catalyzed a paradigm shift towards intent-driven software development, where autonomous agents are expected to design and deliver complete, runnable software systems from scratch. However, existing benchmarks fail to adequately assess this 0-to-1 generation capability due to two fundamental limitations. First, they rely on predefined structural scaffolds, which reduces the task to mere file-filling. Second, they depend on rigid white-box unit testing, which forces generated code to conform to specific internal implementations rather than validating end-to-end user-centric behavior. To bridge this gap, we introduce CLI-Tool-Bench, a novel, structure-agnostic benchmark designed to evaluate the ground-up generation of Command-Line Interface (CLI) tools. Powered by an automated black-box differential testing framework, the benchmark comprises 94 high-quality, real-world repositories spanning diverse programming languages and complexity levels. For each task, agents are provided with an empty workspace, forcing them to autonomously handle repository planning and dependencies. We evaluate the generated software by executing it in isolated sandboxes. The system-level side effects and terminal outputs are then compared against human-written oracles using a rigorous multi-tiered equivalence metric. Extensive evaluation of seven state-of-the-art LLMs reveals that the top-tier models achieve a maximum overall success rate of only 43.8%, highlighting that 0-to-1 software generation remains a highly challenging frontier. Furthermore, we discover that agents exhibit a strong tendency to generate monolithic code structures, and that higher token consumption does not necessarily yield better task performance.

Figures & tables

Fig. 1: Overview of the CLI-Tool-Bench framework.

Dimension	Sub-category	Count	Avg. LOC
Difficulty	Easy ( $\leq 1500$ LOC)	40	635.90
	Medium ( $1500-4000$ LOC)	21	2,687.57
	Hard ( $\geq 4000$ LOC)	33	18,509.97
Language	Python	34	4,473.50
	JavaScript	15	5,770.13
	Go	45	10,090.07

TABLE I: Statistical distribution of our benchmark.

Model	OpenHands						Mini-SWE-Agent						Average (Both Frameworks)
Model	Build	Exec	SP	EM	FM	SM	Build	Exec	SP	EM	FM	SM	Build	Exec	SP	EM	FM	SM
GPT-5.4	79.79	59.66	38.83	21.47	38.55	28.88	85.11	61.05	51.90	23.85	40.74	31.82	82.45	60.35	45.36	22.66	39.65	30.35
Claude-Sonnet-4.6	37.23	18.42	12.34	5.95	9.36	7.21	52.13	31.67	28.15	11.77	21.03	14.02	44.68	25.04	20.24	8.86	15.20	10.61
DeepSeek-V3.2	80.85	61.82	41.53	21.48	37.72	27.44	72.34	60.08	50.67	21.25	40.33	27.41	76.60	60.95	46.10	21.37	39.02	27.43
Qwen-3.5-plus	73.40	58.96	42.71	23.50	38.34	28.86	76.60	61.34	49.38	24.15	39.99	30.75	75.00	60.15	46.05	23.83	39.16	29.81
GLM-5	76.60	59.03	40.05	22.07	37.43	28.54	78.72	63.47	53.44	26.69	43.97	33.23	77.66	61.25	46.75	24.38	40.70	30.89

TABLE II: Overall performance of the evaluated LLMs across two agent frameworks. The rightmost columns and bottom row present macro-averages. Best results are highlighted in bold .

Fig. 2: Performance comparison of seven models across three programming languages. The radar charts illustrate the varying capabilities of each model, with the radial axes representing Semantic Match (

SM

) scores.

Fig. 3: Correlation between repository complexity and agent performance. Each scatter point represents an individual test case. The color of the points denotes the difficulty level: Green for Easy, Orange for Medium, and Red for Hard. The dashed black line represents the LOESS trend line.

Model	Framework	Steps			Prompt Tokens (K)			Comp. Tokens (K)			Cost ($)
Model	Framework	Avg	Min	Max	Avg	Min	Max	Avg	Min	Max	Avg	Min	Max
GPT-5.4	OpenHands	12.03	5	26	159.43	43.75	722.91	9.07	1.69	25.89	0.53	0.15	1.94
GPT-5.4	Mini-SWE-Agent	2.33	2	6	17.68	2.81	82.45	3.21	0.30	8.26	0.09	0.01	0.26
Claude-Sonnet-4.6	OpenHands	7.48	1	34	196.74	31.84	1,917.41	7.09	0.12	81.78	0.70	0.10	6.15
Claude-Sonnet-4.6	Mini-SWE-Agent	7.17	2	26	92.01	8.96	483.82	7.58	0.51	39.36	0.39	0.03	1.77
DeepSeek-V3.2	OpenHands	60.87	24	132	2,031.91	352.04	9,851.36	18.14	4.30	40.04	0.58	0.10	2.77

TABLE III: Generation overhead and estimated API cost per repository, categorized by LLM and agent framework.

Fig. 4: Cost-effectiveness analysis of LLM agents across different frameworks. The scatter plots illustrate the trade-off between task performance and execution cost. Small translucent dots represent individual task runs, while the large cross markers denote the centroid for each model. The dashed gray lines indicate the overall mean score and mean token consumption, dividing the space into four quadrants.

Fig. 5: Distribution of total files generated by the human Oracle and various LLMs. The box plots, combined with overlaid strip plots, show the median and quartile distributions while highlighting individual task runs as semi-transparent dots. The y-axis is truncated at 20 files. Extreme outliers are annotated with red stars and their maximum values at the top.

Fig. 6: Representative failure cases of LLM agents in CLI tool development.

Code Generation

Code generation studies methods for turning natural-language, visual, or formal specifications into executable programs and code edits.

See all 313 papers

Sep 29, 2026

E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch

Coding agents powered by large language models (LLMs) are evolving from making localized code changes to developing complete software repositories. However, evaluating repository-scale generation remains challenging: tasks must demand system-level reasoning while ensuring that all evaluated behaviors are precisely specified and independent of any particular implementation. We introduce E2E-SWE, a benchmark for evaluating whether coding agents can build complete, functional software repositories end to end. E2E-SWE contains 186 whole-repository generation tasks spanning 11 programming languages. Given only a natural-language specification and an empty workspace, an agent must implement a complete, installable project that satisfies a comprehensive suite of hidden tests. Each task is constructed by a software engineer in collaboration with an LLM; together, they develop the test suite and a corresponding implementation-independent specification. To ensure that tasks are well specified and practically solvable, we further subject them to an iterative verification process in which autonomous agents audit and repair task defects using static inspection and failures observed from real model rollouts. Evaluating 13 frontier models, we find substantial variation in end-to-end repository generation ability, with pass@1 ranging from 11.7% to 67.7%, providing strong model differentiation while leaving considerable headroom for future progress. Analysis of agent trajectories further reveals long, front-loaded reasoning patterns, highlighting the planning and system-level reasoning required to construct working codebases from scratch.

Hantian Ding, Chloe Bi, Jiacheng Zhu +5

Meta Superintelligence Labs

Software Engineering Agents Repository-Level Code Generation Code Generation Benchmarks

Ask PDF

Jul 13, 2026

BackendForge: Benchmarking Agentic End-to-End Code Generation with Backend Services

Large language models (LLMs) are increasingly used in agentic coding settings, where they can inspect files, execute commands, run tests, observe failures, and iteratively revise code. This shift raises a central evaluation question: can an agentic LLM generate an end-to-end software artifact that is both deployable and behaviorally correct under execution? Backend services provide a controlled but realistic substrate for this evaluation. Their APIs expose application-level executable semantics, and deployed behavior can be checked deterministically against an OpenAPI contract through black-box HTTP interactions. We introduce BackendForge, a benchmark of 56 contract-defined backend generation tasks rewritten from real open-source applications. Given a visible specification and an OpenAPI contract, an LLM must generate a Dockerized service that is built, deployed, and evaluated only through HTTP tests. To strengthen evaluation without introducing hidden requirements, BackendForge uses a test agent and a code agent to co-evolve the test oracle and reference service, where the test agent proposes specification-grounded backend tests and the code agent repairs the reference implementation. Although the best-performing model, GPT-5.5, succeeds on 55.4% of tasks under the base oracle, it succeeds on only 28.6% under the final oracle. This gap suggests that current LLMs can implement many local API behaviors, but still struggle to produce complete backend services.

Yuzhe Guo, Mengzhou Wu, Yuan Cao +4

1Key Lab of HCST (PKU), MOE; SCS, Peking University · 2Beijing Tongming Lake Information Technology Application Innovation Center (TLAIC) · 3Fudan University Institute of Systems for Advanced Computing +1

Tool-Augmented Language Model Agents Language Model Generation Evaluation AI Agent Benchmarks Agentic Code Generation Code Generation Benchmarks

Ask PDF

Apr 14, 2026

CodeSpecBench: Benchmarking LLMs for Executable Behavioral Specification Generation

Large language models (LLMs) can generate code from natural language, but the extent to which they capture intended program behavior remains unclear. Executable behavioral specifications, defined via preconditions and postconditions, provide a concrete means to assess such understanding. However, existing work on specification generation is constrained in evaluation methodology, task settings, and specification expressiveness. We introduce CodeSpecBench, a benchmark for executable behavioral specification generation under an execution-based evaluation protocol. CodeSpecBench supports both function-level and repository-level tasks and encodes specifications as executable Python functions. Constructed from diverse real-world codebases, it enables a realistic assessment of both correctness (accepting valid behaviors) and completeness (rejecting invalid behaviors). Evaluating 15 state-of-the-art LLMs on CodeSpecBench, we observe a sharp performance degradation on repository-level tasks, where the best model attains only a 20.2% pass rate. We further find that specification generation is substantially more challenging than code generation, indicating that strong coding performance does not necessarily reflect deep understanding of intended program semantics. Our data and code are available at https://github.com/SparksofAGI/CodeSpecBench.

Zaoyu Chen, Jianbo Dai, Boyu Zhu +6

The Hong Kong Polytechnic University · University of Edinburgh · UCL +5

LLM Evaluation Software Engineering Benchmarks

Ask PDF

Abstract

Code Generation

Explore similar work

E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch

BackendForge: Benchmarking Agentic End-to-End Code Generation with Backend Services

CodeSpecBench: Benchmarking LLMs for Executable Behavioral Specification Generation