IndustryCode: A Benchmark for Industry Code Generation
Organizations: Shanghai Jiao Tong University, Shanghai, China · Alibaba Group, Hangzhou, China · Tsinghua University, Beijing, China
Abstract
Code generation and comprehension by Large Language Models (LLMs) have emerged as core drivers of industrial intelligence and decision optimization, finding widespread application in fields such as finance, automation, and aerospace. Although recent advancements have demonstrated the remarkable potential of LLMs in general code generation, existing benchmarks are mainly confined to single domains and languages. Consequently, they fail to effectively evaluate the generalization capabilities required for real-world industrial applications or to reflect the coding proficiency demanded by complex industrial scenarios. To bridge this gap, we introduce IndustryCode, the first comprehensive benchmark designed to span multiple industrial domains and programming languages. IndustryCode comprises 579 sub-problems derived from 125 primary industrial challenges, accompanied by rigorous problem descriptions and test cases. It covers a wide range of fields, including finance, automation, aerospace, and remote sensing-and incorporates diverse programming languages such as MATLAB, Python, C++, and Stata. In our evaluation, the top-performing model, Claude 4.5 Opus, achieved an overall accuracy of 68.1% on sub-problems and 42.5% main problems. The benchmark dataset and automated evaluation code will be made publicly available upon acceptance.
Figures & tables
| Programming Language | Subfields |
|---|---|
| Python | Chemical Engineering (14), Advanced Manufacturing (12), Finance and Business (11), Automation and Control Engineering (9), Artificial Intelligence (9), Electronics and Communication (5), Biomedical Engineering (4), Information Technology (4), Transportation and Logistics (3) |
| C++ | Construction (10), Information Technology (7), Mining (6), Machinery Manufacturing (5), Semiconductors and Microelectronics (4), Transportation and Logistics (4) |
| Matlab | Operations Research (7), Genetic Computation (7), Aerospace Engineering (2), Naval Architecture and Ocean Engineering (1) |
| Stata | Finance and Business (3) |
| Python | C++ | Matlab | Stata | Overall | ||||||
| Models | Sub | Main | Sub | Main | Sub | Main | Sub | Main | Sub | Main |
| Proprietary Models | ||||||||||
| Claude 4.5 Opus | 61.8 | 36.5 | 70.9 | 52.0 | 72.8 | 31.3 | 66.7 | 50.0 | 68.1 | 42.5 |
| Claude 4.5 Sonnet | 58.6 | 36.5 | 58.0 | 36.0 | 74.3 | 37.5 | 66.7 | 50.0 | 64.4 | 33.8 |
| Claude 4.5 haiku | 53.4 | 28.6 | 61.7 | 36.0 | 64.3 | 12.5 | 50.0 | 50.0 | 57.4 | 31.8 |
| GPT-5.2 | 48.5 | 27.0 | 67.9 | 40.0 | 47.1 | 12.5 | 50.0 | 50.0 | 53.4 | 32.4 |
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Industry Domain | Leading Model | Score (%) | Primary Performance Factor |
|---|---|---|---|
| Information Technology | Qwen3-Max | 86.5 | MoE Expert Routing |
| Transportation & Logistics | Claude 4.5 Sonnet | 85.0 | Contextual Understanding |
| Machinery Manufacturing | Qwen3-Max | 83.3 | Pattern Matching (G-code/PLC) |
| Aerospace Engineering | GPT-5.2 | 82.4 | Technical Knowledge Retrieval |
| Artificial Intelligence | Claude 4.5 Sonnet | 81.2 | Algorithm Implementation |
| Mining | GPT-5.2 | 80.6 | Specification Adherence |
| Model Mode | Context Confusion | Problem Misinterp. | Syntax Error | Hallucination | Truncation | Reasoning Failure |
|---|---|---|---|---|---|---|
| No-Thinking | 1.9% | 15.0% | 4.4% | 8.7% | 1.2% | 11.8% |
| Thinking | 29.2% | 30.4% | 9.4% | 12.4% | 1.9% | 0.8% |
Code Generation
Code generation studies methods for turning natural-language, visual, or formal specifications into executable programs and code edits.
See all 304 papers