IndustrialKnowEval: A Benchmark for Evaluating Multi-Level Industrial Knowledge in LLMs
Authors: Haiteng Wang, Haoze Li, Jingwei Guo, Jingbo Zhou, Si Liu, Lei Ren
Organizations: School of Artificial Intelligence, Beihang University, Beijing 100191, China · National College for Excellent Engineers, Beihang University, Beijing 100191, China · School of Automation Science and Electrical Engineering, Beihang University, Beijing 100191, China · Business Intelligence Lab, Baidu Research, Beijing 100085, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou 311115, China · State Key Laboratory of Intelligent Manufacturing System Technology, Beijing 100854, China
Large language models (LLMs) are increasingly explored for industrial tasks, yet few benchmarks evaluate their mastery of industrial knowledge across different types of cognitive demand. We introduce IndustrialKnowEval (IndKnow), a benchmark for industrial knowledge and engineering question answering. It contains 43,282 samples covering 21 task types across seven industrial sectors, including aerospace, electric power, and metallurgy, organized into four levels of cognitive demand: memory, comprehension, reasoning, and engineering application. Enterprise-based domain specialists and experienced researchers participate in all stages of dataset construction and quality control. We conduct a series of experiments on 18 representative LLMs, with domain experts deeply involved in the evaluation. Building upon IndKnow, we propose the Industrial Cognitive Capability Gap (ICCG) metric to characterize the performance differential between foundational industrial knowledge and higher-order reasoning and application, and we analyze it alongside absolute performance to provide a more complete view of model capabilities. We anticipate that IndustrialKnowEval will support the benchmarking of LLMs in industrial scenarios and promote the development of robust industrial intelligence. Code is available at: https://github.com/J4cksonLee/IndKnow
Figures & tables
Fig. 1 : Overview of the IndustrialKnowEval Framework. The framework constructs a four-level cognitive hierarchy (L1–L4) grounded in Bloom’s Taxonomy, mapping cognitive processes to types of industrial questions, from L1 Memory to L4 Engineering Application. The circular schema on the right illustrates the multi-dimensional evaluation matrix, applying this stratified standard across seven core industrial domains to enable a comprehensive assessment of model knowledge.
Benchmark
Size
Knowledge Source
Task Type
Ability
Ind. Domain
MCQ
OEQ
L1
L2
L3
L4
General-Purpose Benchmarks
MMLU [ 9 ]
15.9K
Web corpus
✓
✓
1 (Elec. Eng.)
MMLU-Pro [ 17 ]
12K
Web corpus
✓
✓
1 (Engineering)
C-Eval [ 10 ]
13.9K
Exam + Textbook
✓
✓
✓
3 (Engineering)
GPQA [ 18 ]
0.4K
Expert-curated
✓
✓
0
TABLE I : Comparison between IndustrialKnowEval and representative evaluation benchmarks.
Fig. 2 : An illustration of dataset construction pipeline in IndustrialKnowEval, including I) Raw Data Collecting and Preprocessing, II) Task Construction and III) Quality Control.
Level
Power
Metallurgy
Chemical
Rail Transit
Automotive
Petroleum
Aerospace
Total
L1
1754
1731
1771
700
1203
1846
1940
10945
L2
1732
1740
1765
708
1204
1835
1935
10919
L3
1754
1720
1779
708
1205
1828
1936
10930
L4
1632
1679
1686
691
1143
1738
1919
10488
total
6872
6870
7001
2807
4755
7247
7730
43282
TABLE II : Overview of the dataset.
Level
L1 (Single-Choice Questions)
L2 (Single-Choice Questions)
L3 (Multiple-Answer Questions)
L4 (Open-Ended Questions)
Task Types
Method Recognition
Principle Explanation
Influence Factor Analysis
Condition-Impact Reasoning
Data Extraction
Function Description
Method Analysis
System Optimization
Concept Recall
Structural Feature Description
Performance Comparison
Fault Diagnosis
Entity Recognition
Process Description
Process Flow Analysis
Root Cause Reasoning
Category Identification
Formula Comprehension
Phenomenon Interpretation
Consequence Prediction
-
-
Structure Function Analysis
-
TABLE III : Detailed Questions & Task Types
Model
Power
Metallurgy
Chemical
Rail Transit
Automotive
Petroleum
Aerospace
L1
L2
L3
L4
Mean
Macro
Rank
Deepseek-V3.1
0.7977
0.6591
0.7185
0.7127
0.7609
0.7126
0.7165
0.7481
0.8246
0.5701
0.7649
0.7265
0.7034
7
Deepseek-V4-Pro-0813
0.8201
0.6586
0.7183
0.7079
0.7646
0.7231
0.7128
0.7623
0.8439
0.5301
0.7797
0.7285
0.7014
6
Gemini-2.5-Pro
0.8181
0.6855
0.7295
0.7294
0.7784
0.7344
0.7425
0.7731
0.8361
0.6194
0.7671
0.7487
0.7250
3
GLM-4.5
0.7968
0.6480
0.7068
0.6898
0.7509
0.7022
0.6961
0.7444
0.8129
0.5602
0.7565
0.7181
0.6953
8
GLM-4.5-Air
0.7968
0.6421
0.7023
0.6897
0.7399
0.6997
0.6954
0.7386
0.8125
0.5463
0.7495
0.7113
0.6840
9
GLM-4
0.7499
0.5637
0.6244
0.6143
0.6706
0.6351
0.6100
0.6896
0.7755
0.4444
0.6572
0.6415
0.6087
14
TABLE IV : Overall zero-shot performance of LLMs across four levels in seven domains. Aerospace refers to the Chinese questions; the level columns, Mean, and Macro also include the English Aerospace questions. Mean is micro-averaged (each item has equal weight), Macro gives each of the 21 task types equal weight, and rank follows Mean. L4 scores are from the primary judge. Bold underlined indicates the best results among all models, bold indicates the second-best results, and underline indicates the third-best results. Statistical significance tests are reported in Appendix E .
Subset and grading
L1
L2
L3
L4
Single-choice
0.740
0.805
0.695
0.989, 0.974
Single-choice, L3 in multi-answer form
–
–
0.524
–
Open-ended, L4 rubric, primary judge
0.482
0.561
0.522
0.715
Open-ended, L4 rubric, GPT-6-sol
0.520
0.572
0.525
0.553
Open-ended, all-or-nothing, primary judge
0.219
0.140
0.059
0.180
Open-ended, all-or-nothing, GPT-6-sol
0.356
0.287
0.141
–
TABLE V : Mean scores on the controlled subsets (12 models on the single-choice subset and 10 on the open-ended subset). L4 is shown for GPT-5 and Qwen-Plus in the single-choice row and for Qwen-Plus in the all-or-nothing row.
Comparison
Spearman ρ
ICC
Quadratic κ
Mean diff.
Primary vs. GPT-6-sol
0.831
0.694
0.694
0.140
Primary vs. Qwen3.7-Plus
0.893
0.734
0.734
0.138
GPT-6-sol vs. Qwen3.7-Plus
0.891
0.888
0.888
− 0.002
TABLE VI : Agreement of L4 scores among the LLM judges over all L4 responses. Mean difference is the first judge minus the second.
Judge
ρ
ICC(A,1)
Mean diff.
Primary
0.756 [0.569, 0.869]
0.821 [0.734, 0.879]
0.088 [0.050, 0.127]
GPT-6-sol
0.907 [0.809, 0.956]
0.938 [0.891, 0.965]
− 0.058 [ − 0.082, − 0.035]
Qwen3.7-Plus
0.759 [0.568, 0.882]
0.899 [0.823, 0.943]
− 0.020 [ − 0.057, 0.016]
TABLE VII : L4 judges compared with the mean score of three experts on 50 responses, with 95% bootstrap confidence intervals. ρ is the Spearman correlation, and mean diff. is judge minus experts.
Fig. 3 : Zero-shot score and CoT gain for each of the 11 tested models, averaged over L3 and L4 on the Chinese questions of the seven domains. Lower-scoring models tend to obtain larger gains.
Fig. 4 : CoT gains in Power, Aerospace, and Aerospace (English) at L3 and L4.
Fig. 5 : Comparison of the zero-shot and CoT settings on L3 and L4 tasks for GPT-5 and Qwen3-0.6B.
Fig. 6 : Pass@k curves ( k =1 to 5) for GPT-5, Qwen-Plus, Qwen3-8B, and Llama-3.1-8B on L3 tasks in Chinese (CN) and English (EN).
Fig. 7 : ICCG against foundational accuracy A12=Acc(L1+L2) for the 18 evaluated models. Models with similar ICCG can differ widely in absolute performance.
Models
Power
Metallurgy
Chemical
Rail Transit
Automotive
Petroleum
Aerospace
Mean
A12
Acc(L3+L4)
DeepSeek-V3.1
0.1854
0.1163
0.1478
0.1404
0.1275
0.1524
0.1655
0.1479
0.7858
0.6688
DeepSeek-V4-Pro-0813
0.1578
0.2110
0.1759
0.1895
0.1548
0.1729
0.2255
0.1839
0.8026
0.6557
Gemini 2.5 Pro
0.1477
0.1375
0.1449
0.1217
0.1343
0.1569
0.1157
0.1370
0.8035
0.6934
GLM-4.5
0.1605
0.1335
0.1534
0.1521
0.1331
0.1676
0.1675
0.1525
0.7769
0.6581
GLM-4.5-Air
0.1572
0.1659
0.1713
0.1540
0.1626
0.1654
0.1684
0.1636
0.7747
0.6480
GLM-4
0.1871
0.2893
0.2640
0.2636
0.2415
0.2663
0.2442
0.2509
0.7321
0.5498
TABLE VIII : ICCG of the models across domains, with absolute performance. ICCG =1−r is computed per domain, and Aerospace includes its Chinese and English questions. Mean, A12 , and Acc(L3+L4) are averaged over the seven domains.
TABLE IX : Overview of the IndustrialKnowEval Dataset Across Seven Industrial Domains and Four Cognitive Levels. SCQ means Single-Choice Question, MCQ means Multiple-Choice Question, and QA means Open-ended Question.
Domain
Documents
Questions
Q/Doc
Questions by Level
Top-10 (%)
Count
Share (%)
Count
Share (%)
L1
L2
L3
L4
Aerospace
2,000
16.5
7,730
17.9
3.87
1,940
1,935
1,936
1,919
0.58
Power
2,025
16.7
6,872
15.9
3.39
1,754
1,732
1,754
1,632
0.58
Metallurgy
2,008
16.5
6,870
15.9
3.42
1,731
1,740
1,720
1,679
0.60
Chemical
2,012
16.6
7,001
16.2
3.48
1,771
1,765
1,779
1,686
0.59
Automotive
1,327
10.9
4,755
11.0
3.58
1,203
1,204
1,205
1,143
0.84
Appendix
TABLE X : Source Documents and Questions of IndustrialKnowEval by Domain. Q/Doc is the mean number of retained questions per source document. Top-10 is the share of a domain’s questions derived from its ten most frequently used documents.
ID
Model
Creator
#Parameters
API Version
Access
Reasoning
M1
GPT-5
OpenAI
undisclosed
openai/gpt-5
OpenRouter
Minimal effort
M2
GPT-4.1
OpenAI
undisclosed
openai/gpt-4.1
OpenRouter
–
M3
Gemini 2.5 Pro
Google
undisclosed
google/gemini-2.5-pro
OpenRouter
Lowest effort
M4
DeepSeek-V3.1
DeepSeek AI
671B
deepseek-chat
DeepSeek API
Thinking off
M5
GLM-4.5
ZhipuAI
undisclosed
glm-4.5
Zhipu BigModel
Thinking off
M6
GLM-4.5-Air
ZhipuAI
undisclosed
glm-4.5-air
Zhipu BigModel
–
Appendix
TABLE XI : The 18 LLMs evaluated in our experiments. Reasoning is the setting in the zero-shot evaluation; “–” means that the model has no adjustable reasoning setting.
Attribute
Non-CDFR Example
CDFR Example
Question
What is the primary function of a transformer in a power transmission system?
According to the experiment described in the paper, what was the measured efficiency of the prototype transformer under 85% load?
Triggered Criterion
None
Criterion 1 (Specific Data Dependency) and Criterion 3 (Implicit Context Dependency)
Classification
Non-CDFR (retained)
CDFR (excluded)
Rationale
Answerable from general electrical engineering knowledge without reference to any specific document.
Requires access to a specific numerical measurement (efficiency value) from a particular prototype experiment described only in the source document.
Appendix
Table 25
Status Value
Stage
Definition
auto_pass
Automated Verification
The sample successfully passed automated verification.
manual_pass
Manual Review
The logical chain of the sample and the evidentiary support were confirmed as reasonable during manual review.
auto_fail / manual_fail
Automated Verification / Manual Review
The sample failed verification at either the automated stage ( auto_fail ) or the manual review stage ( manual_fail ).
revised
Post-Review Revision
The sample was initially assigned auto_fail or manual_fail status, subsequently underwent targeted revision by the research team or domain experts, and successfully passed re-verification.
Appendix
Table 26
Stage
Removed
Retained
Initial generation (12,151 per level)
–
48,604
Automated verification
3,100
45,504
Manual review (including 745 CDFR exclusions)
2,222
43,282
Appendix
TABLE XII: Questions removed and retained at each filtering stage.
Model
Acc.
Δ Acc.
95% CI
pdoc
pfrag
GPT-6-sol
0.8509
− 0.1051*
[ − 0.110, − 0.101]
< 0.001
< 0.001
Gemini 2.5 Pro
0.7428
+0.0029
[ − 0.002, 0.007]
0.207
0.213
Qwen-Plus
0.7402
+0.0056
[0.001, 0.010]
0.011
0.010
GPT-4.1
0.7278
+0.0180*
[0.014, 0.022]
< 0.001
< 0.001
DeepSeek-V3.1
0.7142
+0.0315*
[0.027, 0.036]
< 0.001
< 0.001
DeepSeek-V4-Pro
0.7121
+0.0337*
[0.029, 0.038]
< 0.001
< 0.001
Appendix
TABLE XIII : Paired cluster bootstrap comparison of GPT-5 (accuracy 0.7458) with the other models on L1–L3 ( n=32,794 ). Δ Acc. is GPT-5 minus the model. The 95% CI and pdoc use document-level resampling, and pfrag uses fragment-level resampling.
Model
L1
L2
L3
L4
Overall
GPT-6-sol
0.905
0.887
0.706
0.765
0.811
GPT-5
0.785
0.801
0.585
0.847
0.746
Gemini 2.5 Pro
0.781
0.772
0.600
0.772
0.725
Qwen-Plus
0.772
0.797
0.581
0.739
0.715
GPT-4.1
0.758
0.774
0.584
0.768
0.714
DeepSeek-V3.1
0.753
0.767
0.551
0.773
0.703
Appendix
TABLE XIV : Macro-averaged scores by cognitive level. Overall is the mean over the 21 task types.
Model
Aero.
Power
Metall.
Chemical
Auto.
Rail
Petrol.
GPT-6-sol
0.841
0.832
0.773
0.806
0.822
0.811
0.793
GPT-5
0.781
0.786
0.696
0.740
0.765
0.725
0.727
Gemini 2.5 Pro
0.743
0.763
0.683
0.713
0.735
0.715
0.710
Qwen-Plus
0.743
0.765
0.665
0.702
0.726
0.718
0.700
GPT-4.1
0.735
0.757
0.665
0.714
0.733
0.682
0.687
DeepSeek-V3.1
0.721
0.752
0.650
0.701
0.729
0.700
0.696
Appendix
TABLE XV : Macro-averaged scores by domain.
Family
Models
L1
L2
L3
Under
Over
Other
GPT
3
0.810
0.871
0.644
15.3%
69.5%
15.2%
Gemini
1
0.773
0.836
0.619
33.0%
44.6%
22.4%
DeepSeek
2
0.755
0.834
0.550
51.1%
28.6%
20.3%
GLM
4
0.713
0.791
0.519
15.6%
61.8%
22.6%
Llama
2
0.591
0.661
0.443
32.5%
37.1%
29.8%
Qwen
6
0.654
0.735
0.419
16.0%
54.0%
29.9%
Appendix
TABLE XVI : Accuracy by level and L3 error pattern by model family, pooled over the member models. Under, Over, and Other are shares of the L3 errors.
Model
A12
H
ICCG
Δ
ICCGL3
G6
Q3.7
GPT-5
0.815
0.722
0.113
0.093
0.262
0.238
0.200
GPT-4.1
0.788
0.680
0.137
0.108
0.236
0.234
0.219
Gemini 2.5 Pro
0.804
0.693
0.137
0.110
0.229
0.257
0.241
DeepSeek-V3.1
0.786
0.669
0.148
0.117
0.271
0.260
0.253
GLM-4.5
0.777
0.658
0.153
0.119
0.280
0.262
0.249
GLM-4.5-Air
0.775
0.648
0.164
0.127
0.297
0.283
0.269
Appendix
TABLE XVII : ICCG and its variants for each model, ordered by ICCG. H is Acc(L3+L4) . G6 and Q3.7 denote ICCG with L4 scored by GPT-6-sol and Qwen3.7-Plus.
Gap Metric
ρ
Smallest
Largest
Revised ICCG, 1−r
1.00
GPT-5
Llama-3.1-70B
Absolute difference Δ
0.779
GPT-5
Llama-3.1-70B
ICCGL3 (without L4)
0.692
Llama-3.1-8B
Qwen3-0.6B
ICCG, L4 scored by GPT-6-sol
0.880
GPT-6-sol
Qwen3-0.6B
ICCG, L4 scored by Qwen3.7-Plus
0.897
GPT-5
Qwen3-0.6B
Appendix
TABLE XVIII : Robustness of model rankings to the gap metric, based on the mean across domains. ρ is the Spearman correlation with the revised ICCG; Smallest and Largest name the models with the smallest and largest gaps.
Model
Qwen
Claude
Difference [95% CI]
GPT-6-sol
0.890
0.843
+0.047 [ − 0.010, +0.103]
Qwen-Plus
0.787
0.800
− 0.013 [ − 0.080, +0.057]
Gemini 2.5 Pro
0.777
0.783
− 0.007 [ − 0.067, +0.053]
GPT-4.1
0.773
0.773
0.000 [ − 0.067, +0.067]
DeepSeek-V4-Pro
0.760
0.783
− 0.023 [ − 0.090, +0.043]
DeepSeek-V3.1
0.760
0.717
+0.043 [ − 0.027, +0.113]
Appendix
TABLE XIX : Accuracy on the same 100 source texts (300 L1–L3 items) with Qwen-Plus and Claude Opus 5.5 as generators. The difference is Qwen minus Claude, with a 95% bootstrap confidence interval.
Code generation and comprehension by Large Language Models (LLMs) have emerged as core drivers of industrial intelligence and decision optimization, finding widespread application in fields such as finance, automation, and aerospace. Although recent advancements have demonstrated the remarkable potential of LLMs in general code generation, existing benchmarks are mainly confined to single domains and languages. Consequently, they fail to effectively evaluate the generalization capabilities required for real-world industrial applications or to reflect the coding proficiency demanded by complex industrial scenarios. To bridge this gap, we introduce IndustryCode, the first comprehensive benchmark designed to span multiple industrial domains and programming languages. IndustryCode comprises 579 sub-problems derived from 125 primary industrial challenges, accompanied by rigorous problem descriptions and test cases. It covers a wide range of fields, including finance, automation, aerospace, and remote sensing-and incorporates diverse programming languages such as MATLAB, Python, C++, and Stata. In our evaluation, the top-performing model, Claude 4.5 Opus, achieved an overall accuracy of 68.1% on sub-problems and 42.5% main problems. The benchmark dataset and automated evaluation code will be made publicly available upon acceptance.
Puyu Zeng, Zhaoxi Wang, Zhixu Duan +7
Shanghai Jiao Tong University, Shanghai, China · Alibaba Group, Hangzhou, China · Tsinghua University, Beijing, China
In industrial procurement, an LLM answer is useful only if it survives a standards check: recommended material must match operating condition, every parameter must respect a regulated threshold, and no procedure may contradict a safety clause. Partial correctness can mask safety-critical contradictions that aggregate LLM benchmarks rarely capture. We introduce IndustryBench, a 2,049-item benchmark for industrial procurement QA in Chinese, grounded in Chinese national standards (GB/T) and structured industrial product records, organized by seven capability dimensions, ten industry categories, and panel-derived difficulty tiers, with item-aligned English, Russian, and Vietnamese renderings. Our construction pipeline rejects 70.3% of LLM-generated candidates at a search-based external-verification stage, calibrating how unreliable industrial QA remains after LLM-only filtering. Our evaluation decouples raw correctness, scored by a Qwen3-Max judge validated at κw=0.798 against a domain expert, from a separate safety-violation (SV) check against source texts. Across 17 models in Chinese and an 8-model intersection over four languages, we find: (i) the best system reaches only 2.083 on the 0--3 rubric, leaving substantial headroom; (ii) Standards & Terminology is the most persistent capability weakness and survives item-aligned translation; (iii) extended reasoning lowers safety-adjusted scores for 12 of 13 models, primarily by introducing unsupported safety-critical details into longer final answers; and (iv) safety-violation rates reshuffle the leaderboard -- GPT-5.4 climbs from rank 6 to rank 3 after SV adjustment, while Kimi-k2.5-1T-A32B drops seven positions. Industrial LLM evaluation therefore requires source-grounded, safety-aware diagnosis rather than aggregate accuracy. We release IndustryBench with all prompts, scoring scripts, and dataset documentation.
Songlin Bai, Xintong Wang, Linlin Yu +12
Multimodal and Industrial AI Team† Taobao&Tmall, Alibaba Group
We introduce FailureSensorIQ, a novel Multi-Choice Question-Answering (MCQA) benchmarking system designed to assess the ability of Large Language Models (LLMs) to reason and understand complex, domain-specific scenarios in Industry 4.0. Unlike traditional QA benchmarks, our system focuses on multiple aspects of reasoning through failure modes, sensor data, and the relationships between them across various industrial assets. Through this work, we envision a paradigm shift where modeling decisions are not only data-driven using statistical tools like correlation analysis and significance tests, but also domain-driven by specialized LLMs which can reason about the key contributors and useful patterns that can be captured with feature engineering. We evaluate the Industrial knowledge of over a dozen LLMs-including GPT-4, Llama, and Mistral-on FailureSensorIQ from different lens using Perturbation-Uncertainty-Complexity analysis, Expert Evaluation study, Asset-Specific Knowledge Gap analysis, ReAct agent using external knowledge-bases. Even though closed-source models with strong reasoning capabilities approach expert-level performance, the comprehensive benchmark reveals a significant drop in performance that is fragile to perturbations, distractions, and inherent knowledge gaps in the models. We also provide a real-world case study of how LLMs can drive the modeling decisions on 3 different failure prediction datasets related to various assets. We release: (a) expert-curated MCQA for various industrial assets, (b) FailureSensorIQ benchmark and Hugging Face leaderboard based on MCQA built from non-textual data found in ISO documents, and (c) LLMFeatureSelector, an LLM-based feature selection scikit-learn pipeline. The software is available at https://github.com/IBM/FailureSensorIQ.
Christodoulos Constantinides, Dhaval Patel, Shuxin Lin +3