IndustrialKnowEval: A Benchmark for Evaluating Multi-Level Industrial Knowledge in LLMs
Organizations: School of Artificial Intelligence, Beihang University, Beijing 100191, China · National College for Excellent Engineers, Beihang University, Beijing 100191, China · School of Automation Science and Electrical Engineering, Beihang University, Beijing 100191, China · Business Intelligence Lab, Baidu Research, Beijing 100085, China ·
Abstract
Large language models (LLMs) are increasingly explored for industrial tasks, yet few benchmarks evaluate their mastery of industrial knowledge across different types of cognitive demand. We introduce IndustrialKnowEval (IndKnow), a benchmark for industrial knowledge and engineering question answering. It contains 43,282 samples covering 21 task types across seven industrial sectors, including aerospace, electric power, and metallurgy, organized into four levels of cognitive demand: memory, comprehension, reasoning, and engineering application. Enterprise-based domain specialists and experienced researchers participate in all stages of dataset construction and quality control. We conduct a series of experiments on 18 representative LLMs, with domain experts deeply involved in the evaluation. Building upon IndKnow, we propose the Industrial Cognitive Capability Gap (ICCG) metric to characterize the performance differential between foundational industrial knowledge and higher-order reasoning and application, and we analyze it alongside absolute performance to provide a more complete view of model capabilities. We anticipate that IndustrialKnowEval will support the benchmarking of LLMs in industrial scenarios and promote the development of robust industrial intelligence. Code is available at: https://github.com/J4cksonLee/IndKnow
Figures & tables
| Benchmark | Size | Knowledge Source | Task Type | Ability | Ind. Domain | ||||
| MCQ | OEQ | L1 | L2 | L3 | L4 | ||||
| General-Purpose Benchmarks | |||||||||
| MMLU [ 9 ] | 15.9K | Web corpus | 1 (Elec. Eng.) | ||||||
| MMLU-Pro [ 17 ] | 12K | Web corpus | 1 (Engineering) | ||||||
| C-Eval [ 10 ] | 13.9K | Exam + Textbook | 3 (Engineering) | ||||||
| GPQA [ 18 ] | 0.4K | Expert-curated | 0 | ||||||
| Level | Power | Metallurgy | Chemical | Rail Transit | Automotive | Petroleum | Aerospace | Total |
| L1 | 1754 | 1731 | 1771 | 700 | 1203 | 1846 | 1940 | 10945 |
| L2 | 1732 | 1740 | 1765 | 708 | 1204 | 1835 | 1935 | 10919 |
| L3 | 1754 | 1720 | 1779 | 708 | 1205 | 1828 | 1936 | 10930 |
| L4 | 1632 | 1679 | 1686 | 691 | 1143 | 1738 | 1919 | 10488 |
| total | 6872 | 6870 | 7001 | 2807 | 4755 | 7247 | 7730 | 43282 |
| Level | L1 (Single-Choice Questions) | L2 (Single-Choice Questions) | L3 (Multiple-Answer Questions) | L4 (Open-Ended Questions) |
| Task Types | Method Recognition | Principle Explanation | Influence Factor Analysis | Condition-Impact Reasoning |
| Data Extraction | Function Description | Method Analysis | System Optimization | |
| Concept Recall | Structural Feature Description | Performance Comparison | Fault Diagnosis | |
| Entity Recognition | Process Description | Process Flow Analysis | Root Cause Reasoning | |
| Category Identification | Formula Comprehension | Phenomenon Interpretation | Consequence Prediction | |
| - | - | Structure Function Analysis | - |
| Model | Power | Metallurgy | Chemical | Rail Transit | Automotive | Petroleum | Aerospace | L1 | L2 | L3 | L4 | Mean | Macro | Rank |
| Deepseek-V3.1 | 0.7977 | 0.6591 | 0.7185 | 0.7127 | 0.7609 | 0.7126 | 0.7165 | 0.7481 | 0.8246 | 0.5701 | 0.7649 | 0.7265 | 0.7034 | 7 |
| Deepseek-V4-Pro-0813 | 0.8201 | 0.6586 | 0.7183 | 0.7079 | 0.7646 | 0.7231 | 0.7128 | 0.7623 | 0.8439 | 0.5301 | 0.7797 | 0.7285 | 0.7014 | 6 |
| Gemini-2.5-Pro | 0.8181 | 0.6855 | 0.7295 | 0.7294 | 0.7784 | 0.7344 | 0.7425 | 0.7731 | 0.8361 | 0.6194 | 0.7671 | 0.7487 | 0.7250 | 3 |
| GLM-4.5 | 0.7968 | 0.6480 | 0.7068 | 0.6898 | 0.7509 | 0.7022 | 0.6961 | 0.7444 | 0.8129 | 0.5602 | 0.7565 | 0.7181 | 0.6953 | 8 |
| GLM-4.5-Air | 0.7968 | 0.6421 | 0.7023 | 0.6897 | 0.7399 | 0.6997 | 0.6954 | 0.7386 | 0.8125 | 0.5463 | 0.7495 | 0.7113 | 0.6840 | 9 |
| GLM-4 | 0.7499 | 0.5637 | 0.6244 | 0.6143 | 0.6706 | 0.6351 | 0.6100 | 0.6896 | 0.7755 | 0.4444 | 0.6572 | 0.6415 | 0.6087 | 14 |
| Subset and grading | L1 | L2 | L3 | L4 |
| Single-choice | 0.740 | 0.805 | 0.695 | 0.989, 0.974 |
| Single-choice, L3 in multi-answer form | – | – | 0.524 | – |
| Open-ended, L4 rubric, primary judge | 0.482 | 0.561 | 0.522 | 0.715 |
| Open-ended, L4 rubric, GPT-6-sol | 0.520 | 0.572 | 0.525 | 0.553 |
| Open-ended, all-or-nothing, primary judge | 0.219 | 0.140 | 0.059 | 0.180 |
| Open-ended, all-or-nothing, GPT-6-sol | 0.356 | 0.287 | 0.141 | – |
| Comparison | Spearman | ICC | Quadratic | Mean diff. |
| Primary vs. GPT-6-sol | 0.831 | 0.694 | 0.694 | 0.140 |
| Primary vs. Qwen3.7-Plus | 0.893 | 0.734 | 0.734 | 0.138 |
| GPT-6-sol vs. Qwen3.7-Plus | 0.891 | 0.888 | 0.888 | 0.002 |
| Judge | ICC(A,1) | Mean diff. | |
| Primary | 0.756 [0.569, 0.869] | 0.821 [0.734, 0.879] | 0.088 [0.050, 0.127] |
| GPT-6-sol | 0.907 [0.809, 0.956] | 0.938 [0.891, 0.965] | 0.058 [ 0.082, 0.035] |
| Qwen3.7-Plus | 0.759 [0.568, 0.882] | 0.899 [0.823, 0.943] | 0.020 [ 0.057, 0.016] |
| Models | Power | Metallurgy | Chemical | Rail Transit | Automotive | Petroleum | Aerospace | Mean | ||
| DeepSeek-V3.1 | 0.1854 | 0.1163 | 0.1478 | 0.1404 | 0.1275 | 0.1524 | 0.1655 | 0.1479 | 0.7858 | 0.6688 |
| DeepSeek-V4-Pro-0813 | 0.1578 | 0.2110 | 0.1759 | 0.1895 | 0.1548 | 0.1729 | 0.2255 | 0.1839 | 0.8026 | 0.6557 |
| Gemini 2.5 Pro | 0.1477 | 0.1375 | 0.1449 | 0.1217 | 0.1343 | 0.1569 | 0.1157 | 0.1370 | 0.8035 | 0.6934 |
| GLM-4.5 | 0.1605 | 0.1335 | 0.1534 | 0.1521 | 0.1331 | 0.1676 | 0.1675 | 0.1525 | 0.7769 | 0.6581 |
| GLM-4.5-Air | 0.1572 | 0.1659 | 0.1713 | 0.1540 | 0.1626 | 0.1654 | 0.1684 | 0.1636 | 0.7747 | 0.6480 |
| GLM-4 | 0.1871 | 0.2893 | 0.2640 | 0.2636 | 0.2415 | 0.2663 | 0.2442 | 0.2509 | 0.7321 | 0.5498 |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Level | Task Type | Aerospace | Power | Metallurgy | Chemical | Automotive | Rail Transit | Petroleum | Total |
| L1 SCQ | Method Recognition | 406 | 636 | 409 | 448 | 419 | 240 | 499 | 10945 |
| Data Extraction | 271 | 137 | 751 | 579 | 209 | 203 | 553 | ||
| Concept Recall | 501 | 598 | 199 | 274 | 253 | 127 | 301 | ||
| Entity Recognition | 60 | 33 | 87 | 112 | 55 | 97 | 97 | ||
| Category Identification | 702 | 350 | 285 | 358 | 267 | 33 | 396 | ||
| L2 SCQ | Principle Explanation | 815 | 756 | 959 | 857 | 466 | 368 | 936 | 10919 |
| Domain | Documents | Questions | Q/Doc | Questions by Level | Top-10 (%) | |||||
| Count | Share (%) | Count | Share (%) | L1 | L2 | L3 | L4 | |||
| Aerospace | 2,000 | 16.5 | 7,730 | 17.9 | 3.87 | 1,940 | 1,935 | 1,936 | 1,919 | 0.58 |
| Power | 2,025 | 16.7 | 6,872 | 15.9 | 3.39 | 1,754 | 1,732 | 1,754 | 1,632 | 0.58 |
| Metallurgy | 2,008 | 16.5 | 6,870 | 15.9 | 3.42 | 1,731 | 1,740 | 1,720 | 1,679 | 0.60 |
| Chemical | 2,012 | 16.6 | 7,001 | 16.2 | 3.48 | 1,771 | 1,765 | 1,779 | 1,686 | 0.59 |
| Automotive | 1,327 | 10.9 | 4,755 | 11.0 | 3.58 | 1,203 | 1,204 | 1,205 | 1,143 | 0.84 |
| ID | Model | Creator | #Parameters | API Version | Access | Reasoning |
| M1 | GPT-5 | OpenAI | undisclosed | openai/gpt-5 | OpenRouter | Minimal effort |
| M2 | GPT-4.1 | OpenAI | undisclosed | openai/gpt-4.1 | OpenRouter | – |
| M3 | Gemini 2.5 Pro | undisclosed | google/gemini-2.5-pro | OpenRouter | Lowest effort | |
| M4 | DeepSeek-V3.1 | DeepSeek AI | 671B | deepseek-chat | DeepSeek API | Thinking off |
| M5 | GLM-4.5 | ZhipuAI | undisclosed | glm-4.5 | Zhipu BigModel | Thinking off |
| M6 | GLM-4.5-Air | ZhipuAI | undisclosed | glm-4.5-air | Zhipu BigModel | – |
| Attribute | Non-CDFR Example | CDFR Example |
| Question | What is the primary function of a transformer in a power transmission system? | According to the experiment described in the paper, what was the measured efficiency of the prototype transformer under 85% load? |
| Triggered Criterion | None | Criterion 1 (Specific Data Dependency) and Criterion 3 (Implicit Context Dependency) |
| Classification | Non-CDFR (retained) | CDFR (excluded) |
| Rationale | Answerable from general electrical engineering knowledge without reference to any specific document. | Requires access to a specific numerical measurement (efficiency value) from a particular prototype experiment described only in the source document. |
| Status Value | Stage | Definition |
| auto_pass | Automated Verification | The sample successfully passed automated verification. |
| manual_pass | Manual Review | The logical chain of the sample and the evidentiary support were confirmed as reasonable during manual review. |
| auto_fail / manual_fail | Automated Verification / Manual Review | The sample failed verification at either the automated stage ( auto_fail ) or the manual review stage ( manual_fail ). |
| revised | Post-Review Revision | The sample was initially assigned auto_fail or manual_fail status, subsequently underwent targeted revision by the research team or domain experts, and successfully passed re-verification. |
| Stage | Removed | Retained |
| Initial generation (12,151 per level) | – | 48,604 |
| Automated verification | 3,100 | 45,504 |
| Manual review (including 745 CDFR exclusions) | 2,222 | 43,282 |
| Model | Acc. | Acc. | 95% CI | ||
| GPT-6-sol | 0.8509 | 0.1051* | [ 0.110, 0.101] | 0.001 | 0.001 |
| Gemini 2.5 Pro | 0.7428 | +0.0029 | [ 0.002, 0.007] | 0.207 | 0.213 |
| Qwen-Plus | 0.7402 | +0.0056 | [0.001, 0.010] | 0.011 | 0.010 |
| GPT-4.1 | 0.7278 | +0.0180* | [0.014, 0.022] | 0.001 | 0.001 |
| DeepSeek-V3.1 | 0.7142 | +0.0315* | [0.027, 0.036] | 0.001 | 0.001 |
| DeepSeek-V4-Pro | 0.7121 | +0.0337* | [0.029, 0.038] | 0.001 | 0.001 |
| Model | L1 | L2 | L3 | L4 | Overall |
| GPT-6-sol | 0.905 | 0.887 | 0.706 | 0.765 | 0.811 |
| GPT-5 | 0.785 | 0.801 | 0.585 | 0.847 | 0.746 |
| Gemini 2.5 Pro | 0.781 | 0.772 | 0.600 | 0.772 | 0.725 |
| Qwen-Plus | 0.772 | 0.797 | 0.581 | 0.739 | 0.715 |
| GPT-4.1 | 0.758 | 0.774 | 0.584 | 0.768 | 0.714 |
| DeepSeek-V3.1 | 0.753 | 0.767 | 0.551 | 0.773 | 0.703 |
| Model | Aero. | Power | Metall. | Chemical | Auto. | Rail | Petrol. |
| GPT-6-sol | 0.841 | 0.832 | 0.773 | 0.806 | 0.822 | 0.811 | 0.793 |
| GPT-5 | 0.781 | 0.786 | 0.696 | 0.740 | 0.765 | 0.725 | 0.727 |
| Gemini 2.5 Pro | 0.743 | 0.763 | 0.683 | 0.713 | 0.735 | 0.715 | 0.710 |
| Qwen-Plus | 0.743 | 0.765 | 0.665 | 0.702 | 0.726 | 0.718 | 0.700 |
| GPT-4.1 | 0.735 | 0.757 | 0.665 | 0.714 | 0.733 | 0.682 | 0.687 |
| DeepSeek-V3.1 | 0.721 | 0.752 | 0.650 | 0.701 | 0.729 | 0.700 | 0.696 |
| Family | Models | L1 | L2 | L3 | Under | Over | Other |
| GPT | 3 | 0.810 | 0.871 | 0.644 | 15.3% | 69.5% | 15.2% |
| Gemini | 1 | 0.773 | 0.836 | 0.619 | 33.0% | 44.6% | 22.4% |
| DeepSeek | 2 | 0.755 | 0.834 | 0.550 | 51.1% | 28.6% | 20.3% |
| GLM | 4 | 0.713 | 0.791 | 0.519 | 15.6% | 61.8% | 22.6% |
| Llama | 2 | 0.591 | 0.661 | 0.443 | 32.5% | 37.1% | 29.8% |
| Qwen | 6 | 0.654 | 0.735 | 0.419 | 16.0% | 54.0% | 29.9% |
| Model | ICCG | G6 | Q3.7 | ||||
| GPT-5 | 0.815 | 0.722 | 0.113 | 0.093 | 0.262 | 0.238 | 0.200 |
| GPT-4.1 | 0.788 | 0.680 | 0.137 | 0.108 | 0.236 | 0.234 | 0.219 |
| Gemini 2.5 Pro | 0.804 | 0.693 | 0.137 | 0.110 | 0.229 | 0.257 | 0.241 |
| DeepSeek-V3.1 | 0.786 | 0.669 | 0.148 | 0.117 | 0.271 | 0.260 | 0.253 |
| GLM-4.5 | 0.777 | 0.658 | 0.153 | 0.119 | 0.280 | 0.262 | 0.249 |
| GLM-4.5-Air | 0.775 | 0.648 | 0.164 | 0.127 | 0.297 | 0.283 | 0.269 |
| Gap Metric | Smallest | Largest | |
| Revised ICCG, | 1.00 | GPT-5 | Llama-3.1-70B |
| Absolute difference | 0.779 | GPT-5 | Llama-3.1-70B |
| (without L4) | 0.692 | Llama-3.1-8B | Qwen3-0.6B |
| ICCG, L4 scored by GPT-6-sol | 0.880 | GPT-6-sol | Qwen3-0.6B |
| ICCG, L4 scored by Qwen3.7-Plus | 0.897 | GPT-5 | Qwen3-0.6B |
| Model | Qwen | Claude | Difference [95% CI] |
| GPT-6-sol | 0.890 | 0.843 | +0.047 [ 0.010, +0.103] |
| Qwen-Plus | 0.787 | 0.800 | 0.013 [ 0.080, +0.057] |
| Gemini 2.5 Pro | 0.777 | 0.783 | 0.007 [ 0.067, +0.053] |
| GPT-4.1 | 0.773 | 0.773 | 0.000 [ 0.067, +0.067] |
| DeepSeek-V4-Pro | 0.760 | 0.783 | 0.023 [ 0.090, +0.043] |
| DeepSeek-V3.1 | 0.760 | 0.717 | +0.043 [ 0.027, +0.113] |
LLM Evaluation
Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.
See all 2,527 papers