Industrial-Instruction: An End-to-End Framework for Building Instruction-Tuning and Benchmark Datasets from Industrial Technical Reports

Parsa Bakhtiari; Hassan Bashiri; Alireza Khalilipour; Masoud Nasiripour; Moharram Challenger

cs.CLAug 24, 2026

Industrial-Instruction: An End-to-End Framework for Building Instruction-Tuning and Benchmark Datasets from Industrial Technical Reports

Authors: Parsa Bakhtiari, Hassan Bashiri, Alireza Khalilipour, Masoud Nasiripour, Moharram Challenger

Organizations: Department of Computer Engineering, Hamedan University of Technology, Hamedan, Iran. · Department of Computer Science, University of Antwerp, Antwerp, Belgium; Flanders Make Strategic Research Center, Antwerp, Belgium.

Abstract

Industrial technical reports contain high-value knowledge for maintenance, troubleshooting, and product engineering, but their heterogeneous structure (dense prose, specifications, tables) makes them difficult to index and reason over with standard retrieval and QA pipelines, and no public instruction-tuning or benchmark datasets are built from such documents. We address this gap with Industrial-Instruction, contributing (i) two open QA datasets built from real industrial technical reports and (ii) the end-to-end pipeline that produces them. Using 906 public Panasonic documents (7,525 pages), we apply layout-aware extraction, build a semantic retrieval index, and synthesize multiple-choice QA grounded in retrieved evidence under five query-document relationships (irrelevant retrieval, single-/multi-document support, single-/multi-document answer). After filtering an initial 23.9k generated samples, each dataset provides approximately 13.6k QA pairs with source documents and a held-out benchmark split. Fine-tuning small open LLMs (under 10B parameters) improves Set-Match Accuracy from 28.5% to 42.0% and F1 from 46.6% to 63.5% on the Panasonic benchmark. We release two parallel versions built by the same pipeline: one generated with the open-weight Qwen3-30B-A3B-Instruct model and one with the closed, API-based Claude-Opus-4.6 model, enabling a direct comparison of open- versus frontier-model data generation. The Claude-Opus-4.6 dataset yields a cleaner raw corpus and larger fine-tuning gains, at roughly two orders of magnitude higher cost. MMLU evaluation shows models trained on the Claude-Opus-4.6 data retain essentially all general knowledge, versus a small but measurable forgetting effect for the Qwen-generated data. Together, these datasets and pipeline offer a practical, reproducible path toward scalable industrial benchmarks and training data from real-world documentation.

Figures & tables

Feature	Rule-based	Learning-oriented
Application in language models	Suitable for digital government texts; may be fragmented for complex documents.	Suitable for preserving content; can be converted into a Markdown document.
Markdown support	Limited; usually requires post-processing after text extraction.	Native support; designed for outputs like Markdown/LaTeX.
Technical images	Can extract images but has no relation to the content.	Ability to recognize objects and link them appropriately to the content.
Reliability	Highly interpretable; without producing hallucinatory errors.	Higher risk of hallucinations in complex documents.

Table 1: Comparison between Rule-based and Learning-oriented approaches

Figure 1: An example output of a page from PDF documents by the Dots.OCR model; on the right, you can see the JPG document that has identified and categorized the various regions and sections, and the two images below are a sample created with the Markdown format, which has placed the images and various sections present on the page into this file.

Classification	Table	Text	Footnote	Index	Section	Page	Etc.
Count (pages)	236	14	6	4	2	1	28

Table 2: This presents the distribution of document sections that were not successfully processed, categorized by the type of structural element responsible for the processing error. The counts indicate the number of pages affected by each category.

Figure 2: Shows the distribution of tokens across document pages.

Figure 3: Real-world information retrieval system scenarios and how they are used in an information retrieval-enhanced generation system.

Figure 4: The pipeline for generating the Panasonic document dataset.

Filter/Scenario	R0	R1	R2	R3	R4
Filter to detect healthy response options	2472	2526	2406	2476	2379
Filter for identifying options embedded in the question text	427	302	410	315	365
Rest (deleted)	1962	1961	1994	2015	2052
Total remaining	2899	2828	2816	2791	2744

Table 3: Total number of samples after preprocessing in each scenario.

Figure 5: This shows the final set of samples after the filtering process in each scenario.

Sample Type	Sample Size
Training	12,557
Test (Benchmark)	1,000

Table 4: Data split for training and testing.

Figure 6: Per-scenario sample counts for the Panasonic dataset generated with Claude-Opus-4.6, after filtering.

Sample Type	Sample Size
Training	25,252
Test (Benchmark)	1,000

Table 5: Data split for training and benchmarking on the Claude-Opus-4.6-generated dataset.

Model	Set-Match Acc.	F1-Score	Jaccard
Qwen-4B-Instruct	28.5%	46.65%	41.62%
Phi-3-mini-4k-Instruct	17.5%	31.27%	28%
RAG-Instruct-Llama3-8B	0.70%	0.90%	0.86%

Table 6: Performance of base models on the Panasonic industrial dataset. Qwen-4B-Instruct, Phi-3-mini-4k-Instruct [ 1 ] , RAG-Instruct-Llama3-8B

Config	F1-Score	Jaccard Similarity	Set-Match Acc.
Original	46.57%	41.57%	28.50%
R16-A16	46.71%	41.66%	28.50%
R32-A16	46.89%	41.86%	28.70%
R8-A16	46.53%	41.54%	28.50%
R64-A16	46.27%	41.27%	28.20%

Table 7: Evaluation results under different LoRA fine-tuning configurations.

Panel of Table 8 — Table 8: Loss reduction under different LoRA rank configurations.

Configuration	F1-Score	Jaccard Similarity	Set-Match Acc.
Original with RAG	58.55%	54.00%	40.90%
Fully Fine-Tuned with RAG	72.66%	68.88%	56.40%
Original	59.24%	54.71%	41.60%
Fully Fine-Tuned	72.72%	68.93%	56.40%

Table 9: Performance comparison before and after full fine-tuning, with and without RAG, on the Claude-Opus-4.6-generated Panasonic dataset.

Configuration	F1-Score	Jaccard Similarity	Set-Match Acc.
Original with RAG	46.57%	41.57%	28.50%
Fully Fine-Tuned with RAG	63.48%	57.95%	42.00%
Original	46.63%	41.61%	28.50%
Fully Fine-Tuned	63.14%	57.60%	41.70%

Table 10: Performance comparison before and after full fine-tuning, with and without RAG, on the Panasonic dataset.

Model	MMLU Overall
Original (base)	72.13%
finetuned_v1_qwen	70.87%
finetuned_v2_claude	72.08%

Table 11: MMLU accuracy of the base model and the two fine-tuned variants.

Category	Original	finetuned_v1_qwen	finetuned_v2_claude
Humanities	63.66%	61.13%	63.85%
Social Sciences	81.38%	80.44%	81.28%
STEM	72.53%	72.41%	72.44%
Other	75.41%	74.57%	75.09%

Table 12: MMLU accuracy by top-level category for the base model and the two fine-tuned variants.

Figure 7: A side-by-side comparison of FailureSensorIQ and Panasonic metrics for Qwen3-4B-Instruct, Qwen3-4B-Instruct-Panasonic-Qwen, and RAG-Instruct-Llama3-8B.

Metric	Qwen-Org	Qwen-Pana-Qwen	RAG-Instruct
AccOrgIBM	34%	27%	28%
AccPerIBM	0%	0%	33%
F1-Macro (IBM)	40%	43%	62%
F1-Micro (IBM)	66%	74%	68%
Set-Match Acc. (Panasonic)	28%	42%	1%
Jaccard Similarity (Panasonic)	41%	58%	1%

Table 13: Underlying values for Figure 7 .

Figure 8: A side-by-side comparison of FailureSensorIQ and Panasonic metrics for Qwen3-4B-Instruct, Qwen3-4B-Instruct-Panasonic-Claude, and RAG-Instruct-Llama3-8B.

Metric	Qwen-Org	Qwen-Pana-Claude	RAG-Instruct
AccOrgIBM	34.0%	49.6%	28.0%
AccPerIBM	0.0%	0.0%	33.0%
F1-Macro (IBM)	40.0%	33.5%	62.0%
F1-Micro (IBM)	66.0%	50.3%	68.0%
Set-Match Acc. (Panasonic)	40.9%	56.4%	10.7%
Jaccard Similarity (Panasonic)	54.0%	68.9%	15.24%

Table 14: Underlying values for Figure 8 .

Generator	Compute Time	Cost	Notes
Qwen3-30B-A3B-Instruct	1h 43m	$3.2	Local Pro 6000 WS
Claude-Opus-4.6	—	$330	Token-based billing

Table 15: Compute/monetary cost of generating the Panasonic dataset with each generator model.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Subject	Original	finetuned_v1_qwen	$\Delta$
moral_scenarios	51.17%	40.45%	-10.72
public_relations	69.09%	62.73%	-6.36
marketing	91.88%	87.18%	-4.70
high_school_biology	90.65%	86.13%	-4.52
high_school_microeconomics	91.60%	87.82%	-3.78
human_aging	73.54%	69.96%	-3.58

Appendix

Table 16: Largest MMLU regressions for finetuned_v1_qwen (subjects with the biggest accuracy drop vs. the base model).

Subject	Original	finetuned_v2_claude	$\Delta$
public_relations	69.09%	63.64%	-5.45
anatomy	71.11%	66.67%	-4.44
college_chemistry	54.00%	50.00%	-4.00
high_school_computer_science	89.00%	85.00%	-4.00
marketing	91.88%	88.46%	-3.42
global_facts	43.00%	51.00%	+8.00

Appendix

Table 17: Largest MMLU changes for finetuned_v2_claude vs. the base model: regressions (top) and improvements (bottom).

Scenario	Docs	Core requirement (rule 1)
Useless Document (r0)	1	Document is related but provides no useful information for answering $q^{*}$ .
Single-Doc. Support (r1)	1	Document provides supporting clues but not an explicit answer.
Multi-Doc. Support (r2)	$\geq$ 2	Multiple documents jointly provide clues but not an explicit answer.
Single-Doc. Answer (r3)	1	Document contains the complete, explicit answer.
Multi-Doc. Answer (r4)	$\geq$ 2	Answer requires multi-hop reasoning across documents.

Appendix

Table 18: How each scenario prompt differs from the shared template.

Synthetic Benchmark Generation

Synthetic benchmark generation creates controlled, verifiable evaluation tasks and datasets to test models across diverse settings.

See all 281 papers

IndustryBench: Probing the Industrial Knowledge Boundaries of LLMs
May 11, 2026Songlin Bai, Xintong Wang, Linlin Yu +12LLM Evaluation Question Answering
Multimodal and Industrial AI Team† Taobao&Tmall, Alibaba Group
FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale
Jan 29, 2026Ajay Patel, Colin Raffel, Chris Callison-BurchLanguage Model Pretraining Synthetic Data Pretraining
University of Pennsylvania · University of Toronto · Vector Institute +1
IndustryLLM: Failure-Driven LLM Training for Industrial Procurement
Sep 25, 2026Liang Ding, Zhiang Xu, Yuyang Sheng +13LLM Fine-Tuning Domain Adaptation for LLMs
Project Lead · Multimodal and Industrial AI Team†∗ Alibaba

May 11, 2026J/K move · Enter open · S save

IndustryBench: Probing the Industrial Knowledge Boundaries of LLMs

Songlin Bai, Xintong Wang, Linlin Yu, Bin Chen+11

Multimodal and Industrial AI Team† Taobao&Tmall, Alibaba Group

LLM Evaluation Question Answering LLM Safety Evaluation Knowledge-Intensive QA

Open paper AskPDF arXiv

In industrial procurement, an LLM answer is useful only if it survives a standards check: recommended material must match operating condition, every parameter must respect a regulated threshold, and no procedure may contradict a safety clause. Partial correctness can mask safety-critical contradictions that aggregate LLM benchmarks rarely capture. We introduce IndustryBench, a 2,049-item benchmark for industrial procurement QA in Chinese, grounded in Chinese national standards (GB/T) and structured industrial product records, organized by seven capability dimensions, ten industry categories, and panel-derived difficulty tiers, with item-aligned English, Russian, and Vietnamese renderings. Our construction pipeline rejects 70.3% of LLM-generated candidates at a search-based external-verification stage, calibrating how unreliable industrial QA remains after LLM-only filtering. Our evaluation decouples raw correctness, scored by a Qwen3-Max judge validated at

κ_w = 0.798

against a domain expert, from a separate safety-violation (SV) check against source texts. Across 17 models in Chinese and an 8-model intersection over four languages, we find: (i) the best system reaches only 2.083 on the 0--3 rubric, leaving substantial headroom; (ii) Standards & Terminology is the most persistent capability weakness and survives item-aligned translation; (iii) extended reasoning lowers safety-adjusted scores for 12 of 13 models, primarily by introducing unsupported safety-critical details into longer final answers; and (iv) safety-violation rates reshuffle the leaderboard -- GPT-5.4 climbs from rank 6 to rank 3 after SV adjustment, while Kimi-k2.5-1T-A32B drops seven positions. Industrial LLM evaluation therefore requires source-grounded, safety-aware diagnosis rather than aggregate accuracy. We release IndustryBench with all prompts, scoring scripts, and dataset documentation.

Abstract

Synthetic Benchmark Generation

Explore similar work

IndustryBench: Probing the Industrial Knowledge Boundaries of LLMs

FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale

IndustryLLM: Failure-Driven LLM Training for Industrial Procurement