Organizations: Department of Computer Engineering, Hamedan University of Technology, Hamedan, Iran. · Department of Computer Science, University of Antwerp, Antwerp, Belgium; Flanders Make Strategic Research Center, Antwerp, Belgium.
Industrial technical reports contain high-value knowledge for maintenance, troubleshooting, and product engineering, but their heterogeneous structure (dense prose, specifications, tables) makes them difficult to index and reason over with standard retrieval and QA pipelines, and no public instruction-tuning or benchmark datasets are built from such documents. We address this gap with Industrial-Instruction, contributing (i) two open QA datasets built from real industrial technical reports and (ii) the end-to-end pipeline that produces them. Using 906 public Panasonic documents (7,525 pages), we apply layout-aware extraction, build a semantic retrieval index, and synthesize multiple-choice QA grounded in retrieved evidence under five query-document relationships (irrelevant retrieval, single-/multi-document support, single-/multi-document answer). After filtering an initial 23.9k generated samples, each dataset provides approximately 13.6k QA pairs with source documents and a held-out benchmark split. Fine-tuning small open LLMs (under 10B parameters) improves Set-Match Accuracy from 28.5% to 42.0% and F1 from 46.6% to 63.5% on the Panasonic benchmark. We release two parallel versions built by the same pipeline: one generated with the open-weight Qwen3-30B-A3B-Instruct model and one with the closed, API-based Claude-Opus-4.6 model, enabling a direct comparison of open- versus frontier-model data generation. The Claude-Opus-4.6 dataset yields a cleaner raw corpus and larger fine-tuning gains, at roughly two orders of magnitude higher cost. MMLU evaluation shows models trained on the Claude-Opus-4.6 data retain essentially all general knowledge, versus a small but measurable forgetting effect for the Qwen-generated data. Together, these datasets and pipeline offer a practical, reproducible path toward scalable industrial benchmarks and training data from real-world documentation.
Figures & tables
Feature
Rule-based
Learning-oriented
Application in language models
Suitable for digital government texts; may be fragmented for complex documents.
Suitable for preserving content; can be converted into a Markdown document.
Markdown support
Limited; usually requires post-processing after text extraction.
Native support; designed for outputs like Markdown/LaTeX.
Technical images
Can extract images but has no relation to the content.
Ability to recognize objects and link them appropriately to the content.
Reliability
Highly interpretable; without producing hallucinatory errors.
Higher risk of hallucinations in complex documents.
Table 1: Comparison between Rule-based and Learning-oriented approaches
Figure 1: An example output of a page from PDF documents by the Dots.OCR model; on the right, you can see the JPG document that has identified and categorized the various regions and sections, and the two images below are a sample created with the Markdown format, which has placed the images and various sections present on the page into this file.
Classification
Table
Text
Footnote
Index
Section
Page
Etc.
Count (pages)
236
14
6
4
2
1
28
Table 2: This presents the distribution of document sections that were not successfully processed, categorized by the type of structural element responsible for the processing error. The counts indicate the number of pages affected by each category.
Figure 2: Shows the distribution of tokens across document pages.
Figure 3: Real-world information retrieval system scenarios and how they are used in an information retrieval-enhanced generation system.
Figure 4: The pipeline for generating the Panasonic document dataset.
Filter/Scenario
R0
R1
R2
R3
R4
Filter to detect healthy response options
2472
2526
2406
2476
2379
Filter for identifying options embedded in the question text
427
302
410
315
365
Rest (deleted)
1962
1961
1994
2015
2052
Total remaining
2899
2828
2816
2791
2744
Table 3: Total number of samples after preprocessing in each scenario.
Figure 5: This shows the final set of samples after the filtering process in each scenario.
Sample Type
Sample Size
Training
12,557
Test (Benchmark)
1,000
Table 4: Data split for training and testing.
Figure 6: Per-scenario sample counts for the Panasonic dataset generated with Claude-Opus-4.6, after filtering.
Sample Type
Sample Size
Training
25,252
Test (Benchmark)
1,000
Table 5: Data split for training and benchmarking on the Claude-Opus-4.6-generated dataset.
Model
Set-Match Acc.
F1-Score
Jaccard
Qwen-4B-Instruct
28.5%
46.65%
41.62%
Phi-3-mini-4k-Instruct
17.5%
31.27%
28%
RAG-Instruct-Llama3-8B
0.70%
0.90%
0.86%
Table 6: Performance of base models on the Panasonic industrial dataset. Qwen-4B-Instruct, Phi-3-mini-4k-Instruct [ 1 ] , RAG-Instruct-Llama3-8B
Config
F1-Score
Jaccard Similarity
Set-Match Acc.
Original
46.57%
41.57%
28.50%
R16-A16
46.71%
41.66%
28.50%
R32-A16
46.89%
41.86%
28.70%
R8-A16
46.53%
41.54%
28.50%
R64-A16
46.27%
41.27%
28.20%
Table 7: Evaluation results under different LoRA fine-tuning configurations.
Table 8: Loss reduction under different LoRA rank configurations.
Configuration
F1-Score
Jaccard Similarity
Set-Match Acc.
Original with RAG
58.55%
54.00%
40.90%
Fully Fine-Tuned with RAG
72.66%
68.88%
56.40%
Original
59.24%
54.71%
41.60%
Fully Fine-Tuned
72.72%
68.93%
56.40%
Table 9: Performance comparison before and after full fine-tuning, with and without RAG, on the Claude-Opus-4.6-generated Panasonic dataset.
Configuration
F1-Score
Jaccard Similarity
Set-Match Acc.
Original with RAG
46.57%
41.57%
28.50%
Fully Fine-Tuned with RAG
63.48%
57.95%
42.00%
Original
46.63%
41.61%
28.50%
Fully Fine-Tuned
63.14%
57.60%
41.70%
Table 10: Performance comparison before and after full fine-tuning, with and without RAG, on the Panasonic dataset.
Model
MMLU Overall
Original (base)
72.13%
finetuned_v1_qwen
70.87%
finetuned_v2_claude
72.08%
Table 11: MMLU accuracy of the base model and the two fine-tuned variants.
Category
Original
finetuned_v1_qwen
finetuned_v2_claude
Humanities
63.66%
61.13%
63.85%
Social Sciences
81.38%
80.44%
81.28%
STEM
72.53%
72.41%
72.44%
Other
75.41%
74.57%
75.09%
Table 12: MMLU accuracy by top-level category for the base model and the two fine-tuned variants.
Figure 7: A side-by-side comparison of FailureSensorIQ and Panasonic metrics for Qwen3-4B-Instruct, Qwen3-4B-Instruct-Panasonic-Qwen, and RAG-Instruct-Llama3-8B.
Metric
Qwen-Org
Qwen-Pana-Qwen
RAG-Instruct
AccOrgIBM
34%
27%
28%
AccPerIBM
0%
0%
33%
F1-Macro (IBM)
40%
43%
62%
F1-Micro (IBM)
66%
74%
68%
Set-Match Acc. (Panasonic)
28%
42%
1%
Jaccard Similarity (Panasonic)
41%
58%
1%
Table 13: Underlying values for Figure 7 .
Figure 8: A side-by-side comparison of FailureSensorIQ and Panasonic metrics for Qwen3-4B-Instruct, Qwen3-4B-Instruct-Panasonic-Claude, and RAG-Instruct-Llama3-8B.
Metric
Qwen-Org
Qwen-Pana-Claude
RAG-Instruct
AccOrgIBM
34.0%
49.6%
28.0%
AccPerIBM
0.0%
0.0%
33.0%
F1-Macro (IBM)
40.0%
33.5%
62.0%
F1-Micro (IBM)
66.0%
50.3%
68.0%
Set-Match Acc. (Panasonic)
40.9%
56.4%
10.7%
Jaccard Similarity (Panasonic)
54.0%
68.9%
15.24%
Table 14: Underlying values for Figure 8 .
Generator
Compute Time
Cost
Notes
Qwen3-30B-A3B-Instruct
1h 43m
$3.2
Local Pro 6000 WS
Claude-Opus-4.6
—
$330
Token-based billing
Table 15: Compute/monetary cost of generating the Panasonic dataset with each generator model.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Subject
Original
finetuned_v1_qwen
Δ
moral_scenarios
51.17%
40.45%
-10.72
public_relations
69.09%
62.73%
-6.36
marketing
91.88%
87.18%
-4.70
high_school_biology
90.65%
86.13%
-4.52
high_school_microeconomics
91.60%
87.82%
-3.78
human_aging
73.54%
69.96%
-3.58
Appendix
Table 16: Largest MMLU regressions for finetuned_v1_qwen (subjects with the biggest accuracy drop vs. the base model).
Subject
Original
finetuned_v2_claude
Δ
public_relations
69.09%
63.64%
-5.45
anatomy
71.11%
66.67%
-4.44
college_chemistry
54.00%
50.00%
-4.00
high_school_computer_science
89.00%
85.00%
-4.00
marketing
91.88%
88.46%
-3.42
global_facts
43.00%
51.00%
+8.00
Appendix
Table 17: Largest MMLU changes for finetuned_v2_claude vs. the base model: regressions (top) and improvements (bottom).
Scenario
Docs
Core requirement (rule 1)
Useless Document (r0)
1
Document is related but provides no useful information for answering q∗ .
Single-Doc. Support (r1)
1
Document provides supporting clues but not an explicit answer.
Multi-Doc. Support (r2)
≥ 2
Multiple documents jointly provide clues but not an explicit answer.
Single-Doc. Answer (r3)
1
Document contains the complete, explicit answer.
Multi-Doc. Answer (r4)
≥ 2
Answer requires multi-hop reasoning across documents.
Appendix
Table 18: How each scenario prompt differs from the shared template.
In industrial procurement, an LLM answer is useful only if it survives a standards check: recommended material must match operating condition, every parameter must respect a regulated threshold, and no procedure may contradict a safety clause. Partial correctness can mask safety-critical contradictions that aggregate LLM benchmarks rarely capture. We introduce IndustryBench, a 2,049-item benchmark for industrial procurement QA in Chinese, grounded in Chinese national standards (GB/T) and structured industrial product records, organized by seven capability dimensions, ten industry categories, and panel-derived difficulty tiers, with item-aligned English, Russian, and Vietnamese renderings. Our construction pipeline rejects 70.3% of LLM-generated candidates at a search-based external-verification stage, calibrating how unreliable industrial QA remains after LLM-only filtering. Our evaluation decouples raw correctness, scored by a Qwen3-Max judge validated at κw=0.798 against a domain expert, from a separate safety-violation (SV) check against source texts. Across 17 models in Chinese and an 8-model intersection over four languages, we find: (i) the best system reaches only 2.083 on the 0--3 rubric, leaving substantial headroom; (ii) Standards & Terminology is the most persistent capability weakness and survives item-aligned translation; (iii) extended reasoning lowers safety-adjusted scores for 12 of 13 models, primarily by introducing unsupported safety-critical details into longer final answers; and (iv) safety-violation rates reshuffle the leaderboard -- GPT-5.4 climbs from rank 6 to rank 3 after SV adjustment, while Kimi-k2.5-1T-A32B drops seven positions. Industrial LLM evaluation therefore requires source-grounded, safety-aware diagnosis rather than aggregate accuracy. We release IndustryBench with all prompts, scoring scripts, and dataset documentation.
Songlin Bai, Xintong Wang, Linlin Yu +12
Multimodal and Industrial AI Team† Taobao&Tmall, Alibaba Group
Due to limited supervised training data, large language models (LLMs) are typically pre-trained via a self-supervised "predict the next word" objective on a vast amount of unstructured text data. To make the resulting model useful to users, it is further trained on a far smaller amount of "instruction-tuning" data comprised of supervised training examples of instructions and responses. To overcome the limited amount of supervised data, we propose a procedure that can transform the knowledge in internet-scale pre-training documents into billions of synthetic instruction and answer training pairs. The resulting dataset, called FineInstructions, uses ~18M instruction templates created from real user-written queries and prompts. These instruction templates are matched to and instantiated with human-written source documents from unstructured pre-training corpora. With "supervised" synthetic training data generated at this scale, an LLM can be pre-trained from scratch solely with the instruction-tuning objective, which is far more in-distribution with the expected downstream usage of LLMs (responding to user prompts). We conduct controlled token-for-token training experiments and find pre-training on FineInstructions outperforms standard pre-training and other proposed synthetic pre-training techniques on standard benchmarks measuring free-form response quality. Our resources can be found at https://huggingface.co/fineinstructions .
Ajay Patel, Colin Raffel, Chris Callison-Burch
University of Pennsylvania · University of Toronto · Vector Institute +1
Industrial procurement requires language models to bridge informal buyer jargon, sparse marketplace attributes, and authoritative engineering standards under strict safety tolerances. We present IndustryLLM, an open-weight industrial language model trained from Qwen3.5-35B-A3B-Base (35B total parameters with ~3B activated per token, with the vision encoder frozen). Rather than relying on generic text scaling, we introduce a failure-driven adaptation recipe spanning continued pre-training (CPT) and supervised fine-tuning (SFT). CPT leverages a curated ~100B-token corpus integrating 5B tokens of national standards (e.g., GB/T) and technical archives, 10B tokens of de-identified real-world industrial transaction and inquiry records, and 60B tokens of general replay. To overcome register mismatch and factual brittleness, we systematically reconstruct an estimated 20B-token domain subset via multi-register rewriting across 10 genres and 8 writing styles, confidence-routed minimal factual editing, and error-targeted QA synthesis (resolving colloquial typos like '42-luo-mu' -> 42CrMo, expanding ambiguous codes like '16674' -> GB/T 16674, and clarifying conflicting dimensional specs). For downstream deployment, we formalize an evidence-gated constraint-evaluation interface enforcing three-valued logic where unverified product evidence remains unknown rather than satisfied. Offline evaluations demonstrate consistent gains on procurement-query structuring (+2.97 percentage points in exact match, 95% CI [2.11, 3.86] in No-Think mode), while randomized online A/B experiments in production yield substantial improvements (+4.25% GMV, +8.3% satisfied inquiries) alongside a latency reduction from 6-7 s to 1.5 s. Model weights and configs are released at https://huggingface.co/alibaba-multimodal-industrial-ai/IndustryLLM.
Liang Ding, Zhiang Xu, Yuyang Sheng +13
Project Lead · Multimodal and Industrial AI Team†∗ Alibaba