cs.AISep 25, 2026
SaveIndustryLLM: Failure-Driven LLM Training for Industrial Procurement
Organizations: Project Lead · Multimodal and Industrial AI Team†∗ Alibaba
Abstract
Industrial procurement requires language models to bridge informal buyer jargon, sparse marketplace attributes, and authoritative engineering standards under strict safety tolerances. We present IndustryLLM, an open-weight industrial language model trained from Qwen3.5-35B-A3B-Base (35B total parameters with ~3B activated per token, with the vision encoder frozen). Rather than relying on generic text scaling, we introduce a failure-driven adaptation recipe spanning continued pre-training (CPT) and supervised fine-tuning (SFT). CPT leverages a curated ~100B-token corpus integrating 5B tokens of national standards (e.g., GB/T) and technical archives, 10B tokens of de-identified real-world industrial transaction and inquiry records, and 60B tokens of general replay. To overcome register mismatch and factual brittleness, we systematically reconstruct an estimated 20B-token domain subset via multi-register rewriting across 10 genres and 8 writing styles, confidence-routed minimal factual editing, and error-targeted QA synthesis (resolving colloquial typos like '42-luo-mu' -> 42CrMo, expanding ambiguous codes like '16674' -> GB/T 16674, and clarifying conflicting dimensional specs). For downstream deployment, we formalize an evidence-gated constraint-evaluation interface enforcing three-valued logic where unverified product evidence remains unknown rather than satisfied. Offline evaluations demonstrate consistent gains on procurement-query structuring (+2.97 percentage points in exact match, 95% CI [2.11, 3.86] in No-Think mode), while randomized online A/B experiments in production yield substantial improvements (+4.25% GMV, +8.3% satisfied inquiries) alongside a latency reduction from 6-7 s to 1.5 s. Model weights and configs are released at https://huggingface.co/alibaba-multimodal-industrial-ai/IndustryLLM.
Figures & tables
Figure 1: Parameter scale and the evidence-gated procurement boundary. (a) Development diagnostic on IndustryBench: IndustryLLM (35B total, 3B activated/token; evaluated under the reasoning-enabled Think mode) compared against published proprietary and open-weight references across parameter scales. Because evaluation protocols, decoding configurations, and reasoning modes differ across external rows, this panel provides parameter-scale diagnostic context rather than a protocol-matched ranking or system efficiency study. (b) Evidence-gated eligibility interface: A downstream system contract that decouples demand-side structuring from supply-side verification. The model maps requests into typed constraints (hard, soft, unresolved); candidate products are evaluated under three-valued logic ( Satisfied , Unknown , Violated ). Missing, stale, or conflicting product evidence strictly evaluates to unknown rather than satisfied (“unknown is not satisfied”), triggering clarification, retrieval, or abstention. Scores: IndustryBench [ 1 ] ; parameter scales: official reports and model cards [ 2 , 51 , 52 , 53 , 54 , 55 , 56 ] .
| Query / evidence state | Structured state | Safe catalog action |
|---|---|---|
| Corrosive-service transfer through an existing line | Explicit application and compatibility concern; candidate duty and interface constraints | Retrieve operating and compatibility evidence before instantiating material, seal, interface, or protection requirements |
| Decision-critical duty conditions are absent | Add medium, temperature, viscosity/solids, – duty, and suction head to | Trigger targeted clarification; do not hallucinate default industrial specifications |
| Required supplier evidence is absent | Hard-constraint status strictly evaluates to | Prohibit item admission to on an unverified basis |
| Components are plausible; assembly compatibility is unverified | Add cross-component interface and bundle-compatibility constraints | Verify the complete assembly; partial component matches do not establish an eligible product set |
| Several fully eligible pump sets remain | Soft preferences and supply features | Rank exclusively within the hard-constraint-verified set |
Table 1: From an informal pump request to an executable procurement state. Mapping observed evidence states to structured constraints and deterministic catalog actions. Missing operating parameters and unverified supplier claims remain unknown until grounded by authoritative evidence.
Figure 2: Training lineage and the procurement evidence boundary. CPT operates over a 100B-token retained pool (incorporating 5B tokens of national standards, 10B tokens of marketplace records, 20B tokens of transformed text, and 60B tokens of general replay). SFT utilizes task-filtered teacher targets, yielding reasoning-enabled ( Think ) and direct-response ( No-Think ) variants. Downstream product eligibility is decoupled from parametric generation, requiring empirical verification via the evidence gate.
| Item | Value |
|---|---|
| Architecture | Hybrid-attention MoE Transformer (unchanged from base) |
| Language-model parameters | 35B (reported base-model count); the frozen vision encoder is not separately counted here |
| Activated parameters per token | Approximately 3B |
| Experts | 256 routed; 8 routed + 1 shared expert activated per token |
| Layers / hidden size | 40 / 2,048 |
| Full-attention heads | 16 query / 2 key–value; head dimension 256 |
Table 2: Inherited model configuration. IndustryLLM retains the architecture, tokenizer, and vocabulary of Qwen3.5-35B-A3B-Base; the training intervention changes data and supervision, while the vision modules remain frozen.
| Source | Tokens | Role | Key processing |
| Filtered open web | 25B (after filtering) | Coverage and long-tail technical text | Domain classifier and quality scorer over a web-scale candidate pool |
| Curated and purchased technical sources | 5B | Technical depth and standards-oriented content | Targeted collection of encyclopedias, technical corpora, standards, and white papers |
| Platform commerce data | 10B | Marketplace terminology and query patterns | De-identification, cleaning, and model-assisted transformation |
| Synthetic / transformed subset: 20B tokens via three procedures in Section 5.2 ; | |||
| included in—not additional to—the approximately 40B domain subtotal. | |||
| Domain subtotal | 40B | ||
Table 3: Estimated CPT corpus composition. Token counts represent retained source-pool volume rather than cumulative training exposure. The estimated 20B-token synthetic/transformed subset is subsumed within—not additional to—the 40B domain subtotal.
Figure 3: Failure-driven CPT reconstruction and proxy tests. Genre-and-style rewriting diversifies document register, minimal editing targets suspect factual spans, and weakness-targeted QA concentrates synthesis on content that the current checkpoint answers poorly. The reported 20B-token transformed subset is included within the source rows of Table 3 .
| A Genre–style rewriting GS-01 | |
| Original excerpt | Rewritten excerpt |
| “What is the difference between optical fiber and broadband?” […] “Broadband is a standard for Internet access speed.” […] “Optical fiber is a transmission line. Unlike ordinary cables carrying network signals, it uses light to transmit signals, at a speed times faster than electricity.” […] “Fiber costs 5,000 a month; broadband—you know.” | Optical fiber and broadband “Optical fiber is a physical medium that transmits information using light signals; broadband is a collective term for network access methods meeting a certain rate and service-carrying capability. They belong to different conceptual categories: the former describes the transmission medium, the latter the access-rate level.” […] “The group velocity of light in fiber is approximately two-thirds of the speed of light in vacuum (about m/s), of the same order as electrical-signal propagation in copper cables; fiber’s advantages lie in information-carrying capacity and transmission distance” […] |
| Transformation. Web QA to an encyclopedic article: noise removal, concept organization, and knowledge expansion—not style alone. | |
| B Confidence-routed minimal edit ME-01 | |
| Original excerpt | Revised excerpt |
| “When the atomic percentage of chromium in steel is not less than 12.5%, it can cause an abrupt change in the steel’s electrode potential from negative to positive , thereby suppressing electrochemical corrosion.” | “The corrosion resistance of stainless steel mainly arises from a chromium-rich passive film formed on its surface. A chromium mass fraction of at least 10.5% is one basic condition in the compositional definition of stainless steel, but does not imply immunity to corrosion in every environment. The stability of the passive film depends on steel grade, medium composition, temperature, and surface condition; chloride ions can induce pitting and crevice corrosion.” |
Table 4: Illustrative CPT reconstruction examples. Each row links a diagnosed data failure to a source-conditioned transformation and its intended training signal. The examples explain the procedures; they do not establish retained-set membership or measured downstream impact.
| Genre | Share | Style | Share |
|---|---|---|---|
| Technical manual / training material | 22% | Technical practitioner | 22% |
| Industry blog / column | 16% | Introductory explanation | 18% |
| Encyclopedic article / knowledge card | 15% | Procurement specialist | 16% |
| Product comparison / selection pitfalls | 11% | Conversational industry blog | 14% |
| Application case / failure analysis | 10% | Standards-oriented formal | 10% |
| FAQ / multi-turn question answering | 9% | Catalog operations | 8% |
Table 5: Genre and style targets for multi-register rewriting. The transformation prompt draws from ten document genres and eight writing styles to vary how the same industrial knowledge is expressed.
| Setting | IndustryBench score (gain over base) | MMLU-Pro score |
|---|---|---|
| A raw corpus, no rewriting | 54.6 (+2.5) | 63.51 |
| B single genre single style | 57.2 (+5.1) | 62.56 |
| C multi genre multi style | 58.8 (+6.7) | 63.77 |
Table 6: Proxy results for genre-and-style rewriting. On Qwen3.5-4B, IndustryBench cells report the score and gain over the base checkpoint; the general-domain column monitors potential regression.
| Knowledge category | A (original) | B (transformed) | B A | |
| Selection & substitution | 1,585 | 51.2% | 64.8% | |
| Standards & terminology | 1,490 | 48.3% | 66.7% | |
| Process principles | 1,285 | 54.1% | 63.5% | |
| Safety & compliance | 285 | 65.0% | 77.2% | |
| Quality & metrology | 225 | 61.0% | 72.0% | |
| Fault diagnosis † | 75 | 48.0% | 65.3% |
Table 7: Proxy results for minimal-edit reconstruction. On Qwen3.5-2B-Base, accuracy is a forced choice between the original and retained variants of one edited fact (chance: 50%). Category rows are measured after annealing; marks the two smallest strata. The targets inherit the transformation record and have not undergone independent expert factual validation; Appendix A gives the full protocol.
| Group mean | Paired difference | |||||
|---|---|---|---|---|---|---|
| Channel | A | B | C | B A | C A | B C |
| Reference-answer log-probability | ||||||
| Judged answer score, 0–5 | 2.159 | 2.217 | 2.206 | |||
Table 8: Proxy comparison of weakness-targeted and random QA synthesis. On Qwen3.5-2B, A uses documents only, while B and C add weakness-targeted and uniformly sampled QA, respectively, under matched token budgets. Means and paired differences use 5,000 held-out questions; Appendix B reports paired win rates and the full protocol.
| Metric | Result |
|---|---|
| Muon iteration at AdamW endpoint loss | |
| Final loss | ( 2.2% lower) |
| Mean global gradient norm | 12.6% lower |
Table 9: Short-horizon Muon–AdamW comparison. The recorded trace uses Qwen3.5-35B-A3B-Base with a shared seed, 500K samples, and sequence length 8,192. The loss-crossing iteration is not a wall-clock, FLOP, or full-scale efficiency measurement.
Figure 4: Reported loss trajectories for main CPT, annealing, and teacher-target SFT. Gray lines are digitized per-step traces, blue lines reproduce the displayed smoothing, and orange markers denote the reported minima. Because the trajectories were reconstructed from the training-monitor figure rather than exported from raw logger data, the curves document stage progression rather than support a new quantitative analysis.
Figure 5: Teacher-target construction and dual response protocols for SFT. For each prompt from the fixed prompt library, the teacher model generates four candidate trajectories. Task-specific filters evaluate response correctness and enforce a strict chain-of-thought constraint ( tokens) to select a single replacement target. The pipeline branches into reasoning-enabled ( Think ) and direct-response ( No-Think ) checkpoints. The evaluation holds prompts and task distributions invariant, while supervised token exposure and training compute are unmatched across experimental arms.
| Model | Raw (0–3) | Raw (0–100) | Final SV (0–3) |
|---|---|---|---|
| IndustryLLM developmental result (Reasoning-enabled Think mode) | |||
| IndustryLLM 35B-A3B (post-SFT Think) | 2.120 | 70.7 | 2.030 |
| Proprietary reference rows (benchmark-reported setup) | |||
| Gemini 3.1 Pro | 2.253 | 75.1 | 2.083 |
| Qwen3.6-Plus | 2.231 | 74.4 | 2.073 |
| Claude Opus 4.6 | 2.164 | 72.1 | 2.011 |
Table 12: IndustryBench developmental diagnostic scores and external reference rows. IndustryLLM is evaluated under the reasoning-enabled Think mode. Published reference models reflect their original benchmark-reported inference configurations and were not rerun under a shared judge snapshot or synchronized decoding harness. This compilation establishes descriptive parameter-scale context rather than a protocol-matched competitive ranking or system efficiency benchmark.
| No-Think Base-Checkpoint Comparison | ||
|---|---|---|
| Structuring Task | CPT+SFT | Base+SFT |
| Exact match | 14.83% | 11.86% |
| Semantic match | 36.65% | 31.47% |
| Multiple choice | 82.91% | 79.86% |
Table 13: Procurement-query structuring performance across base-checkpoint variants. Evaluated in the direct-response ( No-Think ) mode across generation instances and multiple-choice reasoning items under matched downstream SFT configurations. Metrics assess intent structuring competencies rather than downstream product eligibility.
| Production Metric | Formal Operational Definition |
|---|---|
| PV_L2O | Orders completed per product impression (page view to order); distinct from CPV tuples. |
| UV_L2O | Completed transactions per unique visitor session. |
| A2A inquiries | Buyer procurement inquiries initiated through the conversational assistant flow. |
| Demand-satisfaction rate | Proportion of search sessions where returned items fully cover stated requirements. |
| Transaction-conversion rate | Ratio of fulfilled procurement transactions to active buying sessions. |
| Product CTR per visitor | Ratio of unique item detail clicks to active purchasing visitors. |
Table 14: Definitions of production-study metrics. Public absolute rates are reported directly, whereas commercially sensitive metrics (A2A inquiries, GMV, and advertising revenue) are disclosed exclusively as relative percentage changes.
| Dimension | Standard General LLM Response | IndustryLLM Intent Interpretation |
|---|---|---|
| Raw Buyer Query | “帮我找 42络钼 螺栓 16674 M1035 但要50长” (“Find me 42-luo-mu bolts 16674 M1035 but need 50 length”) | |
| Material Parsing | Retains verbatim typo “42络钼” or fails extraction; zero catalog recall. | Normalized: Maps phonetic typo to canonical alloy grade 42CrMo . |
| Standard Mapping | Interprets “16674” as internal product SKU or arbitrary quantity. | Expanded: Resolves truncated designator to GB/T 16674 ( Hexagon bolts with flange ). |
| Constraint Logic | Arbitrarily selects or via recency bias; risks silent physical mismatch. | Conflict Interception: Identifies geometric contradiction ( vs. ); sets length to . |
| Actionable Output | Directly emits unverified, hallucinated product recommendations. | Deterministic Gating: Generates RFQ payload with missing field flagged; asks targeted clarification: “请确认螺栓公称长度为 35 mm 还是 50 mm？” |
Table 15: Qualitative comparison on an authentic, de-identified industrial fastener inquiry. Demonstrating how IndustryLLM resolves colloquial typos, standard designators, and conflicting geometric constraints compared to typical general LLM failure modes.
Figure 6: Blind multi-dimensional evaluation of conversational inquiry understanding. Score differentials (IndustryLLM No-Think minus Qwen3.5-Plus rubric points) across 1,000 authentic buyer–supplier procurement dialogs, scored on a 70-point rubric by a Qwen3.7-Max-thinking judge. Positive values denote dimensions favoring IndustryLLM.
| Production Endpoint | Observed Shift | Evaluation Methodology & Control Conditions |
|---|---|---|
| End-to-end response time | 6–7 s 1.5 s | Matched GPU hardware pool and serving framework; load distribution unisolated. |
| A2A inquiry volume | +6.50% | Relative percentage increase ( ). |
| Gross merchandise value (GMV) | +4.25% | Relative percentage increase ( ). |
| Advertising revenue | +1.85% | Relative percentage increase ( ). |
| Attribute-standardization rate | 62.5% 85.2% | Absolute percentage rate improvement ( ). |
Table 16: Production outcomes from the 18-day randomized inquiry A/B trial. Evaluated across persistent 50:50 user-level traffic splits ( 100,000 users/arm/day). All reported commercial shifts satisfy under standard production testing. The evaluation reflects the bundled inquiry stack; commercially sensitive baseline levels are withheld.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Stage | Procedure | Retained |
|---|---|---|
| Source documents | industrial documents sampled from the training corpus, so that every question has a source document inside it | 400,000 |
| Question generation | factual and open-ended items from topic clusters and keyword pools, self-contained phrasing, verbatim answer points | 120,000 |
| Cleaning | drop anaphoric items, items under 15 characters, and items whose answer points cannot be matched in the source document | 113,000 |
| Weakness probing | annealing-start checkpoint, eight samples per item ( , top- 0.8), resampled to sixteen inside the ambiguous band | — |
| Response scoring | one judge call rates the sampled responses on a 0–5 scale and labels the dominant defect | pool mean 2.17 |
| Weakness selection | low mean score with dominant defect “knowledge error” | 34,000 (30.1%) |
Table B1: Weakness-targeted QA construction pipeline and retained counts. Qwen3.7-Max performs question generation, response scoring, teacher answering, and quality control; the annealing-start checkpoint performs local weakness probing. Counts distinguish source documents, generated questions, cleaned items, selected weaknesses, and final splits.
| Group mean | Paired contrast | |||||
|---|---|---|---|---|---|---|
| Channel | A | B | C | B A | C A | B C |
| Reference-answer log-probability | ||||||
| Judged answer score, 0–5 | 2.159 | 2.217 | 2.206 | |||
| paired win rate | — | — | — | 54.6% | 50.9% | 51.6% |
Table B2: Proxy results across likelihood and judged-answer channels. Group columns report means over held-out questions; contrast columns report differences of per-question paired means, with paired win rates for the judged channel. Bold marks the best group in each row.
| Case stratum | Currently available | Evidence status |
|---|---|---|
| Alias / standard / conflict | Narrative for 42CrMo, GB/T 16674, and conflicting length expressions | Illustrative only; no frozen paired output or blinded ruling |
| Unit and attribute normalization | Aggregate structuring scores only | No outcome-independent paired case |
| Missing information and abstention | Conceptual pump example only | Formulation example, not empirical evidence |
| Catalog evidence and eligibility | No end-to-end record | Not evaluated in this report |
| Tie or IndustryLLM failure | None displayed | No comparative case set is reported |
Table C1: Scope of the illustrative application cases. The available examples explain intended query-structuring and evidence-gating behavior, but they do not form a frozen comparative case set with paired outputs and blinded adjudication.
| Field | Required content |
|---|---|
| Provenance and selection | Case ID, frozen split, timestamp, category, risk tier, selection rule, and privacy edits |
| Common input | De-identified buyer query, relevant context, schema, candidate records, and accessible evidence |
| Reference state | Expert category; resolved explicit and derived constraints; hard-versus-soft status; required-versus-excluded polarity; operator, value, unit, and origin; critical unresolved fields; eligible set when evaluated |
| Base+SFT output | Verbatim structured and natural-language output, including citations, clarification, or abstention |
| CPT+SFT output | Verbatim output under the identical harness |
| Blinded adjudication | Per-field correctness, unsupported inference, conflict handling, hard-constraint status, error taxonomy, and rationale |
Table C2: Required contents of a paired procurement case. Each case card should preserve common inputs, verbatim outputs from Base+SFT and CPT+SFT, provenance and selection metadata, the expert reference state, and blinded adjudication.
| Source class | Retained | Information not available | Governance status |
|---|---|---|---|
| Filtered open web | 25B tokens | Raw volume; cutoff; language, industry, and category mix; classifier validation, thresholds and yield; deduplication; phase sampling | Source-specific terms apply; corpus not redistributed |
| Curated / purchased technical | 5B tokens | Named source types and revisions; standards cutoff; document count; overlap and sampling | Source-specific access terms apply; corpus not redistributed |
| Platform commerce | 10B tokens | Date range; product/query/inquiry mix; categories; cleaning; deduplication; phase sampling | De-identified internal data; not redistributed |
| General corpus | 60B tokens | Named corpora and revisions; languages; unique and sampled tokens by phase | Source-specific terms apply; corpus not redistributed |
Table D1: Source-level composition of the approximately 100B retained corpus. The rows separate domain and general source classes, approximate retained-token counts, unavailable metadata, and distribution status. These counts do not represent phase-specific training exposure.
| Procedure | Currently reported | Information not available | Teacher/validation status |
|---|---|---|---|
| Genre/style diversification | Ten genres, eight styles; Qwen3-Max | Input documents/tokens; candidates and retained records/tokens; acceptance; deduplication; cross-procedure overlap | Teacher revision, prompt hash, and sampled validation not available |
| Minimal edits | Localized long-document edits; short-document regeneration allowed; Qwen3-Max | Full-scale candidates, retained edits/documents/tokens, non-target equality, edited-token share, drop reasons, source-validation yield | Teacher revision and independent factual validation not available |
| Weakness-targeted QA | M full-scale questions; Qwen3.7-Max | Selected QA pairs/documents/tokens, selection yield, duplicates, source and cross-procedure overlap | Teacher/judge revisions, prompts, and validation results not available |
| Total transformed | M texts; 20B tokens | Reconciliation to source rows without double counting; unique versus sampled exposure | Final manifest revision and hash not available |
Table D2: Transformation procedures, reported scale, and exposure definitions. Generated candidates, retained unique records, and sampled training exposure are distinct quantities; the aggregate metadata do not separate them by procedure or source overlap.
More in this topic
LLM Fine-Tuning
LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.
See all 746 papers