FORGE: Fine-grained Multimodal Evaluation for Manufacturing Scenarios

Xiangru Jian; Hao Xu; Wei Pang; Xinjian Zhao; Chengyu Tao; Qixin Zhang; Xikun Zhang; Chao Zhang

cs.CVApr 8, 2026

FORGE: Fine-grained Multimodal Evaluation for Manufacturing Scenarios

Authors: Xiangru Jian, Hao Xu, Wei Pang, Xinjian Zhao, Chengyu Tao, Qixin Zhang, Xikun Zhang, Chao Zhang,

Organizations: University of Waterloo, Canada · University of Sydney, Australia · Singapore Management University, Singapore · The Chinese University of Hong Kong, Shenzhen, China ·

Abstract

The manufacturing sector is increasingly adopting Multimodal Large Language Models (MLLMs) to transition from simple perception to autonomous execution, yet current evaluations fail to reflect the rigorous demands of real-world manufacturing environments. Progress is hindered by data scarcity and a lack of fine-grained domain semantics in existing datasets. To bridge this gap, we introduce FORGE. Wefirst construct a high-quality multimodal dataset that combines real-world 2D images and 3D point clouds, annotated with fine-grained domain semantics (e.g., exact model numbers). We then evaluate 18 state-of-the-art MLLMs across three manufacturing tasks, namely workpiece verification, structural surface inspection, and assembly verification, revealing significant performance gaps. Counter to conventional understanding, the bottleneck analysis shows that visual grounding is not the primary limiting factor. Instead, insufficient domain-specific knowledge is the key bottleneck, setting a clear direction for future research. Beyond evaluation, we show that our structured annotations can serve as an actionable training resource: supervised fine-tuning of a compact 3B-parameter model on our data yields up to 90.8% relative improvement in accuracy on held-out manufacturing scenarios, providing preliminary evidence for a practical pathway toward domain-adapted manufacturing MLLMs. The code and datasets are available at https://ai4manufacturing.github.io/forge-web.

Figures & tables

Panel of Figure 1 — Figure 1: Benchmark Overview. (a) The main pipeline of FORGE . (b) Compared to previous benchmarks, FORGE primarily evaluates the understanding, reasoning, and decision-making capabilities regarding fine-grained domain semantics in manufacturing scenarios. (c) The performance of all models across all given tasks. From left to right: open-source models and closed-source models.

Benchmarks	Data Modality		Data Source	Granularity $\dagger$			Statistics
	Image	Point cloud	Real / Synthetic	Scenario	Workpiece	Model number	Availability	Samples
MMAD ( Jiang et al., 2024 )	✓	✗	Real-world	✓	✓	✗	✓	39,672
MME-Industry ( Yi et al., 2025 )	✓	✗	Real-world	✗	✗	✗	✓	1,050
DesignQA ( Doris et al., 2025 )	✗	✗	Synthetic	✓	✗	✗	✗	1,451
FailureSensorIQ ( Constantinides et al., 2025 )	✗	✗	Real-world	✓	✗	✗	✓	8,296
EngDesign ( Guo et al., 2025 )	✗	✗	Synthetic	✓	✗	✗	✓	1,717

Table 1: Comparison of existing manufacturing MLLMs benchmarks with FORGE .

Figure 2: Task descriptions, input data, and dialogue examples of FORGE .

Panel of Figure 3 — Figure 3: Qualitative results and analysis. (a) WorkVeri : The top row shows examples of CHS scenario , the second rows are from Nuts scenario , and the bottom row is from PCs scenario . (b) SurfInsp : Randomly selected defect examples across different workpieces. (c) AssyVeri : The first three rows correspond to MES scenario , PES scenario , and CNC scenario , respectively, while the last row shows examples of SWN scenario .

Open-Source / Weights Models		Closed-Source Models
Provider	Model	Provider	Model
Google	Gemma-3-27B	OpenAI	GPT-5 / 5.2
OpenGVLab	InternVL3-78B	OpenAI	GPT-5-Mini
Meta	Llama-4-Maverick	OpenAI	O3
Mistral	Mi(ni)stral-3-8B/14B/Large	Google	Gemini-2.5/3-Flash
Alibaba	Qwen3-VL-8B/235B	Anthropic	Claude-4.5-Opus

Table 2: List of evaluated open-source and closed-source MLLMs.

Task	Mod.	Setting	Open-source Models										Closed-source Models								Random
Task	Mod.	Setting	Gemma-3-27B	InternVL3-78B	Llama-4-MAV	Mistral-3-14B	Mistral-3-8B	Mistral-3-Large	Qwen3-VL-235B	Qwen3-VL-8B	Kimi-K2.5	GLM-4.6V	Claude-4.5-Opus	Gemini-2.5-Flash	Gemini-3-Flash	GPT-5.2	GPT-5	GPT-5 Mini	O3	Seed-1.6	Random
WorkVeri	3V	Zero Shot	27.62	32.59	36.36	32.46	30.97	34.78	52.36	42.01	50.00	47.42	39.19	56.05	69.56	61.29	61.05	37.98	58.22	54.44	25.0
		Ref-Cond	23.59	24.90	20.36	25.00	18.26	20.93	35.67	27.20	36.76	44.29	27.88	43.55	65.25	28.34	30.02	33.94	29.55	26.87	25.0
		ICD	28.83	30.16	35.69	28.54	28.86	37.53	37.86	26.32	60.56	44.90	55.04	53.85	67.34	61.69	56.68	30.71	57.91	46.45	25.0
	Image	Zero Shot	25.94	32.59	38.80	33.56	30.61	25.78	64.08	35.41	66.75	50.44	59.42	55.78	72.22	72.50	74.72	73.78	76.18	67.04	25.0
		Ref-Cond	30.51	53.88	40.65	20.22	24.28	25.50	58.01	24.04	69.67	50.33	56.22	51.66	76.27	55.48	64.22	72.73	68.32	49.66	25.0

Table 3: Main benchmark results (accuracy %) across three manufacturing tasks. Bold orange / underlined blue / light purple indicate best / second / third within each model category (open- or closed-source) per row. Rand. = weighted random-guess baseline.

Figure 4: Performance of Open-Source vs. Closed-Source models.

Figure 5: Two error cases of AssyVeri .

Model	Type	Single-Image			Cross-Image
Model	Type	C $\to$ L	L $\to$ C	Avg.	L $\to$ L	C $\to$ C	Avg.
Gemini-3-Flash	Closed	98.2	99.6	98.9	88.7	79.9	84.3
GPT-5.2	Closed	74.6	97.6	86.1	85.6	75.4	80.5
Qwen3-VL-235B	Open	85.4	98.8	92.1	80.3	72.2	76.3
Seed 1.6	Closed	42.0	99.2	70.6	79.3	71.2	75.2
Mistral-3-8B	Open	66.0	70.6	68.3	62.0	33.9	48.0

Table 4: Visual grounding Bottleneck Analysis results (accuracy %). Single-image : coordinate-to-letter (C

\to

L) and letter-to-coordinate (L

\to

C) on 500 cases. Cross-image : letter-to-letter (L

\to

L) and coordinate-to-coordinate (C

\to

C) on 513 pairs.

Table 10

Figure 6: Domain-specific SFT on held-out scenarios (zero-shot acc. %).

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Case	Mod.	Setting	Open-source Models										Closed-source Models
Task	Case	Mod.	Setting	Gemma-3-27B	InternVL3-78B	Llama-4-MAV	Mistral-3-14B	Mistral-3-8B	Mistral-3-Large	Qwen3-VL-235B	Qwen3-VL-8B	Kimi-K2.5	GLM-4.6V	Claude-4.5-Opus	Gemini-2.5-Flash	Gemini-3-Flash	GPT-5.2	GPT-5	GPT-5 Mini	O3	Seed-1.6
WorkVeri	Model No	Image	Zero-Shot	23.18	27.04	24.55	25.78	20.09	24.46	36.91	21.12	30.41	41.84	30.04	33.48	46.78	48.03	51.93	53.88	55.70	38.96
			Ref-Cond	25.97	30.47	21.33	18.61	23.81	19.40	33.64	25.54	37.04	27.90	34.33	30.04	55.79	32.86	40.52	49.36	45.66	27.07
			ICD	18.03	41.56	31.28	24.24	26.84	24.03	35.34	22.41	38.76	29.00	30.90	31.33	65.67	60.87	71.43	61.90	66.51	44.21
		3V	Zero-Shot	22.87	20.18	29.15	27.35	26.24	27.91	37.79	33.33	35.00	37.50	30.49	39.01	51.12	43.50	40.09	31.39	44.09	40.81
			Ref-Cond	20.18	21.17	19.28	30.94	19.73	20.36	30.05	25.69	22.64	28.96	29.60	31.39	49.10	22.07	25.91	27.35	26.82	22.87

Appendix

Table 7: Performance of different cases for WorkVeri and AssyVeri , different workpieces, and different model numbers. Bold orange / underlined blue / light purple indicate best / second / third within each model category (open- or closed-source) per row.

Case	Method	Open-Source										Closed-Source
Case	Method	Gemma-3-27B	InternVL3-78B	Llama-4-MAV	Mistral-3-14B	Mistral-3-8B	Mistral-3-Large	Qwen3-VL-235B	Qwen3-VL-8B	Kimi-K2.5	GLM-4.6V	Claude-4.5-Opus	Gemini-2.5-Flash	Gemini-3-Flash	GPT-5.2	GPT-5	GPT-5 Mini	O3	Seed-1.6
CRACK	Zero-Shot	31.58	6.32	24.21	56.84	44.21	4.21	4.35	22.11	21.05	53.68	16.84	1.05	7.37	5.26	0.00	5.26	5.26	42.55
	Ref-Cond	15.79	4.21	13.68	60.64	60.00	9.47	8.14	27.37	14.74	44.21	11.58	1.05	34.74	5.26	3.16	2.11	7.37	46.32
	ICD	37.89	9.47	44.21	59.78	69.47	10.64	21.79	41.49	40.00	51.58	57.45	47.37	62.11	20.00	10.53	14.74	25.26	56.38
CUT	Zero-Shot	3.37	1.14	5.62	15.73	4.49	0.00	25.58	12.50	12.36	2.25	5.62	4.49	7.87	30.68	22.47	28.09	14.61	4.49
	Ref-Cond	1.12	4.49	5.62	16.85	2.25	1.12	19.05	11.24	11.24	2.25	6.74	4.49	17.98	32.58	29.21	47.19	20.45	7.87

Appendix

Table 8: Performance of 4 anomaly cases for SurfInsp , different workpieces, and different model numbers. Bold orange / underlined blue / light purple indicate best / second / third within each model category (open- or closed-source) per row.

Workpieces	Model Number	No. of Samples
Corner Bracket	5	25
Countersunk Screw	2	20
Cup Head Screw	22	110
Eye Bolt	5	45
Flat Washer	9	45
Hex Nut	4	33

Appendix

Table 9: Simple data summary of raw point cloud data. The dataset contains 14 workpieces.

Workpiece	Wrong types	No. of samples
MES scenario	No Spring Washers	56
	Hex Nut M14	52
	Flat Washer M14	51
	No Flat Washers	50
	Two Flat Washers	50
	Two Spring Washers	49

Appendix

Table 10: Simple data summary of raw image data (scenario-level).

Task	Scenario	Case Type	Error Type	Samples
WorkVeri	CHS scenario	Model No	M16 longer length 100	42
			M18 longer length 100	35
			M12 shorter length 50	34
			M10 shorter length 50	32
		Workpiece	Self tapping Screw	37
			T Bolt Half thread Screw	44

Appendix

Table 11: Simple data summary of three tasks (scenario-level).

Model	Type	WorkVeri 3V ( Nuts scenario )	AssyVeri Img ( CNC scenario )
Gemini 3 Flash	Closed	63.6	53.5
GPT-5.2	Closed	62.6	37.0
Seed 1.6	Closed	60.0	23.6
Qwen3-VL-235B	Open	54.4	19.7
Mistral 3 8B	Open	35.4	26.0
Qwen2.5-VL-3B (base)	Open	28.2	24.0

Appendix

Table 12: Finetuned 3B model vs. frontier models on held-out scenarios (zero-shot accuracy %). The finetuned Qwen2.5-VL-3B nearly matches Qwen3-VL-235B on WorkVeri (53.8% vs. 54.4%) and surpasses Seed 1.6, Qwen3-VL-235B, and Mistral 3 8B on AssyVeri . Bold / Underlined / Shaded = best/2nd/3rd among the five reference models.

Panel of Figure 7 — Figure 7: Two error cases with red annotations added for clearer interpretation.

Panel of Figure 8 — Figure 8: Random examples from PCs scenario

Panel of Figure 9 — Figure 9: Random examples from CHS scenario

Panel of Figure 10 — Figure 10: Random examples from Nuts scenario

Panel of Figure 11 — Figure 11: Random examples from Corner Bracket

Panel of Figure 12 — Figure 12: Random examples from Countersunk Screw

Panel of Figure 13 — Figure 13: Random examples from Cup Head Screw

Panel of Figure 14 — Figure 14: Random examples from Eye Bolt

Panel of Figure 15 — Figure 15: Random examples from Flat Washer

Panel of Figure 16 — Figure 16: Random examples from Hex Nut

Panel of Figure 17 — Figure 17: Random examples from Rivet Nut

Panel of Figure 18 — Figure 18: Random examples from Self-tapping Screw

Panel of Figure 19 — Figure 19: Random examples from Spring Washer

Panel of Figure 20 — Figure 20: Random examples from T Bolt Half thread Screw

Panel of Figure 21 — Figure 21: Random examples from T Nut

Panel of Figure 22 — Figure 22: Random examples from T Screw

Panel of Figure 23 — Figure 23: Random examples from Wing Nut

Panel of Figure 24 — Figure 24: Random examples from Wing Screw

Panel of Figure 25 — Figure 25: Random examples from MES scenario

Panel of Figure 26 — Figure 26: Random examples from PES scenario

Panel of Figure 27 — Figure 27: Random examples from CNC scenario

Panel of Figure 28 — Figure 28: Random examples from SWN scenario

Panel of Figure 29 — Figure 29: Random grounding examples from MES scenario

Panel of Figure 30 — Figure 30: Random grounding examples from PES scenario

Panel of Figure 31 — Figure 31: Random grounding examples from CNC scenario

Panel of Figure 32 — Figure 32: Random grounding examples from PCs scenario

Multimodal Model Evaluation

Multimodal model evaluation measures how reliably AI systems understand and generate content across text, images, audio, video, and other data.

See all 733 papers

Domain Adapted Large Language Models for Additive Manufacturing
Mar 23, 2026Peter Pak, Amir Barati FarimaniMultimodal Large Language Models Domain Adaptation for LLMs
Department of Mechanical Engineering, Carnegie Mellon University, Pittsburgh, PA, USA · Machine Learning Department, Carnegie Mellon University, Pittsburgh, PA, USA
Is In-Domain Training Enough for Fine-Grained Industrial Anomaly Understanding?
Oct 8, 2026Xingwu Zhang, Duanyang Du, Huiling Zhu +5Industrial Anomaly Detection Multimodal Anomaly Detection
Hunan University · University of Aberdeen · South China Normal University +1
From Instructions to Assistance: a Dataset Aligning Instruction Manuals with Assembly Videos for Evaluating Multimodal LLMs
Mar 20, 2026Federico Toschi, Nicolò Brunello, Andrea Sassella +2Multimodal Large Language Models Multimodal Model Evaluation
DEIB, Politecnico di Milano, Via Ponzio 34/5, 20133, Milano (MI), Italy · KASTEL, Karlsruhe Institute of Technology, Am Fasanengarten, 576131, Karlsruhe, Germany

Mar 23, 2026J/K move · Enter open · S save

Domain Adapted Large Language Models for Additive Manufacturing

Peter Pak, Amir Barati Farimani

Department of Mechanical Engineering, Carnegie Mellon University, Pittsburgh, PA, USA · Machine Learning Department, Carnegie Mellon University, Pittsburgh, PA, USA

Multimodal Large Language Models Domain Adaptation for LLMs Domain-Adaptive Pretraining Additive Manufacturing Multimodal Instruction Tuning

Open paper AskPDF arXiv

This work presents a collection of multi-modal domain adapted large language models built upon the instruction tuned variants of open weight models (Gemma 3, Qwen 3, Gemma 4) using a relatively small dataset of around 50 million tokens. The dataset consists of open-access additive manufacturing journal articles with data extracted for the domain adaptive pretraining and visual instruction tuning processes. Various stages of the developed model are evaluated with the Additive-Manufacturing-Benchmark which consists of additive manufacturing domain specific tasks compiled published resources. Domain adapted and instruction tuned models exhibit proficiency in both language and vision based tasks, achieving accuracies upwards of 90% in general additive manufacturing knowledge. This domain adaptive pretraining and instruction tuning strategy outline an accessible specialization method for large language models to a domain such as additive manufacturing.

Abstract

Multimodal Model Evaluation

Explore similar work

Domain Adapted Large Language Models for Additive Manufacturing

Is In-Domain Training Enough for Fine-Grained Industrial Anomaly Understanding?

From Instructions to Assistance: a Dataset Aligning Instruction Manuals with Assembly Videos for Evaluating Multimodal LLMs