Instruction-Tuning Open-Weight Language Models for BPMN Model Generation

Gökberk Çelikmasat; Atay Özgövde; Fatma Başak Aydemir

cs.SEDec 12, 2025

Instruction-Tuning Open-Weight Language Models for BPMN Model Generation

Authors: Gökberk Çelikmasat, Atay Özgövde, Fatma Başak Aydemir

Organizations: Boğaziçi University, İstanbul, Türkiye · Utrecht University, Utrecht, The Netherlands

Abstract

Domain models are central to software engineering, as they enable a shared understanding, guide implementation, and support automated analyses and model-driven development. Yet, despite these benefits, practitioners often skip modeling because it is time-consuming and demands scarce expertise. We address this barrier by investigating whether open-weight large language models, adapted via instruction tuning, can generate high-quality BPMN process models directly from natural language descriptions in a cost-effective and privacy-preserving way. We introduce InstruBPM, a reproducible approach that prepares paired text-diagram data and instruction tunes an open source large language model with parameter-efficient fine-tuning and quantization for on-prem deployment. We evaluate the tuned model through complementary perspectives: (i) text/code similarity using BLEU, ROUGE-L, and METEOR, (ii) structural fidelity using Relative Graph Edit Distance, (iii) guidelines conformance using external tool checks, and (iv) a small expert review. Using a curated subset of a multi-domain BPMN dataset, we compare the tuned model with untuned open-weight baselines and strong proprietary models under consistent prompting regimes. Our compact tuned model outperforms all baselines across sequence and structural metrics while requiring substantially fewer resources; guideline analysis and expert feedback further indicate that the generated diagrams largely follow BPMN best practices and are useful starting points that reduce modeling effort. Overall, instruction tuning improves structural accuracy and robustness compared to untuned baselines and reduces reliance on heavy prompt scaffolding. We publicly share the trained models and scripts to support reproducibility and further research.

Figures & tables

Approach	Model	Adapt.	Target	Evaluation
Grohs et al. [ 21 ]	GPT-4	Prompt	BPMN/RPA	Qualitative
Klievtsova et al. [ 27 ]	GPT-3.5/4	Prompt	Dialogue/extraction	Empirical
Licardo et al. [ 39 ]	GPT-4+BERT	Prompt+FT	BPMN	R-GED
Nivon & Salaün [ 44 ]	GPT-3.5	FT	Grammar/AST to BPMN	Text + structural
Bernardi et al. [ 4 ]	LLaMA	FT+RAG	BPMN Q&A	Retrieval metrics
Kourani et al. [ 29 ]	GPT-4	Prompt	BPMN/Petri nets	Manual + structural

Table 1: Representative LLM-based BPM approaches.

Figure 1: Design Science Research methodology adopted in this study.

Panel of Figure 2 — Figure 2: End-to-end instruction-tuning example adapted from [ 10 ] : (a) instruction template, (b) sample textual description, (c) model output in DOT, and (d) rendered BPMN model.

ID	Current Role	Years of BPM Experience
E1	System Analyst	2
E2	Business Analyst	4
E3	Workflow Automation Engineer	6
E4	Lead Business Analyst	10

Table 2: Demographics of Experts

DOT element	BPMN 2.0 element
START_NODE (circle)	bpmn2:startEvent
END_NODE (double circle)	bpmn2:endEvent
Box node (labeled)	bpmn2:task
Diamond [label="+"]	bpmn2:parallelGateway
Diamond [label="X"]	bpmn2:exclusiveGateway
Diamond [label="O"]	bpmn2:inclusiveGateway

Table 3: DOT-to-BPMN 2.0 element mapping used by the converter.

Category	ID	Description
General	2	Smaller models improve understandability and reduce cognitive load.
Notation	16	The designer should split or join sequence flows always using gateways.
	18	Match split/join pairs to preserve structuredness.
	20	Do not represent gateways with only one incoming and one outgoing sequence flow.
	21	Minimize the use of inclusive (OR) gateways, which are difficult to synchronize.
	22	Avoid dead ends at exclusive gateways; provide a default branch.

Table 4: Selected BEBoP guidelines, grouped by BEBoP’s own categories, with brief rationale.

Open-Weight LLMs
Models	BLEU	ROUGE-L	METEOR	R-GED
Qwen2.5-7B-Instruct	$5.67$	$43.11$	$45.10$	$37.94$
Qwen2.5-14B-Instruct	$6.66$	$43.01$	$42.96$	$40.00$
Qwen2.5-Coder-7B-Instruct	$6.08$	$41.34$	$39.38$	$38.48$
Qwen2.5-Coder-14B-Instruct	$6.76$	$43.30$	$43.99$	$42.10$
Qwen3-4B-Instruct-2507	$2.89$	$40.31$	$44.16$	$44.47$

Table 5: BPMN Model Generation Textual and Structural Similarity Evaluation Results

Domain	BLEU	ROUGE-L	METEOR	R-GED
Account payable process	$84.02$	$94.69$	$93.65$	$100.0$
Accounts receivable process	$85.62$	$95.64$	$94.17$	$100.0$
Budget preparation process	$81.23$	$94.41$	$92.52$	$100.0$
Churn-rate prevention process	$87.34$	$96.10$	$94.94$	$100.0$
Client onboarding (marketing agency)	$83.63$	$95.49$	$93.31$	$100.0$
Content promotion process	$82.25$	$94.66$	$92.78$	$100.0$

Table 6: Per-domain BPMN Model Generation Average Textual Similarity Results.

Technique	BLEU	ROUGE-L	METEOR
Zero-shot	83.06	94.43	92.82
Chain-of-thought	81.92	93.79	91.25
Tree-of-thoughts	81.80	93.72	91.16

Table 7: Impact of Prompt Engineering Techniques on Trained Model

Training Mode	BLEU	ROUGE-L	METEOR	Model Size (GB)	Throughput (tokens/sec)	Avg Time to 1st Token (s)
Full-Lora-Tuning (BF16)	$83.06$	$94.43$	$92.82$	$\sim$ 8.05	$35.73$	$0.6423$
Q8_0-GGUF	$83.08$	$94.41$	$92.83$	$\sim$ 4.28	$31.04$	$0.6052$
Q6_K-GGUF	$83.05$	$94.39$	$92.77$	$\sim$ 3.31	$29.45$	$0.7249$
Q5_K_M-GGUF	$82.93$	$94.33$	$92.75$	$\sim$ 2.89	$29.30$	$0.6710$
Q4_K_M-GGUF	$80.42$	$93.22$	$91.03$	$\sim$ 2.50	$30.05$	$0.5502$
Q3_K_M-GGUF	$75.58$	$90.59$	$87.58$	$\sim$ 2.08	$28.46$	$0.9269$

Table 8: Quantization and Model Size Impact on BPMN Generation

Alpha ( $\alpha$ )	BLEU	ROUGE-L	METEOR	R-GED (%)
8	41.24	51.13	58.50	55.07
16	74.47	87.77	85.60	98.23
32	83.06	94.43	92.82	99.44
64	14.09	33.78	30.98	31.09

Table 9: Effect of merge-time LoRA alpha on model performance (rank = 16).

Panel of Figure 5 — Figure 5: Effect of LoRA merge alpha ( $\alpha$ ) on evaluation metrics with rank = 16 adapters.

ID	Guideline	OK	KO	Pass (%)
2	Minimize model size	179	0	100.00
16	Use explicit gateways	173	6	96.65
18	Split and join flows consistently	175	4	97.77
20	Use meaningful gateways	178	1	99.44
21	Minimize inclusive OR gateways	81	98	45.25
22	Use default flows	174	5	97.21

Table 10: Verification results on 179 generated BPMN models.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Reference and generated BPMN diagrams illustrating an omission case.

Open-weight LLMs
Models	BLEU	ROUGE-L	METEOR	R-GED
Qwen2.5-7B-Instruct	5.67 [ 4.85, 6.82]	43.11 [41.44, 44.89]	45.10 [42.17, 47.51]	37.94 [32.70, 44.14]
Qwen2.5-14B-Instruct	6.66 [ 5.63, 8.02]	43.01 [41.12, 44.98]	42.96 [39.47, 46.26]	40.00 [34.92, 45.79]
Qwen2.5-Coder-7B-Instruct	6.08 [ 5.30, 6.95]	41.34 [40.01, 42.61]	39.38 [36.41, 42.09]	38.48 [32.99, 44.32]
Qwen2.5-Coder-14B-Instruct	6.76 [ 5.76, 8.01]	43.30 [41.80, 44.85]	43.99 [41.40, 46.21]	42.10 [37.92, 46.95]
Qwen3-4B-Instruct-2507	2.89 [ 2.16, 4.02]	40.31 [38.70, 42.03]	44.16 [41.52, 46.67]	44.47 [39.38, 50.08]

Appendix

Table 11: BPMN model generation: macro means with 95% confidence intervals on the 180-instance seed set.

Domain	BLEU	ROUGE-L	METEOR	R-GED
Account payable process	84.0 [77.9, 89.2]	94.7 [92.6, 96.4]	93.6 [91.4, 95.4]	100.0 [100.0, 100.0]
Accounts receivable process	85.6 [80.6, 89.5]	95.6 [93.7, 97.1]	94.2 [91.6, 96.0]	100.0 [100.0, 100.0]
Budget preparation process	81.2 [77.4, 84.2]	94.4 [93.1, 95.4]	92.5 [90.8, 94.0]	100.0 [100.0, 100.0]
Churn-rate prevention process	87.3 [83.6, 90.2]	96.1 [95.0, 97.0]	94.9 [93.5, 96.1]	100.0 [100.0, 100.0]
Client onboarding (marketing agency)	83.6 [79.3, 87.2]	95.5 [94.3, 96.5]	93.3 [91.4, 94.8]	100.0 [100.0, 100.0]
Content promotion process	82.2 [77.3, 86.2]	94.7 [93.3, 95.8]	92.8 [90.8, 94.4]	100.0 [100.0, 100.0]

Appendix

Table 12: Per-domain micro means with 95% CIs for the trained model on the 180-instance seed set.

Metric	$\boldsymbol{\chi^{2}_{F}}$ (df = 13)	$\boldsymbol{p}$ -value	$\boldsymbol{W}$	$k$ models	$n$ instances
BLEU	1862.52	$<0.001$	0.81	14	177
ROUGE-L	1506.76	$<0.001$	0.65	14	177
METEOR	1727.50	$<0.001$	0.75	14	177
R-GED	1702.32	$<0.001$	0.74	14	177

Appendix

Table 13: Friedman tests and Kendall’s

W

over

k=14

models and

n=177

diagrams.

ID	Guideline	OK	KO	Pass (%) [95% CI]
2	Minimize model size	179	0	100.00 [97.90, 100.00]
16	Use explicit gateways	173	6	96.65 [92.88, 98.45]
18	Split and join flows consistently	175	4	97.77 [94.40, 99.13]
20	Use meaningful gateways	178	1	99.44 [96.90, 99.90]
21	Minimize inclusive OR gateways	81	98	45.25 [38.14, 52.57]
22	Use default flows	174	5	97.21 [93.63, 98.80]

Appendix

Table 14: BEBoP verification results on 179 generated BPMN models. Pass rates are reported as mean and 95% Wilson confidence interval.

Business Process Modeling

See all 18 papers

Assessing the Business Process Modeling Competences of Large Language Models
Jan 29, 2026Chantale Lauer, Peter Pfeiffer, Alexander Rombach +1LLM Evaluation Business Process Modeling
German Research Center for Artificial Intelligence (DFKI), Campus D3 2, Saarbrücken, 66123, Germany · Saarland University, Campus D3 2, Saarbrücken, 66123, Germany · 5Plus GmbH, Goethestraße 1, Würzburg, 97072, Germany
Improving LLM-Generated Process Model Quality Through Reinforcement Learning: The Role of Reward Function Design
Jul 7, 2026Alexander Rombach, Chantale Lauer, Nijat MehdiyevRL for Language Models Multi-Objective Language Model Alignment
German Research Center for Artificial Intelligence (DFKI), Campus D3 2, Saarbrücken, 66123, Saarland, Germany · Saarland University, Campus D3 2, Saarbrücken, 66123, Saarland, Germany
Automatic Generation of Executable BPMN Models from Medical Guidelines
Apr 9, 2026Praveen Kumar Menaka Sekar, Ion Matei, Maksym Zhenirovskyy +4Business Process Modeling Business Process Automation
Dept. of Mechanical Engineering, University of Maryland, College Park, MD 20742, USA · Fujitsu Research of America, Santa Clara, CA 95054, USA · Fujitsu Limited, Nakahara-ku, Kawasaki, Kanagawa 211-8588, Japan

Jan 29, 2026J/K move · Enter open · S save

Assessing the Business Process Modeling Competences of Large Language Models

Chantale Lauer, Peter Pfeiffer, Alexander Rombach, Nijat Mehdiyev

German Research Center for Artificial Intelligence (DFKI), Campus D3 2, Saarbrücken, 66123, Germany · Saarland University, Campus D3 2, Saarbrücken, 66123, Germany · 5Plus GmbH, Goethestraße 1, Würzburg, 97072, Germany

LLM Evaluation Business Process Modeling

Open paper AskPDF arXiv

The creation of Business Process Model and Notation (BPMN) models is a complex and time-consuming task requiring both domain knowledge and proficiency in modeling conventions. Recent advances in large language models (LLMs) have significantly expanded the possibilities for generating BPMN models directly from natural language, building upon earlier text-to-process methods with enhanced capabilities in handling complex descriptions. However, there is a lack of systematic evaluations of LLM-generated process models. Current efforts either use LLM-as-a-judge approaches or do not consider established dimensions of model quality. To this end, we introduce BEF4LLM, a novel LLM evaluation framework comprising four perspectives: syntactic quality, pragmatic quality, semantic quality, and validity. Using BEF4LLM, we conduct a comprehensive analysis of open-source LLMs and benchmark their performance against human modeling experts. Results indicate that LLMs excel in syntactic and pragmatic quality, while humans outperform LLMs in semantic aspects; however, the differences in scores are relatively modest, highlighting LLMs' competitive potential despite challenges in validity and semantic quality. The insights highlight current strengths and limitations of using LLMs for BPMN modeling and guide future model development and fine-tuning. Addressing these areas is essential for advancing the practical deployment of LLMs in business process modeling.

Abstract

Business Process Modeling

Explore similar work

Assessing the Business Process Modeling Competences of Large Language Models

Improving LLM-Generated Process Model Quality Through Reinforcement Learning: The Role of Reward Function Design

Automatic Generation of Executable BPMN Models from Medical Guidelines