Instruction-Tuning Open-Weight Language Models for BPMN Model Generation

Gökberk Çelikmasat; Atay Özgövde; Fatma Başak Aydemir

cs.SEDec 12, 2025

Instruction-Tuning Open-Weight Language Models for BPMN Model Generation

Authors: Gökberk Çelikmasat, Atay Özgövde, Fatma Başak Aydemir

Organizations: Boğaziçi University, İstanbul, Türkiye · Utrecht University, Utrecht, The Netherlands

Abstract

Domain models are central to software engineering, as they enable a shared understanding, guide implementation, and support automated analyses and model-driven development. Yet, despite these benefits, practitioners often skip modeling because it is time-consuming and demands scarce expertise. We address this barrier by investigating whether open-weight large language models, adapted via instruction tuning, can generate high-quality BPMN process models directly from natural language descriptions in a cost-effective and privacy-preserving way. We introduce InstruBPM, a reproducible approach that prepares paired text-diagram data and instruction tunes an open source large language model with parameter-efficient fine-tuning and quantization for on-prem deployment. We evaluate the tuned model through complementary perspectives: (i) text/code similarity using BLEU, ROUGE-L, and METEOR, (ii) structural fidelity using Relative Graph Edit Distance, (iii) guidelines conformance using external tool checks, and (iv) a small expert review. Using a curated subset of a multi-domain BPMN dataset, we compare the tuned model with untuned open-weight baselines and strong proprietary models under consistent prompting regimes. Our compact tuned model outperforms all baselines across sequence and structural metrics while requiring substantially fewer resources; guideline analysis and expert feedback further indicate that the generated diagrams largely follow BPMN best practices and are useful starting points that reduce modeling effort. Overall, instruction tuning improves structural accuracy and robustness compared to untuned baselines and reduces reliance on heavy prompt scaffolding. We publicly share the trained models and scripts to support reproducibility and further research.

Figures & tables

Approach	Model	Adapt.	Target	Evaluation
Grohs et al. [ 21 ]	GPT-4	Prompt	BPMN/RPA	Qualitative
Klievtsova et al. [ 27 ]	GPT-3.5/4	Prompt	Dialogue/extraction	Empirical
Licardo et al. [ 39 ]	GPT-4+BERT	Prompt+FT	BPMN	R-GED
Nivon & Salaün [ 44 ]	GPT-3.5	FT	Grammar/AST to BPMN	Text + structural
Bernardi et al. [ 4 ]	LLaMA	FT+RAG	BPMN Q&A	Retrieval metrics
Kourani et al. [ 29 ]	GPT-4	Prompt	BPMN/Petri nets	Manual + structural

Table 1: Representative LLM-based BPM approaches.

Figure 1: Design Science Research methodology adopted in this study.

Panel of Figure 2 — Figure 2: End-to-end instruction-tuning example adapted from [ 10 ] : (a) instruction template, (b) sample textual description, (c) model output in DOT, and (d) rendered BPMN model.

ID	Current Role	Years of BPM Experience
E1	System Analyst	2
E2	Business Analyst	4
E3	Workflow Automation Engineer	6
E4	Lead Business Analyst	10

Table 2: Demographics of Experts

DOT element	BPMN 2.0 element
START_NODE (circle)	bpmn2:startEvent
END_NODE (double circle)	bpmn2:endEvent
Box node (labeled)	bpmn2:task
Diamond [label="+"]	bpmn2:parallelGateway
Diamond [label="X"]	bpmn2:exclusiveGateway
Diamond [label="O"]	bpmn2:inclusiveGateway

Table 3: DOT-to-BPMN 2.0 element mapping used by the converter.

Category	ID	Description
General	2	Smaller models improve understandability and reduce cognitive load.
Notation	16	The designer should split or join sequence flows always using gateways.
	18	Match split/join pairs to preserve structuredness.
	20	Do not represent gateways with only one incoming and one outgoing sequence flow.
	21	Minimize the use of inclusive (OR) gateways, which are difficult to synchronize.
	22	Avoid dead ends at exclusive gateways; provide a default branch.

Table 4: Selected BEBoP guidelines, grouped by BEBoP’s own categories, with brief rationale.

Open-Weight LLMs
Models	BLEU	ROUGE-L	METEOR	R-GED
Qwen2.5-7B-Instruct	$5.67$	$43.11$	$45.10$	$37.94$
Qwen2.5-14B-Instruct	$6.66$	$43.01$	$42.96$	$40.00$
Qwen2.5-Coder-7B-Instruct	$6.08$	$41.34$	$39.38$	$38.48$
Qwen2.5-Coder-14B-Instruct	$6.76$	$43.30$	$43.99$	$42.10$
Qwen3-4B-Instruct-2507	$2.89$	$40.31$	$44.16$	$44.47$

Table 5: BPMN Model Generation Textual and Structural Similarity Evaluation Results

Domain	BLEU	ROUGE-L	METEOR	R-GED
Account payable process	$84.02$	$94.69$	$93.65$	$100.0$
Accounts receivable process	$85.62$	$95.64$	$94.17$	$100.0$
Budget preparation process	$81.23$	$94.41$	$92.52$	$100.0$
Churn-rate prevention process	$87.34$	$96.10$	$94.94$	$100.0$
Client onboarding (marketing agency)	$83.63$	$95.49$	$93.31$	$100.0$
Content promotion process	$82.25$	$94.66$	$92.78$	$100.0$

Table 6: Per-domain BPMN Model Generation Average Textual Similarity Results.

Technique	BLEU	ROUGE-L	METEOR
Zero-shot	83.06	94.43	92.82
Chain-of-thought	81.92	93.79	91.25
Tree-of-thoughts	81.80	93.72	91.16

Table 7: Impact of Prompt Engineering Techniques on Trained Model

Training Mode	BLEU	ROUGE-L	METEOR	Model Size (GB)	Throughput (tokens/sec)	Avg Time to 1st Token (s)
Full-Lora-Tuning (BF16)	$83.06$	$94.43$	$92.82$	$\sim$ 8.05	$35.73$	$0.6423$
Q8_0-GGUF	$83.08$	$94.41$	$92.83$	$\sim$ 4.28	$31.04$	$0.6052$
Q6_K-GGUF	$83.05$	$94.39$	$92.77$	$\sim$ 3.31	$29.45$	$0.7249$
Q5_K_M-GGUF	$82.93$	$94.33$	$92.75$	$\sim$ 2.89	$29.30$	$0.6710$
Q4_K_M-GGUF	$80.42$	$93.22$	$91.03$	$\sim$ 2.50	$30.05$	$0.5502$
Q3_K_M-GGUF	$75.58$	$90.59$	$87.58$	$\sim$ 2.08	$28.46$	$0.9269$

Table 8: Quantization and Model Size Impact on BPMN Generation

Alpha ( $\alpha$ )	BLEU	ROUGE-L	METEOR	R-GED (%)
8	41.24	51.13	58.50	55.07
16	74.47	87.77	85.60	98.23
32	83.06	94.43	92.82	99.44
64	14.09	33.78	30.98	31.09

Table 9: Effect of merge-time LoRA alpha on model performance (rank = 16).

Panel of Figure 5 — Figure 5: Effect of LoRA merge alpha ( $\alpha$ ) on evaluation metrics with rank = 16 adapters.

ID	Guideline	OK	KO	Pass (%)
2	Minimize model size	179	0	100.00
16	Use explicit gateways	173	6	96.65
18	Split and join flows consistently	175	4	97.77
20	Use meaningful gateways	178	1	99.44
21	Minimize inclusive OR gateways	81	98	45.25
22	Use default flows	174	5	97.21

Table 10: Verification results on 179 generated BPMN models.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Reference and generated BPMN diagrams illustrating an omission case.

Open-weight LLMs
Models	BLEU	ROUGE-L	METEOR	R-GED
Qwen2.5-7B-Instruct	5.67 [ 4.85, 6.82]	43.11 [41.44, 44.89]	45.10 [42.17, 47.51]	37.94 [32.70, 44.14]
Qwen2.5-14B-Instruct	6.66 [ 5.63, 8.02]	43.01 [41.12, 44.98]	42.96 [39.47, 46.26]	40.00 [34.92, 45.79]
Qwen2.5-Coder-7B-Instruct	6.08 [ 5.30, 6.95]	41.34 [40.01, 42.61]	39.38 [36.41, 42.09]	38.48 [32.99, 44.32]
Qwen2.5-Coder-14B-Instruct	6.76 [ 5.76, 8.01]	43.30 [41.80, 44.85]	43.99 [41.40, 46.21]	42.10 [37.92, 46.95]
Qwen3-4B-Instruct-2507	2.89 [ 2.16, 4.02]	40.31 [38.70, 42.03]	44.16 [41.52, 46.67]	44.47 [39.38, 50.08]

Appendix

Table 11: BPMN model generation: macro means with 95% confidence intervals on the 180-instance seed set.

Domain	BLEU	ROUGE-L	METEOR	R-GED
Account payable process	84.0 [77.9, 89.2]	94.7 [92.6, 96.4]	93.6 [91.4, 95.4]	100.0 [100.0, 100.0]
Accounts receivable process	85.6 [80.6, 89.5]	95.6 [93.7, 97.1]	94.2 [91.6, 96.0]	100.0 [100.0, 100.0]
Budget preparation process	81.2 [77.4, 84.2]	94.4 [93.1, 95.4]	92.5 [90.8, 94.0]	100.0 [100.0, 100.0]
Churn-rate prevention process	87.3 [83.6, 90.2]	96.1 [95.0, 97.0]	94.9 [93.5, 96.1]	100.0 [100.0, 100.0]
Client onboarding (marketing agency)	83.6 [79.3, 87.2]	95.5 [94.3, 96.5]	93.3 [91.4, 94.8]	100.0 [100.0, 100.0]
Content promotion process	82.2 [77.3, 86.2]	94.7 [93.3, 95.8]	92.8 [90.8, 94.4]	100.0 [100.0, 100.0]

Appendix

Table 12: Per-domain micro means with 95% CIs for the trained model on the 180-instance seed set.

Metric	$\boldsymbol{\chi^{2}_{F}}$ (df = 13)	$\boldsymbol{p}$ -value	$\boldsymbol{W}$	$k$ models	$n$ instances
BLEU	1862.52	$<0.001$	0.81	14	177
ROUGE-L	1506.76	$<0.001$	0.65	14	177
METEOR	1727.50	$<0.001$	0.75	14	177
R-GED	1702.32	$<0.001$	0.74	14	177

Appendix

Table 13: Friedman tests and Kendall’s

W

over

k=14

models and

n=177

diagrams.

ID	Guideline	OK	KO	Pass (%) [95% CI]
2	Minimize model size	179	0	100.00 [97.90, 100.00]
16	Use explicit gateways	173	6	96.65 [92.88, 98.45]
18	Split and join flows consistently	175	4	97.77 [94.40, 99.13]
20	Use meaningful gateways	178	1	99.44 [96.90, 99.90]
21	Minimize inclusive OR gateways	81	98	45.25 [38.14, 52.57]
22	Use default flows	174	5	97.21 [93.63, 98.80]

Appendix

Table 14: BEBoP verification results on 179 generated BPMN models. Pass rates are reported as mean and 95% Wilson confidence interval.

Business Process Modeling

See all 18 papers

Jan 29, 2026

Assessing the Business Process Modeling Competences of Large Language Models

The creation of Business Process Model and Notation (BPMN) models is a complex and time-consuming task requiring both domain knowledge and proficiency in modeling conventions. Recent advances in large language models (LLMs) have significantly expanded the possibilities for generating BPMN models directly from natural language, building upon earlier text-to-process methods with enhanced capabilities in handling complex descriptions. However, there is a lack of systematic evaluations of LLM-generated process models. Current efforts either use LLM-as-a-judge approaches or do not consider established dimensions of model quality. To this end, we introduce BEF4LLM, a novel LLM evaluation framework comprising four perspectives: syntactic quality, pragmatic quality, semantic quality, and validity. Using BEF4LLM, we conduct a comprehensive analysis of open-source LLMs and benchmark their performance against human modeling experts. Results indicate that LLMs excel in syntactic and pragmatic quality, while humans outperform LLMs in semantic aspects; however, the differences in scores are relatively modest, highlighting LLMs' competitive potential despite challenges in validity and semantic quality. The insights highlight current strengths and limitations of using LLMs for BPMN modeling and guide future model development and fine-tuning. Addressing these areas is essential for advancing the practical deployment of LLMs in business process modeling.

Chantale Lauer, Peter Pfeiffer, Alexander Rombach +1

German Research Center for Artificial Intelligence (DFKI), Campus D3 2, Saarbrücken, 66123, Germany · Saarland University, Campus D3 2, Saarbrücken, 66123, Germany · 5Plus GmbH, Goethestraße 1, Würzburg, 97072, Germany

LLM Evaluation Business Process Modeling

Ask PDF

Jul 7, 2026

Improving LLM-Generated Process Model Quality Through Reinforcement Learning: The Role of Reward Function Design

Large language models (LLMs) can generate BPMN process models from natural-language descriptions, yet supervised fine-tuning (SFT) limits their output quality to the patterns present in the training data. Reinforcement learning (RL) can optimize beyond this ceiling using external quality measures, but how the reward function should be designed when quality is multi-dimensional remains unexplored. We present a systematic investigation of reward function design for RL-based process model generation, training two LLM families (Llama~~3.1 8B, Qwen~~2.5 14B) under 48 configurations using Group Sequence Policy Optimization with rewards derived from an automated evaluation framework comprising 38 metrics across syntactic, pragmatic, and semantic quality. Three findings emerge. First, RL significantly improves pragmatic and syntactic quality while preserving semantic fidelity, reducing output variability by more than sixfold. Second, equal reward weighting consistently outperforms targeted weighting: emphasizing a specific dimension fails to improve it and can collapse the model into a low-quality mode. Third, design choices interact with model architecture in non-trivial ways: the invalidity penalty is essential for one model but irrelevant for the other, and SFT initialization is indispensable for one architecture but counterproductive for another. These results demonstrate that reward composition is a primary determinant of optimization outcomes, with effects as large as the decision to apply RL itself. The findings generalize to any structured generation task where quality is assessed along multiple automated dimensions. We release our implementation and experimental code at https://github.com/chlauer99/RL_for_process_modeling.

Alexander Rombach, Chantale Lauer, Nijat Mehdiyev

German Research Center for Artificial Intelligence (DFKI), Campus D3 2, Saarbrücken, 66123, Saarland, Germany · Saarland University, Campus D3 2, Saarbrücken, 66123, Saarland, Germany

RL for Language Models Multi-Objective Language Model Alignment RL Fine-Tuning Reward Design for RL Business Process Modeling

Ask PDF

Apr 9, 2026

Automatic Generation of Executable BPMN Models from Medical Guidelines

We present an end-to-end pipeline that converts healthcare policy documents into executable, data-aware Business Process Model and Notation (BPMN) models using large language models (LLMs) for simulation-based policy evaluation. We address the main challenges of automated policy digitization with four contributions: data-grounded BPMN generation with syntax auto-correction, executable augmentation, KPI instrumentation, and entropy-based uncertainty detection. We evaluate the pipeline on diabetic nephropathy prevention guidelines from three Japanese municipalities, generating 100 models per backend across three LLMs and executing each against 1,000 synthetic patients. On well-structured policies, the pipeline achieves a 100% ground-truth match with perfect per-patient decision agreement. Across all conditions, raw per-patient decision agreement exceeds 92%, and entropy scores increase monotonically with document complexity, confirming that the detector reliably separates unambiguous policies from those requiring targeted human clarification.

Praveen Kumar Menaka Sekar, Ion Matei, Maksym Zhenirovskyy +4

Dept. of Mechanical Engineering, University of Maryland, College Park, MD 20742, USA · Fujitsu Research of America, Santa Clara, CA 95054, USA · Fujitsu Limited, Nakahara-ku, Kawasaki, Kanagawa 211-8588, Japan

Business Process Modeling Business Process Automation

Ask PDF

Abstract

Business Process Modeling

Explore similar work

Assessing the Business Process Modeling Competences of Large Language Models

Improving LLM-Generated Process Model Quality Through Reinforcement Learning: The Role of Reward Function Design

Automatic Generation of Executable BPMN Models from Medical Guidelines