Towards Standardized Evaluation in Automated Domain Modeling: Introducing a Benchmark

Vasiliy Seibert

cs.AIAug 15, 2026

Towards Standardized Evaluation in Automated Domain Modeling: Introducing a Benchmark

Authors: Vasiliy Seibert

Organizations: Institute for Software and Systems Engineering, TU Clausthal Clausthal-Zellerfeld, Lower Saxony, Germany

Abstract

Domain modeling plays an essential role in domain-driven design, capturing essential entities and their relationships within a specific domain. Despite advancements in automated domain modeling, the absence of standardized benchmarks has hindered the comparative assessment of existing approaches. This paper introduces a benchmark designed to address this gap. The benchmark combines the 45-record Golden UML Modelset (Verbruggen et al., 2025) on Zenodo, as distributed by the Text2UML project of Calamo, Mecella, and Snoeck (Calamo et al., 2025), with the 8-record reference archive of Chen et al. (Chen et al., 2023a,b), enabling the evaluation of automated domain modeling approaches across different levels of complexity and scale. Given a natural language description, the task is to generate a corresponding domain model. For each description, a reference domain model is provided as ground truth. A metric is used to compare the generated domain model with the corresponding ground-truth model. To demonstrate the utility of the benchmark, we evaluate multiple automated domain modeling approaches, including heuristic rule-based methods and LLM-driven strategies. In accordance with the FAIR4RS recommendations (Chue Hong et al., 2022), the benchmark is provided as a research artifact to encourage reuse and support future research on automated domain modeling.

Figures & tables

Figure 1. Overview of the benchmark: a (natural-language text, reference PlantUML class diagram) pair is fed to a candidate strategy, which produces a candidate PlantUML diagram that a metric compares against the reference.

	DS-1 (45)	DS-2 (8)	Total (53)
NLT words (mean)	316	423	—
Classes (mean / total)	9.4 / 425	14.4 / 115	540
Attributes (mean / total)	13.0 / 585	10.3 / 82	667
Enums (mean / total)	2.4 / 109	9.4 / 75	184
Relationships (mean / total)	11.9 / 537	21.6 / 173	710
Association ( -- )	268	87	355

Table 1. Per-dataset overview.

Candidate	LLM	Fail %	Class	Attribute	Relationships
Rule-based ( Abdelnabi et al., 2020 )	—	0.0	0.417 $\pm$ 0.000	0.447 $\pm$ 0.000	0.348 $\pm$ 0.000
Chen et al. ( Chen et al., 2023b ; Chen et al., 2023a )	GLM-5.1 ( Zhipu AI, 2025 )	0.0	0.746 $\pm$ 0.006	0.780 $\pm$ 0.006	0.676 $\pm$ 0.007
Chen et al. ( Chen et al., 2023b ; Chen et al., 2023a )	Kimi K2 ( Moonshot AI, 2025 )	8.9	0.699 $\pm$ 0.013	0.717 $\pm$ 0.014	0.641 $\pm$ 0.014
Calamo et al. ( Calamo et al., 2025 ; Calamo et al., 2026 )	GLM-5.1 ( Zhipu AI, 2025 )	2.2	0.694 $\pm$ 0.024	0.691 $\pm$ 0.025	0.701 $\pm$ 0.024
Calamo et al. ( Calamo et al., 2025 ; Calamo et al., 2026 )	Kimi K2 ( Moonshot AI, 2025 )	4.4	0.613 $\pm$ 0.015	0.603 $\pm$ 0.016	0.637 $\pm$ 0.015

Table 2. Benchmark results on DS-1 (45 records). Scores are 3-run median

\pm

cross-run std; Fail % is the median percentage of records that failed validation. Best results in bold.

Candidate	LLM	Fail %	Class	Attribute	Relationships
Rule-based ( Abdelnabi et al., 2020 )	—	0.0	0.312 $\pm$ 0.000	0.353 $\pm$ 0.000	0.215 $\pm$ 0.000
Chen et al. ( Chen et al., 2023b ; Chen et al., 2023a )	GLM-5.1 ( Zhipu AI, 2025 )	12.5	0.625 $\pm$ 0.013	0.650 $\pm$ 0.014	0.576 $\pm$ 0.030
Chen et al. ( Chen et al., 2023b ; Chen et al., 2023a )	Kimi K2 ( Moonshot AI, 2025 )	25.0	0.596 $\pm$ 0.057	0.603 $\pm$ 0.053	0.581 $\pm$ 0.066
Calamo et al. ( Calamo et al., 2025 ; Calamo et al., 2026 )	GLM-5.1 ( Zhipu AI, 2025 )	0.0	0.655 $\pm$ 0.062	0.640 $\pm$ 0.064	0.689 $\pm$ 0.059
Calamo et al. ( Calamo et al., 2025 ; Calamo et al., 2026 )	Kimi K2 ( Moonshot AI, 2025 )	0.0	0.506 $\pm$ 0.080	0.487 $\pm$ 0.070	0.548 $\pm$ 0.104

Table 3. Benchmark results on DS-2 (8 records). Scores are 3-run median

\pm

cross-run std; Fail % is the median percentage of records that failed validation. Best results in bold.

Benchmark Design

Benchmark design creates tasks, datasets, metrics, and protocols that make AI capabilities measurable and comparisons credible.

See all 389 papers

Are We Making Progress in Multimodal Domain Generalization? A Comprehensive Benchmark Study
May 7, 2026Hao Dong, Hongzhao Li, Shupan Li +3Multimodal Robustness Multimodal Domain Generalization
ETH Zürich · Zhengzhou University · MBZUAI +1
BenchBench: Benchmarking Automated Benchmark Generation
Mar 21, 2026Yandan Zheng, Haoran Luo, Zhenghong Lin +2Benchmark Design Synthetic Benchmark Generation
Nanyang Technological University Singapore, Singapore
CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Benchmarks
Jun 2, 2026Alexander Apartsin, Yehudit ApersteinModel Selection LLM Evaluation
Holon Institute of Technology, Israel · Afeka Tel Aviv Academic College of Engineering, Israel

May 7, 2026J/K move · Enter open · S save

Are We Making Progress in Multimodal Domain Generalization? A Comprehensive Benchmark Study

Hao Dong, Hongzhao Li, Shupan Li, Muhammad Haris Khan+2

ETH Zürich · Zhengzhou University · MBZUAI+1

Multimodal Robustness Multimodal Domain Generalization Domain Generalization Multimodal Model Evaluation

Open paper AskPDF arXiv

Despite the growing popularity of Multimodal Domain Generalization (MMDG) for enhancing model robustness, it remains unclear whether reported performance gains reflect genuine algorithmic progress or are artifacts of inconsistent evaluation protocols. Current research is fragmented, with studies varying significantly across datasets, modality configurations, and experimental settings. Furthermore, existing benchmarks focus predominantly on action recognition, often neglecting critical real-world challenges such as input corruptions, missing modalities, and model trustworthiness. This lack of standardization obscures a reliable assessment of the field's advancement. To address this issue, we introduce MMDG-Bench, the first unified and comprehensive benchmark for MMDG, which standardizes evaluation across six datasets spanning three diverse tasks: action recognition, mechanical fault diagnosis, and sentiment analysis. MMDG-Bench encompasses six modality combinations, nine representative methods, and multiple evaluation settings. Beyond standard accuracy, it systematically assesses corruption robustness, missing-modality generalization, misclassification detection, and out-of-distribution detection. With 7, 402 neural networks trained in total across 95 unique cross-domain tasks, MMDG-Bench yields five key findings: (1) under fair comparisons, recent specialized MMDG methods offer only marginal improvements over ERM baseline; (2) no single method consistently outperforms others across datasets or modality combinations; (3) a substantial gap to upper-bound performance persists, indicating that MMDG remains far from solved; (4) trimodal fusion does not consistently outperform the strongest bimodal configurations; and (5) all evaluated methods exhibit significant degradation under corruption and missing-modality scenarios, with some methods further compromising model trustworthiness.

Abstract

Benchmark Design

Explore similar work

Are We Making Progress in Multimodal Domain Generalization? A Comprehensive Benchmark Study

BenchBench: Benchmarking Automated Benchmark Generation

CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Benchmarks