The rapid emergence of large language models (LLMs), trained on vast and diverse corpora, has positioned them as a promising source of domain knowledge to assist modelers. However, leveraging LLMs for model completion remains challenging: suggestions must respect both the syntax of a domain-specific language (DSL) and the intended domain semantics, while remaining consistent with an evolving model. This paper presents RAFiC, an LLM-based approach for iterative domain model completion that generalizes across DSLs. It drives completion using metamodel information to ensure syntactic validity and enriches prompts with semantic-role annotations to better capture the intended modeling role of DSL elements. The approach provides context-aware, incremental guidance during model construction. We evaluate RAFiC through an ablation-based comparative study over three DSLs and two scenarios per DSL to isolate the contribution of syntactic and semantic guidance. Across these six scenarios, semantic role guidance improves contribution rate in every case, and RAFiC improves completion effectiveness over weaker prompting configurations in most of them. The largest gains occur for DSLs that are less represented in LLM training data. In these cases, iterative guidance leads to earlier and more stable accepted suggestions, while semantic role guidance improves their relevance.
Figures & tables
Figure 1
Figure 2. Ticketing DSL: (a) metamodel, (b) concrete syntax, and (c) example instance.
Figure 3. Semantic annotation scheme used to annotate metamodel elements with their intended modeling role.
Table 4
It.
Rk.
Type
Mini-task
Elements
Reason
S
Iteration 1. Canvas: Ticket TK=T-101 “Login failure”; unnamed Workflow WF containing {TR=Triage, VF=Verify, FX=Fix}; no Roles; no Artifacts.
Table 4. Unified ranking of the next modeling operation on the running example: two-iteration snapshot leading to the model of Fig. 2(c) .
Template
Purpose and instantiated context
Output
ARCH
Proposes candidate names for a focal element type using domain, local instance context, optional semantic hint, known facts, and co-occurrence cues.
1–3 labels
REL
Proposes concise relation labels between a focal element and a trigger element using role phrase, trigger, local context, and optional semantic hint.
1–3 labels
CHAR
Proposes characteristic labels (e.g., attributes, roles, states) for a focal element using relation context, trigger, and optional semantic hints.
1–3 labels
Table 5. Prompt template families used in the approach.
Figure 4. RAFiC in Sirius Web on the Ticketing example: recommendations (left), model under construction (center), and properties of the selected step (right).
Figure 5. Offline evaluation protocol.
Lang.
Scenarios
UML-CD
Medical appointment;
Food delivery application
SCM
Car manufacturing;
Aircraft maintenance routing
SysML-v2
Flashlight assembly;
BatMobile system design
Table 6. Validation test cases.
Cfg
Configuration
Syntax injection
Semantic injection
RAFiC
Tool-based completion
Iterative
Iterative
RAFiC-Abl
Tool-based ablation (no semantics)
Iterative
None
C-Min
Chat baseline (minimal prompt)
None
None
C-Full
Chat baseline (full prompt)
Whole (upfront)
Whole (upfront)
Table 7. Experimental configurations along two prompt-design axes: syntax injection and semantic injection.
DSL
Scen.
RQ1
RQ2
RQ3
RAFiC
C-Min
C-Full
RAFiC-Abl
Ctr. (%)
Cand. (u.)
Red. (%)
Ctr. (%)
Cand. (u.)
Red. (%)
Ctr. (%)
Cand. (u.)
Red. (%)
Ctr. (%)
Cand. (u.)
Red. (%)
UML-CD
Food del.
39%
122
42%
43.3%
20
65.5%
39%
31
65.6%
34%
90
23%
Med. appt.
40%
135
38%
44.8%
33
53.6%
43%
27
56.5%
36%
100
25%
SCM
Table 8. Summary of results across DSLs and scenarios, grouped by configuration.
Figure 6. Cumulative AI-accepted suggestions across modeling progress for the DSLs and scenarios, grouped by DSL.
Work
Year
Target language
LLM
Generation
Tool integration
Syntax inj.
Semantic inj.
Chen et al. (2023b)
2023
UML Class Diagrams
GPT-3.5, GPT-4 (Comm)
Non-iterative, acceptance-loop
No editor integration; offline pipeline
Implicit format constraints
NL descriptions and examples
Cámara et al. (2023)
2023
UML Class Diagrams
ChatGPT (Comm)
Iterative, HITL
No editor integration; PlantUML/USE outputs
OCL constraints
None
Chen et al. (2023a)
2023
Goal Models
GPT-4 (Comm)
Iterative, HITL
No editor integration; interactive ChatGPT prompting
None
Natural-language descriptions
Ben Chaaben et al. (2025)
2024
UML Class Diagrams
GPT-3 (Comm)
Iterative, HITL
MAGDA tool; Eclipse/Sirius-based environment
Implicit constraints
Few-shot examples
Ferrari et al. (2024)
2024
UML Sequence Diagrams
ChatGPT (GPT-3.5/4, Comm)
Non-iterative, acceptance-loop
No editor integration; PlantUML output in PlantText
None
Natural-language requirements
Apvrille and Sultan (2024)
2024
SysML Diagrams
ChatGPT (GPT-3.5, Comm)
Iterative, automated feedback
TTool-AI modeling and verification environment
SysML rules and tool checks
Natural-language specifications
Table 9. Comparison of LLM-assisted domain modeling approaches.
Large language models (LLMs) struggle with relation completion (RC), both with and without retrieval-augmented generation (RAG), particularly when the required information is rare or sparsely represented. To address this, we propose a novel multi-stage paraphrase-guided relation-completion framework, RC-RAG, that systematically incorporates relation paraphrases across multiple stages. In particular, RC-RAG: (a) integrates paraphrases into retrieval to expand lexical coverage of the relation, (b) uses paraphrases to generate relation-aware summaries, and (c) leverages paraphrases during generation to guide reasoning for relation completion. Importantly, our method does not require any model fine-tuning. Experiments with five LLMs on two benchmark datasets show that RC-RAG consistently outperforms several RAG baselines. In long-tail settings, the best-performing LLM augmented with RC-RAG improves by 40.6 Exact Match (EM) points over its standalone performance and surpasses two strong RAG baselines by 16.0 and 13.8 EM points, respectively, while maintaining low computational overhead.
Fahmida Alam, Mihai Surdeanu, Ellen Riloff
Department of Computer Science University of Arizona, USA
Large language models (LLMs) are increasingly used to generate software engineering artifacts written in domain-specific languages (DSLs). Correct DSL instances usually must comply with not only the concrete syntax but also the abstract syntax defined by a metamodel augmented with semantic well-formedness constraints. Existing constrained decoding approaches typically guarantee compliance only with concrete syntax or with a limited set of semantic constraints expressed over the generated text, making metamodel-level constraint consistency difficult to enforce during generation. We present MeMoDecode: a decoding framework that enforces a DSL's concrete syntax, abstract syntax, and well-formedness constraints jointly. Specifically, whenever the LLM generates a complete metamodel concept, MeMoDecode incrementally maps the current output to a partial model and verifies whether it can still be completed into a consistent instance. We combine MeMoDecode with a simple retry mechanism that resumes decoding from the most recent completable snapshot whenever an inconsistency is detected. Under a sound backend verifier, MeMoDecode never performs worse than grammar-constrained decoding on alignment metrics satisfying natural assumptions. Across four DSLs and six open-weight LLMs ranging from 0.6B to 70B parameters, MeMoDecode achieves the highest consistency, improving consistency by 23% on average over unconstrained decoding and by 8% on average over grammar-constrained decoding. It also improves benchmark performance while incurring only modest runtime overhead. These results show that metamodel-guided decoding is an effective approach for enforcing DSL consistency during decoding across diverse DSLs and LLM scales.
Boqi Chen, Aren A. Babikian, José Antonio Hernández López
University of Ottawa, Ottawa, Canada · McGill University, Montreal, Canada · University of Murcia, Murcia, Spain
Large Language Models (LLMs) have rapidly improved in performance across code-related tasks, making their integration into Register Transfer Level (RTL) development increasingly attractive. Mimicking the behavior of inline code assistants, many benchmarks evaluate LLMs' capabilities in code completion, either assessing the generation of entire hardware modules or the completion of a single line within a module. However both of these approaches lack the ability to control the granularity of the code-completion sample size and the syntactic range of completions. To overcome these limitations, we present a framework for language-agnostic rule completion (RuC), a grammar-driven, rule-selectable benchmark generator that automatically produces RTL code-completion tasks from a set of input hardware description sources. RuC uses the target Hardware Description Language (HDL) grammar to mask syntactically defined code regions and prompts a model to regenerate them using the surrounding unmasked code as context, enabling a controlled and scalable evaluation of the domain-specific model's code-understanding capabilities, ranging from assignments to the reconstruction of entire logic blocks. We use RuC to generate two SystemVerilog rule-completion benchmarks from the Tiny Tapeout shuttle TT07 and the CVE2 RISC-V core to demonstrate RuC's applicability to a broad range of designs, and conduct a comparative study of the code completion capabilities of modern open-source LLMs across diverse settings. Results indicate that completion performance strongly depends on the model type, the grammatical structure of the masked region, and the prompting strategy. Specifically, the highest scores are obtained with Fill-in-the-Middle (FIM) prompting. These findings highlight the value of grammar-driven, arbitrarily granular benchmarks for meaningful evaluation of LLM capabilities in RTL development workflows.