Before Blaming the LLM, Diagnose the Workflow: A Role-Level Difficulty Profile for Translating Natural Language into Ontology-Grounded Statements
Organizations: NORCE Norwegian Research Centre, Nygårdsgaten 112, 5008 Bergen, Norway
Abstract
Natural-language-to-domain-specific-language (NL-to-DSL) semantic parsing systems increasingly translate informal user descriptions into structured artifacts such as Structured Query Language (SQL) queries, Application Programming Interface (API) calls, workflows, formal specifications, and ontology-grounded domain-specific language (DSL) statements. This paper focuses on ontology-grounded Assertion Box (ABox) DSL generation: the translation of natural language domain descriptions into assertion-level DSL statements grounded in a pre-existing ontology vocabulary. In this setting, syntactic validity is often easier to achieve than semantic adequacy. Failures are frequently attributed to weak generator models, but recurring sources of difficulty are also structural: insufficient input, weak domain grounding, output template and vocabulary gaps, weak validation, and non-local repair. We propose a role-level difficulty profile for diagnosing difficulty in ontology-grounded NL-to-DSL semantic parsing at the workflow level. The profile organizes diagnostic variables across five recurring workflow roles: Input, Generator, Output, Validator, and Repairer. The seven variables are input sufficiency, context availability, grounding complexity, template fit, vocabulary coverage, validation strength, and error observability. Practitioners construct the difficulty profile through qualitative judgment over available evidence. This experience-informed, non-exhaustive difficulty profile is intended to support workflow diagnosis and guide redesign or human-in-the-loop intervention.
Figures & tables
| Diagnostic variable | Role | High-friction evidence | Redesign lever |
|---|---|---|---|
| Input sufficiency | Input | Missing fields, implicit assumptions, ambiguous intent | Intent gate and clarification |
| Context availability | Generator | Missing terminology descriptions, examples, or modeling conventions | Knowledge base refinement |
| Grounding complexity | Generator | Many plausible nouns, verbs, paths, or templates | Candidate ranking strategy |
| Template fit | Output | Irregular or overlapping assertion patterns | Template or motif library |
| Vocabulary coverage | Output | Missing constructs or forced near-matches | Ontology extension mechanism |
| Validation strength | Validator | Semantic adequacy not deterministically checkable | Layered validation and human review |
Semantic Parsing
Semantic parsing maps natural language into structured representations, such as logical forms, graphs, or queries, that machines can execute or reason over.
See all 30 papers