Metamodel-Guided LLM Decoding for Consistent DSL Generation

Boqi Chen; Aren A. Babikian; José Antonio Hernández López

cs.SEOct 21, 2026

Metamodel-Guided LLM Decoding for Consistent DSL Generation

Authors: Boqi Chen, Aren A. Babikian, José Antonio Hernández López

Organizations: University of Ottawa, Ottawa, Canada · McGill University, Montreal, Canada · University of Murcia, Murcia, Spain

Abstract

Large language models (LLMs) are increasingly used to generate software engineering artifacts written in domain-specific languages (DSLs). Correct DSL instances usually must comply with not only the concrete syntax but also the abstract syntax defined by a metamodel augmented with semantic well-formedness constraints. Existing constrained decoding approaches typically guarantee compliance only with concrete syntax or with a limited set of semantic constraints expressed over the generated text, making metamodel-level constraint consistency difficult to enforce during generation. We present MeMoDecode: a decoding framework that enforces a DSL's concrete syntax, abstract syntax, and well-formedness constraints jointly. Specifically, whenever the LLM generates a complete metamodel concept, MeMoDecode incrementally maps the current output to a partial model and verifies whether it can still be completed into a consistent instance. We combine MeMoDecode with a simple retry mechanism that resumes decoding from the most recent completable snapshot whenever an inconsistency is detected. Under a sound backend verifier, MeMoDecode never performs worse than grammar-constrained decoding on alignment metrics satisfying natural assumptions. Across four DSLs and six open-weight LLMs ranging from 0.6B to 70B parameters, MeMoDecode achieves the highest consistency, improving consistency by 23% on average over unconstrained decoding and by 8% on average over grammar-constrained decoding. It also improves benchmark performance while incurring only modest runtime overhead. These results show that metamodel-guided decoding is an effective approach for enforcing DSL consistency during decoding across diverse DSLs and LLM scales.

Figures & tables

Fig. 2: Example DSL for CLEVR programs [ 37 ]

Fig. 3: Overview of MeMoDecode decoding process; MM: metamodel, WF: well-formedness constraints; PM: partial model

Fig. 4: Different partial models, their verification results (boxed), and color-coded correspondence with syntactic units

Benchmark	Samples	Constraints	Example well-formedness constraints
Non-executable (structural)
Taxonomy	100	6	one root; acyclic; $\leq\!1$ parent/term
Flowchart	326	5	reachability; decisions have $\geq\!2$ guarded out-edges; no self-loops
Executable (operational)
CLEVR	300	17	one source and one sink; acyclic; type consistency; referential integrity
Block	399	11	value type matches property, e.g., height $\to$ integer; type consistency

TABLE I: Summary of evaluation benchmarks.

LLM	Taxonomy						Flowchart						Block						CLEVR
	Unc		Grammar		MeMoDecode		Unc		Grammar		MeMoDecode		Unc		Grammar		MeMoDecode		Unc		Grammar		MeMoDecode
	Syn	Con	Syn	Con	Syn	Con	Syn	Con	Syn	Con	Syn	Con	Syn	Con	Syn	Con	Syn	Con	Syn	Con	Syn	Con	Syn	Con
Qwen3-0.6B	99.7	36.0	100.0	36.0	100.0	66.7	78.5	71.9	99.6	89.1	99.3	98.6	51.4	49.0	100.0	93.7	100.0	98.8	44.3	5.3	99.9	5.6	99.9	31.2
Qwen3-8B	99.3	69.0	99.3	69.0	99.3	78.3	93.8	90.4	100.0	96.1	100.0	96.4	75.0	74.0	100.0	94.9	99.9	95.7	90.8	57.1	99.9	59.8	100.0	65.6
Qwen3-32B	99.7	79.3	99.7	79.3	100.0	88.7	87.7	85.4	100.0	97.2	100.0	97.2	76.8	75.6	100.0	92.6	100.0	96.8	93.4	62.0	100.0	60.3	100.0	67.9
Gemma4-E2B	64.0	22.7	92.7	29.3	90.7	44.3	0.0	0.0	99.7	89.0	99.7	89.6	60.4	55.6	100.0	83.5	100.0	88.6	68.4	19.9	99.2	20.1	99.7	30.8

TABLE II: Syntactic validity ( Syn ) and consistency ( Con ), in %, averaged over three runs, across the four DSL benchmarks. The best consistency per benchmark and LLM is in bold . GPT-5.5 (unconstrained only) is shown as a reference.

Fig. 5: Token efficiency of decoding approaches on Qwen3-8B

LLM	Taxonomy						Flowchart						Block			CLEVR
	Unc		Grammar		MeMoDecode		Unc		Grammar		MeMoDecode		Unc	Grammar	MeMoDecode	Unc	Grammar	MeMoDecode
	F1	F1 ∗	F1	F1 ∗	F1	F1 ∗	F1	F1 ∗	F1	F1 ∗	F1	F1 ∗	Acc	Acc	Acc	Acc	Acc	Acc
Qwen3-0.6B	20.0	8.6	20.0	8.6	19.9	14.4	53.9	49.6	68.1	61.3	68.1	67.7	11.9	18.6	19.3	1.0	1.8	6.1
Qwen3-8B	53.2	36.5	53.2	36.5	53.1	40.7	71.5	69.0	76.2	73.4	76.2	73.7	35.1	38.8	39.0	44.9	46.4	49.8
Qwen3-32B	59.1	46.7	59.0	46.6	59.4	52.3	67.5	65.8	76.9	75.0	76.9	75.0	42.5	45.3	46.2	55.0	53.8	58.7
Gemma4-E2B	26.7	9.1	36.4	11.7	36.6	18.3	0.0	0.0	72.3	66.8	72.3	67.3	30.7	38.9	39.5	12.2	11.8	15.9

TABLE III: Alignment scores (%) per approach across the four benchmarks, best F1 ∗ or accuracy per benchmark and LLM is shown in bold , GPT-5.5 as an unconstrained reference.

Benchmark	W	T	L	$p$
Taxonomy	234	1566	0	$<\!10^{-4}$
Flowchart	109	5759	0	$<\!10^{-4}$
Block	43	7139	0	$<\!10^{-4}$
CLEVR	329	5070	1	$<\!10^{-4}$

TABLE IV: Win/tie/loss of MeMoDecode vs. Grammar (See Theorem 1 ) on the per-sample alignment score, pooled over six LLMs and three runs;

p

is a one-sided Wilcoxon signed-rank test (alternative hypothesis: MeMoDecode

>

Grammar )

Fig. 6: Average runtime breakdown; for each benchmark, the two bars correspond to Grammar and MeMoDecode , respectively

Constrained Decoding

Constrained decoding steers language model outputs to satisfy formal or task-specific rules while balancing validity, quality, and faithful probabilities.

See all 67 papers

The Hidden Cost of Structured Generation in LLMs: Draft-Conditioned Constrained Decoding
Feb 8, 2026Avinash Reddy, Thayne T. Walker, James S. Ide +1Constrained Generative Modeling Constrained Decoding
Department of Computer Science, University of Central Florida, FL, US · Lockheed Martin AI Center, CT, US
Constrained Decoding: Harmful or Beneficial? Revisiting Its Impact on LLM Performance
Oct 26, 2026Ngoc Trinh Hung Nguyen, Alonso Silva, Armen Aghasaryan +1Constrained Decoding Constraint-Guided Reasoning in Language Models
Télécom Paris, Institut Polytechnique de Paris, France · Nokia Bell Labs
Grammar-Aligned Decoding
May 31, 2024Kanghee Park, Jiayu Wang, Taylor Berg-Kirkpatrick +2Constrained Generative Modeling Constrained Decoding
University of Wisconsin-Madison · University of California San Diego

Feb 8, 2026J/K move · Enter open · S save

The Hidden Cost of Structured Generation in LLMs: Draft-Conditioned Constrained Decoding

Avinash Reddy, Thayne T. Walker, James S. Ide, Amrit Singh Bedi

Department of Computer Science, University of Central Florida, FL, US · Lockheed Martin AI Center, CT, US

Constrained Generative Modeling Constrained Decoding LLM Inference Structured Output Generation

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used to generate executable outputs, JSON objects, and API calls, where a single syntax error can make the output unusable. Constrained decoding enforces validity token-by-token via masking and renormalization, but it can distort generation when the model assigns low probability mass to valid continuations, pushing decoding toward locally valid yet semantically incorrect trajectories. We propose Draft-Conditioned Constrained Decoding (DCCD), a simple two-step, training-free inference procedure that decouples semantic planning from structural enforcement: an unconstrained draft is generated first, and constrained decoding is then applied, conditioned on this draft, to guarantee validity. We analyze DCCD through a KL-projection view, showing that draft conditioning increases feasible mass and reduces the cumulative "projection tax" induced by hard constraints, with an optional best-of-

K

draft selection. Across structured reasoning benchmarks, DCCD improves strict structured accuracy by up to +24 percentage points over standard constrained decoding (e.g., 15.2% to 39.0% on GSM8K with a 1B model), and enables smaller model pairs to match or exceed much larger constrained baselines, yielding substantial gains in parameter efficiency.

Abstract

Constrained Decoding

Explore similar work

The Hidden Cost of Structured Generation in LLMs: Draft-Conditioned Constrained Decoding

Constrained Decoding: Harmful or Beneficial? Revisiting Its Impact on LLM Performance

Grammar-Aligned Decoding