Metamodel-Guided LLM Decoding for Consistent DSL Generation

Boqi Chen; Aren A. Babikian; José Antonio Hernández López

cs.SEOct 21, 2026

Metamodel-Guided LLM Decoding for Consistent DSL Generation

Authors: Boqi Chen, Aren A. Babikian, José Antonio Hernández López

Organizations: University of Ottawa, Ottawa, Canada · McGill University, Montreal, Canada · University of Murcia, Murcia, Spain

Abstract

Large language models (LLMs) are increasingly used to generate software engineering artifacts written in domain-specific languages (DSLs). Correct DSL instances usually must comply with not only the concrete syntax but also the abstract syntax defined by a metamodel augmented with semantic well-formedness constraints. Existing constrained decoding approaches typically guarantee compliance only with concrete syntax or with a limited set of semantic constraints expressed over the generated text, making metamodel-level constraint consistency difficult to enforce during generation. We present MeMoDecode: a decoding framework that enforces a DSL's concrete syntax, abstract syntax, and well-formedness constraints jointly. Specifically, whenever the LLM generates a complete metamodel concept, MeMoDecode incrementally maps the current output to a partial model and verifies whether it can still be completed into a consistent instance. We combine MeMoDecode with a simple retry mechanism that resumes decoding from the most recent completable snapshot whenever an inconsistency is detected. Under a sound backend verifier, MeMoDecode never performs worse than grammar-constrained decoding on alignment metrics satisfying natural assumptions. Across four DSLs and six open-weight LLMs ranging from 0.6B to 70B parameters, MeMoDecode achieves the highest consistency, improving consistency by 23% on average over unconstrained decoding and by 8% on average over grammar-constrained decoding. It also improves benchmark performance while incurring only modest runtime overhead. These results show that metamodel-guided decoding is an effective approach for enforcing DSL consistency during decoding across diverse DSLs and LLM scales.

Figures & tables

Fig. 2: Example DSL for CLEVR programs [ 37 ]

Fig. 3: Overview of MeMoDecode decoding process; MM: metamodel, WF: well-formedness constraints; PM: partial model

Fig. 4: Different partial models, their verification results (boxed), and color-coded correspondence with syntactic units

Benchmark	Samples	Constraints	Example well-formedness constraints
Non-executable (structural)
Taxonomy	100	6	one root; acyclic; $\leq\!1$ parent/term
Flowchart	326	5	reachability; decisions have $\geq\!2$ guarded out-edges; no self-loops
Executable (operational)
CLEVR	300	17	one source and one sink; acyclic; type consistency; referential integrity
Block	399	11	value type matches property, e.g., height $\to$ integer; type consistency

TABLE I: Summary of evaluation benchmarks.

LLM	Taxonomy						Flowchart						Block						CLEVR
	Unc		Grammar		MeMoDecode		Unc		Grammar		MeMoDecode		Unc		Grammar		MeMoDecode		Unc		Grammar		MeMoDecode
	Syn	Con	Syn	Con	Syn	Con	Syn	Con	Syn	Con	Syn	Con	Syn	Con	Syn	Con	Syn	Con	Syn	Con	Syn	Con	Syn	Con
Qwen3-0.6B	99.7	36.0	100.0	36.0	100.0	66.7	78.5	71.9	99.6	89.1	99.3	98.6	51.4	49.0	100.0	93.7	100.0	98.8	44.3	5.3	99.9	5.6	99.9	31.2
Qwen3-8B	99.3	69.0	99.3	69.0	99.3	78.3	93.8	90.4	100.0	96.1	100.0	96.4	75.0	74.0	100.0	94.9	99.9	95.7	90.8	57.1	99.9	59.8	100.0	65.6
Qwen3-32B	99.7	79.3	99.7	79.3	100.0	88.7	87.7	85.4	100.0	97.2	100.0	97.2	76.8	75.6	100.0	92.6	100.0	96.8	93.4	62.0	100.0	60.3	100.0	67.9
Gemma4-E2B	64.0	22.7	92.7	29.3	90.7	44.3	0.0	0.0	99.7	89.0	99.7	89.6	60.4	55.6	100.0	83.5	100.0	88.6	68.4	19.9	99.2	20.1	99.7	30.8

TABLE II: Syntactic validity ( Syn ) and consistency ( Con ), in %, averaged over three runs, across the four DSL benchmarks. The best consistency per benchmark and LLM is in bold . GPT-5.5 (unconstrained only) is shown as a reference.

Fig. 5: Token efficiency of decoding approaches on Qwen3-8B

LLM	Taxonomy						Flowchart						Block			CLEVR
	Unc		Grammar		MeMoDecode		Unc		Grammar		MeMoDecode		Unc	Grammar	MeMoDecode	Unc	Grammar	MeMoDecode
	F1	F1 ∗	F1	F1 ∗	F1	F1 ∗	F1	F1 ∗	F1	F1 ∗	F1	F1 ∗	Acc	Acc	Acc	Acc	Acc	Acc
Qwen3-0.6B	20.0	8.6	20.0	8.6	19.9	14.4	53.9	49.6	68.1	61.3	68.1	67.7	11.9	18.6	19.3	1.0	1.8	6.1
Qwen3-8B	53.2	36.5	53.2	36.5	53.1	40.7	71.5	69.0	76.2	73.4	76.2	73.7	35.1	38.8	39.0	44.9	46.4	49.8
Qwen3-32B	59.1	46.7	59.0	46.6	59.4	52.3	67.5	65.8	76.9	75.0	76.9	75.0	42.5	45.3	46.2	55.0	53.8	58.7
Gemma4-E2B	26.7	9.1	36.4	11.7	36.6	18.3	0.0	0.0	72.3	66.8	72.3	67.3	30.7	38.9	39.5	12.2	11.8	15.9

TABLE III: Alignment scores (%) per approach across the four benchmarks, best F1 ∗ or accuracy per benchmark and LLM is shown in bold , GPT-5.5 as an unconstrained reference.

Benchmark	W	T	L	$p$
Taxonomy	234	1566	0	$<\!10^{-4}$
Flowchart	109	5759	0	$<\!10^{-4}$
Block	43	7139	0	$<\!10^{-4}$
CLEVR	329	5070	1	$<\!10^{-4}$

TABLE IV: Win/tie/loss of MeMoDecode vs. Grammar (See Theorem 1 ) on the per-sample alignment score, pooled over six LLMs and three runs;

p

is a one-sided Wilcoxon signed-rank test (alternative hypothesis: MeMoDecode

>

Grammar )

Fig. 6: Average runtime breakdown; for each benchmark, the two bars correspond to Grammar and MeMoDecode , respectively

Constrained Decoding

Constrained decoding steers language model outputs to satisfy formal or task-specific rules while balancing validity, quality, and faithful probabilities.

See all 66 papers

Feb 8, 2026

The Hidden Cost of Structured Generation in LLMs: Draft-Conditioned Constrained Decoding

Large language models (LLMs) are increasingly used to generate executable outputs, JSON objects, and API calls, where a single syntax error can make the output unusable. Constrained decoding enforces validity token-by-token via masking and renormalization, but it can distort generation when the model assigns low probability mass to valid continuations, pushing decoding toward locally valid yet semantically incorrect trajectories. We propose Draft-Conditioned Constrained Decoding (DCCD), a simple two-step, training-free inference procedure that decouples semantic planning from structural enforcement: an unconstrained draft is generated first, and constrained decoding is then applied, conditioned on this draft, to guarantee validity. We analyze DCCD through a KL-projection view, showing that draft conditioning increases feasible mass and reduces the cumulative "projection tax" induced by hard constraints, with an optional best-of-

K

draft selection. Across structured reasoning benchmarks, DCCD improves strict structured accuracy by up to +24 percentage points over standard constrained decoding (e.g., 15.2% to 39.0% on GSM8K with a 1B model), and enables smaller model pairs to match or exceed much larger constrained baselines, yielding substantial gains in parameter efficiency.

Avinash Reddy, Thayne T. Walker, James S. Ide +1

Department of Computer Science, University of Central Florida, FL, US · Lockheed Martin AI Center, CT, US

Constrained Generative Modeling Constrained Decoding LLM Inference Structured Output Generation

Ask PDF

Oct 26, 2026

Constrained Decoding: Harmful or Beneficial? Revisiting Its Impact on LLM Performance

Constrained decoding guarantees that Large Language Models (LLMs) produce outputs conforming to predefined formats, but is often assumed to inherently degrade reasoning. We examine this assumption through a taxonomy of three constraint types: format constraints, which enforce output structure; validity constraints, which mask invalid answer; and reasoning constraints, which restrict reasoning trajectories. We theoretically show that appropriate constraints can preserve or improve performance by eliminating invalid paths. Across four model families and eleven open-source models, causal ablations on diverse benchmarks using state-of-the-art constrained decoding strategies reveal that apparent performance degradation largely stems from non-equivalent prompting and output formats rather than from decoder-level masking. Meanwhile, validity constraints improve reliability and extractability by eliminating invalid answers, while reasoning constraints can improve reasoning and, consequently, accuracy when they enforce beneficial trajectories. These findings show that the impact of constrained decoding depends on the type of constraint and the output space it removes. Our code is available.

Ngoc Trinh Hung Nguyen, Alonso Silva, Armen Aghasaryan +1

Télécom Paris, Institut Polytechnique de Paris, France · Nokia Bell Labs

Constrained Decoding Constraint-Guided Reasoning in Language Models Controllable Text Generation

Ask PDF

May 31, 2024

Grammar-Aligned Decoding

Large Language Models (LLMs) struggle with reliably generating highly structured outputs, such as program code, mathematical formulas, or well-formed markup. Constrained decoding approaches mitigate this problem by greedily restricting what tokens an LLM can output at each step to guarantee that the output matches a given constraint. Specifically, in grammar-constrained decoding (GCD), the LLM's output must follow a given grammar. In this paper, we demonstrate that GCD techniques (and in general constrained decoding techniques) can distort the LLM's distribution, leading to outputs that are grammatical but appear with likelihoods that are not proportional to the ones given by the LLM, and so ultimately are low-quality. We call the problem of aligning sampling with a grammar constraint, grammar-aligned decoding (GAD), and propose adaptive sampling with approximate expected futures (ASAp), a decoding algorithm that guarantees the output to be grammatical while provably producing outputs that match the conditional probability of the LLM's distribution conditioned on the given grammar constraint. Our algorithm uses prior sample outputs to soundly overapproximate the future grammaticality of different output prefixes. Our evaluation on code generation and structured NLP tasks shows how ASAp often produces outputs with higher likelihood (according to the LLM's distribution) than existing GCD techniques, while still enforcing the desired grammatical constraints.

Kanghee Park, Jiayu Wang, Taylor Berg-Kirkpatrick +2

University of Wisconsin-Madison · University of California San Diego

Constrained Generative Modeling Constrained Decoding Adaptive Sampling LLM Code Generation Grammar-Constrained Decoding

Ask PDF

Abstract

Constrained Decoding

Explore similar work

The Hidden Cost of Structured Generation in LLMs: Draft-Conditioned Constrained Decoding

Constrained Decoding: Harmful or Beneficial? Revisiting Its Impact on LLM Performance

Grammar-Aligned Decoding