From Abstraction to Action: DSLs for Natural Language Abstraction Interpretation and Execution

Amit Lisha; Reut Tsarfaty

cs.CLOct 20, 2026

From Abstraction to Action: DSLs for Natural Language Abstraction Interpretation and Execution

Authors: Amit Lisha, Reut Tsarfaty

Organizations: Bar-Ilan University, Israel

Abstract

Natural-language instructions often rely on high-level abstractions, such as asking an agent to ``stack blocks in a pyramid.'' Interpreting abstract instructions requires models to both identify the abstraction intended by the speaker, and unroll it into the many concrete actions that instantiate it. Language models struggle here: prior work has shown that performance degrades as the level of abstraction in the instruction rises. To bridge the gap between natural language abstraction and concrete actions, we propose to borrow from coding environments the notion of a {\em domain-specific language} (DSL), an explicit representation layer whose documented primitives name recurring abstractions that expand into concrete actions. We demonstrate the efficacy of DSLs for NL interpretation on Hexagons, a visually-grounded instruction-following task graded for abstraction. We further ask whether language models can induce such DSLs themselves from domain data. Our results show that while code generation consistently improves over direct inference, code generation over a human-curated DSL yields dramatic gains, up to 10 absolute F1 points, reaching the human skyline, with gains concentrated at {\em higher} abstraction levels. We confirm this pattern on the Text2CAD benchmark. At the same time, model-induced DSLs recover only part of this gain, remaining inconsistent across runs and short of their human-curated counterpart. Overall, the right representation closes much of the gap caused by high abstraction --- but inducing that layer of representation automatically is where models still fall short, and where we see a strong opportunity for the community to make progress.

Figures & tables

Panel of Figure 1 — Figure 1: Abstract instructions compress many low-level drawing actions on the board. (a) Direct prediction must recover this structure directly, in an end-2-end fashion. (b) HexaGen makes task-relevant abstractions explicit, letting the model generate code that executes to the correct output.

Panel of Figure 2 — Figure 2: The Hexagons task illustration: Natural language instructions and respective target board states.

	Proprietary Models								Open-Source Models
	GPT-5.1 (Low)		GPT-5.1 (High)		Gemini 3 Pro (Low)		Gemini 3 Pro (High)		Qwen3 30B		Gemma3 27B
	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM
Tiles	0.683 $\pm$ 0.006	0.236 $\pm$ 0.018	0.765 $\pm$ 0.016	0.322 $\pm$ 0.032	0.743 $\pm$ 0.009	0.301 $\pm$ 0.009	0.745 $\pm$ 0.02	0.382 $\pm$ 0.037	0.556 $\pm$ 0.016	0.177 $\pm$ 0.013	0.407 $\pm$ 0.005	0.123 $\pm$ 0.007
Unrestricted Python	0.708 $\pm$ 0.011	0.231 $\pm$ 0.009	0.757 $\pm$ 0.009	0.296 $\pm$ 0.009	0.807 $\pm$ 0.003	0.376 $\pm$ 0.040	0.793 $\pm$ 0.021	0.419 $\pm$ 0.000	0.587 $\pm$ 0.012	0.172 $\pm$ 0.007	0.495 $\pm$ 0.005	0.145 $\pm$ 0.000
Human-Curated DSL	0.774 $\pm$ 0.021	0.371 $\pm$ 0.000	0.831 $\pm$ 0.010	0.441 $\pm$ 0.025	0.825 $\pm$ 0.002	0.457 $\pm$ 0.024	0.853 $\pm$ 0.016	0.511 $\pm$ 0.046	0.663 $\pm$ 0.019	0.237 $\pm$ 0.033	0.512 $\pm$ 0.007	0.140 $\pm$ 0.007

Table 1: Hexagons execution results across all models and strategies. We report F1 and board-level exact match (EM) as mean

\pm

standard deviation over three runs.

Figure 3: F1 as a function of abstraction level under the 3 strategies.

Figure 4: DSL usage patterns vs. performance, for 5 model configurations sorted by overall F1. Left: breadth of DSL usage (unique methods invoked) against board F1. Right: volume of DSL usage (total method calls) against board F1.

	Qwen3-30B-A3B-Thinking			Gemini-3.1-Pro
Level	DSL	Direct	Undec.	DSL	Direct	Undec.
Expert	43.8%	20.9%	35.3%	13.8%	4.8%	81.4%
Intermediate	25.6%	20.4%	54.0%	17.6%	12.1%	70.3%
Beginner	23.0%	22.1%	54.9%	17.1%	17.5%	65.3%
Abstract	31.1%	30.1%	38.9%	31.1%	24.0%	45.0%

Table 2: Text2CAD pairwise VLM-judge preferences between DSL and direct CadQuery generation, across abstraction levels and two models (single seed, 885 examples/level/model). Percentages are over pairs for which a judgment was reached.

	Proprietary Models				Open-Source Models
	GPT-5.1 (High)		Gemini 3.1 Pro (High)		Qwen3 30B Thinking		Gemma3 27B Instruct
	F1	EM	F1	EM	F1	EM	F1	EM
Minimal DSL	0.740 $\pm$ 0.008	0.282 $\pm$ 0.034	0.839 $\pm$ 0.033	0.484 $\pm$ 0.028	0.542 $\pm$ 0.003	0.145 $\pm$ 0.016	0.371 $\pm$ 0.019	0.097 $\pm$ 0.000
All-at-once	0.757 $\pm$ 0.082	0.314 $\pm$ 0.103	0.776 $\pm$ 0.069	0.321 $\pm$ 0.099	0.565 $\pm$ 0.019	0.152 $\pm$ 0.018	0.416 $\pm$ 0.055	0.129 $\pm$ 0.023
Batches	0.745 $\pm$ 0.088	0.339 $\pm$ 0.160	0.828 $\pm$ 0.032	0.518 $\pm$ 0.023	0.480 $\pm$ 0.041	0.100 $\pm$ 0.034	0.389 $\pm$ 0.060	0.105 $\pm$ 0.034
Semantic Batches	0.704 $\pm$ 0.007	0.225 $\pm$ 0.022	0.788 $\pm$ 0.077	0.396 $\pm$ 0.129	0.531 $\pm$ 0.025	0.149 $\pm$ 0.019	0.441 $\pm$ 0.020	0.121 $\pm$ 0.011

Table 3: Results for induced DSL variants and the human-curated DSL. We report F1 and board-level exact match (EM) as mean

\pm

standard deviation across three independently induced libraries; each library is evaluated over three execution runs. Gemini rows use Gemini 3.1 Pro and are not comparable to Table 1 .

Capability	HexaGen	Run A	Run B
Shape constructors (rays, lines, rings, triangles, fill)	✓	✓	✗
Rotation / mirror / translation	✓	✓	✗
Hex distance metric	✓	✓	–
Hex adjacency (neighbors)	✓	✓	✗
Compositional Shape algebra	✓	–	–
Spatial / topological queries	✓	–	–

Table 4: Capability coverage and board-F1 for HexaGen and the two GPT-5.1 (High) all-at-once induced DSLs. ✓ = supported, – = absent, ✗ = present but broken.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Setup	EM	B-F1	A-F1
Tiles	0.590	0.864	0.819
Tiles + Vision	0.464	0.830	0.782
HexaGen	0.744	0.902	0.867
HexaGen + Vision	0.727	0.896	0.860

Appendix

Table 5: Effect of visual feedback under step-by-step prompting (Gemini 3.1 Pro, validation set, 446 instructions). EM is board-level exact match; B-F1 and A-F1 are board- and action-level F1.

	Original Benchmark Comparison
Setup	GPT-5.1 (Low)	Gemma 3	DeBERTa	T5
Step-by-step + Full History	0.520 $\pm$ 0.003	0.404 $\pm$ 0.002	0.404	0.386
Full-Procedure	0.541 $\pm$ 0.007	0.407 $\pm$ 0.005	–	–

Appendix

Table 6: Comparison to the original Hexagons benchmark in F1.

Figure 5: For each model and induction variant, we report how many methods remain after each pipeline stage: merge, deduplication, validation, and downstream use.

Class	Method / Property	Description
Game	init(width, height)	Initializes a game with board dimensions (defaults to WIDTH=18 , HEIGHT=10 ).
	start(width, height)	Factory method returning a new game instance.
	enter() , exit(...)	Context-manager interface for writing programs inside a Game block.
	record_step(name)	Saves a snapshot of the current painted tiles under a step name.
	get_record(names)	Returns the tiles painted in one or more named steps.
	plot(...)	Visualizes the board state, optionally against a gold reference.

Appendix

Table 7: Public classes, methods, properties, and operators exposed in the HexaGen base library.

Class	Method / Property	Description
Game	init(width, height)	Initializes the board with the given dimensions.
	start(width, height)	Creates a new game instance with the given dimensions.
	enter(), exit(...)	Supports context-managed execution inside a Game block.
	plot(...)	Visualizes the board state, optionally against a gold reference.
Tile	init(row, column, game)	Creates a tile at the specified coordinates.
	draw(color)	Paints the tile with the specified color.

Appendix

Table 8: Public classes, methods, and properties exposed in the minimal HexaGen API.

Instruction Following

Instruction following studies how AI systems interpret and reliably carry out user goals, constraints, formats, and priorities.

See all 134 papers

Jun 21, 2026

Text2DSL: LLM-Based Code Generation for Domain-Specific Languages

Domain-specific languages (DSLs) are widely used for managing operating system security policies, yet manually authoring rules in such languages demands high expertise and is error-prone. This paper formalises the task of automatic DSL code generation from natural language descriptions - Text2DSL - as a distinct problem class, separate from Text-to-SQL and general-purpose code generation. We introduce the PolkitBench dataset comprising 4,204 verified natural-language-to-Polkit-rule pairs, each validated through a three-level AST-based pipeline. Controlled prompt experiments on two MoE models of different scale and provenance - GigaChat-10B-A1.8B (1.8B active parameters) and Nemotron-3-Nano-30B-A3B (3B active) - demonstrate the critical role of structured context (BNF grammar, API specification, permitted identifier vocabulary) for LLM-based DSL code generation. Across both models, supplying context raises syntactic validity to 98.6-99.4%, structural validity by +9.7 to +35.5 pp, and the CodeBLEU score by +60% to +95%. The consistency of the effect across models of different scale and provenance indicates that, for the Text2DSL class of problems, injecting a formal target-language specification into the prompt context is a robust enabling factor for high-quality generation without model fine-tuning.

Alexander V. Kozachok, Alexander M. Nazimov, Shamil G. Magomedov

RTU MIREA — Russian Technological University, 78 Vernadsky Avenue, Moscow 119454, Russia · Academy of the Federal Guard Service of the Russian Federation, 35 Priborostroitelnaya Street, Oryol 302020, Russia

LLM Prompting Code Generation Code Generation Benchmarks LLM Code Generation

Ask PDF

Sep 25, 2026

Up and Down the Abstraction Ladder: Code-Based Skills for Language Agents

Language agents struggle to act and learn in environments that require long sequences of low-level actions. Code-based abstractions can make these agents more productive by letting them invoke reusable skills instead of repeatedly selecting individual actions. The code handles recurring local decisions, while the language model decides which skills to use and how to combine them. Yet abstractions are leaky, and situations beyond a skill's capabilities may require a return to primitive actions. Motivated by this tradeoff between productivity and flexibility, we systematically study how code-based action abstraction affects the performance, inference cost, and learning of language agents. We study this in NetHack, a challenging, long-horizon game environment, using CodeHack, our library of code-based skills with natural-language descriptions. We use this library to compare agents restricted to primitives with those using semantic skills alone or in combination with primitives. We evaluate these agents in three settings: zero-shot prompting, supervised fine-tuning, and reinforcement learning. Across a broad zero-shot evaluation on NetHack, we find that compared with primitives, skills nearly triple game progression, while reducing inference cost per episode by 86%. Combining skills with primitives retains much of this benefit while preserving a path back down to low-level actions. Finally, in RL, we find that skill-based agents learn significantly faster than agents acting on primitives, achieving a 7.2x larger average gain in dungeon level over the same training budget. These results show that a supplied skill library can improve performance, efficiency, and learning, while retaining primitives provides flexibility when the library is insufficient. We release CodeHack together with training and evaluation code.

Bartłomiej Cupiał, Jens Tuyls, Maciej Wołczyk +5

University of Warsaw · AKCES NCBR · Princeton University +6

Long-Horizon LLM Agents LLM Agent Skill Learning RL for LLM Agents

Ask PDF

Jun 21, 2026

Context-Aware Distillation and Ablation for Text2DSL

We extend our prior work on Text2DSL automatic generation of domain-specific language (DSL) code from natural language descriptions along two complementary axes. First, we replace prompt-only synthetic generation with context-aware distillation, in which a teacher large language model (DeepSeek-V4-Flash) operates under an explicitly defined structured context comprising a BNF grammar, an API specification, and a closed identifier vocabulary; the resulting corpus is verified by a two-tier pipeline combining AST validation through esprima and runtime acceptance through the production polkitd daemon and the pkcheck client. This scales the verified PolkitBench corpus from 4,204 to 10,073 natural-language-to-Polkit-rule pairs at 100.0% AST validity and 99.7% runtime pass rate. Second, we conduct the per-component factorial ablation of structured context that was identified as future work in the precursor study: eight conditions C0-C7 are evaluated on GigaChat-10B-A1.8B with the new corpus. Three findings emerge. (i) The new harder corpus collapses the baseline mode (Syntax Valid 97.6% -> 58.5%, Combined Score 0.482 -> 0.252), whereas the context-enhanced mode degrades only marginally (Syntax 98.6% -> 97.4%, Combined 0.801 -> 0.750), confirming that structured context is not a cosmetic improvement but a load-bearing mechanism. (ii) The best absolute condition is the full context C7 across all metrics, while the strongest partial conditions (C5 = BNF + Vocabulary, C6 = API + Vocabulary) both contain the vocabulary. (iii) A Shapley-style decomposition assigns the largest semantic-quality effect to the vocabulary (Combined +0.198), the largest structural-validity effects to API (+24.7 pp) and BNF (+22.3 pp).

Alexander V. Kozachok, Alexander M. Nazimov, Shamil G. Magomedov

MIREA – Russian Technological University, Moscow, Russian Federation · Academy of the Federal Guard Service of the Russian Federation, Oryol, Russian Federation

LLM Prompting LLM-Based Program Synthesis Language Model Distillation Synthetic Data Generation for Language Models Text Generation

Ask PDF

Abstract

Instruction Following

Explore similar work

Text2DSL: LLM-Based Code Generation for Domain-Specific Languages

Up and Down the Abstraction Ladder: Code-Based Skills for Language Agents

Context-Aware Distillation and Ablation for Text2DSL