Interactive Clarification for Cloud Infrastructure-as-Code Synthesis

Zhenning Yang; Kaden Gruizenga; Tongyuan Miao; Patrick Tser Jern Kon; Hui Guan; Andrew Barto; Ang Chen

cs.SEApr 1, 2026

Interactive Clarification for Cloud Infrastructure-as-Code Synthesis

Authors: Zhenning Yang, Kaden Gruizenga, Tongyuan Miao, Patrick Tser Jern Kon, Hui Guan, Andrew Barto, Ang Chen

Organizations: University of Michigan · University of Massachusetts Amherst

Abstract

The scale and complexity of modern cloud infrastructure have made "Infrastructure-as-Code" (IaC) essential for managing deployments through declarative configurations. While large language models (LLMs) are increasingly used to generate IaC configurations from natural language, user requests are often ambiguous and underspecified. Unlike traditional code generation, it is costly and time-consuming to test IaC configurations during the synthesis, forcing the LLMs into an almost one-shot regime. We observe that ambiguity in IaC synthesis exhibits a compositional structure: configurations decompose into three axes (resources, topology, attributes) where higher-level decisions constrain lower-level ones. We propose a training-free, multi-level disambiguation framework that generates diverse candidate specifications, identifies structural disagreements across these axes, ranks them by informativeness, and produces targeted clarification questions that progressively narrow the configuration space. We further introduce Ambig-IaC, an expert-verified benchmark of 300 validated IaC tasks with ambiguous requests, and define evaluation metrics based on graph edit distance and exact attribute matching. Comprehensive experiments show that our method outperforms existing interactive clarification baselines, with gains that scale with the interaction budget and generalize across models. Extensive ablation studies and analyses further demonstrate its robustness for interactive IaC synthesis.

Figures & tables

Figure 1: An underspecified user request can be satisfied by many plausible cloud infrastructure configurations, each represented as a resource dependency graph (left). Ambiguity arises in choosing resource abstractions and services for compute, networking, storage, in inter-resource topology, and in per-resource configuration attributes, when translating intent into Infrastructure-as-Code (right).

Figure 2: Overview of the iterative disambiguation process for interactive Infrastructure-as-Code synthesis. 1. Users provide an initial IaC request. 2. Initially, a pool of diverse structured specifications is generated based on the current interpretations. 3. Symbolically compute disagreements among the current pool of candidates. 4. These disagreements are ranked to identify the most informative differences measured by entropy. 5. Based on the top-ranked disagreement, the system generates a targeted clarification question. 6. Interaction with the user. 7. If the candidate pool is non-empty, prune candidates inconsistent with the user’s answer. 8. The refined candidate pool is fed back into the next iteration, repeating the process until the interaction budget is exhausted. Otherwise, pool regeneration will be triggered.

Dataset	Natural.	Ambig.	Target Cons.
Normalized score ( $0$ – $1$ )
IaC-Eval	0.25	0.24	0.90
Ambig-IaC	0.78	0.89	0.99
Inter-coder agreement (Gwet’s AC)
IaC-Eval	0.71	0.82	0.70
Ambig-IaC	0.65	0.82	0.98

Table 1: Human validation of Ambig-IaC and IaC-Eval on 100 samples, with normalized ratings and inter-coder agreement measured by Gwet’s AC. Higher ratings indicate greater naturalness, ambiguity, and target consistency.

Method	Budget $K$	Struct. (%)	$\Delta$ Struct.	R+A F1 (%)	$\Delta$ F1
Direct Q Generation	5	43.08 [39.90, 46.31]	-5.87 ( $\downarrow$ 11.99% )	17.69 [15.40, 20.05]	-4.35 ( $\downarrow$ 19.74% )
Best-of-N	5	41.28 [37.97, 44.68]	-7.67 ( $\downarrow$ 15.67% )	18.18 [15.80, 20.68]	-3.86 ( $\downarrow$ 17.51% )
Self-Consistency	5	38.59 [35.48, 41.73]	-10.36 ( $\downarrow$ 21.16% )	15.63 [13.40, 17.97]	-6.41 ( $\downarrow$ 29.08% )
ATD	5	42.23 [38.55, 45.85]	-6.72 ( $\downarrow$ 13.73% )	22.70 [20.21, 25.21]	+0.66 ( $\uparrow$ 2.99% )
Ours	5	48.95 [45.95, 51.91]	–	22.04 [19.57, 24.63]	–
Direct Q Generation	10	44.38 [41.31, 47.52]	-6.60 ( $\downarrow$ 12.95% )	18.72 [16.58, 20.97]	-5.49 ( $\downarrow$ 22.68% )

Table 2: Main results on Ambig-IaC using GPT-4o-mini. We report means with 95% task-level bootstrap CIs over 300 tasks (10,000 resamples). Struct. is normalized-GED similarity; R+A F1 is exact-match resource-and-attribute F1; and

\Delta

gives each baseline’s absolute and relative difference from ours. ATD is evaluated at

K\in\{5,10\}

due to its substantially higher cost.

Method	$\Delta$ Struct.	$\Delta$ R+A F1
Direct Q Gen.	+1.62 ( $\uparrow$ 3.68% )	+0.38 ( $\uparrow$ 2.19% )
Best-of-N	+0.92 ( $\uparrow$ 2.25% )	+0.67 ( $\uparrow$ 3.62% )
Self-Consistency	-0.20 ( $\downarrow$ 0.52% )	+0.25 ( $\uparrow$ 1.59% )
Ours	+2.95 ( $\uparrow$ 5.86% )	+2.25 ( $\uparrow$ 9.71% )

Table 3: Question-budget scaling (GPT-4o-mini): average gain in points per 5-question budget increase, with the relative gain in parentheses (

5{\to}10

and

10{\to}15

increments, from Table 2 ).

Figure 3: Per-round dynamics show the candidates converging toward a shared interpretation.

Figure 4: Most tasks require three to five regenerations, with performance generally improving as regenerations increase.

Table 8

$K$	Struct.	$\Delta$ Struct.	R+A F1	$\Delta$ F1
5	47.18	$-$ 1.77 ( $\downarrow$ 3.62% )	21.08	$-$ 0.96 ( $\downarrow$ 4.36% )
10	49.74	$-$ 1.24 ( $\downarrow$ 2.43% )	23.25	$-$ 0.96 ( $\downarrow$ 3.97% )
15	51.65	$-$ 3.19 ( $\downarrow$ 5.82% )	24.34	$-$ 2.19 ( $\downarrow$ 8.25% )

Table 7: Cross-axes balancing ablation (GPT-4o-mini).

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

	Parameter	Value
$K$	question budget	5 / 10 / 15
$N$	candidates per (re)generation	10
$T_{\mathrm{div}}$	diverse-sampling temperature	0.7
$\theta$	entropy floor	0.5

Appendix

Table 8: Hyperparameters of our method.

N

and

T_{\mathrm{div}}

are swept in the main paper’s sensitivity ablation; the rest are fixed across all experiments.

Model	Method	Tokens/task	Time/task
Claude Opus 4.6	Ours	35.8k	75s
	Direct Q Gen.	2.5k	16s
	Best-of-N	10.0k	64s
	Self-Consistency	10.6k	43s
GPT-4.1-mini	Ours	35.0k	58s
	Direct Q Gen.	1.9k	7s

Appendix

Table 9: Per-task cost across five backbone LLMs at

K{=}5

(Ambig-IaC, 300 tasks).

Method	$K{=}5$	$K{=}10$	$K{=}15$
Ours	31.8k / 28s	61.8k / 50s	93.8k / 69s
Direct Q Gen.	1.8k / 3s	2.0k / 4s	2.2k / 6s
Best-of-N	8.2k / 14s	18.0k / 31s	30.0k / 48s
Self-Consistency	8.6k / 13s	25.9k / 38s	55.4k / 91s
Direct ( $K{=}0$ )	0.7k / 1s

Appendix

Table 10: Per-task cost (Tokens/task / Time/task) vs. interaction budget

K

(GPT-4o-mini, Ambig-IaC). Direct is the no-clarification reference.

$K$	Method	Tokens/task	Time/task
5	Ours	31.8k	28s
5	ATD	129.1k	303s
10	Ours	61.8k	50s
10	ATD	1451.6k †	512s

Appendix

Table 11: Per-task cost of ATD vs. ours (GPT-4o-mini, Ambig-IaC). † Includes prompt-cache reads (

\sim

4.1M cached tokens per task);

188.0

k excluding them.

Method	Struct. (%)	R+A F1 (%)
Ours	54.84	26.53
Ours-BaselineOut	49.07 ( $\downarrow$ 10.5% )	26.01 ( $\downarrow$ 2.0% )
Direct Q Generation	46.31 ( $\downarrow$ 15.5% )	18.45 ( $\downarrow$ 30.5% )
Best-of-N	43.12 ( $\downarrow$ 21.4% )	19.52 ( $\downarrow$ 26.4% )
Self-Consistency	38.19 ( $\downarrow$ 30.4% )	16.12 ( $\downarrow$ 39.2% )

Appendix

Table 12: Output-protocol ablation (GPT-4o-mini,

K{=}15

, Ambig-IaC). Ours-BaselineOut keeps our question selection but emits the final spec via the baseline clarified-intent regeneration. Parentheses give the relative drop from Ours .

LLM-Based Program Synthesis

Large language model (LLM)-based program synthesis turns natural-language goals and examples into executable programs, often using tests or formal constraints.

See all 307 papers

CONTRA: Discovering and Qualifying Behavior-Changing Questions for Selective Clarification in LLM Code Generation
Oct 1, 2026Zheng Fang, Yongmin Li, Yichang Zhang +5Clarification Question Generation AI Coding Agents
Peking University · Wuhan University
Don't Guess, Just Ask: Evaluating Interactive Requirement Clarification for Repository-Level Software Development
Oct 9, 2026Mingyu Chen, Zhiqing Zhong, Manyi Wang +3Software Engineering Benchmarks Clarification Question Generation
The Chinese University of Hong Kong, Shenzhen, Guangdong, China
When and How Should an Agent Clarify? CIGAsk: Teaching LLMs to Clarify via Counterfactual Information Gain
Sep 21, 2026Yunxiang Li, Xixin Wu, Helen MengClarification Question Generation LLM Agent Training
The Chinese University of Hong Kong, Hong Kong SAR, China

Oct 1, 2026J/K move · Enter open · S save

CONTRA: Discovering and Qualifying Behavior-Changing Questions for Selective Clarification in LLM Code Generation

Zheng Fang, Yongmin Li, Yichang Zhang, Dongming Jin+4

Peking University · Wuhan University

Clarification Question Generation AI Coding Agents Code Generation Code Generation Evaluation

Open paper AskPDF arXiv

Coding agents can generate code that appears correct but implements behavior the user never intended. This mismatch can arise when an agent silently resolves underspecified requirements through its own assumptions. As subsequent development builds on these assumptions, correcting the resulting behavior can become increasingly costly. Early clarification can help prevent such mismatches, but unnecessary questions can interrupt developers and slow down development. Existing methods struggle to identify key clarification questions while avoiding unnecessary ones. Therefore, we propose CONTRA, a training-free method that combines broad question discovery with semantic and execution-based question qualification. CONTRA first generates candidate questions and filters out those unrelated to required behavior or already resolved by the requirement. For each remaining question, it generates programs conditioned on two plausible answers and checks for stable behavioral differences on shared inputs. It then uses the interaction history to select among qualified questions or stop asking. Experiments on ClarifyCodeBench show that CONTRA achieves the highest F1 with all four coding agents, exceeding the best baseline macro-average F1 by 13.88 percentage points. With the same LLM and evaluation protocol, CONTRA also achieves higher clarification recall and F1 than the coding harnesses Claude Code and OpenHands. To support practical use, we also implement CONTRA as a Claude Code plugin that integrates selective clarification into everyday development.

Abstract

LLM-Based Program Synthesis

Explore similar work

CONTRA: Discovering and Qualifying Behavior-Changing Questions for Selective Clarification in LLM Code Generation

Don't Guess, Just Ask: Evaluating Interactive Requirement Clarification for Repository-Level Software Development

When and How Should an Agent Clarify? CIGAsk: Teaching LLMs to Clarify via Counterfactual Information Gain