cs.CLNov 1, 2026
SaveDon't Guess, Ask Wisely: Benchmarking and Training Proactive Elicitation in Open-Domain LLM Conversations
Organizations: Zhejiang University · Ant Group · City University of Hong Kong
Abstract
When user queries are incomplete, large language models should ask targeted questions before committing to an answer, a capability we call proactive elicitation, yet evaluation on real user-assistant queries across seven LLMs (8B to frontier) shows that fewer than 11% of underspecified queries trigger a follow-up question. To benchmark and train this capability, we introduce VagueBench, where each seed query is annotated with structured intent and detail units then degraded into intent-vague and fully-vague variants via controlled removal of linguistic cues, yielding known hidden information sets for evaluating both when and what to ask. We propose DialogGRPO, an offline reinforcement learning method that organizes multi-turn elicitation as a dialogue-state directed acyclic graph, where candidate questions at each node receive dense turn-level rewards combining immediate information gain with lookahead value through group-relative comparison, enabling fine-grained credit assignment without runtime simulation. DialogGRPO elicits missing information on over 88% of vague queries with substantial gains in both recovery efficiency and answer coverage, without over-asking on complete queries, outperforms prompting and test-time-scaling baselines, and generalizes zero-shot to MultiWOZ 2.4 and AmbigQA.
Figures & tables
Figure 1: Top: The same underlying need expressed as a complete query and a fully-vague query. Bottom: A direct-answer LLM fabricates all missing information, while proactive elicitation recovers it in 3 targeted turns.
| Method | No Runtime Sim. | Open Domain | Turn Reward | Real Queries |
| ClarifyGPT | ✓ | ✗ | ✗ | ✗ |
| ACT | ✗ | ✗ | ✗ | ✗ |
| STaR-GATE | ✓ | ✓ | ✗ | ✗ |
| CollabLLM | ✗ | ✓ | ✓ | ✗ |
| Ours | ✓ | ✓ | ✓ | ✓ |
Table 1: Comparison of proactive elicitation methods. No Runtime Sim.: the parameter-update phase does not invoke an LLM-based user simulator at runtime. Turn Reward: per-turn rather than trajectory-level credit assignment.
Figure 2: VagueBench construction pipeline. Real user queries are filtered for self-containedness and information completeness, annotated with intent and detail units via four clarity anchor types, and then systematically degraded into intent-vague and fully-vague variants with known gold information for evaluation.
Figure 3: DialogGRPO pipeline. Left: A dialogue-state DAG is built by expanding candidate questions at each turn and merging nodes with identical acquired sets. Right: Recovered units ground the final answer (top); turn-level rewards and group-relative advantages guide training (bottom).
| Model | Intent-Vague | Fully-Vague | ||||
| Ask% ( ) | Cov. F1 ( ) | IAR ( ) | Ask% ( ) | Cov. F1 ( ) | IAR ( ) | |
| Closed-source Models | ||||||
| GPT-5.4 | 1.6 | 0.58 | 0.06 | 10.8 | 0.53 | 0.65 |
| Claude Sonnet 4.6 | 8.8 | 0.60 | 0.08 | 8.4 | 0.52 | 0.56 |
| Gemini 3.1 Pro | 2.4 | 0.57 | 0.00 | 3.6 | 0.54 | 0.80 |
| Open-source Models | ||||||
Table 2: Main results on VagueBench. Ask% : proportion of samples on which the model chose to ask. Cov. F1 : coverage F1 of the final answer against the gold information set. IAR : information acquisition rate (Eq. 1 ), computed only on samples where the model asked. Best in bold ; second best underlined .
| Model | Ask% ( ) | Cov. F1 ( ) |
| Qwen3.5-27B | 1.6 | 0.58 |
| + Prompt | 42.8 | 0.59 |
| + UoT | 56.4 | 0.48 |
| + SFT (Ours) | 6.4 | 0.71 |
| + DialogGRPO (Ours) | 5.2 | 0.74 |
Table 3: Behavior on information-complete seed queries. Ask% is lower-the-better. Best in bold ; second best underlined .
| Variant | Intent-Vague | Fully-Vague | ||
| Cov. F1 | IAR | Cov. F1 | IAR | |
| Full (Ours) | 0.64 | 0.76 | 0.67 | 0.87 |
| w/o lookahead reward | 0.58 | 0.74 | 0.60 | 0.84 |
| w/o immediate reward | 0.60 | 0.68 | 0.66 | 0.85 |
Table 4: Reward ablation on VagueBench. The lookahead term primarily affects Cov. F1; the immediate term primarily affects IAR. Best in bold ; second best underlined .
| Model | MultiWOZ 2.4 | AmbigQA | |
| Success% | Ask% | Ask F1% | |
| GPT-5.4 | 39.9 | 5.1 | 9.6 |
| GLM-5 | 23.8 | 0.9 | 1.9 |
| Qwen3.5-27B | 11.4 | 1.1 | 2.1 |
| + DialogGRPO (Ours) | 24.7 | 16.3 | 26.3 |
| Gemma3-27B | 15.1 | 1.1 | 2.2 |
Table 5: Zero-shot transfer. Success% : MultiWOZ task-completion rate. Ask% : fraction of queries on which the model asks in the AmbigQA multipleQAs subset. Ask F1% : ask-versus-answer F1 on the full AmbigQA development set. Best in bold ; second best underlined .
Figure 4: Generalization across base models on VagueBench. Darker/lighter segments denote base model scores and DialogGRPO lifts, respectively. Both 27B models show comparable gains, confirming transferability across model families.
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Flow of queries through the four-stage VagueBench filtering pipeline. Rule-based Filtering removes of the 100K raw queries on length or unsafe-content grounds. LLM-based Filtering then discards greetings, test strings, and context-dependent utterances ( ), followed by detail-sparse queries that lack the structure required for degradation ( ). A final Intent & Detail Annotation stage yields information-complete seeds, of the input. The lighter peach tones on the left mark earlier stages and the deepest tone on the right marks the final seed pool; grey side-flows are the discarded branches at each stage.
Figure 6: Task-family composition of the VagueBench seed queries. Consultation & Analysis and Information Query together account for over of the pool, while Task Execution forms the rarest . The bench inherits a long-tailed mix that mirrors real-world assistant traffic.
Figure 7: Per-seed counts of annotated intents (left) and details (right) in the -seed pool. Seeds carry on average intents and details; have exactly one intent, and detail counts cluster in the 3 to 6 range.
Figure 8: Initial information coverage of each query variant. Coverage is the fraction of gold information units, intents and details combined, that are already expressed in the initial query. Intent-Vague (peach) preserves of units on average since only intent anchors are weakened, while Fully-Vague (coral) drops to as both intent and a sample of detail anchors are removed. KDE smoothed; box-and-whisker overlays show median, interquartile range, and 5/95 percentiles.
| Split | Human 1 Ask% | Human 2 Ask% |
| Intent-Vague | 98.0 | 96.0 |
| Fully-Vague | 100.0 | 100.0 |
Table 6: Human Ask rates on vague queries. Both annotators were given the vague query together with its complete gold query and judged whether the information gap warranted a clarifying question before answering. High Ask rates confirm that the vague queries genuinely warrant clarification.
| Pair | Cohen’s |
| Human 1 vs Human 2 | 0.87 |
| Human 1 vs LLM | 0.83 |
| Human 2 vs LLM | 0.81 |
Table 7: Unit-level agreement on whether each candidate intent or detail unit is present in the query. Computed on a candidate-set pool drawn from 50 queries.
| Pair | Cohen’s |
| Human 1 vs Human 2 | 0.84 |
| Human 1 vs LLM | 0.80 |
| Human 2 vs LLM | 0.79 |
Table 8: Unit-level agreement on whether each candidate unit was disclosed by a simulated-user reply. Computed on a candidate-set pool drawn from 50 dialogue turns.
Figure 9: System prompt example for a Fully-Vague sample. The hidden-information table lists the four detail units and one intent unit that may release on a matching question.
| Provider | Model | Model Link |
| OpenAI | GPT-5.4 | OpenAI Docs |
| GPT-oss-120B | Hugging Face | |
| Anthropic | Claude Sonnet 4.6 | Anthropic |
| Gemini 3.1 Pro | Google DeepMind | |
| Gemma3-27B | Hugging Face | |
| Meta | Llama-3.1-8B-Instruct | Hugging Face |
Table 9: Models evaluated in this work. The same Qwen3.5-27B checkpoint serves as a base-model baseline (Table 2 ); the cross-base experiment in § 5.4 additionally trains Gemma3-27B with identical hyperparameters. GLM-5.1 plays the simulated user (Appendix C ).
| Hyperparameter | Value |
| Learning rate | |
| Warm-up ratio | |
| LR scheduler | cosine |
| Batch size | |
| Epochs | |
| LoRA rank |
Table 10: Training hyperparameters for DialogGRPO. The same configuration is used in the cross-base-model experiment of § 5.4 .
Figure 10: General-capability check on six external benchmarks: LiveCodeBench, MathQA, CommonsenseQA, MATH-500, MMLU, and GSM8K. The two envelopes trace nearly identical shapes, with the trained DialogGRPO model staying at or above the Qwen3.5-27B base on every benchmark and slightly improving on LiveCodeBench and MATH-500. The result indicates that the proactive elicitation behaviour reported in the main text is acquired without sacrificing general task ability.
| Method (Llama-3.1-8B-Instruct) | Intent-Vague | Fully-Vague | ||||
| Ask% | Cov. F1 | IAR | Ask% | Cov. F1 | IAR | |
| Base | 0.8 | 0.50 | 0.00 | 1.2 | 0.35 | 0.23 |
| CollabLLM | 3.2 | 0.59 | 0.00 | 4.4 | 0.37 | 0.27 |
| DialogGRPO | 76.6 | 0.61 | 0.37 | 82.0 | 0.49 | 0.43 |
Table 11: Same-backbone comparison with CollabLLM and transfer of DialogGRPO to a sub-10B model on VagueBench.
Figure 11: Case 1. Drug comparison — the model elicits the user’s intent, then the symptom and the target user, before delivering a focused recommendation.
Figure 12: Case 2. Short-term stock query — the model successively elicits intent, risk preference, and investor identity before delivering a strategy sketch.
Figure 13: Case 3. RPG reward design — the model elicits intent, then the current mechanism and pain point, before delivering targeted design ideas.
Figure 14: Prompt 1: Seed Query Filter.
Figure 15: Prompt 2: Variant Synthesis.
Figure 16: Prompt 3: Behavior Judge.
Figure 17: Prompt 4: Information Extractor.
More in this topic
Clarification Question Generation
Clarification question generation teaches AI systems to identify missing intent and ask targeted follow-ups before answering or acting.
See all 44 papers