CLARA: A Cognitively-Driven Clarification Framework for Grounding Ambiguous Requests
Authors: Dimitrios Bekris, Marios Glytsos, Panagiotis P. Filntisis, Niki Efthymiou, George Retsinas, Petros Maragos
Organizations: Robotics Institute, Athena Research Center, 15125 Maroussi, Athens, Greece · School of ECE, NTUA, Athens, Greece, the Robotics Institute, Athena Research Center, Athens, Greece and the HERON - Hellenic Robotics Center of Excellence, Athens, Greece
Robotics and AI systems are increasingly deploying large vision-language models (LVLMs) to follow high-level instructions, from routine household chores to more complex organizational tasks. Yet, broad directives like "clean the room" or "organize the desk" frequently omit crucial details about user preferences and specific arrangement of objects that influence the desired outcome. Without any further guidance, such models tend to overestimate their grasp of ambiguous instructions and frequently assume they have sufficient understanding despite missing important details. In this paper, we propose CLARA (Conversational cLArification for Robotic Arrangement), a cognitively-driven clarification framework that proactively elicits key information from users, in a similar fashion as humans' clarification seeking behavior. Through targeted questioning, CLARA refines ambiguous requests into structured, detailed instructions describing precisely how objects should be organized or rearranged, according to the user's preferences. These clarifications then guide a diffusion-based image generation pipeline, producing realistic visual renderings of the final, desired scene. This final rendering also serves as a visual confirmation of the system's understanding from the user. Experimental results and user studies demonstrate that CLARA leads to more accurate and consistent outcomes compared to another State-of-the-Art LVLM system, highlighting the importance of interactive clarification for bridging the gap between vague directives and fully grounded task specifications.
Figures & tables
Fig. 2 : The abstract scenarios designed for the experimental setup. An unorganized desk (left), a dinner table with food to serve (middle), and a dirty desk (right).
Fig. 3 : Example of clarification seeking behavior from CLARA. First the system seeks to clarify the abstract goal of tidying up and what exactly it refers to. Then, the system asks to specify how exactly the user wants the objects to be organized. When the dialog ends and CLARA has all the information needed, it generates the new scene and shows it to the user for verification.
System
stdev
cos
Gemini 2.0
0.1
0.92
CLARA+Gemini 2.0
0.14
0.88
CLARA
0.08
0.95
TABLE I : Standard deviation and mean of cosine similarity for each model. CLARA achieves more consistent visual results when prompted with the same instruction.
Fig. 4 : Visual outputs of each model in each of the abstract scenarios described: “Organize a desk”, “Serve food to platters”, and “Clean a desk”. As we can see, CLARA achieves a visual result which is aligned with the textual instruction while retaining the whole scene structure. Gemini either introduces new objects (such as new platters) or cannot capture the user’s purpose (e.g. removing most objects on the table), since it does not perform clarification seeking behavior. CLARA+Gemini performs better than Gemini, but also introduces new objects on the scene (e.g. a new bowl) or erroneously modifies some objects.
System
Q1 (Intent)
Q2 (Scene)
Q3 (Satisfaction)
Gemini 2.0
1.40
3.00
1.30
CLARA+Gemini 2.0
2.97
3.60
3.17
CLARA
2.90
3.89
3.37
TABLE II : User Study Ratings (Mapped to 1–5 Likert Scale) for the three systems.
Enabling robots to explore and act in unfamiliar environments under ambiguous human instructions by interactively identifying task-relevant objects (e.g., cups or beverages for "I'm thirsty") remains challenging for existing vision-language model (VLM)-based methods, due to inefficient reasoning and limited environmental interaction. These factors hinder real-time task planning and execution. To address this, we propose Affordance-Aware Interactive Decision-Making and Execution for Ambiguous Instructions (AIDE), a human-inspired dual-stream framework that integrates interactive exploration with vision-language reasoning: mirroring the complementary deliberative and intuitive processes of human cognition, Multi-Stage Inference (MSI) serves as the decision-making stream and Accelerated Decision-Making (ADM) as the execution stream, enabling zero-shot affordance analysis and interpretation of ambiguous instructions. Experiments in simulation and real-world settings show that AIDE achieves over 80% task planning success rate and more than 95% closed-loop continuous execution accuracy at 10 Hz, surpassing existing VLM-based methods in diverse open scenarios.
Hengxuan Xu, Fengbo Lan, Zhixin Zhao +5
Department of Automation, Tsinghua University, Beijing, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China · Institute for Interdisciplinary Information Sciences, Tsinghua University, Beijing, China +1
Vision-language models (VLMs) often exhibit overconfidence, providing firm answers even when visual evidence is insufficient. A prominent case is referential ambiguity in Visual Question Answering (VQA), where multiple entities in an image are plausible referents of a question. Addressing such cases requires recognizing when ambiguity is present: ambiguous questions can be handled by asking for clarification or enumerating plausible referents, while unambiguous questions should receive a direct answer. Current VLMs struggle with this distinction, often defaulting to a single interpretation. To investigate whether activation steering can serve as an inference-time mechanism to elicit such behaviors, we introduce AmbiSteer, a lightweight framework to selectively elicit ambiguity-aware behavior. Across three VLMs, we find that steering alone has limited and model-dependent effects, while prompting is generally more effective. Combining prompting and steering substantially increases clarification and enumeration behaviors, but also elicits them on unambiguous cases. AmbiSteer addresses this using a linear classifier to predict input ambiguity and condition the steering direction accordingly, increasing the proportion of inputs handled appropriately from 59.0% to 77.0% on average across models. Our work sets the ground for lightweight approaches to reduce model overconfidence in uncertain multimodal settings.
Monica Riedler, Samuele Antonelli, Barbara Plank +1
MaiNLP, Center for Information and Language Processing, LMU Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · University of Trento +1
AI assistants receive requests that leave out information needed for a good outcome, for example about users' preferences or goals. They must then either speculate or ask for more information before proceeding. We reconceptualize this as a value-of-information problem: the assistant should acquire information whose absence causes the greatest avoidable loss in user utility. This is rarely known ex ante; rather, assistants must predict it in order to optimally allocate limited user interactions. We instantiate this problem in image generation and derive a reinforcement learning framework using multi-turn simulated users to maximize utility recovery under uncertainty. In a preregistered study with 456 interactive sessions across 76 human participants, this helped users significantly better match reference images with significantly fewer questions, less total interaction time, and lower cost. This points toward a simple and scalable framework for training language model assistants to better disambiguate user intent by asking more informative questions.