Prompting and steering techniques are well established in general-purpose generative AI, yet assistive visual question answering (VQA) tools for blind users still follow rigid interaction patterns with limited opportunities for customization. User control can be helpful when system responses are misaligned with their goals and contexts, a gap that becomes especially consequential for blind users that may rely on these systems for access. We invite 11 blind users to customize their interactions with a real-world conversational VQA system. Drawing on 418 interactions, reflections, and post-study interviews, we analyze prompting-based techniques participants adopted, including those introduced in the study and those developed independently in real-world settings. VQA interactions were often lengthy: participants averaged 3 turns, sometimes up to 21, with input text typically tenfold shorter than the responses they heard. Built on state-of-the-art LLMs, the system lacked verbosity controls, was limited in estimating distance in space and time, relied on inaccessible image framing, and offered little to no camera guidance. We discuss how customization techniques such as prompt engineering can help participants work around these limitations. Alongside a new publicly available dataset, we offer insights for interaction design at both query and system levels.
Figures & tables
Figure 1. During an in-lab scenario, P11 experimented with stylistic customization to control the level of detail in the system’s description and focus the response on locating the snack box. A blind participant capturing a lab room trying to find the red box of snack, first the application provides a long description of what is included in the scene, after that the participant tries to filter out extra information by asking the app to only provide the red box location relative to laptop, as a result the answer is short and concise and focused on the goal.
Figure 2. Our multi-day study includes an in-lab interview with scenarios, a 10-day diary, and a remote interview. artistic depiction of the study, from left to right: in-lab interview and engineered scenarios, the diary study with user-defined scenarios, and the remote interview. The first illustrates a blind participant taking a photo of an object on top of a table using the phone. The participant is seated close by a researcher, who is wearing a pair of smart glasses for video recording. An open laptop placed in front of the participant on the table is also recording via Zoom. The second illustrates a blind participant with a white cane using a smartphone which displays an image of an urban courtyard and a text description followed by a user question and AI answer about a dome-shaped glass structure. Under this section there is text specifying that this phase included 3 days of optional guided scenarios plus 7 days of user defined scenarios. The third illustrates A computer screen showing a Zoom video call with the researcher and blind participant speaking, indicated by speech bubbles and a red “Rec” icon.
PID
Blindness
Age
Gender
Education
Occupation
AI Fam*
Related Assistive Tech
P1
total
30
W
associate
full-time parent
1
Be My Eyes
P2
legal
39
W
high school
customer service representative
3
Seeing AI; Be My Eyes; Aira
P3
total
52
W
some college
not employed
3
Be My Eyes; Seeing AI
P4
total
38
W
high school
not employed
2
Be My Eyes
P5
legal
53
W
master
not employed
3
-
P6
legal
31
W
some college
assistant technology tutor
3
Be My AI; Aira; Meta Glasses
Table 1. Self-reported participant demographics, familiarity with AI, and related assistive technology use .
Technique
Definition in GenAI
Definition in our use case
Example used
Binary Feedback
Feedback that indicate positive or negative rewards ( Ouyang et al., 2022 ) .
Users provide a simple positive or negative feedback with the response
"Helpful/Not helpful."
Zero-shot: Style Prompting
Prompt to specify desired stylistic attributes of GenAI output (e.g., tone and genre) ( Lu et al., 2023 ) .
Users prompt to specify desired style to adjust following responses.
"More/Less detail."
Zero-shot: Intention Prompting *
Emotion prompt that incorporates phrases of psychological relevance to humans ( Li et al., 2023 ) .
Users express their goals to guide systems tailor responses based on the task context.
"I want to gain muscle. Is this a good option of food?"
CoT Prompting
Offer reasoning behind to guide GenAI systems ( Patel et al., 2022 ; Wei et al., 2022 ) .
Users prompt with reasoning for desirable or undesirable answers.
"I only care about finding the soup box, so do not mention about other items."
Image-as-text Prompting
Generate a textual description of image(s), enabling easy inclusion into a text-based prompt ( Hakimov and Schlangen, 2023 ) .
Users prompt with visual characteristics of objects or scenes, allowing systems to complete tasks with additional information.
"Find my keys with a keychain that is green, round, and with raised letters."
Table 2. Definitions and examples of the customization techniques introduced in the lab.
Figure 3. Lengths of user inputs and system responses (measured in word count) across in-lab session and diary session interactions. Violin chart showing the word count distribution of user inputs and system responses during in-lab and diary session interactions. In the in-lab sessions, user inputs are all under 100 words, while system responses typically range from approximate 80 to 500 words. In the diary sessions, user inputs mostly fall between 0 and 100 words, with a few outliers extending to 150–300 words. System responses during diary sessions show a broader range, primarily between 30 and 400 words, with some responses reaching up to nearly 1,400 words.
Figure 4. Example interaction consisting of five conversational turns with word count for each turn. an interaction example consisting of five turns between a blind user and the VQA system, including two images and three text prompts. The interaction shows a participant locating a box of snack in their cabinet. In the first turn, the user submits a image, and the system responds with a detailed description. The second turn to the fourth turn consist of text prompts from the user, followed by shorter responses from the system. In the fifth turn, the user shares another image, prompting a slightly longer response from the system. Participant input word count and system word count are variable in each turn.
Figure 5. Distribution of interactions across scenarios, with colors indicating presence, absence, or uncertainty and intensity marking customization. Proportionally, interactions involved more customization in familiar environments and with familiar items, slightly more around others than alone, similar indoors and outdoors, and less when in hurry. Stacked horizontal bar chart illustrating preferences for interaction types across various contextual factors, categorized by levels of customization and input types. The y-axis lists seven situational factors: "In a hurry," "Alone," "Indoor," "Leisure," "Familiar environment," "Familiar item(s)," and "Familiar people." Each bar is segmented into multiple color-coded responses indicating different combinations of customization and text input: dark blue (No, with customization), medium blue (No, text input without customization), light blue (No, without text input), dark red (Yes, with customization), orange (Yes, text input without customization), peach (Yes, without text input), and three shades of gray for "Maybe/mix" options. The x-axis is percentage-based from 0% to 100%, showing distribution of responses. The right side features mirrored bars for "Maybe/mix" responses. The chart reveals that participants are least likely to engage when "in a hurry" (high "No" responses) and most likely to engage when in "familiar environments" or during "leisure" (high "Yes" responses with customization or text input). A legend at the bottom provides detailed color definitions for interaction modes.
Figure 6. All the customization techniques observed in Ask & Prompt dataset following the hierarchy from ( Schulhoff et al., 2025 ) . The techniques with solid border are introduced in the lab. Techniques with asterisk come up by participants in the diaries. Techniques with dash border are customization techniques that participants’ explored strategies resemble. Due to the limitation of Be My AI , we use image-as-text prompting instead of few-shot prompting to provide information. Diagram showing different types of AI prompting techniques from Schulhoff et al. It splits into two main categories: text-based prompting and multimodal (MM) prompting. Under text-based prompting, several methods are listed: Binary feedback, Zero-shot, which branches into intention prompting, style prompting, role prompting, and zero-shot with context*, Thought generation, leading to chain-of-thought prompting, Ensembling, Self-criticism**, Decomposition**, Few-shot, which connects to image-as-text prompting. The multimodal prompting branch includes image-as-text and action-oriented* techniques. Some methods are marked with an asterisk to highlight variations. The chart emphasizes how different prompting \revised{techniques} connect and build on each other.
Figure 7. Distribution of question categories in 363 multi-turn interactions from participants in our diary study session and 495 single-turn interactions randomly sampled in VizWiz ( Brady et al., 2013 ) . In participants' dairies, there are 40.38% of interactions involve questions asking for description, 20.33% of interactions involve questions asking for text reading, 9.62% of the interactions involve questions for object localization and orientation, 8.79% of interactions involve questions for object identification, and 5.49% of interactions where the question remained unanswerable despite follow-up attempts.
Figure 8. P11 using (a)) binary feedback along with detailed explanation in the style of chatting, (b) chain-of-thought prompting during the diary session, and (c) explicitly mentioning vision level. An interaction example using three customization techniques. 1) Binary feedback along with detailed reasons in the style of chatting with the system: participant prompted the system with This is very helpful, because the app itself is not sufficiently accessible to me using VoiceOver on my iPhone as a blind user.'' 2) Chain-of-thought and decomposition prompting: participant broke down his goal of assessing vacuuming progress into multiple prompts. First, he asked for factual details such as the area cleaned and the battery level. Then he followed up with an inference question, If that number (the battery level) is low, and based on the purple coloration (cleaned area) of each room on that floor, is it reasonable for me to assume that it is largely or even completely done?'' 3) explicitly mentioned blindness: in the same prompt as binary feedback one.
Figure 9. P9 using decomposition prompting in the diary. An interaction example using decomposition prompting. Participant breaks down her overall goal into a series of questions, asking the system to check its volume, look for air bubbles, assess whether it is tacky, and determine if any flour remains in the bowl.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Question Types
Definition
Example Questions
Description Questions (40.38% * )
Questions asked for the description of a picture or the visual characteristics of an object or a scene.
“Is anyone sitting on the bench?” “What color are the pillows on the couch?” “Please just describe in the greatest possible detail the picture [on the screenshot].”
Reading Questions (20.33%)
Questions asked for reading text or looking for information based on text.
“I need to know who the sender is.” “What is written on the door?” “May you read me the setup instructions?”
Localization Questions (9.62%)
Questions asked for the location or spatial relation of objects.
“Where is the ID card?” “Help me find the remote.” “From where I am, how far is the dumpster and is it to my left or right?”
Identification Questions (8.79%)
Questions asked for identifying an object by name or type ( Brady et al., 2013 ) .
“What is it?” “What is in the boxes?” “What’s the Stonyfield’s product? Is it yogurt”
Unanswerable Questions (5.49% unanswerable till the end)
Questions that cannot be answered based on the information included in previous images in the interaction.
–
Appendix
Table 3. Types of questions asked by participants, with definitions and examples.
Generative AI (GenAI) tools are increasingly woven into how blind and low-vision (BLV) people communicate, not only with digital information, but with the physical world and with other people. Tools such as ChatGPT, Google Gemini, Be My AI, and Seeing AI translate visual and textual content into accessible form, and are beginning to substitute for interpersonal requests for help, such as asking a family member to read a label or describe a scene. Drawing on semi-structured interviews with 19 BLV participants, we examine GenAI as a communication intermediary and how it succeeds and fails as an alternative for reading, describing, and even asking another person for help. We also investigated what BLV users gain and risk when these tools take over that role. We conclude with design and policy implications for GenAI systems that communicate uncertainty honestly, protect information, and support BLV users' independence rather than substitute for it unsafely.
Visual diagrams, figures, and tables are central to scientific papers, and convey information beyond what is captured in text. While blind or low-vision (BLV) scientists have traditionally relied on static alternative text to access figures in papers, the rise of artificial intelligence (AI) has made interactive question-answering (QA) a feasible paradigm for visual exploration; yet little is known about how scientists use visual QA in practice or how to improve its accessibility. In this work, we interview five BLV and five sighted scientists across different STEM fields to understand how they use two AI tools, ChatGPT and Gemini, to query multimodal scientific documents. Our findings characterize how scientists review multimodal content, including existing practices (along with accessibility workarounds) for engaging with visuals, and feedback on the suitability of AI-generated responses to multimodal queries. We further find that vague or incomplete image descriptions, as well as incorrect AI outputs more broadly, can cause both BLV and sighted scientists to abandon AI workflows. To support future research, we additionally contribute a dataset of 115 queries and responses from our participants' interactions with the AI tools for papers in their field. We close by discussing implications for AI-powered scientific QA systems, emphasizing considerations for access across abilities and domains.
Arnavi Chheda-Kothary, Lucy Lu Wang, Joseph Chee Chang +1
Paul G. Allen School of Computer Science & Engineering University of Washington Seattle, WA, USA · Allen Institute for AI · The Information School University of Washington Allen Institute for AI Seattle, WA, USA +1
Vision-language models (VLMs) increasingly power consumer-facing AI search, yet evaluating them on the diversity of everyday visual questions remains challenging. Existing benchmarks often target predefined capabilities, such as multi-hop retrieval or long-form synthesis, whereas users ask photo-grounded questions spanning a long tail of everyday scenarios. Despite advances in VLMs, users on Xiaohongshu, a mainstream Chinese image-sharing platform, continue to turn to other people for help with everyday visual questions. Motivated by this behaviour, we curate NoteVQA from these questions, yielding 252 items across 12 topical categories and 7 user intents. Each item includes a concise reference distilled from expert community responses and a human-audited interleaved reference answer that combines textual explanations with supporting visual evidence. We evaluate both short-answer correctness and interleaved-answer quality. To support the latter, we introduce AgenticInterleave, a single-agent ReAct framework for retrieval-supported answer generation, together with IVR-12, a 12-dimensional rubric for assessing the content, presentation, and image quality of interleaved references and model outputs. Across 9 frontier VLMs, the highest short-answer accuracy is 52.8%, while adding agentic search to Qwen3.5-397B-A17B improves accuracy by only 2.0%. For interleaved answers, the same model running AgenticInterleave scores 3.52 under IVR-12, compared with 4.65 for the human-audited references, with the largest gap in content quality. These results highlight the challenges that everyday visual questions pose for current VLMs in both answer accuracy and the quality of visually grounded explanations.