cs.HCMar 28, 2026
SaveBeyond Descriptions: A Generative Scene2Audio Framework for Blind and Low-Vision Users to Experience Vista Landscapes
Organizations: Augmented Human Lab, Dept. of Computer Science, National University of Singapore · Augmented Human Lab, National University of Singapore · Saarland University, Saarland Informatics Campus Saarbrücken, Germany · IRIT, CNRS, Toulouse, France ·
Abstract
Current scene perception tools for Blind and Low Vision (BLV) individuals rely on spoken descriptions but lack engaging representations of visually pleasing distant environmental landscapes (Vista spaces). Our proposed Scene2Audio framework generates comprehensible and enjoyable nonverbal audio using generative models informed by psychoacoustics, and principles of scene audio composition. Through a user study with 11 BLV participants, we found that combining the Scene2Audio sounds with speech creates a better experience than speech alone, as the sound effects complement the speech making the scene easier to imagine. A mobile app "in-the-wild" study with 7 BLV users for more than a week further showed the potential of Scene2Audio in enhancing outdoor scene experiences. Our work bridges the gap between visual and auditory scene perception by moving beyond purely descriptive aids, addressing the aesthetic needs of BLV users.
Figures & tables
Figure 1. This paper consists of three stages: (a) Designing and evaluating the Scene2Audio Framework, (b) A lab-study with BLV participants to evaluate the sounds generated from Scene2Audio framework, and (c) In-the-wild evaluation of user experience of this framework through a mobile app user study with BLV participants for over a week. This image is a flowchart illustrating the three-stage research process described in the paper. The stages are arranged horizontally from left to right, connected by arrows indicating the progression of the work. From left to right: (a) Stage 1 - Framework Development: This stage is represented by a block diagram representing a scenic image passing through the Scene2Audio framework to generate an audio. A right-facing arrow connects Stage 1 to Stage 2. (b) Stage 2 - Lab Study: This stage is represented by photos of two BLV participants sitting at a computer within a controlled environment, wearing headphones. This symbolizes the controlled lab study with Blind and Low-Vision (BLV) participants to evaluate the quality and effectiveness of the sounds generated by the framework. Another right-facing arrow connects Stage 2 to Stage 3. (c) Stage 3 - Field Study: This stage is represented by the photo of a BLV person holding a mobile phone, and a zoomed in picture of the mobile phone screen is on the right that contains the scene2audio app called sonic vista with a vista scene image and four audio icons at the bottom - Brief, Detail, Speech, Audio. This symbolizes the in-the-wild user study where BLV participants used the framework via a mobile app in their daily lives for over a week to evaluate the real-world usage of the system.
Figure 2. Overview of Scene2Audio Framework. The figure illustrates a pipeline for generating non-verbal audio from a vista scene image. On the left, there is a photo of a grassy field with cows, autumn trees, and a church building. Below it, the label reads “Vista Scene Image.” An arrow points to a pink box titled “Salient Objects Identification.” Inside it, the text reads “Image-to-Text” with a ChatGPT icon. Beneath the box, there is a caption: “Identifying salient objects in the vista scene that should be sonified.” From this box, arrows lead to the right. A list is shown: [“Cows mooing”, discrete] [“Leaves rustling”, continuous] [“Church bell ringing”, discrete] The column headers are PROMPT and EVENT TYPE. The process continues to a section labeled “Audio Scene Composition” (outlined in purple). A diamond-shaped decision box asks: “Discrete?” If Yes, the arrow leads to a Text-to-Audio box. The output goes into a panel labeled “Discrete Event Counter, n” showing a line graph with peaks labeled “Onset Envelope of Audio” and “Threshold Events.” The text says: “Select with minimum n.” This path continues to a label “Foreground (0.8).” If No, the arrow leads to another Text-to-Audio box, labeled PROMPT, which continues to a label “Background (0.2).” Both foreground and background paths feed into a Combine box. The final output points to a soundwave icon with the label “Vista Scene Audio (non-verbal).” At the bottom of the purple section, a caption reads: “Generating sounds for the salient objects, and deciding the foreground and background sounds.”
| ID | Age | Gender | Visual Status/ Condition | Visual Acuity | Onset Age | Type of Earphones used | Apps to Know About Surroundings |
|---|---|---|---|---|---|---|---|
| 1 | 30 | M | Legally Blind | <2% | 25 | Wired and wireless earphones | BeMyEyes, Seeing AI |
| 2 | 73 | M | Retinitis Pigmentosa - “I can differentiate between light and shadow” | not known | 59 | Bone conduction headphones (open ear) | Moovit - public transport navigation, BeMyEyes, BeMyAI, Lazarillo |
| 3 | 45 | M | Legally Blind | <2% | 16 | Over the ear headphones | Seeing AI |
| 4 | 27 | M | Retinoblastoma, “I can still see colors and shapes, but cannot see details” | 20/200 | Since birth | AirPods | None |
| 5 | 60 | M | Retinitis Pigmentosa - a genetic disorder | <10% | 41 | Wired earphones | None |
| 6 | 35 | M | Legally Blind | <2% | 19 | In-ear headphones | BeMyEyes, BeMyAI |
Table 1. Details of the BLV participants
| Scene | Category | Density | Num. sonic objects | Example generated |
| Noun+Verb Phrase | ||||
| seabeach | nature | low | 2 | waves crashing, |
| wind blowing | ||||
| mountains | nature | low | 2 | river flowing |
| wind blowing | ||||
| countryside | nature | high | 4 | cows mooing, |
Table 2. Details of the scene images used for the first two studies. Check out the images and generated sounds on our webpage: https://scenetoaudio.github.io/scenetoaudio/#/section-4 .
Figure 5. Mobile App Screenshots (a) Take/Select a photo, (b) Audio playback page The figure shows two screenshots of a mobile application. (a) The first screenshot is labeled “Sonic Vista.” It shows a photo of tall buildings with greenery on multiple levels, along with a road and traffic at the bottom. At the bottom of the screen, there is a large yellow button labeled “Choose from photo gallery.” In the middle, there is also a circular camera button with a camera icon. (b) The second screenshot is labeled “Brief Mode.” It shows the same photo of the buildings with greenery. At the top, navigation arrows and the text “Brief Mode” are visible. At the bottom, there are four menu options in a purple bar: Brief, Detail, Speech, and Audio, each with an icon above the text. The Brief option is highlighted.
| PID | Image | Clearest info | Least clear info | Most enjoyable | Least enjoyable | Most preferred | Why | What did you want to know | Did you get the info you wanted |
|---|---|---|---|---|---|---|---|---|---|
| P4 | Detail | Audio | Detail | Audio | detail | gave the most detail and had some sound effects enhance the experience. | description of the scene | True | |
| P2 | Detail | Audio | Detail | Audio | detail | because it gives me clearest possible picture in my mind | True | ||
| P3 | Detail | Brief | Detail | Brief | detail | Details and precise location that I can visualise | I wish it could describe colours of the environment as well | True | |
| P5 | Detail | Audio | Detail | Speech | detail | Canal at the park. | I especially enjoyed the sounds of the birds, water, etc. However, the app failed to inform me if there were fishes in the water. | True | |
| P7 | Detail | Audio | Detail | Speech | detail | I got all the details, especially the pool | the swimming pool | True | |
| P11 | Detail | Audio | Brief | Audio | Brief | It was concise and easy to understand. It gave a good overview of the scene ahead of me. | False |
Table 3. Subset of in-the-wild user study participant responses. To refer to all the responses, see 16
Figure 6. Mode (in percentage) that : (a) provided the clearest info; (b) provided the least clear info; (c) provided the most enjoyable experience; (d) provided the least enjoyable experience; and (e) was overall preferred the most. This figure, titled Preference Analysis, is composed of five bar charts that show user preferences for different modes of interaction. Each chart compares four modes: speech-only, audio-only, overlay, and overlay-concat. The y-axis on each chart represents the percentage of responses for each mode. Error bars indicate a 95% confidence interval, and asterisks above the brackets show statistically significant differences (p<0.05). Key Findings by Chart Chart (a): Mode with clearest info. The overlay-concat mode was overwhelmingly rated as providing the clearest information, with a score of 92%, which is significantly higher than the other modes. The other three modes (speech-only, audio-only, and overlay) each scored around 1-5%, with no significant difference among them. Chart (b): Mode with least clear info. In contrast, the audio-only mode was rated as providing the least clear information by 83% of users, significantly higher than all other modes. The overlay-concat and overlay modes were rated as least clear by only 5% of users, while the speech-only mode was rated as least clear by 10%. Chart (c): Mode most enjoyable. The overlay-concat mode was also rated as the most enjoyable experience, receiving a score of 88%. This is significantly higher than the other three modes, which all scored below 6%. There were no significant differences in ratings among the speech-only, audio-only, and overlay modes for this category. Chart (d): Mode with least enjoyable experience. The audio-only mode was rated as the least enjoyable by 58% of users, significantly higher than the other three modes. The speech-only mode was rated as least enjoyable by 32% of users, while the overlay and overlay-concat modes were rated as least enjoyable by 9% and 2%, respectively. Chart (e): Mode most preferred overall. The overlay-concat mode was the most preferred mode overall, with a preference score of 81%. This is significantly higher than all other modes. The speech-only, audio-only, and overlay modes were preferred by 3%, 5%, and 11% of users, respectively, with no significant difference in preference among them. Across all five charts, the overlay-concat mode consistently received the most positive ratings, excelling in clarity, enjoyability, and overall preference. Conversely, the audio-only mode was consistently rated the worst, providing the least clear and least enjoyable experience.
Figure 7. Indoor vs. outdoor scenes comparison of (a) mode providing the clearest information, (b) mode rated as most enjoyable, and (c) average clicks per image for each mode. Figure 7 compares participant responses for indoor versus outdoor vista scenes across four audio modes: Speech-only, Audio-only, Overlay, and Overlay Concat. Panel (a) shows which mode provided the clearest information. Overlay Concat is overwhelmingly chosen for both indoor and outdoor scenes, with very low selections for all other modes. Panel (b) shows which mode was rated most enjoyable. Again, Overlay Concat dominates for both scene types, with Overlay receiving a small number of votes and the other modes near zero. Panel (c) shows the average number of replays per image. Speech-only and Overlay require the most replays, while Audio-only and Overlay Concat require fewer replays, with indoor scenes generally showing slightly higher replay counts than outdoor scenes. Overall, the figure illustrates consistent preferences for Overlay Concat in clarity and enjoyment, and moderate differences in replay behavior across scene types.
Figure 8. User experience questionnaire results, (a) Average rating ( std.) for each question across all participants, (b) Average UEQ scores ( std.) over time across all participants. This figure is composed of two sub-figures, both displaying results from a User Experience Questionnaire (UEQ). Each chart uses bar graphs to show average scores on a 1 to 7 scale, with error bars representing the standard deviation. \parSub-figure (a): UEQ Results This chart shows the average rating for eight different questions, or qualities, of the user experience. \parSupportive: The highest score, with an average of 5.62±0.43. \parEasy: A score of 5.82±1.16, which is the highest average rating in this chart. \parEfficient: A score of 5.19±0.79. \parClear: A score of 5.50±0.55. \parExciting: A score of 5.29±0.54. \parInteresting: A score of 5.44±0.50. \parInventive: A score of 5.25±0.94. \parLeading Edge: The lowest average score, with 4.75±0.91. \parOverall, the ratings for all qualities are above 4.5 on the 7-point scale, indicating a generally positive user experience. The qualities rated highest were "Easy" and "Supportive," while the lowest rating was for "Leading Edge." \parSub-figure (b): UEQ Results Over Time This chart shows how the average UEQ score changes at different points in the user's interaction with the system. The x-axis marks the different points in time: "after 1st image," "after 4th image," "after 7th image," and "after 10th image." \parAfter 1st image: The average score is 5.39±0.71. \parAfter 4th image: The average score is 4.96±1.02. This is the lowest score, representing a slight dip in user experience. \parAfter 7th image: The average score is 5.46±0.42. This score shows an improvement, nearly reaching the initial score. \parAfter 10th image: The average score is 5.45±0.92, which is nearly identical to the score after the 7th image. \parThe results suggest that the user experience starts strong, dips slightly in the middle of the task, and then recovers and remains stable towards the end. The scores are consistently above 4.5, indicating a positive user experience throughout the process.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Scene | Im2wav (a) | Im2text2audio (b) | Ours (c) |
| seabeach | 0 | 95 (a∗) | 76 (a∗) |
| mountains | 0 | 29 (a) | 62 (a∗,b) |
| countryside | 0 | 14 | 81 (a∗,b∗) |
| reservoir | 24 (b) | 0 | 57 (b) |
| park | 5 | 14 | 90 (a∗,b∗) |
| train station | 33 | 14 | 33 |
Table 4. Comparison of our framework with baseline methods in scene selection. All numbers are in % accuracy in selecting the correct scene image that was used to generate that sound, out of four scene image options. Each sound-type for every scene-type was evaluated by a total of 21 participants. Pairwise statistical significance using Wilcoxon signed rank test is shown as superscript of the corresponding audio type on the higher score (p<0.05). A superscript with * means p<0.005.
| Criteria | Questions | Speech-only (a) | Audio-only (b) | Overlay (c) | Overlay Concat (d) |
| Comprehension | Intent Similarity | 5.71 0.70 | 5.57 0.44 | 6.00 0.40 (b,d) | 5.70 0.38 |
| Comp Rating | 5.91 1.23 (b) | 4.41 1.97 | 6.23 0.87 (b∗) | 5.68 1.29 (b) | |
| Qual. of Experience | Enjoyable | 5.05 1.36 (b) | 3.91 1.63 | 5.50 1.34 (b∗,d) | 4.55 1.65 |
| Curious | 5.00 1.41 | 5.77 1.19 (a) | 5.18 1.37 | 5.05 1.62 | |
| Imagination | 4.95 1.56 | 5.41 1.79 | 5.59 1.22 | 5.32 1.52 | |
| Memorable | 4.27 1.61 | 4.50 1.50 | 5.18 1.26 (a) | 4.68 1.81 |
Table 5. Comparison of Ratings for Different Audio Types. Pairwise statistical significance using Wilcoxon signed rank test is shown as superscript of the corresponding audio type on the higher score (p<0.05). A superscript with * means p<0.005. For the four criteria, means higher is better and means lower is better.
Figure 13. In-app questionnaire after each photo submission and audition of the sounds. Three mobile app screens show the post-task questionnaire used after photo submission and listening to generated audio. The first screen (“Mode Preference One”) asks users to select which mode—brief, detail, speech, or audio—provided the clearest and least clear information, and the most and least enjoyable experience, using radio buttons. The second screen (“Mode Preference Two”) asks users to select their most preferred mode (brief, detail, audio, or speech) and provide a short explanation. The third screen (“General Questions”) includes an open-ended question about desired information, a yes/no question on whether the information was obtained, and a 1–7 satisfaction rating scale. Each screen includes a “Next” button for navigation.
Figure 14. In-app questionnaire: User Experience Questionnaire (UEQ). Two mobile app screens show an in-app questionnaire evaluating overall app experience after each task. The questionnaire presents pairs of contrasting attributes rated on a 1–7 scale, including obstructive–supportive, complicated–easy, inefficient–efficient, confusing–clear, boring–exciting, not interesting–interesting, conventional–inventive, and usual–leading edge. Each item uses radio buttons with validation prompts if unanswered. The final screen includes a “Submit” button to complete the questionnaire. Overall, the interface collects user ratings of usability, clarity, engagement, and perceived innovation.