Organizations: Augmented Human Lab, Dept. of Computer Science, National University of Singapore · Augmented Human Lab, National University of Singapore · Saarland University, Saarland Informatics Campus Saarbrücken, Germany · IRIT, CNRS, Toulouse, France · IPAL, CNRS, Singapore
Current scene perception tools for Blind and Low Vision (BLV) individuals rely on spoken descriptions but lack engaging representations of visually pleasing distant environmental landscapes (Vista spaces). Our proposed Scene2Audio framework generates comprehensible and enjoyable nonverbal audio using generative models informed by psychoacoustics, and principles of scene audio composition. Through a user study with 11 BLV participants, we found that combining the Scene2Audio sounds with speech creates a better experience than speech alone, as the sound effects complement the speech making the scene easier to imagine. A mobile app "in-the-wild" study with 7 BLV users for more than a week further showed the potential of Scene2Audio in enhancing outdoor scene experiences. Our work bridges the gap between visual and auditory scene perception by moving beyond purely descriptive aids, addressing the aesthetic needs of BLV users.
Figures & tables
Figure 1. This paper consists of three stages: (a) Designing and evaluating the Scene2Audio Framework, (b) A lab-study with BLV participants to evaluate the sounds generated from Scene2Audio framework, and (c) In-the-wild evaluation of user experience of this framework through a mobile app user study with BLV participants for over a week. This image is a flowchart illustrating the three-stage research process described in the paper. The stages are arranged horizontally from left to right, connected by arrows indicating the progression of the work. From left to right: (a) Stage 1 - Framework Development: This stage is represented by a block diagram representing a scenic image passing through the Scene2Audio framework to generate an audio. A right-facing arrow connects Stage 1 to Stage 2. (b) Stage 2 - Lab Study: This stage is represented by photos of two BLV participants sitting at a computer within a controlled environment, wearing headphones. This symbolizes the controlled lab study with Blind and Low-Vision (BLV) participants to evaluate the quality and effectiveness of the sounds generated by the framework. Another right-facing arrow connects Stage 2 to Stage 3. (c) Stage 3 - Field Study: This stage is represented by the photo of a BLV person holding a mobile phone, and a zoomed in picture of the mobile phone screen is on the right that contains the scene2audio app called sonic vista with a vista scene image and four audio icons at the bottom - Brief, Detail, Speech, Audio. This symbolizes the in-the-wild user study where BLV participants used the framework via a mobile app in their daily lives for over a week to evaluate the real-world usage of the system.
Figure 2. Overview of Scene2Audio Framework. The figure illustrates a pipeline for generating non-verbal audio from a vista scene image. On the left, there is a photo of a grassy field with cows, autumn trees, and a church building. Below it, the label reads “Vista Scene Image.” An arrow points to a pink box titled “Salient Objects Identification.” Inside it, the text reads “Image-to-Text” with a ChatGPT icon. Beneath the box, there is a caption: “Identifying salient objects in the vista scene that should be sonified.” From this box, arrows lead to the right. A list is shown: [“Cows mooing”, discrete] [“Leaves rustling”, continuous] [“Church bell ringing”, discrete] The column headers are PROMPT and EVENT TYPE. The process continues to a section labeled “Audio Scene Composition” (outlined in purple). A diamond-shaped decision box asks: “Discrete?” If Yes, the arrow leads to a Text-to-Audio box. The output goes into a panel labeled “Discrete Event Counter, n” showing a line graph with peaks labeled “Onset Envelope of Audio” and “Threshold Events.” The text says: “Select with minimum n.” This path continues to a label “Foreground (0.8).” If No, the arrow leads to another Text-to-Audio box, labeled PROMPT, which continues to a label “Background (0.2).” Both foreground and background paths feed into a Combine box. The final output points to a soundwave icon with the label “Vista Scene Audio (non-verbal).” At the bottom of the purple section, a caption reads: “Generating sounds for the salient objects, and deciding the foreground and background sounds.”
ID
Age
Gender
Visual Status/ Condition
Visual Acuity
Onset Age
Type of Earphones used
Apps to Know About Surroundings
1
30
M
Legally Blind
<2%
25
Wired and wireless earphones
BeMyEyes, Seeing AI
2
73
M
Retinitis Pigmentosa - “I can differentiate between light and shadow”
not known
59
Bone conduction headphones (open ear)
Moovit - public transport navigation, BeMyEyes, BeMyAI, Lazarillo
3
45
M
Legally Blind
<2%
16
Over the ear headphones
Seeing AI
4
27
M
Retinoblastoma, “I can still see colors and shapes, but cannot see details”
20/200
Since birth
AirPods
None
5
60
M
Retinitis Pigmentosa - a genetic disorder
<10%
41
Wired earphones
None
6
35
M
Legally Blind
<2%
19
In-ear headphones
BeMyEyes, BeMyAI
Table 1. Details of the BLV participants
Scene
Category
Density
Num. sonic objects
Example generated
Noun+Verb Phrase
seabeach
nature
low
2
waves crashing,
wind blowing
mountains
nature
low
2
river flowing
wind blowing
countryside
nature
high
4
cows mooing,
Table 2. Details of the scene images used for the first two studies. Check out the images and generated sounds on our webpage: https://scenetoaudio.github.io/scenetoaudio/#/section-4 .
Table 5
Figure 5. Mobile App Screenshots (a) Take/Select a photo, (b) Audio playback page The figure shows two screenshots of a mobile application. (a) The first screenshot is labeled “Sonic Vista.” It shows a photo of tall buildings with greenery on multiple levels, along with a road and traffic at the bottom. At the bottom of the screen, there is a large yellow button labeled “Choose from photo gallery.” In the middle, there is also a circular camera button with a camera icon. (b) The second screenshot is labeled “Brief Mode.” It shows the same photo of the buildings with greenery. At the top, navigation arrows and the text “Brief Mode” are visible. At the bottom, there are four menu options in a purple bar: Brief, Detail, Speech, and Audio, each with an icon above the text. The Brief option is highlighted.
PID
Image
Clearest info
Least clear info
Most enjoyable
Least enjoyable
Most preferred
Why
What did you want to know
Did you get the info you wanted
P4
Detail
Audio
Detail
Audio
detail
gave the most detail and had some sound effects enhance the experience.
description of the scene
True
P2
Detail
Audio
Detail
Audio
detail
because it gives me clearest possible picture in my mind
True
P3
Detail
Brief
Detail
Brief
detail
Details and precise location that I can visualise
I wish it could describe colours of the environment as well
True
P5
Detail
Audio
Detail
Speech
detail
Canal at the park.
I especially enjoyed the sounds of the birds, water, etc. However, the app failed to inform me if there were fishes in the water.
True
P7
Detail
Audio
Detail
Speech
detail
I got all the details, especially the pool
the swimming pool
True
P11
Detail
Audio
Brief
Audio
Brief
It was concise and easy to understand. It gave a good overview of the scene ahead of me.
False
Table 3. Subset of in-the-wild user study participant responses. To refer to all the responses, see 16
Figure 6. Mode (in percentage) that : (a) provided the clearest info; (b) provided the least clear info; (c) provided the most enjoyable experience; (d) provided the least enjoyable experience; and (e) was overall preferred the most. This figure, titled Preference Analysis, is composed of five bar charts that show user preferences for different modes of interaction. Each chart compares four modes: speech-only, audio-only, overlay, and overlay-concat. The y-axis on each chart represents the percentage of responses for each mode. Error bars indicate a 95% confidence interval, and asterisks above the brackets show statistically significant differences (p<0.05). Key Findings by Chart Chart (a): Mode with clearest info. The overlay-concat mode was overwhelmingly rated as providing the clearest information, with a score of 92%, which is significantly higher than the other modes. The other three modes (speech-only, audio-only, and overlay) each scored around 1-5%, with no significant difference among them. Chart (b): Mode with least clear info. In contrast, the audio-only mode was rated as providing the least clear information by 83% of users, significantly higher than all other modes. The overlay-concat and overlay modes were rated as least clear by only 5% of users, while the speech-only mode was rated as least clear by 10%. Chart (c): Mode most enjoyable. The overlay-concat mode was also rated as the most enjoyable experience, receiving a score of 88%. This is significantly higher than the other three modes, which all scored below 6%. There were no significant differences in ratings among the speech-only, audio-only, and overlay modes for this category. Chart (d): Mode with least enjoyable experience. The audio-only mode was rated as the least enjoyable by 58% of users, significantly higher than the other three modes. The speech-only mode was rated as least enjoyable by 32% of users, while the overlay and overlay-concat modes were rated as least enjoyable by 9% and 2%, respectively. Chart (e): Mode most preferred overall. The overlay-concat mode was the most preferred mode overall, with a preference score of 81%. This is significantly higher than all other modes. The speech-only, audio-only, and overlay modes were preferred by 3%, 5%, and 11% of users, respectively, with no significant difference in preference among them. Across all five charts, the overlay-concat mode consistently received the most positive ratings, excelling in clarity, enjoyability, and overall preference. Conversely, the audio-only mode was consistently rated the worst, providing the least clear and least enjoyable experience.
Figure 7. Indoor vs. outdoor scenes comparison of (a) mode providing the clearest information, (b) mode rated as most enjoyable, and (c) average clicks per image for each mode. Figure 7 compares participant responses for indoor versus outdoor vista scenes across four audio modes: Speech-only, Audio-only, Overlay, and Overlay Concat. Panel (a) shows which mode provided the clearest information. Overlay Concat is overwhelmingly chosen for both indoor and outdoor scenes, with very low selections for all other modes. Panel (b) shows which mode was rated most enjoyable. Again, Overlay Concat dominates for both scene types, with Overlay receiving a small number of votes and the other modes near zero. Panel (c) shows the average number of replays per image. Speech-only and Overlay require the most replays, while Audio-only and Overlay Concat require fewer replays, with indoor scenes generally showing slightly higher replay counts than outdoor scenes. Overall, the figure illustrates consistent preferences for Overlay Concat in clarity and enjoyment, and moderate differences in replay behavior across scene types.
Figure 8. User experience questionnaire results, (a) Average rating ( ± std.) for each question across all participants, (b) Average UEQ scores ( ± std.) over time across all participants. This figure is composed of two sub-figures, both displaying results from a User Experience Questionnaire (UEQ). Each chart uses bar graphs to show average scores on a 1 to 7 scale, with error bars representing the standard deviation. \parSub-figure (a): UEQ Results This chart shows the average rating for eight different questions, or qualities, of the user experience. \parSupportive: The highest score, with an average of 5.62±0.43. \parEasy: A score of 5.82±1.16, which is the highest average rating in this chart. \parEfficient: A score of 5.19±0.79. \parClear: A score of 5.50±0.55. \parExciting: A score of 5.29±0.54. \parInteresting: A score of 5.44±0.50. \parInventive: A score of 5.25±0.94. \parLeading Edge: The lowest average score, with 4.75±0.91. \parOverall, the ratings for all qualities are above 4.5 on the 7-point scale, indicating a generally positive user experience. The qualities rated highest were "Easy" and "Supportive," while the lowest rating was for "Leading Edge." \parSub-figure (b): UEQ Results Over Time This chart shows how the average UEQ score changes at different points in the user's interaction with the system. The x-axis marks the different points in time: "after 1st image," "after 4th image," "after 7th image," and "after 10th image." \parAfter 1st image: The average score is 5.39±0.71. \parAfter 4th image: The average score is 4.96±1.02. This is the lowest score, representing a slight dip in user experience. \parAfter 7th image: The average score is 5.46±0.42. This score shows an improvement, nearly reaching the initial score. \parAfter 10th image: The average score is 5.45±0.92, which is nearly identical to the score after the 7th image. \parThe results suggest that the user experience starts strong, dips slightly in the middle of the task, and then recovers and remains stable towards the end. The scores are consistently above 4.5, indicating a positive user experience throughout the process.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Scene
Im2wav (a)
Im2text2audio (b)
Ours (c)
seabeach
0
95 (a∗)
76 (a∗)
mountains
0
29 (a)
62 (a∗,b)
countryside
0
14
81 (a∗,b∗)
reservoir
24 (b)
0
57 (b)
park
5
14
90 (a∗,b∗)
train station
33
14
33
Appendix
Table 4. Comparison of our framework with baseline methods in scene selection. All numbers are in % accuracy in selecting the correct scene image that was used to generate that sound, out of four scene image options. Each sound-type for every scene-type was evaluated by a total of 21 participants. Pairwise statistical significance using Wilcoxon signed rank test is shown as superscript of the corresponding audio type on the higher score (p<0.05). A superscript with * means p<0.005.
Table 12Table 13
Criteria
Questions
Speech-only (a)
Audio-only (b)
Overlay (c)
Overlay Concat (d)
Comprehension ↑
Intent Similarity
5.71 ± 0.70
5.57 ± 0.44
6.00 ± 0.40 (b,d)
5.70 ± 0.38
Comp Rating
5.91 ± 1.23 (b)
4.41 ± 1.97
6.23 ± 0.87 (b∗)
5.68 ± 1.29 (b)
Qual. of Experience ↑
Enjoyable
5.05 ± 1.36 (b)
3.91 ± 1.63
5.50 ± 1.34 (b∗,d)
4.55 ± 1.65
Curious
5.00 ± 1.41
5.77 ± 1.19 (a)
5.18 ± 1.37
5.05 ± 1.62
Imagination
4.95 ± 1.56
5.41 ± 1.79
5.59 ± 1.22
5.32 ± 1.52
Memorable
4.27 ± 1.61
4.50 ± 1.50
5.18 ± 1.26 (a)
4.68 ± 1.81
Appendix
Table 5. Comparison of Ratings for Different Audio Types. Pairwise statistical significance using Wilcoxon signed rank test is shown as superscript of the corresponding audio type on the higher score (p<0.05). A superscript with * means p<0.005. For the four criteria, ↑ means higher is better and ↓ means lower is better.
Figure 13. In-app questionnaire after each photo submission and audition of the sounds. Three mobile app screens show the post-task questionnaire used after photo submission and listening to generated audio. The first screen (“Mode Preference One”) asks users to select which mode—brief, detail, speech, or audio—provided the clearest and least clear information, and the most and least enjoyable experience, using radio buttons. The second screen (“Mode Preference Two”) asks users to select their most preferred mode (brief, detail, audio, or speech) and provide a short explanation. The third screen (“General Questions”) includes an open-ended question about desired information, a yes/no question on whether the information was obtained, and a 1–7 satisfaction rating scale. Each screen includes a “Next” button for navigation.
Figure 14. In-app questionnaire: User Experience Questionnaire (UEQ). Two mobile app screens show an in-app questionnaire evaluating overall app experience after each task. The questionnaire presents pairs of contrasting attributes rated on a 1–7 scale, including obstructive–supportive, complicated–easy, inefficient–efficient, confusing–clear, boring–exciting, not interesting–interesting, conventional–inventive, and usual–leading edge. Each item uses radio buttons with validation prompts if unanswered. The final screen includes a “Submit” button to complete the questionnaire. Overall, the interface collects user ratings of usability, clarity, engagement, and perceived innovation.
Navigational aids for blind and low vision individuals struggle conveying dynamic real-world environments, leading to cognitive overload from continuous, undifferentiated feedback. We present AMAVA, a novel real-time video-to-audio framework that converts mobile device video into contextually relevant sound effects or text-to-speech descriptions. We propose a motion-aware pipeline using a lightweight AI classification model to distinguish between low and high-movement scenes followed by a real-time text-to-audio synthesis pipeline to enhance environmental perception more efficiently. In static environments, AMAVA generates spoken audio scene descriptions for situational awareness. In high-movement situations, it prioritizes safety by delivering sound cues, such as spoken hazard alerts and environmental sound effects. These audio outputs are produced by a decoder-only transformer-based vision-language model with mixture-of-experts and cross-modal attention for visual understanding, in conjunction with neural text-to-speech and natural sound synthesis networks. The proposed framework uses prompt-based caching and category-specific throttling to avoid auditory clutter and minimize latency. We present a comprehensive evaluation of the system, including a real-time navigation study comparing a white cane alone versus with AMAVA, that shows a significant increase in user confidence and perceived safety.
Benjamin Klein, Kazi Ruslan Rahman, Sanchita Ghose
Department of Computer Science, San Francisco State University, San Francisco, CA 94132, USA · Department of Mathematics, San Francisco State University, San Francisco, CA 94132, USA · Department of Computer Engineering, San Francisco State University, San Francisco, CA 94132, USA
Tremendous progress in visual scene generation now turns a single image into an explorable 3D world, yet immersion remains incomplete without sound. We introduce Image2AVScene, the task of generating a 3D audio-visual scene from a single image, and present SonoWorld, the first framework to tackle this challenge. From one image, our pipeline outpaints a 360° panorama, lifts it into a navigable 3D scene, places language-guided sound anchors, and renders ambisonics for point, areal, and ambient sources, yielding spatial audio aligned with scene geometry and semantics. Quantitative evaluations on a newly curated real-world dataset and a controlled user study confirm the effectiveness of our approach. Beyond free-viewpoint audio-visual rendering, we also demonstrate applications to one-shot acoustic learning and audio-visual spatial source separation. Project website: https://humathe.github.io/sonoworld/
Immersive 360-degree educational environments often lack accessible spatial structure, limiting visually impaired learners' ability to orient, explore, and construct mental representations. This paper proposes EscFOA, a geometry-aware spatial audio generation framework designed as an acoustic scaffolding to support spatial cognition. By integrating 3D Gaussian Splatting (3DGS) with conditional diffusion models, EscFOA reconstructs scene geometry from 360-degree videos to synthesize high-fidelity spatial audio consistent with the environmental structure. Explicitly targeting learning outcomes like independent spatial orientation and reduced cognitive load, EscFOA significantly outperforms conventional monaural and stereo audio in supporting spatial learning behaviors among blindfolded sighted participants (simulating visually impaired learners). These findings demonstrate that geometry-consistent generative audio can effectively enable inclusive access to complex spatial learning materials.
Ziyu Luo, Xiaowei Dai, Siying Zhu +1
School of Computer and Artificial Intelligence, Beijing Technology and Business University, Beijing, China · College of Fundamental Studies, Beijing Vocational College of Labour and Social Security, Beijing, China