cs.CVOct 8, 2026
SaveFloorSAV: Elucidating Spatial Audio-Visual Context with 2D Floormap for AV-LLMs
Organizations: Korea Advanced Institute of Science and Technology
Abstract
While 3D spatial reasoning in dynamic egocentric environments is crucial for embodied intelligence, audio-visual large language models (AV-LLMs) lack explicit mechanisms to process and internalize global geometry directly from raw sensory streams. Existing approaches either require costly fine-tuning or underutilize the model's cross-modal reasoning capacities. In this paper, we propose FloorSAV, a novel framework that explicitly grounds spatial audio-visual context by rendering a dynamic 2D floormap. By integrating 3D point clouds, camera trajectories, spatial audio cues, and semantically grounded object landmarks, we inject this floormap into the AV-LLM as a synchronized stream with an egocentric video. AV-LLMs utilize their multi-modal capabilities to jointly reason over visual, auditory, and geometric cues in a single inference with floormap interpretation guidance. We further introduce SAVED-Bench (Spatial Audio-Visual Egocentric Benchmark with Dynamic Agents), constructing essential tasks of spatial capability in real-world scenarios: dynamic relativity, regional, and path reasoning QAs. FloorSAV improves AV-LLMs' spatial reasoning on various tasks from both SAVED-Bench and SAVVY-Bench. Studies with ground-truth floormaps demonstrate the substantial potential of FloorSAV with accurate spatial information.
Figures & tables
Figure 2: Semantic grounding process. Semantically enriched floormap is generated by a pipeline of keyframe selection, mapping visual objects via VLMs, and filtering, providing semantic context for better reasoning.
Figure 3: SAVED-Bench at a glance. Illustrative question-answer pairs cover (a) dynamic relativity, (b) regional, and (c) path reasoning tasks. Sound events ( ) identify the relevant moments ( ) or intervals, while synchronized egocentric and floormap views provide spatial context.
Figure 4: Region annotation process. (Left) For each scene, we prompt an LLM to assign a region category to every ground-truth object. (Middle) We group objects by category and use their coordinates to automatically compute the smallest enclosing rectangle for each group, with its sides parallel to the walls. (Right) Human annotators cross-check the egocentric video and 2D floormap and refine these initial rectangles by adjusting their vertices to obtain region polygons. These are only used for the QA synthesis, not provided to the model.
Figure 5: SAVED-Bench configurations and distributions. Unlike SAVVY-Bench, whose answers are heavily focused on front-side ( ) and short distances (< 1m), SAVED-Bench is constructed with a broader range of answer cases, covering various spatial configurations and temporal intervals.
| Task | Baseline | FloorSAV | Partial GT Map † | Full GT Map ‡ |
|---|---|---|---|---|
| Dynamic Relativity | ||||
| Ego-to-Exo Direction | 56.0 | 65.0 {}_{{\color[rgb]{0,0.75,0.16}\text{+9.0}}} | 76.5 {}_{{\color[rgb]{0,0.75,0.16}\text{+20.5}}} | 72.0 {}_{{\color[rgb]{0,0.75,0.16}\text{+16.0}}} |
| Ego-to-Exo Distance (AbsMRA) | 34.3 | 34.6 {}_{{\color[rgb]{0,0.75,0.16}\text{+0.3}}} | 47.9 {}_{{\color[rgb]{0,0.75,0.16}\text{+13.6}}} | 45.9 {}_{{\color[rgb]{0,0.75,0.16}\text{+11.6}}} |
| Exo-to-Ego Direction | 32.0 | 50.5 {}_{{\color[rgb]{0,0.75,0.16}\text{+18.5}}} | 50.5 {}_{{\color[rgb]{0,0.75,0.16}\text{+18.5}}} | 53.5 {}_{{\color[rgb]{0,0.75,0.16}\text{+21.5}}} |
| Exo-to-Ego Distance (AbsMRA) | 46.4 | 44.7 {}_{{\color[rgb]{1,0,0}\text{--1.7}}} | 57.5 {}_{{\color[rgb]{0,0.75,0.16}\text{+11.1}}} | 55.0 {}_{{\color[rgb]{0,0.75,0.16}\text{+8.6}}} |
| 4-task average | 42.2 | 48.7 {}_{{\color[rgb]{0,0.75,0.16}\text{+6.5}}} | 58.1 {}_{{\color[rgb]{0,0.75,0.16}\text{+15.9}}} | 56.6 {}_{{\color[rgb]{0,0.75,0.16}\text{+14.4}}} |
Table 1: SAVED-Bench results with Gemini-3.6-Flash. FloorSAV improves over the baseline in all categories. Subscripts indicate the difference from Baseline. *Pooled Visit Order and Unvisited questions. † Ground-truth position of the sound source onto the map. ‡ Partial GT plus the ground-truth object names on the map.
Figure 6: Dynamic relativity by visibility. FloorSAV helps more on FoV-out ( i.e., other agent invisible at the queried moment). Both strategies are weaker on Exo-to-Ego direction tasks.
| Qwen3-Omni-30B | Gemini-2.5-Flash | Gemini-2.5-Pro | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Task | Base | FloorSAV | Base | FloorSAV | Base | SAVVY | FloorSAV | Partial † | Full ‡ |
| Ego Direction | 71.7 | 72.8 | 74.2 | 71.3 | 75.2 | 84.7 | 75.8 | 70.2 | 68.3 |
| Ego Distance | 59.4 | 61.8 | 49.7 | 55.4 | 59.6 | 62.9 | 55.4 | 59.0 | 61.0 |
| Exo Direction | 29.4 | 32.9 | 29.8 | 44.5 | 31.7 | 44.0 | 52.8 | 52.3 | 69.9 |
| Exo Distance | 31.8 | 31.0 | 29.0 | 32.2 | 37.0 | 40.2 | 34.9 | 37.9 | 52.2 |
| Avg. | 48.1 | 49.6 | 45.7 | 50.9 | 50.9 | 58.0 | 54.7 | 54.9 | 62.8 |
Table 4: SAVVY-Bench results. For FloorSAV, Gemini also supplies semantic grounding. SAVVY scores are from Chen et al. (2025) . Direction uses accuracy, and distance uses AbsMRA. Avg. is the four-task mean.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Pseudo prompt for object visual mapping stage. The prompt is designed to leverage the VLM’s visual grounding prior on egocentric view images, enabling accurate coordinate mapping, the extraction of representative visual landmarks, and the simplification of semantic labels.
Figure 8: Simple and hard direction configurations. Relative to the target agent, simple direction utilizes three options (left, right, back), and hard direction utilizes four (front-left, front-right, back-left, back-right).
Figure 9: Comparison of visual input formats for floormap prompting. The strategies include (a) the RGB-only baseline, (b) adding a static floormap image, (c) merging both views into a single-stream side-by-side video, and (d) our proposed dual-stream injection using separate synchronized videos.
| Egocentric | Exocentric | ||||
|---|---|---|---|---|---|
| # of Frames | Direction | Distance | Direction | Distance | Overall |
| 32 | 73.9 | 63.9 | 31.6 | 31.0 | 50.1 |
| 64 | 70.6 | 62.8 | 31.9 | 28.9 | 48.6 |
| 128 | 72.8 | 61.8 | 32.9 | 31.0 | 49.6 |
Table 5: Ablation study on input video frame counts. The frame count indicates the number of uniformly sampled frames from each input video stream, an egocentric view and a floormap.
| Egocentric | Exocentric | ||||
| Method | Direction | Distance | Direction | Distance | Overall |
| Gemini-2.5-Flash ( Comanici et al., 2025 ) | 74.2 | 49.7 | 29.8 | 29.0 | 45.7 |
| Gemini-2.5-Flash- FloorSAV | 71.3 | 55.4 | 44.5 | 32.2 | 50.9 |
| (–) Semantic Grounding | 63.3 | 58.0 | 34.4 | 30.8 | 46.6 |
| Gemini-2.5-Pro ( Comanici et al., 2025 ) | 75.2 | 59.6 | 31.7 | 37.0 | 50.9 |
| Gemini-2.5-Pro- FloorSAV | 75.8 | 55.4 | 52.8 | 34.9 | 54.7 |
Table 6: Ablation study on semantic grounding with Gemini-2.5-Flash and Pro on SAVVY-Bench. The (–) Semantic Grounding setting evaluates our FloorSAV framework without explicit object labels overlaid on the rendered 2D floormap on SAVVY-Bench.
| Task | Gemini-3.6-Flash | Gemini-3.6-Flash- FloorSAV | ( ) Semantic Grounding |
|---|---|---|---|
| Dynamic Relativity | |||
| Ego-to-Exo Direction | 56.0 | 65.0 | 59.5 {}_{{\color[rgb]{1,0,0}\text{--5.5}}} |
| Ego-to-Exo Distance (AbsMRA) | 34.3 | 34.6 | 32.1 {}_{{\color[rgb]{1,0,0}\text{--2.5}}} |
| Exo-to-Ego Direction | 32.0 | 50.5 | 47.5 {}_{{\color[rgb]{1,0,0}\text{--3.0}}} |
| Exo-to-Ego Distance (AbsMRA) | 46.4 | 44.7 | 41.1 {}_{{\color[rgb]{1,0,0}\text{--3.6}}} |
| 4-task average | 42.2 | 48.7 | 45.0 {}_{{\color[rgb]{1,0,0}\text{--3.7}}} |
Table 7: Ablation study on semantic grounding with Gemini-3.6-Flash on SAVED-Bench. The (–) Semantic Grounding setting evaluates FloorSAV framework without explicit object labels overlaid on the rendered 2D floormap on SAVED-Bench. ∗ Pooled Visit Order and Unvisited questions.
Figure 11: Qualitative example of an exocentric direction question. The baseline sees only the egocentric video and has to imagine the robot’s viewpoint, answering “ left ”. FloorSAV reads the robot, the facing object, and the green sound-source marker off the floormap grid and answers “ right ”.
Figure 12: Qualitative example of the exocentric distance task. The baseline model suffers from severe depth distortion inherent in monocular egocentric vision, overestimating the metric distance by approximately 1.5 meters. In contrast, FloorSAV extracts precise metric coordinates from the top-down floormap to compute the Euclidean distance, significantly reducing the measurement error to about 0.5 meters.
Figure 13: Qualitative example of the semantically grounded floormap. Our semantically grounded floormap, constructed through the proposed pipeline with Gemini, offers semantic labels as visually grounded cues that enhance the model’s spatial reasoning.
Figure 14: Qualitative example of dynamic relativity QA. In this FoV-out Exo-to-Ego direction example, both methods correctly identify the sound event time. However, the baseline incorrectly assumes that the other agent is directly ahead ( ) of the camera wearer. FloorSAV estimates the other agent’s position from adjacent frames and utilizes the floormap as a shared global coordinate frame for viewpoint transformation.
Figure 15: Qualitative example of regional QA. For the Location Awareness task, both methods correctly identify the sound event time. However, the baseline incorrectly assumes that the other agent remains near the dining table. FloorSAV jointly reasons about the other agent’s movement from nearby frame cues and physical layout on the floormap, identifies the region (kitchen area) from key objects in the floormap to answer correctly.
Figure 16: Qualitative example of path reasoning QA. In this Line-Path Search (static) example, both methods correctly identify the sound event time. The baseline focuses on the nearby window and fails to relate a finite line segment to other candidates, which results in an inappropriate approach and answer. FloorSAV utilizes a 2D floormap as a reference frame and explicitly grounds the finite line segment and objects. With subsequent arithmetic calculations, FloorSAV determines the closest object to the finite line segment.
Figure 17: Failure case analysis: Temporal localization error. The model incorrectly points out the queried speech event (“ acknowledging the watermelon ”, occurred at 02:19) to an earlier, incorrect timestamp (01:53). Consequently, it retrieves the spatial geometry from the wrong video frame, leading to a flawed directional prediction despite having access to the correct mapping mechanism.
Figure 18: Failure case analysis: Context length issue in the base model. We compare high-density (270 frames, 1 FPS) and low-density (128 frames, 0.5 FPS) egocentric inputs. Contrary to the expectation that more frames provide better visual cues, the excessively long 270-frame input overwhelms the model and leads to an incorrect prediction, whereas the shorter 128-frame context successfully guides the model to the correct answer.
Figure 19: Failure case analysis: Context length issue in the FloorSAV framework. As FloorSAV doubles the input video tokens by incorporating the floormap stream, the model fails to process the basic RGB view correctly, inexplicably hallucinating the other person’s location as “ right ” despite them sitting on the left for almost the entire timeline.
Figure 20: Failure case analysis: Metric map misinterpretation. Although FloorSAV accurately localizes the temporal window of the queried event, it misreads the 2D coordinate grid. It incorrectly specifies the locations of the sound source and reference objects, leading to wrong spatial calculation.
Figure 21: Instruction prompt of our FloorSAV framework for Qwen-Omni models. Gemini models use similar prompt with their own system prompt and multimodal token sequence.