SAW-Bench: Learning Situated Awareness in the Real World

Chuhan Li; Rilyn Han; Joy Hsu; Yongyuan Liang; Rajiv Dhawan; Jiajun Wu; Ming-Hsuan Yang; Xin Eric Wang

cs.CVFeb 18, 2026

SAW-Bench: Learning Situated Awareness in the Real World

Authors: Chuhan Li, Rilyn Han, Joy Hsu, Yongyuan Liang, Rajiv Dhawan, Jiajun Wu, Ming-Hsuan Yang, Xin Eric Wang

Organizations: University of California, Santa Barbara · Yale University · Stanford University · University of Maryland, College Park ·

Abstract

A core aspect of human perception is situated awareness, the ability to relate ourselves to the surrounding physical environment and reason over possible actions in context. However, most existing benchmarks for multimodal foundation models (MFMs) emphasize environment-centric spatial relations (relations among objects in a scene), while largely overlooking observer-centric relationships that require reasoning relative to agent's viewpoint, pose, and motion. To bridge this gap, we introduce SAW-Bench (Situated Awareness in the Real World), a novel benchmark for evaluating egocentric situated awareness using real-world videos. SAW-Bench comprises 786 self-recorded videos captured with Ray-Ban Meta (Gen 2) smart glasses spanning diverse indoor and outdoor environments, and over 2,071 human-annotated question-answer pairs. It probes a model's observer-centric understanding with six different awareness tasks. Our comprehensive evaluation reveals a human-model performance gap of 37.66%, even with the best-performing MFM, Gemini 3 Flash. Beyond this gap, our in-depth analysis uncovers several notable findings; for example, while models can exploit partial geometric cues in egocentric videos, they often fail to infer a coherent camera geometry, leading to systematic spatial reasoning errors. We position SAW-Bench as a benchmark for situated spatial intelligence, moving beyond passive observation to understanding physically grounded, observer-centric dynamics.

Figures & tables

Figure 2: Overview of SAW-Bench . We illustrate six representative tasks (§ 3.1 ) evaluating different aspects of situated awareness: Localization , Relative Direction , Route Shape , Reverse Route Plan , Spatial Memory , and Spatial Affordance . During data collection, human annotators follow pre-defined trajectories when recording egocentric videos (§ 3.2 ); these trajectories are visualized as purple dashed arrows. For all tasks, the model input consists solely of egocentric video without access to any bird’s-eye or global scene representations; the visualizations shown here are provided for illustrative purposes only.

Benchmark	Ego.	Self-Col.	Mod.	Self-Loc.	Motion	Traj.	Mem.	Act. Feas.
Spatial-MM ( Shiri et al., 2024 )	✓ –	✗	Images	✗	✗	✗	✗	✗
ViewSpatial Bench ( Li et al., 2025b )	✓ –	✗	Images	✗	✗	✗	✗	✗
Open3D-VQA ( Zhang et al., 2025a )	✗	✗	Images	✗	✗	✗	✗	✗
SpatialBench ( Xu et al., 2025 )	✓	✓	Videos	✗	✗	✗	✗	✗
MindCube ( Wang et al., 2026 )	✗	✓ –	Images	✗	✗	✗	✗	✗
3SDRBench ( Ma et al., 2025 )	✗	✗	Images	✗	✗	✗	✗	✗

Table 1: Summary of Visual-Spatial Reasoning Benchmarks. We compare existing works from: 1 Egocentric setting, 2 Self-collected data, 3 Input modality, 4 Self-localization tasks, 5 Motion Understanding, 6 Trajectory Reasoning, 7 Spatial Memory, and 8 Action Feasibility.

Figure 3: Benchmark Curation Pipeline. We first pre-define 46 camera trajectories and annotate their metadata. Human video collectors then record egocentric videos by following these trajectories in selected scenes. Low-quality recordings are filtered and re-captured to ensure consistent video quality.

Baselines
Model	All	Self- Localization	Relative Direction	Route Shape	Reverse Route Plan	Spatial Memory	Spatial Affordance
Human Level ∗	91.55	94.00	89.39	97.62	93.01	88.50	79.01
Chance Level (Random)	27.49	34.00	25.90	21.43	27.51	28.00	56.17
Chance Level (Frequent)	29.55	38.00	25.90	27.11	27.51	27.00	50.62
Blind LLM (GPT-5.2)	31.34	38.00	23.02	36.63	24.02	38.00	54.32
Socratic Model (GPT-5.2)	31.34	40.50	20.62	41.58	24.02	32.00	50.62

Table 2: Evaluation Results on SAW-Bench . Unless otherwise specified, all models process videos at 2 fps (frames per second). Frame level sensitivity analyses are provided in § F . Bold and underlined numbers indicate the best and second-best performance in each category, respectively. Model configurations are provided in Table 4 . ∗ : Human baseline details are provided in § C.3 . § : Models do not support fps-based sampling and process a fixed total of 32 frames per video. ¶ : 8 frames per video sampling due to compute limitations.

Figure 4: Error Case Analysis. ( Left ) Reverse Route Plan : Gemini 3 Flash successfully reconstructs the return path by systematically inverting the actions from the forward pass. In contrast, Qwen3-VL 235B attempts to exploit a shortcut between the first and last frames, thereby neglecting the transitive dynamics and spatial transformations occurring throughout the frame sequence. ( Right ) Route Shape : While both Gemini 3 Flash and Qwen3-VL 235B effectively identify camera rotations, they falsely integrate these rotational pans into the observer’s physical movement trajectory, leading to incorrect shape understanding.

Figure 5: Camera Rotation and Observer’s Trajectory. Visualization of three controlled scenarios used to isolate the impact of head rotation on Route Shape . (Left) a straight path with steady head orientation; (Middle) the same straight path with frequent left-and-right head rotations; and (Right) a true zigzag trajectory.

Models	Straight	Single Turn	Two Turns
Human	100.00	96.67 (-3.33%)	90.00 (-10.00%)
Gemini 3 Flash	73.33	70.69 (-3.60%)	40.61 (-44.63%)
Gemini 3 Pro	63.33	56.90 (-10.16%)	36.46 (-42.44%)
Gemini 2.5 Pro	73.33	55.17 (-24.76%)	33.41 (-54.44%)
GPT-5.2	30.00	39.66 (+32.20%)	22.49 (-25.03%)
Qwen3-VL 235B	90.00	8.62 (-90.42%)	27.85 (-69.06%)

Table 3: Accuracy (%) on Relative Direction Tasks Stratified by the Number of Turns . Performance for most models degrades significantly as geometric complexity increases.

Figure 6: Model Responses in Spatial Memory . Across multiple models, non-visibility is incorrectly treated as non-existence: objects that exit the camera’s field of view are inferred to have disappeared or changed, revealing a gap between what is seen and what exists .

Figure 7: Indoor vs. Outdoor Performance . Comparison of zero-shot accuracy across six situated awareness tasks for Gemini 3 Flash, Gemini 2.5 Pro, GPT-5.2, and Qwen3-VL 235B.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Proprietary Multimodal Foundation Models
Model	API Checkpoint / HF Checkpoint	Max New Tokens	Temp.	Top-P	Top-K	Video Sampling	Thinking Level
Gemini 3 Pro †	gemini-3-pro-preview	16384	0.0	1.0	1.0	2 fps	low †
Gemini 3 Flash	gemini-3-flash-preview	16384	0.0	1.0	1.0	2 fps
Gemini 2.5 Pro	gemini-2.5-pro	16384	0.0	1.0	1.0	2 fps
Gemini 2.5 Flash	gemini-2.5-flash	16384	0.0	1.0	1.0	2 fps
GPT-5.2 ∗	gpt-5.2-2025-12-11	16384				2 fps	medium

Appendix

Table 4: Model Configurations for Evaluation. Unset values indicate that their default values are being used. Configurations are based on official model repositories where available. Temp. : temperature. ‡ : GPT-5 mini only accepts a temperature value of 1.0. ∗ : GPT-5.2 does not support adjusting the temperature or top-p parameters. † : The thinking_level parameter for Gemini 3 Pro was set to low . When configured to dynamic , Gemini 3 Pro consistently generated reasoning traces that exceeded the output context window, resulting in incomplete or Null responses. § : Models do not support fps-based sampling and process a fixed total of 32 frames per video. ¶ : 8 frames per video sampling due to compute limitations.

Figure 8: Example Input to the Socratic Baseline. Each $4\times 4$ grid contains 16 frames sampled at 2 fps, with frame indices shown in the bottom-left corner of each frame.

Model	All	Self- Localization	Relative Direction	Route Shape	Reverse Route Plan	Spatial Memory	Spatial Affordance
Annotator 1	91.07	98.50	86.69	98.53	91.70	94.00	76.54
Annotator 2	92.03	89.50	92.09	96.70	94.32	83.00	81.48
Average	91.55	94.00	89.39	97.62	93.01	88.50	79.01

Appendix

Table 5: Human Baseline Performance on SAW-Bench .

Figure 9: Human Evaluation Interface. Each annotator answers all questions independently, with unlimited time and full access to the corresponding videos. Annotators are instructed to rely solely on the visual information provided, without external tools or discussion.

Spatial Location	Video ID	Q1	A1	Q2	A2
Corner 1	0	Self-Localization : Are you positioned near the corner, along the side, or near the center of the $\langle$ Scene Name $\rangle$ ?	Corner	Route Shape : What’s the shape of my moving trajectory?	In-place orientation
Corner 2	1	Self-Localization : Are you positioned near the corner, along the side, or near the center of the $\langle$ Scene Name $\rangle$ ?	Corner	Route Shape : What’s the shape of my moving trajectory?	In-place orientation
Corner 3	2	Self-Localization : Are you positioned near the corner, along the side, or near the center of the $\langle$ Scene Name $\rangle$ ?	Corner	Route Shape : What’s the shape of my moving trajectory?	In-place orientation
Corner 4	3	Self-Localization : Are you positioned near the corner, along the side, or near the center of the $\langle$ Scene Name $\rangle$ ?	Corner	Route Shape : What’s the shape of my moving trajectory?	In-place orientation
Side 1	4	Self-Localization : Are you positioned near the corner, along the side, or near the center of the $\langle$ Scene Name $\rangle$ ?	Side	Route Shape : What’s the shape of my moving trajectory?	In-place orientation
Side 2	5	Self-Localization : Are you positioned near the corner, along the side, or near the center of the $\langle$ Scene Name $\rangle$ ?	Side	Route Shape : What’s the shape of my moving trajectory?	In-place orientation

Appendix

Table 6: Video Filming Protocol for In-Place Orientation. Camera wearers remain at a fixed spatial location ( i.e., corner, side, or center of a scene) and only changes their viewing orientation, without introducing translational motion. Whenever possible, we include instances from each spatial location to ensure coverage across different positional contexts. For each video, we asked one Self-Localization and one Route Shape question.

Trajectory	Video ID	Q1	A1	Q2	A2	Q3	A3
$\uparrow$ $\hskip 7.3612pt\leftarrow$ $\hskip 18.40302pt\uparrow$	18	Relative Direction : From my viewpoint at the $\langle$ beginning / end $\rangle$ of this video, where am I located at the $\langle$ end / beginning $\rangle$ of the video?	$\langle$ front left / back right $\rangle$	Reverse Route Plan : From my viewpoint at the end of the video, how can I go back to my starting point?	Turn around, then go straight. Turn left , go straight, then turn right and continue straight.	Route Shape : What’s the shape of my moving trajectory?	Manhattan-style piecewise linear
$\hskip 7.3612pt\leftarrow$ $\downarrow\hskip 14.72241pt\uparrow$	19	Relative Direction : From my viewpoint at the $\langle$ beginning / end $\rangle$ of this video, where am I located at the $\langle$ end / beginning $\rangle$ of the video?	$\langle$ left / left $\rangle$	Reverse Route Plan : From my viewpoint at the end of the video, how can I go back to my starting point?	Turn around, go straight. Turn right , go straight, then turn right and continue go straight.	Route Shape : What’s the shape of my moving trajectory?	Manhattan-style piecewise linear
$\hskip 18.40302pt\uparrow$ $\hskip 7.3612pt\rightarrow$ $\uparrow$	20	Relative Direction : From my viewpoint at the $\langle$ beginning / end $\rangle$ of this video, where am I located at the $\langle$ end / beginning $\rangle$ of the video?	$\langle$ front right / back left $\rangle$	Reverse Route Plan : From my viewpoint at the end of the video, how can I go back to my starting point?	Turn around, go straight. Turn right , go straight, then turn left and continue go straight.	Route Shape : What’s the shape of my moving trajectory?	Manhattan-style piecewise linear
$\hskip 7.3612pt\rightarrow$ $\uparrow\hskip 14.72241pt\downarrow$	21	Relative Direction : From my viewpoint at the $\langle$ beginning / end $\rangle$ of this video, where am I located at the $\langle$ end / beginning $\rangle$ of the video?	$\langle$ right / right $\rangle$	Reverse Route Plan : From my viewpoint at the end of the video, how can I go back to my starting point?	Turn around, go straight. Turn left , go straight, then turn left and continue go straight.	Route Shape : What’s the shape of my moving trajectory?	Manhattan-style piecewise linear
$\leftarrow$ $\hskip 11.04181pt\uparrow$ $\hskip 18.40302pt\leftarrow$	22	Relative Direction : From my viewpoint at the $\langle$ beginning / end $\rangle$ of this video, where am I located at the $\langle$ end / beginning $\rangle$ of the video?	$\langle$ front left / back left $\rangle$	Reverse Route Plan : From my viewpoint at the end of the video, how can I go back to my starting point?	Turn around, go straight. Turn right , go straight, then turn left and continue go straight.	Route Shape : What’s the shape of my moving trajectory?	Manhattan-style piecewise linear
$\hskip 7.3612pt\rightarrow$ $\uparrow$ $\hskip 7.3612pt\leftarrow$	23	Relative Direction : From my viewpoint at the $\langle$ beginning / end $\rangle$ of this video, where am I located at the $\langle$ end / beginning $\rangle$ of the video?	$\langle$ front / right $\rangle$	Reverse Route Plan : From my viewpoint at the end of the video, how can I go back to my starting point?	Turn around, go straight. Turn left , go straight, then turn left and continue go straight.	Route Shape : What’s the shape of my moving trajectory?	Manhattan-style piecewise linear

Appendix

Table 7: Video Filming Protocol for Manhattan-Style Piecewise Linear. Observers’ moving trajectories are visualized using egocentric action arrows. For each video, we asked two Relative Direction questions, one Reverse Route Plan question, and one Route Shape question.

Trajectory	Video ID	Q1	A1	Q2	A2
Straight	34	Relative Direction : From my viewpoint at the $\langle$ beginning / end $\rangle$ of this video, where am I located at the $\langle$ end / beginning $\rangle$ of the video?	$\langle$ left / right $\rangle$	Route Shape : What’s the shape of my moving trajectory?	Straight
Straight w/ turning head frequently	35	Relative Direction : From my viewpoint at the $\langle$ beginning / end $\rangle$ of this video, where am I located at the $\langle$ end / beginning $\rangle$ of the video?	$\langle$ front / back $\rangle$	Route Shape : What’s the shape of my moving trajectory?	Straight
$\hskip 8.19447pt\rightarrow$ $\uparrow$ (L-Shape)	36	Relative Direction : From my viewpoint at the $\langle$ beginning / end $\rangle$ of this video, where am I located at the $\langle$ end / beginning $\rangle$ of the video?	$\langle$ front right / back right $\rangle$	Route Shape : What’s the shape of my moving trajectory?	L-Shape
$\leftarrow$ $\hskip 8.19447pt\uparrow$ (L-Shape)	37	Relative Direction : From my viewpoint at the $\langle$ beginning / end $\rangle$ of this video, where am I located at the $\langle$ end / beginning $\rangle$ of the video?	$\langle$ front left / back left $\rangle$	Route Shape : What’s the shape of my moving trajectory?	L-Shape
$\subset$ (U-Shape)	38	Relative Direction : From my viewpoint at the $\langle$ beginning / end $\rangle$ of this video, where am I located at the $\langle$ end / beginning $\rangle$ of the video?	$\langle$ front / right $\rangle$	Route Shape : What’s the shape of my moving trajectory?	U-Shape
$\supset$ (U-Shape)	39	Relative Direction : From my viewpoint at the $\langle$ beginning / end $\rangle$ of this video, where am I located at the $\langle$ end / beginning $\rangle$ of the video?	$\langle$ back / right $\rangle$	Route Shape : What’s the shape of my moving trajectory?	U-Shape

Appendix

Table 8: Video Filming Protocol for Other Simple Trajectories. For each video, we asked two Relative Direction questions and one Route Shape question.

Task	Cohen’s $\kappa$
Relative Direction	0.987
Reverse Route Plan	0.952

Appendix

Table 9: Inter-Annotator Agreement Measured by Cohen’s

\kappa

Figure 10: Video duration distribution of SAW-Bench .

Figure 11: Question scene distribution of SAW-Bench .

Statistics	Value
Unique Videos	786
Indoor	317 (40.33%)
Outdoor	469 (59.67%)
Number of Scenes	15
Number of Videos per scene ( avg )	42
Number of Videos in Unassigned Scene	155

Appendix

Table 10: Key statistics of SAW-Bench .

Figure 12: Sensitivity to the Number of Input Frames Across Reasoning Types. Each subplot reports accuracy as a function of the number of input frames (8, 16, 32) for a fixed reasoning type. Lines correspond to different open-source MFMs evaluated with varying temporal context lengths. Tasks that require longer-horizon spatial integration, such as Spatial Memory and Route Shape , generally benefit from increased temporal context, whereas short-horizon tasks ( e.g., Self-Localization and Relative Direction exhibit earlier saturation and more limited improvements beyond 16 frames. Nevertheless, additional frames yield only marginal gains across all tasks.

Model	FPS	All	Self- Localization	Relative Direction	Route Shape	Reverse Route Plan	Spatial Memory	Spatial Affordance
Gemini 3 Flash	1	53.55	45.00	39.93	69.05	55.90	64.00	72.22
Gemini 3 Flash	2	53.89	48.50	41.13	64.84	61.57	66.00	70.99
Gemini 3 Flash	3	51.57	42.50	40.29	60.99	63.76	53.00	70.99
Gemini 3 Flash	4	52.25	47.50	39.93	63.55	65.50	55.00	62.96
Gemini 3 Flash	6	49.25	40.50	39.33	56.96	59.83	54.00	67.28

Appendix

Table 11: Sensitivity to Frame Sampling Rate (FPS) Across Reasoning Types. We evaluate the top-performing model, Gemini 3 Flash, on SAW-Bench by varying the number of input frames through different temporal sampling rates (FPS

\in\{1,2,3,4,6\}

), and report performance across reasoning tasks. All other configurations are held constant, as detailed in Table 4 . Overall, increasing the frame sampling rate does not substantially improve situated spatial reasoning. Notably, performance on Spatial Memory decreases at FPS = 3, potentially due to redundant or temporally condensed visual information, while Reverse Route Plan performs worst at FPS = 1, suggesting a greater reliance on temporal visual context for this reasoning type.

Spatial Reasoning Benchmarks

Spatial reasoning benchmarks test whether AI systems can perceive, represent, and act on geometric, relational, and viewpoint-dependent information.

See all 162 papers

May 14, 2026

Minerva-Ego: Spatiotemporal Hints for Egocentric Video Understanding

Video reasoning models are a core component of egocentric and embodied agents. However, standard benchmarks for assessing models provide only evaluation of the output (e.g. the answer to a question), without evaluation of intermediate reasoning steps, and most provide answers only in the text domain. We introduce Minerva-Ego, a benchmark for evaluating complex egocentric visual reasoning. We extend recent high-quality video data sources recorded from egocentric / embodied settings with a set of challenging, multi-step multimodal questions and spatiotemporally-dense human-annotated reasoning traces. Benchmarking experiments show that state-of-the-art models still have a large gap to human performance. To investigate this gap in detail, we annotate each reasoning trace in the dataset with the objects of interest required to solve the question, as spatiotemporal mask annotations. Through extensive evaluations, we identify that prompting frontier models with hints of 'where' and 'when' to look yields substantial improvements in performance. Minerva-Ego can be downloaded at https://github.com/google-deepmind/neptune.

Arsha Nagrani, Jasper Uijilings, Shyamal Buch +6

Google DeepMind

Egocentric Video QA Egocentric Video Understanding Multimodal QA Video Reasoning Multimodal CoT Reasoning Visually Grounded Reasoning

Ask PDF

May 4, 2026

From Where Things Are to What They Are For: Benchmarking Spatial-Functional Intelligence in Multimodal LLMs

Human-level agentic intelligence extends beyond low-level geometric perception, evolving from recognizing where things are to understanding what they are for. While existing benchmarks effectively evaluate the geometric perception capabilities of multimodal large language models (MLLMs), they fall short of probing the higher-order cognitive abilities required for grounded intelligence. To address this gap, we introduce the Spatial-Functional Intelligence Benchmark (SFI-Bench), a video-based benchmark with over 1,500 expert-annotated questions derived from diverse egocentric indoor video scans. SFI-Bench systematically evaluates two complementary dimensions of advanced reasoning: (1) Structured Spatial Reasoning, which requires understanding complex layouts and forming coherent spatial representations, and (2) Functional Reasoning, which involves inferring object affordances and their context-dependent utility. The benchmark includes tasks such as conditional counting, multi-hop relational reasoning, functional pairing, and knowledge-grounded troubleshooting, directly challenging models to integrate perception, memory, and inference. Our experiments reveal that current MLLMs consistently struggle to combine spatial memory with functional reasoning and external knowledge, highlighting a critical bottleneck in achieving grounded intelligence. SFI-Bench therefore provides a diagnostic tool for measuring progress toward more cognitively capable and truly grounded multimodal agents.

Le Zhang, Jihan Yang, Soundarya Krishnan +11

1Mila - Qu´ebec AI Institute, UdeM · 2NYU · 3Apple +1

Spatial Reasoning Benchmarks Visual Spatial Reasoning Multimodal Large Language Models Video Reasoning Multimodal Model Evaluation

Ask PDF

Oct 7, 2026

SAVU-BENCH: A Real-World Benchmark for Spatial Audio-Visual Understanding

Spatial audio-visual understanding requires models to recognize not only what is present, but also where events occur and how they relate across modalities. Existing benchmarks often rely on simulated scenes, evaluate isolated spatial skills, and provide limited diagnostic insight into failure modes. We introduce SAVU-Bench, a real-world benchmark that systematically evaluates spatial audio-visual understanding across three capability levels and seven evaluation tasks. We further introduce SAVU-Diag, a scene-linked diagnostic set that decomposes reasoning questions into their prerequisite grounding and alignment sub-tasks. Evaluation of 12 representative models on SAVU-Bench reveals that while visual spatial grounding is relatively mature, spatial perception involving audio remains a primary bottleneck. SAVU-Diag further demonstrates that most reasoning errors co-occur with failures on these prerequisite tasks, though reasoning gaps persist even when prerequisites are correctly resolved. Motivated by these findings, we introduce SAVU-EA, a training-free evidence-augmented baseline that makes spatial cues more explicit. While SAVU-EA substantially improves spatial grounding and joint matching, high-level spatial reasoning remains challenging. Our findings highlight the urgent need for both robust spatial audio perception and deeper integration of cross-modal spatial relations.

Yu Chen, Ruihang Liu, Yangguang Xu +5

University of Western Australia · University of Science and Technology Beijing · Monash University

Audio-Visual Understanding Spatial Reasoning Benchmarks Spatial Audio Audio-Visual Alignment Visual Spatial Reasoning

Ask PDF

Abstract

Spatial Reasoning Benchmarks

Explore similar work

Minerva-Ego: Spatiotemporal Hints for Egocentric Video Understanding

From Where Things Are to What They Are For: Benchmarking Spatial-Functional Intelligence in Multimodal LLMs

SAVU-BENCH: A Real-World Benchmark for Spatial Audio-Visual Understanding