SituRecBench: Evaluating Frontier Models as Situated Recommendation Assistants in 3D Environments
Organizations: Department of Computing, The Hong Kong Polytechnic University · College of Computer Science, Sichuan University
Abstract
Recommender systems are transitioning from static retrieval engines into interactive assistants grounded in physical environments. However, current studies largely neglect the combined complexities of spatial navigation, visual grounding, and multi-turn social interaction. To bridge this gap, we introduce SituRecBench, an interactive benchmark for evaluating frontier vision-language models (VLMs) as situated recommendation assistants within high-fidelity 3D environments. We reformulate recommendation as a situated assistance process in which an assistant interacts with a user while sharing the user's egocentric view. Our benchmark simulates real-world shopping environments (e.g., supermarkets and home furnishings stores), requiring assistants to elicit preferences, ground recommendations in the environment, and guide the user to relevant locations through dialogue. The benchmark evaluates dialogue quality, recommendation utility, and navigation effectiveness through a unified interactive protocol. Extensive experiments reveal persistent challenges in spatial grounding and preference elicitation, highlighting limitations in their ability to integrate global contextual information with local observations. SituRecBench provides a rigorous foundation for advancing the development of situated assistants in the real world.
Figures & tables
| Benchmark | Spatial Env. | Ego. View | Dialog | Phys. Action | Rec. | Interactive Eval. | Scenario |
| SIMMC-VR [ 34 ] | ✓ | ✓ | ✓ | ✗ | ✓ | ✗ | VR Shopping |
| SCREEN [ 14 ] | ✓ | ✓ | ✓ | ✗ | ✓ | ✗ | Multimodal Rec. |
| MUSE [ 33 ] | ✗ | ✗ | ✓ | ✗ | ✓ | ✗ | Multimodal Rec. |
| MarketGen [ 11 ] | ✓ | ✓ | ✗ | ✓ | ✗ | ✗ | Embodied Retail |
| DeliveryBench [ 18 ] | ✓ | ✓ | ✗ | ✓ | ✗ | ✗ | Food Delivery |
| S 3 IT [ 27 ] | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ | Situated Social Test |
| Environment | Supermarket | Home Furnishings Store | Total |
| # Products | 233 | 222 | 455 |
| # Instances | 1,324 | 244 | 1,568 |
| # Nav. Targets | 29 | 31 | 60 |
| # Waypoints | 42 | 38 | 80 |
| # User Profiles | 25 | 25 | 50 |
| # Tasks | 100 | 100 | 200 |
| Environment | Model | Outcome | Navigation | Recommendation | |||
| SR (%) | Turn | AFR (%) | LNSR (%) | R@5 (%) | NDCG@10 (%) | ||
| Supermarket | Gemma 4 E4B | 28.0 ±5.7 | 9.0 ±0.7 | 49.2 ±2.0 | 10.0 ±3.7 | 17.3 ±4.5 | 15.0 ±4.4 |
| Qwen3-VL-8B | 34.0 ±9.5 | 11.8 ±1.2 | 34.5 ±3.6 | 8.8 ±3.0 | 20.3 ±4.6 | 17.7 ±5.0 | |
| GPT-4o | 45.0 ±6.8 | 12.5 ±0.6 | 62.3 ±2.1 | 15.1 ±3.2 | 31.6 ±2.3 | 26.6 ±1.5 | |
| Qwen3.6-Plus | 65.0 ±11.9 | 12.1 ±1.7 | 65.9 ±4.3 | 28.5 ±5.6 | 49.6 ±10.2 | 42.4 ±9.1 | |
| Claude Opus 5 | 66.0 ±10.1 | 11.6 ±1.1 | 78.6 ±5.5 | 41.0 ±5.0 | 49.8 ±10.7 | 43.5 ±9.4 | |
| Coefficient | SR | AFR | LNSR | R@5 | NDCG @10 | Overall | Inform. | Natur. | Ground. | Prefer. | Proact. | Nonrep. |
| Pearson ( ) | 0.800 | 0.287 | 0.742 | 0.619 | 0.574 | 0.773 | 0.774 | 0.726 | 0.698 | 0.807 | 0.753 | 0.744 |
| Spearman ( ) | 0.462 | 0.100 | 0.800 | 0.300 | 0.300 | 0.700 | 0.359 | 0.667 | 0.564 | 0.700 | 0.600 | 0.900 |
| Condition | SR (%) | AFR (%) | LNSR (%) | R@5 (%) | NDCG@10 (%) |
| Full Context | 69.0 ±20.8 | 89.0 ±1.2 | 53.5 ±3.9 | 56.3 ±16.0 | 48.2 ±13.2 |
| w/o Ego. Visual View (EVV) | 71.0 ±11.9 | 84.8 ±3.5 | 42.4 ±10.2 | 55.5 ±13.2 | 49.5 ±10.9 |
| w/o EVV & Textual Caption | 65.0 ±6.8 | 69.9 ±3.1 | 58.7 ±5.3 | 50.0 ±5.6 | 41.7 ±3.4 |
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Interface | Function |
| Connection | Maintains communication between the runner and the UE5 scene. |
| User Pose | Initializes and updates the embodied user’s pose. |
| Action Execution | Executes predefined UE5 actions for movement, view, and interaction. |
| State Query | Queries object positions and orientations in the scene. |
| Camera Pose | Aligns the first-person camera with the user’s position and viewing direction. |
| Visual Capture | Captures RGB, depth, and object-mask observations. |
| Environment | Model | Outcome | Navigation | Recommendation | |||
| SR (%) | Turn | AFR (%) | LNSR (%) | R@5 (%) | NDCG@10 (%) | ||
| Supermarket | Qwen3.6-Plus | 52.0 | 11.9 | 68.8 | 32.6 | 41.7 | 35.1 |
| GPT-4o | 68.0 | 9.9 | 68.6 | 15.0 | 35.1 | 30.5 | |
| GPT-5.6 Sol | 72.0 | 10.7 | 89.1 | 32.3 | 59.0 | 50.0 | |
| Claude Opus 5 | 76.0 | 10.8 | 72.7 | 42.7 | 51.6 | 46.3 | |
| Gemini 3.1 Pro | 92.0 | 9.4 | 74.8 | 50.1 | 73.4 | 57.7 | |
| Environment | Model | Outcome | Recommendation | ||
| RPR (%) | RPR Cart (%) | R-MRR@10 (%) | Catalog (%) | ||
| GPT-5.6 Sol Users | |||||
| Supermarket | Qwen3-VL-8B | 26.0 | 76.5 | 53.7 ±6.7 | 97.9 ±1.0 |
| Gemma 4 E4B | 19.0 | 67.9 | 54.7 ±4.8 | 96.4 ±0.3 | |
| GPT-4o | 42.0 | 93.3 | 70.6 ±3.6 | 99.7 ±0.2 | |
| Qwen3.6-Plus | 58.0 | 89.2 | 74.6 ±5.3 | 99.4 ±0.3 | |
| Condition | RPR (%) | RPR Cart (%) |
| Full Context | 64.0 | 92.8 |
| w/o EVV | 64.0 | 90.1 |
| w/o EVV & Textual Caption | 59.0 | 90.8 |
| Stage | Failure Trajectory | Success Trajectory |
| Initial Need | User, T0 “I’m looking for a chair to make our apartment feel more comfortable.” | User, T0 “I’m looking for comfortable seating for reading, ideally something I can sit in for a while.” |
| Inspection | User, T3 “This chair looks promising for the apartment; I’d like to see whether its comfort and upholstery feel right.” | User, T2 “This looks like the kind of chair I could settle into with a book. I’d like to confirm its comfort, size, and price before deciding.” |
| Decision | Assistant, T4 “This one checks out nicely — it’s a Scandinavian midcentury lounge chair with a natural wood frame and gray fabric upholstery with charcoal piping, priced at $227. It’s sized for a single seat and works well in a living room or bedroom. I’d go ahead and add it to your cart.” User, T4 “The chair seems comfortable, but the cool gray and sharper shape aren’t quite right for warming up our apartment, so I’ll skip it.” | Assistant, T3 “This chair is a strong fit for extended reading: it’s a single lounge chair with gray fabric upholstery, natural wood open arms, and a splayed-leg frame. The confirmed price is 227 USD, so I recommend adding it to your cart.” User, T3 “This chair sounds comfortable enough for longer reading, and at $227 it fits my budget. I’ll take it.” |
| Terminal | Recorded end recommendation_exhausted | User action add_to_cart Recorded end success_cart_added |
Recommender Systems
Recommender systems predict and rank items for people using interaction histories, content, context, and feedback.
See all 564 papers