From Vision to Interaction: Evaluating Dynamic Social Intelligence of VLM Agents in ALIVE-World
Organizations: Department of Computer Science and Technology, Tsinghua University, Beijing, China · BNRist, Tsinghua University · Key Laboratory of Pervasive Computing, Ministry of Education
Abstract
As AI agents are increasingly deployed in real-world applications and take on social roles, dynamic social intelligence, the ability to continuously perceive multimodal cues, infer latent cognitive states, intervene through embodied actions, and ultimately achieve social goals, has become essential. However, existing evaluations largely fragment social intelligence into text-based abstractions, static reasoning tasks, or embodied simulators that lack social dynamics. These approaches fail to capture the multimodal, continuous, and interpersonal complexity of authentic human interactions. To bridge this gap, we introduce ALIVE-World, an embodied simulation environment supporting dynamic multimodal interactions encompassing egocentric vision, language, and non-verbal action. To provide socially grounded interactive feedback, we populate this environment with counterpart agents driven by a profile-conditioned, dual-system (intuitive-affective and logical-cognitive) architecture. Building upon this platform, we propose ALIVE-Bench to transform the assessment of social intelligence into a trackable quantitative framework. By deeply leveraging underlying environmental states and agents' internal variables, ALIVE-Bench systematically evaluates Vision-Language Model (VLM) agents across four critical dimensions: perception, understanding, influence, and outcome. Extensive evaluations reveal that dynamic social intelligence is fundamentally multi-dimensional and scenario-dependent. We identify a critical bottleneck between upstream perception and downstream social intervention, exposing VLM vulnerabilities in fine-grained proxemic awareness, adaptive intervention, and multi-party coordination. By uncovering these limitations and the diverse interaction strategies they produce, this benchmark provides actionable insights for the future behavioral alignment of multimodal agents.
Figures & tables
| Dimension | Action Example | Parameters | Duration |
| Navigation ( ) | MOVE | Target, Distance, Direction | 10.0s |
| CHANGE | Top, Bottom, Dress | 60.0s | |
| Gesture ( ) | POINT_AT | Target Object / Agent | Instant † |
| RAISE_HAND | None | Instant † | |
| HANDS_RELAX_ON_LAP | None | Instant † | |
| Gaze ( ) | LOOK_AT | Target Object / Agent | Instant † |
| Hierarchy | Dimension | Shopping Scenario Instantiation | LGD Scenario Instantiation |
|---|---|---|---|
| Perception | Kinesic Cue Recognition | Accuracy of bodily action recognition. | Accuracy of bodily action recognition. |
| Proxemic Awareness | Accuracy of spatial position identification. | Accuracy of gaze target identification. | |
| Understanding | Affective State Inference | Accuracy of inferring customers’ modeled affective states. | Accuracy of inferring teammates’ modeled affective states. |
| Cognitive Intent Inference | Accuracy of inferring customers’ purchase intention and item-level interest. | Accuracy of inferring teammates’ stance tendency and core claims. | |
| Influence | Rapport Building | Ability to improve customers’ willingness to engage. | Ability to gain teammates’ interpersonal favor. |
| Persuasive Efficacy | Ability to increase customers’ purchase intention. | Ability to increase teammates’ acceptance intention toward the agent’s proposal. |
| Method | Profile Consistency | Behavioral Naturalness | Social Cue Sensitivity | Affect-Cognition Coherence |
|---|---|---|---|---|
| ALIVE-World ’s Counterpart (Ours) | 4.56 | 4.31 | 4.62 | 4.81 |
| Baseline | ||||
| Profile-Only | 4.31 | 3.25 * | 2.75 *** | 3.88 ** |
| Profile + CoT | 3.75 * | 3.06 * | 3.00 *** | 3.56 ** |
| Profile + Memory-Reflection- Planning-React | 4.00 * | 3.31 ** | 2.94 *** | 3.75 * |
| Ablation | ||||
| Model | Perception | Understanding | Influence | Outcome | ||||
|---|---|---|---|---|---|---|---|---|
| Kinesic | Proxemic | Affect | Intent | Rapport | Persuasion † | Attainment | Depth | |
| Gemini-3-Pro | 80.43% | 86.60% | 63.67% | 80.92% | 7.85 | 12.77 | 43.75% | 3.43 |
| Gemini-3-Flash | 73.80% | 78.95% | 45.69% | 74.16% | 7.72 | 12.07 | 46.88% | 3.13 |
| Claude-Opus-4.6 | 75.27% | 70.77% | 40.63% | 76.66% | 8.09 | 14.46 | 62.50% | 2.45 |
| Claude-Sonnet-4.6 | 77.57% | 71.69% | 55.38% | 68.77% | 8.10 | 13.58 | 56.25% | 3.44 |
| GPT-5 | 71.00% | 78.00% | 41.76% | 69.22% | 7.77 | 14.33 | 53.13% | 3.06 |
| Model | Perception | Understanding | Influence | Outcome | ||||
|---|---|---|---|---|---|---|---|---|
| Kinesic | Proxemic | Affect | Intent | Rapport | Persuasion † | Attainment | Depth | |
| Gemini-3-Pro | 68.15% | 53.27% | 52.14% | 68.93% | 7.41 | 6.80 | 64.00% | 2.25 |
| Gemini-3-Flash | 70.19% | 49.13% | 46.20% | 66.76% | 6.64 | 6.53 | 73.23% | 1.88 |
| Claude-Opus-4.6 | 50.61% | 52.22% | 49.96% | 70.49% | 6.95 | 6.70 | 67.71% | 2.13 |
| Claude-Sonnet-4.6 | 41.09% | 33.25% | 43.25% | 59.76% | 6.60 | 6.11 | 60.96% | 1.81 |
| GPT-5 | 64.47% | 47.52% | 40.36% | 69.91% | 4.97 | 6.94 | 71.60% | 3.67 |
| Model | Novelty- seeking | Impulsive | Choice- overloaded | Quality- perfectionist | Price-value conscious |
|---|---|---|---|---|---|
| GPT-5 | 29.24 | 23.83 | 3.76 | 7.48 | 5.20 |
| o3 | 21.28 | 28.75 | 10.54 | 7.36 | 5.88 |
| Gemini-3-Pro | 25.11 | 13.05 | 5.27 | 2.41 | 4.61 |
| Claude-Opus-4.6 | 14.63 | 10.53 | 9.33 | 14.72 | 11.22 |
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Property | ToMBench ( Chen et al., 2024 ) | Generative Agents ( Park et al., 2023 ) | SOTOPIA ( Zhou et al., 2024 ) | MoMentS ( Villa-Cueva et al., 2025 ) | SoMi-ToM ( Fan et al., 2025b ) | EmbodiedBench ( Yang et al., 2025 ) | SimWorld ( Ye et al., 2025 ) | VirtualCommunity ( Zhou et al., 2026 ) | ALIVE-Bench (Ours) |
|---|---|---|---|---|---|---|---|---|---|
| Embodied 3D Environment | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Egocentric Visual Perception | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Dynamic Real-time Multi-agent Interaction | ✗ | ✓ | ✓ | ✗ | ✗ | ✗ | ✓ | ✓ | ✓ |
| Systematic Social Intelligence Evaluation | ✓ | ✗ | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✓ |
| Fine-Grained Nonverbal Social Cues | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ |
| Human-like Internal Social Dynamics | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ |
| Shelf Zone | Product List |
|---|---|
| Women’s Tops | Green Hooded Sweatshirt (19), Beige Floral Shirt (54), Pink Short-Sleeve T-shirt ($13) |
| Women’s Bottoms | White Slim-Fit Trousers (35), Blue Distressed Shorts (19), Dark Blue Skinny Jeans ($70) |
| Women’s Dresses | Pink Chiffon Dress (42), White Sleeveless Dress (1860), Blue-and-White Paneled Dress ($52) |
| Men’s Tops | Khaki Trench Coat (52), Black Suit Jacket (58), White Graphic T-shirt ($19) |
| Men’s Bottoms | Dark Brown Dress Pants (155), Khaki Casual Trousers (77), Black Side-Stripe Track Pants ($27) |
| Accessories | Blue Sneakers (1090), Black Leather Shoes (27), White Bucket Hat (33), Brown Newsboy Cap (14), Black Handbag (990), Red Backpack (26), Polka-Dot Handbag ($7) |
| Task Type | Representative Prompt |
|---|---|
| Multiple-choice | Hotel profit diagnosis. A resort hotel failed to meet its fourth-quarter profit target. Ten possible management problems have been identified, including weak performance incentives, poor room hygiene, inaccurate reporting, inventory mismatch, and internal disorder. As the newly appointed general manager, select the three most important causes. Public-welfare entrepreneurship funding. A city-level competition has shortlisted ten social-enterprise proposals, such as a vegetarian restaurant, a health-care business, an eco-laundry service, removable wallpaper, elderly clothing retail, environmentally friendly cleaning, an aromatherapy shop, battery rental, eco-tourism guiding, and a cake shop. As a reviewer, choose the three projects that should receive funding. |
| Dilemma | Rule compliance vs. unconventional innovation. In enterprise management, which is more important: strictly following institutional rules, or encouraging innovation that may challenge existing rules? Ability vs. attitude. In employee recruitment, which matters more: professional competence, or attitude and cooperative spirit? |
| Resource Allocation | Emergency departmental budget. A unit has only a limited budget and must divide it across four urgent needs: repairing the office telephone system, renovating a conference room for an important superior-led meeting, covering an employee’s large medical expense, and providing holiday benefits. Give a concrete allocation plan. Company bonus allocation. A company has a fixed bonus pool to distribute across four departments—engineering, marketing, design, and general administration—but equal division is not allowed. Propose a reasonable allocation strategy. |
| Dimension | Action | Parameters | Duration | Note |
|---|---|---|---|---|
| Navigation | MOVE | Target, Distance, Direction | 10s | Move relative to an object or another agent. |
| CHANGE | Top, Bottom, Dress | 60s | Used by customer agents when changing clothes in the fitting room. | |
| LEAVE | None | 10s | Exit the current scene. | |
| Gesture | POINT | Target | Instant | Point toward an object or another agent. |
| RAISE_HAND | None | Instant | Typically used to request the floor in LGD. | |
| THUMB_UP | Target | Instant | Express approval or encouragement. |
| Category | Group | Count |
| Gender | Female | 15 |
| Male | 15 | |
| Age | 20–24 | 5 |
| 25–29 | 6 | |
| 30–34 | 7 | |
| 35–39 | 2 |
| Analytical | Directive | Behavioral | Conceptual | |||||
|---|---|---|---|---|---|---|---|---|
| Model | att | acc | att | acc | att | acc | att | acc |
| GPT-5 | 5.81 | 6.49 | 4.33 | 6.87 | 4.79 | 7.20 | 4.07 | 6.30 |
| o3 | 6.81 | 6.14 | 5.67 | 5.70 | 6.75 | 7.10 | 5.06 | 7.00 |
| Gemini-3-Pro | 6.59 | 6.28 | 7.28 | 6.06 | 7.64 | 7.04 | 7.66 | 6.81 |
| Claude-Opus-4.6 | 6.97 | 5.94 | 5.73 | 5.79 | 7.17 | 7.12 | 7.13 | 7.20 |