Grounding Social Perception in Intuitive Physics

Lance Ying; Aydan Y. Huang; Aviv Netanyahu; Andrei Barbu; Boris Katz; Joshua B. Tenenbaum; Tianmin Shu

q-bio.NCMar 28, 2026

Grounding Social Perception in Intuitive Physics

Authors: Lance Ying, Aydan Y. Huang, Aviv Netanyahu, Andrei Barbu, Boris Katz, Joshua B. Tenenbaum, Tianmin Shu

Organizations: Massachusetts Institute of Technology · Harvard University · Johns Hopkins University · Amazon

Abstract

People infer rich social information from others' actions. These inferences are often constrained by the physical world: what agents can do, what obstacles permit, and how the physical actions of agents causally change an environment and other agents' mental states and behavior. We propose that such rich social perception is more than visual pattern matching, but rather a reasoning process grounded in an integration of intuitive psychology with intuitive physics. To test this hypothesis, we introduced PHASE (PHysically grounded Abstract Social Events), a large dataset of procedurally generated animations, depicting physically simulated two-agent interactions on a 2D surface. Each animation follows the style of the Heider and Simmel movie, with systematic variation in environment geometry, object dynamics, agent capacities, goals, and relationships (friendly/adversarial/neutral). We then present a computational model, SIMPLE, a physics-grounded Bayesian inverse planning model that integrates planning, probabilistic planning, and physics simulation to infer agents' goals and relations from their trajectories. Our experimental results showed that SIMPLE achieved high accuracy and agreement with human judgments across diverse scenarios, while feedforward baseline models -- including strong vision-language models -- and physics-agnostic inverse planning failed to achieve human-level performance and did not align with human judgments. These results suggest that our model provides a computational account for how people understand physically grounded social scenes by inverting a generative model of physics and agents.

Figures & tables

Figure 1: ( A ) Examples of real-life social interactions in physical environments: a basketball player trying to block an opponent from shooting a ball; two people carrying a couch together. ( B ) The classic Heider-Simmel animation abstracts such real-life interactions in animated displays of simple geometric shapes. ( C ) PHASE models social interactions as physically grounded abstract events, where animated agents with limited fields of view move in a physics-based environment with objects, landmarks, and obstacles, enabling behaviors such as helping, hindering, or collaborating toward goals.

Figure 2: Example PHASE animations for different scenario types. A). In this collaborative scenario, agents collaborate to move the pink circle to the yellow landmark. The green agent is too large to move past the barrier. Therefore, the red agent first moves the circle to the left of the barrier and then carries it together with the green agent to the yellow landmark. B). Two agents move towards different landmarks with no interaction. C). In this competing scenario, the agents want to move the blue circle to different landmarks, but there is only one blue circle. They therefore pull the circle in different directions. D). In this opposing scenario, the red agent tries to chase the green agent, while the green agent tries to get away from the red agent. The videos for these examples can be found at https://osf.io/fkp5m/ .

Scenario Type	Description	Relation
Collaborative	Two agents collaborate on a shared goal.	Friendly
Independent	The two agents have different, non-conflicting goals.	Neutral
Competing	The two agents have conflicting goals about the same object.	Adversarial
Opposing	The agents have opposite goals, where one agent’s goal is to intentionally prevent the other agent from reaching its goal.	Adversarial

Table 1: Types of agent goal scenarios, their descriptions, and the corresponding relations.

Figure 3: Consistent human responses showing how many videos (percentages) were assigned with an interaction category by at least 50% of the participants who have watched the videos.

Panel of Figure 4 — Figure 4: ( A ) Pseudo code for the SIMPLE model. ( B ) Illustration of the key model components of SIMPLE. ( C ) Illustration of the joint physical and social simulator in SIMPLE.

Figure 5: Accuracy results for goal classification task across all 100 scenarios and grouped by 4 distinct scenario goal types. The number of goal judgments is shown in the bracket. Humans and models are evaluated on 100 test videos, each with two goal classification tasks and one relation classification task. Error bars show 95% confidence interval from 1000 bootstrapped samples.

Figure 6: Confusion matrix of SIMPLE and SocialGNN on goal classification tasks. (Col = Collaborative, Ind = Independent, Com = Competing, Opp = Opposite)

Figure 7: Correlation between model and human judgments tasks.

Figure 8: Scatterplots comparing human judgment against SIMPLE model judgment on goals (left) and relations (right). The colors indicate the scenario types.

Figure 9: Qualitative examples showing human and model judgments on PHASE animations. Examples A and B show a competing and collaborative scenario, respectively. Across both scenarios, SIMPLE aligns with human distribution on goal and relation classifications, whereas SocialGNN fits less well. Example C shows a competitive scenario where SIMPLE misaligns with humans. In this example, the two agents want to push the blue circle to different landmarks. The green agent eventually steals the blue circle from the red agent. SIMPLE was able to correctly recognize the goal of the red agent, but misclassified the green agent’s goal as hindering the red agent (move the blue circle away from the yellow landmark) and other independent goals (e.g., moving to the red landmark).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure B1 : Interface used to collect human judgments of agent goals and social relations from PHASE videos. Participants selected the most likely goal for each agent and indicated the agents’ relationship after viewing each video.

Predicate	Definition
On ( agent/object , landmark )	An entity is on a landmark
Touch ( agent , agent/object)	An agent touches another entity
Attach ( agent , object )	An object is attached to an agent’s body
Close ( agent/object , agent/object/landmark )	An entity is within a certain distance away from another entity or a landmark

Appendix

Table C1 : Predicates and their definitions. Note that we also consider their negations, which are not shown in the table for brevity.

Figure C1 : Illustration of the effect of the estimated value function for the high-level planner. The numbers indicate the temporal order of the frames. In both sequences, the green agent’s goal is to move the blue object to the red landmark in the bottom-right corner. Since it does not see the blue object initially, it needs to first find the object. ( A ) The sequence when $\lambda=0$ . Since there is more unseen space in the left part of the environment, it is more likely that the blue object is in the left part. So the green agent first searches the left part when not considering the cost of doing so. ( B ) The sequence when $\lambda=0.05$ . When considering the cost, it is worthwhile for the green agent to search the nearby region first. The chance of finding the blue object there is slightly lower than in the left region, but the resulting cost is considerably lower. In particular, it first looks around (frame #2) and then proceeds to search the upper-right part (frame #3 and #4). This comparison demonstrates that an appropriate $\lambda$ could give us more natural agent behaviors under partial observability.

Figure C2 : Illustration of how the red agent updates its belief using $K=10$ particles. ( A ) True states $s^{t}$ (top) and $s^{t+1}$ (bottom). The bright pixels indicate the red agent’s field of view. ( B ) $\{b^{t}_{\text{red},k}\}^{K}_{k=1}$ (top) and $\{b^{t+1}_{\text{red},k}\}^{K}_{k=1}$ (bottom). The states in all the particles are visualized together. At step $t+1$ the red agent observes the blue object via its field of view. All particles are then updated accordingly with the ground truth properties of the blue object, and the inconsistent belief states are also resampled. ( C ) The state in one of the belief particles, $b^{t}_{\text{red},k}$ (top) and $b^{t+1}_{\text{red},k}$ (bottom). The particle is updated with ground truth properties blue object at step $t+1$ . The properties of the pink object are resampled at step $t+1$ since its believed position in step $t$ conflicts with the observation at step $t+1$ .

Cognitive Modeling

Cognitive modeling uses computational theories to explain and predict how people perceive, learn, remember, reason, and decide.

See all 119 papers

Social Intuition vs. Machine Reasoning: Anticipating Human-Robot Interaction from multiple modalities
Sep 7, 2026Raphael Lorenzo-Louis, Bertrand Luvison, Serena IvaldiHuman Behavior Prediction Human-Robot Interaction
Inria, CNRS, UL, Loria, HUCEBOT F-54600 Villers-les-Nancy France
Human-Level Accuracy, Non-Human Strategies: Revealing Model-Human Divergence in Video Physical Reasoning
Sep 19, 2026Fanhong Li, Shurui Zheng, Zi Yin +3Physical Reasoning in Video Physical Foundation Models
Tsinghua University, Beijing, China · ModelBest Inc., Beijing, China · Microsoft, Beijing, China
PhysMind: From Video to Executable Worlds for Training-Free Physical Reasoning
Aug 5, 2026Chen Yang, Shenxiang Zeng, Haoyang Zhao +6Counterfactual Reasoning Physical Reasoning in Video
1Tsinghua University · 2The University of Hong Kong

Sep 7, 2026J/K move · Enter open · S save

Social Intuition vs. Machine Reasoning: Anticipating Human-Robot Interaction from multiple modalities

Raphael Lorenzo-Louis, Bertrand Luvison, Serena Ivaldi

Inria, CNRS, UL, Loria, HUCEBOT F-54600 Villers-les-Nancy France

Human Behavior Prediction Human-Robot Interaction VLMs for Robotics

Open paper AskPDF arXiv

Anticipating whether a person will interact from one's own perspective is a highly intuitive task for humans, that relies on a combination of cues. We investigate how humans perform at predicting a person's intention to interact from a service robot's point of view, using pose-only or full video input, then benchmark different lightweight pose-based models and state-of-the-art vision-language models. We conducted our benchmark on the HUI360 dataset on a fixed pilot subset of 100 test tracks (25 positive, 75 negative). We found that with pose-only input, human annotators outperform lightweight trained pose models but not by large margins (+0.08 in F1-Score). But when given full egocentric video with a target bounding box, human annotators perform substantially better and largely outperform the Vision-Language Models (+0.2 in F1-Score). We also compared VLMs of different size and under different input conditions, and found that the best results do not correlate with model size. Our result confirms that predicting interactions is a challenging task for social robots and that reasoning-capable models are necessary but their actual reasoning capabilities alone do not suffice to match the social intuition of humans.

Abstract

Cognitive Modeling

Explore similar work

Social Intuition vs. Machine Reasoning: Anticipating Human-Robot Interaction from multiple modalities

Human-Level Accuracy, Non-Human Strategies: Revealing Model-Human Divergence in Video Physical Reasoning

PhysMind: From Video to Executable Worlds for Training-Free Physical Reasoning