q-bio.NCMar 28, 2026

Grounding Social Perception in Intuitive Physics

Authors: Lance Ying, Aydan Y. Huang, Aviv Netanyahu, Andrei Barbu, Boris Katz, Joshua B. Tenenbaum, Tianmin Shu

Organizations: Massachusetts Institute of Technology · Harvard University · Johns Hopkins University · Amazon

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

People infer rich social information from others' actions. These inferences are often constrained by the physical world: what agents can do, what obstacles permit, and how the physical actions of agents causally change an environment and other agents' mental states and behavior. We propose that such rich social perception is more than visual pattern matching, but rather a reasoning process grounded in an integration of intuitive psychology with intuitive physics. To test this hypothesis, we introduced PHASE (PHysically grounded Abstract Social Events), a large dataset of procedurally generated animations, depicting physically simulated two-agent interactions on a 2D surface. Each animation follows the style of the Heider and Simmel movie, with systematic variation in environment geometry, object dynamics, agent capacities, goals, and relationships (friendly/adversarial/neutral). We then present a computational model, SIMPLE, a physics-grounded Bayesian inverse planning model that integrates planning, probabilistic planning, and physics simulation to infer agents' goals and relations from their trajectories. Our experimental results showed that SIMPLE achieved high accuracy and agreement with human judgments across diverse scenarios, while feedforward baseline models -- including strong vision-language models -- and physics-agnostic inverse planning failed to achieve human-level performance and did not align with human judgments. These results suggest that our model provides a computational account for how people understand physically grounded social scenes by inverting a generative model of physics and agents.

Figures & tables

Figure 1: ( A ) Examples of real-life social interactions in physical environments: a basketball player trying to block an opponent from shooting a ball; two people carrying a couch together. ( B ) The classic Heider-Simmel animation abstracts such real-life interactions in animated displays of simple geometric shapes. ( C ) PHASE models social interactions as physically grounded abstract events, where animated agents with limited fields of view move in a physics-based environment with objects, landmarks, and obstacles, enabling behaviors such as helping, hindering, or collaborating toward goals.
Figure 2: Example PHASE animations for different scenario types. A). In this collaborative scenario, agents collaborate to move the pink circle to the yellow landmark. The green agent is too large to move past the barrier. Therefore, the red agent first moves the circle to the left of the barrier and then carries it together with the green agent to the yellow landmark. B). Two agents move towards different landmarks with no interaction. C). In this competing scenario, the agents want to move the blue circle to different landmarks, but there is only one blue circle. They therefore pull the circle in different directions. D). In this opposing scenario, the red agent tries to chase the green agent, while the green agent tries to get away from the red agent. The videos for these examples can be found at https://osf.io/fkp5m/ .
Scenario TypeDescriptionRelation
CollaborativeTwo agents collaborate on a shared goal.Friendly
IndependentThe two agents have different, non-conflicting goals.Neutral
CompetingThe two agents have conflicting goals about the same object.Adversarial
OpposingThe agents have opposite goals, where one agent’s goal is to intentionally prevent the other agent from reaching its goal.Adversarial
Table 1: Types of agent goal scenarios, their descriptions, and the corresponding relations.
Figure 3: Consistent human responses showing how many videos (percentages) were assigned with an interaction category by at least 50% of the participants who have watched the videos.
Figure 4: ( A ) Pseudo code for the SIMPLE model. ( B ) Illustration of the key model components of SIMPLE. ( C ) Illustration of the joint physical and social simulator in SIMPLE.
Figure 5: Accuracy results for goal classification task across all 100 scenarios and grouped by 4 distinct scenario goal types. The number of goal judgments is shown in the bracket. Humans and models are evaluated on 100 test videos, each with two goal classification tasks and one relation classification task. Error bars show 95% confidence interval from 1000 bootstrapped samples.
Figure 6: Confusion matrix of SIMPLE and SocialGNN on goal classification tasks. (Col = Collaborative, Ind = Independent, Com = Competing, Opp = Opposite)
Figure 7: Correlation between model and human judgments tasks.
Figure 8: Scatterplots comparing human judgment against SIMPLE model judgment on goals (left) and relations (right). The colors indicate the scenario types.
Figure 9: Qualitative examples showing human and model judgments on PHASE animations. Examples A and B show a competing and collaborative scenario, respectively. Across both scenarios, SIMPLE aligns with human distribution on goal and relation classifications, whereas SocialGNN fits less well. Example C shows a competitive scenario where SIMPLE misaligns with humans. In this example, the two agents want to push the blue circle to different landmarks. The green agent eventually steals the blue circle from the red agent. SIMPLE was able to correctly recognize the goal of the red agent, but misclassified the green agent’s goal as hindering the red agent (move the blue circle away from the yellow landmark) and other independent goals (e.g., moving to the red landmark).
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure B1 : Interface used to collect human judgments of agent goals and social relations from PHASE videos. Participants selected the most likely goal for each agent and indicated the agents’ relationship after viewing each video.
PredicateDefinition
On ( agent/object , landmark )An entity is on a landmark
Touch ( agent , agent/object)An agent touches another entity
Attach ( agent , object )An object is attached to an agent’s body
Close ( agent/object , agent/object/landmark )An entity is within a certain distance away from another entity or a landmark
Appendix
Table C1 : Predicates and their definitions. Note that we also consider their negations, which are not shown in the table for brevity.
Figure C1 : Illustration of the effect of the estimated value function for the high-level planner. The numbers indicate the temporal order of the frames. In both sequences, the green agent’s goal is to move the blue object to the red landmark in the bottom-right corner. Since it does not see the blue object initially, it needs to first find the object. ( A ) The sequence when λ=0\lambda=0 . Since there is more unseen space in the left part of the environment, it is more likely that the blue object is in the left part. So the green agent first searches the left part when not considering the cost of doing so. ( B ) The sequence when λ=0.05\lambda=0.05 . When considering the cost, it is worthwhile for the green agent to search the nearby region first. The chance of finding the blue object there is slightly lower than in the left region, but the resulting cost is considerably lower. In particular, it first looks around (frame #2) and then proceeds to search the upper-right part (frame #3 and #4). This comparison demonstrates that an appropriate λ\lambda could give us more natural agent behaviors under partial observability.
Figure C2 : Illustration of how the red agent updates its belief using K=10K=10 particles. ( A ) True states sts^{t} (top) and st+1s^{t+1} (bottom). The bright pixels indicate the red agent’s field of view. ( B ) {bred,kt}k=1K\{b^{t}_{\text{red},k}\}^{K}_{k=1} (top) and {bred,kt+1}k=1K\{b^{t+1}_{\text{red},k}\}^{K}_{k=1} (bottom). The states in all the particles are visualized together. At step t+1t+1 the red agent observes the blue object via its field of view. All particles are then updated accordingly with the ground truth properties of the blue object, and the inconsistent belief states are also resampled. ( C ) The state in one of the belief particles, bred,ktb^{t}_{\text{red},k} (top) and bred,kt+1b^{t+1}_{\text{red},k} (bottom). The particle is updated with ground truth properties blue object at step t+1t+1 . The properties of the pink object are resampled at step t+1t+1 since its believed position in step tt conflicts with the observation at step t+1t+1 .
More in this topic

Cognitive Modeling

Cognitive modeling uses computational theories to explain and predict how people perceive, learn, remember, reason, and decide.

See all 119 papers

Explore similar work

CardsList
Sep 7, 2026

Social Intuition vs. Machine Reasoning: Anticipating Human-Robot Interaction from multiple modalities

Anticipating whether a person will interact from one's own perspective is a highly intuitive task for humans, that relies on a combination of cues. We investigate how humans perform at predicting a person's intention to interact from a service robot's point of view, using pose-only or full video input, then benchmark different lightweight pose-based models and state-of-the-art vision-language models. We conducted our benchmark on the HUI360 dataset on a fixed pilot subset of 100 test tracks (25 positive, 75 negative). We found that with pose-only input, human annotators outperform lightweight trained pose models but not by large margins (+0.08 in F1-Score). But when given full egocentric video with a target bounding box, human annotators perform substantially better and largely outperform the Vision-Language Models (+0.2 in F1-Score). We also compared VLMs of different size and under different input conditions, and found that the best results do not correlate with model size. Our result confirms that predicting interactions is a challenging task for social robots and that reasoning-capable models are necessary but their actual reasoning capabilities alone do not suffice to match the social intuition of humans.

Raphael Lorenzo-Louis, Bertrand Luvison, Serena Ivaldi

Inria, CNRS, UL, Loria, HUCEBOT F-54600 Villers-les-Nancy France

Human Behavior PredictionHuman-Robot InteractionVLMs for Robotics
AskPDF
Sep 19, 2026

Human-Level Accuracy, Non-Human Strategies: Revealing Model-Human Divergence in Video Physical Reasoning

Video foundation models now reach human-level accuracy on physical-reasoning benchmarks, yet such tasks require predicting unobserved physical outcomes. Do these models perform human-like forward simulation, or do they exploit statistical regularities in visible scenes? Accuracy alone cannot distinguish these strategies. We introduce a distributional evaluation framework that treats model seeds and human raters as populations, enabling comparison of consensus, uncertainty, and strategy. On the Physion benchmark, we evaluate three ViT-L architectures (V-JEPA2, VideoMAEv2, DINOv2). V-JEPA2 narrows the accuracy gap to ~1 percentage point (73.2% vs. 74.2%), yet model-human disagreement reaches 26.4%, far exceeding human-human disagreement (4.8%), with substantially lower agreement (kappa ~ 0.48 vs. 0.91). The divergence follows forward-simulation demands: models outperform humans on geometric reasoning (linking, +11.8 pp) but underperform on gravitational dynamics (rolling, -11.8 pp) and causal chains (dominoes, -10.5 pp). Strategy fingerprinting confirms all three architectures share non-human strategies while none aligns with humans. Attribution analysis suggests that unobservable outcome features, rather than visible scene properties, predict this divergence, consistent with models relying more on scene-level statistical regularities than on explicit forward simulation, a systematic divergence that accuracy alone cannot reveal. Code is available at https://github.com/fanhong-li/model-human-divergence.

Fanhong Li, Shurui Zheng, Zi Yin +3

Tsinghua University, Beijing, China · ModelBest Inc., Beijing, China · Microsoft, Beijing, China

Physical Reasoning in VideoPhysical Foundation ModelsPhysical ReasoningVision Foundation Models
AskPDF
Aug 5, 2026

PhysMind: From Video to Executable Worlds for Training-Free Physical Reasoning

Reliable physical reasoning from video requires understanding how objects move, interact, and respond to interventions. Existing vision-language models (VLMs) often struggle to interpret these dynamics and reason reliably about future and counterfactual outcomes. We introduce PhysMind, a training-free agentic framework that constructs one reusable, question-agnostic executable world per video. PhysMind recovers a temporally consistent dynamic scene through object segmentation, mesh reconstruction, and 6D pose tracking, then fits analytic continuous-time dynamics and latent physical parameters without unrolling a time-stepped simulator. Given a question, it inspects, continues, or edits the world and answers from the resulting trajectories and interactions. Relative to direct chain-of-thought (CoT) reasoning with the same VLM, PhysMind improves accuracy by 38.23 points on CLEVRER and 8.08 points on Physion++. On counterfactual questions, it exceeds the strongest evaluated VLM baseline, GPT-5.5, by 19.25 points.

Chen Yang, Shenxiang Zeng, Haoyang Zhao +6

1Tsinghua University · 2The University of Hong Kong

Counterfactual ReasoningPhysical Reasoning in VideoEmbodied World ModelsTemporal Reasoning in VLMs
AskPDF