PhysMem: Scaling Test-Time Memory for Embodied Physical Reasoning

Haoyang Li; Yang You; Hao Su; Leonidas Guibas

cs.ROFeb 23, 2026

PhysMem: Scaling Test-Time Memory for Embodied Physical Reasoning

Authors: Haoyang Li, Yang You, Hao Su, Leonidas Guibas

Organizations: UC San Diego · Stanford University

Abstract

Reliable object manipulation requires understanding physical properties that vary across objects and environments. Vision-language model (VLM) planners can reason about friction and stability in general terms; however, they often cannot predict how a specific ball will roll on a particular surface or which stone will provide a stable foundation without direct experience. We present PhysMem, a memory framework that enables VLM robot planners to learn physical principles from interaction at test time, without updating model parameters. The system records experiences, generates candidate hypotheses, and verifies them through targeted interaction before promoting validated knowledge to guide future decisions. A central design choice is verification before application: the system tests hypotheses against new observations rather than applying retrieved experience directly, reducing rigid reliance on prior experience when physical conditions change. We evaluate PhysMem on three real-world manipulation tasks and simulation benchmarks across four VLM backbones. On a controlled brick insertion task, principled abstraction achieves 76% success compared to 23% for direct experience retrieval, and real-world experiments show consistent improvement over 30-minute deployment sessions.

Figures & tables

Fig. 1 : PhysMem learns physical principles through interaction. (a) Continually Learn via the memory consolidation system maintains principles $\mathcal{P}$ , hypotheses $\mathcal{H}$ , and experiences $\mathcal{E}$ , which guide the embodied agent’s actions $a$ ; world feedback (observations $o$ , rewards $r$ ) generates new experiences $e$ that refine knowledge. (b) Test-time learning on Parts Organization : PhysMem (blue) improves continuously while no-memory baseline (gray) remains flat. (c) Qualitative results on Parts Organization : green boxes show covered cells, red dashed boxes indicate potential collisions avoided, and blue circles highlight learned space-saving strategies.

Fig. 2 : System overview of PhysMem . Left (Consolidation): A three-tier memory system stores raw experiences in episodic memory, clusters them into testable hypotheses in working memory, and promotes verified knowledge to long-term memory as principles. The consolidation process continuously refines memory through interaction. Top-right (Embodied Agent): A Vision-Language Model receives language instructions along with retrieved principles and active hypotheses from memory, then outputs high-level decisions that are executed by a low-level policy. Bottom-right (World Interaction): The agent interacts with challenging physical tasks (Parts Organization, Ball Navigation, and Balanced Stacking), and outcomes feed back into the memory system as new experiences.

Paradigm	Examples	Param. update	Memory content	Interp.	Inf. cost
Online RL	RECAP, $\pi_{0.6}$	✓ Gradient	Implicit weights	✗	sec
Trained memory VLA	MemER, MemoryVLA, MEM	✓ Imit. FT	Visual / latent / text	✗	ms
Reflection (in-episode)	Reflexion, Self-Refine	✗	Free-form text	✓	sec
Retrieval	RAG, MAP-VLA	✗	Raw experiences	$\triangle$	ms
Verified abstraction (ours)	PhysMem	✗	Verified principles	✓	ms

TABLE I: Adaptation paradigms for deployed VLM/VLA agents. PhysMem sits in a separate row: frozen base, structured memory, and a verification step before commitment.

Fig. 3 : Memory injection into VLM prompts. Verified principles (blue) and active hypotheses (yellow) are inserted into the planner’s context with confidence scores and typed constraints ( Prefer , Avoid , Sequence ).

Fig. 4 : Experimental environments. (a) Left: Real-world platform with xArm6 robot, fin-ray soft grippers, and multi-view RealSense cameras in an enclosed workspace.Right: The partial props used in the experiments. (b) Reflect-VLM simulation [ 20 ] with Franka Panda robot for large-scale experiments.

Fig. 5 : Real-world tasks. Top: symbolic representations; bottom: actual setups. (a) Grid layout and placement trajectories for Parts Organization. (b) Obstacle course and ball trajectory for Ball Navigation. (c) Stone arrangement and stacking position for Balanced Stacking.

	Success Rate
Method	Easy	Med.	Hard	Tokens
No Memory	83%	53%	28%	0.40 $\times$
Direct Retrieval	48%	23%	8%	0.55 $\times$
MemER-style Retrieval	85%	53%	19%	1.0 $\times$
MemoryVLA-style Token Memory	69%	35%	13%	2.5 $\times$
Reflexion-style	86%	61%	31%	1.2 $\times$

TABLE II: Main results: PhysMem vs. memory paradigms on Brick Insertion (100 episodes per cell, 3 seeds). MemER-style retrieval lands at the No-Memory baseline on medium (53%), and only PhysMem ’s verified principles exceed No Memory on hard. Best per-difficulty in bold.

Fig. 6 : End-state keyframes across tasks and OOD variants. A $2{\times}2$ ablation of prior knowledge and on-the-fly adaptation, shown as final states for four task instances. Rows: (i) Parts Organization on its OOD configuration (covered cells, lower is better); (ii) Ball Navigation with the in-distribution soccer ball; (iii) Ball Navigation with an OOD tennis ball; (iv) Balanced Stacking. Columns: (a) no prior, no adaptation; (b) prior only; (c) adaptation only; (d) full PhysMem . Across all four rows, the full setting reaches the most efficient or most stable end state; either component alone leaves a visible gap that Section V-D quantifies.

Fig. 7 : Resonance score evolution across tasks. Each panel shows RGB keyframes alongside the resonance curve. Resonance $\rho$ measures prediction-outcome alignment: low values indicate surprising outcomes that trigger learning; high values (green region) indicate principled reasoning. Left : Parts Organization progresses from scattered placement to efficient packing. Middle : Ball Navigation learns ball dynamics through trial. Right : Balanced Stacking discovers stable stone combinations. All tasks transition from reactive ( $\rho<0.5$ ) to rational ( $\rho>0.7$ ) behavior within 10 episodes.

Fig. 8 : Test-time evolution across experience utilization levels. Each panel shows learning curves with shaded standard deviation bands (3 runs per condition). Blue : Full memory (100%); Green : 50% experience; Yellow : 25% experience; Gray : No memory (0%). Without memory, performance remains flat across all tasks. Complex dynamics (Ball Navigation) benefit most from full experience, while simpler tasks (Balanced Stacking) show diminishing returns.

		Parts Organization		Ball Navigation		Balanced Stacking
Prior	Adapt	Score	Succ.	Score	Succ.	Score	Succ.
✗	✗	$-0.6$	0/10	1.6	1/10	6.7	4/10
✗	✓	3.3	1/10	5.5	2/10	8.3	7/10
✓	✗	6.9	3/10	2.9	1/10	9.2	8/10
✓	✓	8.3	4/10	7.1	4/10	12.3	9/10

TABLE III: Memory transfer to OOD task variants. We ablate prior knowledge (Prior) and test-time adaptation (Adapt). Prior knowledge helps when physics are similar (Parts Organization, Balanced Stacking). For Ball Navigation with new ball types, adaptation is essential since dynamics differ substantially. Best results in bold . Green : better; Red : worse.

Fig. 9 : Success rates across VLMs and difficulty levels. Bars show baseline performance (w/o PhysMem ); annotations indicate improvement with test-time learning. Gemini-3-Flash dominates across all difficulties and shows the largest improvement. Weaker models benefit less from test-time learning, suggesting that memory amplifies existing capabilities rather than compensating for fundamental limitations.

Fig. 10 : Principle scaling across difficulty levels. Medium tasks (blue) show rapid learning between 2 and 8 principles before stabilizing. Easy tasks (green) saturate quickly with moderate improvement. Hard tasks (gray) show substantial improvement potentially through accumulated failure cases. Shaded region marks the “rapid learning” phase.

	Success Rate
Configuration	Easy	Med.	Hard	Tokens
PhysMem (Full)	89%	76%	39%	1.0 $\times$
Direct Retrieval	48%	23%	8%	0.55 $\times$
w/o Resonance	81%	58%	21%	1.3 $\times$
w/o Verification	85%	64%	27%	0.85 $\times$
w/o Forgetting	91%	78%	36%	3.4 $\times$

TABLE IV: Ablation study across difficulty levels. Success rates and token consumption (on medium) for brick insertion. Direct retrieval validates the need for principle abstraction. Component importance scales with task complexity.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 11 : PhysMem overall pipeline. PhysMem enables VLM robot planners to learn physical principles through interaction. The scientific loop transforms raw experiences into verified knowledge: collecting observations, generating hypotheses, verifying through action-level attribution, and promoting validated principles to guide future decisions.

Memory Capacity
Parameter	Value	Description
$N_{\max}$	3000	Max episodic experiences
Folded TTL	100	Episodes before folded exp. removal
Consolidation
Interval	50	Episodes between consolidation
Min cluster size	2	Experiences per cluster

Appendix

TABLE V : Hyperparameter settings.

Memory Architecture Ablations
	Easy (2–3 bricks)			Medium (4–5 bricks)			Hard (6–8 bricks)
Configuration	Success	$\Delta$	Tokens	Success	$\Delta$	Tokens	Success	$\Delta$	Tokens
PhysMem (Full)	89%	—	1.0 $\times$	76%	—	1.0 $\times$	39%	—	1.0 $\times$
w/o Episodic Memory	54%	$-35$	0.3 $\times$	37%	$-39$	0.25 $\times$	14%	$-25$	0.2 $\times$
w/o Working Memory	84%	$-5$	0.85 $\times$	69%	$-7$	0.8 $\times$	28%	$-11$	0.75 $\times$
w/o Long-term Memory	81%	$-8$	1.15 $\times$	64%	$-12$	1.25 $\times$	26%	$-13$	1.35 $\times$

Appendix

TABLE VI : Complete ablation results across difficulty levels. Success rates and token consumption on simulation benchmark (100 episodes each). Token consumption is normalized relative to the full system at each difficulty level.

\Delta

indicates absolute change from the full system. Best accuracy in bold .

Config	H/C	P/P	Success	Principles	Tokens
Sparse	1	3	71%	12	0.7 $\times$
Balanced	2	5	76%	18	1.0 $\times$
Dense	3	10	74%	31	1.6 $\times$
Hypo-Heavy	3	3	73%	26	1.2 $\times$
Principle-Heavy	1	10	72%	14	1.1 $\times$

Appendix

TABLE VII : Quantity configuration results. H/C = hypotheses per cluster, P/P = principles in prompt. Evaluated on medium difficulty (100 episodes each).

Fig. 12 : Assembly task demonstration across difficulty levels. Each row shows the oracle planner’s execution sequence from initial state to completion. Step numbers indicate execution order. Easy (green, 2 bricks): 5 steps total, minimal dependencies. Medium (blue, 5 bricks): 11 steps with moderate dependency chains. Hard (gray, 7 bricks): 15 steps requiring complex ordering to avoid blocking. The VLM must learn correct insertion sequences through trial and error, as inserting bricks in the wrong order causes failures that provide only sparse error signals.

Fig. 13 : Parts Organization: Component shapes and grid occupancy. Each panel shows a 3D-printed part overlaid on its $2\times 2$ template grid, illustrating which cells ( [a,b,c,d] ) each part occupies. Top row : red-L (2 cells, occupies [a,c] ), red-q (3 cells, [a,c,d] ), white-q with red-L interlocking example. Bottom row : white-U (4 cells, [a,b,c,d] ), black-U (4 cells), black-I (2 cells, [a,c] ). Red grid lines mark template boundaries. Note the internal geometric biases: U-shapes have empty space in [a,c] (the “U” opening), while q-shapes have opposite edge alignments (white-q left-aligned, red-q right-aligned).

Fig. 14 : Out-of-distribution experimental settings and props. (a) Parts Organization OOD : Initial grid configuration is modified with parts pre-placed at different positions. Green cells (14–16, 23–25) are pre-occupied; hatched area indicates the constrained placement region ( $\pm 3$ columns from occupied boundary). Dashed trajectory shows example placement paths. (b) Ball Navigation OOD : Top shows stacking blocks (illustrative only, not used in ball task). Bottom shows all balls: soccer ball (center) is the in-distribution training ball; tennis ball (yellow) and moon-textured plastic ball (red boxes) are OOD items with different friction and elasticity. (c) Balanced Stacking OOD : Top rows show balance stones with varying surfaces—some treated with tapes (3M, black duct, white PVC; shown in bottom), others with natural WOOD or PAINT surfaces. Red-boxed stones (top-right) represent OOD combinations with novel shape/material/size configurations not seen during in-distribution training.

Model	Baseline	+ PhysMem	$\Delta$
Gemini-3-Flash	53%	76%	+23%
GPT-5.1	43%	57%	+14%
Qwen3-VL-235B	38%	50%	+12%
Gemini-ER-1.5	29%	34%	+5%

Appendix

TABLE VIII : Cross-model results on medium difficulty. Success rates with and without PhysMem across different VLM planners (100 episodes each). All models use thinking mode. Consolidation model fixed to Qwen3-VL.

Memory-Augmented Agents

Memory-augmented agents use stored experience and knowledge to act coherently across long tasks, changing environments, and repeated interactions.

See all 347 papers

PhysMRV: Physical Memory Retrieval and Verification for Physics Plausibility Reasoning
Jul 11, 2026Wenyuan Wang, Lianyu Hu, Hao Wang +1Memory-Augmented VLMs Physical Reasoning
1Nanyang Technological University · 2Rutgers University · University of Illinois Urbana-Champaign
Plan-Bench: Can Existing VLMs Go Well Beyond Perception to Reason and Plan in Embodied Worlds?
Oct 13, 2026Juntong Feng, Qingan Liu, Jingru Dong +8VLM Reasoning VLMs for Robotics
Dalian University of Technology · Nanyang Technological University
PhysMind: From Video to Executable Worlds for Training-Free Physical Reasoning
Aug 5, 2026Chen Yang, Shenxiang Zeng, Haoyang Zhao +6Counterfactual Reasoning Physical Reasoning in Video
1Tsinghua University · 2The University of Hong Kong

Jul 11, 2026J/K move · Enter open · S save

PhysMRV: Physical Memory Retrieval and Verification for Physics Plausibility Reasoning

Wenyuan Wang, Lianyu Hu, Hao Wang, Yang Liu

1Nanyang Technological University · 2Rutgers University · University of Illinois Urbana-Champaign

Memory-Augmented VLMs Physical Reasoning VLM Reasoning Physical Reasoning in Video

Open paper AskPDF arXiv

Video-language models (VLMs) have achieved remarkable performance on video understanding and visual question answering, yet they remain unreliable in reasoning about physical plausibility, where understanding object interactions, causal dynamics, and fundamental physical principles is essential. This limitation is particularly evident on challenging physical reasoning benchmarks, revealing a persistent gap in physical commonsense reasoning. To address this challenge, we propose PhysMRV, a training-free physical memory and verification framework for physical plausibility reasoning. Unlike retrieval-augmented VLMs that retrieve semantically similar videos as additional context, PhysMRV transforms training videos into a Hierarchical Memory Bank of structured physical knowledge comprising three complementary levels: scene descriptions capturing visual context, physical-event graphs modeling object interactions and causal structure, and physics-rule summaries distilling reusable physical principles and cues. During inference, PhysMRV retrieves physically relevant memories and leverages their structured physical evidence to guide a frozen VLM in verifying physical plausibility, requiring neither fine-tuning nor parameter updates. We evaluate PhysMRV on three challenging physical reasoning benchmarks, ImplausiBench, IntPhys2, and GRASP Level 2, across multiple state-of-the-art VLMs. Experimental results demonstrate consistent improvements over direct prompting across diverse VLMs and evaluation benchmarks, showing that structured physical memories provide an effective and scalable means of enhancing physical plausibility reasoning without additional training.

Abstract

Memory-Augmented Agents

Explore similar work

PhysMRV: Physical Memory Retrieval and Verification for Physics Plausibility Reasoning

Plan-Bench: Can Existing VLMs Go Well Beyond Perception to Reason and Plan in Embodied Worlds?

PhysMind: From Video to Executable Worlds for Training-Free Physical Reasoning