MOST: Memory of Object States and Task Progress for Long-Horizon Mobile Manipulation

Diqi He; Jinbang Guo; Yijun Shen; Zhiwei Yu; Huan Yang; Dingwen Zhang; Junwei Han; Zhongyuan Wang

cs.ROOct 31, 2026

MOST: Memory of Object States and Task Progress for Long-Horizon Mobile Manipulation

Authors: Diqi He, Jinbang Guo, Yijun Shen, Zhiwei Yu, Huan Yang, Dingwen Zhang, Junwei Han, Zhongyuan Wang

Organizations: Northwestern Polytechnical University · Beijing Academy of Artificial Intelligence · East China Normal University · Chongqing University of Posts and Telecommunications

Abstract

Long-horizon embodied manipulation requires policies to act on information that is no longer visible. Modified objects may leave the field of view, while completed subtasks continue to constrain future actions. Existing approaches extend temporal context through recurrent state updates, test-time training, or storage and retrieval of past observations. As horizons grow, these mechanisms can still leave the policy to infer task state from long, redundant, and viewpoint-dependent visual experience. We argue that long-horizon agents benefit not merely from remembering more, but from remembering task-relevant state changes and progress. We propose Memory of Object States and Task Progress (MOST), a policy-internal memory framework that compresses multi-view observations into temporal tokens. Historical and predictive future tokens are explicitly supervised with object states and subtask progress, structuring memory around what has changed in the world and what has already been accomplished. The resulting memory directly conditions action generation within a single end-to-end policy, without an external planner or retrieval stage. We further introduce RoboCasa-Long, a benchmark of multi-minute, multi-stage mobile manipulation tasks. Across RoboCasa365, RoboCasa-Long, and real-world robot experiments, MOST consistently outperforms Xiaomi-Robotics-1 (XR-1), the previous state-of-the-art model. It raises average success from 65.9% to 72.1% on RoboCasa365, average maximum task progress from 20.83% to 42.22% on RoboCasa-Long, and real-world success from 6.66% to 43.33%. MOST also outperforms a matched visual-history memory variant, showing that effective long-horizon memory depends not only on how much experience is retained, but on whether it preserves decision-relevant progress and world-state changes.

Figures & tables

Figure 1: MOST structures policy-internal memory around object states and task progress for long-horizon mobile manipulation. We evaluate it on RoboCasa365, our newly introduced RoboCasa-Long benchmark of multi-minute, multi-stage tasks, and real-world deployment. M-Raw in the plots denotes MOST-Raw, a matched visual-history memory variant without task-state supervision or future prediction. Across these settings, MOST outperforms both the memoryless Xiaomi-Robotics-1 (XR-1) baseline and MOST-Raw.

Figure 2: MOST overview. MOST integrates a Vision Language Model, a Memory Expert, and an Action Expert within one end-to-end policy. The Vision Language Model encodes the current observation and instruction. The Memory Expert represents sampled history as object states and task progress, and the Action Expert uses both contexts to generate continuous actions. Task-state supervision is training-only.

Figure 3: Memory Expert architecture. Historical multi-view observations are encoded into history tokens, and future queries are appended to form the memory sequence. The sequence is processed with block-causal self-attention and conditioned on the instruction through cross-attention. During training, each token predicts its subtask and the state of every relevant object. The output of each Memory Expert layer conditions the corresponding Action Expert layer.

Data	Method	Atomic $\uparrow$	Composite-S $\uparrow$	Composite-U $\uparrow$	Average $\uparrow$
Target 10%	GR00T N1.5	57.0	28.7	32.3	40.0
	XR-1	60.1	44.3	43.1	49.6
	MOST-Raw	61.1	48.3	45.2	51.9
	MOST (Ours)	63.3	51.0	48.7	54.6
Target 100%	GR00T N1.5	70.0	41.2	44.0	52.4
	XR-1	78.5	57.9	60.0	65.9

Table 1: Success rate (%) on the 50-task RoboCasa365 target suite under Target-10% and Target-100%. GR00T N1.5 split rates are published results, while XR-1, MOST-Raw, and MOST use our matched protocol. Averages are recomputed from the displayed split rates with 18/16/16 task weighting and truncated to one decimal place. The best result in each setting is shown in bold .

Data	Method	SR $\uparrow$	MC@25 $\uparrow$	MC@50 $\uparrow$	MC@75 $\uparrow$	MP $\uparrow$	PPL $\uparrow$	OPS $\uparrow$	UPS $\uparrow$
Com-S-10%	XR-1	44.37	88.12	77.29	61.45	72.34	57.64	66.42	79.22
	MOST-Raw	48.33	86.45	77.70	64.16	73.78	58.54	67.12	79.46
	MOST (Ours)	51.04	89.16	82.50	68.75	77.29	62.00	71.86	81.76
Com-U-10%	XR-1	43.12	94.37	80.83	58.75	73.30	56.02	66.92	81.08
	MOST-Raw	45.20	94.16	81.66	61.04	74.41	56.99	68.69	81.05
	MOST (Ours)	48.75	95.41	83.33	60.83	75.79	57.55	69.11	83.49

Table 2: Progress-aware evaluation (%) on RoboCasa365 composite tasks. XR-1, MOST-Raw, and MOST use matched data, training budgets, and evaluation protocols. Best results within each setting are highlighted in blue and bold . Second-best results use a lighter blue.

Figure 4: Horizon analysis on RoboCasa365 Composite-Seen under Target-100%. We sort 16 tasks by demonstration length into four groups from shortest (G1) to longest (G4) and report mean changes from XR-1. MOST separates most clearly from unstructured visual history on G3 and G4.

Task	Method	SR $\uparrow$	MC@25 $\uparrow$	MC@50 $\uparrow$	MC@75 $\uparrow$	MP $\uparrow$	PPL $\uparrow$	OPS $\uparrow$	UPS $\uparrow$
PackFruit	XR-1	0.00	43.33	6.66	0.00	18.05	10.93	14.16	27.50
	MOST-Raw	0.00	76.66	40.00	0.00	38.61	23.72	34.16	57.50
	MOST (Ours)	10.00	93.33	76.66	30.00	61.84	43.57	58.84	82.99
Prepare Breakfast	XR-1	0.00	66.66	0.00	0.00	26.66	17.93	20.33	35.00
	MOST-Raw	0.00	76.66	10.00	0.00	27.66	15.76	19.33	42.33
	MOST (Ours)	6.66	86.66	16.66	6.66	37.14	36.19	35.95	71.66

Table 3: Evaluation on our RoboCasa-Long benchmark. We compare XR-1, MOST-Raw, and MOST on three multi-minute, multi-stage manipulation tasks. Best results, including ties, are highlighted in blue and bold ; a distinct second-best result uses lighter blue.

Figure 5: RoboCasa-Long task demonstrations. From top to bottom, each row follows one PackFruit , PrepareBreakfast , or CleanAndStoreAfterMeal demonstration from left to right.

Memory Representation	SR $\uparrow$	MC@25 $\uparrow$	MC@50 $\uparrow$	MC@75 $\uparrow$	MP $\uparrow$	PPL $\uparrow$	OPS $\uparrow$	UPS $\uparrow$
No Memory (XR-1)	57.91	90.83	84.37	70.41	79.48	63.55	74.83	84.56
Visual History (MOST-Raw)	63.33	90.62	85.00	73.95	81.68	65.06	76.67	86.82
Current Task State	56.45	89.37	81.66	68.54	77.50	61.35	72.24	83.00
Task-State History	65.20	91.04	84.79	75.83	82.18	66.23	77.88	86.67
Subtask Trajectory	67.08	92.91	86.45	76.66	83.67	67.88	79.25	88.02
Object-State Trajectory	66.04	92.70	87.08	76.66	83.67	68.09	79.31	86.87

Table 4: Controlled memory-representation study on RoboCasa365 Target-100% Composite-Seen. Darker blue denotes higher performance; the lowest result is unshaded and the best is bold .

Task	Method	SR $\uparrow$	MC@25 $\uparrow$	MC@50 $\uparrow$	MC@75 $\uparrow$	MP $\uparrow$	PPL $\uparrow$
Pack 1 Banana	XR-1	0.00	80.00	50.00	0.00	43.33	38.33
	MOST-Raw	40.00	70.00	50.00	40.00	53.33	50.00
	MOST (Ours)	40.00	80.00	70.00	40.00	63.33	61.67
Pack 2 Bananas	XR-1	10.00	60.00	50.00	50.00	50.00	48.00
	MOST-Raw	40.00	90.00	70.00	50.00	68.00	65.00
	MOST (Ours)	50.00	90.00	90.00	80.00	82.00	79.50

Table 5: Real-world long-horizon evaluation on three count-conditioned task variants. For each instruction, the robot must transfer the requested number of bananas one by one and then press the completion button.

Figure 6: Representative successful real-world rollouts on a wheeled manipulator. MOST transfers one banana and stops in the top row. For Pack 3 Bananas , the trajectory continues from the middle row into the bottom row, which is read from right to left, before the robot presses the completion button.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Task	SR $\uparrow$	OPS $\uparrow$	UPS $\uparrow$	NCT $\downarrow$	MP $\uparrow$	MC@25 $\uparrow$	MC@50 $\uparrow$	MC@75 $\uparrow$	PPL $\uparrow$	CRA $\downarrow$	DRR $\uparrow$	FNS $\uparrow$
DeliverStraw	33.33	50.00	75.55	86.80	68.88	83.33	73.33	50.00	50.81	11.05	25.00	46.16
GetToastedBread	83.33	90.83	95.83	68.03	90.83	93.33	93.33	90.00	89.72	0.00	100.00	46.50
KettleBoiling	70.00	81.11	86.66	67.96	83.33	90.00	90.00	70.00	83.33	0.00	–	35.55
LoadDishwasher	66.66	78.66	90.00	73.75	84.66	93.33	90.00	76.66	62.40	6.94	77.27	50.00
PackIdenticalLunches	20.00	47.50	69.16	92.56	67.50	100.00	76.66	43.33	41.96	18.87	62.20	52.60
PreSoakPan	53.33	86.66	89.33	76.38	87.33	96.66	93.33	93.33	81.50	1.12	100.00	79.28

Appendix

Table 6: Per-task performance (%) of MOST on RoboCasa365 Target-10% Composite-Seen.

Task	SR $\uparrow$	OPS $\uparrow$	UPS $\uparrow$	NCT $\downarrow$	MP $\uparrow$	MC@25 $\uparrow$	MC@50 $\uparrow$	MC@75 $\uparrow$	PPL $\uparrow$	CRA $\downarrow$	DRR $\uparrow$	FNS $\uparrow$
ArrangeBreadBasket	86.66	88.00	94.83	40.12	89.50	93.33	86.66	86.66	51.16	2.57	96.42	10.62
ArrangeTea	60.00	80.66	89.33	85.53	85.33	100.00	93.33	73.33	72.97	3.45	57.14	58.33
BreadSelection	50.00	72.50	79.16	76.79	75.83	100.00	93.33	60.00	47.06	5.90	87.50	49.16
CategorizeCondiments	13.33	40.83	73.33	96.22	55.83	100.00	76.66	33.33	37.69	21.05	69.56	46.05
CuttingToolSelection	63.33	68.33	80.00	64.37	73.33	83.33	83.33	63.33	42.50	3.91	85.00	24.54
GarnishPancake	80.00	87.77	93.33	64.05	92.22	100.00	96.66	80.00	82.59	3.27	55.55	47.22

Appendix

Table 7: Per-task performance (%) of MOST on RoboCasa365 Target-10% Composite-Unseen.

Task	SR $\uparrow$	OPS $\uparrow$	UPS $\uparrow$	NCT $\downarrow$	MP $\uparrow$	MC@25 $\uparrow$	MC@50 $\uparrow$	MC@75 $\uparrow$	PPL $\uparrow$	CRA $\downarrow$	DRR $\uparrow$	FNS $\uparrow$
DeliverStraw	73.33	81.11	94.44	67.17	92.22	96.66	96.66	83.33	66.78	7.26	65.00	64.16
GetToastedBread	86.66	88.33	94.16	67.99	89.16	90.00	90.00	90.00	83.20	0.72	80.00	21.87
KettleBoiling	90.00	94.44	96.66	57.19	96.66	100.00	100.00	90.00	89.11	0.08	100.00	53.33
LoadDishwasher	96.66	99.33	99.33	58.80	99.33	100.00	100.00	100.00	73.32	0.23	100.00	90.00
PackIdenticalLunches	40.00	66.25	78.75	82.80	78.33	100.00	93.33	60.00	62.69	13.69	66.66	59.72
PreSoakPan	73.33	90.66	93.33	75.65	92.66	96.66	96.66	96.66	86.54	1.62	66.66	80.00

Appendix

Table 8: Per-task performance (%) of MOST on RoboCasa365 Target-100% Composite-Seen.

Task	SR $\uparrow$	OPS $\uparrow$	UPS $\uparrow$	NCT $\downarrow$	MP $\uparrow$	MC@25 $\uparrow$	MC@50 $\uparrow$	MC@75 $\uparrow$	PPL $\uparrow$	CRA $\downarrow$	DRR $\uparrow$	FNS $\uparrow$
ArrangeBreadBasket	86.66	89.33	91.33	40.80	90.00	90.00	86.66	86.66	53.16	1.38	96.29	12.50
ArrangeTea	76.66	90.66	93.33	79.05	92.00	100.00	96.66	86.66	83.86	1.32	77.77	62.85
BreadSelection	83.33	91.66	95.00	66.05	92.50	100.00	100.00	86.66	64.37	0.68	95.65	53.50
CategorizeCondiments	46.66	70.00	85.83	88.36	76.66	100.00	90.00	66.66	67.36	6.93	54.16	55.62
CuttingToolSelection	46.66	50.00	71.66	74.73	65.00	83.33	83.33	46.66	45.83	8.01	43.75	30.93
GarnishPancake	70.00	81.11	91.11	72.22	90.00	100.00	100.00	70.00	78.74	6.92	33.33	53.33

Appendix

Table 9: Per-task performance (%) of MOST on RoboCasa365 Target-100% Composite-Unseen.

Figure 7: RoboCasa365 Composite-Seen visualizations under Target-100%. Each row shows ten sampled frames from one rollout of a different task. The staircase beneath each frame traces exact simulator-derived task progress over the rollout, and the red marker indicates the displayed timestep.

Figure 8: RoboCasa365 Composite-Unseen visualizations under Target-100%. Each row shows ten sampled frames from one rollout of a different task. The staircase beneath each frame traces exact simulator-derived task progress over the rollout, and the red marker indicates the displayed timestep.

Long-Horizon Robotic Manipulation

Long-horizon robotic manipulation studies how robots plan, remember, and recover across extended sequences of physical tasks.

See all 449 papers

Mar 4, 2026

MEM: Multi-Scale Embodied Memory for Vision Language Action Models

Conventionally, memory in end-to-end robotic learning involves inputting a sequence of past observations into the learned policy. However, in complex multi-stage real-world tasks, the robot's memory must represent past events at multiple levels of granularity: from long-term memory that captures abstracted semantic concepts (e.g., a robot cooking dinner should remember which stages of the recipe are already done) to short-term memory that captures recent events and compensates for occlusions (e.g., a robot remembering the object it wants to pick up once its arm occludes it). In this work, our main insight is that an effective memory architecture for long-horizon robotic control should combine multiple modalities to capture these different levels of abstraction. We introduce Multi-Scale Embodied Memory (MEM), an approach for mixed-modal long-horizon memory in robot policies. MEM combines video-based short-horizon memory, compressed via a video encoder, with text-based long-horizon memory. Together, they enable robot policies to perform tasks that span up to fifteen minutes, like cleaning up a kitchen, or preparing a grilled cheese sandwich. Additionally, we find that memory enables MEM policies to intelligently adapt manipulation strategies in-context.

Marcel Torne, Karl Pertsch, Homer Walke +14

Physical Intelligence · Stanford University · UC Berkeley +1

Memory-Augmented Agents Long-Term Memory for Embodied Agents Robotic Control Long-Horizon Robotic Manipulation Vision-Language-Action Models ICL for Robotics

Ask PDF

Oct 15, 2026

Mem4Act: Learning Effective Latent Memory for Long-Horizon Robotic Manipulation

Long-horizon robotic manipulation fundamentally requires memory: agents must retain task-relevant information from past observations and interactions and use it to guide future actions. Existing approaches typically preserve raw historical observations, compressed visual tokens, or language summaries, but these representations either expose the action policy to increasingly long histories or risk discarding task-relevant temporal, semantic, and perceptual information. We introduce Mem4Act, a framework for learning effective latent memory as a fixed-size representation explicitly optimized for long-horizon robotic control. Mem4Act addresses three coupled challenges: how to efficiently cover long histories, what information memory should retain, and how to ensure the policy actually relies on memory. To this end, it organizes history via multi-granularity temporal aggregation, preserves complementary semantic and perceptual information through question-answering supervision and key-frame DINO feature reconstruction, and employs memory-first two-stage training to encourage greater use of memory by the action policy. The resulting memory captures both high-level task state and fine-grained perceptual evidence while providing a bounded interface to the action policy. Across RoboMME, RMBench, and four real-world manipulation tasks, Mem4Act achieves state-of-the-art performance on memory-dependent control, with success rates of 49.88%, 61.0%, and 81.5%, respectively. Extensive ablations further demonstrate the importance of temporal aggregation, complementary memory supervision, and memory-first training.

Yun Zhang, Zhiyu Huang, Tianhui Cai +3

University of California, Los Angeles · North Carolina State University

Long-Horizon Robotic Manipulation Memory-Augmented Agents

Ask PDF

Nov 4, 2026

Robocasa-Mem: Benchmarking Environment and Process Memory in Long-Horizon Mobile Manipulation

In the real world, mobile manipulation robots must perform complex object interactions and make multi-step decisions, often relying on task-relevant information that is no longer observable. However, existing benchmarks provide limited evaluation of such memory-dependent decision-making in mobile manipulation. To address this gap, we introduce RoboCasa-Mem, a new benchmark comprising nine mobile manipulation tasks that require agents to use interaction history at key decision points. Specifically, the tasks are grouped into two categories of memory: One is environment memory, which captures spatial locations, object attributes, and object relationships. The other is process memory, which captures single-task progress and multi-task scheduling. Particularly, we introduce Behavior-Conditioned Evaluation. For tasks with multiple valid execution paths, instead of fixing a single target or trajectory, the evaluator derives later task constraints from the robot's earlier actions. This allows us to evaluate whether later behavior is consistent with the robot's own interaction history while reducing shortcuts based on fixed target assignments. We provide 1,800 successful demonstrations for policy training. We also develop a baseline on our benchmark, namely Hierarchical Memory-Augmented Policy(HMAP). Its high-level planner uses structured entity and process memories to generate subtasks. Its low-level VLA controller uses subgoal-conditioned queries to incorporate VGGT-Omega geometry into action generation. Together, RoboCasa-Mem and HMAP baseline support systematic evaluation of memory-dependent behavior and the development of memory-aware mobile manipulation policies.

Sihang Chen, Chen Ye, Wei Chen +8

University of Science and Technology of China · Tongji University · The Chinese University of Hong Kong +2

Mobile Manipulation Robot Manipulation Benchmarks Long-Horizon Robotic Manipulation Memory-Augmented Agents Robotic Spatial Memory

Ask PDF

Abstract

Long-Horizon Robotic Manipulation

Explore similar work

MEM: Multi-Scale Embodied Memory for Vision Language Action Models

Mem4Act: Learning Effective Latent Memory for Long-Horizon Robotic Manipulation

Robocasa-Mem: Benchmarking Environment and Process Memory in Long-Horizon Mobile Manipulation