MOST: Memory of Object States and Task Progress for Long-Horizon Mobile Manipulation

Diqi He; Jinbang Guo; Yijun Shen; Zhiwei Yu; Huan Yang; Dingwen Zhang; Junwei Han; Zhongyuan Wang

cs.ROOct 31, 2026

MOST: Memory of Object States and Task Progress for Long-Horizon Mobile Manipulation

Authors: Diqi He, Jinbang Guo, Yijun Shen, Zhiwei Yu, Huan Yang, Dingwen Zhang, Junwei Han, Zhongyuan Wang

Organizations: Northwestern Polytechnical University · Beijing Academy of Artificial Intelligence · East China Normal University · Chongqing University of Posts and Telecommunications

Abstract

Long-horizon embodied manipulation requires policies to act on information that is no longer visible. Modified objects may leave the field of view, while completed subtasks continue to constrain future actions. Existing approaches extend temporal context through recurrent state updates, test-time training, or storage and retrieval of past observations. As horizons grow, these mechanisms can still leave the policy to infer task state from long, redundant, and viewpoint-dependent visual experience. We argue that long-horizon agents benefit not merely from remembering more, but from remembering task-relevant state changes and progress. We propose Memory of Object States and Task Progress (MOST), a policy-internal memory framework that compresses multi-view observations into temporal tokens. Historical and predictive future tokens are explicitly supervised with object states and subtask progress, structuring memory around what has changed in the world and what has already been accomplished. The resulting memory directly conditions action generation within a single end-to-end policy, without an external planner or retrieval stage. We further introduce RoboCasa-Long, a benchmark of multi-minute, multi-stage mobile manipulation tasks. Across RoboCasa365, RoboCasa-Long, and real-world robot experiments, MOST consistently outperforms Xiaomi-Robotics-1 (XR-1), the previous state-of-the-art model. It raises average success from 65.9% to 72.1% on RoboCasa365, average maximum task progress from 20.83% to 42.22% on RoboCasa-Long, and real-world success from 6.66% to 43.33%. MOST also outperforms a matched visual-history memory variant, showing that effective long-horizon memory depends not only on how much experience is retained, but on whether it preserves decision-relevant progress and world-state changes.

Figures & tables

Figure 1: MOST structures policy-internal memory around object states and task progress for long-horizon mobile manipulation. We evaluate it on RoboCasa365, our newly introduced RoboCasa-Long benchmark of multi-minute, multi-stage tasks, and real-world deployment. M-Raw in the plots denotes MOST-Raw, a matched visual-history memory variant without task-state supervision or future prediction. Across these settings, MOST outperforms both the memoryless Xiaomi-Robotics-1 (XR-1) baseline and MOST-Raw.

Figure 2: MOST overview. MOST integrates a Vision Language Model, a Memory Expert, and an Action Expert within one end-to-end policy. The Vision Language Model encodes the current observation and instruction. The Memory Expert represents sampled history as object states and task progress, and the Action Expert uses both contexts to generate continuous actions. Task-state supervision is training-only.

Figure 3: Memory Expert architecture. Historical multi-view observations are encoded into history tokens, and future queries are appended to form the memory sequence. The sequence is processed with block-causal self-attention and conditioned on the instruction through cross-attention. During training, each token predicts its subtask and the state of every relevant object. The output of each Memory Expert layer conditions the corresponding Action Expert layer.

Data	Method	Atomic $\uparrow$	Composite-S $\uparrow$	Composite-U $\uparrow$	Average $\uparrow$
Target 10%	GR00T N1.5	57.0	28.7	32.3	40.0
	XR-1	60.1	44.3	43.1	49.6
	MOST-Raw	61.1	48.3	45.2	51.9
	MOST (Ours)	63.3	51.0	48.7	54.6
Target 100%	GR00T N1.5	70.0	41.2	44.0	52.4
	XR-1	78.5	57.9	60.0	65.9

Table 1: Success rate (%) on the 50-task RoboCasa365 target suite under Target-10% and Target-100%. GR00T N1.5 split rates are published results, while XR-1, MOST-Raw, and MOST use our matched protocol. Averages are recomputed from the displayed split rates with 18/16/16 task weighting and truncated to one decimal place. The best result in each setting is shown in bold .

Data	Method	SR $\uparrow$	MC@25 $\uparrow$	MC@50 $\uparrow$	MC@75 $\uparrow$	MP $\uparrow$	PPL $\uparrow$	OPS $\uparrow$	UPS $\uparrow$
Com-S-10%	XR-1	44.37	88.12	77.29	61.45	72.34	57.64	66.42	79.22
	MOST-Raw	48.33	86.45	77.70	64.16	73.78	58.54	67.12	79.46
	MOST (Ours)	51.04	89.16	82.50	68.75	77.29	62.00	71.86	81.76
Com-U-10%	XR-1	43.12	94.37	80.83	58.75	73.30	56.02	66.92	81.08
	MOST-Raw	45.20	94.16	81.66	61.04	74.41	56.99	68.69	81.05
	MOST (Ours)	48.75	95.41	83.33	60.83	75.79	57.55	69.11	83.49

Table 2: Progress-aware evaluation (%) on RoboCasa365 composite tasks. XR-1, MOST-Raw, and MOST use matched data, training budgets, and evaluation protocols. Best results within each setting are highlighted in blue and bold . Second-best results use a lighter blue.

Figure 4: Horizon analysis on RoboCasa365 Composite-Seen under Target-100%. We sort 16 tasks by demonstration length into four groups from shortest (G1) to longest (G4) and report mean changes from XR-1. MOST separates most clearly from unstructured visual history on G3 and G4.

Task	Method	SR $\uparrow$	MC@25 $\uparrow$	MC@50 $\uparrow$	MC@75 $\uparrow$	MP $\uparrow$	PPL $\uparrow$	OPS $\uparrow$	UPS $\uparrow$
PackFruit	XR-1	0.00	43.33	6.66	0.00	18.05	10.93	14.16	27.50
	MOST-Raw	0.00	76.66	40.00	0.00	38.61	23.72	34.16	57.50
	MOST (Ours)	10.00	93.33	76.66	30.00	61.84	43.57	58.84	82.99
Prepare Breakfast	XR-1	0.00	66.66	0.00	0.00	26.66	17.93	20.33	35.00
	MOST-Raw	0.00	76.66	10.00	0.00	27.66	15.76	19.33	42.33
	MOST (Ours)	6.66	86.66	16.66	6.66	37.14	36.19	35.95	71.66

Table 3: Evaluation on our RoboCasa-Long benchmark. We compare XR-1, MOST-Raw, and MOST on three multi-minute, multi-stage manipulation tasks. Best results, including ties, are highlighted in blue and bold ; a distinct second-best result uses lighter blue.

Figure 5: RoboCasa-Long task demonstrations. From top to bottom, each row follows one PackFruit , PrepareBreakfast , or CleanAndStoreAfterMeal demonstration from left to right.

Memory Representation	SR $\uparrow$	MC@25 $\uparrow$	MC@50 $\uparrow$	MC@75 $\uparrow$	MP $\uparrow$	PPL $\uparrow$	OPS $\uparrow$	UPS $\uparrow$
No Memory (XR-1)	57.91	90.83	84.37	70.41	79.48	63.55	74.83	84.56
Visual History (MOST-Raw)	63.33	90.62	85.00	73.95	81.68	65.06	76.67	86.82
Current Task State	56.45	89.37	81.66	68.54	77.50	61.35	72.24	83.00
Task-State History	65.20	91.04	84.79	75.83	82.18	66.23	77.88	86.67
Subtask Trajectory	67.08	92.91	86.45	76.66	83.67	67.88	79.25	88.02
Object-State Trajectory	66.04	92.70	87.08	76.66	83.67	68.09	79.31	86.87

Table 4: Controlled memory-representation study on RoboCasa365 Target-100% Composite-Seen. Darker blue denotes higher performance; the lowest result is unshaded and the best is bold .

Task	Method	SR $\uparrow$	MC@25 $\uparrow$	MC@50 $\uparrow$	MC@75 $\uparrow$	MP $\uparrow$	PPL $\uparrow$
Pack 1 Banana	XR-1	0.00	80.00	50.00	0.00	43.33	38.33
	MOST-Raw	40.00	70.00	50.00	40.00	53.33	50.00
	MOST (Ours)	40.00	80.00	70.00	40.00	63.33	61.67
Pack 2 Bananas	XR-1	10.00	60.00	50.00	50.00	50.00	48.00
	MOST-Raw	40.00	90.00	70.00	50.00	68.00	65.00
	MOST (Ours)	50.00	90.00	90.00	80.00	82.00	79.50

Table 5: Real-world long-horizon evaluation on three count-conditioned task variants. For each instruction, the robot must transfer the requested number of bananas one by one and then press the completion button.

Figure 6: Representative successful real-world rollouts on a wheeled manipulator. MOST transfers one banana and stops in the top row. For Pack 3 Bananas , the trajectory continues from the middle row into the bottom row, which is read from right to left, before the robot presses the completion button.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Task	SR $\uparrow$	OPS $\uparrow$	UPS $\uparrow$	NCT $\downarrow$	MP $\uparrow$	MC@25 $\uparrow$	MC@50 $\uparrow$	MC@75 $\uparrow$	PPL $\uparrow$	CRA $\downarrow$	DRR $\uparrow$	FNS $\uparrow$
DeliverStraw	33.33	50.00	75.55	86.80	68.88	83.33	73.33	50.00	50.81	11.05	25.00	46.16
GetToastedBread	83.33	90.83	95.83	68.03	90.83	93.33	93.33	90.00	89.72	0.00	100.00	46.50
KettleBoiling	70.00	81.11	86.66	67.96	83.33	90.00	90.00	70.00	83.33	0.00	–	35.55
LoadDishwasher	66.66	78.66	90.00	73.75	84.66	93.33	90.00	76.66	62.40	6.94	77.27	50.00
PackIdenticalLunches	20.00	47.50	69.16	92.56	67.50	100.00	76.66	43.33	41.96	18.87	62.20	52.60
PreSoakPan	53.33	86.66	89.33	76.38	87.33	96.66	93.33	93.33	81.50	1.12	100.00	79.28

Appendix

Table 6: Per-task performance (%) of MOST on RoboCasa365 Target-10% Composite-Seen.

Task	SR $\uparrow$	OPS $\uparrow$	UPS $\uparrow$	NCT $\downarrow$	MP $\uparrow$	MC@25 $\uparrow$	MC@50 $\uparrow$	MC@75 $\uparrow$	PPL $\uparrow$	CRA $\downarrow$	DRR $\uparrow$	FNS $\uparrow$
ArrangeBreadBasket	86.66	88.00	94.83	40.12	89.50	93.33	86.66	86.66	51.16	2.57	96.42	10.62
ArrangeTea	60.00	80.66	89.33	85.53	85.33	100.00	93.33	73.33	72.97	3.45	57.14	58.33
BreadSelection	50.00	72.50	79.16	76.79	75.83	100.00	93.33	60.00	47.06	5.90	87.50	49.16
CategorizeCondiments	13.33	40.83	73.33	96.22	55.83	100.00	76.66	33.33	37.69	21.05	69.56	46.05
CuttingToolSelection	63.33	68.33	80.00	64.37	73.33	83.33	83.33	63.33	42.50	3.91	85.00	24.54
GarnishPancake	80.00	87.77	93.33	64.05	92.22	100.00	96.66	80.00	82.59	3.27	55.55	47.22

Appendix

Table 7: Per-task performance (%) of MOST on RoboCasa365 Target-10% Composite-Unseen.

Task	SR $\uparrow$	OPS $\uparrow$	UPS $\uparrow$	NCT $\downarrow$	MP $\uparrow$	MC@25 $\uparrow$	MC@50 $\uparrow$	MC@75 $\uparrow$	PPL $\uparrow$	CRA $\downarrow$	DRR $\uparrow$	FNS $\uparrow$
DeliverStraw	73.33	81.11	94.44	67.17	92.22	96.66	96.66	83.33	66.78	7.26	65.00	64.16
GetToastedBread	86.66	88.33	94.16	67.99	89.16	90.00	90.00	90.00	83.20	0.72	80.00	21.87
KettleBoiling	90.00	94.44	96.66	57.19	96.66	100.00	100.00	90.00	89.11	0.08	100.00	53.33
LoadDishwasher	96.66	99.33	99.33	58.80	99.33	100.00	100.00	100.00	73.32	0.23	100.00	90.00
PackIdenticalLunches	40.00	66.25	78.75	82.80	78.33	100.00	93.33	60.00	62.69	13.69	66.66	59.72
PreSoakPan	73.33	90.66	93.33	75.65	92.66	96.66	96.66	96.66	86.54	1.62	66.66	80.00

Appendix

Table 8: Per-task performance (%) of MOST on RoboCasa365 Target-100% Composite-Seen.

Task	SR $\uparrow$	OPS $\uparrow$	UPS $\uparrow$	NCT $\downarrow$	MP $\uparrow$	MC@25 $\uparrow$	MC@50 $\uparrow$	MC@75 $\uparrow$	PPL $\uparrow$	CRA $\downarrow$	DRR $\uparrow$	FNS $\uparrow$
ArrangeBreadBasket	86.66	89.33	91.33	40.80	90.00	90.00	86.66	86.66	53.16	1.38	96.29	12.50
ArrangeTea	76.66	90.66	93.33	79.05	92.00	100.00	96.66	86.66	83.86	1.32	77.77	62.85
BreadSelection	83.33	91.66	95.00	66.05	92.50	100.00	100.00	86.66	64.37	0.68	95.65	53.50
CategorizeCondiments	46.66	70.00	85.83	88.36	76.66	100.00	90.00	66.66	67.36	6.93	54.16	55.62
CuttingToolSelection	46.66	50.00	71.66	74.73	65.00	83.33	83.33	46.66	45.83	8.01	43.75	30.93
GarnishPancake	70.00	81.11	91.11	72.22	90.00	100.00	100.00	70.00	78.74	6.92	33.33	53.33

Appendix

Table 9: Per-task performance (%) of MOST on RoboCasa365 Target-100% Composite-Unseen.

Figure 7: RoboCasa365 Composite-Seen visualizations under Target-100%. Each row shows ten sampled frames from one rollout of a different task. The staircase beneath each frame traces exact simulator-derived task progress over the rollout, and the red marker indicates the displayed timestep.

Figure 8: RoboCasa365 Composite-Unseen visualizations under Target-100%. Each row shows ten sampled frames from one rollout of a different task. The staircase beneath each frame traces exact simulator-derived task progress over the rollout, and the red marker indicates the displayed timestep.

Long-Horizon Robotic Manipulation

Long-horizon robotic manipulation studies how robots plan, remember, and recover across extended sequences of physical tasks.

See all 449 papers

MEM: Multi-Scale Embodied Memory for Vision Language Action Models
Mar 4, 2026Marcel Torne, Karl Pertsch, Homer Walke +14Memory-Augmented Agents Long-Term Memory for Embodied Agents
Physical Intelligence · Stanford University · UC Berkeley +1
Mem4Act: Learning Effective Latent Memory for Long-Horizon Robotic Manipulation
Oct 15, 2026Yun Zhang, Zhiyu Huang, Tianhui Cai +3Long-Horizon Robotic Manipulation Memory-Augmented Agents
University of California, Los Angeles · North Carolina State University
Robocasa-Mem: Benchmarking Environment and Process Memory in Long-Horizon Mobile Manipulation
Nov 4, 2026Sihang Chen, Chen Ye, Wei Chen +8Mobile Manipulation Robot Manipulation Benchmarks
University of Science and Technology of China · Tongji University · The Chinese University of Hong Kong +2

Mar 4, 2026J/K move · Enter open · S save

MEM: Multi-Scale Embodied Memory for Vision Language Action Models

Marcel Torne, Karl Pertsch, Homer Walke, Kyle Vedder+13

Physical Intelligence · Stanford University · UC Berkeley+1

Memory-Augmented Agents Long-Term Memory for Embodied Agents Robotic Control Long-Horizon Robotic Manipulation Vision-Language-Action Models ICL for Robotics

Open paper AskPDF arXiv

Conventionally, memory in end-to-end robotic learning involves inputting a sequence of past observations into the learned policy. However, in complex multi-stage real-world tasks, the robot's memory must represent past events at multiple levels of granularity: from long-term memory that captures abstracted semantic concepts (e.g., a robot cooking dinner should remember which stages of the recipe are already done) to short-term memory that captures recent events and compensates for occlusions (e.g., a robot remembering the object it wants to pick up once its arm occludes it). In this work, our main insight is that an effective memory architecture for long-horizon robotic control should combine multiple modalities to capture these different levels of abstraction. We introduce Multi-Scale Embodied Memory (MEM), an approach for mixed-modal long-horizon memory in robot policies. MEM combines video-based short-horizon memory, compressed via a video encoder, with text-based long-horizon memory. Together, they enable robot policies to perform tasks that span up to fifteen minutes, like cleaning up a kitchen, or preparing a grilled cheese sandwich. Additionally, we find that memory enables MEM policies to intelligently adapt manipulation strategies in-context.

Abstract

Long-Horizon Robotic Manipulation

Explore similar work

MEM: Multi-Scale Embodied Memory for Vision Language Action Models

Mem4Act: Learning Effective Latent Memory for Long-Horizon Robotic Manipulation

Robocasa-Mem: Benchmarking Environment and Process Memory in Long-Horizon Mobile Manipulation