Organizations: Center on Frontiers of Computing Studies, School of Computer Science, Peking University · Jingdong Technology Information Technology Co., Ltd
Operating household appliances requires long-horizon planning that is state-dependent and robust to disturbances, yet existing large models fall short, as no sufficiently diverse, task-oriented dataset exists to support such planning. To bridge this gap, we propose MAGE, a scalable data synthesis pipeline that introduces a novel Hierarchical Appliance Graph (HAG) to automatically generate part grounding, long-horizon planning, and closed-loop recovery data from appliance manuals. With MAGE, we build UseAppliance, the first large-scale dataset for manual-grounded appliance manipulation planning, spanning 22 appliance categories with 89K+ part annotations, 53K+ manipulation tasks, and 33K+ closed-loop adjustment steps. Built on UseAppliance, we develop AppliancePlan, an end-to-end model for manual-grounded appliance manipulation planning. On RealAppliance-Bench, AppliancePlan with only 7B parameters achieves over 10x the best baseline on open-loop planning and consistently outperforms state-of-the-art models across all tasks. Real-robot experiments on six household appliances further confirm effective sim-to-real transfer, marking an important step toward general-purpose household robotics.
Figures & tables
Figure 1. Capabilities of our AppliancePlan model. Left: AppliancePlan plans manipulation for diverse appliances and updates the plan when interrupted. Right: AppliancePlan outperforms strong large-model baselines on RealAppliance-Bench ( Gao et al., 2026 ) .
Figure 2. Overview of MAGE. The first two stages (Secs. 3.2 – 3.3 ) construct the Hierarchical Appliance Graph (HAG) via automated collection and human-verified annotation; the latter two (Secs. 3.4 – 3.5 ) consume the completed HAG to synthesize three types of data—part grounding annotations, open-loop manipulation plans, and observation images with closed-loop recovery data.
Figure 3. Statistics of UseAppliance. (a) Long-tail distribution of 89K+ part bounding boxes. (b) Action-transition flow of 53K+ open-loop planning tasks.
Figure 4. Overview of AppliancePlan model. AppliancePlan is an end-to-end model for manual-grounded appliance manipulation planning, trained with three main objectives and three auxiliary objectives.
School of Electronic Engineering, Kumoh National Institute of Technology, Gyeongbuk, S. Korea. · Kinetic Intelligent Machine Lab (KIMLAB), University of Illinois Urbana-Champaign, Champaign, IL 61801 USA.