Organizations: MAIS, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Zhongguancun Academy, Beijing, China · Noematrix · Shanghai Jiao Tong University, Shanghai, China
Precision manipulation with contact-critical interactions is often history-dependent: visually similar observations can correspond to different latent interaction states and therefore require different actions, while small execution errors can alter task outcomes. Policies relying on the current visual observation alone cannot resolve such ambiguity; force-aware and memory-augmented methods enrich physical or temporal context, while reactive high-rate policies improve local contact response, yet long-horizon temporal reasoning and precision execution remain largely decoupled in existing methods, limiting reliable progression in visually aliased precision manipulation. To bridge this gap, we introduce History-Conditioned Interaction Reasoning and Execution (HIRE), a cross-rate framework comprising a history-conditioned Interaction-State Reasoner (ISR) and a high-rate Interaction-Manifold Executor (IME). ISR encodes ordered wrench history with a temporal wrench encoder and Force Perceiver as persistent physical evidence for state-consistent action generation, while IME structures contact-critical motion into intrinsic progress and transverse correction for precise execution; their cross-rate loop allows the resulting physical traces to inform subsequent reasoning. In real-robot experiments across surface, insertion, and rotational interactions, HIRE achieves at least 90% completion across all evaluated task stages while improving interaction-state disambiguation, execution precision, and generalization. More broadly, HIRE provides a unified reasoning--execution perspective on precision manipulation under history-dependent partial observability, where physical interaction both realizes task intent and reveals latent-state evidence for future decisions. Code will be released upon publication.
Figures & tables
Fig. 2: Overview of HIRE. (a) ISR compresses long-horizon wrench traces into force-event tokens for history-conditioned action generation. (b) IME combines local sensing with a latched CCE for high-rate manifold-structured interaction execution. (c) Representative interaction-manifold instantiations for surface, insertion, and rotational interactions. (d) Cross-rate coupling passes the CCE and embodied contact-entry condition forward, while measured physical interaction evidence returns to subsequent ISR reasoning.
Fig. 3: Representative real-robot execution flows and visually aliased decision states across the three evaluation regimes. The large external-view images are captured from closer visualization viewpoints than the policy’s global-camera observations and are used only for visualization.
Fig. 4: Balanced reasoning accuracy under different inputs.
TABLE III: Generalization to unseen transparent objects.
Fig. 5: Wrench-history signatures under visual interaction-state aliasing.
H.I.-Robot Lab., School of Automation Science and Engineering, South China University of Technology. · Faculty of Mechanical and Aerospace Engineering, The Hong Kong University of Science and Technology, HKSAR. · School of Astronautics, Northwestern Polytechnical University, Xi’an, China. +1