MatchingPolicy: Correspondence-Aware Policy Enables Cross-Object In-Context Learning

Qijin She; Hanyang Yu; Zeming Li; Ping Tan

cs.ROAug 17, 2026

MatchingPolicy: Correspondence-Aware Policy Enables Cross-Object In-Context Learning

Authors: Qijin She, Hanyang Yu, Zeming Li, Ping Tan

Organizations: Hong Kong University of Science and Technology Hong Kong SAR, China

Abstract

In-context imitation learning enables few-shot policy generalization but struggles to maintain performance on unseen objects and novel scenarios. To address this, we introduce MatchingPolicy, a correspondence-driven framework that explicitly decouples demonstration-to-scene matching from policy learning. Central to our method is a correspondence-aware diffusion policy that conditions robotic actions directly on dense semantic correspondences. This architectural separation resolves the inherent conflict between correspondence identification and action adaptation, enabling robust out-of-distribution transfer. Our framework integrates vision foundation models with a novel two-stage matching algorithm to dynamically establish reliable correspondences. Extensive evaluations on RLBench and real-world manipulation tasks confirm that MatchingPolicy achieves superior few-shot performance, generalizing reliably across unseen object instances and semantic categories.

Figures & tables

Figure 1: The overview of MatchingPolicy. Correspondence points are extracted using a two-stage feature matching algorithm with VFMs and subsequently passed into a graph-based diffusion policy model to predict 6D gripper poses. These predicted motions are then translated into robot joint commands, with the model trained end-to-end on our synthetic dataset.

Figure 2: Our Correspondence-Aware Policy Model. Left: The selected correspondence key points on 2D images of demonstration frames, current frame, and action frames. Right: The structure of our graph-based policy model. The red lines with two-way arrow indicate the one-to-one mapping between the graph nodes in the network and the key points in the current frame. Node lines indicate data transfer. The lines connecting the two frames are a simplified view; the detailed node-to-node links are omitted for visual clarity

Figure 3: Overview of MatchingDataset , MatchingEngine spatially retargets each reference demonstration to produce task-consistent trajectories with ground-truth 3D correspondences.

Figure 4: Real-world Setup and Matching (a) Camera setup (b) View matching based on the average features of segmented objects. (c) Point matching on objects for matched image pairs.

Adaptation Tasks			OOD-Easy Tasks			OOD-Hard Tasks
Task	InstantPolicy	MatchingPolicy	Task	InstantPolicy	MatchingPolicy	Task	InstantPolicy 1 1 footnotemark: 1	MatchingPolicy
Open Box	0.94/0.99	1.00/1.00	Slide Buzzer	0.35/0.94	0.71/0.74	Close Drawer	–/0.00	0.87/0.90
Close Jar	0.58/0.93	0.78/0.92	Plate Out	0.81/0.97	0.96/0.99	Close Grill	–/0.00	0.98/0.97
Toilet Seat Down	0.85/0.93	1.00/1.00	Close Laptop	0.91/0.95	1.00/1.00	Open Grill	–/0.00	0.97/0.99
Close Microwave	1.00/1.00	0.97/1.00	Close Box	0.77/0.99	1.00/1.00	Pick Up Cup	–/0.00	0.90/0.94
Phone on Base	0.98/1.00	1.00/1.00	Open Jar	0.52/0.78	0.89/0.91	Open Door	–/0.01	0.99/0.98

Table 1: Success rates on 36 RLBench tasks. OOD-Easy (Hard) denotes unseen tasks with similar (dissimilar) interaction patterns to the Adaptation Tasks; Results are reported as NoFT/FT, indicating performance without/with fine-tuning on the Adaptation Tasks.

Task	KAT (ICL)		RDT-1B (VLA)		MatchingPolicy
Task	$D{=}2$ 1	$D{=}10$ 2	$D{=}0$ 3	$D{=}20$ 2	$D{=}2$ 1
Plate Out	0.12	0.36	0.08	0.67	0.96
Slide Buzzer	0.04	0.19	0.00	0.47	0.71
Toilet Seat Up	0.25	0.38	0.12	0.69	1.00
Meat off Grill	0.33	0.54	0.05	0.73	0.77
Pick Up Cup	0.28	0.47	0.09	0.81	0.94

Table 2: Comparison with KAT and RDT-1B on six challenging RLBench tasks. MatchingPolicy achieves the highest average success rate using only two demonstrations.

Figure 5: Generalization capabilities of MatchingPolicy with a single demonstration. Left: Novel layouts; Right: Unseen object instances

Level	Method	Pour Water	Put Lid	Open Box	Cut Egg	Averge
Layout-Easy	InstantPolicy	0/10	9/10	0/10	0/10	22.5%
Layout-Easy	MatchingPolicy	7/10	8/10	9/10	7/10	77.5%
Layout-Hard	InstantPolicy	0/10	0/10	0/10	0/10	0.0%
Layout-Hard	MatchingPolicy	6/10	6/10	8/10	6/10	65.0%
Shape	InstantPolicy	0/10	4/10	0/10	0/10	10.0%
Shape	MatchingPolicy	8/10	6/10	8/10	7/10	72.5%

Table 3: Results on real-world tasks, evaluated across different levels of generalization.

Matching	Planning	Layout-Easy	Layout-Hard	Shape
Naive	Overlap	60.0%	20.0%	45.0%
2-Stage	Whole	54.0%	47.5%	57.5%
2-Stage	Half	60.0%	45.0%	45.0%
2-Stage	Overlap	77.5%	65.0%	72.5%

Table 4: Average success rates of our model and ablated variants.

Figure 6: Data generation pipeline of MatchingDataset

Figure 7: The real-world four tasks for evaluation.

Figure 8: The diverse set of test objects

Figure 10: Interaction following shown in the demonstrations.

Explore similar work

Beyond Pixels: Learning Invariant Rewards for Real-World Robotics From a Few Demonstrations
May 21, 2026Tengye Xu, Yangting Sun, Ziju Shen +5Reinforcement Learning Generalization in Robotic Manipulation
School of Computing and Data Science, The University of Hong Kong · 2LimX Dynamics Technology Co., Ltd · 4Peking University +2
ContextFlow: In-Context Flow Matching for Robot Manipulation
Sep 6, 2026Jian Ding, Xianjie Dai, Roei Herzig +5ICL for Robotics Robot Imitation Learning
KAUST, Thuwal, Saudi Arabia · University of California, Berkeley, USA · Zurich, Switzerland
RA-VLA: Retrieval-Augmented VLA for Test-Time Adaptation
Aug 26, 2026Sanghwan Jang, Minjin Jeon, Minsoo Kim +3Test-Time Adaptation of VLA Models Vision-Language-Action Models
Department of Computer Science and Engineering, POSTECH, Pohang, South Korea · Graduate School of Artificial Intelligence, POSTECH, Pohang, South Korea

May 21, 2026J/K move · Enter open · S save

Beyond Pixels: Learning Invariant Rewards for Real-World Robotics From a Few Demonstrations

Tengye Xu, Yangting Sun, Ziju Shen, Guanqi Chen+4

School of Computing and Data Science, The University of Hong Kong · 2LimX Dynamics Technology Co., Ltd · 4Peking University+2

Reinforcement Learning Generalization in Robotic Manipulation Robotic Manipulation Robot Policy Generalization

Open paper AskPDF arXiv

Designing reward functions that generalize beyond controlled laboratory settings remains a fundamental challenge in reinforcement learning for robotics. In open-world manipulation problems, a single task can appear in numerous variants through different object instances, positions, and camera viewpoints. Recent vision-based reward models tend to memorize specific pixel distributions and fail to generalize beyond their training conditions. To address this, we propose a framework that learns invariant symbolic reward functions from as few as five demonstrations. The insight is to shift from visual feature-fitting to the discovery of behavioral invariants: task-level properties that remain constant across diverse visual instantiations. The framework has two coupled components: a structural reward formulation that encodes task-level strategies and physical constraints while preserving optimal policy invariance, and a hybrid symbolic-numerical procedure that distills these invariants from demonstrations without online interaction. Experiments on eight Meta-World tasks and three Franka manipulation tasks demonstrate that our method achieves stronger process alignment and policy rollout ranking abilities compared to baselines, accelerating downstream policy learning. Three real-world out-of-distribution experiments further show that the same learned reward generalizes zero-shot to position, viewpoint, and object variations, enabling a single reward representation to be reused across diverse task variants in practice.

Abstract

Explore similar work

Beyond Pixels: Learning Invariant Rewards for Real-World Robotics From a Few Demonstrations

ContextFlow: In-Context Flow Matching for Robot Manipulation

RA-VLA: Retrieval-Augmented VLA for Test-Time Adaptation