Robots learn reward functions from user demonstrations, but these rewards often fail to generalize to new environments. This failure occurs because learned rewards latch onto spurious correlations in training data rather than the underlying human intent that demonstrations represent. Existing methods leverage visual or semantic similarity to improve robustness, yet these surface-level cues often diverge from what humans actually care about. We present Generalizing Intent for Flexible Test-Time Rewards (GIFT), a framework that grounds reward generalization in human intent rather than surface cues. GIFT leverages language models to infer high-level intent from user demonstrations by contrasting preferred with non-preferred behaviors. At deployment, GIFT maps novel test states to behaviorally equivalent training states via intent-conditioned similarity, enabling learned rewards to generalize across distribution shifts without retraining. We evaluate GIFT on tabletop manipulation tasks with new objects and layouts. Across four simulated tasks with over 50 unseen objects, GIFT consistently outperforms visual and semantic similarity baselines in test-time pairwise win rate and state-alignment F1 score. Real-world experiments on a 7-DoF Franka Panda robot demonstrate that GIFT reliably transfers to physical settings. Further discussion can be found at https://mit-clear-lab.github.io/GIFT/
Figures & tables
Fig. 1: Top. During training, the robot gets task demonstrations (loading a paintbrush) and uses them to infer the user’s high-level intent (“load art supplies”). Bottom. At test time, the robot encounters three unseen objects. GIFT uses the inferred intent to correctly identify that the molding clay is relevant. In contrast, visual-similarity baselines incorrectly prefer the dish scrubber due to its appearance, and language-similarity baselines make an analogous mistake (“ tooth brush ” and “ paint brush. ”)
Fig. 2: G eneralizing I ntent for F lexible T est-Time rewards. Left. GIFT infers the human’s intent given pairs of human-preferred demonstrations and reference trajectories. Right. During inference, GIFT deduces which objects in the unseen states should behave as objects in the training states. Afterwards, the unseen state components are aligned to training states so that the reward function learned before deployment can be used for planning.
Fig. 3: Similarity Between LM-Inferred and Ground Truth Intent. We gave the LM 3 demonstration pairs from a virtual Jaco robot and a real-world Franka robot, and tasked it with deducing the human’s intent. We found that LMs produced an acceptable conditioning variable for alignment.
Fig. 4: Test-Time Pairwise Win Rate. Across our tasks, the rewards learned via GIFT achieved a higher win rate than all other baselines. These results were aggregated over 250 trajectory pairs per scene, with randomization of various state components along with random sampling from a pool of over 50 unseen objects. Black bars denote standard errors.
Fig. 5: Test-Time State Alignment F1 Score. GIFT achieves a superior F1 score by aligning along intent, reducing confounds from superficial language/visual similarity.
Fig. 6: FP/FN (%) on the Confounding States, Sconf . GIFT remains low across categories by using intent-relevance. On the other hand, the oracle baselines merely retune thresholds and trade errors across confounds. Thresholding cannot correct a misaligned similarity signal, and therefore they show high errors. GIFT’s ablation, LM NoIntent , is less performant due to not recognizing which test states are intent relevant; leading to a high FN a rate.
Fig. 7: Real World Behavior. By utilizing intent, GIFT planned behavior that was more aligned with the human preferring to pack art supplies and store valuables . The x-axis labels correspond with test trajectories that have the Franka arm store/pack those unseen items. We randomly sampled sets of four trajectories corresponding to the four items and computed the resulting Boltzmann distribution for each induced reward along with the standard error.
Fig. 8: Language–Vision Similarity Plot for Pack Backpack with Art Supplies . Similarities are measured relative to the training-time anchor ( paintbrushes ) at (1.0,1.0) . Test objects such as the tablet with stylus and molding clay fall closer to distractor , highlighting failure cases where relevant art supplies are confused with unrelated objects. Items like broomstick and toothbrush are incorrectly pulled toward paintbrushes , showing how superficial visual or language similarity can lead to false positives.
Fig. 9: Language–Vision Similarity Plot for Store Valuables . Similarities are measured relative to the training-time anchor ( diamond ring ) at (1.0,1.0) . Test objects such as the MacBook Pro and iPad Pro fall close to the anchor, reflecting correct generalization to unseen valuable items. In contrast, irrelevant objects like the paper ring cluster further away, indicating proper separation between true valuables and distractors.
Learning reward functions from demonstrations assumes that demonstrations provide adequate supervision over all features -- or task-relevant aspects of behavior. In practice, demonstrations are often imperfect: humans may under-emphasize certain features due to cognitive load or physical difficulty, or the training regime may fail to sufficiently cover all relevant situations. In either case, important features may be underspecified, leading to ambiguity in the learned reward function and misaligned behavior at deployment. We propose a framework that detects such underspecified features and actively solicits targeted corrective demonstrations. Our key insight is that demonstrations implicitly reveal which features are well specified: features that are consistently optimized show little variation across demonstrations, while features that are underspecified vary widely. We leverage this statistical signal to infer which features may have been insufficiently demonstrated. The robot then explains which features it is uncertain about in natural language and queries for demonstrations that explicitly address the identified gaps. We evaluate our approach in a simulated tabletop manipulation domain and in a user study with a real Franka robot. Targeted, explanation-guided queries significantly improve reward recovery compared to random querying and passive data collection, reducing ambiguity that would otherwise persist in learning from imperfect demonstrations.
Designing reward functions that generalize beyond controlled laboratory settings remains a fundamental challenge in reinforcement learning for robotics. In open-world manipulation problems, a single task can appear in numerous variants through different object instances, positions, and camera viewpoints. Recent vision-based reward models tend to memorize specific pixel distributions and fail to generalize beyond their training conditions. To address this, we propose a framework that learns invariant symbolic reward functions from as few as five demonstrations. The insight is to shift from visual feature-fitting to the discovery of behavioral invariants: task-level properties that remain constant across diverse visual instantiations. The framework has two coupled components: a structural reward formulation that encodes task-level strategies and physical constraints while preserving optimal policy invariance, and a hybrid symbolic-numerical procedure that distills these invariants from demonstrations without online interaction. Experiments on eight Meta-World tasks and three Franka manipulation tasks demonstrate that our method achieves stronger process alignment and policy rollout ranking abilities compared to baselines, accelerating downstream policy learning. Three real-world out-of-distribution experiments further show that the same learned reward generalizes zero-shot to position, viewpoint, and object variations, enabling a single reward representation to be reused across diverse task variants in practice.
Tengye Xu, Yangting Sun, Ziju Shen +5
School of Computing and Data Science, The University of Hong Kong · 2LimX Dynamics Technology Co., Ltd · 4Peking University +2
Reinforcement learning relies on accurate reward functions, which are often hand-crafted or even unavailable in real-world applications, such as robotics. Recent work has explored the zero-shot reasoning capabilities of pre-trained Vision-Language Models (VLMs) as reward models. However, without careful prompt engineering, these approaches tend to produce suboptimal rewards, where false positive predictions can severely degrade downstream policy learning. In robotics, limited datasets comprising expert demonstrations are often collected to bootstrap policy learning. This scenario provides an opportunity to optimize a reward model prior policy training. We propose Demo2Reward a test-time adaptation technique to optimize the language instruction of a reward model based on a few demonstrations (3-10 trajectories) to reduce false positives while preserving true positives. Crucially, this requires no additional model training or computation resources during policy learning. We show that Demo2Reward consistently outperforms existing zero- and few-shot VLM reward models across a range of simulated robotic tasks and policy backbones. Finally, we demonstrate that Demo2Reward effectively transfers to a real-world robotic learning scenario, enabling policy learning without manually engineering a reward function.
Christian Gumbsch, Leonardo Barcellona, Lennard Schünemann +7
University of Amsterdam · 2Catholic University of Leuven · 3Toyota Research Institute +1