Vision-language models (VLMs) have shown impressive capabilities across diverse tasks, motivating efforts to leverage these models to supervise robot learning. However, when used as evaluators in reinforcement learning (RL), today's strongest models often fail under partial observability and distribution shift, enabling policies to exploit perceptual errors rather than solve the task. We introduce SOLE-R1 (Self-Observing LEarner), a video-language reasoning model explicitly designed to serve as the sole reward signal for online RL. Given only raw video observations and a natural-language goal, SOLE-R1 performs per-timestep spatiotemporal chain-of-thought (CoT) reasoning and produces dense estimates of task progress that can be used directly as rewards. To train SOLE-R1, we develop a large-scale video trajectory and reasoning synthesis pipeline that generates temporally grounded CoT traces aligned with continuous progress supervision. This data is combined with foundational spatial and multi-frame temporal reasoning, and used to train the model with a hybrid framework that couples supervised fine-tuning with RL from verifiable rewards. Across four different simulation environments and a real-robot setting, SOLE-R1 enables zero-shot online RL from random initialization: robots learn previously unseen manipulation tasks without ground-truth rewards, success indicators, demonstrations, or task-specific tuning. SOLE-R1 succeeds on 24 unseen tasks and substantially outperforms strong vision-language rewarders, including Robometer, RoboReward, ReWiND, GPT-5, and Gemini-3-Pro, while exhibiting markedly greater robustness to reward hacking. We release all models, data, code, and demos at the anonymous page: https://philip-mit.github.io/sole-r1/
Figures & tables
Figure 1: SOLE-R1 is a video-language reasoning model designed to guide online RL with per-timestep chain-of-thought reasoning and progress prediction. In large-scale experiments across 40 tasks, SOLE-R1 outperforms strong baseline models with zero-shot online RL.
Figure 2: SOLE-R1 training data mixture. The dataset combines foundational spatial reasoning, multi-frame temporal reasoning, and our synthesized video trajectories with chain-of-thought explanations and dense progress supervision, jointly enabling reasoning over space and time for progress prediction.
Figure 3: Zero-shot success rate of online RL across 40 tasks. We plot the mean and standard error across three random seeds (real-world experiments use a single seed, shown as a single value). In all experiments, the robot begins with a random policy and learns entirely through interaction with the task, guided only by the predicted rewards.
Figure 4: Perceived vs true success in zero-shot RL. Perceived success is the average max progress predicted (RoboReward is excluded as it does not provide dense rewards). True success is the average max ground-truth reward achieved.
Figure 5
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6
Figure 7: Extended reasoning examples across all models.
Figure 8: Zero-shot RL success vs correlation between predicted and ground-truth progress
Primary qualitative error mode
SOLE-R1
GPT-5
Gemini-3-Pro
Gemini Robotics-ER
Temporal under-detection of contact / state change
34%
14%
17%
19%
Ambiguous object state under partial observability
29%
11%
13%
15%
Relies on goal-consistent appearance cues
15%
18%
20%
21%
Perceptual hallucination of success
9%
42%
38%
35%
Other / unclassified
13%
15%
12%
10%
Appendix
Table 1: Quantitative summary of failure modes from qualitative review of real-robot and simulation rollouts.
Figure 9: SFT-only vs SFT+RLVR model in zero-shot success rate of online RL
DrQv2
Meta-World
RoboSuite, LIBERO, ManiSkill
Real-world
Buffer size
200000
200000
200000
Batch size
256
256
256
Critic to actor steps
4
4
4
Initial random steps
1000
2500
100
Start learning at
1000
2500
100
Critic ensemble size
10
10
10
Appendix
Table 2: Reinforcement learning agent details
Task Name
Natural Language Task Specification
Real-World Tasks
Pick can
Pick up the can
Touch strawberry in clutter
Touch the strawberry on the table
Push can to cube
Push the can near the cube
Close drawer
Close the drawer
Open drawer
Open the drawer
Appendix
Table 3: Task groups and their natural language specifications
Figure 10: Zero-shot success rate of online RL on 10 simple Meta-World tasks across varying prompting strategies.
Figure 11: Left: workcell with two Franka FR3 arms. Right: example shoulder and wrist views from the right robot.
Non-expert trajectory levels
Percentage
Pick-only
1: Fail to approach {obj}
25%
2: Approach {obj}
25%
3: Contact {obj}
25%
4: Pick up {obj}
25%
Pick-and-place
1: Fail to approach {obj}
14%
2: Approach {obj}
14%
Appendix
Table 4: Levels for non-expert trajectories and video dataset counts. For each level, we assume that all task components corresponding to the preceding levels have already been successfully completed.
Data source ID
Total Dataset Count
Video-based CoT Reasoning and Progress Prediction
OXE berkeley_autolab_ur5
4874
OXE berkeley_fanuc_manipulation
2689
OXE bridge
4602
OXE bridge_set2
32596
OXE bridge_set3
51939
Appendix
Table 5: Dataset mixture
SpatialBench
SSRBench
CV-Bench
Method
Size
General
Spatial
Qwen2.5-VL
7B
64.7
69.4
53.6
73.0
Qwen3-VL
8B
65.3
73.1
57.2
73.7
SSR
7B
67.0
82.1
76.1
73.3
SOLE-R1 (Ours)
8B
69.4
85.6
79.2
74.4
Appendix
Table 6: Performance improvement of SOLE-R1 compared to SSR 7B model and the backbone model on SpatialBench, SSRBench, and CV-Bench.
Table 7: Value-Order-Correlation analysis (as done in on [ 34 ] ) comparing SOLE-R1 to GVL on 1,000 videos from OpenX Embodiment datasets.
Figure 12: Example of a successful (top) and failed (bottom) manipulation trajectory generated by SmolVLA in Libero. We predict and plot (red) SOLE-R1 task progress predictions based on the prompt “put the bowl on the cabinet”.
Figure 13: Example of a successful (top) and failed (bottom) manipulation trajectory generated by SmolVLA in Libero. We predict and plot (red) SOLE-R1 task progress predictions based on the prompt “put the cream cheese in the bowl”.
General-purpose robot learning requires dense, instruction-conditioned feedback that can distinguish meaningful task progress from stalled, failed, or partially completed behavior. Yet obtaining such feedback at scale remains difficult, since existing approaches often rely on manual progress annotations, task-specific demonstrations, or reward models trained on curated robot datasets. We introduce TOPReward, a training-free progress reward method that probes pretrained Video-Language Models (VLMs) through their internal token probabilities rather than asking them to generate numerical progress values. Given a video prefix and a language instruction, TOPReward measures the model's likelihood that the instructed task has been completed, converting latent video-language understanding into a dense reward signal without task-specific reward-model training or manually annotated progress labels. We evaluate TOPReward on ManiRewardBench, our real-world manipulation benchmark spanning 130 unique tasks and four robot platforms, as well as on Open X-Embodiment datasets. Across these settings, TOPReward substantially outperforms prior training-free VLM reward methods on open-source models and is competitive with a trained reward-model baseline on progress-estimation metrics, while requiring no reward-model training. Additional analyses show that the reward is sensitive to the specified instruction and is not explained by time index alone. Finally, TOPReward supports downstream applications including success detection and offline reward-weighted behavior cloning.
Shirui Chen, Cole Harrison, Ying-Chun Lee +6
University of Washington · Allen Institute for AI · Amazon +1
Reinforcement Learning (RL) has shown strong potential for improving robotic manipulation policies, yet its practical use remains bottlenecked by the difficulty of specifying reward functions that are both semantically meaningful and reusable across tasks. In this paper, we propose Large Reward Models (LRMs), a framework that adapts foundation VLMs into frame-level reward generators for robot policy refinement. We specialize a state-of-the-art VLM on a multi-source dataset spanning real-world robot trajectories, human-object interactions, and simulated manipulation environments. Unlike prior approaches that mainly evaluate trajectories post-hoc, LRMs expose multiple reward interfaces from visual observations: progress estimation, task completion, and temporal contrastive comparison. Starting from an imitation-learned policy, we use these VLM-derived rewards to guide PPO refinement on held-out long-horizon manipulation tasks. Our experiments show that LRM progress rewards provide the strongest non-privileged online refinement signal, improving the IL baseline and narrowing the gap to privileged environment rewards. We further deploy progress rewards for progress-weighted behavioral cloning on four real-world manipulation tasks spanning two robot platforms, improving over SFT on all four tasks. These results suggest that modality-specific specialization of foundation VLMs can provide practical visual reward signals for both simulated policy refinement and physical robot self-improvement without hand-coded task rewards.
Video generation models produce visually coherent content but struggle with tasks requiring spatial reasoning and multi-step planning. Reinforcement learning (RL) offers a path to improve generalization, but its effectiveness in video reasoning hinges on reward design -- a challenge that has received little systematic study. We investigate this problem by adapting Group Relative Policy Optimization (GRPO) to flow-based video models and training them on maze-solving and robotic navigation tasks. We first show that multimodal reward models fail catastrophically in this setting. To address this, we design verifiable reward functions grounded in objective task metrics. For structured game environments, we introduce a multi-component trajectory reward. For robotic navigation, we propose an embedding-level verifiable reward. Our experiments show that RL fine-tuning with verifiable rewards improves generalization. For example, on complex 3D mazes, our model improves exact match accuracy by 29.1% over the SFT baseline, and on trap-avoidance tasks by 51.4%. Our systematic reward analysis reveals that verifiable rewards are critical for stable training, while multimodal reward models could lead to degenerate solutions. These findings establish verifiable reward design as a key enabler for robust video reasoning. Code will be publicly available.
Ming Liu, Yunbei Zhang, Shilong Liu +2
Iowa State University, Ames, IA 50011, USA · Tulane University, New Orleans, LA 70118, USA · Princeton University, Princeton, NJ 08544, USA