Vision-language models (VLMs) have shown impressive capabilities across diverse tasks, motivating efforts to leverage these models to supervise robot learning. However, when used as evaluators in reinforcement learning (RL), today's strongest models often fail under partial observability and distribution shift, enabling policies to exploit perceptual errors rather than solve the task. We introduce SOLE-R1 (Self-Observing LEarner), a video-language reasoning model explicitly designed to serve as the sole reward signal for online RL. Given only raw video observations and a natural-language goal, SOLE-R1 performs per-timestep spatiotemporal chain-of-thought (CoT) reasoning and produces dense estimates of task progress that can be used directly as rewards. To train SOLE-R1, we develop a large-scale video trajectory and reasoning synthesis pipeline that generates temporally grounded CoT traces aligned with continuous progress supervision. This data is combined with foundational spatial and multi-frame temporal reasoning, and used to train the model with a hybrid framework that couples supervised fine-tuning with RL from verifiable rewards. Across four different simulation environments and a real-robot setting, SOLE-R1 enables zero-shot online RL from random initialization: robots learn previously unseen manipulation tasks without ground-truth rewards, success indicators, demonstrations, or task-specific tuning. SOLE-R1 succeeds on 24 unseen tasks and substantially outperforms strong vision-language rewarders, including Robometer, RoboReward, ReWiND, GPT-5, and Gemini-3-Pro, while exhibiting markedly greater robustness to reward hacking. We release all models, data, code, and demos at the anonymous page: https://philip-mit.github.io/sole-r1/
Figures & tables
Figure 1: SOLE-R1 is a video-language reasoning model designed to guide online RL with per-timestep chain-of-thought reasoning and progress prediction. In large-scale experiments across 40 tasks, SOLE-R1 outperforms strong baseline models with zero-shot online RL.
Figure 2: SOLE-R1 training data mixture. The dataset combines foundational spatial reasoning, multi-frame temporal reasoning, and our synthesized video trajectories with chain-of-thought explanations and dense progress supervision, jointly enabling reasoning over space and time for progress prediction.
Figure 3: Zero-shot success rate of online RL across 40 tasks. We plot the mean and standard error across three random seeds (real-world experiments use a single seed, shown as a single value). In all experiments, the robot begins with a random policy and learns entirely through interaction with the task, guided only by the predicted rewards.
Figure 4: Perceived vs true success in zero-shot RL. Perceived success is the average max progress predicted (RoboReward is excluded as it does not provide dense rewards). True success is the average max ground-truth reward achieved.
Figure 5
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6
Figure 7: Extended reasoning examples across all models.
Figure 8: Zero-shot RL success vs correlation between predicted and ground-truth progress
Primary qualitative error mode
SOLE-R1
GPT-5
Gemini-3-Pro
Gemini Robotics-ER
Temporal under-detection of contact / state change
34%
14%
17%
19%
Ambiguous object state under partial observability
29%
11%
13%
15%
Relies on goal-consistent appearance cues
15%
18%
20%
21%
Perceptual hallucination of success
9%
42%
38%
35%
Other / unclassified
13%
15%
12%
10%
Appendix
Table 1: Quantitative summary of failure modes from qualitative review of real-robot and simulation rollouts.
Figure 9: SFT-only vs SFT+RLVR model in zero-shot success rate of online RL
DrQv2
Meta-World
RoboSuite, LIBERO, ManiSkill
Real-world
Buffer size
200000
200000
200000
Batch size
256
256
256
Critic to actor steps
4
4
4
Initial random steps
1000
2500
100
Start learning at
1000
2500
100
Critic ensemble size
10
10
10
Appendix
Table 2: Reinforcement learning agent details
Task Name
Natural Language Task Specification
Real-World Tasks
Pick can
Pick up the can
Touch strawberry in clutter
Touch the strawberry on the table
Push can to cube
Push the can near the cube
Close drawer
Close the drawer
Open drawer
Open the drawer
Appendix
Table 3: Task groups and their natural language specifications
Figure 10: Zero-shot success rate of online RL on 10 simple Meta-World tasks across varying prompting strategies.
Figure 11: Left: workcell with two Franka FR3 arms. Right: example shoulder and wrist views from the right robot.
Non-expert trajectory levels
Percentage
Pick-only
1: Fail to approach {obj}
25%
2: Approach {obj}
25%
3: Contact {obj}
25%
4: Pick up {obj}
25%
Pick-and-place
1: Fail to approach {obj}
14%
2: Approach {obj}
14%
Appendix
Table 4: Levels for non-expert trajectories and video dataset counts. For each level, we assume that all task components corresponding to the preceding levels have already been successfully completed.
Data source ID
Total Dataset Count
Video-based CoT Reasoning and Progress Prediction
OXE berkeley_autolab_ur5
4874
OXE berkeley_fanuc_manipulation
2689
OXE bridge
4602
OXE bridge_set2
32596
OXE bridge_set3
51939
Appendix
Table 5: Dataset mixture
SpatialBench
SSRBench
CV-Bench
Method
Size
General
Spatial
Qwen2.5-VL
7B
64.7
69.4
53.6
73.0
Qwen3-VL
8B
65.3
73.1
57.2
73.7
SSR
7B
67.0
82.1
76.1
73.3
SOLE-R1 (Ours)
8B
69.4
85.6
79.2
74.4
Appendix
Table 6: Performance improvement of SOLE-R1 compared to SSR 7B model and the backbone model on SpatialBench, SSRBench, and CV-Bench.
Table 7: Value-Order-Correlation analysis (as done in on [ 34 ] ) comparing SOLE-R1 to GVL on 1,000 videos from OpenX Embodiment datasets.
Figure 12: Example of a successful (top) and failed (bottom) manipulation trajectory generated by SmolVLA in Libero. We predict and plot (red) SOLE-R1 task progress predictions based on the prompt “put the bowl on the cabinet”.
Figure 13: Example of a successful (top) and failed (bottom) manipulation trajectory generated by SmolVLA in Libero. We predict and plot (red) SOLE-R1 task progress predictions based on the prompt “put the cream cheese in the bowl”.