Many robot manipulation videos do not indicate whether the task was successfully completed. RoboJev-27B is an open-weight vision-language model that takes a task instruction, 10 frames of a manipulation attempt, and (if available) final frames from auxiliary cameras and predicts whether the robot accomplished the task. RoboJev-27B is a variant of JEV-27B-VL, trained with a rank-32 LoRA adapter on the language model only (keeping the vision encoder frozen) using 52,598 episode records. It also uses a source-specific KL penalty that applies only to research data, anchoring predictions closer to the original model on those sources. It uses a calibrated two-token readout that outputs success/failure probabilities in a single forward pass. On 268 held-out episodes of the same robot model (SO-101), RoboJev-27B attains 95.2% balanced accuracy and 0.993 AUROC. It also generalises to episodes collected by unseen labs/robots (AUROC = 0.890 over 459 episodes) as well as to clearly labelled community-collected episodes (accuracy = 93.3% over 135 episodes), where the base model attains an AUROC = 0.900 on the unseen-lab/robot subset with the same inputs. It takes 0.26 seconds per episode (median, serial) to infer on a single H100, which is about $0.19 per 1,000 success predictions at $2.69 per GPU-hour (not including media processing and serving costs). RoboJev-27B is released together with its inference code, along with a transparent evaluation: 20,628 predictions organized into 29 model/protocol entries are made available to facilitate browsing and to enable reproducible benchmarking of success detection. We also compare against strong API-based frontier VLMs, which take in fewer input frames (4 images from the primary camera only); note that this comparison is not strictly like-for-like because RoboJev-27B is given more frames at inference.
Figures & tables
Figure 2: An existing vision-language backbone with a task-specific decision adapter. LoRA (rank 32) is applied only to the language-model component, while the vision encoder and merger remain frozen. Success targets can be binary or soft, and KL anchoring to the base model is applied on a selected subset of research sources. At inference, the merged model scores the two answer tokens with the released calibration applied; the teacher branch is used only during training and is not part of the serving graph.
Figure 3: Performance depends on the evaluation domain and input protocol. Shown are results on the held-out SO-101 runs, on the unseen lab/robot domains, and on the clearly labeled community subset. Error bars represent episode bootstrap intervals. All baselines using the API are given the 4 primary camera frames; RoboJev and the dense base receive 10 frames plus available final auxiliary views. There are 458 recorded answers in the unseen-domain plot for Sol. The same-input base achieves a higher unseen-domain AUROC, and Opus achieves higher community accuracy.
Setting
Success
Failure
Scored
SO-101
111
157
268
Unseen domains
263
196
459
Community
60
75
135
Total
434
428
862
Table 1: Evaluation composition. “Held out” has a different meaning in each setting; the SO-101 task-held-out subset contains 80 of the 268 runs.
Figure 4: Matched-input adaptation and deployment cost. Left: adapted minus base, paired at the verified identical inputs. The interval includes zero for the unseen domain. Right: Cost per 1,000 answers and response time per answer (medians). API costs are recorded charges; the H100 point uses a GPU-hour estimate. These are not controlled speedups, because the infrastructure and input budget differ.
Figure 5: Good calibration in one setting does not transfer automatically. The reliability diagram for the released model uses ten equal-width bins; the vertical error bars represent 95% Wilson intervals on the observed fraction of successes. In the histograms, we show the fraction of episodes falling into each bin, separated by outcome. The ECE and Brier score are computed over all scored episodes in a given panel. A diagonal line indicates perfect calibration. This represents a post-hoc diagnostic of the evaluation set, rather than another calibration fit.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value and scope
Starting model
autotrust/JEV-27B-VL ; its decision adapter is merged before robot adaptation.
Architecture class
Qwen3_5ForConditionalGeneration ; model_type=qwen3_5 in the pinned checkpoint. The model card’s Qwen3.8 naming is not used to infer an independently verified foundation checkpoint lineage.
Adapter
Rank 32, language component; frozen vision encoder; full-strength exported merge.
Optimization
One epoch, learning rate 5×10−5 ; binary log loss plus source-specific KL anchor of weight 1.0.
Input protocol
Ten uniformly spaced primary-camera frames and the last frame of available auxiliary cameras.
Answer tokens
false : 3721; true : 1802.
Appendix
Table 2: Inference and adaptation settings recorded for the released model.
SO-101
Unseen domains
Model / historical variant
Input
n
AUC
BA
n
AUC
BA
RoboJev-27B
D
268
0.993
95.2
459
0.890
78.2
GPT-6 Astra
C
268
0.987
93.3
459
0.862
77.8
GPT-6.1 Sol
C
268
0.989
92.8
458
0.865
79.1
Claude Opus 5.5
C
268
0.978
91.9
459
0.861
73.9
Claude Sonnet 5.5
C
268
0.980
92.5
459
0.869
74.4
Appendix
Table 3: Complete available SO-101 and unseen-domain results. BA is balanced accuracy (%); AUC is AUROC. n is the number of available binary-labelled predictions. D: dense-v1; C: clip-v1; V: whole video.
Model / historical variant
Input
n
AUROC
Accuracy
RoboJev-27B
D
135
0.984
93.3
GPT-6 Astra
C
135
0.991
92.6
GPT-6.1 Sol
C
135
0.990
92.6
Claude Opus 5.5
C
135
0.988
94.1
Claude Sonnet 5.5
C
135
0.972
92.6
Appendix
Table 4: Available results on the 135 clearly labelled community episodes. Accuracy is ordinary accuracy (%), not balanced accuracy. The other 65 community clips are marked unclear.
Model family / recorded variant
Exact model or endpoint identifier (linked)
RoboJev-27B and historical 27B/4B variants
josefchen/RoboJev-27B (release documentation)
GPT-6 Astra
openai/gpt-6-astra
GPT-6.1 Sol
openai/gpt-6.1-sol
Claude Opus 5.5
anthropic/claude-opus-5.5
Claude Sonnet 5.5
anthropic/claude-sonnet-5.5
JEV-27B-VL
autotrust/JEV-27B-VL
Appendix
Table 5: Primary model sources. Multiple recorded variants share a source.
Processed subset identifier
Fit
Validation
aliangdw_metaworld_metaworld_eval
107
11
aliangdw_metaworld_metaworld_train
97
3
jesbu1_auto_eval_rfm_auto_eval_rfm
0
800
jesbu1_failsafe_rfm_failsafe
600
40
jesbu1_fino_net_rfm_fino_net
0
229
jesbu1_molmoact_rfm_molmoact_dataset_household
300
40
Appendix
Table 6: Exact processed RBM source identifiers, linked to the verified repository. Fit counts are after local and whole-source holdouts.