cs.ROOct 24, 2026

RoboJev: An Open Vision-Language Judge for Robot Success Detection

Authors: Josef Liyanjun Chen

Organizations: Independent Researcher

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Many robot manipulation videos do not indicate whether the task was successfully completed. RoboJev-27B is an open-weight vision-language model that takes a task instruction, 10 frames of a manipulation attempt, and (if available) final frames from auxiliary cameras and predicts whether the robot accomplished the task. RoboJev-27B is a variant of JEV-27B-VL, trained with a rank-32 LoRA adapter on the language model only (keeping the vision encoder frozen) using 52,598 episode records. It also uses a source-specific KL penalty that applies only to research data, anchoring predictions closer to the original model on those sources. It uses a calibrated two-token readout that outputs success/failure probabilities in a single forward pass. On 268 held-out episodes of the same robot model (SO-101), RoboJev-27B attains 95.2% balanced accuracy and 0.993 AUROC. It also generalises to episodes collected by unseen labs/robots (AUROC = 0.890 over 459 episodes) as well as to clearly labelled community-collected episodes (accuracy = 93.3% over 135 episodes), where the base model attains an AUROC = 0.900 on the unseen-lab/robot subset with the same inputs. It takes 0.26 seconds per episode (median, serial) to infer on a single H100, which is about $0.19 per 1,000 success predictions at $2.69 per GPU-hour (not including media processing and serving costs). RoboJev-27B is released together with its inference code, along with a transparent evaluation: 20,628 predictions organized into 29 model/protocol entries are made available to facilitate browsing and to enable reproducible benchmarking of success detection. We also compare against strong API-based frontier VLMs, which take in fewer input frames (4 images from the primary camera only); note that this comparison is not strictly like-for-like because RoboJev-27B is given more frames at inference.

Figures & tables

Figure 2: An existing vision-language backbone with a task-specific decision adapter. LoRA (rank 32) is applied only to the language-model component, while the vision encoder and merger remain frozen. Success targets can be binary or soft, and KL anchoring to the base model is applied on a selected subset of research sources. At inference, the merged model scores the two answer tokens with the released calibration applied; the teacher branch is used only during training and is not part of the serving graph.
Figure 3: Performance depends on the evaluation domain and input protocol. Shown are results on the held-out SO-101 runs, on the unseen lab/robot domains, and on the clearly labeled community subset. Error bars represent episode bootstrap intervals. All baselines using the API are given the 4 primary camera frames; RoboJev and the dense base receive 10 frames plus available final auxiliary views. There are 458 recorded answers in the unseen-domain plot for Sol. The same-input base achieves a higher unseen-domain AUROC, and Opus achieves higher community accuracy.
SettingSuccessFailureScored
SO-101111157268
Unseen domains263196459
Community6075135
Total434428862
Table 1: Evaluation composition. “Held out” has a different meaning in each setting; the SO-101 task-held-out subset contains 80 of the 268 runs.
Figure 4: Matched-input adaptation and deployment cost. Left: adapted minus base, paired at the verified identical inputs. The interval includes zero for the unseen domain. Right: Cost per 1,000 answers and response time per answer (medians). API costs are recorded charges; the H100 point uses a GPU-hour estimate. These are not controlled speedups, because the infrastructure and input budget differ.
Figure 5: Good calibration in one setting does not transfer automatically. The reliability diagram for the released model uses ten equal-width bins; the vertical error bars represent 95% Wilson intervals on the observed fraction of successes. In the histograms, we show the fraction of episodes falling into each bin, separated by outcome. The ECE and Brier score are computed over all scored episodes in a given panel. A diagonal line indicates perfect calibration. This represents a post-hoc diagnostic of the evaluation set, rather than another calibration fit.
Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

SettingValue and scope
Starting modelautotrust/JEV-27B-VL ; its decision adapter is merged before robot adaptation.
Architecture classQwen3_5ForConditionalGeneration ; model_type=qwen3_5 in the pinned checkpoint. The model card’s Qwen3.8 naming is not used to infer an independently verified foundation checkpoint lineage.
AdapterRank 32, language component; frozen vision encoder; full-strength exported merge.
OptimizationOne epoch, learning rate 5×10−55\times 10^{-5} ; binary log loss plus source-specific KL anchor of weight 1.0.
Input protocolTen uniformly spaced primary-camera frames and the last frame of available auxiliary cameras.
Answer tokensfalse : 3721; true : 1802.
Appendix
Table 2: Inference and adaptation settings recorded for the released model.
  SO-101Unseen domains
Model / historical variantInputnnAUCBAnnAUCBA
RoboJev-27BD2680.99395.24590.89078.2
GPT-6 AstraC2680.98793.34590.86277.8
GPT-6.1 SolC2680.98992.84580.86579.1
Claude Opus 5.5C2680.97891.94590.86173.9
Claude Sonnet 5.5C2680.98092.54590.86974.4
Appendix
Table 3: Complete available SO-101 and unseen-domain results. BA is balanced accuracy (%); AUC is AUROC. nn is the number of available binary-labelled predictions. D: dense-v1; C: clip-v1; V: whole video.
Model / historical variantInputnnAUROCAccuracy
RoboJev-27BD1350.98493.3
GPT-6 AstraC1350.99192.6
GPT-6.1 SolC1350.99092.6
Claude Opus 5.5C1350.98894.1
Claude Sonnet 5.5C1350.97292.6
Appendix
Table 4: Available results on the 135 clearly labelled community episodes. Accuracy is ordinary accuracy (%), not balanced accuracy. The other 65 community clips are marked unclear.
Model family / recorded variantExact model or endpoint identifier (linked)
RoboJev-27B and historical 27B/4B variantsjosefchen/RoboJev-27B (release documentation)
GPT-6 Astraopenai/gpt-6-astra
GPT-6.1 Solopenai/gpt-6.1-sol
Claude Opus 5.5anthropic/claude-opus-5.5
Claude Sonnet 5.5anthropic/claude-sonnet-5.5
JEV-27B-VLautotrust/JEV-27B-VL
Appendix
Table 5: Primary model sources. Multiple recorded variants share a source.
Processed subset identifierFitValidation
aliangdw_metaworld_metaworld_eval10711
aliangdw_metaworld_metaworld_train973
jesbu1_auto_eval_rfm_auto_eval_rfm0800
jesbu1_failsafe_rfm_failsafe60040
jesbu1_fino_net_rfm_fino_net0229
jesbu1_molmoact_rfm_molmoact_dataset_household30040
Appendix
Table 6: Exact processed RBM source identifiers, linked to the verified repository. Fit counts are after local and whole-source holdouts.
More in this topic

VLMs for Robotics

Vision-language models for robotics connect visual and linguistic understanding to grounded navigation, manipulation, and robot control.

See all 664 papers

Explore similar work

CardsList
Sep 3, 2026

FailBench: How Reliable are VLMs at Judging Robot Task Success?

Vision-Language Models (VLMs) are increasingly used to evaluate robot manipulation outcomes, but existing benchmarks offer limited evidence of cross-domain generalization. We introduce FailBench, a benchmark for robot failure detection comprising 2,197 manipulation attempts across 14 public sources (12 real-world, 2 simulated). In FailBench, 75% of failures occur naturally, and six real-world sources come from non-failure-detection datasets. Evaluating 13 VLM-based detectors, we find the best model achieves only 0.77 mean balanced accuracy. Notably, models fine-tuned for failure detection consistently underperform general-purpose VLMs and their own pretrained baselines. Performance depends heavily on required visual evidence: models approach saturation when outcomes depend on observable object motion, but degrade to near-chance (<0.60 balanced accuracy) on contact-intensive assembly tasks. Error analysis reveals a systematic bias toward predicting success under ambiguous evidence, which persists even with increased reasoning effort. Finally, we show that input-level intervention--spatially localizing and cropping outcome-relevant regions--improves the top detector by 2.4 percentage points without extra training.

Zaruhi Navasardyan, Tatul Danielyan, Hrant Davtyan

Metric AI Lab

VLM EvaluationRobot Failure DetectionRobot Manipulation BenchmarksVLMs for RoboticsAnomaly Detection
AskPDF
Jun 11, 2026

Robo2Process: Benchmarking and Learning Process-Aware Understanding in Vision-Language Robotic Manipulation

Vision-language models (VLMs) used as robotic critics, reward generators, and failure detectors must understand how manipulation unfolds, not only whether it succeeds. Yet fine-grained process supervision is not readily available in a unified form, and existing evaluations do not systematically diagnose these abilities. We introduce Robo2Process, a trace-grounded framework for scalable construction of process-aware supervision from heterogeneous robotic execution data. It aligns visual observations with source-native signals and annotations, extracts local process units (LPUs), and generates structured QA supervision through task-specific sampling, deterministic label construction, and ambiguity filtering. We instantiate this framework as RoboProcessBench, a diagnostic benchmark with 12 question families spanning static monitoring and dynamic reasoning, and ProcessData, a corpus of 57,892 QA pairs across four data sources and 260 manipulation tasks, split into ProcessData-SFT and ProcessData-Eval. Evaluation of a broad suite of VLMs reveals uneven process understanding. Supervision from ProcessData-SFT improves Qwen2.5-VL-7B and InternVL-3-8B on local state, motion, progress, and primitive-aware cues. External evaluation on RoboRewardBench shows Robo2Process enables out-of-distribution process-aware reasoning abilities. Overall, Robo2Process provides a source-aware pipeline, diagnostic benchmark, and learnable substrate for process-aware understanding in robotic manipulation.

Dayu Xia, Yue Shi, Yao Mu +7

Shanghai AI Laboratory · Zhejiang University · Shanghai Jiao Tong University +2

VLM EvaluationRobotic ManipulationRobot Manipulation BenchmarksVLMs for Robotics
AskPDF
Dec 1, 2025

Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Models

Robust robotic manipulation requires reliable failure detection and recovery. Although recent Vision-Language Models (VLMs) show promise in robot failure detection, their generalization is severely limited by the scarcity and narrow coverage of failure data. To address this bottleneck, we propose an automatic framework for generating diverse robotic planning and execution failures across both simulated and real-world environments. Our approach perturbs successful manipulation trajectories to synthesize failures that reflect realistic failure distributions, and leverages VLMs to produce structured step-by-step reasoning traces. This yields GuardianFail-36k, a large-scale failure reasoning dataset built upon the RLBench simulator and the BridgeDataV2 real-robot dataset. Using GuardianFail-36k, we train Guardian, a multi-view reasoning VLM for unified planning and execution verification. Guardian achieves state-of-the-art performance on three unseen real-world benchmarks: RoboFail, RoboVQA, and our newly introduced UR5-Fail. When integrated with a state-of-the-art LLM-based manipulation policy, it consistently boosts task success rates in both simulation and real-world deployment. These results demonstrate that scaling high-quality failure reasoning data is critical for improving generalization in robotic failure detection. Code, Data, and Models are available at https://www.di.ens.fr/willow/research/guardian/.

Paul Pacaud, Ricardo Garcia, Shizhe Chen +1

Inria, École normale supérieure, CNRS, PSL Research University

Failure Detection for VLA ModelsRobot Manipulation DatasetsSynthetic Data Generation for RoboticsGeneralization in Robotic ManipulationReasoning for VLA Models
AskPDF