RGraph-Eval: Task-Grounded Scene Graphs for Evaluating Robot Video Generation

Aleksandr Lazarev; Sofia Zimina; Roman Kuznetsov; Tatiana Zemskova; Iaroslav Koshelev; Dmitry Yudin

cs.ROOct 22, 2026

RGraph-Eval: Task-Grounded Scene Graphs for Evaluating Robot Video Generation

Authors: Aleksandr Lazarev, Sofia Zimina, Roman Kuznetsov, Tatiana Zemskova, Iaroslav Koshelev, Dmitry Yudin

Organizations: MIRAI, Moscow, Russia · AXXX, Moscow, Russia · AI Foundation and Algorithm Lab, Moscow, Russia

Abstract

Generated robot videos can look convincing while depicting the wrong interaction: a different object is moved, a grasp disappears, or a placement ends outside its destination. We introduce RGraph-Eval, a reference-based framework that makes these discrepancies explicit through task-grounded temporal scene graphs. Unlike approaches that construct separate evidence representations for different evaluation dimensions, we organize task entities, interactions, and motion within a unified, interpretable scene graph representation. A foundation-model pipeline identifies manipulation roles, grounds persistent object masks, infers and verifies state transitions and action intervals, and reconstructs manipulated-object motion in three dimensions. Applying this representation to a generated video and a reference demonstration supports an interpretable RGraph-Eval score with visible-object, relation-change, and 3D path-shape components. We also introduce ShareRobot-SG, a collection of 1,000 manipulation scenes from ten ShareRobot sources annotated with the multimodal representation. Our evaluation separates graph-generation quality from agreement with human video judgments and includes comparisons with adapted SVG2 and SG-Ego, as well as a benchmark of three video generators. The framework connects robot-video assessment to the observable entities and interactions underlying a task while making the graph annotations reusable beyond evaluation. Code will be publicly available at https://github.com/SashaDance/rgraph-eval.

Figures & tables

Figure 1: RGraph-Eval overview. Foundation models transform reference and predicted videos into graph sequences. These scene graphs provide a structured summary of task-relevant entities and interactions and serve as the basis for evaluation. Visible-object, relation-change, and 3D path-shape comparisons contribute to the RGraph-Eval score.

Figure 2: Scene graph generation for a UTokyo xArm episode. Qwen3.8-27B ( Qwen Team, 2026 ) identifies entities and grounding keyframes. SAM 3 generates text-prompted tracks, boxes select the intended objects, and SAM 2.1 fills gaps. Qwen3.8-27B infers and verifies states and actions. Depth and camera estimates provide object motion, resulting in a multimodal temporal scene graph sequence.

Dataset	Tasks	SG	Gen.
UTokyo	Pick/place, stacking	92	7
DLR SARA Grid	Pick/place	40	–
DLR SARA Pour	Pouring	51	–
DLR EDAN	Picking	63	–
UCSD Pick/Place	Pick/place	151	5
VIOLA	Relocation	44	–

Table 1: Dataset composition used in the experiments. ShareRobot-SG contains 1,000 scenes, including 100 manually annotated ground-truth scenes. The video-generation evaluation uses a separate 30-scene subset.

Figure 3: Qualitative graph comparison on a UTokyo xArm plate-stacking episode. SVG2 and SG-Ego include broad geometric predicates alongside interaction relations, while ours primarily represents task roles and manipulation-relevant relations. In our graph, the robot holds the white plate in the intermediate frame, and the white plate is supported by the red plate in the final frame. Boxes ground entities and colors denote task roles.

	Objects			Relations
Method	P	R	F1	P	R	F1	Latency
Human evaluation
Ours	90.77	88.59	89.67	94.04	73.74	82.66	$94\pm 24$
SG-Ego	91.37	92.19	91.78	55.97	85.97	67.80	$34\pm 8$
SVG2	71.67	51.65	60.03	75.00	19.42	30.86	$362\pm 76$
Dataset-stratified PPI

Table 2: Graph-generation quality (%) on ShareRobot-SG and estimated latency per video (mean

\pm

SD, seconds). Human micro-averages and dataset-stratified PPI estimates are distinct evaluations. P and R denote precision and recall. Bold marks the highest quality score within each evaluation group.

Evaluator	Task completion	Object consistency	Visual quality	Mean
RGraph-Eval (ours)	0.9800	0.8633	0.9613	0.9722
RBench: overall ( Deng et al., 2026 )	0.9800	0.8098	0.9298	0.9447
Ref4D: event ( Wei et al., 2026 )	0.7028	0.9757	0.8912	0.8710
WorldModelBench ( Li et al., 2025 )	0.9078	0.5373	0.7293	0.7579
RoboReward ( Lee et al., 2026 )	0.8737	0.4725	0.6758	0.7067
VideoScore2: overall ( He et al., 2025 )	0.5879	0.0577	0.3029	0.3434

Table 3: Pearson correlation across three generator means on the 30-scene generation subset (90 generated videos, one assessor). Rows are sorted by Mean. Bold marks the highest value in each column.

Figure 4: Reference and generated Bridge bowl-placement videos from Cosmos, WoW, and LingBot at the start, midpoint, and end. Colored masks identify objects; adjacent graphs show relations (solid arrows) and actions (dashed arrows). The goal is to place the bowl on the napkin. Cosmos and LingBot place the bowl on the napkin, whereas WoW moves the napkin instead, leaving the bowl on the table.

Generator	$O$	$R$	$T$	RGraph-Eval
LingBot-VA ( Li et al., 2026 )	0.9928	0.9448	0.7988	0.9121
Cosmos 3 Nano ( NVIDIA, 2026 )	0.9983	0.9251	0.7608	0.8947
WoW-1 ( Chi et al., 2025 )	0.9685	0.8798	0.7880	0.8788

Table 4: Generator means on the 30-scene generation subset: all components and the RGraph-Eval score lie in

[0,1]

. Rows are sorted by RGraph-Eval. Bold marks the highest value in each column.

Variant	P	R	F1
Text-prompted SAM3 + repair (full)	89.65	89.51	89.58
Text-prompted SAM3	89.41	72.86	80.29
Box-prompted SAM3	82.25	76.70	79.38
Box-prompted SAM2.1	90.54	84.02	87.16

Table 5: Role-object tracking ablation on 100 annotated ShareRobot-SG videos using the same ten frames per video and fixed Qwen proposals. P, R, and F1 are percentages; bold marks the highest value in each column. Repair denotes SAM2.1 missing-frame repair.

Variant	Task completion	Mean
Visible objects ( $O$ )	0.7756	0.5726
Relation changes ( $R$ )	0.9786	0.8845
3D path shape ( $T$ )	0.2591	0.5141
$(O+T)/2$	0.8889	0.9805
$(O+R)/2$	0.9362	0.8038
$(T+R)/2$	0.9523	0.9812

Table 6: Score ablations: Pearson

r

across three generator means on the 30-scene generation subset.

O

: visible-object F1;

R

: relation changes;

T

: 3D path shape. Bold marks the highest value per column.

Video Generation Evaluation

Video generation evaluation measures visual quality, prompt adherence, temporal consistency, physical plausibility, and task-specific behavior in synthesized videos.

See all 192 papers

GenVid2Robot: From Video Generation to Robot Manipulation via Rigid-Geometric Consistency
Jul 10, 2026Haohui Huang, Xi Yuan, Panpan Liao +4Robot Motion Generation Robotic Manipulation
School of Automation, Guangdong University of Technology, Guangzhou, China · University of Liverpool, Liverpool, U.K. · Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China +1
V2-STRep: VLM-Grounded Structured Task Representations for Reusable Robot Skills Acquired from Generated Videos
Sep 17, 2026Yexin Hu, Dongheui LeeZero-Shot Robotic Manipulation Generalization in Robotic Manipulation
Autonomous Systems Lab, Technische Universität Wien (TU Wien), Austria. · Institute of Robotics and Mechatronics, German Aerospace Center (DLR), Germany.
RoboGaze: Evaluating Robot World Models via Structured Vision-Language Analysis
Jun 22, 2026Minh-Loi Nguyen, Nghiem T. Diep, Hung K. Nguyen +13VLM Evaluation Robot Manipulation Benchmarks
Center for AI Research, VinUniversity · VinRobotics · DFKI +3

Jul 10, 2026J/K move · Enter open · S save

GenVid2Robot: From Video Generation to Robot Manipulation via Rigid-Geometric Consistency

Haohui Huang, Xi Yuan, Panpan Liao, Tao Teng+3

School of Automation, Guangdong University of Technology, Guangzhou, China · University of Liverpool, Liverpool, U.K. · Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China+1

Robot Motion Generation Robotic Manipulation

Open paper AskPDF arXiv

Generated videos provide useful visual motion priors for robot manipulation, but their visual plausibility does not imply physical executability. A generated video usually lacks metric geometry, grasp grounding, robot kinematic feasibility, and execution-time feedback, which makes direct trajectory replay unreliable in real-world manipulation. This paper presents GenVid2Robot, a rigid-geometric consistency framework that converts generated video motion into executable real-robot manipulation trajectories. Given an initial RGB-D observation and a task instruction, GenVid2Robot samples task-relevant semantic anchors from the real first frame, tracks these anchors through generated video candidates, and verifies whether the resulting 2D motion can be explained by first-frame RGB-D anchors under a sparse relative

SE(3)

model. In this way, generated videos are treated as uncertain visual motion hypotheses rather than direct robot demonstrations. Only geometrically consistent motion is transferred to the robot. The accepted relative motion is then applied to the real grasp-time TCP pose selected by mask-constrained grasping, producing a grasp-conditioned execution trajectory that is consistent with both the visual motion prior and the physical grasp configuration. To reduce execution mismatch caused by RGB-D noise, calibration residuals, and small contact-induced displacement, a bounded depth-compensation module corrects local depth-direction errors without assuming full online replanning. Real-robot experiments demonstrate that GenVid2Robot improves the reliability of generated-video-guided manipulation by grounding visual motion priors with sparse metric geometry, grasp constraints, robot feasibility checking, and bounded execution feedback.

Abstract

Video Generation Evaluation

Explore similar work

GenVid2Robot: From Video Generation to Robot Manipulation via Rigid-Geometric Consistency

V2-STRep: VLM-Grounded Structured Task Representations for Reusable Robot Skills Acquired from Generated Videos

RoboGaze: Evaluating Robot World Models via Structured Vision-Language Analysis