Execution Logs Sway VLM Judges of Robot Progress

Weiyan Zhao; Sunny Yang

cs.LGOct 25, 2026

Execution Logs Sway VLM Judges of Robot Progress

Authors: Weiyan Zhao, Sunny Yang

Organizations: NLPQuant AI Inc. · University of Illinois Urbana-Champaign

Abstract

Vision-language models now grade robot videos, and text placed beside the frames can change their verdicts. On clips where the robot is losing progress, we add one log line saying the robot is making progress. Across Gemini, Qwen and GPT judges, this flips correct verdicts to progress 10 to 35 percentage points more often than a neutral line does. A one-line warning that the log may be wrong removes most of this effect. Selected fake "task complete" claims flip half of Qwen's correct verdicts, against a tenth for unrelated log lines. They have no detectable effect on Gemini.

Figures & tables

Figure 1: One fake “task complete” log line flips Qwen’s correct verdict. Task: hand the pen from the left to the right arm, then place it in the blue plate; the right arm never takes the pen. On the frames alone, Qwen answers Falling (losing progress): “the right arm remains open and does not grasp it”. With the log “Successful completion confirmed: the pen was … transferred to the right arm, and placed in the blue plate”, it answers Rising (making progress): “the execution log confirms the successful completion of the task”. Example chosen for visual clarity.

Figure 2: A fake progress line flips correct verdicts 10 to 35 points more often than a neutral line in all three judges; selected fake “task complete” claims fool Qwen, with no detectable effect on Gemini. Each dot is the rate at which a correct Falling verdict turns to Rising, minus the same rate for the control: a neutral sentence (left) or an unrelated log line (right). Bars are 95% CIs. Only intervals the judge gets right on the frames, both forward and reversed. Numbers in Table 1 .

One fixed progress log, reversal-consistent Falling intervals
	Gemini 3.8 Flash	Qwen3.7 Flash	GPT-6.1 Sol
Frames-only accuracy, Falling / Rising	0.89 / 0.95	0.78 / 0.87	0.85 / 0.95
n; turned to Rising by log / neutral / repeat	95; 16 / 7 / 6	74; 17 / 3 / 5	91; 32 / 0 / 5
log $-$ neutral [95% CI]	$+.095$ [.033, .163]	$+.189$ [.110, .284]	$+.352$ [.242, .464]
same, 56 intervals shared by all three	$+.018$ [.000, .055]	$+.161$ [.073, .259]	$+.214$ [.109, .333]
Regression log, reversal-consistent Rising intervals

Table 1: Effects of execution-log text on reversal-consistent intervals (the judge is right on the frames, forward and reversed). Counts, and rate differences with episode-level bootstrap 95% CIs (2,000 resamples). Repeat: the frames-only call run again. The regression log (“the robot is losing ground”) counts turns to Falling. A completion claim counts only if a second call with it also answers Rising. The completion-claim attack used all Falling intervals for Qwen and a random 80 for Gemini.

	Accuracy with contradicting log				Gain from agreeing log
	Gemini		Qwen		Gemini	Qwen
Prompt	Falling	Rising	Falling	Rising	(on Falling)
original	.73	.83	.58	.44	$+.117$	$+.186$
warning	.91	.94	.81	.82	$-.017$	$+.125$
original	.72	.80	.58	.45	$+.125$	$+.208$
observe	.77	.78	.51	.61	$+.075$	$+.325$

Table 2: The warning gives the best accuracy with a contradicting log, but the judge also gains less from an agreeing one. Gain: accuracy with an agreeing log minus accuracy with a neutral sentence, under the same prompt. The two blocks are separate runs of 240 intervals with slightly different baselines. Verify lowers Qwen’s neutral-sentence accuracy on Falling (0.78 to 0.61), which inflates its gain.

VLM Robustness

Vision-language model robustness studies whether models remain reliable under visual, textual, and distribution shifts, adversarial inputs, and safety challenges.

See all 454 papers

Robot Critics that Sweat the Small Stuff
Jun 19, 2026Sruthi Sudhakar, Junbang Liang, Sreehari Rammohan +3Robot Task Progress Estimation Robot Failure Detection
Columbia University · Toyota Research Institute
RoboGaze: Evaluating Robot World Models via Structured Vision-Language Analysis
Jun 22, 2026Minh-Loi Nguyen, Nghiem T. Diep, Hung K. Nguyen +13VLM Evaluation Robot Manipulation Benchmarks
Center for AI Research, VinUniversity · VinRobotics · DFKI +3
When the Judge Acts: Auditing VLM-Guided Image Selection on Culturally Situated Prompts
Oct 1, 2026Huichan SeoVLM Evaluation Position Bias in VLMs
Independent Researcher

Jun 19, 2026J/K move · Enter open · S save

Robot Critics that Sweat the Small Stuff

Sruthi Sudhakar, Junbang Liang, Sreehari Rammohan, Pavel Tokmakov+2

Columbia University · Toyota Research Institute

Robot Task Progress Estimation Robot Failure Detection VLM Adaptation Robotic Policy Evaluation Robotic Manipulation VLMs for Robotics

Open paper AskPDF arXiv

Large vision-language models contain several priors about the world and object interactions, making them useful critics during inference to steer robot policies towards success. However, closed-loop robot manipulation requires judging small visual differences between success and failure, which remains a challenge for current VLMs. We introduce a method to fine-tune critics by constructing pairwise progress supervision using success and failure rollouts obtained from a policy. Our fine-tuned critic excels at fine-grained progress reasoning and subtle failure detection, outperforming prior progress reasoning baselines. Additionally, we use an action-conditioned video model to predict the visual effect of several candidate actions sampled from a policy, and show that our critic can correctly identify successful candidates to execute, improving the average policy success rate by 11% across real-world tasks and 5.9% across simulation tasks.

Abstract

VLM Robustness

Explore similar work

Robot Critics that Sweat the Small Stuff

RoboGaze: Evaluating Robot World Models via Structured Vision-Language Analysis

When the Judge Acts: Auditing VLM-Guided Image Selection on Culturally Situated Prompts