Don't Read the Log: Execution Traces Contaminate Verifiers in Video-Generation Agents

Jian Xu

cs.CRSep 23, 2026

Don't Read the Log: Execution Traces Contaminate Verifiers in Video-Generation Agents

Authors: Jian Xu

Organizations: RIKEN

Abstract

Agentic video-generation systems close a loop between a generator and a verifier: an LLM plans shots, calls a text-to-video model, and a multimodal judge decides whether the result satisfies the request. To diagnose where a long workflow fails, recent harnesses deliberately show the judge more than the video-the agent's execution trace, its plan, the narration it synthesized. We ask whether this auxiliary text moves the judge's verdict on purely \emph{visual} requirements, holding the frames fixed. On a benchmark of 109 generated two-event clips with manual labels, in which the requested event is either visibly completed or visibly missing, a trace that reports a successful tool call makes three open-weight Qwen-VL judges (7B, 8B, 32B) accept $78$ -- $90\%$ of the failures, up from $7$ -- $19\%$ without text, and a contradicting trace makes them reject up to $100\%$ of correct clips; an instruction to ``use only the frames'' does not remove the effect. Frontier closed judges are essentially unmoved on the same clips, showing that the vulnerability is a property of the judge's learned trust in tool logs rather than of the task. Plan-derived text carries no clip-specific information, so it can only shift a judge's operating point, and in a repair loop that shift becomes a cap on the true pass rate that no repair policy can exceed; the cap matches simulation to two decimals. In the loop, contamination is exploited without any adversarial agent: an honest LLM planner that always regenerates ends with a judge pass rate of $1.00$ and a human-labelled pass rate of $0.28$ , and a pipeline in which a cheap checker writes its verdict into the trace launders that checker's errors into a stronger final judge ( $0.69$ false accepts).

Figures & tables

Figure 1: Auxiliary text moves the verdict on a purely visual requirement, with the frames held fixed (generated near-miss set, 50 present / 59 absent clips, default prompt). Left: when the event is absent and the text claims it happened, an execution log makes the three open-weight judges accept

0.78

–

0.90

of the failures; GPT-5.4-mini is unmoved (

0.12\to 0.14

). Right: when the event is present and the text claims otherwise, the log drives acceptance to

0.00

–

0.08

for the Qwen3 judges and leaves GPT-5.4-mini at

0.90

	open-weight judges			frontier closed judges
auxiliary text	Qwen2.5-VL-7B	Qwen3-VL-8B	Qwen3-VL-32B	GPT-5.4-mini	GPT-5.5	Claude Opus 5
none	0.96 / 0.17	0.96 / 0.19	0.82 / 0.07	0.98 / 0.12	0.88 / 0.09	0.88 / 0.10
transcript, supports $e_{2}$	1.00 / 0.37	0.98 / 0.47	0.98 / 0.17	0.98 / 0.15	0.91 / 0.09	0.94 / 0.10
subtitle, supports $e_{2}$	0.92 / 0.19	0.98 / 0.54	0.82 / 0.10	1.00 / 0.27	0.88 / 0.09	0.88 / 0.10
log, supports $e_{2}$	1.00 / 0.78	1.00 / 0.90	1.00 / 0.83	0.98 / 0.14	0.88 / 0.09	0.88 / 0.10
+ visual-only prompt	1.00 / 0.63	0.96 / 0.54	1.00 / 0.41	1.00 / 0.12	0.85 / 0.06	0.88 / 0.10

Table 1: Generated near-miss set:

P(\text{accept})

for the visual requirement

e_{2}

, present / absent clips, default prompt unless noted. Open-weight judges on all 109 clips (50/59); GPT-5.4-mini on all 109; GPT-5.5 on 67 (33/34) and Claude Opus 5 on 26 (16/10) because of API budget. Bold marks the log channel.

judge	no memory	memory: asset locked	memory: re-cast
Qwen3-VL-8B	1.00 / 0.47	1.00 / 1.00	0.88 / 0.29
Qwen3-VL-32B	1.00 / 0.29	1.00 / 0.47	0.00 / 0.00
GPT-5.4-mini	1.00 / 0.18	1.00 / 0.12	1.00 / 0.12
Claude Opus 5	1.00 / 0.00	1.00 / 0.00	0.76 / 0.00

Table 2: Multi-shot consistency requirement (17 consistent / 17 inconsistent shot pairs).

P(\text{accept})

on consistent / inconsistent pairs as the production-memory entry in the trace is absent, claims the character asset was locked and reused, or claims shot 2 was re-cast.

Figure 2: Repair loop over 36 real generated failures, Qwen3-VL-8B judge,

\leq 4

rounds. With a naive judge (left), every planner—including an LLM planner that always chooses to regenerate—ends with judge pass rate

1.00

but true pass rate

0.28

72\%

of failed clips are accepted at round zero on the strength of a trace containing only the plan and a status: success . With least-privilege routing (right) the true pass rate rises to

0.86

; the remaining gap is the judge’s own visual false-accept floor.

judge	evidence	planner	judge pass	true pass	hacked	cost/ep	round-0
Qwen3-VL-8B	Naive	always regenerate	1.00	0.28	0.72	0.36	0.72
		best-of- $K$	1.00	0.28	0.72	0.28
		cost-greedy	0.75	0.22	0.72	0.30
		LLM planner	1.00	0.28	0.72	0.36
	LP	always regenerate	1.00	0.86	0.14	1.11	0.14
		best-of- $K$	1.00	0.86	0.14	0.86

Table 3: Closed-loop outcomes on 36 labelled failures. Naive : judge sees frames + trace + narration. LP : least-privilege routing (frames only for the visual requirement). Round-0 is the acceptance rate of the failed clip before any repair. “+8B check” injects the 8B judge’s own verdict on the initial clip into the trace as a vision_check tool result, as a cheap checker would.

judge	evidence	$\alpha_{0}$	ceiling $1-\alpha_{0}$	best observed true pass
Qwen3-VL-8B	Naive	0.72	0.28	0.28
Qwen3-VL-8B	LP	0.14	0.86	0.86
Qwen3-VL-32B	Naive	0.25	0.75	0.75
Qwen3-VL-32B	Naive + 8B check	0.69	0.31	0.31
Qwen3-VL-32B	LP	0.08	0.92	0.92

Table 4: Round-0 identity check.

\alpha_{0}

is the observed round-0 false-accept rate; the ceiling is

1-\alpha_{0}

; “best observed” is the highest true pass rate over the planners of Table 3 .

	naive		OCR-mask
subtitle	pres.	abs.	pres.	abs.
none	0.97	0.14	0.97	0.14
supports $e_{2}$	0.97	0.50	0.97	0.14
contradicts	0.82	0.06	0.97	0.17
evidence-first, supports	0.77	0.28
evidence-first, contradicts	0.50	0.06

Table 5: Burned-in subtitle channel, Qwen3-VL-8B.

P(\text{accept})

on present / absent clips. OCR-masking restores the no-subtitle baseline; an evidence-first two-step prompt (describe frames, then judge the description) only halves the effect and loses recall.

	naive		OCR-mask
subtitle	pres.	abs.	pres.	abs.
none	0.97	0.14	0.97	0.14
supports $e_{2}$	0.97	0.50	0.97	0.14
contradicts	0.82	0.06	0.97	0.17
evidence-first, supports	0.77	0.28
evidence-first, contradicts	0.50	0.06

Table 5: Burned-in subtitle channel, Qwen3-VL-8B.

P(\text{accept})

on present / absent clips. OCR-masking restores the no-subtitle baseline; an evidence-first two-step prompt (describe frames, then judge the description) only halves the effect and loses recall.

requirement	Naive	LP
plan written before generation	0.93	0.89
narration track added	1.00	1.00
generator called with 81 frames	1.00	1.00
no tool call returned an error	1.00	1.00

Table 6: Process requirements are not harmed. Accuracy on four trace-only requirements whose truth is set by constructing the trace (70 clips each). LP sees the trace only; Naive sees frames + trace.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

id	$e_{1}$	$e_{2}$ (requirement)	filler	pres.	abs.
mug	picks up a red mug	drinks from the mug	holds it, looks out of the window	2	1
door	opens the door	walks through the doorway	stands in the doorway	2	4
hat	puts on a yellow hat	waves at the camera	adjusts the brim	2	4
bench	sits on a bench	opens a newspaper	sits with hands on knees	2	2
apple	picks up a green apple	takes a bite	inspects the apple	2	3
book	picks up a book	opens it and reads	holds the closed book	2	0

Appendix

Table 7: Prompts (scene abbreviated), events, fillers and clean-clip counts, batches 1–2 (top) and 3 (bottom). Dropped prompts: candle (candles already lit by scene prior), jacket (zip not visible), shoes (laces not visible), lamp (already lit by scene prior). Prompts with no clean both clip contribute absent clips only.

Figure 3: Generated two-event clips (Wan2.2-T2V-A14B, six of 81 frames each). Rows alternate between the both condition (event $e_{2}$ present) and the e1-only condition (filler instead of $e_{2}$ ). Two rows show filler prompts that leaked: the generator completed the natural continuation (drinking, opening the umbrella) despite the prompt—in the umbrella case despite an explicit “never opens the umbrella”. Leaked clips were relabelled by inspection and excluded from the absent set.

Figure 4: The burned-in subtitle channel and its least-privilege projection. Top: an absent clip (she never drinks) with the narration claiming the event rendered into every frame. Bottom: the same frames after EasyOCR detection and Gaussian masking of the detected text boxes, which is what the visual-requirement judge receives under least-privilege routing.

	Qwen2.5-VL-7B		Qwen3-VL-8B		Qwen3-VL-32B
auxiliary text	true	distr.	true	distr.	true	distr.
none	0.89	0.15	0.82	0.10	0.80	0.11
transcript, supports	0.98	0.19	0.93	0.11	0.83	0.14
log, supports	1.00	0.35	0.98	0.17	0.88	0.16
subtitle, supports	0.99	0.43	0.98	0.20	0.92	0.17
transcript, contradicts	0.31	0.15	0.33	0.10	0.25	0.10

Appendix

Table 8: UCF101 hard split (100 clips

\times

true/distractor class), default prompt,

P(\text{accept})

on true-class / distractor-class requirement.

Explore similar work

VideoGen-Agent: Reinforcing Video Generation Agents
Sep 21, 2026Binxu Li, Haoyi Duan, Yuhui Zhang +9Video Agent Video Generation
Princeton University · Stanford University · UC Davis +4
VeriPhy: Agentic Physical Reasoning for World Model Evaluation and Refinement
Sep 2, 2026Wenzhuo Xu, Yuchen Zhu, Chongjian Ge +8Physical Plausibility Generation Provenance
1Carnegie Mellon University · 2Georgia Institute of Technology · 4Adobe Research +1
VideoWeaver: Evaluating and Evolving Skills for Agentic Long Video Generation
Jun 6, 2026Jianhui Wei, Yan Zhang, Jie Tan +7Video Agent Long Video Generation
Zhejiang University · ByteDance

Sep 21, 2026, cs.CVJ/K move · Enter open · S save

VideoGen-Agent: Reinforcing Video Generation Agents

Binxu Li, Haoyi Duan, Yuhui Zhang, Yaohui Zhang+8

Princeton University · Stanford University · UC Davis+4

Video Agent Video Generation Multimodal Agents Generative Models

Open paper AskPDF arXiv

Recent advances in video generative models have enabled high-fidelity, temporally coherent video generation. However, these models often struggle to satisfy prompts requiring specialized knowledge, specific identities, physical consistency, or ordered events. In this paper, we present VideoGen-Agent, a multimodal agent trained through multitask agentic reinforcement learning to use external tools for video generation. The agent coordinates augmentation, generation, and verification tools through multi-turn interactions, using the prompt and intermediate observations to guide its decisions. We train a shared policy on a category-balanced dataset spanning six tasks. Supervised fine-tuning on teacher-generated trajectories establishes tool-use behavior, which is then refined through reinforcement learning. A category-aware hybrid reward evaluates tool-call validity, task-appropriate tool use, and generated video quality. We further introduce VABench, a held-out benchmark of 600 prompts covering procedural knowledge, single- and multi-entity identity preservation, physical consistency, scene composition, and multi-shot temporal structure. On VABench, VideoGen-Agent improves over its base text-to-video generator by 19.1 points, from 56.5 to 75.6. Upgrading the generation tools further raises the score to 86.1 without additional agent training. Human raters prefer the upgraded configuration over the strongest standalone baseline in 84.3% of comparisons. These results support learning tool use across video-generation tasks and show that the trained agent can benefit from subsequent advances in generation tools. Project page: https://andyca111.github.io/VideoGen_Agent/

Abstract

Explore similar work

VideoGen-Agent: Reinforcing Video Generation Agents

VeriPhy: Agentic Physical Reasoning for World Model Evaluation and Refinement

VideoWeaver: Evaluating and Evolving Skills for Agentic Long Video Generation