Don't Read the Log: Execution Traces Contaminate Verifiers in Video-Generation Agents

Jian Xu

cs.CRSep 23, 2026

Don't Read the Log: Execution Traces Contaminate Verifiers in Video-Generation Agents

Authors: Jian Xu

Organizations: RIKEN

Abstract

Agentic video-generation systems close a loop between a generator and a verifier: an LLM plans shots, calls a text-to-video model, and a multimodal judge decides whether the result satisfies the request. To diagnose where a long workflow fails, recent harnesses deliberately show the judge more than the video-the agent's execution trace, its plan, the narration it synthesized. We ask whether this auxiliary text moves the judge's verdict on purely \emph{visual} requirements, holding the frames fixed. On a benchmark of 109 generated two-event clips with manual labels, in which the requested event is either visibly completed or visibly missing, a trace that reports a successful tool call makes three open-weight Qwen-VL judges (7B, 8B, 32B) accept $78$ -- $90\%$ of the failures, up from $7$ -- $19\%$ without text, and a contradicting trace makes them reject up to $100\%$ of correct clips; an instruction to ``use only the frames'' does not remove the effect. Frontier closed judges are essentially unmoved on the same clips, showing that the vulnerability is a property of the judge's learned trust in tool logs rather than of the task. Plan-derived text carries no clip-specific information, so it can only shift a judge's operating point, and in a repair loop that shift becomes a cap on the true pass rate that no repair policy can exceed; the cap matches simulation to two decimals. In the loop, contamination is exploited without any adversarial agent: an honest LLM planner that always regenerates ends with a judge pass rate of $1.00$ and a human-labelled pass rate of $0.28$ , and a pipeline in which a cheap checker writes its verdict into the trace launders that checker's errors into a stronger final judge ( $0.69$ false accepts).

Figures & tables

Figure 1: Auxiliary text moves the verdict on a purely visual requirement, with the frames held fixed (generated near-miss set, 50 present / 59 absent clips, default prompt). Left: when the event is absent and the text claims it happened, an execution log makes the three open-weight judges accept

0.78

–

0.90

of the failures; GPT-5.4-mini is unmoved (

0.12\to 0.14

). Right: when the event is present and the text claims otherwise, the log drives acceptance to

0.00

–

0.08

for the Qwen3 judges and leaves GPT-5.4-mini at

0.90

	open-weight judges			frontier closed judges
auxiliary text	Qwen2.5-VL-7B	Qwen3-VL-8B	Qwen3-VL-32B	GPT-5.4-mini	GPT-5.5	Claude Opus 5
none	0.96 / 0.17	0.96 / 0.19	0.82 / 0.07	0.98 / 0.12	0.88 / 0.09	0.88 / 0.10
transcript, supports $e_{2}$	1.00 / 0.37	0.98 / 0.47	0.98 / 0.17	0.98 / 0.15	0.91 / 0.09	0.94 / 0.10
subtitle, supports $e_{2}$	0.92 / 0.19	0.98 / 0.54	0.82 / 0.10	1.00 / 0.27	0.88 / 0.09	0.88 / 0.10
log, supports $e_{2}$	1.00 / 0.78	1.00 / 0.90	1.00 / 0.83	0.98 / 0.14	0.88 / 0.09	0.88 / 0.10
+ visual-only prompt	1.00 / 0.63	0.96 / 0.54	1.00 / 0.41	1.00 / 0.12	0.85 / 0.06	0.88 / 0.10

Table 1: Generated near-miss set:

P(\text{accept})

for the visual requirement

e_{2}

, present / absent clips, default prompt unless noted. Open-weight judges on all 109 clips (50/59); GPT-5.4-mini on all 109; GPT-5.5 on 67 (33/34) and Claude Opus 5 on 26 (16/10) because of API budget. Bold marks the log channel.

judge	no memory	memory: asset locked	memory: re-cast
Qwen3-VL-8B	1.00 / 0.47	1.00 / 1.00	0.88 / 0.29
Qwen3-VL-32B	1.00 / 0.29	1.00 / 0.47	0.00 / 0.00
GPT-5.4-mini	1.00 / 0.18	1.00 / 0.12	1.00 / 0.12
Claude Opus 5	1.00 / 0.00	1.00 / 0.00	0.76 / 0.00

Table 2: Multi-shot consistency requirement (17 consistent / 17 inconsistent shot pairs).

P(\text{accept})

on consistent / inconsistent pairs as the production-memory entry in the trace is absent, claims the character asset was locked and reused, or claims shot 2 was re-cast.

Figure 2: Repair loop over 36 real generated failures, Qwen3-VL-8B judge,

\leq 4

rounds. With a naive judge (left), every planner—including an LLM planner that always chooses to regenerate—ends with judge pass rate

1.00

but true pass rate

0.28

72\%

of failed clips are accepted at round zero on the strength of a trace containing only the plan and a status: success . With least-privilege routing (right) the true pass rate rises to

0.86

; the remaining gap is the judge’s own visual false-accept floor.

judge	evidence	planner	judge pass	true pass	hacked	cost/ep	round-0
Qwen3-VL-8B	Naive	always regenerate	1.00	0.28	0.72	0.36	0.72
		best-of- $K$	1.00	0.28	0.72	0.28
		cost-greedy	0.75	0.22	0.72	0.30
		LLM planner	1.00	0.28	0.72	0.36
	LP	always regenerate	1.00	0.86	0.14	1.11	0.14
		best-of- $K$	1.00	0.86	0.14	0.86

Table 3: Closed-loop outcomes on 36 labelled failures. Naive : judge sees frames + trace + narration. LP : least-privilege routing (frames only for the visual requirement). Round-0 is the acceptance rate of the failed clip before any repair. “+8B check” injects the 8B judge’s own verdict on the initial clip into the trace as a vision_check tool result, as a cheap checker would.

judge	evidence	$\alpha_{0}$	ceiling $1-\alpha_{0}$	best observed true pass
Qwen3-VL-8B	Naive	0.72	0.28	0.28
Qwen3-VL-8B	LP	0.14	0.86	0.86
Qwen3-VL-32B	Naive	0.25	0.75	0.75
Qwen3-VL-32B	Naive + 8B check	0.69	0.31	0.31
Qwen3-VL-32B	LP	0.08	0.92	0.92

Table 4: Round-0 identity check.

\alpha_{0}

is the observed round-0 false-accept rate; the ceiling is

1-\alpha_{0}

; “best observed” is the highest true pass rate over the planners of Table 3 .

	naive		OCR-mask
subtitle	pres.	abs.	pres.	abs.
none	0.97	0.14	0.97	0.14
supports $e_{2}$	0.97	0.50	0.97	0.14
contradicts	0.82	0.06	0.97	0.17
evidence-first, supports	0.77	0.28
evidence-first, contradicts	0.50	0.06

Table 5: Burned-in subtitle channel, Qwen3-VL-8B.

P(\text{accept})

on present / absent clips. OCR-masking restores the no-subtitle baseline; an evidence-first two-step prompt (describe frames, then judge the description) only halves the effect and loses recall.

	naive		OCR-mask
subtitle	pres.	abs.	pres.	abs.
none	0.97	0.14	0.97	0.14
supports $e_{2}$	0.97	0.50	0.97	0.14
contradicts	0.82	0.06	0.97	0.17
evidence-first, supports	0.77	0.28
evidence-first, contradicts	0.50	0.06

Table 5: Burned-in subtitle channel, Qwen3-VL-8B.

P(\text{accept})

on present / absent clips. OCR-masking restores the no-subtitle baseline; an evidence-first two-step prompt (describe frames, then judge the description) only halves the effect and loses recall.

requirement	Naive	LP
plan written before generation	0.93	0.89
narration track added	1.00	1.00
generator called with 81 frames	1.00	1.00
no tool call returned an error	1.00	1.00

Table 6: Process requirements are not harmed. Accuracy on four trace-only requirements whose truth is set by constructing the trace (70 clips each). LP sees the trace only; Naive sees frames + trace.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

id	$e_{1}$	$e_{2}$ (requirement)	filler	pres.	abs.
mug	picks up a red mug	drinks from the mug	holds it, looks out of the window	2	1
door	opens the door	walks through the doorway	stands in the doorway	2	4
hat	puts on a yellow hat	waves at the camera	adjusts the brim	2	4
bench	sits on a bench	opens a newspaper	sits with hands on knees	2	2
apple	picks up a green apple	takes a bite	inspects the apple	2	3
book	picks up a book	opens it and reads	holds the closed book	2	0

Appendix

Table 7: Prompts (scene abbreviated), events, fillers and clean-clip counts, batches 1–2 (top) and 3 (bottom). Dropped prompts: candle (candles already lit by scene prior), jacket (zip not visible), shoes (laces not visible), lamp (already lit by scene prior). Prompts with no clean both clip contribute absent clips only.

Figure 3: Generated two-event clips (Wan2.2-T2V-A14B, six of 81 frames each). Rows alternate between the both condition (event $e_{2}$ present) and the e1-only condition (filler instead of $e_{2}$ ). Two rows show filler prompts that leaked: the generator completed the natural continuation (drinking, opening the umbrella) despite the prompt—in the umbrella case despite an explicit “never opens the umbrella”. Leaked clips were relabelled by inspection and excluded from the absent set.

Figure 4: The burned-in subtitle channel and its least-privilege projection. Top: an absent clip (she never drinks) with the narration claiming the event rendered into every frame. Bottom: the same frames after EasyOCR detection and Gaussian masking of the detected text boxes, which is what the visual-requirement judge receives under least-privilege routing.

	Qwen2.5-VL-7B		Qwen3-VL-8B		Qwen3-VL-32B
auxiliary text	true	distr.	true	distr.	true	distr.
none	0.89	0.15	0.82	0.10	0.80	0.11
transcript, supports	0.98	0.19	0.93	0.11	0.83	0.14
log, supports	1.00	0.35	0.98	0.17	0.88	0.16
subtitle, supports	0.99	0.43	0.98	0.20	0.92	0.17
transcript, contradicts	0.31	0.15	0.33	0.10	0.25	0.10

Appendix

Table 8: UCF101 hard split (100 clips

\times

true/distractor class), default prompt,

P(\text{accept})

on true-class / distractor-class requirement.

Explore similar work

Sep 21, 2026cs.CV

VideoGen-Agent: Reinforcing Video Generation Agents

Recent advances in video generative models have enabled high-fidelity, temporally coherent video generation. However, these models often struggle to satisfy prompts requiring specialized knowledge, specific identities, physical consistency, or ordered events. In this paper, we present VideoGen-Agent, a multimodal agent trained through multitask agentic reinforcement learning to use external tools for video generation. The agent coordinates augmentation, generation, and verification tools through multi-turn interactions, using the prompt and intermediate observations to guide its decisions. We train a shared policy on a category-balanced dataset spanning six tasks. Supervised fine-tuning on teacher-generated trajectories establishes tool-use behavior, which is then refined through reinforcement learning. A category-aware hybrid reward evaluates tool-call validity, task-appropriate tool use, and generated video quality. We further introduce VABench, a held-out benchmark of 600 prompts covering procedural knowledge, single- and multi-entity identity preservation, physical consistency, scene composition, and multi-shot temporal structure. On VABench, VideoGen-Agent improves over its base text-to-video generator by 19.1 points, from 56.5 to 75.6. Upgrading the generation tools further raises the score to 86.1 without additional agent training. Human raters prefer the upgraded configuration over the strongest standalone baseline in 84.3% of comparisons. These results support learning tool use across video-generation tasks and show that the trained agent can benefit from subsequent advances in generation tools. Project page: https://andyca111.github.io/VideoGen_Agent/

Binxu Li, Haoyi Duan, Yuhui Zhang +9

Princeton University · Stanford University · UC Davis +4

Video Agent Video Generation Multimodal Agents Generative Models

Ask PDF

Sep 2, 2026cs.CV

VeriPhy: Agentic Physical Reasoning for World Model Evaluation and Refinement

Visual fluency in generated video does not imply physical reliability, and a scalar quality score alone is incapable of indicating the obligation a clip violates or the moment it fails. We present VeriPhy, an auditable physical-verification system in which a text-only planner compiles the prompt into typed physical obligations and a statically validated execution plan before any frame is observed. During execution, observations gate and scope only declared calls to frozen low-level experts (e.g., segmentation and tracking, counting, eleven typed physical measurements over the resulting tracks, depth, OCR, and audio-event detection). Each action returns a provenance-carrying evidence record whose payload, when usable, is either a typed measurement or an explicitly tagged learned state. Typed resolvers and fixed composition map usable records to a three-valued state (supported, contradicted, or unknown, surfaced as plausible, implausible, or abstain) with full provenance, so that every verdict is traceable to the evidence that produced it. We anchor evaluation in a 1,500-clip corpus of human-annotated flaw records that localize real generation failures in prompt reference, space, and time. On a 149-clip core carrying 304 such records, VeriPhy accounts for 228, against 164 for a published question-decomposition evaluator given the same clips and the same claims. Recall alone does not separate it from prompting the same backbone monolithically, which reaches 222; what separates them is that each decision retains its evidence record and provenance, making the traces auditable one verdict at a time and usable as the interface through which a critic verdict could be written back into generation.

Wenzhuo Xu, Yuchen Zhu, Chongjian Ge +8

1Carnegie Mellon University · 2Georgia Institute of Technology · 4Adobe Research +1

Physical Plausibility Generation Provenance Visual Evidence Provenance Agentic Reasoning Fluency+1

Ask PDF

Jun 6, 2026cs.CV

VideoWeaver: Evaluating and Evolving Skills for Agentic Long Video Generation

Agentic long video generation requires planning, tool orchestration, and cross-clip coordination over a long horizon. Most existing video agents either rely on static, human-crafted workflows, which require substantial manual effort and poorly adapt across tasks, or iteratively refine the output of the current task without persistently distilling execution experience into reusable skills for future tasks. We introduce VideoWeaver, an agent harness and benchmark that evaluates and evolves skills for long video generation. Given a single high-level instruction, an agent dynamically composes foundation skills into its own workflow rather than following a predefined pipeline. We construct a benchmark of 16 task categories and 285 cases, with references spanning text, image, audio, video, and their combinations. We further propose an evidence-grounded agent-as-judge that inspects both the execution trace and the final video to diagnose process and output failures. Based on this feedback, our evolution algorithm progressively refines category-level composition and creator skills, allowing recurring experience to guide dynamically constructed workflows for unseen cases. Experiments show that explicit composition skills improve the generation process over foundation skills alone, while skill evolution further improves output quality and generalizes to unseen cases. Incorporating judge feedback yields additional gains, especially on output metrics, and the agent-as-judge aligns well with human, particularly on process metrics. Code is available at https://github.com/JianhuiWei7/VideoWeaver.

Jianhui Wei, Yan Zhang, Jie Tan +7

Zhejiang University · ByteDance

Video Agent Long Video Generation Long Videos Skill Evolution

Ask PDF