When Fine-Grained Credit Assignment Helps and Hurts: Multimodal Agentic Reinforcement Learning through Video Editing

Lei Zhao; Zongheng Cao; Xinyu Hu

cs.LGOct 26, 2026

When Fine-Grained Credit Assignment Helps and Hurts: Multimodal Agentic Reinforcement Learning through Video Editing

Authors: Lei Zhao, Zongheng Cao, Xinyu Hu

Organizations: University of Pennsylvania · Philo Labs

Abstract

Reinforcement learning (RL) has shown promise for training multimodal agents to reason and use tools. However, it remains unclear when fine-grained credit assignment improves learning and when it fails to improve task performance. Video editing provides a useful testbed for this question, combining multimodal perception, long-horizon planning, and tool use with delayed outcome feedback. We construct four movie-trailer editing task types---Content Selection, Narrative Sequencing, Rough Cut, and Fine Cut---that reflect real-world editing workflows and span different levels of task complexity and interaction horizon. Decaying process rewards provides the largest benefit on Fine Cut, with a 45.4% relative gain over outcome-only GRPO on tasks averaging more than 60 turns under outcome-only GRPO. On the three shorter-horizon task types averaging 30--40 turns under the same baseline, outcome-only GRPO remains competitive, with decaying process rewards yielding at most a 10.4% relative gain. Process rewards from an LLM judge initially improve tool use and encourage visual inspection. As training progresses, however, GRPO+PRM with a fixed process-reward weight increasingly exhibits excessive reasoning and redundant tool calls: process rewards continue to rise while task completion rates and outcome scores decline. Decaying the process-reward weight to zero mitigates this reward-hacking behavior while preserving the early gains. Despite dense token-level supervision, on-policy self-distillation (OPSD) yields little improvement or substantial degradation and is highly sensitive to the choice of privileged information. Our analysis reveals differences in teacher--student disagreement and gradient magnitudes, and suggests possible dilution of informative signals over long trajectories.

Figures & tables

Figure 1: Task-dependent effects of process rewards for Qwen3.5-9B. (a) Independent task schematics, ordered by base-policy horizon. Turns and base scores are task-type averages, not example-specific results; evaluation metrics average four repetitions. Fine Cut keeps picture and timing fixed. (b) Final score differences from outcome-only GRPO. (c) Fixed-weight process and outcome rewards, and decayed-weight outcome rewards available from update 21. Faint lines connect recorded observations; thick lines show centered five-observation means. The aligned strip shows fixed and cubic-decay weights $\lambda$ . Shading marks the decayed variant’s 20–30 decay window.

Task stage	Task type	Input	Output	Turns	Score
Understanding	CS	Candidate shots	Selected shot IDs	22.3	0.657
	NS	Shuffled shots	Ordered shot IDs	15.3	0.303
Editing	RC	Ordered shots, target duration	Clip in/out points	21.8	0.205
	FC	Rough cut, audio assets	Rendered edit	87.2	0.152

Table 1: Task stages, inputs and outputs of the four task types. CS: Content Selection; NS: Narrative Sequencing; RC: Rough Cut; FC: Fine Cut. Each task also receives an editorial brief. The RC target specifies total duration. Turns and score are averaged over four independent evaluations of all 81 validation tasks with the untrained Qwen3.5-9B.

Split	CS	NS	RC	FC	Total
Train	122	132	231	231	716
Validation	14	15	26	26	81

Table 2: Task counts by type. The training and validation films are disjoint.

	Mean task score (avg@4)
Method	CS	NS	RC	FC	Overall
Qwen3.5-9B	0.657	0.303	0.205	0.152	0.284
Outcome-only GRPO	0.821	0.500	0.356	0.201	0.413
	$(+24.9\%)$	$(+65.1\%)$	$(+73.4\%)$	$(+32.9\%)$	$(+45.5\%)$
GRPO+PRM (fixed $\lambda$ )	0.482	0.457	0.276	0.239	0.334
	$(-26.6\%)$	$(+51.0\%)$	$(+34.8\%)$	$(+57.8\%)$	$(+17.4\%)$

Table 3: Validation scores for the final checkpoints of all training variants across the four task types. Each score is the mean over four repetitions (avg@4). Percentages below scores show relative changes from the base model, computed using the four-decimal evaluation means. Best result per column in bold.

Task type	Positive	Neutral	Negative	Mixing
CS	55.3%	36.0%	8.7%	0.297
NS	42.7%	44.9%	12.4%	0.355
RC	44.5%	43.6%	11.9%	0.390
FC	16.7%	47.5%	35.8%	0.651

Table 4: Turn-level quality of the untrained model’s rollouts from the second evaluation repetition on all 81 validation tasks, scored by the process-reward judge.

Method	Turns	Tool calls	Truncated	Frames	Blind	Length	Reflection
	Interaction			Visual inspection		Reasoning
Qwen3.5-9B	41.7	61.4	1.5%	32.2	7.4%	644	17.7%
Outcome-only GRPO	45.1	52.2	0.3%	84.2	4.6%	618	19.0%
GRPO+PRM (fixed $\lambda$ )	82.4	165.8	44.1%	143.6	0.3%	761	24.8%
GRPO+PRM (decayed $\lambda$ )	56.9	99.5	6.2%	129.8	0.0%	1,035	36.0%
OPSD (reference answers)	41.1	62.5	1.5%	36.7	3.4%	657	17.8%

Table 5: Validation behavior across four repetitions of all 81 tasks. Fixed-weight PRM has a truncation rate of 44.1%; decaying the weight reduces it to 6.2%, with longer reasoning and more frequent reflection markers.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 2: Data generation from a film–trailer pair. (a) Two real source clips and their reference-aligned cuts illustrate shot matching. (b) Each local chunk stays within a story beat and receives a clip-independent brief. (c) Four tasks share the brief and initialize from reference states, so errors in sampled outputs cannot propagate between tasks. Timeline and asset layouts are schematic. D/M/S denote dialogue, music, and production sound.

Film	CS	NS	RC	FC
Heretic	4	2	5	4
Sinners	4	2	4	4
The Drama	2	3	3	5
They Will Kill You	1	2	3	4
Whiplash	3	3	7	4
Deadpool & Wolverine	0	3	4	5

Appendix

Table 6: Final validation tasks by film and task type. These counts describe the 81 tasks, before repeated evaluation.

Tool	What it returns
view_image	Prepared or policy-generated still images selected by path.
inspect_video	Sampled video frames selected by timestamps or an interval; returns actual decoded times rather than continuous playback.
inspect_frames	Frame-index or time-based inspection near a cut, optionally cropped; returns a sheet or individual images.
contact_sheet	A numbered sheet of up to 16 requested source images or frames.
inspect_audio	Duration, level, silence and an offline English transcript for a requested audio window.
analyze_audio	Loudness, true peak, channel-level measurements, silence, and onset or pulse candidates for an audio window.

Appendix

Table 7: Media tools available to the policy. None of them exposes a task label, a hidden reference decision or an editing-specific oracle.

Setting	Value
Tasks per update	16
Optimizer	AdamW, constant learning rate $1\times 10^{-6}$
Micro-batch per GPU	2
Training updates	50
Clipping thresholds $(\delta_{\mathrm{low}},\delta_{\mathrm{high}})$	$(0.2,\,0.28)$
Loss reduction	token mean

Appendix

Table 8: Training settings shared by all training variants.

Qwen3.5-9B
Setting	Outcome-only GRPO	GRPO+PRM (fixed $\lambda$ )	GRPO+PRM (decayed $\lambda$ )	OPSD
Rollouts per task	8	8	8	1
Process-reward weight $\lambda$	0	0.5	$0.5\!\to\!0$	0
Advantage estimator	GRPO	GRPO	GRPO	—
Policy loss	rollout importance sampling			top- $K$ KL
Zero-variance filter	yes	yes	yes	no

Appendix

Table 9: Per-variant training configuration. The schedule for GRPO+PRM (decayed

\lambda

) takes

\lambda

from 0.5 to zero with a cubic curve over updates 20 to 30 and holds it there through update 50. OPSD optimizes the top-

K

distillation objective of Equation 3.4 against a frozen copy of the initial policy, retaining the teacher’s top 64 tokens, and uses no outcome or process reward. Our OPSD runs sample one rollout per task rather than eight, so their configured trajectory count per update is one eighth of that of the reward-based variants.

Score	Evidence required
$+1$	New task-relevant information or inference supported by observations; a verified correction; or a substantive edit that demonstrably advances a task requirement.
$0$	Routine file operations, an unchanged check, or an honest status update without verified new progress. A successful command or valid JSON alone does not earn credit.
$-1$	An observed error, a demonstrated task violation, an invalid claim of verification, or a supported persistent repetition of already-resolved work.
Abstain	Material evidence is missing or inadequate for the proposed judgment. Unavailable media or an infrastructure failure alone does not establish an agent error.

Appendix

Table 10: Process-judgment decision rules. These summarize the implemented rubric; they specify intended supervision rather than guarantee the correctness of every judgment.

Setting	Value
Validation tasks	81, from six held-out films
Evaluations per task	4 independent repetitions
Evaluated checkpoint	after 50 updates, unless stated otherwise
Sampling	temperature 1.0, top- $p$ 1.0, top- $K$ $-1$
Thinking	enabled
Turn budget	100

Appendix

Table 11: Evaluation configuration. The outcome grader applies the rubric formulation of Section 4.1 ; the process-reward judge is used for training the GRPO+PRM variants and for the turn-quality analysis.

Method	CS	NS	RC	FC	Overall
Qwen3.5-9B	22.3	15.3	21.8	87.2	41.7
Outcome-only GRPO	34.8	32.3	37.7	65.4	45.1
GRPO+PRM (fixed $\lambda$ )	70.4	84.6	85.1	84.9	82.4
GRPO+PRM (decayed $\lambda$ )	31.6	52.5	54.3	75.7	56.9
OPSD (reference answers)	23.0	13.5	22.6	85.2	41.1
OPSD (turn-level suggestions)	11.8	9.0	12.2	13.0	11.8

Appendix

Table 12: Mean turns by task type over four independent repetitions of all 81 validation tasks (324 rollouts per method). Overall averages over all tasks, weighted by the number of tasks of each type.

Privileged information	Mean KL	Mean norm	Max norm	Clipped
Reference answers	0.0019	0.084	0.249	0
Outcome-only GRPO	—	0.240	0.296	0
Turn-level suggestions	0.0279	1.177	3.981	22

Appendix

Table 13: Mean teacher–student KL per token and gradient-norm summaries. The two OPSD mean gradient norms are those reported in Section 6.3 . Mean KL, maximum norms, clipping counts and the outcome-only GRPO mean norm are computed over the 50 retained optimizer updates, excluding superseded restart attempts. Gradient norms are measured before global clipping. Rows are ordered by the mean norm. The clipping threshold is

1.0

, and the Clipped column counts retained updates whose norm exceeded it. Outcome-only GRPO trains on no distillation target and therefore has no teacher–student KL, and its row pools the 50 updates of that run across its resumed segments.

	Share of total KL held by			First fifth
Privileged information	top $1\%$ turns	top $5\%$ turns	Tokens in top $5\%$	over last
Reference answers	19.8%	47.4%	17.9%	3.91
Turn-level suggestions	5.6%	19.6%	14.2%	0.72

Appendix

Table 14: Concentration of the teacher–student KL across the turns of a run, over the 800 recorded training trajectories per run. Ranking uses total KL per turn; the positional ratio uses the mean of per-turn, per-token KL values.

Privileged information	$r$	$r^{2}$	$t$
Reference answers	$-0.165$	0.027	$-4.71$
Turn-level suggestions	$-0.007$	0.000	$-0.19$

Appendix

Table 15: Correlation between the per-token KL of a rollout and the outcome reward it earns, over the

800

rollouts of each run. The reported

t

is the nominal Pearson-correlation statistic. These pooled rollouts share training updates and task identities; the statistic does not account for this dependence and should not be interpreted as a causal test.

Type	Channel	Items with weights
RC	Blind	cut placement 0.35, beat coverage 0.40, join continuity 0.25
RC	Reference	picture entries 0.24, picture exits 0.24, pacing 0.32, mechanical duration 0.20
FC	Blind	dialogue 0.32, music 0.20, continuity 0.20, transition 0.28
FC	Reference	dialogue 0.40, continuity 0.35, music 0.25

Appendix

Table 16: Rubric items and weights inside each judge channel. The blind channel asks whether the edit works on its own terms, and the reference channel asks how far it departs from the released trailer.

Item and weight	Evidence and task requirement	Failure criteria and scope
B: cut placement (0.35)	Submitted picture entries and exits, inspected with the mapped source cut-context reel; whether the selected windows make the intended actions legible.	Visible premature cuts, truncated actions, missed moments, or dead holds. Equal shot lengths, starts at zero, or static similarity alone do not show a defect. Finished sound is not required.
B: visual coverage (0.40)	The full assembled sequence against the prompt’s explicit visual actions, relationships, and requested beat.	Required content is absent, unclear, or too brief to understand. An action visible only in discarded source context does not count as delivered; ASR does not substitute for visual coverage.
B: join continuity (0.25)	Actual consecutive submitted shots in motion; spatial and action continuity, rhythm, and the style requested by the task.	Distracting discontinuities or confusing joins that disrupt the intended sequence. Static likeness is not proof of a jump cut, and deliberate montage can be appropriate.
R: picture entries (0.24)	Submitted entries, source context, and available reference entry evidence; whether useful openings and essential picture moments remain legible.	Entries that miss the useful opening or linger before it. A numeric departure from the reference is not itself a defect; effective alternative entries can receive full credit.
R: picture exits (0.24)	Submitted exits and available reference exit evidence; whether actions have enough time to read without unnecessary tails.	Visibly clipped actions or prolonged dead tails. Credit depends on the actual loss or preservation of the shot’s purpose, not exact reference matching.
R: pacing (0.32)	Continuous playback, the requested beat, and available reference pacing evidence; the distribution of time and emphasis across shots.	Concrete rushing, stalling, or misplaced emphasis. Total duration is scored separately; uniform lengths or a different duration-variation statistic are not faults by themselves.

Appendix

Table 17: RC rubric details. B and R denote judgment without the reference and reference-informed judgment. Weights are within the respective channel; the R weights include the mechanical duration item. Source context can contain discarded footage and artificial reel joins, neither of which is submitted content.

Item and weight	Evidence and task requirement	Failure criteria and scope
B: dialogue (0.32)	Listen to the submitted mix while watching the picture; assess intelligibility, placement, and narrative relevance against the prompt’s explicit requirements.	Required speech is masked, missing, irrelevant, or poorly placed. Missing ASR words alone do not prove inaudibility; lip-sync claims require visible mouth and audible speech evidence. Appropriate absence is allowed.
B: music (0.20)	The audible cue’s entry, exit, phrasing, balance, and relationship to the picture and dialogue, where the brief requires those functions.	Music fights the picture, masks speech, or fails a required musical function. A declared cue offset is not delivered alignment. A steady bed or silence can be correct; there is no requirement to hit every cut.
B: continuity choices (0.20)	Actual picture–sound continuity across joins; whether hard cuts, sustained beds, dialogue overlaps, or other bridges serve the requested stretch.	Observed continuity weaknesses disrupt the flow. There is no bridge quota: one bed may span many joins, and a deliberate hard cut may be appropriate. Unconfirmed crossings are not automatically missing bridges.
B: transition execution (0.28)	The continuous render at actual transition locations, with measured discontinuities used only to locate possible problems.	Audible chops, distracting level jumps, unintended black, inappropriate fades, or disruptive seams. Source black, title cards, and required pauses are not errors; signal discontinuities alone do not establish defects.
R: dialogue (0.40)	The same delivered-dialogue evidence, informed by the reference’s dialogue choices and the task’s intended narrative function.	An actual loss of required meaning, intelligibility, or effective placement. A different line selection is not penalized solely for differing from the reference or leaving a source opportunity unused.
R: continuity (0.35)	Delivered continuity choices and their execution, considered with the available reference evidence for the intended beat.	Disruptive joins or transition defects that weaken the actual sequence. Reference bridge annotations describe one solution; they do not impose a transition-count target or require identical bridge types.

Appendix

Table 18: FC rubric details. B and R denote judgment without the reference and reference-informed judgment. Weights are within each channel. All items evaluate the delivered audiovisual result: a declared JSON operation or a possible source-media alignment is not evidence that it was realized.

Panel of Figure 3 — Figure 3: Training dynamics of GRPO+PRM (fixed $\lambda$ ) and GRPO+PRM (decayed $\lambda$ ). Faint lines show recorded values; thick lines show centered five-observation moving averages. With a fixed weight, the process reward keeps rising while the outcome reward falls after update 30. The decayed-weight outcome curve uses the same observations as Figure 1 (c), starting at update 21; lines connect available observations.

Figure 4: Training dynamics of the five Qwen3.5-9B variants. Each point is an unsmoothed, recorded optimizer update: (a) outcome reward on training rollouts, (b) policy turns divided by the logged rollout count, and (c) logged mean assistant tokens per rollout. Outcome reward is monitored but not optimized by OPSD. Records are selected along the final checkpoint lineage. The missing fixed-weight update 30 is left as a gap. The decayed run starts at update 21 from the fixed-weight update-20 checkpoint; its earlier prefix is not replotted. These are individual training runs, with different rollout batches, not repeated held-out evaluation scores.

Figure 5: Optimization diagnostics for the two reported OPSD runs over 50 retained optimizer updates. (a) Token-mean forward KL on the teacher’s top 64 tokens plus a tail class; (b) gradient norm before global clipping, with the threshold of 1.0 dashed; and (c) logged policy entropy. Points are unsmoothed records from the final checkpoint lineage, with repeated or superseded restart attempts excluded. The suggestion-conditioned run has larger KL and gradient norms and generally lower later entropy. These diagnostics accompany the outcome degradation in Figure 4 ; they do not establish a causal mechanism.

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,727 papers

Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents
Aug 13, 2026Zechuan Wang, Siyuan Lu, Hongxuan Zhang +3Token-Level Credit Assignment Step-Level Credit Assignment in RL
Zhejiang University · AWorld Team, Inclusion AI · Shanghai Innovation Institute +2
From Visual Affinity to Learning Credit in Multimodal Reinforcement Learning
Oct 23, 2026Yuhang Han, Dingning Liu, Yujie Chen +7RL for VLMs Step-Level Credit Assignment in RL
The Hong Kong University of Science and Technology (Guangzhou) · Shanghai Jiao Tong University · Shadoweave +2
Targeting Pivotal Decisions for Credit Assignment in Agentic Reinforcement Learning
Sep 28, 2026Dongwon Jung, Hemanth Neelgund Ramesh, Yifan Wang +7Agentic RL Group Relative Policy Optimization
University of California, Davis · Microsoft · University of Washington

Aug 13, 2026J/K move · Enter open · S save

Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents

Zechuan Wang, Siyuan Lu, Hongxuan Zhang, Linjian Mo+2

Zhejiang University · AWorld Team, Inclusion AI · Shanghai Innovation Institute+2

Token-Level Credit Assignment Step-Level Credit Assignment in RL Reinforcement Learning with Verifiable Rewards Step-Level Credit Assignment On-Policy Distillation RL for Tool Use

Open paper AskPDF arXiv

Reinforcement learning with verifiable rewards (RLVR) offers a verifier-bounded performance ceiling for training multi-turn tool-use agents, yet its trajectory-level credit assignment conflates heterogeneous per-turn outcomes into a single reward signal. On-policy distillation provides dense per-token supervision but is either teacher-bounded or prone to gradient concentration collapse. We introduce

\textbf{CrEST}

, a hierarchical credit assignment framework that retains RL's verifier-bounded ceiling while incorporating dense token-level signals from a privileged self-teacher.

\textbf{CrEST}

resolves credit at two levels: turn-segmented verified advantages address inter-turn dilution, while entropy-gated self-teacher modulation refines intra-turn token contributions. Experiments on BFCL V3 and WildToolBench show that

\textbf{CrEST}

consistently outperforms both RL and distillation baselines across two model scales, with the largest gains on long-trajectory and strict session-level metrics. Our work demonstrates that the teacher's role in policy optimization can be reduced from determining update directions to modulating update magnitudes, unlocking dense credit assignment without sacrificing the verifier-bounded ceiling.

Abstract

Reinforcement Learning

Explore similar work

Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents

From Visual Affinity to Learning Credit in Multimodal Reinforcement Learning

Targeting Pivotal Decisions for Credit Assignment in Agentic Reinforcement Learning