Organizations: School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen · vivo AI Lab · University of the Chinese Academy of Sciences · Southeast University · Peking University · School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen · Shenzhen Future Network of Intelligence Institute (FNii-Shenzhen) · Guangdong Provincial Key Laboratory of Future Networks of Intelligence, CUHK-Shenzhen
Practical video editing is not only pixel generation: an editor must turn a brief, a clip pool, music metadata, and hard constraints into an executable timeline. We study this decision layer as executable video-editing planning and introduce RefineCut, which, unlike workflow systems that wrap a prompted frontier model, trains a compact open-weight planner for it. The planner edits a typed timeline through structured patches covering clip selection, trimming, ordering, transitions, and duration and music alignment; a deterministic verifier applies each patch and checks it against an explicit constraint ledger. Because editing has no single ground-truth repair, we do not imitate teachers directly: RefineCut replays every multi-teacher branch through the verifier and keeps verifier-best repairs as supervision. A second stage, RefineCut-Evo, lets the student score its own repairs with the verifier and a task rubric and trains on high-margin preference pairs, so the final 8B planner runs in a closed verifier loop with no teacher calls at inference. On RefineCut-Bench (3,578 tasks, 7,971 captioned clips, 499 music tracks, explicit ledgers), verifier-replayed distillation lifts the planner from 0.620 to 0.858 on the protocol-specific Video-Editing Score and RefineCut-Evo reaches 0.924; the gain transfers to Llama-3.1-8B and GLM-4-9B, and in the same closed loop the 8B planner matches or exceeds its frontier teachers. Code and RefineCut-Bench are publicly released; see the Data Availability statement.
Figures & tables
Figure 1: From implicit to explicit planning. RefineCut turns implicit prompted editing into explicit, verifier-trained executable planning.
Figure 2: RefineCut-Bench overview. The benchmark ties real clip/music metadata, explicit ledger constraints, multi-teacher trajectories, and verifier-based evaluation into one planning-level protocol.
Figure 3: Framework. Noisy teacher trajectories become verified supervision; verifier- and rubric-scored student repairs drive self-improvement and closed-loop planning.
Figure 4: Training-stage progression. Common-100 VES across checkpoints; annotations mark the Raw → Verified and Mixed-Pref → Evo aggregate gains.
Constraint adherence
Loop quality
Overall
Model
FCSR
Hard
ReqCR
Dur
Cvg
VES
Prompted backbone and raw imitation
Prompted
0.608
0.150
0.273
0.660
0.210
0.594
Raw
0.559
0.160
0.723
0.440
0.250
0.620
Verifier replay distillation
Verified
0.880
0.630
0.975
0.820
0.790
0.858
Table 1: Closed-loop planning results on Common-100. The Δ row gives aggregate changes relative to Mixed-Pref; ⋆ marks the final RefineCut checkpoint.
Replay scale
Verifier components
Teacher
N
Branches
FCSR
Hard
PASR
ReqCR
VES
GPT-5.4
2,000
6,000
0.379
0.015
0.872
0.698
0.521
Qwen3-Max
1,946
3,892
0.446
0.016
0.989
0.712
0.557
DeepSeek-V4-Pro
1,959
3,920
0.473
0.028
0.946
0.703
0.559
Table 2: Verifier-best replay diagnostics by teacher. Per-branch values are computed from replay logs; this diagnoses the teacher resource, not student closed-loop performance.
Component metrics
Overall
Variant
FCSR
Hard
Dur
Cvg
VES
Rubric-margin DPO
0.9530
0.820
0.980
0.950
0.9237
Verifier-only DPO
0.9395
0.780
0.940
0.900
0.9085 ↓ 0.015
No-margin DPO
0.9178
0.760
0.910
0.870
0.8965 ↓ 0.027
UCPO-lite DPO
0.9012
0.740
0.890
0.840
0.8800 ↓ 0.044
Table 3: RefineCut-Evo ablation under matched backbone, candidate pool, DPO recipe, and wall-clock budget. Verifier-scored self-improvement is the primary signal; rubric-structured margins add 0.015 .
VES by training stage
Gain
Backbone
Prompted
Raw
Verified
Δ V − R
Qwen3-8B
0.594
0.620
0.858
+0.238
Llama-3.1-8B
0.566
0.496
0.649
+0.153
GLM-4-9B
0.643
0.607
0.686
+0.079
Table 4: Cross-backbone transfer. Same data, recipe, and frozen protocol; 95% CIs: +0.153[0.109,0.197] , +0.079[0.036,0.120] . Full components in Appendix G , Table 13 .
Policy (identical loop)
VES
Hard
Cvg
Frontier teachers (online)
GPT-5.4
0.893
0.76
0.86
Qwen3-Max
0.773
0.63
0.72
DeepSeek-V4-Pro
0.936
0.89
0.95
Untrained Qwen3-8B: prompting and search
Direct prompting ( T=1 )
0.502
0.07
0.08
Table 5: Same-loop external comparison. All policies use the identical state, ledger, RefinePatch interface, Apply/Verify loop, stopping rule, and T=3 budget. Full tables with latency, W/T/L, CIs, and R=1 sampling are in Appendix G .
Figure 5: Failure composition. RefineCut-Evo increases OK plans from 74 to 91 and reduces duration mismatches from 16 to 1 on Common-100; the small residual tail shows all remaining closed-loop failure types.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Item
Value
Raw video clips
7,971
Extracted caption frames
23,913
Music tracks
499
Raw task records
3,960
Canonical unique tasks
3,578
Train / Dev / Test (records)
2,773 / 596 / 591
Appendix
Table 6: Dataset, splits, and teacher coverage. Teacher counts are GPT-5.4 / Qwen3-Max / DeepSeek-V4-Pro.
Source
Captioned clips
panda70m_sample
3,990
pexels
3,219
pixabay
255
openvid
100
long_video_segment
407
Total
7,971
Appendix
Table 7: Captioned clip counts per source. The total matches the asset count reported in Section 2.4 .
Family
Subtypes
A (multi-clip
themed_montage, pacing_progression,
composition)
motion_dynamics_montage, scene_traversal,
character_focus, abstract_rhythm
B (local
b_roll_insert, clip_swap, cut_extend,
edit/repair)
transition_repair
C (generative
generated_opener, generated_bridge,
Appendix
Table 8: The 15 task subtypes grouped by family.
repair_operator
Intent
modify_prompt
rewrite the generative prompt
change_tool
switch the downstream tool slot
adjust_parameter
tweak a numeric or enum field
reselect_clip
replace a clip in the sequence
retry_with_fallback
redo with a safer default
abort_and_skip
drop the step
Appendix
Table 9: The six repair_operator values.
ID
Criterion
Avg. weight
ER1
Intent and Story Alignment
0.09
ER2
Ledger Satisfaction
0.25
ER3
Clip Grounding and Relevance
0.22
ER4
Timeline and Segment Coherence
0.11
ER5
Duration and Pacing Alignment
0.15
ER6
Music / Beat Alignment
0.05
Appendix
Table 11: Average rubric weights across the train set (rounded; per-task weights sum to 1 ). Family A puts more mass on ER4; families B and C put more mass on ER3.
Instruction-based general video editing seeks to unify diverse editing operations within a single, intuitive interface. Existing approaches often rely on resource-intensive conditioning, using either heavyweight branches or costly source concatenation. Is there any efficient way to model editing intent? Thus, we introduce GRNEdit, a lightweight two-stage framework. GRN inspires our approach by encoding visual semantics through combinations of bits. Through task-specific fine-tuning, we take this representation further and recast editing semantics as local retain-or-flip decisions over individual bits. Source information is consequently modeled as coordinate-wise evidence supporting the observed binary states, while the GRN backbone remains responsible for resolving their global composition into coherent generative semantics. In Stage I, a compact encoder translates discrete source codes into continuous evidence signals, which GRN assimilates throughout binary refinement. Inspired by null-prompt training for classifier-free guidance, we further assign the null condition an editing-specific meaning: an empty instruction denotes no edit and is supervised through source reconstruction. This identity pathway not only implicitly strengthens evidence utilization and content preservation in Stage I, but also produces a source-preserving state in the same representation space as the edited state. Stage II can therefore directly compare each edited state with its source-preserving counterpart and use their discrepancy to revise unresolved target-bit decisions. Trained on only 0.6M pairs with less than 3% conditioning parameters, GRNEdit-2B and GRNEdit-8B achieve scores of 4.03 and 4.18 on OpenVE-Bench. The 2B model outperforms multiple 14B open-source editors, while the 8B model performs on par with leading open-source editors.
AI agents increasingly carry out long-horizon professional work, but their evaluations rarely require a finished creative deliverable. To this end, we introduce Timeline-Bench, a benchmark of 56 real video-editing tasks, each asking an agent to turn raw production material into a finished video. Tasks range from selecting dialog takes and shaping interview footage into a story to cutting commercials from product shots, voiceovers and graphics. Every task provides a brief, source assets, a container and a set of tests. A task is resolved when the output passes every test. The tests check the delivery format, the content and the brief's explicit requirements, and include a quality test calibrated on 2,582 blind judgments by 43 video editors. We evaluate 16 agents that pair frontier models with coding-agent harnesses such as Codex, Claude Code and OpenCode. The best, GPT-6 Astra in Codex with curated editorial guidance, resolves only 15 of the 56 tasks (26.8%), and the average agent resolves 14.0%. Human editors prefer the reference edit in 83.5% of judgments. Most unresolved runs (562 of 771) fail only the quality test: agents perceive footage through stills and transcripts and check their renders for defects, not craft. We release the tasks, verifier and per-run results at https://timelinebench.tensortest.com.
Modern image editing models produce realistic results but struggle with abstract, multi step instructions (e.g., ``make this advertisement more vegetarian-friendly''). Prior agent based methods decompose such tasks but rely on handcrafted pipelines or teacher imitation, limiting flexibility and decoupling learning from actual editing outcomes. We propose an experiential framework for long-horizon image editing, where a planner generates structured atomic decompositions and an orchestrator selects tools and regions to execute each step. A vision language judge provides outcome-based rewards for instruction adherence and visual quality. The orchestrator is trained to maximize these rewards, and successful trajectories are used to refine the planner. By tightly coupling planning with reward driven execution, our approach yields more coherent and reliable edits than single-step or rule-based multistep baselines.
Anirudh Sundara Rajan, Krishna Kumar Singh, Yong Jae Lee
University of Wisconsin–Madison, WI, USA · Adobe Research, San Jose, CA, USA