Coding agents work in an environment built for them: a compiler or a test checks each change and points to the fault, and version control keeps every change in a project people can continue. Video editing agents lack one: an edit's correctness shows only in the costly rendered video, which does not point to the decision at fault, and existing systems offer either editing operations that cover only anticipated edits or a script whose output is a video the user cannot keep editing. We present Cassette-Editor, a complete agent harness for video editing that lets the agent edit the way a person does: knowing what the project allows, seeing what each change affects, and revising from there. The agent writes patches of typed changes to the editor's timeline document. A schema check and a rulebook shared with manual editing check each patch before it commits; a rejected patch leaves the project unchanged and returns each problem located with repair options, and an accepted one becomes a step of the history the user also edits. Reads map source analysis to the frames of the current edit. Because manual gestures are writes in the same language, folded by the same executor and rulebook, the agent can express every change a manual edit makes to the project. On 31 editing requests with the model held fixed, the complete harness passed 89% of trials against 47% for the same model with ten basic editing operations and without the rulebook, at less than half the agent time and cost per successful request; the loss fell on requests at rule boundaries, composite edits, and undo, not on finding content.
Figures & tables
Figure 1: System overview and coding-agent analogy. ❶ The agent writes one patch. ❷ Field checks type-check it against the document schema, the executor constructs a candidate document, and the rulebook supplies semantic checks and lint diagnostics; manual edits enter the same executor and rulebook, and a blocking violation returns diagnostics without changing the shared document. ❸ An accepted patch commits as one history step. ❹ read_resource answers from the timeline graph the editor compiles from the document, with code the rulebook, preview, and renderer share, and from source analysis. Version checks and patch translation are omitted.
Figure 3: Cassette-Editor after one agent request. The user asks in the chat panel (right) for a five-shot teaser with a title and dissolves; the agent commits one patch, version v1, which the timeline (bottom) and preview (left) show and which the user can undo or keep editing by hand. Footage: Brewing Coffee with Moka Alessi and Peugeot Bresil Mill , Shokuiku Cuisine, Wikimedia Commons, CC BY 3.0; resized and cut.
Figure 4: How the user and the agent take turns on the shared history (schematic); each step is one commit. ❶ An editing run holds the project’s write lease, and the user’s edits and undo are refused until it ends. ❷ The user trimmed c3 between runs, so a patch made on the earlier version is refused; the agent reads again and commits. ❸ An edit after an undo starts a new branch and keeps the old one; ❹ the user can switch to a kept branch, and the line it leaves is kept too.
Figure 5: A recorded run on a project of three shots with a dissolve on each cut. ❶ The rulebook rejects the first patch whole because a cut takes one transition, and its receipt names the dissolve to change or remove; ❷ the agent amends only the second change, and ❸ a further patch adds the new dissolve.
Figure 6: Pass rate by request family with one component removed. Catalog: small catalog; No conseq.: no derived consequences; Src.-time: source-time reads. Filled points are targeted families, hollow points control families. Lines show the full harness and basic tools on the whole family. ∗ The six requests that locate a phrase. On find requests every condition passed 83%.
Pass rate by request family (%) ↑
Per successful task ↓
Per trial ↓
Condition
Find
Conseq.
Rule
Comp.
Undo
All
All 3 trials passed ↑
Agent time (s)
Cost (USD)
Write calls
Rejec- tions
Basic tools
83
85
6
11
25
47
13/31
158.5
0.092
5.28
0.72
Full harness
83
100
94
72
92
89
25/31
65.4
0.042
1.33
0.01
Difference
0
− 15
− 89
− 61
− 67
− 42
− 12
2.42 ×
2.21 ×
+3.95
+0.71
Table 1: The full harness against the same model with basic tools on all 31 requests, three trials each. Per successful task, time and cost sum all trials and divide by the passes. The last row gives basic tools minus the full harness (ratios for time and cost). Arrows indicate preferred directions; the better result is bold.
Measure
Pass rate, other tasks
Condition
Tasks
Measure
Full → cond.
Difference
pHolm
Targeted
Control
Basic tools
31
Pass rate (%)
89 → 47
− 42
0.0004
–
–
One component removed
Small catalog
15+6
Pass rate, targeted (%)
89 → 56
− 33
0.16
–
0
No rulebook
15+6
Violations left
0 → 0
0
1.00
− 11
0
No derived consequences
15+6
Rejections per trial
0.02 → 0.08
0.06
0.50
− 4
0
Table 2: Each condition against the full harness on the requests it runs (targeted+control). Measures were fixed before the runs, except the repair’s time and cost. Differences are condition minus full harness (pass rates in points; geometric-mean ratios for the repair). Exact permutation tests are Holm-corrected over six primary comparisons, the sixth being repairs within three attempts (100% in both).
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Trials passed out of three for every request and condition; grey cells are requests a condition does not run.
Figure 8: The timeline graph the editor compiles from the running example. Each sequence has its own graph, and containment is drawn as nesting. Values in gray italics are derived when the graph is compiled: each node’s frames, each instance’s mapping into compound A, and the source paths through which a1, defined once, plays at two ranges of the root.
Figure 9: The read path in full, computed by the editor on the running example’s storyline. Top: root frame 211 lies in the second instance of compound A, which plays its frame 21; a1 plays that frame backwards, so the read reports one source time, and stored evidence is found by that time; the steps are numbered as in Figure 2 . The plot gives the source time at every frame of A. Bottom: a subject box is carried through cover fit, a1’s clip transform, and the instance transform of A#2 into root-composition pixels; when a projection cannot be computed exactly, the read says so. Frames, source times, and pixel coordinates are the editor’s output; the document’s windows and transforms are chosen, and the subject box is placed by hand.
AI agents increasingly carry out long-horizon professional work, but their evaluations rarely require a finished creative deliverable. To this end, we introduce Timeline-Bench, a benchmark of 56 real video-editing tasks, each asking an agent to turn raw production material into a finished video. Tasks range from selecting dialog takes and shaping interview footage into a story to cutting commercials from product shots, voiceovers and graphics. Every task provides a brief, source assets, a container and a set of tests. A task is resolved when the output passes every test. The tests check the delivery format, the content and the brief's explicit requirements, and include a quality test calibrated on 2,582 blind judgments by 43 video editors. We evaluate 16 agents that pair frontier models with coding-agent harnesses such as Codex, Claude Code and OpenCode. The best, GPT-6 Astra in Codex with curated editorial guidance, resolves only 15 of the 56 tasks (26.8%), and the average agent resolves 14.0%. Human editors prefer the reference edit in 83.5% of judgments. Most unresolved runs (562 of 771) fail only the quality test: agents perceive footage through stills and transcripts and check their renders for defects, not craft. We release the tasks, verifier and per-run results at https://timelinebench.tensortest.com.
Editing a long-form video from heterogeneous footage requires more than selecting clips: an agent must preserve narrative intent across material preparation, timeline construction, post-production, and revision while leaving enough evidence to diagnose failures. We present Crayotter, an open-source multimodal multi-agent system for prompt-driven video editing. Crayotter organizes production into three phases: coverage-aware material preparation, artifact-based editing research, and tool-grounded timeline execution. Each phase externalizes inspectable artifacts, including coverage reports, multimodal analyses, editing blueprints, tool calls, and intermediate renders. These artifacts make an editing run traceable and allow failed segments to be diagnosed and selectively revised instead of requiring a full restart. We evaluate Crayotter on 23 editing themes against CapCut-Mate and CutClaw. Under human evaluation, Crayotter achieves an average score of 3.40/5, compared with 2.44 and 1.70 for the two baselines, with consistent gains in theme alignment, narrative coherence, and editing smoothness. We additionally describe a replayable trajectory schema and verifiable reward design that prepare these workflows for future policy optimization. Code, traces, and examples are publicly available at https://github.com/idwts/Crayotter.
Video editing has become essential in digital media creation, yet existing automated systems are restricted to short segment processing and domain-specific tasks. They face two critical limitations: i) inability to handle diverse video comprehension and editing operations, and ii) lack of long-video understanding for coherent narrative creation. We propose VideoAgent, an all-in-one agentic framework addressing these challenges through two key innovations. First, we develop automated video shot creation with shot planning agents for coherent narratives and cross-modal retrieval for aligned visual content. Second, we design a multi-agent orchestration framework integrating over thirty specialized editing agents. Intent parsing filters relevant tools while textual-gradient graph optimization assembles complex editing pipelines. Extensive experiments on our newly-proposed VideoEdit benchmark and public datasets demonstrate VideoAgent's superiority over existing multimodal LLMs and agentic systems. VideoAgent achieves 87-95% orchestration success rates while reducing API costs by 60%. Human evaluation across six video categories shows VideoAgent produces professional-quality content approaching human-level performance, with ratings only 4% below human-created videos. We release our code at https://github.com/HKUDS/VideoAgent.
Hengji Zhou, Lingxuan Huang, Jian Wang +4
1Harbin Institute of Technology, Shenzhen · 2South China University of Technology · 3The University of Hong Kong +1