Multimodal Concert Video Editing: System Design and Exploratory Evaluation

Lennart Schiweck; Adrian Kempf; Michelle Retig; Moritz Feucht; Tim Mangliers; Benjamin Himpel

cs.MMOct 26, 2026

Multimodal Concert Video Editing: System Design and Exploratory Evaluation

Authors: Lennart Schiweck, Adrian Kempf, Michelle Retig, Moritz Feucht, Tim Mangliers, Benjamin Himpel

Organizations: Department of Informatics, Reutlingen University 72762 Reutlingen, Germany

Abstract

Editing multi-camera concerts requires coordinated decisions about musical relevance, visual quality and cut timing. We present a modular framework that combines instrument activity from multitrack audio, visible instruments and image sharpness with beat-based timing and probabilistic camera selection. An interpretable scoring model links these components to the choice of views over complete shot intervals. Component tests and an exploratory expert study on a four-camera concert yield mixed assessments across the editing variants. The findings motivate extending local audiovisual relevance with contextual and sequence-level criteria. The architecture provides a basis for testing these extensions and studying how component behaviour affects the quality of complete edits.

Figures & tables

Axis	Wang et al. (2023)	Our system
Audio	Instrument classification from mixed audio.	Named stems; relative energy for primary/secondary roles.
Instrument matching	Audio–visual instrument agreement contributes when permitted by shot scale.	Explicit audible–visible role bonuses and missing/inaudible-label penalties.
Image criteria	Blur and shot-scale information.	Fixed gradient blur thresholds and box-area roles; no framing optimisation.
Temporal handling	Downbeat information and an estimated switching rate.	Every-fourth-beat grid and tempo-based target duration; conservative full-interval scoring.
Cut positions	Downbeat-based timing.	Next grid point after a tempo-dependent target duration.
Camera selection	Highest eligible score under shot-scale transition rules.	Proportional sampling with no immediate repeat and uniform fallback.

Table 1: Concert-directing comparison with Wang et al. (2023). The table compares system designs rather than performance on a common dataset.

Panel of Figure 1 — Figure 1: Modular concert-editing architecture. Named stems supply activity analysis and are mixed into a mono waveform for timing. Visual roles and sharpness combine with audible roles in a camera utility; shot boundaries define the aggregation intervals. Camera selection produces the edit. Reference annotations form an alternative label input for the annotation-driven evaluation variant.

Camera	View	$n$	Class supports
CAM1	Audience-oriented view	481	7 / 474 / 0
CAM2	Close-up view	480	54 / 386 / 40
CAM3	Close-up view	481	4 / 453 / 24
CAM4	Wide view	481	481 / 0 / 0

Table 2: Camera mapping and blur reference supports.

n

is the number of one-second evaluation samples used in the blur evaluation. Supports are ordered as not blurred / partially blurred / fully blurred.

Variant	Window	Cuts	$n$	Mean	Median	SD
V1	Full	35	36	12.99	6.94	19.70
V2	Full	65	66	7.09	6.72	2.27
V3	Full	65	66	7.09	6.72	2.27
V4	Full	65	66	7.09	6.72	2.27
V1	Music	34	35	10.86	7.14	9.68
V2	Music	55	56	6.79	6.67	2.18

Table 3: Shot-duration summaries from the complete rated videos and the approximately 380-s music window (80–460 s). Durations are in seconds. Cuts are internal camera changes in the window;

n

is the number of positive displayed segments, including clipped boundary segments. SD is the sample SD, not uncertainty across concerts. V2–V4 are listed separately, not pooled.

Variant	E1	E2	E3	Mean	Median	SD
V1: student reference	3	3	3	3.00	3	0.00
V2: random	4	3	1	2.67	3	1.53
V3: predicted scores	5	4	3	4.00	4	1.00
V4: annotated scores	4	4	1	3.00	4	1.73

Table 4: Complete overall grades, 1 = very good and 5 = poor. Mean, median and sample SD summarise three ordinal responses descriptively; they do not estimate uncertainty across concerts.

Concern	Observation and counterpoint	Example locators
Musical action and view choice	E1 describes V3’s cut positions as matching V2 but its chosen views as worse, often missing the musician carrying the theme.	E1/V3: cut comment, remarks
Appropriate moments	Nose blowing and people standing or leaving attract criticism in several variants; this is not exclusive to V3.	E1/V2: cut comment; E1/V4: cut comment; E3/V3: impression comment
Framing and view diversity	E2 repeatedly requests better framing and close-ups; E3 criticises the detail-poor wide view. These comments do not prove that better raw alternatives were unavailable.	E2/V1–V4: remarks; E3/V1: sharpness comment
Interaction and reactions	E3 praises reactions, laughter and musical responses in V2. Such views can be valuable without showing only the leading instrument.	E3/V2: impression comment, remarks
Different pacing criteria	E1/E2 find V2’s pattern schematic or unmotivated, while E3 finds it coherent and dynamic.	E1/V2: cut, transition comments; E2/V2: cut comment; E3/V2: cut comment
Contextual audience value	E1/E2 criticise V3’s audience choices; E3 also values some audience reactions while rejecting unmotivated views.	E1/V3: remarks; E2/V3: cut comment; E3/V3: cut and impression comments

Table 5: Themes in the written expert feedback. Locators identify expert, video and comment field.

Component and score pathway	Expert assessment	Architectural change and isolated test
Audio roles: smoothed relative energy changes match bonuses and missing-label penalties.	E1/V3 remarks report missing musical action; primary-role F1 is 0.654.	Activity and musical relevance could be separated; smoothing or relevance input could vary independently, with score/choice changes logged.
Visible roles: box area supplies prominence, without framing, action suitability or reaction context.	E2/V1–V4 remarks criticise framing; E1/V4 cut comment reports an unsuitable moment; E3/V2 remarks praise reactions.	Shot eligibility could be distinguished from musical/contextual relevance, with each tested using independently annotated inputs before introducing a new detector.
Quality map: partial blur and not blurred both have multiplier one.	CAM1 has 474 partial-blur reference samples but is predicted not blurred; E1/V2 and E3/V3 sharpness comments mention focus.	Correcting those labels alone has no score effect. A calibrated quality map could be tested separately from label accuracy.
Interval/policy: short intervals use their minimum score clipped at zero; zero utility is not a veto under fallback.	E1/V3 cut comment reports poorer views at V2-like cut times; the internal reason is unobserved.	Eligibility, aggregation and sampling could be varied one at a time, with complete decision logs.
Timing: a beat/tempo heuristic also covers speech; automatic pacing is matched.	Nine announcement cuts; E2/V2 and V4 cut comments criticise speech timing; E3/V2 cut comment praises rhythm.	A speech/music/applause state could support timing tests separately from view selection.
Cue interface: V4 adds pan/clapping labels alongside reference roles.	V4 improves on V3 for two raters but does not outgrade V2.	Label sources could be compared under cue parity, with optional cues then tested as a separate factor.

Table 6: Component-to-evaluation links and proposed architectural tests. Operational consequences follow from the scoring and selection rules; feedback locators identify reported outcomes. Proposed improvements have not been evaluated.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Symbol / condition	Value
Visual detector	D-FINE / Objects365	X
Detection confidence	retained predictions	0.55
Primary visual role	fraction of frame area	$>0.20$
Audio RMS frame / hop	seconds	3.0 / 1.5
Audio smoothing	centred RMS frames	9
Activity normalisation	shared percentiles	1 / 99

Appendix

Table A.1: System parameters for audio analysis, visual cues, camera scoring and selection.

Instrument	$N$	P	R	F1	$N$	P	R	F1	$N$	P	R	F1
	Not active				Primary				Secondary
Bass	189	.639	1.000	.779	0	—	—	—	292	1.000	.599	.749
Drums	189	.508	1.000	.674	95	.667	.021	.041	197	.541	.299	.386
Guitar	154	.726	1.000	.842	95	.528	.905	.667	232	.953	.435	.598
Piano	481	1.000	.981	.991	0	—	—	—	0	—	—	—
Speech	413	.998	.983	.990	68	.905	.985	.944	0	—	—	—

Appendix

Table B.1: Audio-role classification against reference labels. Each group reports reference support

N

, precision, recall and F1. The 481 time samples form a multilabel task. Dashes indicate absent reference classes (

N=0

); the underlying evaluator assigns zero to undefined scores.

Figure B.1: Audio-role timelines (top, global labels) and visible instrument roles (bottom, CAM3). Coloured rows are predictions; hatched rows are reference annotations. Saturated colours denote primary roles and pale colours secondary roles. The original axis label “Zeit [s]” means time in seconds. The plots illustrate this case and do not establish visual detector accuracy.

	CAM1			CAM2			CAM3			CAM4
Class	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Not blurred	.01	1.00	.03	.16	.98	.28	.01	1.00	.02	1.00	1.00	1.00
Partially blurred	.00	.00	.00	.90	.25	.39	.94	.30	.46	—	—	—
Fully blurred	—	—	—	.38	.45	.41	.44	.29	.35	—	—	—
Weighted	.00	.01	.00	.77	.35	.38	.91	.31	.45	1.00	1.00	1.00

Appendix

Table B.2: Blur-classification precision, recall and F1. Reference supports and camera-specific

n

are given in Table 2 . Dashes denote absent reference classes. Weighted summaries use class supports, including the poor results on CAM1.

Figure B.2: Per-second camera scores from predicted labels (top) and reference annotations (bottom). Background shading indicates the instantaneous highest-scoring camera, while the selector samples cameras over complete intervals. The annotation-driven scores additionally use pan/clapping cues.

Variant	Window / inclusion	$n$	Mean	Median	SD
V1	Announcement	1	80.00	80.00	—
V2	Announcement	10	8.00	8.86	3.20
V3	Announcement	10	8.00	8.86	3.21
V4	Announcement	10	8.00	8.86	3.21
V1	Applause	2	3.85	3.85	4.09
V2	Applause	2	3.85	3.85	3.99

Appendix

Table C.1: Additional shot-duration summaries (seconds). Announcement and applause rows include clipped boundary segments; “Music: complete” excludes both clipped music-boundary segments. SD is the sample standard deviation.

Field	Assessed dimension
Ovr	Overall quality and readiness for publication or broadcast.
Ed	Appropriateness of cut timing to the musical and content progression.
Tr	Coherence and smoothness of image transitions; disruptive changes.
Sh	Overall picture quality and sharpness of the selected views.
Im	Coherence and aesthetic quality of the overall impression.

Appendix

Table D.1: Dimensions assessed in the expert evaluation.

Variant	Ovr	Ed	Tr	Sh	Im
V1	3 / 3 / 3	3 / 2 / 2	3 / 5 / 2	2 / 1 / 1	2 / 3 / 3
V2	4 / 3 / 1	4 / 4 / 2	3 / 3 / 1	2 / 1 / 3	3 / 3 / 2
V3	5 / 4 / 3	5 / 4 / —	— / 5 / —	— / 1 / —	— / 4 / —
V4	4 / 4 / 1	— / 4 / 1	— / 3 / 1	2 / 1 / 2	— / 4 / 1

Appendix

Table E.1: All recorded numbers, E1 / E2 / E3. A dash means no numeric answer.

Variant	Ovr	Ed	Tr	Sh	Im
V1	3	3	3	3	3
V2	3	3	3	3	3
V3	3	2	1	1	1
V4	3	2	2	3	2

Appendix

Table E.2: Number of explicit numeric answers. Missing numeric answers remain missing even where a comment refers to another video.

Video Editing

Video editing uses AI to alter, compose, relight, or reorganize moving footage while controlling how changes persist across frames.

See all 179 papers

AVENUE: Audio-Video EditiNg Understanding and Evaluation
Aug 31, 2026Hayeon Kim, Yoojin Jang, Jaejun YooVideo Editing Benchmarks Instruction-Based Video Editing
Ulsan National Institute of Science and Technology (UNIST)
DIRECT: Video Mashup Creation via Hierarchical Multi-Agent Planning and Intent-Guided Editing
Apr 6, 2026Ke Li, Maoliang Li, Jialiang Chen +4Video Editing Instruction-Based Video Editing
School of Electronics Engineering and Computer Science, Peking University · School of Computer Science, Peking University · Huazhong University of Science and Technology
AVE-Compass: Towards Holistic Evaluation for Audio-Video Editing Abilities
Jul 17, 2026Yuqing Wen, Yukai Huang, Qianqian Xie +6Video Editing Benchmarks Agentic Video Editing
National University of Singapore · Beijing University of Posts and Telecommunications · NJU-LINK Team, Nanjing University +3

Aug 31, 2026J/K move · Enter open · S save

AVENUE: Audio-Video EditiNg Understanding and Evaluation

Hayeon Kim, Yoojin Jang, Jaejun Yoo

Ulsan National Institute of Science and Technology (UNIST)

Video Editing Benchmarks Instruction-Based Video Editing

Open paper AskPDF arXiv

Audio-video (AV) editing aims to modify audio and video content according to a target prompt. Unlike single-modality editing, AV editing requires models to infer a modality-selective edit scope from the prompt alone: determining not only what should change, but also which modality should be preserved. Faithfully evaluating such models therefore requires both (i) benchmarks that span diverse edit types and modality categories, and (ii) evaluation that is itself modality-aware and sample-specific. However, existing AV editing benchmarks provide limited coverage of edit types and modality combinations, while current evaluation systems are often modality-blind and sample-agnostic, making it difficult to assess whether models faithfully preserve the unintended modality. To address these gaps, we introduce AVENUE, Audio-Video EditiNg Understanding and Evaluation, comprising two contributions: (1) a benchmark of 1,291 source clips and 7,957 editing instructions across audio-targeted, video-targeted, and AV-coupled edit types, curated and human-verified from VGGSound; and (2) a sample-specific, modality-aware evaluation framework that specifies, for each sample, both the intended change and the content that must remain intact. We evaluate representative AV editing models spanning three editing paradigms : joint, sequential, and separate, providing the first systematic analysis of modality-selectivity across paradigms. Our findings reveal a fundamental open challenge: when editing one modality, existing models frequently induce unintended changes in the other, regardless of paradigm. AVENUE provides a benchmark and modality-aware evaluation framework to drive progress toward more controllable AV editing models. Our dataset is publicly available on Hugging Face: https://huggingface.co/datasets/AVENUE-dataset/AVENUE.

Abstract

Video Editing

Explore similar work

AVENUE: Audio-Video EditiNg Understanding and Evaluation

DIRECT: Video Mashup Creation via Hierarchical Multi-Agent Planning and Intent-Guided Editing

AVE-Compass: Towards Holistic Evaluation for Audio-Video Editing Abilities