VioTech: Frame-Level Violin Playing Technique Detection with Multi-Scale Mixture-of-Experts
Authors: Ting-Kang Wang, Vincent K. M. Cheung, Nan-Tien Lai, Li Su
Organizations: Institute of Information Science, Academia Sinica, Taipei, Taiwan · Original Content Center, Gamania Inc., Taipei, Taiwan · Sony Computer Science Laboratories, Inc., Tokyo, Japan · National Taiwan University, Taipei, Taiwan
Playing technique is crucial for expressive violin performance, yet few violin technique detection models exist. Current systems mainly rely on note-level pipelines that first estimate note boundaries and then classify techniques. This is especially problematic for bowed string instruments such as the violin, where soft onsets and continuous bow-string interaction render note segmentation unreliable. Here, we propose VioTech, a frame-level violin technique detection framework that predicts techniques directly at 10 ms resolution without explicit note segmentation. To handle the large temporal-scale variation among techniques, VioTech combines a frozen MERT encoder with a multi-scale Mixture-of-Experts (MoE) architecture and task-specific routing for tonal, articulation, and legato-related predictions. We also introduce VioTech120, the first manually annotated dataset of violin playing techniques constructed from real solo performance recordings, as well as an accompanying note-labeling user interface. Results demonstrate that VioTech achieved consistent improvements over prior methods and significantly enhanced cross-dataset generalization, establishing a new state-of-the-art in violin technique-aware transcription.
Figures & tables
Figure 1: Overview of proposed violin technique transcription system with multi-scale mixture of experts.
Figure 2: Proposed multi-scale mixture-of-experts model architecture for frame-level violin technique detection.
Figure 3: (a) Configuration and corresponding temporal context window size of our multi-scale mixture-of-experts with convolution kernel size K and dilation size D . (b) Architecture of a single expert.
Figure 4: We provide a user-friendly annotation interface with fine-grained temporal alignment and control between MIDI and audio to enable data extension beyond playing technique and include additional fingering information.
Figure 5
Model
Bach10
URMP
P
R
F1
F1 w/o
P
R
F1
F1 w/o
VioPTT [ 1 ]
48.7
53.0
50.7
62.5
44.4
44.9
44.6
54.9
VioTech w/o MoE
54.8
58.0
56.1
65.2
39.1
57.5
46.3
59.7
VioTech (Ours)
60.9
65.0
62.8
79.1
45.6
60.4
51.9
69.3
Table 3: Note transcription performance on Bach10 [ 22 ] and URMP [ 23 ] datasets. P = precision, R = recall, F1 w/o = F1-score without note offsets.
Figure 6: Technique-wise F1-score versus number of experts evaluated on VioTech120 test and RWC [ 21 ] datasets. Stars indicate highest mean performance over three iterations and error bars indicate standard deviation. pp denotes percentage-point improvement over the no-MoE baseline.
Figure 7: Gate distribution across experts for each violin playing technique.
Figure 8: UMAP visualization of frame-level embeddings of each playing technique from the RWC dataset [ 21 ] .
Neural synthesis for musical instruments has the potential to revolutionize current practices that use concatenative synthesis and a sample library. However, most research focused on piano synthesis and expressive performance generation; little work has been done on continuously articulated instruments like the violin, let alone rendering them with playing techniques and dynamics. We present VIOLET, a latent-diffusion framework for controllable violin synthesis, which uses a Diffusion Transformer (DiT) with rectified flow to synthesize high-fidelity audio from MIDI notes, playing techniques, and continuous dynamics. To train VIOLET, in addition to using a few existing datasets, we curate a new dataset named CSV-TD, which contains 39 h of 48 kHz synthetic audio and time-aligned annotations of MIDI notes, note-level techniques, and continuous dynamics curves. Objective and subjective evaluations show that VIOLET synthesizes violin performances with high technique adherence, accurate pitch and timing alignment, and good dynamics control. It outperforms the current state-of-the-art neural violin synthesis system and approaches a top commercial virtual instrument in terms of technique clarity, naturalness, and dynamics following.
Baotong Tian, Cynthia Lu, Vincent K. M. Cheung +3
University of Rochester, USA · Sony Computer Science Laboratories, Tokyo, Japan · National Taiwan University, Taiwan +1
Learning the violin is harder than it looks. Unlike piano keys or guitar frets, the violin neck has no markings at all, so a beginner cannot tell by looking where to place each finger. MusicSynth is an open-source web tool that tries to fix that: user uploads a photo of any violin sheet music (or a digital score file), and the system automatically produces a video showing a violin fingerboard with each note highlighted at the right moment -- no software to install, no manual note entry required. The system connects three existing open-source tools into one pipeline: an optical music recognition (OMR) library reads the notes from the uploaded image, a MusicXML parser extracts timing information from digital scores, and a video renderer draws the fingerboard frame by frame. The only part built from scratch is the lookup table that maps each musical note to a string and finger position on the violin. Tested across 110 public-domain violin scores, MusicSynth correctly identified 91.2,% of notes in clean printed music and assigned the right finger position 99.1,% of the time when given a digital score file. To the author's knowledge, no freely available tool currently turns a sheet music image into an animated violin fingerboard tutorial automatically and in a single browser-based step.
Audio-based piano transcription performs well on onset, pitch, and velocity, but the sustain pedal lets sound persist long after key release, so audio systems predict pedal-extended offsets rather than physical key release. Yet existing Visual Piano Transcription (VPT) systems focus on onset detection from short video windows, offset accuracy lags onset by a wide margin, and note-level velocity has not been reported. To address these gaps, we present V2N (Video to Notes), the first complete VPT system: a shared temporal backbone feeds task-specific heads for onset, offset, key hold, and velocity, jointly trained with per-frame supervision rather than only at the window center. Ablations show that multi-task supervision enables offset and velocity prediction while improving onset accuracy; longer temporal context yields further improvements. V2N sets new state-of-the-art results on PianoVAM and R3.
Yonghyun Kim, Hoyeol Sohn, Juhan Nam +1
Music Informatics Group, Georgia Institute of Technology, USA · Graduate School of Culture Technology, KAIST, South Korea