Score-based multimodal music retrieval requires representations that are both discriminative within each modality and compatible across audio recordings, sheet music images, and symbolic scores. This paper investigates the effectiveness of parameter-efficient fine-tuning (PEFT) for adapting pretrained music encoders and examines whether modality-specific improvements translate into better multimodal representations. We compare full fine-tuning, LoRA, and QLoRA across nine pretrained audio, image, and symbolic score encoders on TriScore and the Musical Theme Dataset. Encoders are first evaluated and selected through supervised unimodal classification, using frozen-encoder linear probing as a reference. We then introduce a controlled 2x2 design crossing encoder adaptation with task-specific shared projection, comparing frozen and adapted representations before and after projection. LoRA or QLoRA is selected in 14 of 18 modality-target combinations, frequently matching or outperforming full fine-tuning while optimizing substantially fewer parameters. However, these unimodal gains do not transfer uniformly to multimodal classification. Adaptation and projection are complementary for composer classification on TriScore, largely redundant for TriScore instrument and the Musical Theme Dataset instrument and ensemble tasks, and partially interfering for composer classification on the Musical Theme Dataset. The final adapted-and-projected title representation on TriScore supports piece-level retrieval, reaching average Top-5 and Top-10 accuracies of 55.32 and 64.68, respectively. Cross-modal exclusion and neighborhood analyses further reveal strong modality-specific local structure in the projected spaces. Overall, PEFT provides an effective and computationally efficient alternative to full fine-tuning, but improved unimodal discrimination alone does not guarantee better multimodal organization.
Figures & tables
Figure 1: High-level overview of the proposed evaluation pipeline. In Stage I, modality-specific audio, image, and symbolic score encoders are adapted and selected through unimodal supervision. In Stage II, their representations are mapped through a task-specific shared projection and evaluated in three downstream settings: category-level classification, piece-level retrieval, and cross-modal analysis.
Figure 2: Overview of the proposed two-stage methodology. Stage I adapts each candidate pretrained encoder E independently through supervised unimodal classification. Frozen-encoder linear probing is used as a reference, while full fine-tuning, LoRA, and QLoRA are compared as adaptation strategies; the best-performing encoder–adaptation configuration is selected using validation macro F1. Stage II extracts representations from both the selected adapted encoder Eϕadaptm for each modality m .
Property
TriScore
MTD
Aligned triplets
2,287
2,068
Composers
17
30
Instrument categories
8
13
Ensemble categories
–
10
Piece-level retrieval
111 works
–
Musical content
Polyphonic excerpts
Monophonic themes
Table 1: Summary of the multimodal datasets used in this work. Sample counts refer to aligned triplets, each comprising audio, symbolic score, and sheet music image modalities.
Modality
Encoder
HuggingFace model ID
Pretraining domain
Hidden size
Audio
MERT [ 29 ]
m-a-p/MERT-v1-330M
Music audio
1,024
Whisper [ 35 ]
openai/whisper-medium
Multilingual speech
1,024
Wav2Vec2 [ 27 ]
ALM/wav2vec2-large-audioset
General audio
1,024
Image
ViT [ 14 ]
google/vit-base-patch16-224
General vision
768
DiT [ 28 ]
microsoft/dit-base-finetuned-rvlcdip
Document images
768
PaliGemma [ 38 ]
google/paligemma2-3b-pt-896
Vision–language
4,096
Table 2: Summary of the nine pretrained encoders evaluated in this work. Hidden size denotes the dimensionality of the extracted representation.
Configuration
Rank r
Scaling factor α
Dropout
1
2
4 ( 2r )
0.1
2
2
8 ( 4r )
0.1
3
4
8 ( 2r )
0.1
4
4
8 ( 2r )
0.0
5
4
16 ( 4r )
0.1
6
8
16 ( 2r )
0.1
Table 3: Hyperparameter configurations evaluated for LoRA and QLoRA . The same search space is used for both methods.
Figure 3: Validation macro F1 for all unimodal adaptation configurations on TriScore . These scores are used for encoder selection and do not correspond to the final multimodal test evaluation. Each row corresponds to a target label and each column to a modality. Lines represent the three encoders evaluated for each modality, starting from the frozen-encoder LP baseline and followed by the full fine-tuning and LoRA / QLoRA configurations. Stars indicate the best encoder–adaptation configuration within each panel.
Figure 4: Validation macro F1 for all unimodal adaptation configurations on MTD . These scores are used for encoder selection and do not correspond to the final multimodal test evaluation. Each row corresponds to a target label and each column to a modality. Lines represent the three encoders evaluated for each modality, starting from the frozen-encoder LP baseline and followed by the full fine-tuning and LoRA / QLoRA configurations. Stars indicate the best encoder–adaptation configuration within each panel.
Modality
Label
Encoder
Method
r
α
Trainable Params
Macro F1
Audio
Instrument
MERT
QLoRA
4
2r
1.6M
0.832 ( ↑ 41.6pp )
Whisper
Full
–
–
306M
0.741 ( ↑ 21.9pp )
Wav2Vec2
LoRA
2
2r
1.4M
0.860 ( ↑ 46.3pp )
Composer
MERT
Full
–
–
316M
0.891 ( ↑ 66.1pp )
Whisper
Full
–
–
306M
0.863 ( ↑ 53.4pp )
Wav2Vec2
LoRA
2
2r
1.4M
0.847 ( ↑ 67.4pp )
Table 4: Best validation configuration per encoder and target label on TriScore . Method indicates the selected adaptation strategy (full fine-tuning, LoRA , or QLoRA ); r and α report the adapter rank and scaling factor and are not applicable to full fine-tuning; Trainable Params reports the number of optimized parameters; and Macro F1 reports validation performance, with the gain over the linear-probing baseline shown in parentheses (in pp ). The best encoder–adaptation configuration for each modality and target is highlighted in bold.
Modality
Label
Encoder
Method
r
α
Trainable Params
Macro F1
Audio
Instrument
MERT
Full
–
–
316M
0.482 ( ↑ 23.6pp )
Whisper
QLoRA
8
4r
1.4M
0.433 ( ↑ 19.3pp )
Wav2Vec2
QLoRA
2
2r
1.4M
0.484 ( ↑ 23.0pp )
Composer
MERT
Full
–
–
316M
0.172 ( ↑ 8.2pp )
Whisper
QLoRA
2
2r
1.4M
0.259 ( ↑ 19.5pp )
Wav2Vec2
LoRA
4
2r
1.4M
0.221 ( ↑ 15.6pp )
Table 5: Best validation configuration per encoder and target label on MTD . Method indicates the selected adaptation strategy (full fine-tuning, LoRA , or QLoRA ); r and α report the adapter rank and scaling factor and are not applicable to full fine-tuning; Trainable Params reports the number of optimized parameters; and Macro F1 reports validation performance, with the gain over the linear-probing baseline shown in parentheses (in pp ). The best encoder–adaptation configuration for each modality and target is highlighted in bold. In the case of tied validation macro F1, the configuration with fewer trainable parameters is selected.
Frozen ( ϕ0 )
Adapted ( ϕadapt )
Δadapt
Dataset
Label
Raw ( Z )
Projected ( H )
Δprojection
Raw ( Z )
Projected ( H )
Δprojection
Raw ( Z )
Projected ( H )
TriScore
Composer
42.09
50.63
↑ 8.54pp
61.55
74.99
↑ 13.44pp
↑ 19.46pp
↑ 24.36pp
Instrument
59.72
74.39
↑ 14.67pp
69.26
70.03
↑ 0.77pp
↑ 9.54pp
↓ 4.36pp
MTD
Instrument
15.38
17.28
↑ 1.90pp
16.15
16.60
↑ 0.45pp
↑ 0.77pp
↓ 0.68pp
Ensemble
11.91
16.24
↑ 4.33pp
16.64
16.29
↓ 0.35pp
↑ 4.73pp
↑ 0.05pp
Composer
4.38
7.34
↑ 2.96pp
11.41
8.58
↓ 2.83pp
↑ 7.03pp
↑ 1.24pp
Table 6: Controlled 2×2 factorial evaluation crossing encoder adaptation (frozen vs. adapted) with shared projection (raw vs. projected). Scores are macro F1 values in pp , combined-gallery average. Δadapt reports the effect of adaptation without and with projection, while Δprojection reports the effect of projection without and with adaptation.
Figure 5: t-SNE visualization of the four factorial conditions for TriScore composer: {F1eϕ}(e,ϕ)∈S×Φ={F1zϕ0,F1hϕ0,F1zϕadapt,F1hϕadapt}. . Colors indicate composer class; marker shapes indicate modality.
Figure 6: t-SNE visualization of the frozen–projected ((ϕ0,H) and adapted–projected ((ϕadapt,z) conditions for TriScore instrument. Colors indicate instrument class; marker shapes indicate modality.
Figure 7: t-SNE visualization of the adapted–raw (ϕadapt,Z) and adapted–projected (ϕadapt,Z) conditions for MTD composer. Colors indicate composer class; marker shapes indicate modality.
Dataset
Label
Audio query
Image query
Score query
Avg.
TriScore
Composer
83.04
47.22
94.70
74.99
Instrument
80.40
47.24
82.46
70.03
MTD
Instrument
29.11
9.68
11.01
16.60
Ensemble
27.36
10.80
10.73
16.29
Composer
10.03
7.17
8.54
8.58
Table 7: Classification performance in the task-specific projected spaces condition (ϕadapt,H)) . Scores are macro F1 values in pp . A separate projected space is evaluated for each dataset and target label. The gallery contains all modalities, and columns indicate the query modality.
Metric
Audio query
Image query
Score query
Avg.
Top-5
61.62
34.59
69.73
55.32
Top-10
69.19
45.95
78.92
64.68
Table 8: Piece-level retrieval on TriScore using the title label. Scores are Top- N accuracies in pp in the title-specific projected space.
Dataset
Label
Audio query
Image query
Score query
Avg.
TriScore
Composer
77.47
17.54
89.25
61.42
Instrument
67.77
35.97
68.49
57.41
MTD
Instrument
14.39
8.90
22.12
15.14
Ensemble
20.30
10.30
14.75
15.11
Composer
2.49
7.69
8.00
6.06
Table 9: Cross-modal exclusion classification in the task-specific projected spaces. Scores are macro F1 values in pp . For each query modality, samples from that same modality are removed from the gallery.
Figure 8: Neighborhood modality composition in the task-specific projected spaces. Each heatmap reports, for a given query modality (rows), the proportion of its five nearest neighbors belonging to each neighbor modality (columns). (a) All three modalities are available in the gallery; the diagonal is the same-modality proportion. (b) The query modality is excluded from the gallery; the diagonal cells are not defined and are shown in grey.
Brain encoding models aim to predict neural responses to stimuli. The most widely used approach fits readouts from frozen pretrained backbones. Using neural data to train large models from scratch or fully fine-tune them after task-pretraining has proven difficult because neural data is costly and limited. We show that parameter-efficient fine-tuning (PEFT) enables us to effectively adapt very large pretrained backbones for brain encoding with a small number of trainable parameters. To make it computationally practical, we exploit the backbone's causal structure to predict multiple timepoints in a single pass instead of using sliding windows, significantly reducing feature-extraction cost. This enables efficient fine-tuning of multimodal backbones with up to 104B parameters, the largest model adapted for brain encoding to our knowledge. Trained with a low-rank adapter (LoRA) and a lightweight readout, our approach consistently improves brain encoding performance across nine diverse multimodal backbones. On the same Qwen3-Omni backbone as the prior state-of-the-art (SoTA), our method BrainLoRA achieves the best published performance on the Algonauts 2025 Challenge. We also show the adapted multimodal backbones remain interpretable: modality attribution recovers the canonical sensory topography that prior work often obtains by construction from separate unimodal encoders, and neural adaptation increases the contribution of audio inputs and cross-modal mixing within the backbone. To scale out the number of adapted models, we develop a multi-LoRA kernel that accelerates joint training of multiple LoRAs on a shared backbone. This enables an ensemble of 200+ adapted models that ranks first on the public Algonauts post-challenge leaderboard at the time of writing. Together, our results highlight PEFT as an efficient and powerful paradigm for brain encoding.
Zhuofan Josh Ying, Guanglong Sun, Nolan Ho +6
Columbia University · Mind Lab · Tsinghua University +1
We empirically analyze two recently proposed encoder-free multimodal LLMs, Gemma-4 12B and Inkling-Small. Unlike previous multimodal LLMs, these models do not use modality-specific encoders to extract tokenized embeddings representing audio and image input; instead, they directly feed windows of samples, pixels, or spectral features to the Transformer language model. We analyze whether the early layers of these models learn to function similarly to dedicated audio and visual encoders. We find that these early layers do in fact act as ``implicit encoders,'' encoding audio and visual inputs in a way that is largely invariant to their cross-modal context. Furthermore, when performing multimodal reasoning tasks, we show that independently encoding the modalities throughout these early model layers achieves the same accuracy as jointly encoding them from the first layer onwards. Finally, we use layer-wise probing to analyze the salient information captured by the implicit encoders and find their behavior to be similar to several popular encoder models.
David Harwath, Desh Raj, Xavier Velez +5
The University of Texas at Austin · NVIDIA · Johns Hopkins University +2
Parameter-Efficient Fine-Tuning (PEFT) methods provide a streamlined and efficient tool for adapting large models to domain-specific multimodal downstream tasks. Although these methods proved their tangible effects in practice, their principal aspects remain under-explored. Therefore we remain curious about the underlying generalization mechanisms in various PEFT methods and how they can be further enhanced. In this paper, we reveal the flatness preference widely present in various PEFTs, where a small fraction of sharp dimensions dominates the generalization of PEFT. This finding suggests an appealing possibility: we may be satisfied with a better generalization by merely attending to this small fraction of sharp dimensions instead of all of them. Furthermore, we propose Flatness Preference Optimization (FlatPO) to flatten these key sharpness dimensions, leading various PEFTs toward better generalization. Extensive experiments demonstrate the effectiveness of our findings and the proposed method. Code is available at https://github.com/Can-Lin/FlatPO.
Yifan Zhu, Can Lin, Hangjie Yuan +4
School of Computer Sciences, Beijing University of Posts and Telecommunications, Beijing 100876, China · College of Computer Science and Technology, Zhejiang University, Zhejiang 310027, China · ETH Zürich, Switzerland