Organizations: University of Bari Aldo Moro, Bari, Italy · Catalonia’s Telecommunications Technology Centre, Barcelona, Spain · University of Picardie Jules Verne, Amiens, France · Artificial Intelligence Venture Builder (AIVB), London, England
Music generation has advanced markedly through multimodal deep learning, enabling models to synthesize audio from text and, more recently, from images. However, existing image-conditioned systems suffer from two fundamental limitations: (i) they are typically trained on natural photographs, limiting their ability to capture the richer semantic, stylistic, and cultural content of artworks; and (ii) most rely on an image-to-text conversion stage, using language as a semantic shortcut that simplifies conditioning but prevents direct visual-to-audio learning. Motivated by these gaps, we introduce ArtSound, a large-scale multimodal dataset of 105,884 artwork-music pairs enriched with dual-modality captions, obtained by extending ArtGraph and the Free Music Archive. We further propose ArtToMus, the first framework explicitly designed for direct artwork-to-music generation, which maps digitized artworks to music without image-to-text translation or language-based semantic supervision. The framework projects visual embeddings into the conditioning space of a latent diffusion model, enabling music synthesis guided solely by visual information. Experimental results show that ArtToMus generates musically coherent and stylistically consistent outputs that reflect salient visual cues of the source artworks. While absolute alignment scores remain lower than those of text-conditioned systems-as expected given the substantially increased difficulty of removing linguistic supervision-ArtToMus achieves competitive perceptual quality and meaningful cross-modal correspondence. This work establishes direct visual-to-music generation as a distinct and challenging research direction, and provides resources that support applications in multimedia art, cultural heritage, and AI-assisted creative practice. Code and dataset will be publicly released upon acceptance.
Figures & tables
Figure 1. Unified captioning pipeline for images and audio. Images are captioned using an MLLM guided by a structured prompt; audio is captioned through a segment-level music captioner and then fused by an LLM. All captions are evaluated using the proposed ICScore and ACScore metrics; captions below threshold are regenerated with refined prompts. The figure illustrates the end-to-end captioning workflow for both images and audio, highlighting their parallel processing and convergence. It emphasizes structured prompting, segment-level fusion, and automatic quality control via iterative regeneration driven by evaluation metrics.
Statistic
Image Captions
Audio Captions
Length
CLIP-Score
PAC-Score
ICScore
Length
BERT-Score
ROUGE-1
ACScore
Minimum
234.00
0.3699
0.5165
0.4918
82.00
0.8474
0.1000
0.6234
Average
950.18
0.7821
0.8431
0.8217
411.76
0.9321
0.6894
0.8593
Maximum
2302.00
1.0000
1.0000
1.0000
1084.00
0.9940
0.9771
0.9889
> Threshold
–
–
–
78,845
–
–
–
95,837
< Threshold
–
–
–
37,630
–
–
–
10,047
Table 1. Statistics for image and audio captions in ArtSound , including caption length (expressed as number of characters), and quality metrics. For image captions: CLIP-Score, PAC-Score, and the composite ICScore . For audio captions: BERT-Score, ROUGE-1, and the composite ACScore . Values above or below the acceptance threshold (0.80) are reported.
Image
Captions
Desert Richard Pousette Dart
The artwork is a vibrant and abstract composition that appears to be a painting. It features a dense arrangement of geometric shapes, lines, and dots in a variety of colors, including red, blue, yellow, and black. The shapes and lines intersect and overlap, creating a sense of complexity and depth. The mood or emotion conveyed by the artwork is one of chaos and complexity, with the abstract nature of the shapes and lines leaving room for interpretation. The key artistic techniques and style evident in the painting are reminiscent of the Cubist style, which is characterized by the use of geometric shapes and lines to create a sense of depth and dimensionality. The use of bold, contrasting colors also adds to the overall visual impact of the artwork. The painting could be associated with the Cubist period or school, which was active in the early 20th century. This period was known for its emphasis on abstract forms and the use of geometric shapes to represent the three-dimensional world. The artwork’s abstract nature and the use of geometric shapes and lines suggest that it may be influenced by or a part of the Cubist movement.
The Hills of Thierceville Camille Pissarro
The artwork depicts a serene rural landscape. The scene is dominated by a field of tall grass, with a winding path cutting through it. The path leads the viewer’s eye towards a line of trees in the distance, suggesting a sense of depth and perspective. The sky above is filled with clouds, adding a sense of tranquility and calmness to the scene. The mood or emotion conveyed by the artwork is one of peace and solitude. The natural elements, such as the grass, trees, and sky, are rendered with a loose, impressionistic style, characterized by visible brushstrokes and a focus on the interplay of light and color. This style is often associated with the Impressionist period, which sought to capture the fleeting effects of light and color in the natural world. The artwork’s subject matter, the rural landscape, is a common theme in many art movements, including the Impressionists, who often sought to capture the beauty of the natural world. The loose, impressionistic style, combined with the tranquil subject matter, suggests that this artwork could be a representation of the Impressionist style, particularly if it was created during the late 19th or early 20th century.
Table 2. Examples of ArtSound image captions generated using LLaVA1.6-Mistral-7B.
Audio
Captions
Audio #1 (001/001556)
This is a music piece that features a low quality recording with an ambient song consisting of sustained synth pad chords, sustained synth bass, mellow piano chords, and a short plucked melody. It sounds calming, relaxing, and hypnotic—like something you would hear in yoga sessions or a spa center.
Audio #2 (001/001140)
This is a music piece that features a male voice singing the main melody, accompanied by percussion playing a simple beat in common time, with the bass playing the root notes of the chords and a piano playing arpeggiated chords in the background. The mood is romantic, and it could be played in a romantic movie. Additionally, it has elements of an instrumental rock music piece recorded in the style of blues, with a male vocalist singing melodically in the lead, the main tune played by a piano, and the electric guitar and bass guitar playing in the background. The rhythmic background consists of a slow tempo acoustic drum beat, and the atmosphere is melancholic, making it suitable for the soundtrack of a drama TV series, especially during scenes where a character is trying to break free.
Table 3. Examples of ArtSound audio captions generated using LP-MusicCaps + Qwen3.
LLM
ROUGE-1
BERT-Score
ACScore
LLaMa3.1-8B-Instruct
0.5373
0.9056
0.7951
Qwen3-8B
0.6301
0.9260
0.8372
Qwen3-14B
0.6870
0.9312
0.8579
Table 4. Average metrics computed during the LLM evaluation for audio caption fusion. Metrics include ROUGE-1, BERT-Score, and the custom ACScore . Best scores are in bold , while underlined values indicate those that are close to the best.
Pairing Approach
Image
Image Caption
Audio
Audio Caption
Similarity Values
Min
Max
Avg
Std Dev
Image–Audio
✓
—
✓
—
− 0.0216
0.5518
0.1864
0.0697
Image–Audio Caption
✓
—
—
✓
− 0.0676
0.4482
0.0644
0.0970
Image Caption–Audio
—
✓
✓
—
− 0.1011
0.1794
0.0658
0.0344
Image Caption–Audio Caption
—
✓
—
✓
0.0699
1.0000
0.7081
0.3613
Table 5. Summary of similarity statistics (minimum, maximum, mean, and standard deviation) for the pairing approaches across images, audio, image captions, and audio captions. All approaches use ImageBind embeddings for the employed modalities. Bold numbers denote the highest values, while underlined numbers mark the second highest values.
Figure 2. Violin plots of similarity score distributions for image–audio pairing strategies. Scores, computed from combinations of image, audio, and caption embeddings using ImageBind, are grouped into low ( ≤ 0.25), medium (0.25–0.6), and high ( ≥ 0.6) ranges, highlighting which model–modality pairs produce stronger or weaker alignments. This figure shows violin plots of similarity scores for different image–audio pairing strategies, illustrating how combinations of image, audio, and caption embeddings from ImageBind distribute across low, medium, and high alignment ranges. Highlights which model–modality pairs achieve stronger semantic alignment.
Figure 3. Overview of the Art2Mus architecture. Inspired by the generative paradigm of AudioLDM 2, Art2Mus reformulates the conditioning interface to enable direct artwork-to-music generation. A Visual Conditioning Extractor and an Image Aligner learn to project image embeddings into GPT-2’s LoA embedding space, establishing cross-modal alignment without textual intermediates. The pretrained AudioLDM 2 backbone is used as a frozen generative prior. GPT-2 operates as a modality-agnostic LoA translator, integrating visual conditioning within the latent diffusion process. Music is encoded into mel-spectrogram latents by the Latent Extractor and synthesized into audio by the LoA-to-Audio Generator . The figure illustrates ArtToMus architecture, explaining how visual information is injected into a pretrained text-to-audio diffusion backbone with minimal architectural changes. It highlights the role of GPT-2 as a shared intermediate representation that aligns visual, textual, and audio modalities while keeping most components frozen.
Figure 4. The Visual Conditioning Extractor and Image Aligner modules. The figure focuses on the visual conditioning pathway, showing how image embeddings are normalized and projected into a common token-based representation. It clarifies how different vision encoders are handled to ensure compatibility with GPT-2.
Figure 5. Distribution of artworks by music genre, segmented by artistic style. Abstract Art , Abstract Expressionism , Art Nouveau (Modern) , and Expressionism appear prominently across most genres. Conversely, Early Renaissance and Symbolism are less frequent. This figure illustrates the relationship between artistic styles and associated music genres, revealing dominant and underrepresented styles across the dataset. It helps contextualize stylistic biases and coverage in the artwork–music pairing.
Image Encoder
Tok/Img
FAD ↓
KL-Div ↓
IBScArtw–GeMus↑
IBScGtMus–GeMus↑
ImageBind
4
19
0.00266
0.05716
0.19363
6
19
0.00273
0.05693
0.21398
8
19
0.00284
0.05946
0.21341
10
19
0.00386
0.05288
0.18045
16
19
0.00279
0.05986
0.21661
CLIP
4
22
0.00320
0.05185
0.18021
Table 6. Average training metrics for different visual encoders and token counts. Arrows indicate whether higher ( ↑ ) or lower ( ↓ ) values are desirable. Best results per encoder are in bold ; second-best are underlined . Abbreviations: Tok/Img – Tokens per Image; Artw – Artwork; GeMus – Generated Music; GtMus – Ground-Truth Music.
Model
Emb. Model
Train Size (hrs)
Tok/Img
FAD ↓
KL-Div ↓
IBScArtw–GeMus↑
IBScGtMus–GeMus↑
Art2Mus
ImageBind
≈ 2.6
16
13
0.00296
0.04671
0.20368
Art2Mus
CLIP
6
15
0.00374
0.04173
0.16716
AudioLDM 2
–
≈ 1,150,000
–
14
0.00425
0.09822
0.17412
AudioLDM 2-Music
–
≈ 665,000
–
15
0.08449
0.18683
0.18683
Mozart’s Touch
–
≈ 20,000
–
19
0.00238
0.08348
0.21629
Table 7. Objective evaluation results for Art2Mus and baselines: AudioLDM 2 (sound effects, human speech, and music), AudioLDM 2-Music (music only), and Mozart’s Touch (music only). Arrows indicate whether higher ( ↑ ) or lower ( ↓ ) values are preferable. Best values per row block are in bold ; second-best are underlined . Abbreviations: Emb. Model – Image Embedding Model; Tok/Img – Tokens per Image; Artw – Artwork; GeMus – Generated Music; GtMus – Ground-Truth Music.
ID
Question
Scale
Q1
How well does the music sound?
1–5
Q2
How expressive or emotionally engaging is the music?
1–5
Q3
How relevant is the music to the artwork?
1–5
Q4
How well does the music reflect the mood/content of the artwork?
1–5
Table 8. Questions used for the subjective evaluation. Participants scored each generated track on audio quality, expressiveness, pertinence, and alignment.
Model
Audio Quality (Q1) ↑
Expressiveness (Q2) ↑
Pertinence (Q3) ↑
Alignment (Q4) ↑
Art2Mus (CLIP only artws)
2.72
3.10
2.91
2.88
Art2Mus (ImageBind only artws)
2.74
3.02
2.87
2.79
Table 9. Subjective evaluation results (mean scores across 15 participants). Each track was evaluated on audio quality, expressiveness, pertinence to the artwork, and alignment with the artwork. For each model, we computed each participant’s average rating and averaged these across all participants to obtain the final score for each question. Best values are in bold .
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Image
Captions
The Hills of Thierceville Camille Pissarro
Qwen2.5-VL: This artwork depicts a serene rural landscape, likely painted in an impressionistic style. The scene features rolling hills, a meandering stream, and scattered trees under a soft sky. The use of light and color suggests a tranquil setting. The mood conveyed is one of peace and calmness. The artist employs loose brushstrokes and a palette dominated by earthy tones, which contribute to the overall impressionistic feel. The light appears to be natural. The style and technique suggest that this work could belong to the Impressionist movement. This period was characterized by a focus on capturing the fleeting effects of light and atmosphere, often through the use of vibrant colors and visible brushstrokes. Qwen2.5-Omni: Well, this artwork seems to be a landscape painting. The subject matter is a field with a winding path or stream running through it. There are some trees in the background.The mood it conveys is quite peaceful and serene. You can almost feel the calmness of the scene. The artistic techniques used might include loose brushstrokes, which gives it a sort of impressionistic feel. The colors are soft and blended together, creating a harmonious look.As for the style, it could be Impressionism. And the possible artistic period would be the late 19th century. If you want to know more about this painting or other artworks, feel free to ask me. LAVIS: a painting of a grassy field with a stream running through it.
Appendix
Table 10. Example of artwork captions generated using discarded models . LAVIS does not require a prompt. All remaining models were given the following instructions: “You are an art expert. Provide a concise description of this artwork, addressing its subject matter, the mood or emotion it conveys, key artistic techniques and style, and its possible artistic period or school”.
Audio
Captions
Audio #1 (001/001556)
Qwen2-Audio: The provided audio clip is an experimental drone piece within the electronic genre. It features synthesizers, bass, guitar, and drums. The instrumentation contributes to evoking a sense of space and timelessness. The structure of the piece adheres to a repetitive pattern. This approach allows each instrument’s timbre to resonate deeply. Qwen2.5-Omni: Well, from what I can tell, the music genre seems to be ambient. You can hear a lot of synthesizers in there. There are also some electronic pads. The overall sound is quite dreamy and atmospheric. It has a slow tempo which adds to that sense of calm and spaciousness. It’s like you’re floating in a dreamy, otherworldly place. What do you think? Do you have any other music clips you want me to analyze? Human: What what Musilingo: The audio clip is a recording of a live performance of a song. The genre of the song is a blend of rock and pop. The musical instruments that can be heard in the recording are a guitar, bass guitar, and a synthesizer. The overall sound of the song is energetic. The style of the song is upbeat and lively. The recording is of high quality.
Appendix
Table 11. Example of audio captions generated using discarded models . All models were given the following prompt: “You are a music expert. Analyze the provided audio clip and respond with a precise description (approximately 200 words). Your response should include: ( i ) the identified music genre, ( ii ) a list of discernible musical instruments, and ( iii ) a brief description of the overall sound and style. Do not ask any questions or seek further information”.
Approach
CLIP Embs. Sims.
Min
Max
Avg
Std Dev
Artwork-Spectrogram
0.2466
0.7183
0.4355
0.0783
Recolored Artwork-Spectrogram
0.2100
0.9200
0.7045
0.1422
Artwork-Music Image
0.1274
0.7759
0.4578
0.0931
Appendix
Table 12. Summary statistics (minimum, maximum, mean, and standard deviation) of cosine-similarity scores between CLIP embeddings for each pairing method (using spectrograms or audio-generated images). Best results appear in bold , and second best are underlined .
Figure 6. Comparison between artworks and their recolored versions using the Inferno colormap. Each subfigure shows the original artwork (left) and its recolored version (right). The figure demonstrates the effect of recoloring artworks using the Inferno colormap. Each subfigure pairs the original artwork (left) with its recolored version (right), highlighting visual transformations.
Figure 7. Examples of images generated by Stable Diffusion 3.5 based on the generated audio captions. The figure shows examples of images generated by Stable Diffusion 3.5 from generated audio captions, demonstrating how audio-driven captions can guide visual synthesis.
Figure 8. Distribution of artworks by music genre, segmented by artistic style. Romanticism and Expressionism are prevalent across most genres, whereas Minimalism and Abstract art occur less frequently. The figure illustrates the distribution of artworks across music genres segmented by artistic style, highlighting that Romanticism and Expressionism dominate most genres, while Minimalism and Abstract Art are less frequent.
Recent advances in multimodal audio generation have enabled music synthesis from text, visual cues, and other high-level conditions. However, most systems are designed for a single operating mode: either generating music without a reference mixture or extracting a target source from an existing mixture. This fixed-task design limits their use when different combinations of text, visual, and mixture inputs are available. To address this gap, we propose MAGE, a modality-agnostic framework for conditional music generation and mixture-grounded target-source extraction within a shared continuous latent space. Our approach introduces three key components. First, a Controlled Multimodal FluxFormer models the conditional flow from noise to a target audio latent, enabling the same backbone to operate with or without a mixture condition. Second, Audio-Visual Nexus Alignment maps frame-level visual features onto the audio latent sequence, allowing visual evidence to condition the generation process at the audio-token level. Third, a cross-gated modulation mechanism uses the aligned visual representation to regulate intermediate audio features, while text provides separate semantic guidance. We further train MAGE with dynamic modality masking, exposing the same model to text-only, visual-only, joint text-visual, mixture-conditioned, and unconditional configurations. Experiments on the MUSIC benchmark evaluate MAGE under separate protocols for mixture-free generation and mixture-grounded target-source extraction. The results show that MAGE provides a shared conditioning interface across both settings, and that the proposed alignment and gating components improve interference suppression in the extraction task.
Muhammad Usama Saleem, Tejasvi Ravi, Tianyu Xu +4
Google · University of North Carolina at Charlotte
As an important subfield of cross-modal generation, synthesizing static visual content in the form of images from audio, namely audio-to-image (A2I) generation, has attracted increasing research attention in recent years. Nevertheless, despite the remarkable visual quality of modern text-to-image (T2I) models, the performance of A2I remains fundamentally limited by traditional datasets, which often lack both high-fidelity images and precise cross-modal alignment. As a result, existing methods still struggle to achieve high-quality audio-to-image generation through finetuning strong T2I models, thereby constraining practical applications in this area. Motivated by this gap, we introduce A2I-Set, a unified, high-quality tri-modal dataset consisting of 323K paired audio, images, and detailed text captions, specifically designed for audio-visual research, including audio-conditioned image generation. Besides, we developed a new mixed-source test set for the A2I task through human supervision. We further propose an A2I model, AudioCanvas, fine-tuned on our A2I-Set. Experiments show that AudioCanvas achieves more visually expressive as well as cross-modal alignment results that generally outperforming existing approaches. Our dataset and source code are available at https://github.com/gdx012/A2I-Generation.
Dongxu Ge, Shansong Liu, Cheng Gong +3
University of Science and Technology of China Hefei, China · Institute of Artificial Intelligence, China Telecom (TeleAI) Beijing, China · Northwest Polytechnical University Xi’an, China
Text-to-music generation has advanced rapidly, with modern autoregressive and diffusion-based models producing convincing music from natural-language prompts. However, much of this progress relies on large-scale training data and external pretraining, making it difficult to isolate which design choices remain effective when data and pretraining are controlled. We study this setting using a Diffusion Transformer backbone with lyric and timbre conditioning, adapted to an instrumental-only text-to-music task in which the auxiliary lyric and timbre branches receive only degenerate conditioning signals. Through controlled ablations, we find that models retrained without these branches score lower across AudioBox aesthetics, LLM-as-judge, and human MOS, and that reinvesting the saved parameters as additional DiT depth recovers only marginally. This suggests the auxiliary branches may act as training-time architectural anchors whose contribution goes beyond their explicit conditioning content. We validate the same model through comparisons with external instrumental baselines and through our submission to the ICME 2026 Academic Text-to-Music (ATTM) Grand Challenge, where our Performance submission ranked first under both the objective metrics and the subsequent organizer-administered MOS over 35 raters, attaining the highest overall MOS across all challenge submissions, while our Efficiency submission was a finalist that tied for second under the objective metrics.
Junyoung Koh
Department of Artificial Intelligence, Yonsei University MAAP KRAFTON Seoul, Republic of Korea