Exploring modality-agnostic representations for music classification Article Swipe

MAKE IMPACT

Ho-Hsiang Wu , Magdalena Fuentes , Juan Pablo Bello · YOU? · · 2021 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2106.01149

Music information is often conveyed or recorded across multiple data modalities including but not limited to audio, images, text and scores. However, music information retrieval research has almost exclusively focused on single modality recognition, requiring development of separate models for each modality. Some multi-modal works require multiple coexisting modalities given to the model as inputs, constraining the use of these models to the few cases where data from all modalities are available. To the best of our knowledge, no existing model has the ability to take inputs from varying modalities, e.g. images or sounds, and classify them into unified music categories. We explore the use of cross-modal retrieval as a pretext task to learn modality-agnostic representations, which can then be used as inputs to classifiers that are independent of modality. We select instrument classification as an example task for our study as both visual and audio components provide relevant semantic information. We train music instrument classifiers that can take both images or sounds as input, and perform comparably to sound-only or image-only classifiers. Furthermore, we explore the case when there is limited labeled data for a given modality, and the impact in performance by using labeled data from other modalities. We are able to achieve almost 70% of best performing system in a zero-shot setting. We provide a detailed analysis of experimental results to understand the potential and limitations of the approach, and discuss future steps towards modality-agnostic classifiers.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2106.01149

PDF

https://arxiv.org/pdf/2106.01149

OA Status

green

Cited By

4

References

31

Related Works

10

OpenAlex ID

https://openalex.org/W3172444685

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W3172444685

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2106.01149

Digital Object Identifier

Title

Exploring modality-agnostic representations for music classification

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2021

Year of publication

Publication date

2021-06-02

Full publication date if available

Authors

Ho-Hsiang Wu, Magdalena Fuentes, Juan Pablo Bello

List of authors in order

Landing page

https://arxiv.org/abs/2106.01149

Publisher landing page

PDF URL

https://arxiv.org/pdf/2106.01149

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2106.01149

Direct OA link when available

Concepts

Modality (human–computer interaction), Computer science, Psychology, Natural language processing, Artificial intelligence

Top concepts (fields/topics) attached by OpenAlex

Cited by

4

Total citation count in OpenAlex

Citations by year (recent)

2024: 1, 2023: 3

Per-year citation counts (last 5 years)

References (count)

31

Number of works referenced by this work

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload