public.loading
MERGE: Music Emotion Recognition - Next Generation

MERGE: Music Emotion Recognition - Next Generation

Synopsis

  • Context
    Current digital music repositories are enormous and continue to grow. More advanced, flexible and user-friendly search mechanisms adapted to individual user requirements are urgent. This has led to increased awareness in the Music Information Retrieval (MIR) area.
    Within MIR, Music Emotion Recognition (MER) has emerged as a significant sub-field. In fact, “music’s preeminent functions are social and psychological”, and so “the most useful retrieval indexes are those that facilitate searching in conformity with such social and psychological functions. Typically, such indexes will focus on stylistic, mood, and similarity information” (David Huron).
    Google, Pandora, Spotify and Apple have MIR/MER research agendas, with existing commercial applications. However, services like the Spotify API which purport to categorise emotion have been shown not to be state of the art. To the best of our knowledge, there is no Portuguese industrial entity working on the field; however, international players may benefit from the outcomes of this research, e.g., Pandora, which endorses this proposal (attached letter).
    MER research promises significant social, economic, and cultural repercussions, as well as substantial scientific impact. Several open and complex research problems exist in this multidisciplinary field, touching upon audio signal processing, natural language processing, feature engineering and machine learning.
  • Problem
    Current MER solutions, both in the audio and lyrics domains, are still unable to solve fundamental problems, such as the classification of static samples (i.e., with uniform emotion) with a single label into a few emotion classes (e.g., 4-5). This is supported by existing studies and the stagnation in the Music Information Retrieval Evaluation eXchange (MIREX) Audio Mood Classification (AMC) task (music-ir.org/mirex/), where accuracy stabilized at ~69%.
    Nevertheless, more complex problems have been addressed, e.g., music emotion variation detection (MEVD), with even lower reported results (~20%).
    We believe that, at the current stage of MER, rather than tackling complex problems, effort should be refocused on simpler static single-label classification, with a low number of classes, and exploiting the combination of audio and lyrics via a bi-modal approach.
  • Current Research Gaps
    There exists a significant corpus of research on MER, ranging from fundamental problems of static single-label classification to higher-level work on multi-label classification or MEVD.
    However, current results are low and limited by a “glass ceiling”. In previous work, we demonstrated this to be partly due to two core problems: i) lack of emotionally-relevant features; ii) absence of public, sizeable and quality datasets. We created public robust (although somewhat small) MER datasets and emotionally-relevant audio and lyrics features (e.g., music expressivity and text stylistic features, such as vibrato or slang). This strategy was successful and led to ~10% increase in classification performance,
    and, thus, should be further pursued.
    The lack of robust and sizeable datasets impacts deep learning (DL) research on MER as well. While DL is widespread in MIR, MER datasets are not sufficiently large or well-annotated to train fully end-to-end models.
  • Objectives and Approaches
    We seek to advance MER research by following an explicitly bi-modal approach which models lyrics and audio simultaneously, and define the following scientific objectives:
    1) Feature engineering/learning. To devise meaningful MER features (both handcrafted and via feature/deep learning) simultaneously in the audio and lyrics domains following a bi-modal approach.
    2) Robust public datasets. To collect and robustly annotate data for MER based on audio + lyrics and to release it to the MIR community.
    3) Static MER and MEVD. To combine these contributions to advance both static MER and MEVD, addressing bi-modal approaches and dimensional MER.
    As a technological objective, we will develop two MER software applications (a standalone and a web app) to demonstrate our scientific innovations.

Funding

FCT: PTDC/CCI-COM/3171/2021

Total budget

€ 195 892.00

Keywords

Music Emotion Recognition, Music Information Retrieval, Feature Engineering, Audio Signal Processing, Natural Language Processing, Machine Learning, Deep Learning

Start Date

2022-01-01

CISUC budget

€ 195 892.00

End Date

2025-09-30

Publications

2026

P. L. Louro and H. Redinho and R. Malheiro and R. Panda and R. P. Paiva and R. Santos and T. F. Ribeiro, "MERGE: A Bimodal Audio-Lyrics Dataset For Static Music Emotion Recognition", IEEE Transactions on Affective Computing, pp. 1–18-1–18, 2026

2025

P. L. Louro and A. C. d. Santos and R. Malheiro and R. P. Paiva and R. Panda and H. Redinho, "Percussion and Instrumentation in Music Emotion Recognition: a Feature Engineering Approach", IEEE Transactions on Affective Computing, vol. (early access), 2025

R. P. Paiva and P. Brutti and S. Chieppa, "Automatic Guitar Transcription With Deep Neural Networks", IEEE Access, vol. 13, pp. 113573–113585-113573–113585, 2025

P. L. Louro and T. F. Ribeiro and R. Malheiro and R. Panda and R. P. Paiva, "BEE-MER: Bimodal Embeddings Ensemble for Music Emotion Recognition", in 22nd Sound and Music Computing Conference – SMC 2025, 2025

2024

R. Malheiro and R. P. Paiva and R. Panda and P. L. Louro and R. Hugo, "A Comparison Study of Deep Learning Methodologies for Music Emotion Recognition", Sensors, in Sensors, vol. 24, no. 7, 2024

T. G. Ferreira and H. Redinho and P. L. Louro and R. Malheiro and R. P. Paiva and R. Panda, "Exploring Song Segmentation for Music Emotion Variation Detection", in 15th International Workshop on Machine Learning and Music – MML 2024 – in conjunction with the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases – ECML/PKDD 2024, 2024

P. L. Louro and H. Redinho and R. Malheiro and R. P. Paiva and R. Panda, "Exploring Deep Learning Methodologies for Music Emotion Recognition", in Sound and Music Computing Conference – SMC 2024, 2024

P. L. Louro and H. Redinho and R. Malheiro and R. P. Paiva and R. Panda, "Improving Deep Learning Methodologies for Music Emotion Recognition", in 30th Portuguese Conference on Pattern Recognition – RECPAD 2024, 2024

P. L. Louro and G. Branco and H. Redinho and R. Santos and R. Malheiro and R. Panda and R. P. Paiva, "MERGE App: A Prototype Software for Multi-User Emotion-Aware Music Management", in 16th International Conference on Knowledge Discovery and Information Retrieval (KDIR), part of the 16th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management (IC3K), 2024

P. L. Louro and H. Redinho and R. Malheiro and R. P. Paiva and R. Panda, "Improving Music Emotion Recognition by Leveraging Handcrafted and Learned Features", in 1st Latin American Music Information Retrieval Workshop – LAMIR 2024, 2024

P. L. Louro and A. C. d. Santos and R. Malheiro and R. P. Paiva and R. Panda and H. Redinho, "Exploiting Automatic Source Separation and Music Transcription for Music Emotion Recognition", in Workshop on Affective Artificial Intelligence – A2I 2024 – at the International Conference on Pattern Recognition – ICPR 2024, in Pattern Recognition. ICPR 2024 International Workshops and Challenges, pp. 67–82-67–82, 2024

2022

M. Davies and S. Sulun and P. Viana, "Symbolic Music Generation Conditioned on Continuous-Valued Emotions", IEEE Access, in IEEE Access, vol. 10, pp. 44617, pp. 44617–44626-44626-44617–44626, 2022

This website uses cookies to improve your experience. Read More