Hendrik Vincent Koops

A functional approach to automatic melody harmonisation

Melody harmonisation is a centuries-old problem of long tradition, and a core aspect of composition in Western tonal music. In this work we describe FHarm, an automated system for melody harmonisation based on a functional model of harmony. Our system first generates multiple harmonically well-formed chord sequences for a given melody. From the generated sequences, the best one is chosen, by picking the one with the smallest deviation from the harmony model. Unlike all existing systems, FHarm guarantees that the generated chord sequences follow the basic rules of tonal harmony. We carry out two experiments to evaluate the quality of our harmonisations. In one experiment, a panel of harmony experts is asked to give its professional opinion and rate the generated chord sequences for selected melodies. In another experiment, we generate a chord sequence for a selected melody, and compare the result to the original harmonisation given by a harmony scholar. Our experiments confirm that FHarm generates realistic chords for each melody note. However, we also conclude that harmonising a melody with individually well-formed chord sequences from a harmony model does not guarantee a well-sounding coherence between the chords and the melody. We reflect on the experience gained with our experiment, and propose future improvements to refine the quality of the harmonisation.

Music Outlier Detection Using Multiple Sequence Alignment and Independent Ensembles

The automated retrieval of related music documents, such as cover songs or folk melodies belonging to the same tune, has been an important task in the field of Music Information Retrieval (MIR). Yet outlier detection, the process of identifying those documents that deviate significantly from the norm, has remained a rather unexplored topic. Pairwise comparison of music sequences (e.g. chord transcriptions, melodies), from which outlier detection can potentially emerge, has been always in the center of MIR research but the connection has remained uninvestigated. In this paper we firstly argue that for the analysis of musical collections of sequential data, outlier detection can benefit immensely from the advantages of Multiple Sequence Alignment (MSA). We show that certain MSA-based similarity methods can better separate inliers and outliers than the typical similarity based on pairwise comparisons. Secondly, aiming towards an unsupervised outlier detection method that is data-driven and robust enough to be generalizable across different music datasets, we show that ensemble approaches using an entropy-based diversity measure can outperform supervised alternatives.

A data-driven approach to chord similarity and chord mutability

Assessing the relationship between chord sequences is an important ongoing research topic in the fields of music cognition and music information retrieval. Heuristic and cognitive models of chord similarity have been investigated but none has aimed to capture the collective perception of chord similarity from a large dataset of user-generated content. Devising a largescale experiment to gather sufficient data from human subjects has always been a major stumbling block. We present a novel chord similarity model based on a large amount of crowd-sourced transcriptions from a popular automatic chord estimation service. We show that our model outperforms heuristic-based models in a song identification task. Secondly, a model of chord mutations based on a large amount of crowd-sourced cover songs transcriptions is introduced. From crowd-sourced data, we create substitution matrices that capture the perceived similarity and mutability between chords. These results show that modelling the collective perception can not only substitute alternative, sophisticated models but also further enhance performance in various music information retrieval tasks.

Automatic Segmentation and Deep Learning of Bird Sounds

We present a study on automatic birdsong recognition with deep neural networks using the birdclef2014 dataset. Through deep learning, feature hierarchies are learned that represent the data on several levels of abstraction. Deep learning has been applied with success to problems in fields such as music information retrieval and image recognition, but its use in bioacoustics is rare. Therefore, we investigate the application of a common deep learning technique deep neural networks in a classification task using songs from Amazonian birds. We show that various deep neural networks are capable of outperforming other classification methods. Furthermore, we present an automatic segmentation algorithm that is capable of separating bird sounds from non-bird sounds.

Automatic chord label personalization through deep learning of shared harmonic interval profiles

Current automatic chord estimation systems are trained and tested using datasets that contain single reference annotations , i.e., for each corresponding musical segment (e.g., audio frame or section), the reference annotation contains a single chord label. Nevertheless, theoretical insights on harmonic ambiguity from harmony theory, experimental studies on annotator subjectivity in harmony annotations, and the availability of vast amounts of heterogeneous (subjective) harmony annotations in crowd-sourced repositories make the notion of a single-harmonic “ground truth” reference annotation a tenuous one. Recent studies suggest that subjectivity is intrinsic to harmonic reference annotations that should be embraced in automatic chord estimation rather than resolved. We introduce the first approach to automatic chord label personalization by modeling annotator subjectivity through harmonic interval-based chord representations. We integrate these representations from multiple annotators and deep learn them from audio. From a single trained model and the annotators’ chord-label vocabulary, we can accurately personalize chord labels for individual annotators. Furthermore, we show that chord personalization using multiple reference annotations outperforms using just a single reference annotation. Our results show that annotator subjectivity should inform future research on automatic chord estimation to improve the state of the art.

Multirotor UAV state prediction through multi-microphone side-channel fusion

Improving trust in the state of Cyber-Physical Systems becomes increasingly important as more Cyber-Physical Systems tasks become autonomous. Research into the sound of Cyber-Physical Systems has shown that audio side-channel information from a single microphone can be used to accurately model traditional primary state sensor measurements such as speed and gear position. Furthermore, data integration research has shown that information from multiple heterogeneous sources can be integrated to create improved and more reliable data. In this paper, we present a multi-microphone machine learning data fusion approach to accurately predict ascending/hovering/descending states of a multi-rotor UAV in flight. We show that data fusion of multiple audio classifiers predicts these states with accuracies over 94%. Furthermore, we significantly improve the state predictions of single microphones, and outperform several other integration methods. These results add to a growing body of work showing that microphone side-channel approaches can be used in Cyber-Physical Systems to accurately model and improve the assurance of primary sensors measurements.

Chord Label Personalization through Deep Learning of Integrated Harmonic Interval-based Representations

Proceedings of the First International Workshop on Deep Learning and Music, joint with IJCNN, Anchorage, US, May 17-18, 2017

Melody Retrieval and Classification Using Biologically-Inspired Techniques

Retrieval and classification are at the center of Music Information Retrieval research. Both tasks rely on a method to assess the similarity between two music documents. In the context of symbolically encoded melodies, pairwise alignment via dynamic programming has been the most widely used method. However, this approach fails to scale-up well in terms of time complexity and insufficiently models the variance between melodies of the same class. Compact representations and indexing techniques that capture the salient and robust properties of music content, are increasingly important. We adapt two existing bioinformatics tools to improve the melody retrieval and classification tasks. On two datasets of folk tunes and cover song melodies, we apply the extremely fast indexing method of the Basic Local Alignment Search Tool (BLAST) and achieve comparable classification performance to exhaustive approaches. We increase retrieval performance and efficiency by using multiple sequence alignment algorithms for locating variation patterns and profile hidden Markov models for incorporating those patterns into a similarity model.