Erica Volta

A multimodal corpus for technology-enhanced learning of violin playing

Learning to play a musical instrument is a difficult task, mostly based on the master-apprentice model. Technologies are rarely employed and are usually restricted to audio and video recording and playback. Nevertheless, multimodal interactive systems can complement actual learning and teaching practice, by offering students guidance during self-study and by helping teachers and students to focus on details that would be otherwise difficult to appreciate from usual audiovisual recordings. This paper introduces a multimodal corpus consisting of the recordings of expert models of success, provided by four professional violin performers. The corpus is publicly available on the repoVizz platform, and includes synchronized audio, video, motion capture, and physiological (EMG) data. It represents the reference archive for the EU-H2020-ICT Project TELMI, an international research project investigating how we learn musical instruments from a pedagogical and scientific perspective and how to develop new interactive, assistive, self-learning, augmented-feedback, and social-aware systems to support musical instrument learning and teaching.

Automatically measuring biomechanical skills of violin performance: an exploratory study

This evaluation study explores how automated movement analysis can be used to catch the biomechanical skills needed for a physically accurate violin performance, maximizing efficiency and minimizing injuries. Starting from a previously recorded multimodal dataset, we compute movement features from motion captured data of five violinists performing three violin exercises: octave shift, string crossing, and a Romantic repertoire piece. Three violin teachers were asked to evaluate audio, video, and both audio and video stimuli of the selected exercises. We correlated their ratings with automatically extracted movement features. Whereas these features are purely visual (i.e., they are computed from motion captured data only), we asked teachers to also evaluate audio because it can be considered as the direct translation of movement skills into another modality. In this way, we can also look at possible relations between evaluation of the audio aspects of the performance and biomechanical skills of violin playing. Results show that the proposed movement features can be partially used to measure the biomechanical skills of the violin players to support learning and mitigate the risk of injuries.

Exploiting multimodal integration in adaptive interactive systems and game-based learning interfaces

The main purpose of my work is to investigate multisensory and multimodal integration in the design and development of adaptive systems and interfaces for game-based learning applications in the areas of education and rehabilitation. To this aim, I contributed to the creation of a multimodal dataset of violin performances, integrating motion capture, video, audio, and on-body sensors (accelerometers and EMG), and I worked closely with psychophysicists and educators on the design of paradigms and technologies for multisensory and embodied learning of mathematics in primary school children. Main theoretical foundations of my research are multisensory processing and integration, psychophysics analysis, embodied cognition theories, computational models of non-verbal and emotion communication in full-body movement, and human-computer interaction models for adaptive interfaces and serious-games.

Enhancing children understanding of mathematics with multisensory technology

Recent results from psychophysics and developmental psychology show that children do not integrate and use the same sensory modalities in the same way, but they rather have a preferential sensory channel to learn specific concepts. In schools, however, the visual channel is often the only one exploited for teaching, whereas the other channels are left to a marginal role. The weDRAW project, an ongoing European two-years project, explores the chance to create and evaluate new methodologies to teaching, grounded on multisensory technologies. In particular, the project targets a deeper understanding of arithmetic and geometry. A major novelty of such a new technology is that it is based on the renewed understanding of what communication between senses is and how it occurs during child development, that is, that specific sensory systems have specific roles for learning specific concepts. Neuroscientific prospectives suggest that it is possible to develop new teaching/learning channels, personalized for each student based on the childs sensory skills. This practice work consists of a demo simulating a set-up in a primary school, where children with any level of previous mathematical knowledge can try the technologies themselves. The demo includes full-body activities to play and draw arithmetical and geometrical concepts, such as playing a fraction using body rhythms and drawing shapes and symmetries.

Enhancing Music Learning with Smart Technologies

Learning to play a musical instrument is a difficult task, requiring the development of sophisticated skills. Nowadays, such a learning process is mostly based on the master-apprentice model. Technologies are rarely employed and are usually restricted to audio and video recording and playback. The TELMI (Technology Enhanced Learning of Musical Instrument Performance) Project seeks to design and implement new interaction paradigms for music learning and training based on state-of-the-art multimodal (audio, image, video, and motion) technologies. The project focuses on the violin as a case study. This practice work is intended as demo, showing to MOCO attendants the results the project obtained along two years of work. The demo simulates a setup at a higher education music institution, where attendants with any level of previous violin experience (and even with no experience at all) are invited to try the technologies themselves, performing basic tests of violin skill and pre-defined exercises under the guidance of the researchers involved in the project.

What cognitive and affective states should technology monitor to support learning

This paper discusses self-efficacy, curiosity, and reflectivity as cognitive and affective states that are critical to learning but are overlooked in the context of affect-aware technology for learning. This discussion sits within the opportunities offered by the weDRAW project aiming at an embodied approach to the design of technology to support exploration and learning of mathematical concepts. We first review existing literature to clarify how the three states facilitate learning and how, if not supported, they may instead hinder learning. We then review the literature to understand how bodily expressions communicate these states and how technology could be used to monitor them. We conclude by presenting initial movement cues currently explored in the context of weDRAW.

An open platform for full-body multisensory serious-games to teach geometry in primary school

Recent results from psychophysics and developmental psychology show that children have a preferential sensory channel to learn specific concepts. In this work, we explore the possibility of developing and evaluating novel multisensory technologies for deeper learning of arithmetic and geometry. The main novelty of such new technologies comes from the renewed understanding of the role of communication between sensory modalities during development that is that specific sensory systems have specific roles for learning specific concepts. Such understanding suggests that it is possible to open a new teaching/learning channel, personalized for each student based on the child’s sensory skills. Multisensory interactive technologies exploiting full-body movement interaction and including a hardware and software platform to support this approach will be presented and discussed. The platform is part of a more general framework developed in the context of the EU-ICT-H2020 weDRAW Project that aims to develop new multimodal technologies for multisensory serious-games to teach mathematics concepts in the primary school.

A multimodal serious-game to teach fractions in primary school

Multisensory learning is considered a relevant pedagogical framework for education since a very long time and several authors support the use of a multisensory and kinesthetic approach in children learning. Moreover, results from psychophysics and developmental psychology show that children have a preferential sensory channel to learn specific concepts (spatial and/or temporal), hence a further evidence for the need of a multisensory approach. In this work, we present an example of serious game for learning a particularly complicated mathematical concept: fractions. The main novelty of our proposal comes from the role covered by the communication between sensory modalities in particular, movement, vision, and sound. The game has been developed in the context of the EU-ICT-H2020 weDRAW Project aiming at developing new multimodal technologies for multisensory serious-games on mathematical concepts for primary school children.

MIE 2017: 1st international workshop on multimodal interaction for education (workshop summary)

The International Workshop on Multimodal Interaction for Education aims at investigating how multimodal interactive systems, firmly grounded on psychophysical, psychological, and pedagogical bases, can be designed, developed, and exploited for enhancing teaching and learning processes in different learning environments, with a special focus on children in the classroom. Whilst the usage of multisensory technologies in the education area is rapidly expanding, the need for solid scientific bases, design guidelines, and appropriate procedures for evaluation is emerging. Moreover, the introduction of multimodal interactive systems in the learning environment needs to develop at the same time suitable pedagogical paradigms. This workshop aims at bringing together researchers and practitioners from different disciplines, including pedagogy, psychology, psychophysics, and computer science - with a particular focus on human-computer interaction, affective computing, and social signal processing - to discuss such challenges under a multidisciplinary perspective. The workshop is partially supported by the EU-H2020-ICT Project weDRAW (http://www.wedraw.eu).