Charles Gondre

The effect of real-time auditory feedback on learning new characters

The present study investigated the effect of handwriting sonification on graphomotor learning. Thirty-two adults, distributed in two groups, learned four new characters with their non-dominant hand. The experimental design included a pre-test, a training session, and two post-tests, one just after the training sessions and another 24h later. Two characters were learned with and two without real-time auditory feedback (FB). The first group first learned the two non-sonified characters and then the two sonified characters whereas the reverse order was adopted for the second group. Results revealed that auditory FB improved the speed and fluency of handwriting movements but reduced, in the short-term only, the spatial accuracy of the trace. Transforming kinematic variables into sounds allows the writer to perceive his/her movement in addition to the written trace and this might facilitate handwriting learning. However, there were no differential effects of auditory FB, neither long-term nor short-term for the subjects who first learned the characters with auditory FB. We hypothesize that the positive effect on the handwriting kinematics was transferred to characters learned without FB. This transfer effect of the auditory FB is discussed in light of the Theory of Event Coding.

An intuitive synthesizer of continuous-interaction sounds: Rubbing, scratching, and rolling

In this article, we propose a control strategy for synthesized continuous-interaction sounds. The framework of our research is based on the action–object paradigm that describes the sound as the result of an action on an object and that presumes the existence of sound invariants (i.e., perceptually relevant signal morphologies that carry information about the actions or the objects attributes). Auditory cues are investigated here for the evocations of rubbing, scratching, and rolling interactions. A generic sound-synthesis model that simulates these interactions is detailed. We then suggest an intuitive control strategy that enables users to navigate continuously from one interaction to another in an “action space,” thereby offering the possibility to simulate morphed interactions—for instance, ones that morph between rubbing and rolling.

Movement Sonification for the Diagnosis and the Rehabilitation of Graphomotor Disorders

The dynamic features of sounds make them particularly appropriate for assessing the spatiotemporal characteristics of movements. Furthermore, sounds can inform about the correctness of an ongoing movement without directly interfering with the visual and proprioceptive feedback. Finally, because of their playful characteristics, sounds are potentially effective for motivating writers in particular need of any writing assistance. By associating relevant sounds to the specific variables of handwriting movement, the present chapter aimed at reporting how supplementary auditory information allows an examiner (teacher or therapist) to assess the movement quality from his/her hearing. Furthermore, a writer could also improve his/her movement from this real-time auditory feedback. Sonification of some movement characteristics would be a relevant tool for the diagnosis and the rehabilitation of some developmental disabilities (e.g. dysgraphia) or acquired disorders (e.g. Parkinson’s disease).

“Let Me Hear Your Handwriting!” Evaluating the Movement Fluency from Its Sonification

The quality of handwriting is evaluated from the visual inspection of its legibility and not from the movement that generates the trace. Although handwriting is achieved in silence, adding sounds to handwriting movement might help towards its perception, provided that these sounds are meaningful. This study evaluated the ability to judge handwriting quality from the auditory perception of the underlying sonified movement, without seeing the written trace. In a first experiment, samples of a word written by children with dysgraphia, proficient children writers, and proficient adult writers were collected with a graphic tablet. Then, the pen velocity, the fluency, and the axial pen pressure were sonified in order to create forty-five audio files. In a second experiment, these files were presented to 48 adult listeners who had to mark the underlying unseen handwriting. In order to evaluate the relevance of the sonification strategy, two experimental conditions were compared. In a first ‘implicit’ condition, the listeners made their judgment without any knowledge of the mapping between the sounds and the handwriting variables. In a second ‘explicit’ condition, they knew what the sonified variables corresponded to and the evaluation criteria. Results showed that, under the implicit condition, two thirds of the listeners marked the three groups of writers differently. In the explicit condition, all listeners marked the dysgraphic handwriting lower than that of the two other groups. In a third experiment, the scores given from the auditory evaluation were compared to the scores given by 16 other adults from the visual evaluation of the trace. Results revealed that auditory evaluation was more relevant than the visual evaluation for evaluating a dysgraphic handwriting. Handwriting sonification might therefore be a relevant tool allowing a therapist to complete the visual assessment of the written trace by an auditory control of the handwriting movement quality.