Brian C. Wesolowski

Rater fairness in music performance assessment: Evaluating model-data fit and differential rater functioning

The purpose of this study was to investigate model-data fit and differential rater functioning in the context of large group music performance assessment using the Many-Facet Rasch Partial Credit Measurement Model. In particular, we sought to identify whether or not expert raters’ (N = 24) severity was invariant across four school levels (middle school, high school, collegiate, professional). Interaction analyses suggested that differential rater functioning existed for both the group of raters and some individual raters based on their expected locations on the logit scale. This indicates that expert raters did not demonstrate invariant levels of severity when rating subgroups of ensembles across the four school levels. Of the 92 potential pairwise interactions examined, 14 (15.2%) interactions were found to be statistically significant, indicating that 10 individual raters demonstrated differential severity across at least one school level. Interpretations of meaningful systematic patterns emerged for some raters after investigating individual pairwise interactions. Implications for improving the fairness and equity in large group music performance evaluations are discussed.

Assessing jazz big band performance: The development, validation, and application of a facet-factorial rating scale

The purpose of this study was to develop a valid and reliable rating scale to assess jazz big band performance and to evaluate the psychometric properties of jazz big bands at three performance achievement levels (e.g., low, moderate, high). The pool of initial scale items (N = 22) was gleaned from jazz big band research and instructional literature. Using a four-point Likert scale, rating responses (N = 102) were gathered for jazz ensemble performances (N = 102) from volunteer raters (N = 102). A factor analysis produced a reduced 18-item scale with a four-factor structure: blend/balance, time-feel, idiomatic nuance, and expression. The four factors accounted for 63.32% of the variance and had a total alpha reliability of .84. Discriminant function analyses revealed that four specific items contributed most to identifying ensembles with low and moderate performance ratings. The factor structure was able to predict group membership with 88.5% accuracy.

Exploring the Effects of Rater Linking Designs and Rater Fit on Achievement Estimates Within the Context of Music Performance Assessments

When good model-data fit is observed, the Many-Facet Rasch (MFR) model acts as a linking and equating model that can be used to estimate student achievement, item difficulties, and rater severity on the same linear continuum. Given sufficient connectivity among the facets, the MFR model provides estimates of student achievement that are equated to control for differences in rater severity. Although several different linking designs are used in practice to establish connectivity, the implications of design differences have not been fully explored. Research is also limited related to the impact of model-data fit on the quality of MFR model-based adjustments for rater severity. This study explores the effects of linking designs and model-data fit for raters on the interpretation of student achievement estimates within the context of performance assessments in music. Results indicate that performances cannot be effectively adjusted for rater effects when inadequate linking or model-data fit is present.

The Tight-interlocked Rhythm Section: Production and Perception of Synchronisation in Jazz Trio Performance

Abstract This study investigates the production and perception of timing, synchronisation and dynamics in jazz trio performances. In a production experiment, six trio combinations of one saxophonist, two bassists, and three drummers were recorded while they performed three popular jazz songs. Onset timing and dynamics of each performer were extracted and analysed. Results showed that the tempo was significantly influenced by the timing of the drummers and all performers showed higher temporal precision on the backbeats. The drummers demonstrated individual swing-ratios, accentuations of beats and intrapersonal asynchronies between simultaneous hi-hat and ride cymbal onsets, which resulted in a hi-hat played 2–26 ms ahead of the pulse of the music. In a subsequent perception test, participants () rated 12 excerpts of the jazz recordings. They selected their preferred version from a pool of stimuli containing the original version, but also manipulations with artificially increased or reduced asynchronies. Stimuli with reduced asynchronies smaller than 19 ms were preferred by the listeners over the original or the fully quantised timing. This suggests that listeners endorse a ‘tight-interlocked’ jazz rhythm section, with asynchronies smaller than the perceptual threshold (temporal masking), but with natural timing variabilities that makes it distinguishable from a computer-generated playback.

There’s More to Groove than Bass in Electronic Dance Music: Why Some People Won’t Dance to Techno

The purpose of this study was to explore the relationship between audio descriptors for groove-based electronic dance music (EDM) and raters’ perceived cognitive, affective, and psychomotor responses. From 198 musical excerpts (length: 15 sec.) representing 11 subgenres of EDM, 19 low-level audio feature descriptors were extracted. A principal component analysis of the feature vectors indicated that the musical excerpts could effectively be classified using five complex measures, describing the rhythmical properties of: (a) the high-frequency band, (b) the mid-frequency band, and (c) the low-frequency band, as well as overall fluctuations in (d) dynamics, and (e) timbres. Using these five complex audio measures, four meaningful clusters of the EDM excerpts emerged with distinct musical attributes comprising music with: (a) isochronous bass and static timbres, (b) isochronous bass with fluctuating dynamics and rhythmical variations in the mid-frequency range, (c) non-isochronous bass and fluctuating timbres, and (d) non-isochronous bass with rhythmical variations in the high frequencies. Raters (N = 99) were each asked to respond to four musical excerpts using a four point Likert-Type scale consisting of items representing cognitive (n = 9), affective (n = 9), and psychomotor (n = 3) domains. Musical excerpts falling under the cluster of “non-isochronous bass with rhythmical variations in the high frequencies” demonstrated the overall highest composite scores as evaluated by the raters. Musical samples falling under the cluster of “isochronous bass with static timbres” demonstrated the overall lowest composite scores as evaluated by the raters. Moreover, music preference was shown to significantly affect the systematic patterning of raters’ responses for those with a musical preference for “contemporary” music, “sophisticated” music, and “intense” music.

The Development of a Secondary-Level Solo Wind Instrument Performance Rubric Using the Multifaceted Rasch Partial Credit Measurement Model

The purpose of this study was to describe the development of a valid and reliable rubric to assess secondary-level solo instrumental music performance based on principles of invariant measurement. The research questions that guided this study included (1) What is the psychometric quality (i.e., validity, reliability, and precision) of a scale developed to assess secondary-level solo music performance? (2) Do the proposed items fit the measurement model, and if so, how do the items vary in difficulty? and (3) How does the structure of the rating scale vary across individual items? The psychometric considerations in this study included calibrations of items, persons, raters, school level, musical instrument, and rating scale structure using the Multifaceted Rasch Partial Credit Measurement Model. A 13-member cohort of music content experts participated as raters in this study. A total of 75 video performances of secondary-level solo and ensemble performances were evaluated. The result was the development of the Music Performance Rubric for Secondary-Level Instrumental Solos (MPR-2L-INSTSOLO), a 30-item rubric consisting of rating scale categories ranging from two to four performance criteria. Implications for consequential validity, rater training, standard setting, and benchmarking are discussed.

Timing deviations in jazz performance: The relationships of selected musical variables on horizontal and vertical timing relations: A case study

The purpose of this study was to examine the effects and relationships of selected musical variables on the horizontal (i.e., successive eighth note timing relationships) and vertical (i.e., the degree of ensemble synchronicity between separate parts) timing properties of jazz rhythm. A total of 949 eighth note samples from five improvised solos by saxophonist Chris Potter were analysed. Musical variables included metrical beat placement, melodic character, intervals, articulation, underlying harmony, and tempo. Results of the simultaneous multiple regression analyses revealed that intervals preceding and intervals succeeding the sample eighth notes had a significant effect on eighth note durations. Articulation had a significant effect on upbeat beat ratio. The relationships between the relative onset timing data of the saxophone, bass, and drums yielded significant results. There was a large, negative correlation between the relative timing onsets of saxophone and bass, a large negative correlation between saxophone and drums, and a moderate, negative correlation between bass and drums. Implications for the fields of music education, pedagogy, and cognitive psychology are discussed.

Exploring rater cognition: A typology of raters in the context of music performance assessment

This manuscript sought to investigate rater cognition by exploring rater types based upon differential severity and leniency associated with rating scale items, rating scale category functioning, and dimensions of music performance assessment. The purpose of this study was to empirically identify typologies of operational raters based upon systematic differential severity indices in the context of large ensemble music performance assessment. A rater cognition information-processing model was explored based upon two frameworks: a framework for scoring and a framework for audition. Rater scoring behavior was examined using a framework for scoring, where raters’ mental processes compare auditory images to the scoring criteria used to generate a scoring decision. The scoring decisions were evaluated using the Multifaceted Rasch Partial Credit Measurement Model. A rater typology was then examined under the framework of audition, where similar schemata were defined through raters’ clustering of differential severity indices related to items and compared across performance dimensions. The results provided three distinct rater-types: (a) the syntactical rater; (b) the expressive rater; and (c) the mental representation rater. Implications for fairness and precision in the assessment process are discussed as well as considerations for validity of scoring processes.

A facet-factorial approach towards the development and validation of a jazz rhythm section performance rating scale

The purpose of this study was to develop a valid and reliable rating scale to assess jazz rhythm sections in the context of jazz big band performance. The research questions that guided this study included: (a) what central factors contribute to the assessment of a jazz rhythm section? (b) what items should be used to describe and assess a jazz rhythm section performance? (c) how should the items be categorized? (d) what differences among jazz rhythm sections exist at three performance achievement levels? and (e) what criteria best predict group membership into three performance achievement levels? Items were gathered from research and literature related to the assessment, teaching, and general discussions related to the jazz rhythm section. Twenty-nine item statements were paired with a four-point Likert scale. One hundred and twenty-two responses were gathered from 41 volunteer raters. The results of the data were factor analyzed and yielded a two-factor structure including rhythmic support/drive and style/clarity. The 16-item scale accounted for 79.23% of the variance and the alpha reliability was estimated at 0.986. The rhythmic support/drive factor contributed most to discriminating between overall group differences. More specifically, five out of the 16 items contributed most to discriminating between groups.

Evaluating the Psychometric Qualities of a Rating Scale to Assess Pre-Service Teachers’ Lesson Plan Development in the Context of a Secondary-Level Music Performance Classroom

The purpose of this study was to evaluate the psychometric quality (i.e., validity and reliability) of a rating scale to assess pre-service teachers’ lesson plan development in the context of secondary-level music performance classrooms. The research questions that guided this study include: (1) What items demonstrate acceptable model fit for the construct of lesson plan development in the context of a secondary-level music performance classroom? (2) How does the structure of the rating scale vary across items? and (3) Does differential severity emerge for academic administrators or music education content specialists across items? Using multiple teacher effectiveness frameworks, the lesson plans in this study were evaluated using a 4-point Likert-type rating scale (e.g., strongly agree, agree, disagree, strongly disagree) consisting of five domains: (a) instructional planning, (b) instructional delivery, (c) differentiated instruction, (d) assessment uses, and (e) assessment strategies. Secondary-level school administrators (n = 8) and music education content specialists (n = 8) rated 32 lesson plans using a balanced incomplete assessment network. The multifaceted Rasch measurement partial credit model was used in this study. Results suggest higher rater severity among administrators than music specialists. Of the 68 potential pairwise interactions examined in the study, 5 (7.4 %) of those were found to be statistically significant, which indicates that 5 raters demonstrated differential severity across at least one lesson plan. Implications for student teacher preparation, teacher effectiveness, and the validity of measures are discussed.