Shelley Gillis

The use of scoring rubrics to determine clinical performance in the operating suite

UNLABELLED This research evolved out of the need to examine the validity and inter-rater reliability of a set of performance-based scoring rubrics designed to measure competencies within the operating suite. METHOD Both holistic and analytical rubrics were developed aligned to the ACORN Standard [Australian College of Operating Room Nurses Standard NR4, 2004. ACORN Competency Standards for Perioperative Nurses: Standard NR4: The Instrument Nurse in the Perioperative Environment. Australian College of Operating Room Nurses Ltd, Adelaide] and underpinned by the Dreyfus model (1981). Three video clips that captured varying performance of nurses performing as instrument nurses in the operating suite were recorded and used as prompts by expert raters, who judged the performance using the rubrics. RESULTS The study found that the holistic rubrics led to more consistent judgments than the analytical rubrics, yet the latter provided more diagnostic information for intervention purposes. Despite less consistency, the Analytical Observation Form had sufficient construct validity to satisfy the requirements of criterion referencing as determined by the Item Separation Index (Rasch, 1960), including high internal consistency and greater inter-rater reliability when average ratings were used. CONCLUSION The study was an empirical investigation of the use of concomitant Analytical and Holistic Rubrics to determine various levels of performance in the operating suite including inter-rater reliability. The methodology chosen was theoretically sound and sufficiently flexible to be used to develop other competencies within the operating suite.

Standards-referenced assessment for vocational education and training in schools

This study examined a model of assessment that could be applied nationally for Year Twelve Vocational Education and Training (VET) subjects and which could yield both a differentiating score and recognition of competence. More than fifty colleges across all states and territories of Australia field-tested the approach over one school year. Results showed that the model allowed for a standards-referenced model to be used: that the approach was compatible with the diverse range of senior secondary assessment systems in use throughout Australia and that there were considerable cost benefits to be had in adopting the logic of item response modelling for the development of rubrics for scoring performances on units of competence from National Training Packages. A change in the logic of competency assessment was proposed, in that the performance indicators were not rated using a dichotomy but with a series of quality ordered criteria to indicate how well students performed specified tasks in the workplace or its simulation. The study validated the method of assessment development, demonstrated the methods consistency, and showed how the method could address the issue of consistency across states. The study also proposed a set of principles for a joint assessment of both quality and competence.

Principles underpinning graded assessment in VET: a critique of prevailing perceptions

Abstract To help achieve national consistency ofassessment and reporting in the Australian Vocational Education and Training sector, there is a need to develop a set of national principles for graded performance assessment This paper challenges a number of prevailing principles from both a theoretical and assessmentperspective, namely that grades must be criterion referenced (Williams & Bateman, 2003), meaningful (Rumsey, 2003) and applied once competence has been achieved (Williams & Bateman, 2003). This paper argues that the use of generic criteria cannot be defended in terms of their validity and reliability and that a clear understanding of the underlying developmental continuum of learning is required to inform the development of meaningful and valid criteria and descriptors of quality performance. Finally, the paper proposes a set ofprinciples that have been grounded in theory, have been put to the test in large-scale research, and are consistent with international literature on competence assessment.

Educational inequality and transitions to university in Australia: aspirations, agency and constraints

Abstract This paper is based on research into the destinations and aspirations of school leavers in Australia. It investigates the relationship between the transition to university for different groups of students and their own and their parents’ and teachers’ expectations. It draws on Bourdieu, Boudon, Nussbaum and others to investigate the way young people construct their aspirations. It examines the limits of young people’s agency, which is bound by their understanding of the hidden and informal rules that govern access to different spaces within the curriculum as well as access to post-school destinations. Navigating these transitions is becoming more important and more complex with the increasing emphasis on higher level qualifications in an education market.