C.P.M. van der Vleuten

The assessment of professional competence: Developments, research and practical implications

Educational achievement testing is an area of turmoil in the health sciences. Examinations are a constant source of problems for many teachers, curriculum designers and educationalists. The evaluation of student achievement is continuously debated at educational meetings, conferences and workshops. It is an area in which tradition, personal values and experiences tend to dominate discussions. On the other hand, the number of scientific publications on assessment over the last decade has exploded. The number of proposed instruments, each preferably using an intriguing acronym, is countless. The literature is however often difficult to access since the psychometrics usually involved in educational testing discourage the average health professions reader. Assessment in the health professions education is nevertheless an area which is fortunately well-researched and has delivered a number of well-documented outcomes. The purpose of this article is to highlight these outcomes and to attempt to translate them into practical implications and research suggestions. We will not review individual methods and instruments in detail, but will describe some classes of methods contingent on a (supposed) theoretical framework. We will delineate what we consider paramount findings within and across these classes and discuss their theoretical implications and their effect on the evolution of new testing instruments. By using a simple conceptual framework the utility of assessment methods will subsequently be defined in a generic sense meant to be helpful in deciding how to compromise and make trade-offs in assessment practice. To improve the utility of assessment a number of practical suggestions and research recommendations are proposed using this framework.

A model for programmatic assessment fit for purpose

We propose a model for programmatic assessment in action, which simultaneously optimises assessment for learning and assessment for decision making about learner progress. This model is based on a set of assessment principles that are interpreted from empirical research. It specifies cycles of training, assessment and learner support activities that are complemented by intermediate and final moments of evaluation on aggregated assessment data points. A key principle is that individual data points are maximised for learning and feedback value, whereas high-stake decisions are based on the aggregation of many data points. Expert judgement plays an important role in the programme. Fundamental is the notion of sampling and bias reduction to deal with the inevitable subjectivity of this type of judgement. Bias reduction is further sought in procedural assessment strategies derived from criteria for qualitative research. We discuss a number of challenges and opportunities around the proposed model. One of its prime virtues is that it enables assessment to move, beyond the dominant psychometric discourse with its focus on individual instruments, towards a systems approach to assessment design underpinned by empirically grounded theory.

Fifteen years of experience with progress testing in a problem-based learning curriculum

This article reports on educational experiences with an assessment method to assess knowledge in a problem-based learning context. This so-called progress test is a comprehensive test sampling knowledge across all content areas of medicine reflecting the end objectives of the curriculum. The test is periodically given to all medical students in the curriculum regardless of their year of training. The format precludes the possibility for students to prepare themselves specifically, therefore preventing the often reported undesirable effects of objective tests such as memorization of facts and interference with tutorial group functioning. The many years of experience indicate that this testing format works effectively. After the introduction of progress tests a number of other, to some extent unexpected, educational advantages became apparent, as well as a few disadvantages. The additional advantages make progress testing also useful in non-problem-based curricula.

How residents learn: qualitative evidence for the pivotal role of clinical activities

Objectives  Medical councils worldwide have outlined new standards for postgraduate medical education. This means that residency programmes will have to integrate modern educational views into the clinical workplace. Postgraduate medical education is often characterised as a process of learning from experience. However, empirical evidence regarding the learning processes of residents in the clinical workplace is lacking. This qualitative study sought insight into the intricate process of how residents learn in the clinical workplace.

The use of qualitative research criteria for portfolio assessment as an alternative to reliability evaluation: a case study

Aim  Because it deals with qualitative information, portfolio assessment inevitably involves some degree of subjectivity. The use of stricter assessment criteria or more structured and prescribed content would improve interrater reliability, but would obliterate the essence of portfolio assessment in terms of flexibility, personal orientation and authenticity. We resolved this dilemma by using qualitative research criteria as opposed to reliability in the evaluation of portfolio assessment.

The use of clinical simulations in assessment.

Context  Simulation‐based testing methods have been developed to meet the need for assessment procedures that are both authentic and well‐structured. It is widely acknowledged that, although the authenticity of a procedure may be a contributing factor to its validity, authenticity alone never is a sufficient factor.

The need for evidence in education

In this article a plea is made to use evidence in education. A remarkable difference in attitude is noted between university staff in their role as scientists in their discipline and in their role as teachers. Whereas evidence is the key to guide scientists in the development of their discipline, evidence on teaching and learning hardly affects their role as teachers. Teaching is, rather, dominated by intuition and tradition. However, particularly in education, intuitions and traditions are not always correct when they are submitted to empirical verification. It even often turns out that our intuitions are not justified or that assumed relations are far more complex. To illustrate the fallacy of our (implicit) intuitions and beliefs, a few of these assumptions are held against the available evidence. Two assumptions related to the learning of students and two assumptions related to the assessment of student achievement are discussed. The illustrations make clear that we do need to use evidence in educatio...In this article a plea is made to use evidence in education. A remarkable difference in attitude is noted between university staff in their role as scientists in their discipline and in their role as teachers. Whereas evidence is the key to guide scientists in the development of their discipline, evidence on teaching and learning hardly affects their role as teachers. Teaching is, rather, dominated by intuition and tradition. However, particularly in education, intuitions and traditions are not always correct when they are submitted to empirical verification. It even often turns out that our intuitions are not justified or that assumed relations are far more complex. To illustrate the fallacy of our (implicit) intuitions and beliefs, a few of these assumptions are held against the available evidence. Two assumptions related to the learning of students and two assumptions related to the assessment of student achievement are discussed. The illustrations make clear that we do need to use evidence in education, just as we do in any other professional area. Being a professional teacher requires more than being an expert in a content area; it also requires familiarity, use, and perhaps production of educational evidence and theory.

Workplace-based assessment: effects of rater expertise.

Traditional psychometric approaches towards assessment tend to focus exclusively on quantitative properties of assessment outcomes. This may limit more meaningful educational approaches towards workplace-based assessment (WBA). Cognition-based models of WBA argue that assessment outcomes are determined by cognitive processes by raters which are very similar to reasoning, judgment and decision making in professional domains such as medicine. The present study explores cognitive processes that underlie judgment and decision making by raters when observing performance in the clinical workplace. It specifically focuses on how differences in rating experience influence information processing by raters. Verbal protocol analysis was used to investigate how experienced and non-experienced raters select and use observational data to arrive at judgments and decisions about trainees’ performance in the clinical workplace. Differences between experienced and non-experienced raters were assessed with respect to time spent on information analysis and representation of trainee performance; performance scores; and information processing––using qualitative-based quantitative analysis of verbal data. Results showed expert-novice differences in time needed for representation of trainee performance, depending on complexity of the rating task. Experts paid more attention to situation-specific cues in the assessment context and they generated (significantly) more interpretations and fewer literal descriptions of observed behaviors. There were no significant differences in rating scores. Overall, our findings seemed to be consistent with other findings on expertise research, supporting theories underlying cognition-based models of assessment in the clinical workplace. Implications for WBA are discussed.

Acquisition of communication skills in postgraduate training for general practice.

Purpose  The evidence suggests that a longitudinal training of communication skills embedded in a rich clinical context is most effective. In this study we evaluated the acquisition of communication skills under such conditions.

Measurement of clinical reflective capacity early in training as a predictor of clinical reasoning performance at the end of residency: an experimental study on the script concordance test

The script concordance (SC) test was conceived to measure knowledge organization, the presence of links between items of knowledge which allow for interpretation of data in clinical decision making situations. Earlier studies have shown that the SC test has good psychometric qualities and overcomes some of the limitations of simulation clinical testing. This study explores the predictive validity of the test.