Michaela Wagner-Menghin

Research in assessment: Consensus statement and recommendations from the Ottawa 2010 conference

Medical education research in general is a young scientific discipline which is still finding its own position in the scientific range. It is rooted in both the biomedical sciences and the social sciences, each with their own scientific language. A more unique feature of medical education (and assessment) research is that it has to be both locally and internationally relevant. This is not always easy and sometimes leads to purely ideographic descriptions of an assessment procedure with insufficient general lessons or generalised scientific knowledge being generated or vice versa. For medical educational research, a plethora of methodologies is available to cater to many different research questions. This article contains consensus positions and suggestions on various elements of medical education (assessment) research. Overarching is the position that without a good theoretical underpinning and good knowledge of the existing literature, good research and sound conclusions are impossible to produce, and that there is no inherently superior methodology, but that the best methodology is the one most suited to answer the research question unambiguously. Although the positions should not be perceived as dogmas, they should be taken as very serious recommendations. Topics covered are: types of research, theoretical frameworks, designs and methodologies, instrument properties or psychometrics, costs/acceptability, ethics, infrastructure and support.

Twelve tips for designing and implementing a structured rater training in OSCEs

Background: Observation, rating, and grading are parts of the routine of the faculty when in contact with medical students. Most interestingly, there is a dearth of proper instructions on rater training for objective structured clinical examinations (OSCEs) in medical education literature. Aims: We assembled 12 tips for implementing a structured rater training program. To highlight these tips, we provided examples of our implementation of rater training for an OSCE in 2009. Methods: We developed a tailored rater training program by compiling information from medical education and other related research disciplines. Results: The structured rater training at hand proved feasible, and the acceptance rate of this training by examiners was high.

Taking the history in patients with swallowing disorders: an international multidisciplinary survey

PurposeClinical assessment of swallowing disorders (dysphagia) requires accurate and comprehensive medical history-taking to further tailor the diagnostic work-up, but functional health care questionnaires show a large variability and various limitations. The aim of this study was to assess the way in which international swallowing experts from various disciplines asses swallowing problems in order to improve the radiologist´s ability to take a thorough medical history in this specific patient group.MethodsA two-step Delphi method was used to collect swallowing experts’ ways of taking the medical history in patients with swallowing disorders. The questions obtained in a first interview round were pooled and structured by dividing them into general and specific questions, including several subcategories, and these were scored by the experts in a second step based on to their clinical relevance.ResultsEighteen experts provided 25 different questions categorized as general questions and 34 dimension-specific questions (eight attributed to ‘suspicion of aspiration,’ 13 to ‘dysphagia,’ six to ‘globus sensation,’ four to ‘non-cardiac chest pain,’ and three to ‘effect of life.’) In the second interview round, the experts´ average predictive values attributed to those questions showed the varying importance of the presented items. Seven general and 13 specific questions (six of them attributed to ‘effect on life’ and seven ‘others’) were also added.ConclusionsThis collection of questions reflects the fact that a multidisciplinary approach when obtaining the medical history in patients with swallowing disorders may contribute to an improved technique for performing a symptom-oriented medical history-taking for radiologists of all training levels.

Information technology and social sciences: how can health IT be used to support the health professional?

Keeping up to date with the increasing amount of health‐related knowledge and managing the increasing numbers of patients with more complex clinical problems is a challenge for healthcare professionals and healthcare systems. Health IT applications, such as electronic health records or decision‐support systems, are meant to support both professionals and their support systems. However, for physicians using these applications, the applications often cause new problems, such as the impracticality of their use in clinical practice. This review adopts a social sciences perspective to understand these problems and derive suggestions for further development. Indeed, humans use tools to remediate the brains weaknesses and enhance thinking. Available health IT tools have been shaped to fit administrative needs rather than physicians’ needs. To increase the beneficial effect of health IT applications in health care, clinicians’ style of thinking and their learning needs must be considered when designing and implementing such systems. New health IT tools must be shaped to fit health professionals’ needs. To further ease the integration of new health IT tools into clinical practice, we must also consider the effects of implementing new tools on the wider social framework.

Desire and reality – teaching and assessing communicative competencies in undergraduate medical education in German-speaking Europe – a survey

Objectives: Increasingly, communicative competencies are becoming a permanent feature of training and assessment in German-speaking medical schools (n=43; Germany, Austria, Switzerland – ”D-A-CH”). In support of further curricular development of communicative competencies, the survey by the “Communicative and Social Competencies” (KusK) committee of the German Society for Medical Education (GMA) systematically appraises the scope of and form in which teaching and assessment take place. Methods: The iterative online questionnaire, developed in cooperation with KusK, comprises 70 questions regarding instruction (n=14), assessment (n=48), local conditions (n=5), with three fields for further remarks. Per location, two to three individuals who were familiar with the respective institute’s curriculum were invited to take part in the survey. Results: Thirty-nine medical schools (40 degree programmes) took part in the survey. Communicative competencies are taught in all of the programmes. Ten degree programmes have a longitudinal curriculum for communicative competencies; 25 programmes offer this in part. Sixteen of the 40 programmes use the Basler Consensus Statement for orientation. In over 80% of the degree programmes, communicative competencies are taught in the second and third year of studies. Almost all of the programmes work with simulated patients (n=38) and feedback (n=37). Exams are exclusively summative (n=11), exclusively formative (n=3), or both summative and formative (n=16) and usually take place in the fifth or sixth year of studies (n=22 and n=20). Apart from written examinations (n=15) and presentations (n=9), practical examinations are primarily administered (OSCE, n=31); WPA (n=8), usually with self-developed scales (OSCE, n=19). With regards to the examiners’ training and the manner of results-reporting to the students, there is a high variance. Conclusions: Instruction in communicative competencies has been implemented at all 39 of the participating medical schools. For the most part, communicative competencies instruction in the D-A-C-H region takes place in small groups and is tested using the OSCE. The challenges for further curricular development lie in the expansion of feedback, the critical evaluation of appropriate assessment strategies, and in the quality assurance of exams.

The Effects of Reusing Written Test Items: A Study Using the Rasch Model

Background. Relevant literature reports no increase in individual scores when test items are reused, but information on change in item difficulty is lacking. Purpose. To test an approach for quantifying the effects of reusing items on item difficulty. Methods. A total of 671 students sat a newly introduced exam in four testing shifts. The test forms experimentally combined published, unused, and reused items. Figures quantifying reuse effects were obtained using the Rasch model to compare item difficulties from different person samples. Results. The observed decrease in mean item difficulty for reused items was insignificant. Students who self-scheduled to the last test performed worse than other students did. Conclusion. Availability of leaked material did not translate into higher individual scores, as mastering leaked material does not guarantee transfer of knowledge to new exam items. Exam quality will not automatically deteriorate when a low ratio of randomly selected items is reused.

Monitoring communication with patients: analyzing judgments of satisfaction (JOS)

Medical students struggle to put into practice communication skills learned in medical school. In order to improve our instructional designs, better insight into the cause of this lack of transfer is foundational. We therefore explored students’ cognitions by soliciting self-evaluations of their history-taking skills, coined ‘judgments of satisfaction (JOSs)’. Our cognitive-psychological approach was guided by Koriat’s cue-utilization framework (J Exp Psychol Gen 126:349–370. doi:10.1037/0096-3445.126.4.349, 1997) which rests on the assumption that internal and external cues inform learners’ metacognitive judgments, which, in turn, steer their actions. Judgments based on unsuitable cues will cause ineffective behavior. Consequently, students are unable to adequately master these skills or properly apply them in similar situations. For the analysis, we had 524 medical undergraduates select scenes they were satisfied or dissatisfied with from their video-recorded simulated-patient encounters and explain why. Twenty transcripts were sampled for directed content analysis. We found that approximately one-third of students’ judgments focused on content (JOS-type-a); about half on the quality of the communication skills (JOS-type-b); and about ten percent targeted the appropriateness of the skills harnessed (JOS-type-c). This lack of reflection on appropriateness may explain why students experience problems adapting to new situations. It was primarily high-performance students who formed type-c judgments; poor performers tended to give type-a and type-b judgments. Future research would benefit from the use of our modified version of Koriat’s framework in order to further explore how high and poor performing medical students differ in the way they form JOSs during communications skills training.

Comparison of rest-break interventions during a mentally demanding task

Abstract Research is scarce on ways to enhance the effect of rest breaks during mentally demanding tasks. The present study investigated the effectiveness of two rest‐break interventions on well‐being during an academic lecture. Sixty‐six students (53 females, mean age 22.5 years) enrolled in two different university classes of 4‐hr duration participated in the study. Two measures of well‐being (fatigue and vigor) were assessed immediately before, after, and 20 minutes after the break. A control condition without a break as well as an unstructured break was compared with breaks either encompassing physical activity or a relaxation exercise. Compared with the nonbreak condition, the unstructured rest break led to an increase in vigor, the exercise break as well as the relaxation break both to an increase in vigor and a decrease in fatigue at 20‐min post break. Compared with the unstructured break, exercise led to an (additional) increase in vigor and relaxation to an (additional) decrease in fatigue at 20‐min post break. Thus, the effects of rest breaks during mentally demanding tasks can be enhanced by engaging in physical activity or relaxation exercises, with effects lasting at least as long as 20 min into the continuation of the task.

Using patient cases to educate health professionals, patients, institutions, and society: the swallowing disorder example

The problems and symptoms of patients are at the heart of medicine and pertinent to information, communication, and education. Patients present their problems to health professionals looking for relief and a cure, while institutions collect data about symptoms of patients and the way health professionals treated them to support societys care providers in administrating and managing care. The information on the problems of patients and the way health professionals treated them, which is gathered and stored in patient files, is valuable as educational material in the field of medicine. Driven by the fast developments of new technical tools case‐based information, communication and education starts to enter educational domains beyond the field of medicine, such as school systems and public health services. In our discussion we seek to understand and outline how to use the material gathered and stored in patient files to educate health professionals. Additionally, this paper gives examples of case‐based education for all members of society.

Psychologische Tätigkeiten im Bereich der Qualitätssicherung für Medizinische Prüfungen der Medizinischen Universität Wien

Was kennzeichnet den guten Arzt/die gute Arztin, und wie bildet man sie aus? Die Diskussion um die Merkmale „Arztlicher Professionalitat“ sowie die Strategien zur Gestaltung von Curricula, welche die Entwicklung der gewunschten Merkmale fordern, ist seit Ende der 80er Jahre Gegenstand der Diskussion in einschlagigen internationalen Fachgesellschaften (z. B. Arbeitskreises Medizinerausbildung der Robert Bosch Stiftung, 1989; Association of American Medical Colleges, 1998) und wissenschaftlichen Zeitschriften (z. B. Hawkins et al.2009). In Osterreich wird diese Diskussion auch unter Beteiligung der Offentlichkeit gefuhrt (siehe z. B. den Kommentar von Smolle, 2009). Obwohl weder die internationale, noch die nationale Diskussion daruber, welche Merkmale ein guter Arzt/eine gute Arztin aufweisen soll, abgeschlossen ist, herrscht Einigkeit daruber, dass angehende Arzt- Innen medizinisches Wissen und Fertigkeiten, sowie adaquate Haltung gegenuber PatientInnen und KollegInnen im Studium erwerben und anschliesend erweitern und aufrechterhalten sollen.