Matthias Holzer

A modified electronic key feature examination for undergraduate medical students: validation threats and opportunities

The purpose of our study was the development and validation of a modified electronic key feature exam of clinical decision-making skills for undergraduate medical students. Therefore, the reliability of the test (15 items), the item difficulty level, the item-total correlations and correlations to other measures of knowledge (40 item MC-test and 580 items of German MC-National Licensing Exam, Part II) were calculated. Based on the guidelines provided by the Medical Council of Canada, a modified electronic key feature exam for internal medicine consisting of 15 key features (KFs) was developed for fifth year German medical students. Long menu (LM) and short menu (SM) question formats were used. Acceptance was assessed through a questionnaire. Thirty-seven students from four medical schools voluntarily participated in the study. The reliability of the key feature exam was 0.65 (Cronbachs alpha). The items’ difficulty level scores were between 0.3 and 0.8 and the item-total correlations between 0.0 and 0.4. Correlations between the results of the KF exam and the other measures of knowledge were intermediate (r between 0.44 and 0.47) as well as the learners’ level of acceptance. The modified electronic KF examination is a feasible and reliable evaluation tool that may be implemented for the assessment of clinical undergraduate training.

Using electronic flashcards to promote learning in medical students: retesting versus restudying

Medical Education 2011: 45: 1101–1110

Student learning preferences and the impact of a multimedia learning tool in the dissection course at the University of Munich

How do students use multimedia tools to support their learning during a gross anatomy dissection course? We investigated this question in the anatomic dissection course for first year medical students at the University of Munich (n=850) by giving all participating students access to an anatomical multimedia CD-ROM. The use of this multimedia learning tool was voluntary and it was not essential for the end-of-course examination. After the examination, two questionnaires (return rate first questionnaire n(1)=347, approximately 41%; return rate second questionnaire n(2)=644, approximately 76%) were given to the surveyed students with the following content: evaluation of the multimedia learning tool, details about the usage of different kinds of available learning media, and finally an evaluation of the media used for teaching during the course. Furthermore we collected personal data from participants such as age, gender and the score achieved in the examination. Classical textbooks and anatomical atlases were used by 84% of students for preparation. The multimedia learning tool was used by 34% as an additional media for learning. The multimedia learning tool was not used alone. The data showed differences with regard to gender and performance of students, but not relating to age. Students rated the computer-specific features, e.g. three-dimensional (3D) models, virtual simulations, and an interactive quiz module, as major reasons for using the multimedia learning tool. Our results show that medical students use anatomic multimedia learning tools primarily as an additional medium for learning, and thus lead to the conclusion that the main learning media are still textbooks and anatomic atlases.

Learning the facts in medical school is not enough: which factors predict successful application of procedural knowledge in a laboratory setting?

BackgroundMedical knowledge encompasses both conceptual (facts or “what” information) and procedural knowledge (“how” and “why” information). Conceptual knowledge is known to be an essential prerequisite for clinical problem solving. Primarily, medical students learn from textbooks and often struggle with the process of applying their conceptual knowledge to clinical problems. Recent studies address the question of how to foster the acquisition of procedural knowledge and its application in medical education. However, little is known about the factors which predict performance in procedural knowledge tasks. Which additional factors of the learner predict performance in procedural knowledge?MethodsDomain specific conceptual knowledge (facts) in clinical nephrology was provided to 80 medical students (3rd to 5th year) using electronic flashcards in a laboratory setting. Learner characteristics were obtained by questionnaires. Procedural knowledge in clinical nephrology was assessed by key feature problems (KFP) and problem solving tasks (PST) reflecting strategic and conditional knowledge, respectively.ResultsResults in procedural knowledge tests (KFP and PST) correlated significantly with each other. In univariate analysis, performance in procedural knowledge (sum of KFP+PST) was significantly correlated with the results in (1) the conceptual knowledge test (CKT), (2) the intended future career as hospital based doctor, (3) the duration of clinical clerkships, and (4) the results in the written German National Medical Examination Part I on preclinical subjects (NME-I). After multiple regression analysis only clinical clerkship experience and NME-I performance remained independent influencing factors.ConclusionsPerformance in procedural knowledge tests seems independent from the degree of domain specific conceptual knowledge above a certain level. Procedural knowledge may be fostered by clinical experience. More attention should be paid to the interplay of individual clinical clerkship experiences and structured teaching of procedural knowledge and its assessment in medical education curricula.

Pick-N Multiple Choice-Exams: A Comparison of Scoring Algorithms.

To compare different scoring algorithms for Pick-N multiple correct answer multiple-choice (MC) exams regarding test reliability, student performance, total item discrimination and item difficulty. Data from six 3rd year medical students’ end of term exams in internal medicine from 2005 to 2008 at Munich University were analysed (1,255 students, 180 Pick-N items in total). Scoring Algorithms: Each question scored a maximum of one point. We compared: (a) Dichotomous scoring (DS): One point if all true and no wrong answers were chosen. (b) Partial credit algorithm 1 (PS50): One point for 100% true answers; 0.5 points for 50% or more true answers; zero points for less than 50% true answers. No point deduction for wrong choices. (c) Partial credit algorithm 2 (PS1/m): A fraction of one point depending on the total number of true answers was given for each correct answer identified. No point deduction for wrong choices. Application of partial crediting resulted in psychometric results superior to dichotomous scoring (DS). Algorithms examined resulted in similar psychometric data with PS50 only slightly exceeding PS1/m in higher coefficients of reliability. The Pick-N MC format and its scoring using the PS50 and PS1/m algorithms are suited for undergraduate medical examinations. Partial knowledge should be awarded in Pick-N MC exams.

Cognitive problem solving patterns of medical students correlate with success in diagnostic case solutions.

Context Problem-solving in terms of clinical reasoning is regarded as a key competence of medical doctors. Little is known about the general cognitive actions underlying the strategies of problem-solving among medical students. In this study, a theory-based model was used and adapted in order to investigate the cognitive actions in which medical students are engaged when dealing with a case and how patterns of these actions are related to the correct solution. Methods Twenty-three medical students worked on three cases on clinical nephrology using the think-aloud method. The transcribed recordings were coded using a theory-based model consisting of eight different cognitive actions. The coded data was analysed using time sequences in a graphical representation software. Furthermore the relationship between the coded data and accuracy of diagnosis was investigated with inferential statistical methods. Results The observation of all main actions in a case elaboration, including evaluation, representation and integration, was considered a complete model and was found in the majority of cases (56%). This pattern significantly related to the accuracy of the case solution (φ = 0.55; p<.001). Extent of prior knowledge was neither related to the complete model nor to the correct solution. Conclusions The proposed model is suitable to empirically verify the cognitive actions of problem-solving of medical students. The cognitive actions evaluation, representation and integration are crucial for the complete model and therefore for the accuracy of the solution. The educational implication which may be drawn from this study is to foster students reasoning by focusing on higher level reasoning.

Knowledge is not enough to solve the problems - The role of diagnostic knowledge in clinical reasoning activities

BackgroundClinical reasoning is a key competence in medicine. There is a lack of knowledge, how non-experts like medical students solve clinical problems. It is known that they have difficulties applying conceptual knowledge to clinical cases, that they lack metacognitive awareness and that higher level cognitive actions correlate with diagnostic accuracy. However, the role of conceptual, strategic, conditional, and metacognitive knowledge for clinical reasoning is unknown.MethodsMedical students (n = 21) were exposed to three different clinical cases and instructed to use the think-aloud method. The recorded sessions were transcribed and coded with regards to the four different categories of diagnostic knowledge (see above). The transcripts were coded using the frequencies and time-coding of the categories of knowledge. The relationship between the coded data and accuracy of diagnosis was investigated with inferential statistical methods.ResultsThe use of metacognitive knowledge is correlated with application of conceptual, but not with conditional and strategic knowledge. Furthermore, conceptual and strategic knowledge application is associated with longer time on task. However, in contrast to cognitive action levels the use of different categories of diagnostic knowledge was not associated with better diagnostic accuracy.ConclusionsThe longer case work and the more intense application of conceptual knowledge in individuals with high metacognitive activity may hint towards reduced premature closure as one of the major cognitive causes of errors in medicine. Additionally, for correct case solution the cognitive actions seem to be more important than the diagnostic knowledge categories.

Improvements in teaching behavior at two German medical schools resulting from a modified Flanders interaction analysis feedback intervention process

Abstract Background: Person-centered teachers who are more empathic and “indirect” (accept, encourage, praise and ask questions) tend to be more effective than those who are “direct” (lecture, give directions and criticize) (Amidon & Flanders 1991). The Flanders Interaction Analysis (FIA) is a tool for diagnosing these teaching aspects, though not yet used to improve lecturing in undergraduate medical education. Aims: Does structured expert feedback to volunteer lecturers lead to improvement in person-centered teaching behavior as measured by a Modified Flanders Interaction Analysis (MFIA) and student questionnaires? Methods: Twenty-one volunteer lecturers from two German medical faculties were stratified by past teaching experience and randomized into two groups. The intervention group received MFIA diagnoses of their lectures plus feedback by an expert observer after winter and summer semester lectures, respectively. The control group was only diagnosed with the MFIA. Teaching behavior changes for both groups were compared and teacher feedback about the intervention process was assessed. Results: Faculty in the intervention group improved significantly in their summer lectures regarding person-centered teaching behavior while controls did not. Conclusions: A structured individual expert feedback intervention using a MFIA as a teaching diagnostic tool is a powerful, cost-effective faculty development process for improving teaching behavior of volunteer lecturers in undergraduate medical education.

Exams at medical faculties - quality, responsibility and perspectives.

Der Ausschuss Prüfungen der GMA hat zusammen mit dem Kompetenzzentrum für Prüfungen in der Medizin in Baden-Württemberg 2008 eine Leitlinie für Fakultäts-interne schriftliche Leistungsnachweise während des Medizinstudiums publiziert [1]. In der Folge wurde die Umsetzung dieser Leitlinie als Qualitätsstandard lebhaft diskutiert. Eine Umfrage zum Ist-Zustand der Prüfungen in fakultärer Verantwortung wurde im Auftrag desMedizinischen Fakultätentages (MFT) von Heidelberg aus durchgeführt und veröffentlicht [2]. Dabei wurde klar, dass bisher schriftliche Prüfungen im Multiple-Choice (MC) Format vorherrschen und eine Reihe von Verbesserungsmöglichkeiten imSinne eines allgemeinenQualitätsstandards für Planung, Fragenerstellung, Review, Durchführung, Auswertung und Ergebnismitteilung bestehen. Nur eine kleine Minderheit der Fächer setzt bisher klinisch-praktische Prüfungsformate wie den OSCE ein, wenn auch mittlerweile fast jede Fakultät in zumindest einem Fach einen OSCE durchführt. Eine Vergleichbarkeit der Noten zwischen den Fakultäten ist aufgrund formal und inhaltlich unterschiedlicher Prüfungen nicht gegeben. Zur Qualitätssicherung und zur Verbesserung des interfakultären Austausches wurden Kompetenznetzwerke für die medizinische Lehre gegründet – zunächst in BadenWürttemberg und dann in Bayern. Das Thema Prüfungen spielt dabei eine zentrale Rolle. Beide Netzwerke mit jeweils fünfmedizinischen Fakultäten haben die Umsetzung der GMA-Leitlinien für fakultäre Prüfungen abgefragt [3], [4]. Erfreulicherweise werden die Leitlinien von allen Fakultäten beachtet und zu einem größeren Teil bereits umgesetzt. Wie bei der MFT-Umfrage [2] ergibt sich aber auch hier ein differenziertes Bild mit unterschiedlichen Umsetzungsrealitäten zwischen den Fakultäten. Das Erreichte ist erfreulich, aber die Weiterentwicklung der Prüfungsqualität und der überfakultäre Austausch dazu gehören weiterhin ganz oben auf die Agenda. Nie-

Using the Progress Test Medizin (PTM) for evaluation of the Medical Curriculum Munich (MeCuM)

Aims: The Medical Curriculum Munich (MeCuM) has been implemented since 2004 and was completely established in 2007. In this study the clinical part of MeCuM was evaluated with respect to retention of the knowledge in internal medicine (learning objectives of the 6th/7th semester). Methods: In summer of 2009 and winter of 2009/2010 1065 students participated in the Progress Test Medizin (PTM) from Charité Medical School Berlin. Additionally the students answered a questionnaire regarding the acceptance and rating of the progress test and basic demographic data. Results: The knowledge of internal medicine continuously increases during the clinical part of the medical curriculum in Munich. However, significant differences between the sub-disciplines of internal medicine could be observed. The overall acceptance of the PTM was high and increased further with the study progress. Interestingly, practical experiences like clinical clerkships positively influenced the test score. Conclusions: The PTM is a useful tool for the evaluation of knowledge retention in a specific curriculum.