Elisabeth A. van Hell

Necessary steps in factor analysis : Enhancing validation studies of educational instruments. The PHEEM applied to clerks as an example

Background: The validation of educational instruments, in particular the employment of factor analysis, can be improved in many instances. Aims: To demonstrate the superiority of a sophisticated method of factor analysis, implying an integration of recommendations described in the factor analysis literature, over often employed limited applications of factor analysis. We demonstrate the essential steps, focusing on the Postgraduate Hospital Educational Environment Measure (PHEEM). Method: The PHEEM was completed by 279 clerks. We performed Principal Component Analysis (PCA) with varimax rotation. A combination of three psychometric criteria was applied: scree plot, eigenvalues >1.5 and a minimum percentage of additionally explained variance of approximately 5%. Furthermore, four interpretability criteria were used. Confirmatory factor analysis was performed to verify the original scale structure. Results: Our method yielded three interpretable and practically useful dimensions: learning content and coaching, beneficial affective climate and external regulation. Additionally, combining several criteria reduced the risk of overfactoring and underfactoring. Furthermore, the resulting dimensions corresponded with three learning functions essential to high-quality learning, thus strengthening our findings. Confirmatory factor analysis disproved the original scale structure. Conclusions: Our sophisticated approach yielded several advantages over methods applied in previous validation studies. Therefore, we recommend this method in validation studies to achieve best practice.

Transition to clinical training : influence of pre-clinical knowledge and skills, and consequences for clinical performance

Context  Many students experience a tough transition from pre‐clinical to clinical training and previous studies suggest that this may constrict students’ progress. However, clear empirical evidence of this is lacking. The aim of this study was to determine: whether the perceived difficulty of transition influences student performance during the first 2 weeks of clerkships; whether it influences students’ overall performance in their first clerkship, and the degree to which the difficulty of transition is influenced by students’ pre‐clinical knowledge and skills levels.

Key Elements in Assessing the Educational Environment: Where Is the Theory?.

The educational environment has been increasingly acknowledged as vital for high-quality medical education. As a result, several instruments have been developed to measure medical educational environment quality. However, there appears to be no consensus about which concepts should be measured. The absence of a theoretical framework may explain this lack of consensus. Therefore, we aimed to (1) find a comprehensive theoretical framework defining the essential concepts, and (2) test its applicability. An initial review of the medical educational environment literature indicated that such frameworks are lacking. Therefore, we chose an alternative approach to lead us to relevant frameworks from outside the medical educational field; that is, we applied a snowballing technique to find educational environment instruments used to build the contents of the medical ones and investigated their theoretical underpinnings (Study 1). We found two frameworks, one of which was described as incomplete and one of which defines three domains as the key elements of human environments (personal development/goal direction, relationships, and system maintenance and system change) and has been validated in different contexts. To test its applicability, we investigated whether the items of nine medical educational environment instruments could be mapped unto the framework (Study 2). Of 374 items, 94% could: 256 (68%) pertained to a single domain, 94 (25%) to more than one domain. In our context, these domains were found to concern goal orientation, relationships and organization/regulation. We conclude that this framework is applicable and comprehensive, and recommend using it as theoretical underpinning for medical educational environment measures.

Time spent on clerkship activities by students in relation to their perceptions of learning environment quality

Context  Students’ perceptions of their learning environment are of great importance to their learning process. In this study we assessed the time allocated by students to clerkship activities and the relationship between students’ allocations of time and their perceptions of the quality of their clinical learning environment.

Instructiveness of feedback during clerkships: Influence of supervisor, observation and student initiative

Background: Several authors assume that the supervisors role, observation of behaviour and students’ active participation are important factors in the instructiveness of feedback. Aim: This study aims to provide empirical evidence for these expectations. Methods: For two weeks, 142 clerks from eight hospitals recorded for each individual feedback event: who provided the feedback, whether the feedback was based on observation of behaviour, who initiated the feedback moment and the perceived instructiveness of the feedback. Data were analysed with multilevel techniques. Results: The perceived instructiveness of feedback provided by specialists and residents did not differ significantly. However, both were perceived to be more instructive than feedback from nursing and paramedical staff (βspecialists = 0.862, p < 0.01; βresidents = 0.853, p < 0.01). Feedback on behaviour that had been directly observed was reported to be more instructive than feedback on behaviour that had not been observed (βobserved = 0.314, p < 0.001). Feedback which stemmed from student initiative or a joint initiative was experienced to be more instructive than feedback which ensued from the supervisors initiative (βstudent = 0.441, p < 0.01; βjoint = 0.392, p < 0.01). Conclusions: The expectations concerning the influence of observation and student initiative on the instructiveness of feedback were confirmed in this empirical study. Expected differences in instructiveness between feedback from specialists and residents were not confirmed.

The effect of implementing undergraduate competency-based medical education on students' knowledge acquisition, clinical performance and perceived preparedness for practice: a comparative study

BackgroundLittle is known about the gains and losses associated with the implementation of undergraduate competency-based medical education. Therefore, we compared knowledge acquisition, clinical performance and perceived preparedness for practice of students from a competency-based active learning (CBAL) curriculum and a prior active learning (AL) curriculum.MethodsWe included two cohorts of both the AL curriculum (n = 453) and the CBAL curriculum (n = 372). Knowledge acquisition was determined by benchmarking each cohort on 24 interuniversity progress tests against parallel cohorts of two other medical schools. Differences in knowledge acquisition were determined comparing the number of times CBAL and AL cohorts scored significantly higher or lower on progress tests. Clinical performance was operationalized as students’ mean clerkship grade. Perceived preparedness for practice was assessed using a survey.ResultsThe CBAL cohorts demonstrated relatively lower knowledge acquisition than the AL cohorts during the first study years, but not at the end of their studies. We found no significant differences in clinical performance. Concerning perceived preparedness for practice we found no significant differences except that students from the CBAL curriculum felt better prepared for ‘putting a patient problem in a broad context of political, sociological, cultural and economic factors’ than students from the AL curriculum.ConclusionsOur data do not support the assumption that competency-based education results in graduates who are better prepared for medical practice. More research is needed before we can draw generalizable conclusions on the potential of undergraduate competency-based medical education.

Alternating skills training and clerkships to ease the transition from preclinical to clinical training

Background: The transition from preclinical to clinical training is perceived as stressful with a high workload being the main difficulty. To ease this transition, we implemented a dual learning year, where just-in-time skills training and clerkships alternated. Aims: To examine the effect of the dual learning year on students’ stress and perceptions of workload and skills level, and to compare these data with a baseline measurement from a curriculum in which skills training was provided in advance of clerkships. Method: During the first Master year, students completed four rotations in which 5 weeks of skills training and 5-week clerkships alternated. In the second clerkship week of each rotation, students (n = 476) completed questionnaires measuring their experienced workload, perceived skills level and stress. Analysis of variance was used for trend analysis and to determine differences with the baseline measurement (n = 83). Results: During the dual learning year, ‘experienced workload’ decreased (F(1,471) = 9.408, p < 0.01), ‘perceived skills level’ increased (F(1,471) = 94.202, p < 0.001) and stress decreased (F(1,471) = 3.309, p < 0.10). ‘Experienced workload’ was lower (F(5,553) = 7.599, p < 0.001) and ‘perceived skills level’ was comparable to the baseline measurement. Conclusions: Compared to the baseline measurement and the results of earlier studies on transition difficulties, alternating just-in-time skills training and clerkships seem to ease the transition from preclinical to clinical training.

Exploring cultural differences in feedback processes and perceived instructiveness during clerkships: Replicating a Dutch study in Indonesia

Context: Cultural differences between countries may entail differences in feedback processes. Aims: By replicating a Dutch study in Indonesia, we analysed whether differences in processes influenced the perceived instructiveness of feedback. Methods: Over a two-week period, Indonesian students (n = 215) recorded feedback moments during clerkships, noting who provided the feedback, whether the feedback was based on observations, who initiated the feedback, and its perceived instructiveness. Data were compared with the earlier Dutch study and analysed with χ2 tests, t-tests and multilevel techniques. Cultural differences were explored using Hofstedes Model, with Indonesia and the Netherlands differing on “power distance” and “individualism.” Results: Perceived instructiveness of feedback did not differ significantly between both countries. However, significant differences were found in feedback provider, observation and initiative. Indonesian students perceived feedback as more instructive if provided by specialists and initiated jointly by the supervisor and student (βresidents = −0.201, p < 0.001 and βjoint = 0.193, p = 0.001). Dutch students appreciated feedback more when it was based on observation. Conclusions: We obtained empirical evidence that one model of feedback does not necessarily translate to another culture. Further research is necessary to unravel other possible influences of culture in implementing feedback procedures in different countries.

The digital pen as a novel device to facilitate the feedback process

Background: To improve clinical performance assessment, checklist data should be immediately available to students to offer them detailed feedback and be stored in a database for quality assurance purposes. Aim: To introduce the digital pen as clinical performance assessment tool, report examiner satisfaction and explore the utility of generated checklist data for quality assurance purposes. Methods: The digital pen technology transmits examiners’ handwritten assessments to a database and exports PDF-files to students’ mailboxes. Descriptive statistical analysis of examiner satisfaction and the generated checklist data was performed. Results: The examiners were satisfied with the digital pen. Valuable data were obtained to improve objective structured clinical examination stations and rating criteria, identify training needs for future students and provide examiners with feedback on their rating skills. Conclusion: The digital pen technology is a practical device for sending completed checklists to students and providing valuable data for quality assurance purposes.

Evidence-based practice for individuals or groups

The aim of applying science into practice is to deliver high-quality health care. Thinking about teaching the necessary accompanying skills, a distinction can be made between using evidence for individual patient care and using scientific knowledge for the development of protocols or guidelines for groups of patients or professionals. In this paper, these two ways of applying science into practice are being considered. We plea for explicating the differences between the individual patient and a group of patients or professionals when applying scientific knowledge in the decision-making process. The acknowledgment of these differences facilitates the teaching of the accompanying competences and different CanMEDS roles.