Nu Viet Vu

Six years of comprehensive, clinical, performance-based assessment using standardized patients at the Southern Illinois University School of Medicine

By the end of 1990–91, the Southern Illinois University School of Medicine had had six years of experience with comprehensive, performance-based examinations of senior medical students levels of clinical competence; this report assesses the psychometric aspects of the six examinations given during that period. The examinations were aimed at determining the students readiness for postgraduate training. Compared with other clinical performance-based assessments that use standardized patients (SPs), these examinations had two important and unique features: (1) the examinations assessed a comprehensive range of clinical skills and reasoning; and (2) they approximated the challenges of real clinical practice wherein a practitioners skills need to be orchestrated and prioritized in order to meet the challenges of the case encountered. Each year, the performance-based assessment given was an intensive clinical examination requiring each student to work up 13 to 18 SP problems over a three-day period. To administer an examination to an entire class of students took three weeks. Because all students after the first year of administration (1986) were required to pass these examinations, the fairness of test design and scoring and the setting of performance standards for the examinations became important issues for the faculty. The results, accumulated over six years and based on a total of 6,804 student-patient encounters involving 405 students, indicate that this kind of clinical performance-based examination can discriminate a wide range of students clinical performances. The results provide evidence for the examinations test security, content validity, construct validity, and reliability.

Development of clinical reasoning from the basic sciences to the clerkships: a longitudinal assessment of medical students' needs and self-perception after a transitional learning unit

Background u2002To facilitate students transition from basic, science‐oriented, problem‐based learning (PBL) to clinical reasoning‐oriented PBL, the University of Geneva School of Medicine introduced a 12‐week unit of Introduction to Clinical Reasoning (ICR) at the beginning of its fourth or clerkship year.

Effects of Examinee Gender, Standardized-Patient Gender, and Their Interaction on Standardized Patients' Ratings of Examinees' Interpersonal and Communication Skills.

PURPOSE. To assess the effects of examinee gender, standardized-patient (SP) gender, and, in particular, their interaction on ratings made by SPs of examinees interpersonal and communication skills in a performance-based examination of clinical competence. METHOD. The examination was administered to four classes of senior medical students (about 70 per class) at Southern Illinois University School of Medicine, 1988–1991. The skill dimensions tested were clarity of communication, thoroughness of explanation, professional manner, personal manner, and overall patient satisfaction. Split-plot analyses of variance were used. RESULTS. There was no interaction of examinee gender and SP gender for any of the five rating scales. There was no main effect of examinee gender for four of the five scales; however, for personal manner, women students were rated slightly higher than men students. There was a main effect of SP gender, but the effect was not consistent from rating scale to rating scale or from class to class. Nevertheless, differences in ratings given by men and women SPs should not be of psychometric concern, since the ratings of men and women examinees are necessarily affected alike. CONCLUSIONS. Except for the women examinees higher performance in personal manner, the men and women examinees generally performed equally well with respect to interpersonal and communication skills, and they performed equally well regardless of the gender of the SP.

Developing and evaluating the student assessment system in the preclinical problem-based curriculum at Sherbrooke.

BACKGROUND: Students learning was used as an outcome measure in the first phases of the major curriculum reform started in 1987 by the Universite de Sherbrooke Faculty of Medicine, which shifted from a traditional to a student-centered, problem-based learning (PBL) and community-oriented program. The system for evaluating preclinical students learning is intended to reinforce the integration of basic and clinical sciences. METHOD: To discover whether the evaluation system was fulfilling its intended goals, the authors used data from the classes of 1991-1993 to assess the reliability and validity of three evaluation instruments. The three instruments were (1) written examinations composed of multiple-choice questions (MCQs), short-answer questions (SAQs), and problem-analysis questions (PAQs); (2) PBL tutor rating forms that evaluate students reasoning skills, communication and group-interaction skills, and autonomy and humanism; and (3) clinical skills evaluations, including objective structured clinical examinations (OSCEs). The weights allocated to the instruments reflected how the faculty valued each evaluation dimension in each of the three phases of the preclinical curriculum. RESULTS: Reliability indexes improved throughout the system implementation. The written examinations proved to have content validity according to the PBL learning objectives. As evaluated by students, the PAQs were found to be at a taxonomic level that assessed ability to analyze information a third of the time in the first year of implementation of the PBL curriculum and 17% in the second year. Variations and correlations of students mean performances across instructional units and between the evaluation instruments led to the development of a student longitudinal performance profile to be used before yearly promotion decisions are proposed. The profile was introduced in the fifth year of PBL implementation. CONCLUSION: The system allows students to learn higher-taxonomic-level content and fulfills the institutions social responsibility of judging program outcomes and promoting qualified students, although evaluation by PBL tutors is still psychometrically questionable and the measurement of students reasoning and ability to analyze problems is still an unfinished evaluation task.

Interprofessional Collaboration on an Internal Medicine Ward: Role Perceptions and Expectations among Nurses and Residents

Background Effective interprofessional collaboration requires that team members share common perceptions and expectations of each others roles. Objective Describe and compare residents’ and nurses’ perceptions and expectations of their own and each other’s professional roles in the context of an Internal Medicine ward. Methods A convenience sample of 14 residents and 14 nurses volunteers from the General Internal Medicine Division at the University Hospitals of Geneva, Switzerland, were interviewed to explore their perceptions and expectations of residents’ and nurses’ professional roles, for their own and the other profession. Interviews were analysed using thematic content analysis. The same respondents also filled a questionnaire asking their own intended actions and the expected actions from the other professional in response to 11 clinical scenarios. Results Three main themes emerged from the interviews: patient management, clinical reasoning and decision-making processes, and roles in the team. Nurses and residents shared general perceptions about patient management. However, there was a lack of shared perceptions and expectations regarding nurses’ autonomy in patient management, nurses’ participation in the decision-making process, professional interdependence, and residents’ implication in teamwork. Results from the clinical scenarios showed that nurses’ intended actions differed from residents’ expectations mainly regarding autonomy in patient management. Correlation between residents’ expectations and nurses’ intended actions was 0.56 (pu200a=u200a0.08), while correlation between nurses’ expectations and residents’ intended actions was 0.80 (p<0.001). Conclusions There are discordant perceptions and unmet expectations among nurses and residents about each other’s roles, including several aspects related to the decision-making process. Interprofessional education should foster a shared vision of each other’s roles and clarify the boundaries of autonomy of each profession.

Comparing checklists and databases with physicians' ratings as measures of students' history and physical-examination skills

PURPOSE. To compare two methods of rating students performances on history and physical examination: (1) by using checklists completed by standardized patients (SPs) and databases completed by students, and (2) by using ratings of students by three physicians for each SP-student encounter. METHOD. Four cases were chosen for the study, and 30 students were examined per case. The students were all in their fourth year at the Southern Illinois University School of Medicine in the spring of 1991. Two of the cases had both checklists and databases, and the remaining two had databases only. Each SP-student encounter was videotaped and was viewed independently by three physicians unfamiliar with the contents of the checklists and databases. The physicians pooled ratings were then compared with the checklist and database scores. Uncorrected and corrected correlations were obtained, with the generalizability coefficient used as the index of reliability. RESULTS. Interrater generalizability of physicians ratings was very good, ranging from .65 to .93 for overall ratings. Generalizability of physicians ratings pooled across the four cases was .85. Checklist scores tended to correlate higher with physicians ratings than did database scores: across the cases, correlation coefficients between physicians ratings and checklist scores and database scores were .65 and .39, respectively. CONCLUSION. The checklist scores correlated strongly with the physicians ratings of history and physical-examination skills, providing some evidence of validity for their use. The checklist scores correlated much better with the physicians ratings than did the database scores. Possible explanations for this finding are discussed.

Test security in examinations that use standardized-patient cases at one medical school

The use of performance-based examinations consisting of standardized-patient (SP) cases has increased greatly in recent years. These examinations are typically long and thus require the presentation of the same SP cases to several consecutive examinee groups. Consequently, concerns have arisen about the potential for violations of test security whereby students who were tested early in the examination period pass on information to students tested later. These concerns are addressed using data from the SP-based examinations administered to five classes (1986–1990) of senior medical students at Southern Illinois University School of Medicine. Because of the length of the examinations, each class was randomly divided into five groups and the examination was administered to one group at a time, requiring three days of testing time per group and three weeks of testing time per class. The results showed no consistent, systematic changes in case means across the five groups tested at different times throughout the examination period, and thus provide no evidence of serious, widespread violations of test security.

Interprofessional Collaboration between Residents and Nurses in General Internal Medicine: A Qualitative Study on Behaviours Enhancing Teamwork Quality

Background Effective teamwork is necessary for optimal patient care. There is insufficient understanding of interactions between physicians and nurses on internal medicine wards. Objective To describe resident physicians’ and nurses’ actual behaviours contributing to teamwork quality in the setting of a simulated internal medicine ward. Methods A volunteer sample of 14 pairs of residents and nurses in internal medicine was asked to manage one non-urgent and one urgent clinical case in a simulated ward, using a high-fidelity manikin. After the simulation, participants attended a stimulated-recall session during which they viewed the videotape of the simulation and explained their actions and perceptions. All simulations were transcribed, coded, and analyzed, using a qualitative method (template analysis). Quality of teamwork was assessed, based on patient management efficiency and presence of shared management goals and of team spirit. Results Most resident-nurse pairs tended to interact in a traditional way, with residents taking the leadership and nurses executing medical prescriptions and assuming their own specific role. They also demonstrated different types of interactions involving shared responsibilities and decision making, constructive suggestions, active communication and listening, and manifestations of positive team building. The presence of a leader in the pair or a truly shared leadership between resident and nurse contributed to teamwork quality only if both members of the pair demonstrated sufficient autonomy. In case of a lack of autonomy of one member, the other member could compensate for it, if his/her own autonomy was sufficiently strong and if there were demonstrations of mutual listening, information sharing, and positive team building. Conclusions Although they often relied on traditional types of interaction, residents and nurses also demonstrated readiness for increased sharing of responsibilities. Interprofessional education should insist on better redefinition of respective roles and reinforce behaviours shown to enhance teamwork quality.

Teaching Diagnostic Skills: Clinical Vignettes or Chief Complaints?

Two formats of case presentation are traditionally used for teaching problem-solving skills: clinical vignette or chief complaint formats. While the first one is more commonly used, it does not completely reflect the actual problem-solving process during a real encounter, which may hamper the learners to integrate separately acquired data gathering skills into their reasoning process and affect their diagnostic performance in practice. The present study compared diagnostic accuracy when the reasoning stimulus was a case vignette containing all diagnostic information versus the patients chief complaint only. Forty-two medical students, 53 residents and 60 general internists participated in the study. Diagnostic accuracy was significantly lower for the chief complaint format at the student, resident, and practitioner levels. Analysis of the data gathered in the chief-complaint format revealed that faulty diagnostic decisions resulted from a failure to gather critical data. The results suggest that data gathering techniques, semiology, and medical reasoning should be trained in association and that this effort should be pursued beyond medical school.

Effect of teaching context and tutor workshop on tutorial skills

Effective faculty development workshops are essential to develop and sustain the quality of facultys teaching. In an integrated problem-based curriculum, tutors expressed the needs to further develop their skills in facilitating students’ content learning and small-group functioning. Based on the authors’ prior observations that tutors’ performance depends on their teaching context, a workshop was designed not only tailored to the tutors’ needs but also organized within their respective teaching unit. The purposes of this study are (1) to evaluate whether this workshop is effective and improves tutors’ teaching skills, and (2) to assess whether workshop effectiveness depends on tutors’ performance before the workshop and on their teaching unit environment. Workshop effectiveness was assessed using (a) tutors’ perception of workshop usefulness and of their improvement in tutorial skills, and (b) students’ ratings of tutor performance before and after the workshop. In addition, an analysis of variance model was designed to analyse how tutors’ performance before the workshop and their teaching unit influence workshop effectiveness. Tutors judged the workshop as helpful in providing them with new teaching strategies and reported having improved their tutorial skills. Workshop attendance enhanced students’ ratings of tutors’ knowledge of problem content and ability to guide their learning. This improvement was also long-lasting. The workshop effect on tutor performance was relative: it varied across teaching units and was higher for tutors with low scores before the workshop. A workshop tailored to tutors’ needs and adapted to their teaching unit improves their tutorial skills. Its effectiveness is, however, influenced by tutors’ level of performance before the workshop and by the environment of their teaching unit. Thus, to be efficient, the design of a workshop should consider not only individual tutors’ needs, but also the background of their teaching units, with special attention to their internal organization and tutor group functioning.