Alan L. Hull

Validity of Three Clinical Performance Assessments of Internal Medicine Clerks.

PURPOSE. To analyze the construct validity of three methods to assess the clinical performances of internal medicine clerks. METHOD. A multitrait-multimethod (MTMM) study was conducted at the Case Western Reserve University School of Medicine to determine the convergent and divergent validity of a clinical evaluation form (CEF) completed by faculty and residents, an objective structured clinical examination (OSCE), and the medicine subject test of the National Board of Medical Examiners. Three traits were involved in the analysis: clinical skills, knowledge, and personal characteristics. A correlation matrix was computed for 410 third-year students who completed the clerkship between August 1988 and July 1991. RESULTS. There was a significant (p < .01) convergence of the four correlations that assessed the same traits by using different methods. However, the four convergent correlations were of moderate magnitude (ranging from .29 to .47). Divergent validity was assessed by comparing the magnitudes of the convergence correlations with the magnitudes of correlations among unrelated assessments (i.e., different traits by different methods). Seven of nine possible coefficients were smaller than the convergent coefficients, suggesting evidence of divergent validity. A significant CEF method effect was identified. CONCLUSION. There was convergent validity and some evidence of divergent validity with a significant method effect. The findings were similar for correlations corrected for attenuation. Four conclusions were reached: (1) the reliability of the OSCE must be improved, (2) the CEF ratings must be redesigned to further discriminate among the specific traits assessed, (3) additional methods to assess personal characteristics must be instituted, and (4) several assessment methods should be used to evaluate individual student performances.

Methods to assess students' acquisition, application and integration of basic science knowledge in an innovative competency-based curriculum.

Background: The Cleveland Clinic Lerner College of Medicine was designed to encourage medical students to pursue careers as physician investigators. Our faculty decided that assessment should enhance learning and adopted only formative assessments to document student performance in relation to nine broad-based competencies. No grades are used to judge student performance throughout the 5-year program. Instead, assessments are competency-based, relate directly to performance standards, and are stored in e-Portfolios to track progress and document student achievement. The class size is limited to 32 students a year. Aims: Schools with competency-based curricula must provide students with formative feedback to identify performance gaps and monitor progress. We describe a systematic approach to assess medical knowledge using essay-type questions (CAPPs) and multiple choice questions (SAQs) to provide medical students with weekly, formative feedback about their abilities to acquire, apply and integrate basic and clinical science concepts. Method: Processes for developing performance standards, creating assessment items, training faculty, reporting student performance and monitoring outcomes are described. A case study of a Year 1 course is presented with specific examples of CAPPs and SAQs to illustrate how formative assessment data are interpreted and reported in students’ e-Portfolios. Results: Preliminary evidence suggests that CAPPs and SAQs have a positive impact on students’ education, a justifiable cost in light of obtained benefits and growing acceptance among stakeholders. Two student cohorts performed significantly above the population mean on USMLE Step 1, which suggests that these assessment methods have not disadvantaged students. More evidence is needed to assess the reliability and validity of these tools for formative purposes. Conclusions: Using assessment data for formative purposes may encourage application and integration of knowledge, help students identify performance gaps, foster student development of learning plans and promote student responsibility for learning. Discussion provides applications for institutions with larger classes to consider.

Examination of a Clinical Teaching Effectiveness Instrument Used for Summative Faculty Assessment

This study explores whether a clinical teaching effectiveness (CTE) instrument provides valid scores for summative faculty assessment. The sample included all CTE instruments (n = 10,087) that learners (N = 1,194) completed to assess clinical teachers (N = 872) during 1 academic year. The authors investigated response processes (e.g., missing data, straight-line responses, level of learner), internal structure (e.g., confirmatory and exploratory factor analysis), teaching ratings by learner group (medical student or resident), and relation to other variables (e.g., correlation with global rating). Response processes identified a high prevalence of straight-line responses (same rating across all items) and differential patterns of missing data by learner group. Medical students rated their teachers higher than residents, and CTE scores had different factor structures depending on learner group. High correlation coefficients of CTE items with a single rating of overall teaching performance suggest that learners consider global performance when assessing clinical teaching performance.

Humanistic teaching attributes of primary care physicians

Little research exists on the humanistic qualities of the physician as teacher. This study examined the humanistic teaching attributes used by 30 primary care physicians when teaching medical students in the ambulatory setting. Three research questions were addressed: What humanistic teaching behaviors are demonstrated by physician‐teachers? How do these observed behaviors compare with medical students’ perceptions of physician behavior? How do these observed behaviors compare with the physician‐teachers’ perceptions of their own behaviors? Data from direct observation of the clinical teaching encounter between the physician‐teacher and medical student were compared with students’ perceptions of the humanistic behaviors demonstrated by the teacher and the teachers perception of the frequency with which these behaviors were used. Humanistic teaching attributes were observed to occur with moderate to high frequency. Findings are consistent with other studies suggesting little correlation between self‐repor...

Psychometric Properties of an Instrument Designed to Measure the Educational Quality of Graduate Training Programs

To assess the quality of residency education programs at an academic medical center for purposes of enhancing individual graduate medical education programs, we asked residents and fellows (N = 419) to evaluate their training programs using a Web-based questionnaire (response rate = 70%). Kruskal-Wallis tests, factor analysis, correlations, generalizability/decision studies, and mean plots were used to examine trainee responses and to assess the questionnaire’s measurement properties. Exploratory factor analysis indicated that the instrument had a threefactor structure that correlated highly with overall program rating. Cronbach’s alpha exceeded .80 for all factors, and decision studies revealed that 13 to 23 raters were needed to obtain G-coefficients greater than .70. Mean plots showed that the instrument could discriminate within and among training programs at the item level and the factor level, which should help target improvements across graduate training programs within large institutions.

Use of Patient Encounter Documentation (Log) Systems at Three Medical Schools.

This article describes uses and limitations of patient encounter documentation (log) systems in four programs at three medical schools. Curricular, psychometric, and administrative issues affecting their successful implementation are discussed, and recommendations to facilitate their use in program evaluation are offered.

The relationship between teachers’ confidence and use of clinical teaching skills in ambulatory care settings

Background: As medical schools expand their teaching programs into ambulatory care settings, faculty must learn to use new and more effective teaching skills. They must therefore be convinced of the importance of those skills and confident in using them. Purpose: Teachers’ confidence in their own teaching abilities (i.e., self‐efficacy) is examined in this study. Methods: We surveyed 145 primary care faculty about their feelings of confidence in and use of 28 specific teaching skills. Results: Faculty self‐ratings indicated that they were less confident in using learning‐enhancing teaching skills than in using supervisory behaviors that focus on patient care. Degree of confidence was most strongly associated with self‐ratings of frequency of use; physician experience and estimated length of time spent in teaching encounters did not mediate this relation. These findings suggest that physicians’ teaching behaviors may depend largely on their self‐confidence. Conclusions: We suggest that teacher attitudes an...

Case Western Reserve University School of Medicine and Cleveland Clinic.

University-based program (University track) was established in 1843 and Cleveland Clinic-based Cleveland Clinic Lerner College of Medicine (College track) was established in 2002. University track implemented a major curriculum revision in 2006, focusing on development of lifelong learners and incorporating themes of scholarship, clinical mastery, leadership, and civic professionalism. A dedicated four-month research requirement culminating in an MD thesis was incorporated into the four-year curriculum. A hybrid of active, student-centered PBL groups and teacher-centered large and medium group sessions form the backbone of learning venues. Five-year competency-based College track is designed to train graduates with excellent clinical skills, expertise in research, and a passion for scientific inquiry. Students complete a master’s level thesis and graduate with an MD degree with Special Qualification in Biomedical Research. While course format, assessment approaches, and curricular topics beyond core content of University and College tracks differ, clinical rotations for both tracks were designed collaboratively and implemented jointly.

A Comprehensive Research Curriculum to Prepare Physician-Investigators: Design, Implementation and Early Outcomes

The Cleveland Clinic Lerner College of Medicine (CCLCM) was established as one of three tracks in the Case Western Reserve University (CWRU) School of Medicine Program to prepare physician investigators through rigorous, comprehensive, longitudinal research training in a five-year undergraduate medical education setting. The curriculum includes specific research activities and dedicated research time, which culminates in a masterslevel thesis. The track has an administrative structure to facilitate student access to research-related opportunities. Basic scientists and clinicians participate in all curricular activities thereby fostering communication across the clinical medicine/research divide. The track features a centralized, portfolio-based assessment system. In this track, students submit portfolios to demonstrate their progress toward mastering nine competencies including research. A comprehensive program evaluation is in place to monitor the curricular impact on students. Results show that the CCLCM track has attracted a subset of students interested in research careers. Two cohorts of students (n = 56) have graduated. The Promotions Committee determined that all graduates successfully met the performance standards for the research competency. These graduates scored above mean USMLE Step 1 subscores in biostatistics and epidemiology. They successfully completed a research thesis where approximately 60% of the topics addressed basic science and translational research. Over 40% of graduates also obtained a masters degree concurrently with their medical degrees. Furthermore, 23% of CCLCM graduates received competitive awards to support their medical school research activities. Recommendations are included to expose students to research in multiple venues to create and sustain a passion for rigorous problem-solving Conclusions offer lessons learned from our experience.

Medical school applicants' essays as predictors of primary care career choice.

No abstract available.