Kent G. Hecker

How to Use Structural Equation Modeling in Medical Education Research: A Brief Guide

Background: Structural equation modeling (SEM) is a family of statistical techniques used for the analysis of multivariate data to measure latent variables and their interrelationships. SEM has potential to advance theory and research in medical education. Purpose: The purpose of this article is to introduce SEM to medical education researchers and provide procedural information for applying SEM. Methods: We outline the basic tenets of SEM, principles of model creation, identification, estimation, and model fit to data, and the use of SEM in medical education research. Results: Although it is a powerful statistical research tool, SEM has had only limited use in medical education research. We explicate a five-step procedure for applying SEM to research problems and summarize an example of SEM to test a hypothetical model. Conclusions: Notwithstanding some pitfalls, SEM does provide promise for testing complex, integrated theoretical models and advance research in medical education.

How Much Do Differences in Medical Schools Influence Student Performance? A Longitudinal Study Employing Hierarchical Linear Modeling

Background: Medical school curricula have undergone considerable change in the past half century. There is little evidence, however, for the impact of various curricula and educational policies on student learning once incoming performance and the nonrandom nature of students nested within schools has been accounted for. Purpose: To investigate effects of school variables on United States Medical Licensing Examination (USMLE) Step 1–3 scores over an 11-year period (1994–2004). Methods: Using Association of American Medical Colleges and USMLE longitudinal data for 116 medical schools, hierarchical linear modeling was used to study the effects of school variables on Step 1–3. Results: Mean unadjusted between school variance was 14.74%, 10.50%, and 11.25%, for USMLE Step 1–3. When student covariates were included, between-school variation was less than 5%. The proportion of variance accounted for in-student-level performance by the covariates ranged from 27.58% to 36.51% for Step 1, 16.37% to 24.48% for Step 2, and 19.22% to 25.32% for Step 3. The variance accounted for in student performance by the student covariates ranged between 81.22% and 88.26% for Step 1, 48.44% and 79.77% for Step 2, and 68.41% and 80.78% for Step 3. School-level variables did not consistently predict for adjusted mean school Step performance. Conclusions: Individual student differences account for most of the variation in USMLE performance with small contributions from between-school variation and even smaller contribution from curriculum and educational policies.

Assessment of Applicants to the Veterinary Curriculum Using a Multiple Mini-Interview Method

This study describes the development, implementation, and psychometric assessment of the multiple mini-interview (MMI) for the inaugural class of veterinary medicine applicants at the University of Calgary Faculty of Veterinary Medicine (UCVM). The MMI is a series of approximately five to 12 10-minute interviews that consist of situational events. Applicants are given a scenario and asked to work through an issue or behavioral-type questions that are meant to assess one attribute (e.g., empathy) at a time. This structure allows for multiple assessments of the applicants by trained interviewers on the same questions. MMI scenario development was based on a review of the noncognitive attributes currently assessed by the 31 veterinary schools across Canada and the United States and the goals and objectives of UCVM. The noncognitive attributes of applicants (N=110) were assessed at five stations, by two interviewers within each station, on three items using a standardized rating form on an anchored 1-5 scale. The method was determined to be reliable (G-coefficient=0.88) and demonstrated evidence of validity. The MMI score did not correlate with grade-point average (r=0.12, p=0.22). While neither the applicants nor interviewers had participated in an MMI format before, both groups reported the process to be acceptable in a post-interview questionnaire. This analysis provides preliminary evidence of the reliability, validity, and acceptability of the MMI in assessing the noncognitive attributes of applicants for veterinary medical school admissions.

Reliability of 400-cell and 5-field leukocyte differential counts for equine bronchoalveolar lavage fluid

BACKGROUND Reliable enumeration of mast cells and eosinophils in equine bronchoalveolar lavage (BAL) fluid is important because small increases in the percentages of these cells support the clinical diagnosis of inflammatory airway disease (IAD). Increases in BAL neutrophils also occur with IAD but are not specific due to overlap between IAD and recurrent airway obstruction (RAO). OBJECTIVES The objectives of this study were to evaluate the reliability of a standard 400-cell leukocyte differential count and an alternate method evaluating 5 microscopic fields at 500× magnification in equine BAL fluid cytocentrifuged preparations. METHODS BAL samples from 60 horses with and without pulmonary inflammation were evaluated using 400-cell and 5-field leukocyte differential counting methods. Reliability of enumeration of each leukocyte type was assessed by calculating and comparing intraclass correlation coefficients (ICC). Reliability of mast cell enumeration was further evaluated by comparing ICCs of slides with different cell densities. RESULTS Reliability was higher for all cell types with the 5-field method; however, overall the difference between methods was not statistically significant. Neutrophil reliability was high (ICC > 0.90) with both methods. Adequate reliability (ICC > 0.85) for mast cells was achieved only with the 5-field method on slides with higher cell density. CONCLUSION Enumeration of mast cells is unreliable when the standard 400-cell differential counting method is used, whereas the 5-field method on slides with higher cell density reached acceptable reproducibility. Neutrophil percentages were highly reliable with both methods.

A Generalizability Analysis of a Veterinary School Multiple Mini Interview: Effect of Number of Interviewers, Type of Interviewers, and Number of Stations

Background: The number of Multiple Mini Interview (MMI) stations and the type and number of interviewers required for an acceptable level of reliability for veterinary admissions requires investigation. Purpose: The goal is to investigate the reliability of the 2009 MMI admission process at the University of Calgary. Methods: Each applicant (n = 103; female = 80.6%; M age = 23.05 years, SD = 3.96) participated in a 7-station MMI. Applicants were rated independently by 2 interviewers, a faculty member, and a community veterinarian, within each station (total interviewers/applicant N = 14). Interviewers scored applicants on 3 items, each on a 5-point anchored scale. Results: Generalizability analysis resulted in a reliability coefficient of G = 0.79. A Decision study (D-study) indicated that 10 stations with 1 interviewer would produce a G = 0.79 and 8 stations with 2 interviewers would produce a G = 0.81; however, these have different resource requirements. A two-way analysis of variance showed that there was a nonsignificant main effect of interviewer type (between faculty member and community veterinarian) on interview scores, F(1, 1428) = 3.18, p = .075; a significant main effect of station on interview scores, F(6, 1428) = 4.34, p < .001; and a nonsignificant interaction effect between interviewer-type and station on interview scores, F(6, 1428) = 0.74, p = .62. Conclusions: Overall reliability was adequate for the MMI. Results from the D-study suggest that the current format with 7 stations provides adequate reliability given that there are enough interviewers; to achieve the same G-coefficient 1 interviewer per station with 10 stations would suffice and reduce the resource requirements. Community veterinarians and faculty members demonstrated an adequate level of agreement in their assessments of applicants.

Understanding Reliability: A Review for Veterinary Educators

Veterinary medical faculty and administrators routinely administer student assessments and conduct surveys to make decisions regarding student performance and to assess their courses/curricula. The decisions that are made are a result of the scores generated. However, how reliable are the scores and how confident can we be about these decisions? Reliability is one of the hallmarks of validity evidence, but what does this mean and what affects the reliability of scores? The purpose of this article is to provide veterinary medical educators and administrators with fundamental information regarding the concept of reliability. Specifically, we review what sources of error reduce the reliability of scores and we describe the different types of reliability coefficients that are reported.

What should we be selecting for? a systematic approach for determining which personal characteristics to assess for during admissions

BackgroundAdmission committees are responsible for creating fair, defensible, reliable, and valid processes that assess those attributes considered important for professional success. There is evidence for the continuing use of academic ability as a selection criterion for health professional schools; however, there is little evidence for the reliability and validity of measures currently in place to assess personal characteristics. The Ontario Veterinary College (OVC) initiated a review of its admissions criteria in order to implement an evidence-based method to determine which characteristics veterinary stakeholders consider important to assess for admission.MethodsEleven characteristics were identified by the OVC Admissions Committee and a survey was sent to all licensed veterinarians in Ontario (n=4,068), OVC students (n=450), and OVC faculty, interns and residents (n=192). A paired comparison method was used to identify the relative rank order of the characteristics, and multivariate analysis of variance with post hoc analyses was used to determine between group differences in the returned survey data.ResultsSurveys were returned from 1,312 participants (27.86% response rate; female 59.70%). The relative rank of the characteristics was reasonably consistent among participant groups, with ethical behaviour, sound judgment, communication, and critical and creative thinking being ranked as the top four. However, the importance of certain characteristics like communication and empathy were perceived differently by groups. For instance, females scored communication (F(1, 1289) = 20.24, p < .001, d = .26) and empathy (F(1, 1289) = 55.41, p < .001, d = 0.42) significantly higher than males, while males scored knowledge of profession (F(1, 1289) = 12.81, p < .001, d = 0.20), leadership (F(1, 1289) = 10.28, p = .001, d = 0.18), and sound judgment (F(1, 1289) = 13.56, p < .001, d = 0.21) significantly higher than females.ConclusionsThe data from the paired comparison method provide convergent evidence for the characteristics participant groups identify as most important in determining who should be admitted to a veterinary program. The between group analyses provides important information regarding characteristics most important to various subgroups; this has implications for what characteristics are selected for at admission as well as on who is selecting for them.

Validity, Reliability, and Defensibility of Assessments in Veterinary Education

In this article, we provide an introduction to and overview of issues of validity, reliability, and defensibility related to measurement of student performance in veterinary medical education. Validity has to do with the extent to which the instrument measures whatever it is supposed to measure, reliability has to do with the consistency of measurement, and defensibility deals with the appropriate standards of the assessment procedure. An explanation of the methods that can be used to determine reliability and validity are given and examples of how they have been used in recent research findings are provided. Veterinary educators should have an understanding of each of these concepts because evaluation of veterinary students and performance measures provides evidence of utility of assessment tools, informs teaching practices, and can guide curriculum development and revision.

Student Attainment of Proficiency in a Clinical Skill: The Assessment of Individual Learning Curves

The aims of this study were to determine if the learning curve cumulative summation test (LC-CUSUM) can differentiate proficiency in placing intravenous catheters by novice learners, and identify the cause of failure when it occurred. In a prospective, observational study design 6 undergraduate students with no previous experience of placing intravenous catheters received standardized training by a board certified veterinary anesthesiologist in intravenous catheter placement technique. Immediately following training, each student attempted 60 intravenous catheterizations in a dog mannequin thoracic limb model. Results were scored as a success or failure based upon completion of four specific criteria, and where catheter placement failure occurred, the cause was recorded according to pre-defined criteria. Initial acceptable and unacceptable failure rates were set by the study team and the LC-CUSUM was used to generate a learning curve for each student. Using 10% and 25% acceptable and unacceptable failure rates, 3 out of 6 students attained proficiency, requiring between 26 to 48 attempts. Applying 25% and 50% acceptable and unacceptable failure rates, 5 of 6 students obtained proficiency, requiring between 18 and 55 attempts. Wide inter-individual variability was observed and the majority of failed catheterisation attempts were limited to two of the four pre-defined criteria. These data indicate that the LC-CUSUM can be used to generate individual learning curves, inter-individual variability in catheter placement ability is wide, and that specific steps in catheter placement are responsible for the majority of failures. These findings may have profound implications for how we teach and assess technical skills.

Leadership competencies for medical education and healthcare professions: Population-based study

Objective To identify and empirically investigate the dimensions of leadership in medical education and healthcare professions. Design A population-based design with a focus group and a survey were used to identify the perceived competencies for effective leadership in medical education. Setting The focus group, consisting of five experts from three countries (Austria n=1; Germany n=2; Switzerland n=2), was conducted (all masters of medical education), and the survey was sent to health professionals from medical schools and teaching hospitals in six countries (Austria, Canada, Germany, Switzerland, the UK and the USA). Participants The participants were educators, physicians, nurses and other health professionals who held academic positions in medical education. A total of 229 completed the survey: 135 (59.0%) women (mean age=50.3 years) and 94 (41.0%) men (mean age=51.0 years). Measures A 63-item survey measuring leadership competencies was developed and administered via electronic mail to participants. Results Exploratory principal component analyses yielded five factors accounting for 51.2% of the variance: (1) social responsibility, (2) innovation, (3) self-management, (4) task management and (5) justice orientation. There were significant differences between physicians and other health professionals on some factors (Wilks λ=0.93, p<0.01). Social responsibility was rated higher by other health professionals (M=71.09) than by physicians (M=67.12), as was innovation (health professionals M=80.83; physicians M=76.20) and justice orientation (health professionals M=21.27; physicians M=20.46). Conclusions The results of the principal component analyses support the theoretical meaningfulness of these factors, their coherence, internal consistency and parsimony in explaining the variance of the data. Although there are some between-group differences, the competencies appear to be stable and coherent.