Kenneth D. Royal

Psychometric Properties of the Death Anxiety Scale (DAS) Among Terminally Ill Cancer Patients

Research conducted with the terminally ill population in relation to death anxiety is rare and mostly outdated. The purpose of this study was to evaluate the psychometric properties of the widely used Death Anxiety Scale (DAS) on a sample of terminal cancer patients. Additionally, validation studies of the DAS have exclusively used traditional statistical methods for analysis. The current study utilized an item response theory technique (IRT), namely the Rasch Rating Scale model for data analysis. The methodology employed may be useful for other researchers conducting validation studies from an IRT perspective.

Understanding Reliability: A Review for Veterinary Educators

Veterinary medical faculty and administrators routinely administer student assessments and conduct surveys to make decisions regarding student performance and to assess their courses/curricula. The decisions that are made are a result of the scores generated. However, how reliable are the scores and how confident can we be about these decisions? Reliability is one of the hallmarks of validity evidence, but what does this mean and what affects the reliability of scores? The purpose of this article is to provide veterinary medical educators and administrators with fundamental information regarding the concept of reliability. Specifically, we review what sources of error reduce the reliability of scores and we describe the different types of reliability coefficients that are reported.

THE CONSEQUENTIAL VALIDITY OF ABFM EXAMINATIONS

Measurement scholar, Samuel Messick, defines validity as “an integrated evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores….”[1][1] (p13) Messick’s definition of

Applying the Rasch Rating Scale Model to Gain Insights into Students' Conceptualisation of Quality Mathematics Instruction

Teacher quality has become a national policy concern in the US, especially in mathematics. This study provides insights into the conceptualisation of high quality mathematics teaching from the perspective of approximately 750 students in grades nine through twelve. Results from Rasch analysis yield information about the quality of theMathematics Quality Survey constructed for this study and the hierarchy of items representing varying levels of quality as perceived by the students. Group interviews with teachers and open-ended responses from students are used to support the findings. This study lays the groundwork for understanding the difference between being qualified and being a quality teacher.

Using rasch measurement to score, evaluate, and improve examinations in an anatomy course

Any examination that involves moderate to high stakes implications for examinees should be psychometrically sound and legally defensible. Currently, there are two broad and competing families of test theories that are used to score examination data. The majority of instructors outside the high‐stakes testing arena rely on classical test theory (CTT) methods. However, advances in item response theory software have made the application of these techniques much more accessible to classroom instructors. The purpose of this research is to analyze a common medical school anatomy examination using both the traditional CTT scoring method and a Rasch measurement scoring method to determine which technique provides more robust findings, and which set of psychometric indicators will be more meaningful and useful for anatomists looking to improve the psychometric quality and functioning of their examinations. Results produced by the more robust and meaningful methodology will undergo a rigorous psychometric validation process to evaluate construct validity. Implications of these techniques and additional possibilities for advanced applications are also discussed. Anat Sci Educ 7: 450–460.

Achievement Goal Validation among African American High School Students: CFA and Rasch Results.

Achievement goal theory helps describe how and why students engage in various academic behaviors. Historically, achievement goals have been examined almost exclusively with undergraduate, nonminority samples, and predominately with factor analytic techniques. The present study adds to a growing literature by providing initial validation of a leading achievement goal measure, the Achievement Goal Questionnaire-Revised (AGQ-R; Elliot & Murayama, 2008), among rural (N = 186) and urban (N = 197) African American high school students. Collectively, results from both confirmatory factor and Rasch analyses highlight issues that should be considered when using the AGQ-R among African American high school students.

Performance on the American Board of Family Medicine (ABFM) Certification Examination: Are Superior Test-Taking Skills Alone Sufficient to Pass?

Introduction: Certification examinations used by American specialty boards have been the sine qua non for demonstrating the knowledge sufficient for attainment of board certification in the United States for more than 75 years. Some people contend that the examination is predominantly a test of superior test-taking skills rather than of family medicine decision-making ability. In an effort to explore the validity of this assertion, we administered the American Board of Family Medicine (ABFM) Certification to examinees who had demonstrated proficiency in taking standardized tests but had limited medical knowledge. Methods: Four nonphysician experts in the field of measurement and testing were administered one version of the 2009 ABFM certification examination. Scaled scores were calculated for each examinee, and psychometric analyses were performed on the examinees responses to examination items and compared with the performance of physicians who took the same examination. Results: The minimum passing threshold for the examination was a scaled score of 390, corresponding to 57.7% to 61.0% of questions answered correctly, depending on the version of the examination. The 4 nonphysician examinees performed poorly, with scaled scores that ranged from 20 to 160 (mean, 87.5; SD, 57.4). The number of questions answered correctly ranged from 24.0% to 35.1% (mean, 29.2%; SD, 0.05%). Rasch analyses of the examination items revealed that the nonphysician examinees were more likely to use guessing strategies in an effort to answer questions correctly. Distracter analysis suggest near-complete randomness in the nonphysician responses. Conclusions: Though all 4 nonphysician examinees performed better than would have been predicted by chance alone, none performed well enough to even fall within 8 SE below the passing thresholds; their performance was far below that of almost all physicians who completed the examination. Given that the nonphysicians relied heavily on the identifying cues in the phrasing of items and the manner in which response options were presented, the results affirm the notion that the ABFM certification examination is not primarily a measure of generic test-taking ability but measures information critical to the estimation of a family physicians knowledge sufficient for certification. Item analysis confirmed that items were well written, provided minimal cueing, and required medical knowledge to answer correctly.

A Closer Look at Recertification Candidate Pass Rates

We continue to receive a number of inquiries from physicians regarding the low pass rate statistics for recertification candidates for the reporting periods from 2010 to 2012. Many physicians fear the examination has changed in some way, putting those attempting to maintain their certification at a

Results of a Flipped Classroom Teaching Approach in Anesthesiology Residents

BACKGROUND In a flipped classroom approach, learners view educational content prior to class and engage in active learning during didactic sessions. OBJECTIVE We hypothesized that a flipped classroom improves knowledge acquisition and retention for residents compared to traditional lecture, and that residents prefer this approach. METHODS We completed 2 iterations of a study in 2014 and 2015. Institutions were assigned to either flipped classroom or traditional lecture for 4 weekly sessions. The flipped classroom consisted of reviewing a 15-minute video, followed by 45-minute in-class interactive sessions with audience response questions, think-pair-share questions, and case discussions. The traditional lecture approach consisted of a 55-minute lecture given by faculty with 5 minutes for questions. Residents completed 3 knowledge tests (pretest, posttest, and 4-month retention) and surveys of their perceptions of the didactic sessions. A linear mixed model was used to compare the effect of both formats on knowledge acquisition and retention. RESULTS Of 182 eligible postgraduate year 2 anesthesiology residents, 155 (85%) participated in the entire intervention, and 142 (78%) completed all tests. The flipped classroom approach improved knowledge retention after 4 months (adjusted mean = 6%; P = .014; d = 0.56), and residents preferred the flipped classroom (pre = 46%; post = 82%; P < .001). CONCLUSIONS The flipped classroom approach to didactic education resulted in a small improvement in knowledge retention and was preferred by anesthesiology residents.

Academic outcomes of a community-based longitudinal integrated clerkships program

Abstract Background: Longitudinal integrated clerkships (LICs) receive recognition internationally as effective, innovative alternatives to traditional block rotations (TBRs) in undergraduate medical education. No studies of LICs in the USA have assessed how students perform on all the standardized exams. Aim: To compare performance on standardized tests of students in the first four years of LICs at the University of North Carolina School of Medicine–Asheville (UNC SOM–Asheville) with students from UNC SOM’s Chapel Hill main campus in TBRs. Methods: LIC and TBR students’ previous academic performance was considered using Medical College Admissions Test (MCAT) and United States Medical Licensing Examination (USMLE) Step 1 scores. Step 1 exam tests students’ pre-clinical, basic science knowledge. Outcome measures included all eight standardized National Board of Medical Examiners (NBME) Subject Shelf Examinations and USMLE Step 2 Clinical Knowledge (CK) examinations, which are used widely in the US to assess students’ progress and as prerequisites to eventual licensure. TBR students were selected using propensity scores to match LIC students. Groups were also compared on the required core clinical conditions documented, and on residency specialty choice. Results: Asheville LIC students earned higher scores on the Step 2 CK examination and the six shelf examinations linked to longitudinal clerkships than the matched TBR students (Step 2 CK exam, Family Medicine and Ambulatory Medicine shelf exams reached statistical significance). LIC students logged greater percentages of core conditions than TBR students and more often chose primary care residencies. Conclusions: UNC School of Medicine medical students participating in a longitudinal integrated curriculum in a community setting outperformed fellow students who completed a more TBR curriculum within the school’s academic medical center. Differences were found in performance on standard tests of clinical knowledge (six NBME exams and Step 2 CK exam), documented breadth of clinical experiences, and likelihood of choosing primary care residency programs.