Karen J. Mitchell

Traditional predictors of performance in medical school

This paper addresses the predictive value for performance in medical school of undergraduate grades, the Medical College Admission Test (MCAT), information on the selectivity of the undergraduate institution, and selected transcript data. The performance data examined were basic science grades; clinical science grades; scores on National Board of Medical Examiners examinations, Parts I, II, and III; and information on academic difficulty. Methodological sources of differences in validity data, including restriction in range, criterion attenuation, and method specificity, are discussed. Reported validity data affirm the substantial value of traditional academic predictors of performance in medical school. Selection committees should, nevertheless, supplement academic data with nonacademic and interview information. Their inclusion is particularly important for minority and disadvantaged applicants. Medical schools should assess the validity of their selection systems. Additional research on the relations between academic predictors and performance-based indexes of clinical competence and on the role of traditional predictors with regard to the recently declining medical school applicant pool is suggested.

Factorial Validity of the Armed Services Vocational Aptitude Battery (ASVAB), Forms 8, 9 and 10: 1981 Army Applicant Sample

The purpose of this paper was to examine the factor structure of the Armed Services Vocational Aptitude Battery (ASVAB), Forms 8, 9, and 10. This standardized multiple cognitive abilities battery is the primary selection and classification instrument used by all military services. It consists of eight power and two speed tests. In this investigation the ASVAB 8/9/10 factor structure was compared with that observed for previous samples and for previous forms of the ASVAB. The similarity of factor structure for racial/ethnic and sex subgroups was examined for the purpose of establishing the possible invariance of ASVAB factorial validity across groups. A factor analysis of a sample of 98,689 male and female Army applicants yielded four orthogonal factors accounting for 93% of the total variance: Verbal Ability, Speeded Performance, Quantitative Ability, and Technical Knowledge. Factor analyses of male, female, white, black, and Hispanic subgroups yielded similar results. These analyses provided evidence of the replicability of ability constructs across diverse samples.

An Interim Report on the MCAT Essay Pilot Project.

Results from four pilot administrations of the Medical College Admission Test essay question are reported. Analyses focused on (a) the performance characteristics of sample groups differentiated by gender, size of hometown, race/ethnicity, and dominant language; (b) the relationships between essay scores and academic/demographic characteristics; and (c) the reliability of one 45-minute versus two 30-minute essays. No differences were found for examinees grouped by gender and size of home community. Mean differences among the racial/ethnic groups were explained largely by reading level differences. Differences in essay performance by language group were large and unexplained by reading level differences. No relationship was found between the essay score and the academic/demographic characteristics. Reliability estimates for two 30-minute essays were higher than for one 45-minute essay; however, the 30-minute period yielded writing of poorer quality. Test-retest reliabilities for the 45-minute topics will remain the focus of future studies as will performance by examinees for whom English is a second language. The impact of the essay on the selection process will also be assessed.

Reliability of Holistic Scoring for the MCAT Essay

The Association of American Medical Colleges is conducting research on the incremental validity associated with adding an essay to the Medical College Admission Test (MCAT). The present research examined the reliability of holistic scoring for a sample of essays written during the Spring 1985 MCAT administration. Analysis of variance techniques were used to estimate the reliability of scoring and to partition score variance into that due to level differences between papers and to context-specific factors such as rater, time of scoring, etc. Recommendations for structuring and managing large-scale scoring sessions are presented based on the results.

Score Reporting for the 1991 Medical College Admission Test.

In December 1988, a 16-member advisory panel, made up of medical school deans, basic science faculty, clinical scientists, medical education researchers, medical students, premedical advisers, and practicing physicians, completed a full-scale review of the format, content, and score-reporting system of the Medical College Admission Test (MCAT). The advisory panels recommendations will shape the MCAT to be introduced in 1991. Presented here are data the panel used in deciding to keep the score scale for the upcoming MCAT. Survey data on student selection practices in U.S. and Canadian medical schools and acceptance information for 1987 applicants showed that medical schools used the scores differently within each school and between the schools in deciding whom to admit. The data demonstrated the value of the current reporting system and led the advisory panel to recommend retention of the 15-point MCAT score scale in 1991.

Factor Structure of the MCAT and Pilot Essay

The Association of American Medical Colleges is examining the desirability and feasibility of including an essay on the Medical College Admission Test (MCAT). This endeavor calls for administration of a trial essay with the 1985 through 1987 MCAT examinations. Research reported in this paper considered the construct validity of the essay and its effect on the measurement domain tapped by the operational MCAT. The objective was to examine the factor structure of the MCAT and pilot essay and compare it to structures observed for previous MCAT versions and examinee groups. Factor analyses of data for 2,876 Spring 1985 examinees yielded two factors: 1. a science/quantitative factor defined by the science tests and Skills Analysis: Quantitative, and 2. an analysis/communication factor including the two Skills Analysis exams and pilot essay. The addition of the essay caused the non-science factor observed in previous MCAT research to be more strongly defined. Factor analyses of data for male, female, black, white, Hispanic, and Asian examinees supported the invariance of factor content across samples.