Michael J. Cullen

On Interpreting Stereotype Threat as Accounting for African American-White Differences on Cognitive Tests.

C. M. Steele and J. Aronson (1995) showed that making race salient when taking a difficult test affected the performance of high-ability African American students, a phenomenon they termed stereotype threat. The authors document that this research is widely misinterpreted in both popular and scholarly publications as showing that eliminating stereotype threat eliminates the African American-White difference in test performance. In fact, scores were statistically adjusted for differences in students prior SAT performance, and thus, Steele and Aronsons findings actually showed that absent stereotype threat, the two groups differ to the degree that would be expected based on differences in prior SAT scores. The authors caution against interpreting the Steele and Aronson experiment as evidence that stereotype threat is the primary cause of African American-White differences in test performance. Language: en

Using SAT-Grade and Ability-Job Performance Relationships to Test Predictions Derived from Stereotype Threat Theory

To examine the generalizability of stereotype threat theory findings from laboratory to applied settings, the authors developed models of the pattern of relationships between cognitive test scores and outcome criteria that would be expected if the test scores of women and minority group members were affected by stereotype threat. Two large data sets were used to test these models, one in an education setting examining SAT-grade relationships by race and gender and the other in a military job setting examining Armed Services Vocational Aptitude Battery-job performance relationships by race. Findings were not supportive of the predictions arising from stereotype threat theory, suggesting caution in positing threat as a key determinant of subgroup mean test score differences in applied settings.

Testing Stereotype Threat Theory Predictions for Math-Identified and Non-Math-Identified Students by Gender

This study follows in the tradition of Cullen, Hardison, and Sackett (2004) by testing the generalizability of stereotype threat theory findings from laboratory to applied settings. Like Cullen et al., the authors developed models of the pattern of relationships between Scholastic Assessment Test (SAT) math scores and English grades that would be expected for math-identified and non-math-identified men and women if stereotype threat were operating to suppress the scores of math-identified women during SAT test administration. The study builds on Cullen et al. by creating an alternative measure of identification with the math domain that is premised on high school students intention to major in math or a math-related discipline during college. Results using this alternative measure of identification were not supportive of predictions arising from stereotype threat theory, reinforcing Cullen et al.s call for caution in generalizing stereotype threat theory lab findings to real-world testing environments.

Racial/ethnic subgroup differences in cognitive ability test range restriction: Implications for differential validity

Recent meta-analyses demonstrated that the observed correlation between cognitive ability test scores and performance criteria was lower for Black and Hispanic subgroups than for Asian and White subgroups in college admissions, civilian employment, and military domains (i.e., differential validity). Given mean score differences between racial/ethnic subgroups, these observed validities may have been confounded by subgroup differences in range restriction. The present study draws on data from hundreds of cognitive ability test validity studies including more than 1 million persons to investigate whether Asian, Black, Hispanic, and White subgroups have differed in amounts of range restriction. We first replicated observed differential validity results and also extended them by presenting the first meta-analytic evidence that observed cognitive ability test validity is lower for the Hispanic subgroup in civilian employment settings. All subgroups were approximately equivalently restricted in range in college admissions and civilian employment domains, but the Black subgroup was more restricted in range than the White subgroup in military studies. In all 3 domains, any differences in range restriction could not account for observed validity differences between subgroups. We also provide estimates of range-restriction-corrected validities; Black and Hispanic subgroups corrected validities were 11.3-18.0% lower than White corrected validities across domains.

Individual Differences in the Effectiveness of Error Management Training for Developing Negotiation Skills

This study investigated whether the effectiveness of an error management approach to training negotiation knowledge and skill depended on individual differences in conscientiousness, extraversion, and openness to experience. Participants were randomly assigned to two training programs that incorporated key elements of an error management and behavioral modeling approach to training, and were trained in the complex interpersonal skill of negotiation. At the end of training, declarative knowledge acquisition, procedural knowledge acquisition, declarative knowledge retention, and transfer performance were assessed at different points in time. Results suggested that the effectiveness of the error management training program was dependent on individual levels of conscientiousness and extraversion. For several learning outcomes, the performance of highly conscientious and extraverted individuals was superior in the error management condition, while the performance of less conscientious and introverted individuals was superior in the behavioral modeling condition. The implications of these findings, and suggestions for future research, are discussed.

Operational threats to the use of SJTS: Faking, coaching, and retesting issues

Contents: R. Pritchard, Series Foreword. Preface. J.A. Weekley, R.E. Ployhart, An Introduction to Situational Judgment Testing. Part I:Theory. T.L. Gessner, R.J. Klimoski, Making Sense of Situations. M.E. Brooks, S. Highhouse, Can Good Judgment Be Measured? S.J. Motowidlo, A.C. Cooper, H.L. Jackson, A Theoretical Basis for Situational Judgment Tests. R.E. Ployhart, The Predictor Response Process Model. S.E. Stemler, R.J. Sternberg, Using Situational Judgment Tests to Measure Practical Intelligence. Part II: Measurement. N. Schmitt, D. Chan, Situational Judgment Tests: Method or Construct? J.A. Weekley, R.E. Ployhart, B.C. Holtz, On the Development of Situational Judgment Tests: Issues in Item Development, Scaling, and Scoring. M.A. McDaniel, D.L. Whetzel, N.S. Hartman, N.T. Nguyen, W.L. Grubb, III, Situational Judgment Tests: Validity and an Integrative Model. A.C. Hooper, M.J. Cullen, P.R. Sackett, Operational Threats to the Use of SJTs: Faking, Coaching, and Retesting Issues. T.N. Bauer, D.M. Truxillo, Applicant Reactions to Situational Judgment Tests: Research and Related Practical Issues. Part III: Application. J.B. Olson-Buchanan, F. Drasgow, Multimedia Situational Judgment Tests: The Medium Creates the Message. F. Lievens, International Situational Judgment Tests. B.A. Fritzsche, K.C. Stagl, E. Salas, C.S. Burke, Enhancing the Design, Delivery, and Evaluation of Scenario-Based Training: Can Situational Judgment Tests Contribute? T.V. Mumford, M.A. Campion, F.P. Morgeson, Situational Judgment in Work Teams: A Team Role Typology. R.E. Ployhart, J.A. Weekley, Situational Judgment: Some Suggestions for Future Science and Practice.

Not all unprofessional behaviors are equal: The creation of a checklist of bad behaviors

Abstract Introduction: Professionalism is a key component of medical education and training. However, there are few tools to aid educators in diagnosing unprofessional behavior at an early stage. The purpose of this study was to employ policy capturing methodology to develop two empirically validated checklists for identifying professionalism issues in early-career physicians. Method: In a series of workshops, a professionalism competency model containing 74 positive and 70 negative professionalism behaviors was developed and validated. Subsequently, 23 subject matter experts indicated their level of concern if each negative behavior occurred 1, 2, 3, 4, or 5 or more times during a six-month period. These ratings were used to create a “brief” and “extended” professionalism checklist for monitoring physician misconduct. Results: This study confirmed the subjective impression that some unprofessional behaviors are more egregious than others. Fourteen negative behaviors (e.g. displaying obvious signs of substance abuse) were judged to be concerning if they occurred only once, whereas many others (e.g. arriving late for conferences) were judged to be concerning only when they occurred repeatedly. Discussion: Medical educators can use the professionalism checklists developed in this study to aid in the early identification and subsequent remediation of unprofessional behavior in medical students and residents.