Michael T. Brannick

Exploratory and confirmatory factor analysis: guidelines, issues, and alternatives

AMY E. HURLEY, TERRI A. SCANDURA, CHESTER A. SCHRIESHEIM, MICHAEL T. BRANNICK, ANSON SEERS, ROBERT J. VANDENBERG AND LARRY J. WILLIAMS Department of Professional Studies, Chapman University, U.S.A. Department of Management, University of Miami, U.S.A. Department of Psychology, University of South Florida, U.S.A. Department of Management, Virginia Commonwealth University, U.S.A. Department of Management, The University of Georgia, U.S.A. Department of Management, University of Tennessee, U.S.A.

Methodological Urban Legends: The Misuse of Statistical Control Variables

The automatic or blind inclusion of control variables in multiple regression and other analyses, intended to purify observed relationships among variables of interest, is widespread and can be considered an example of practice based on a methodological urban legend. Inclusion of such variables in most cases implicitly assumes that the control variables are somehow either contaminating the measurement of the variables of interest or affecting the underlying constructs, thus distorting observed relationships among them. There are, however, a number of alternative mechanisms that would produce the same statistical results, thus throwing into question whether inclusion of control variables has led to more or less accurate interpretation of results. The authors propose that researchers should be explicit rather than implicit regarding the role of control variables and match hypotheses precisely to both the choice of variables and the choice of analyses. The authors further propose that researchers avoid testing models in which demographic variables serve as proxies for variables that are of real theoretical interest in their data.

When Two Factors Don’t Reflect Two Constructs: How Item Characteristics Can Produce Artifactual Factors

Factor analyses of scales that contain items written in opposite directions sometimes show two factors, each of which contains items written in only one direction. Such item direction factors have been found in scales of affect and personality that have been used in organizational research. We discuss how patterns of subject responses to items that vary in direction and extremity can produce an arttfactual two factor structure in the absence of multiple constructs. Response patterns are demonstrated in Study 1 with job satisfaction data gathered from employed subjects. The production of two factors is illustrated in Study 2 with simulated data based on item response characteristic equations.

REAL-TIME HUMAN PERCEPTIONS: TOWARD A BICYCLE LEVEL OF SERVICE

The primary focus of this study by Sprinkle Consulting Engineers, Inc. is to develop a bicycle-quality, or level-of-service, model for applications in U.S. metropolitan areas. Although there are several model forms being used throughout the United States that attempt to quantify road suitability or the quality of service afforded bicyclists traveling the street and roadway networks of urbanized areas, to date there have been no statistically calibrated models published. The statistically calibrated level-of-service model described here is based on real-time perceptions from bicyclists traveling in actual urban traffic and roadway conditions. The study’s participants represented a cross section of age, gender, experience level, and geographic origin of the population of cyclists that use the metropolitan road networks in the United States. The test course is representative of the collector and arterial street systems of North American urban areas. Although further hypothesis testing is being conducted and additional studies are planned to test the need for disaggregate models for central business district streets with high turnover parking, truck routes, and two-lane high-speed rural highways, the general bicycle level-of-service model reported here is highly reliable, has a high correlation coefficient (R2 = 0.73), and is transferable to the vast majority of United States metropolitan areas. The study reveals that pavement-surface conditions and striping of bicycle lanes are important factors in the quality of service.

Publication bias in psychological science: Prevalence, methods for identifying and controlling, and implications for the use of meta-analyses.

The issue of publication bias in psychological science is one that has remained difficult to address despite decades of discussion and debate. The current article examines a sample of 91 recent meta-analyses published in American Psychological Association and Association for Psychological Science journals and the methods used in these analyses to identify and control for publication bias. Of the 91 studies analyzed, 64 (70%) made some effort to analyze publication bias, and 26 (41%) reported finding evidence of bias. Approaches to controlling publication bias were heterogeneous among studies. Of these studies, 57 (63%) attempted to find unpublished studies to control for publication bias. Nonetheless, those studies that included unpublished studies were just as likely to find evidence for publication bias as those that did not. Furthermore, authors of meta-analyses themselves were overrepresented in unpublished studies acquired, as compared with published studies, suggesting that searches for unpublished studies may increase rather than decrease some sources of bias. A subset of 48 meta-analyses for which study sample sizes and effect sizes were available was further analyzed with a conservative and newly developed tandem procedure of assessing publication bias. Results indicated that publication bias was worrisome in about 25% of meta-analyses. Meta-analyses that included unpublished studies were more likely to show bias than those that did not, likely due to selection bias in unpublished literature searches. Sources of publication bias and implications for the use of meta-analysis are discussed.

What is method variance and how can we cope with it? A panel discussion

A panel of experts describes the nature of, and remedies for, method variance. In an attempt to help the reader understand the nature of method variance, the authors describe their experiences with method variance both on the giving and the receiving ends of the editorial review process, as well as their interpretation of other reviewers’ comments. They then describe methods of data analysis and research design, which have been used for detecting and eliminating the effects of method variance. Most methods have some utility, but none prevent the researcher from making faulty inferences. The authors conclude with suggestions for resolving disputes about method variance.

A systematic review of the reliability of objective structured clinical examination scores

Medical Education 2011: 45: 1181–1189

Estimation problems in the block diagonal model of the multitrait-multimethod matrix

The most popular method used to analyze the multitrait-multimethod (MTMM) matrix has been confirmatory factor analysis (CFA). The block-diagonal model, in which trait effects, trait correlations, method effects, and method correlations are simultaneously estimated is examined in detail. Analysis of published data from 18 correlation matrices showed estimation problems in all but one case. Simulations were used to show how identification and specification difficulties may account for these problems. Even trivial misspecification of a single parameter can prevent program convergence. These problems render the CFA block-diagonal approach to analyzing MTMM data less useful than has generally been thought. Index terms:

Games teams play: A method for investigating team coordination and performance

Teams are playing an increasingly important role in the workplace. However, reviews of the team performance literature have suggested that there are serious deficiencies in our understanding of team processes and performance (e.g., Dyer, 1984). These difficulties may be attributable, in part, to the lack of laboratory methodologies to investigate team performance. This paper describes the use of low-fidelity simulations as a potentially useful paradigm for researching team coordination and performance. This paradigm is advantageous in that it offers relatively high levels of experimental control and task representation at a low cost.

Comparison of trait and ability measures of emotional intelligence in medical students

Context  Emotional intelligence (EI), the ability to perceive emotions in the self and others, and to understand, regulate and use such information in productive ways, is believed to be important in health care delivery for both recipients and providers of health care. There are two types of EI measure: ability and trait. Ability and trait measures differ in terms of both the definition of constructs and the methods of assessment. Ability measures conceive of EI as a capacity that spans the border between reason and feeling. Items on such a measure include showing a person a picture of a face and asking what emotion the pictured person is feeling; such items are scored by comparing the test‐taker’s response to a keyed emotion. Trait measures include a very large array of non‐cognitive abilities related to success, such as self‐control. Items on such measures ask individuals to rate themselves on such statements as: ‘I generally know what other people are feeling.’ Items are scored by giving higher scores to greater self‐assessments. We compared one of each type of test with the other for evidence of reliability, convergence and overlap with personality.