James M. LeBreton

Answers to 20 Questions About Interrater Reliability and Interrater Agreement

The use of interrater reliability (IRR) and interrater agreement (IRA) indices has increased dramatically during the past 20 years. This popularity is, at least in part, because of the increased role of multilevel modeling techniques (e.g., hierarchical linear modeling and multilevel structural equation modeling) in organizational research. IRR and IRA indices are often used to justify aggregating lower-level data used in composition models. The purpose of the current article is to expose researchers to the various issues surrounding the use of IRR and IRA indices often used in conjunction with multilevel models. To achieve this goal, the authors adopt a question-and-answer format and provide a tutorial in the appendices illustrating how these indices may be computed using the SPSS software.

History and Use of Relative Importance Indices in Organizational Research

The search for a meaningful index of the relative importance of predictors in multiple regression has been going on for years. This type of index is often desired when the explanatory aspects of regression analysis are of interest. The authors define relative importance as the proportionate contribution each predictor makes to R2, considering both the unique contribution of each predictor by itself and its incremental contribution when combined with the other predictors. The purposes of this article are to introduce the concept of relative importance to an audience of researchers in organizational behavior and industrial/organizational psychology and to update previous reviews of relative importance indices. To this end, the authors briefly review the history of research on predictor importance in multiple regression and evaluate alternative measures of relative importance. Dominance analysis and relative weights appear to be the most successful measures of relative importance currently available. The authors conclude by discussing how importance indices can be used in organizational research.

The restriction of variance hypothesis and interrater reliability and agreement: Are ratings from multiple sources really dissimilar?

The fundamental assumption underlying the use of 360-degree assessments is that ratings from different sources provide unique and meaningful information about the target manager’s performance. Extant research appears to support this assumption by demonstrating low correlations between rating sources. This article reexamines the support of this assumption, suggesting that past research has been distorted by a statistical artifact—restriction of variance in job performance. This artifact reduces the amount of between-target variance in ratings and attenuates traditional correlation-based estimates of rating similarity. Results obtained from a Monte Carlo simulation and two field studies support this restriction of variance hypothesis. Noncorrelation-based methods of assessing interrater agreement indicated that agreement between sources was about as high as agreement within sources. Thus, different sources did not appear to be furnishing substantially unique information. The authors conclude by questioning common practices in 360-degree assessments and offering suggestions for future research and application.

Destructive Leadership A Theoretical Review, Integration, and Future Research Agenda

In this article, we propose a framework for understanding destructive leadership that summarizes the extant destructive leadership research and extends it in new directions. By reviewing the current literature on destructive leadership and drawing on organizational leadership theory and the more general research on deviant behaviors in organizations, we identify the underlying features and mechanisms that define destructive leadership. Recognizing that each form of destructive leadership currently studied (e.g., abusive supervision, petty tyranny, and pseudo-transformational leadership) addresses aspects of destructive leadership but fails to capture the complete picture of the phenomenon, we clarify the boundaries among the constructs studied within the domain of destructive leadership, address some ambiguities about the nature of destructive leadership, make explicit some characteristics of destructive leadership that set it apart from other forms of leading, and integrate this thinking into a theoretical model that helps us understand the manifestations of destructive leadership, and their antecedents and consequences.

Determining the Statistical Significance of Relative Weights

Relative weight analysis is a procedure for estimating the relative importance of correlated predictors in a regression equation. Because the sampling distribution of relative weights is unknown, researchers using relative weight analysis are unable to make judgments regarding the statistical significance of the relative weights. J. W. Johnson (2004) presented a bootstrapping methodology to compute standard errors for relative weights, but this procedure cannot be used to determine whether a relative weight is significantly different from zero. This article presents a bootstrapping procedure that allows one to determine the statistical significance of a relative weight. The authors conducted a Monte Carlo study to explore the Type I error, power, and bias associated with their proposed technique. They illustrate this approach here by applying the procedure to published data.

A Monte Carlo Comparison of Relative Importance Methodologies

This article reports the results of a Monte Carlo simulation comparing four different indices of relative importance (squared correlation, squared beta, product measure, epsilon) to a relatively new method called dominance analysis. Conceptually and empirically, dominance analysis represents an improvement over traditional indices of relative importance. Eight experimental factors were manipulated in the simulations: mean and standard deviation of validity for each predictor, mean and standard deviation of collinearity for two sets of predictors, number of predictors, and presence of simple structure. Of these factors, the number of predictors and the mean collinearity were most strongly related to discrepancies among the rank orders computed using the different importance methods. Across all experimental conditions, the epsilon statistic demonstrated the greatest convergence and beta weights and correlation coefficients the greatest divergence with dominance.

Risk factors for sexual aggression in young men: an expansion of the confluence model

There are many explanations for high rates of sexual aggression, with no one theory dominating the field. This study extends past research by evaluating an expanded version of the confluence model with a community sample. One-hour audio computer-assisted self-interviews were completed by 470 young single men. Using structural equation analyses, delinquency, hostile masculinity, impersonal sex, and misperception of womens sexual cues were positively and directly associated with the number of sexually aggressive acts committed. There were also indirect effects of childhood victimization, personality traits associated with subclinical levels of psychopathy, and alcohol consumption. These findings demonstrate the usefulness of the confluence model, as well as the importance of broadening this theory to include additional constructs.

Multivariate relative importance: extending relative weight analysis to multivariate criterion spaces.

For years, organizational scholars have sought effective ways to evaluate the importance of predictors included in a regression analysis. Recent techniques, such as general dominance weights and relative weights, have shown great promise for guiding evaluations of predictor importance. Nevertheless, questions remain regarding how one should investigate relative importance in the presence of a multidimensional criterion variable. The purpose of this article is to extend understanding of relative importance statistics to multivariate designs. The authors review the concept of relative importance and discuss a new procedure for calculating estimates of importance in the presence of multiple correlated criteria. Finally, a published correlation matrix is reanalyzed and a Monte Carlo simulation conducted to compare the new procedure with another technique for estimating importance. Unlike canonical solutions, which are often uninterpretable, the proposed multivariate relative weights provide an intuitive index regarding the relationship between predictors and criteria. Implications for organizational researchers are discussed.

A Conditional Reasoning Measure for Aggression

This article describes a new approach for assessing cognitive precursors to aggression. Referred to as the Conditional Reasoning Measurement System, this procedure focuses on how people solve what on the surface appear to be traditional inductive reasoning problems. The true intent of the problems is to determine if solutions based on implicit biases (i.e., biases that operate below the surface of consciousness) are logically attractive to a respondent. The authors focus on the types of implicit biases that underlie aggressive individuals’attempts to justify aggressive behavior. People who consistently select solutions based on these types of biases are scored as being potentially aggressive because they are cognitively prepared to rationalize aggression. Empirical tests of the conditional reasoning system are interpreted in terms of Ozer’s criteria for ideal personality instruments. Noteworthy findings are that the system has acceptable psychometric properties and an average, uncorrected empirical validity of 0.44 against behavioral indicators of aggression (based on 11 studies).

Integrating Implicit and Explicit Social Cognitions for Enhanced Personality Assessment A General Framework for Choosing Measurement and Statistical Methods

The current article advocates integrating implicit and explicit social cognitions for enhanced personality assessment in organizational contexts (e.g., personnel selection settings). Several methods for measuring implicit cognitions are reviewed, and their strengths and limitations are discussed. The most widely used method for measuring explicit cognitions, the self-report questionnaire, also is described along with its strengths and limitations. Implicit and explicit cognitions then are integrated to form a general model of personality prototypes. The authors describe several mechanisms by which implicit and explicit cognitions may operate (e.g., coact, interact) to predict criteria, depending on the nature of the personality construct assessed and the outcome of interest. These different operations implicate different statistical methodologies. The authors then present specific examples of this integrative procedure for enhancing personality assessment using the construct of achievement motivation. They conclude by discussing how future research could extend and apply this general framework for use with other personality constructs.