Ryan K. Gottfredson

Best-Practice Recommendations for Estimating Cross-Level Interaction Effects Using Multilevel Modeling

Multilevel modeling allows researchers to understand whether relationships between lower-level variables (e.g., individual job satisfaction and individual performance, firm capabilities and performance) change as a function of higher-order moderator variables (e.g., leadership climate, market-based conditions). We describe how to estimate such cross-level interaction effects and distill the technical literature for a general readership of management researchers, including a description of the multilevel model building process and an illustration of analyses and results with a data set grounded in substantive theory. In addition, we provide 10 specific best-practice recommendations regarding persistent and important challenges that researchers face before and after data collection to improve the accuracy of substantive conclusions involving cross-level interaction effects. Our recommendations provide guidance on how to define the cross-level interaction effect, compute statistical power and make research design decisions, test hypotheses with various types of moderator variables (e.g., continuous, categorical), rescale (i.e., center) predictors, graph the cross-level interaction effect, interpret interactions given the symmetrical nature of such effects, test multiple cross-level interaction hypotheses, test cross-level interactions involving more than two levels of nesting, compute effect-size estimates and interpret the practical importance of a cross-level interaction effect, and report results regarding the multilevel model building process.

Best-Practice Recommendations for Defining, Identifying, and Handling Outliers

The presence of outliers, which are data points that deviate markedly from others, is one of the most enduring and pervasive methodological challenges in organizational science research. We provide evidence that different ways of defining, identifying, and handling outliers alter substantive research conclusions. Then, we report results of a literature review of 46 methodological sources (i.e., journal articles, book chapters, and books) addressing the topic of outliers, as well as 232 organizational science journal articles mentioning issues about outliers. Our literature review uncovered (a) 14 unique and mutually exclusive outlier definitions, 39 outlier identification techniques, and 20 different ways of handling outliers; (b) inconsistencies in how outliers are defined, identified, and handled in various methodological sources; and (c) confusion and lack of transparency in how outliers are addressed by substantive researchers. We offer guidelines, including decision-making trees, that researchers can follow to define, identify, and handle error, interesting, and influential (i.e., model fit and prediction) outliers. Although our emphasis is on regression, structural equation modeling, and multilevel modeling, our general framework forms the basis for a research agenda regarding outliers in the context of other data-analytic approaches. Our recommendations can be used by authors as well as journal editors and reviewers to improve the consistency and transparency of practices regarding the treatment of outliers in organizational science research.