Craig J. Russell

In pursuit of moderation: nine common errors and teir solutions

One result of the increasing sophistication and complexity of MIS theory and research is the number of studies hypothesizing and testing for moderation effects. A review of the MIS and broader management literatures suggests researchers investigating moderated relationships often commit one or more errors falling into three broad categories: inappropriate use or interpretation of statistics, misalignment of research design with phenomena of interest, and measurement or scaling issues. Examples of nine common errors are presented. Commission of these errors is expected to yield literatures characterized by mixed results at best, and thoroughly erroneous results at worse. Procedures representing examples of best practice and reporting guidelines are provided to help MIS investigators avoid or minimize these errors.

Faking Biodata Tests Are Option-Keyed Instruments More Resistant?

Response biases in biodata scores derived with option-keying and item-keying procedures were investigated. Results indicated that (a) when subjects simulated responding as job applicants they distorted their responses in a socially desirable direction; (b) itemkeyed scores were susceptible to inflation due to socially desirable responding and specific job-title knowledge, but option-keyed scores were not; and (c) response biases were not reflected in response latencies. A supplementary analysis indicated that the two keying procedures may capture different aspects of criterion variance. Implications for reconciling conflicting reports about the susceptibility of biodata scores to response biases are discussed. Issues related to reliability and validity of the two keying procedures, and the generalizability of the results to personality tests, are also discussed.

Predictive validity of biodata items generated from retrospective life experience essays

Biographical information was generated from retrospective lifehistory essays completed in the fall of 1986 by first-year students at the U.S. Naval Academy. Essays were targeted to aspects of four generic life experiences (individual accomplishments, group accomplishments, disappointing situations, and stressful situations) that might reflect the processes or outcomes of antecedent developmental episodes. Essays were coded to generate biodata items, which were then given to 917 midshipmen entering the Naval Academy in the summer of 1987. Scales were developed that predicted criteria of military performance, academic performance, and peer ratings of leadership in validation and cross-validation samples. Implications for item development and biodata construct validity are discussed.

To Log or Not to Log: Bootstrap as an Alternative to the Parametric Estimation of Moderation Effects in the Presence of Skewed Dependent Variables

When gross deviations from parametric assumptions are observed, conventional data transformations are often applied with little regard for substantive theoretical implications. One such transformation involves using the logarithm of positively skewed dependent variables. Log transformations were shown to severely decrease estimates of true moderator effects using moderated regression procedures in a Monte Carlo simulation. Estimates of moderator effect sizes were substantially better estimates of the true latent moderator effect (i.e., larger by a multiple of 2.6 to 534) when estimated using a simple percentile bootstrap procedure in the original, positively skewed data. Conclusions with regard to the presence or absence of a true moderator effect using a simple bootstrap procedure were unaffected by the violation of parametric assumptions in the original, positively skewed data. In contrast, moderated regression analysis performed on a log-transformed dependent variable severely increased Type-II error. Implications are drawn for applied psychological and management research.

On Theory, Statistics, and the Search for Interactions in the Organizational Sciences

This study is based on a review of several papers on multiple regression (with an emphasis on moderated regression) that have recently been published in the Journal of Management and elsewhere. Independent contributions from each of these articles suggest six sets of questions and/or conclusions that might motivate future researchers in this area, and we call for research which integrates the questions into unified studies.

Appropriate moderated regression and inappropriate research strategy: A demonstration of information loss due to scale coarseness

Paunonen and Jackson (1988) demonstrated that stepwise moderated regression provides a test of interaction effects that protects the nominal Type I error rate. However, the stepwise procedure has also been characterized as failing to detect interaction effects in empirical studies. This issues has led to questions regarding the methods statistical power in applied research. It is demonstrated that, because of a research strategy frequently used in empirical investigations, the probability of Type II error in detecting a true interaction effect is unknown.

Person Characteristic Versus Role Congruency Explanations for Assessment Center Ratings

The article presents information on the use of personal characteristics and the role congruency explanations for assessment center ratings. A discussion is presented about the moral and ethical imp...

Supervisor-Subordinate Work Value Congruence and Subordinate Performance: A Pilot Study

The relationship of superior-subordinate work value congruence to subordinate performance was examined in a retail setting. Additionally, because of on-going interest in the role of justice or fairness in the performance appraisal process, the relationship between the supervisors value of fairness and subordinate performance was examined. Superior-subordinate value congruence was not related to subordinate performance. The supervisors level of the value of fairness was related to supervisor-rated dimensions of performance, but not to sales and profits. Implications for future research are presented.

Why Meta-Analysis Doesn’t Tell Us What the Data Really Mean: Distinguishing between Moderator Effects and Moderator Processes

Traditional approaches to detecting the presence of moderators in meta-analyses involve inferences drawn from the residual variance in criterion-related validities (a) after correcting for sampling error and statistical artifacts. James, Demaree, Mulaik, and Ladd (1992) argued that these residualized interpretations of meta-analytic results may be spurious when statistical artifacts covary with true moderators. We extend their model to suggest that situational moderators might also covary with sample size and content (i.e., nonrandom sample selection error), causing meta-analysis to be uninterpretable and a significant correlation between criterion-related validities and ni. We investigate this possibility on studies examining criterion-related validities ofpeer nominations originally reported by Kane and Lawler (1978). Application of residualized meta-analysis suggests the presence of moderator effects, but a significant correlation between ri and ni precludes interpretation of the moderator process behi...

Selecting Top Corporate Leaders: An Example of Biographical Information

A system of structured interviews, superior-peer-subordinate ratings, and consensus discussion is describedfor the selection of top level managers in a multi-national Fortune 50 firm. The interviews and superior-peer-subordinate ratings were used to gather biographical information on 66 candidates. The biographical information was used to generate consensus ratings on nine performance dimensions for the job of division general manager. Concurrent validity evidence provides preliminary support for the selection system. Implications for the personnel selection and leadership literatures are discussed.