Robert J. Boik

Effect size and power in assessing moderating effects of categorical variables using multiple regression: a 30-year review.

The authors conducted a 30-year review (1969-1998) of the size of moderating effects of categorical variables as assessed using multiple regression. The median observed effect size (f(2)) is only .002, but 72% of the moderator tests reviewed had power of .80 or greater to detect a targeted effect conventionally defined as small. Results suggest the need to minimize the influence of artifacts that produce a downward bias in the observed effect size and put into question the use of conventional definitions of moderating effect sizes. As long as an effect has a meaningful impact, the authors advise researchers to conduct a power analysis and plan future research designs on the basis of smaller and more realistic targeted effect sizes.

A Generalized Solution for Approximating the Power to Detect Effects of Categorical Moderator Variables Using Multiple Regression

Investigators in numerous organization studies disciplines are concerned about the low statistical power of moderated multiple regression (MMR) to detect effects of categorical moderator variables. The authors provide a theoretical approximation to the power of MMR. The theoretical result confirms, synthesizes, and extends previous Monte Carlo research on factors that affect the power of MMR tests of categorical moderator variables and the low power of MMR in typical research situations. The authors develop and describe a computer program, which is available on the Internet, that allows researchers to approximate the power of MMR to detect the effects of categorical moderator variables given user-input information (e.g., sample size, reliability of measurement). The approximation also allows investigators to determine the effects of violating certain assumptions required for MMR. Given the typically low power of MMR, researchers are encouraged to use the computer program to approximate power while planning their research design and methodology.

Scheffés mixed model for multivariate repeated measures:a relative efficiency evaluation

Scheffe’s mixed model, generalized for application to multivariate repeated measures, is known as the multivariate mixed model (MMM). The primary advantages the MMM are (1) the minimum sample size required to conduct an analysis is smaller than for competing procedures and (2) for certain covariance structures, the MMM analysis is more powerful than its competitors. The primary disadvantage is that the MMM makes a very restrictive covariance assumption; namely multivariate sphericity. This paper shows, first, that even minor departures from multivariate sphericity inflate the size of MMM based tests. Accordingly, MMM analyses, as computed in release 4.0 of SPSS MANOVA (SPSS Inc., 1990), can not be recommended unless it is known that multivariate sphericity is satisfied. Second, it is shown that a new Box-type (Box, 1954) Δ-corrected MMM test adequately controls test size unless departure from multivariate sphericity is severe or the covariance matrix departs substantially from a multiplicative-Kronecker s...

The mixed model for multivariate repeated measures: validity conditions and an approximate test

Repeated measures on multivariate responses can be analyzed according to either of two models: a doubly multivariate model (DMM) or a multivariate mixed model (MMM). This paper reviews both models and gives three new results concerning the MMM. The first result is, primarily, of theoretical interest; the second and third have implications for practice. First, it is shown that, given multivariate normality, a condition called multivariate sphericity of the covariance matrix is both necessary and sufficient for the validity of the MMM analysis. To test for departure from multivariate sphericity, the likelihood ratio test can be employed. The second result is an approximation to the null distribution of the likelihood ratio test statistic, useful for moderate sample sizes. Third, for situations satisfying multivariate normality, but not multivariate sphericity, a multivariate ε correction factor is derived. The ε correction factor generalizes Boxs ε and can be used to construct an adjusted MMM test.

Testing the Rank of a Matrix with Applications to the Analysis of Interaction in ANOVA

Abstract This article develops some general theory for testing the rank of a matrix. Applications include tests of interaction in two-factor experiments that are more powerful than the usual F test for certain reasonable alternatives. The particular tests studied are (a) the likelihood ratio (LR) test of rank(M) = 0 versus rank(M) = r, where M is a matrix of expectations, and (b) the union-intersection (UI) test of rank(M) = 0 versus rank(M) ≥ r. In the two-factor application, M is the a X b matrix of interaction parameters. The UI test that the interaction has rank = 0 versus rank ≥ 1 is a simultaneous test that all product interaction contrasts are zero. It is shown that the asymptotic distributions of the UI and LR test statistics are identical. A distinct advantage of the UI test is that the small-sample null distribution of the test statistic is known and can be computed. Tables of exact percentiles of the UI test statistic for testing rank(M) = 0 against rank(M) ≥ 1 are given. The UI test is illustr...

The analysis of two-factor interactions in fixed effects linear models.

This article considers two related issues concerning the analysis of interactions in complex linear models. The first issue concerns the omnibus test for interaction. Apparently, it is not well known that the usual F test for interaction can be replaced, in many applications, by a test that is more powerful against a certain class of alternatives. The competing test is based on the maximal product interaction contrast F statistic and achieves its power advantage by focusing solely on product contrasts. The maximal product interaction F test is reviewed and three new results are reported: (a) An extended table of exact critical values is computed, (b) a table of moment functions useful for approximating the p-value corresponding to an observed maximal F statistic is computed, and (c) a simulation study concerning the null distribution of the maximal F statistic when data are unbalanced or covariates are present is reported. It is conjectured that lack of balance or presence of covariates has no effect on the null distribution. The simulation results support the conjecture. The second issue concerns follow-up tests when the omnibus test is significant. It appears that researchers, in general, do not perform coherent follow-up tests on interactions. To make it easier for researchers to do so, an exposition on the use of product interaction contrasts and partial interactions in complex fixed-effects models is provided. The recommended omnibus and follow-up tests are illustrated on an educational data set analyzed using SAS (SAS Institute, 1988) andSPSS (1990).

Testing additivity in two-way classifications with no replications:the locally best invariant test

Row x column interaction is frequently assumed to be negligible in two-way classifications having one observation per cell. Absence of interaction allows the researcher to estimate experimental error and to proceed with making inferences about row and column effects. If additivity is suspect, it is conventional to test it against a structured alternative. If the structured alternative missspecifies the existing nonadditivity, then the power of the test is low, even if the magnitude of the existing nonadditivity is large. The locally best invariant (LBI) test of additivity is less subject to model misspecification because a particular structural alternative need not be hypothesized. This paper illustrates the LBI test of additivity and compares its power to that of the Johnson-Graybill likelihood ratio (LR) test. The LBI test performs as well as the LR test under a Johnson-Graybill alternative and performs better than the LR test under more general alternatives.

Identifiable finite mixtures of location models for clustering mixed-mode data

For clustering mixed categorical and continuous data, Lawrence and Krzanowski (1996) proposed a finite mixture model in which component densities conform to the location model. In the graphical models literature the location model is known as the homogeneous Conditional Gaussian model. In this paper it is shown that their model is not identifiable without imposing additional restrictions. Specifically, for g groups and m locations, (g!)m−1 distinct sets of parameter values (not including permutations of the group mixing parameters) produce the same likelihood function. Excessive shrinkage of parameter estimates in a simulation experiment reported by Lawrence and Krzanowski (1996) is shown to be an artifact of the models non-identifiability. Identifiable finite mixture models can be obtained by imposing restrictions on the conditional means of the continuous variables. These new identified models are assessed in simulation experiments. The conditional mean structure of the continuous variables in the restricted location mixture models is similar to that in the underlying variable mixture models proposed by Everitt (1988), but the restricted location mixture models are more computationally tractable.

An examination of the robustness of the empirical Bayes and other approaches for testing main and interaction effects in repeated measures designs.

In a previous paper, Boik presented an empirical Bayes (EB) approach to the analysis of repeated measurements. The EB approach is a blend of the conventional univariate and multivariate approaches. Specifically, in the EB approach, the underlying covariance matrix is estimated by a weighted sum of the univariate and multivariate estimators. In addition to demonstrating that his approach controls test size and frequently is more powerful than either the epsilon-adjusted univariate or multivariate approaches, Boik showed how conventional multivariate software can be used to conduct EB analyses. Our investigation examined the Type I error properties of the EB approach when its derivational assumptions were not satisfied as well as when other factors known to affect the conventional tests of significance were varied. For comparative purposes we also investigated procedures presented by Huynh and by Keselman, Carriere, and Lix, procedures designed for non-spherical data and covariance heterogeneity, as well as an adjusted univariate and multivariate test statistic. Our results indicate that when the response variable is normally distributed and group sizes are equal, the EB approach was robust to violations of its derivational assumptions and therefore is recommended due to the power findings reported by Boik. However, we also found that both the EB approach and the adjusted univariate and multivariate procedures were prone to depressed or elevated rates of Type I error when data were non-normally distributed and covariance matrices and group sizes were either positively or negatively paired with one another. On the other hand, the Huynh and Keselman et al. procedures were generally robust to these same pairings of covariance matrices and group sizes.

A comparison of three invariant tests of additivity in two-way classifications with no replications

Abstract Tusell (1990) proposed an innovative test of additivity in non-replicated two-way classifications. Tusells procedure consists of employing the well-known likelihood ratio (LR) test of sphericity as a test of additivity. Ordinarily, additivity is assumed when testing sphericity. Tusell deduced that if sphericity is assumed, then application of the LR criterion results in a test of additivity. This article gives the limiting distribution of the LR sphericity criterion under nonadditivity. Simulation studies that compare Tusells test with Johnson and Graybills (1972) LR test against a rank-1 alternative and with Boiks (1990) locally best invariant test of addivity are reported. The LR sphericity test is the most powerful test (among the three) when the noncentrality matrix has multiple non-zero eigenvalues which are comparable in magnitude. For most other conditions, the locally best invariant test is the most powerful test. In practice, prior information about the structure of the noncentrality matrix is generally not available. For these cases, the locally best invariant test is recommended.