Scott E. Maxwell

Testing Mediational Models With Longitudinal Data: Questions and Tips in the Use of Structural Equation Modeling.

R. M. Baron and D. A. Kenny (1986; see record 1987-13085-001) provided clarion conceptual and methodological guidelines for testing mediational models with cross-sectional data. Graduating from cross-sectional to longitudinal designs enables researchers to make more rigorous inferences about the causal relations implied by such models. In this transition, misconceptions and erroneous assumptions are the norm. First, we describe some of the questions that arise (and misconceptions that sometimes emerge) in longitudinal tests of mediational models. We also provide a collection of tips for structural equation modeling (SEM) of mediational processes. Finally, we suggest a series of 5 steps when using SEM to test mediational processes in longitudinal designs: testing the measurement model, testing for added components, testing for omitted paths, testing the stationarity assumption, and estimating the mediational effects.

Bias in Cross-Sectional Analyses of Longitudinal Mediation.

Most empirical tests of mediation utilize cross-sectional data despite the fact that mediation consists of causal processes that unfold over time. The authors considered the possibility that longitudinal mediation might occur under either of two different models of change: (a) an autoregressive model or (b) a random effects model. For both models, the authors demonstrated that cross-sectional approaches to mediation typically generate substantially biased estimates of longitudinal parameters even under the ideal conditions when mediation is complete. In longitudinal models where variable M completely mediates the effect of X on Y, cross-sectional estimates of the direct effect of X on Y, the indirect effect of X on Y through M, and the proportion of the total effect mediated by M are often highly misleading.

Designing Experiments and Analyzing Data

Whereas there are many experimental design books available, the overall approach, content, and presentation of statistical experimental design in this text are in many ways more complete and broader based than the typical example. Of particular interest is the approach in the first two chapters dealing with statistics in the philosophical, experimental, and scientific sense rather than the strictly mathematical approach so often implied by other authors. The chapter on the Fisher tradition was especially refreshing and certainly interesting in the context of scientific experimentation. The authors have attempted to create a text on experimental design that can be used either as a textbook or as a reference book. It would be difficult to do justice to the content in a typical 18-week course on experimental design. Therefore, I would be more inclined to use the book for a reference and, possibly selected portions, for an introduction to graduate students embarking on their own research study. It will be noticed that the form of the presentation is different from the more computationally oriented texts. The authors have employed a modelcomparison approach that opens up the opportunity to introduce various research models as extenuations or adaptations of simpler models. The method is unique to most texts on research design, and the information presented is more conceptual than computational. The authors rightly assume that the mathematics will be performed by the computer and that the researcher needs to be familiar with the concept of the design rather than the computational techniques. The heart of the presentation comes in chapters 3 and 4. The underlying statistical formulas are introduced in these beginning chapters of Part II. These formulas are used as a basis for all successive chapters. In fact, one would not be required to use the book sequentially even in a classroom environment. But it would be necessary to have the information presented in chapters 3 and 4 well in mind before departing from the sequence outlined. This points to the usefulness of the book as a reference text in that most researchers can turn to any chapter if they have sufficient background or they can read chapters 3 and 4 and then move to specific research models of their choice.

Implications of Recent Developments in Structural Equation Modeling for Counseling Psychology

We review recent developments in structural equation modeling (SEM) since Fassinger’s (1987) introductory article. We discuss issues critical to designing and evaluating SEM studies. Our review includes recent technological developments in SEM such as new approaches from hypothesis testing to determining statistical power and assessing model fit. Moreover, we discuss innovations in applying SEM to different research contexts and designs (e.g., experimental and longitudinal designs and interactions among latent variables). Finally, we discuss procedures for redressing common problems and misunderstandings in the application of SEM procedures to counseling research.

Internal invalidity in pretest-posttest self-report evaluations and a re-evaluation of retrospective pretests.

True experimental designs (Designs 4, 5, and 6 of Campbell & Stanley, 1963) are thought to provide internally valid results. This paper describes five studies involving the evaluation of various treat ment interventions and identifies a source of inter nal invalidity when self-report measures are used in a Pretest-Posttest manner. An alternative approach (Retrospective Pretest-Posttest design) to measuring change is suggested, and data comparing its ac curacy with the traditional Pretest-Posttest design in measuring treatment effects is presented. Finally, the implications of these findings for evaluation re search using self-report instruments and the strengths and limitations of retrospective measures are discussed.

Bias in Cross-Sectional Analyses of Longitudinal Mediation: Partial and Complete Mediation Under an Autoregressive Model

Maxwell and Cole (2007) showed that cross-sectional approaches to mediation typically generate substantially biased estimates of longitudinal parameters in the special case of complete mediation. However, their results did not apply to the more typical case of partial mediation. We extend their previous work by showing that substantial bias can also occur with partial mediation. In particular, cross-sectional analyses can imply the existence of a substantial indirect effect even when the true longitudinal indirect effect is zero. Thus, a variable that is found to be a strong mediator in a cross-sectional analysis may not be a mediator at all in a longitudinal analysis. In addition, we show that very different combinations of longitudinal parameter values can lead to essentially identical cross-sectional correlations, raising serious questions about the interpretability of cross-sectional mediation data. More generally, researchers are encouraged to consider a wide variety of possible mediation models beyond simple cross-sectional models, including but not restricted to autoregressive models of change.

The Development of Multiple Domains of Child and Adolescent Self‐Concept: A Cohort Sequential Longitudinal Design

The development of child and adolescent self-concept was examined as a function of the self-concept domain, social/developmental/educational transitions, and gender. In two overlapping age cohorts of public school students (Ns = 936 and 984), five dimensions of self-concept were evaluated every 6 months in a manner that spanned grades 3 through 11 (representing the elementary, middle, and high school years). Domains of self-concept included academic competence, physical appearance, behavioral conduct, social acceptance, and sports competence. Structural equation modeling addressed questions about the stability of individual differences over time. Multilevel modeling addressed questions about mean-level changes in self-concept over time. Significant effects emerged with regard to gender, age, dimension of self-concept, and educational transition.

Sample size and multiple regression analysis.

Despite the development of procedures for calculating sample size as a function of relevant effect size parameters, rules of thumb tend to persist in designs of multiple regression studies. One explanation for their persistence may be the difficulty in formulating a reasonable a priori value of an effect size to be detected. This article presents methods for calculating effect sizes in multiple regression from a variety of perspectives and also introduces a new method based on an exchangeability structure among predictor variables. No single method is deemed superior, but rather examples show that a combination of methods is likely to be most valuable in many situations. A simulation provides a 2nd explanation for why rules of thumb for choosing sample size have persisted but also shows that the outcome of such underpowered studies will be a literature consisting of seemingly contradictory results.

Analyzing and Interpreting Significant MANOVAs

Multivariate statistical methods have been strongly recommended in educational and psychological research, which employs multiple dependent variables. While the techniques are readily available there is still controversy as to the proper use of the methods. This paper reviews the available methods for analyzing and interpreting data with multivariate analysis of variance and provides some guidelines for their use. In addition, causal models that underlie the various methods are presented to facilitate the use and understanding of the methods

Sample Size for Multiple Regression: Obtaining Regression Coefficients That Are Accurate, Not Simply Significant

An approach to sample size planning for multiple regression is presented that emphasizes accuracy in parameter estimation (AIPE). The AIPE approach yields precise estimates of population parameters by providing necessary sample sizes in order for the likely widths of confidence intervals to be sufficiently narrow. One AIPE method yields a sample size such that the expected width of the confidence interval around the standardized population regression coefficient is equal to the width specified. An enhanced formulation ensures, with some stipulated probability, that the width of the confidence interval will be no larger than the width specified. Issues involving standardized regression coefficients and random predictors are discussed, as are the philosophical differences between AIPE and the power analytic approaches to sample size planning.