Bradley E. Huitema

Design Specification Issues in Time-Series Intervention Models

It has been recognized that the two-phase version of the interrupted time-series design can be frequently modeled using a four-parameter design matrix. There are differences across writers, however, in the details of the recommended design matrices to be used in the estimation of the four parameters of the model. Various writers imply that different methods of specifying the four-parameter design matrix all lead to the same conclusions; they do not. The tests and estimates for level change are dramatically different under the various seemingly equivalent design specifications. Examples of egregious errors of interpretation are presented and recommendations regarding the correct specification of the design matrix are made. The recommendations hold whether the model is estimated using ordinary least squares (for the case of approximately independent errors) or some more complex time-series approach (for the case of autocorrelated errors).

Autocorrelation Estimation and Inference With Small Samples

The small sample properties of 6 autocorrelation estimators were investigated in an extensive Monte Carlo study. It was demonstrated that conventional estimators yield problems of estimation and inference in the form of(a) inconsistencies between theoretical and empirical expectations, (b) inconsistencies between theoretical and empirical error variances, and (c) dramatic differences between nominal and empirical Type I errors

A Simple and Powerful Test for Autocorrelated Errors in OLS Intervention Models

The important assumption of independent errors should be evaluated routinely in the application of interrupted time-series regression models. The two most frequently recommended tests of this assumption [Moods runs test and the Durbin-Watson (D-W) bounds test] have several weaknesses. The former has poor small sample Type I error performance and the latter has the bothersome property that results are often declared to be “inconclusive.” The test proposed in this article is simple to compute (special software is not required), there is no inconclusive region, an exact p-value is provided, and it has good Type I error and power properties relative to competing procedures. It is shown that these desirable properties hold when design matrices of a specified form are used to model the response variable. A Monte Carlo evaluation of the method, including comparisons with other tests (viz., runs, D-W bounds, and D-W beta), and examples of application are provided.

Autocorrelation Effects on Least-Squares Intervention Analysis of Short Time Series

Several issues regarding the effects of autocorrelated errors on Type I error in ordinary least-squares models are clarified. Although autocorrelated errors have a large effect on both omnibus F tests and tests on individual intervention effect coefficients in many applications, there are exceptions that have not been pointed out previously. It is demonstrated that under certain conditions, distortion in Type I error is far less than is predicted by asymptotic theory. It is shown that these exceptions occur because the effect of autocorrelated errors is dependent on (a) the type of parameters (e.g., level change and/or slope change) required in the model, (b) the number of variables in the design matrix, and (c) the sample size. Because existing time-series methods perform poorly with small samples, this may be a useful finding in some situations; however, a better general solution is to use a recently developed small-sample method.

Identifying Autocorrelation Generated by Various Error Processes in Interrupted Time-Series Regression Designs: A Comparison of AR1 and Portmanteau Tests.

Regression models used in the analysis of interrupted time-series designs assume statistically independent errors. Four methods of evaluating this assumption are the Durbin-Watson (D-W), Huitema-McKean (H-M), Box-Pierce (B-P), and Ljung-Box (L-B) tests. These tests were compared with respect to Type I error and power under a wide variety of error models and sample sizes. Although the B-P and L-B tests are portmanteau methods that incorporate information from a large portion of the autocorrelation function, the more focused D-W and H-M first-order autoregressive tests are shown to be considerably more powerful. The popular L-B test has unacceptable Type I error and should not be used in the context of the intervention model applied in this study.

Reduced Bias Autocorrelation Estimation: Three Jackknife Methods

The effectiveness of jackknife methods in reducing bias in the estimation of the lag-I autocorrelation parameter Pi was evaluated. A Monte Carlo investigation was carried out to study the empirical bias, mean-square error, and variance properties of three jackknife estimators using sample sizes that ranged from 6 through 500. The results demonstrated that these estimators are far less biased in the small sample case than are many other estimators that have been recently investigated. Results on the mean-squared error revealed that the advantage of greatly reduced bias associated with the jackknife estimators does not overcome the disadvantage of increased error variance. Three previously investigated estimators yield smaller mean-squared error than do the jackknife estimators or the conventional estimator at most sample sizes.

Validity of the GRE without Restriction of Range

Restriction of range is a frequently acknowledged issue in estimating the validity of predictors of academic performance in graduate school. Data obtained from a doctoral program in a psychology department where graduate students were admitted without regard to Graduate Record Examination (GRE) scores yielded essentially identical standard deviations on this test for the 204 applicants and 138 enrolled students. The GRE-Total validity coefficients obtained on subjects in the enrolled sample ranged from .55 through .70; these values are considerably higher than those typically reported. The data are congruent with the argument that uncorrected GRE validity coefficients yield biased estimates of the unknown validity in unrestricted applicant pools.

Statistical Analysis and Single-Subject Designs

In the beginning there were no statistical analyses of operant experiments. It was like a breath of fresh air for many psychologists when, many years ago, Skinner (1963) said: Statistical methods are unnecessary....When a variable is changed and the effect on performance observed, it is for most purposes idle to prove statistically that a change has indeed occurred.... rate of responding and changes in rate can be directly observed... The effect is similar to increasing the resolving power of a microscope: A new subject matter is suddenly open to direct inspection, (p. 508)

An improved portmanteau test for autocorrelated errors in interrupted time-series regression models.

A new portmanteau test for autocorrelation among the errors of interrupted time-series regression models is proposed. Simulation results demonstrate that the inferential properties of the proposedQH-M test statistic are considerably more satisfactory than those of the well known Ljung-Box test and moderately better than those of the Box-Pierce test. These conclusions generally hold for a wide variety of autoregressive (AR), moving averages (MA), and ARMA error processes that are associated with time-series regression models of the form described in Huitema and McKean (2000a, 2000b).

Two reduced-bias autocorrelation estimators : rF1 and rF2

Among the problems associated with the application of time-series analysis to typical psychological data are difficulties in parameter estimation. For example, estimates of autocorrelation coefficients are known to be biased in the small-sample case. Previous work by the present authors has shown that, in the case of conventional autocorrelation estimators of ρ1 the degree of bias is more severe than is predicted by formulas that are based on large-sample theory. Two new autocorrelation estimators, rF1 and rF2, were proposed; a Monte Carlo experiment was carried out to evaluate the properties of these statistics. The results demonstrate that both estimators provide major reduction of bias. The average absolute bias of rF2 is somewhat smaller than that of rF1 at all sample sizes, but both are far less biased than is the conventional estimator found in most time-series software. The reduction in bias comes at the price of an increase in error variance. A comparison of the properties of these estimators with those of other estimators suggested in 1991 shows advantages and disadvantages for each. It is recommended that the choice among autocorrelation estimators be based upon the nature of the application. The new estimator rF2 is especially appropriate when pooling estimates from several samples.