Melinda R. Hess

Multilevel Modeling: A Review of Methodological Issues and Applications

This study analyzed the reporting of multilevel modeling applications of a sample of 99 articles from 13 peer-reviewed journals in education and the social sciences. A checklist, derived from the methodological literature on multilevel modeling and focusing on the issues of model development and specification, data considerations, estimation, and inference, was used to analyze the articles. The most common applications were two-level models where individuals were nested within contexts. Most studies were non-experimental and used nonprobability samples. The amount of data at each level varied widely across studies, as did the number of models examined. Analyses of reporting practices indicated some clear problems, with many articles not reporting enough information for a reader to critique the reported analyses. For example, in many articles, one could not determine how many models were estimated, what covariance structure was assumed, what type of centering if any was used, whether the data were consistent with assumptions, whether outliers were present, or how the models were estimated. Guidelines for researchers reporting multilevel analyses are provided.

Making treatment effect inferences from multiple-baseline data: The utility of multilevel modeling approaches

Multiple-baseline studies are prevalent in behavioral research, but questions remain about how to best analyze the resulting data. Monte Carlo methods were used to examine the utility of multilevel models for multiplebaseline data under conditions that varied in the number of participants, number of repeated observations per participant, variance in baseline levels, variance in treatment effects, and amount of autocorrelation in the Level 1 errors. Interval estimates of the average treatment effect were examined for two specifications of the Level 1 error structure (σ2I and first-order autoregressive) and for five different methods of estimating the degrees of freedom (containment, residual, between—within, Satterthwaite, and Kenward—Roger). When the Satterthwaite or Kenward—Roger method was used and an autoregressive Level 1 error structure was specified, the interval estimates of the average treatment effect were relatively accurate. Conversely, the interval estimates of the treatment effect variance were inaccurate, and the corresponding point estimates were biased.

Effectiveness of Interactive Online Algebra Learning Tools

This study of student performance in an online Algebra course looked at the development, implementation, and evaluation of interactive tools for graphing linear equations. The study focused on an interactive tool that was evaluated with virtual school Algebra students for a challenging component of the course. The performance of these students in the course on the component was compared to the performance of students who did not use the intervention. The performance of students learning in the online course with the interactive tools was equivalent to that of not using the tools. The implications of the unique nature of the online Algebra course for teacher preparation are discussed.

Estimation in SEM: A Concrete Example.

A concrete example is used to illustrate maximum likelihood estimation of a structural equation model with two unknown parameters. The fitting function is found for the example, as are the vector of first-order partial derivatives, the matrix of second-order partial derivatives, and the estimates obtained from each iteration of the Newton-Raphson algorithm. The goal is to provide a concrete illustration to help those learning structural equation modeling bridge the gap between the verbal descriptions of estimation procedures and the mathematical definition of these procedures provided in the technical literature.

Interval Estimates of Multivariate Effect Sizes Coverage and Interval Width Estimates Under Variance Heterogeneity and Nonnormality

Monte Carlo methods were used to examine techniques for constructing confidence intervals around multivariate effect sizes. Using interval inversion and bootstrapping methods, confidence intervals were constructed around the standard estimate of Mahalanobis distance (D 2), two bias-adjusted estimates of D 2, and Huberty’s I. Interval coverage and width were examined across conditions by adjusting sample size, number of variables, population effect size, population distribution shape, and the covariance structure. The accuracy and precision of the intervals varied considerably across methods and conditions; however, the interval inversion approach appears to be promising for D 2, whereas the percentile bootstrap approach is recommended for the other effect size measures. The results imply that it is possible to obtain fairly accurate coverage estimates for multivariate effect sizes. However, interval width estimates tended to be large and uninformative, suggesting that future efforts might focus on investigating design factors that facilitate more precise estimates of multivariate effect sizes.

Assessing Online Resources for an Engineering Course in Numerical Methods

To determine, improve, and refine the quality of the online resources for an engineering course in numerical methods, three assessment instruments were used to gather feedback from (1) the independent instructors of the numerical methods course, (2) the students who use the majority of the resources, and (3) the general students worldwide who use resources on an as‐per‐need basis. The findings of this study provide strong evidence that the use of the website modules is a valued aide to most students. The availability of information in multiple modes and formats, at any time, for the students provides them with accessible and convenient learning material that enhances traditional methods. In addition, the analyses of the open‐ended items by both faculty reviewers and students provided insights into how a website used in a technical course such as Numerical Methods can be effectively organized and implemented to enhance further student learning. Results from the instructor surveys found highest ratings for the perceptions of the degree of helpfulness the modules provided in supplementing student readings and with class presentations, while the results from student surveys found highest ratings in the technological domain.