Joop J. Hox

Sufficient Sample Sizes for Multilevel Modeling

An important problem in multilevel modeling is what constitutes a sufficient sample size for accurate estimation. In multilevel analysis, the major restriction is often the higher-level sample size. In this paper, a simulation study is used to determine the influence of different sample sizes at the group level on the accuracy of the estimates (regression coefficients and variances) and their standard errors. In addition, the influence of other factors, such as the lowest-level sample size and different variance distributions between the levels (different intraclass correlations), is examined. The results show that only a small sample size at level two (meaning a sample of 50 or less) leads to biased estimates of the second-level standard errors. In all of the other simulated conditions the estimates of the regression coefficients, the variance components, and the standard errors are unbiased and accurate.

A checklist for testing measurement invariance

The analysis of measurement invariance of latent constructs is important in research across groups, or across time. By establishing whether factor loadings, intercepts and residual variances are equivalent in a factor model that measures a latent concept, we can assure that comparisons that are made on the latent variable are valid across groups or time. Establishing measurement invariance involves running a set of increasingly constrained structural equation models, and testing whether differences between these models are significant. This paper provides a step-by-step guide to analysing measurement invariance.

Multilevel Analysis Methods

This special issue of SMR is about the analysis of data collected at different levels of observation, such as groups and individuals within these groups, and about the methodological problems that are present when natural experimentation and observations nested within existing social groups are the object of study. The methodological problems are summarized in the term multilevel problems. A multilevel problem is a problem that inquires into the relationships between a set of variables that are measured at a number of different levels of a hierarchy. This article discusses some traditional approaches to the analysis of multilevel data and their statistical shortcomings. The random coefficient linear model is presented, which resolves many of these problems, and the currently available software is discussed. Next, some more general developments in multilevel modeling are discussed. The authors end with an overview of this special issue.

Multilevel modeling: When and why

Multilevel models have become popular for the analysis of a variety of problems. This chapter gives a summary of the reasons for using multilevel models, and provides examples why these reasons are indeed valid. Next, recent (simulation) research is reviewed on the robustness and power of the usual estimation procedures with varying sample sizes.

Meta-analysis of randomized response research.

This article discusses two meta-analyses on randomized response technique (RRT) studies, the first on 6 individual validation studies and the second on 32 comparative studies. The meta-analyses focus on the performance of RRTs compared to conventional question-and-answer methods. The authors use the percentage of incorrect answers as effect size for the individual validation studies and the standardized difference score (d-probit) as effect size for the comparative studies. Results indicate that compared to other methods, randomized response designs result in more valid data. For the individual validation studies, the mean percentage of incorrect answers for the RRT condition is .38; for the other conditions, it is .49. The more sensitive the topic under investigation, the higher the validity of RRT results. However, both meta-analyses have unexplained residual variances across studies, which indicates that RRTs are not completely under the control of the researcher.

The accuracy of multilevel structural equation modeling with pseudobalanced groups and small samples.

Hierarchical structured data cause problems in analysis, because the usual assumptions of independently and identically distributed variables are violated. Muthén (1989) described an estimation method for multilevel factor and path analysis with hierarchical data. This article assesses the robustness of the method with unequal groups, small sample sizes at both the individual and the group level, in the presence of a low or a high intraclass correlation (ICC). The within-groups part of the model poses no problems. The most important problem in the between-groups part of the model is the occurrence of inadmissible estimates, especially when group level sample size is small (50) while the intracluster correlation is low. This is partly compensated by using large group sizes. When an admissible solution is reached, the factor loadings are generally accurate. However, the residual variances are underestimated, and the standard errors are generally too small. Having more or larger groups or a higher ICC does not effectively compensate for this. Therefore, although the nominal alpha level is 5%, the operating alpha level is about 8% in all simulated conditions with unbalanced groups. The strongest factor is an inadequate sample size at the group level. Imbalance is only a problem for the overall fit test. For balanced data, the chi-square fit test is accurate. The size of the biases is comparable to the effect of moderate nonnormality in ordinary modeling, and in our view, the approximate solution remains a useful analysis tool, provided the group level sample size is at least 100.

International handbook of survey methodology

Foundations. J.J. Hox, E.D. de Leeuw, D. Dillman, The Cornerstones of Survey Research. N. Schwarz, B. Knauper, D. Oyserman, C. Stich, The Psychology of Asking Questions. P. Lynn, The Problem of Nonresponse. J. A. Harkness, Comparative Survey Research: Goal and Challenges. E. Singer, Ethical Issues in Surveys. Design. S.L. Lohr, Coverage and Sampling. E.D. de Leeuw, Choosing the Method of Data Collection. F.J. Fowler, C. Cosenza, Writing Effective Questions. D.A. Dillman, The Logic and Psychology of Constructing Questionnaires. P. Campanelli, Testing Survey Questions. Implementation. G. Loosveldt, Face-to-Face interviews. C. Steeh, Telephone Surveys. E.D. de Leeuw, J.J. Hox, Self-Administered Questionnaires: Mail Surveys and Other Applications. K.L. Manfreda, V. Vehovar, Internet Surveys. D.M. Steiger, B. Conroy, IVR: Interactive Voice Response. E.D. de Leeuw, D.A. Dillman, J.J. Hox, Mixed Mode Surveys: When and Why. Data Analysis. P.P. Biemer, S.L. Christ, Weighting Survey Data. L.M. Stapleton, Analysis of Data from Complex Surveys. S. Rassler, D.B. Rubin, N. Schenker, Incomplete Data: Diagnosis, imputation, and estimation. J.J. Hox, Accommodating Measurement Errors. Special Issues. P. Mohler, B. Pennell, F. Hubbard, Survey documentation: Towards Professional Knowledge Management in Sample Surveys. L.E. Lyberg, P.P. Biemer, Quality Assurance and Quality Control in Surveys. J.T. Lessler, J. Eyerman, K. Wang, Interviewer Training. G. Lensvelt-Mulders, Surveying Sensitive Topics. D. Sikkel, A. Hoogendoorn, Panel Surveys. J. Betlehem, Surveys Without Questions.

Present but sick : a three-wave study on job demands, presenteeism and burnout

Purpose – The opposite of absenteeism, presenteeism, is the phenomenon of employees staying at work when they should be off sick. Presenteeism is an important problem for organizations, because employees who turn up for work, when sick, cause a reduction in productivity levels. The central aim of the present study is to examine the longitudinal relationships between job demands, burnout (exhaustion and depersonalization), and presenteeism. We hypothesized that job demands and exhaustion (but not depersonalization) would lead to presenteeism, and that presenteeism would lead to both exhaustion and depersonalization over time.Design/methodology/approach – The hypotheses were tested in a sample of 258 staff nurses who filled out questionnaires at three measurement points with 1.5 years in‐between the waves.Findings – Results were generally in line with predictions. Job demands caused more presenteeism, while depersonalization was an outcome of presenteeism over time. Exhaustion and presenteeism were found to...

Facing Off with Scylla and Charybdis: A Comparison of Scalar, Partial, and the Novel Possibility of Approximate Measurement Invariance

Measurement invariance (MI) is a pre-requisite for comparing latent variable scores across groups. The current paper introduces the concept of approximate MI building on the work of Muthén and Asparouhov and their application of Bayesian Structural Equation Modeling (BSEM) in the software Mplus. They showed that with BSEM exact zeros constraints can be replaced with approximate zeros to allow for minimal steps away from strict MI, still yielding a well-fitting model. This new opportunity enables researchers to make explicit trade-offs between the degree of MI on the one hand, and the degree of model fit on the other. Throughout the paper we discuss the topic of approximate MI, followed by an empirical illustration where the test for MI fails, but where allowing for approximate MI results in a well-fitting model. Using simulated data, we investigate in which situations approximate MI can be applied and when it leads to unbiased results. Both our empirical illustration and the simulation study show approximate MI outperforms full or partial MI In detecting/recovering the true latent mean difference when there are (many) small differences in the intercepts and factor loadings across groups. In the discussion we provide a step-by-step guide in which situation what type of MI is preferred. Our paper provides a first step in the new research area of (partial) approximate MI and shows that it can be a good alternative when strict MI leads to a badly fitting model and when partial MI cannot be applied.

Take Care! The Evaluation of a Team-Based Burnout Intervention Program for Oncology Care Providers

In this quasi-experimental study among staff of 29 oncology wards, the authors evaluated the effects of a team-based burnout intervention program combining a staff support group with a participatory action research approach. Nine wards were randomly selected to participate in the program. Before the program started (Time 1), directly after the program ended (Time 2), and 6 months later (Time 3), study participants filled out a questionnaire on their work situation and well-being. Results of multilevel analyses showed that staff in the experimental wards experienced significantly less emotional exhaustion at both Time 2 and Time 3 and less depersonalization at Time 2, compared with the control wards. Moreover, changes in burnout levels were significantly related to changes in the perception of job characteristics over time.