Willem E. Saris

Power of the likelihood ratio test in covariance structure analysis

A procedure for computing the power of the likelihood ratio test used in the context of covariance structure analysis is derived. The procedure uses statistics associated with the standard output of the computer programs commonly used and assumes that a specific alternative value of the parameter vector is specified. Using the noncentral Chi-square distribution, the power of the test is approximated by the asymptotic one for a sequence of local alternatives. The procedure is illustrated by an example. A Monte Carlo experiment also shows how good the approximation is for a specific case.

Testing Structural Equation Models or Detection of Misspecifications

Assessing the correctness of a structural equation model is essential to avoid drawing incorrect conclusions from empirical research. In the past, the chi-square test was recommended for assessing the correctness of the model but this test has been criticized because of its sensitivity to sample size. As a reaction, an abundance of fit indexes have been developed. The result of these developments is that structural equation modeling packages are now producing a large list of fit measures. One would think that this progression has led to a clear understanding of evaluating models with respect to model misspecifications. In this article we question the validity of approaches for model evaluation based on overall goodness-of-fit indexes. The argument against such usage is that they do not provide an adequate indication of the “size” of the models misspecification. That is, they vary dramatically with the values of incidental parameters that are unrelated with the misspecification in the model. This is illustrated using simple but fundamental models. As an alternative method of model evaluation, we suggest using the expected parameter change in combination with the modification index (MI) and the power of the MI test.

The validity and reliability of survey questions: a meta analysis of MTMM studies

Inspired by the research of Frank Andrews on the reliability and validity of survey questions, a large-scale research project was conducted in the Netherlands. The project was comprised of two different stages. For this project, more than 600 survey questions were included in different surveys according to a multitrait-multimethod design. The resulting data were analyzed in two steps. In the first step, estimates of validity and reliability were obtained for each question. The second step was a meta-analysis of the variation in data quality found in the first step. This variation was related to question-specific characteristics, response scale characteristics, context characteristics, and design characteristics. The article describes how the results of this study can be of practical use. In addition, the authors compare them to results of similar studies in the United States, Austria, and other Western, Central, and Eastern European countries.

Alternative approaches to structural modeling of ordinal data: a Monte Carlo study

In practice, several measures of association are used when analyzing structural equation models with ordinal variables: ordinary Pearson correlations (PE approach), polychoric and polyserial correlations (PO approach), and conditional polychoric correlations (CPO approach). In the case of structural equation models without latent variables, the literature has shown that the PE approach is outperformed by the alternatives. In this article we report a Monte Carlo study showing the comparative performance of the aforementioned alternative approaches under deviations from their respective assumptions in the case of structural equation models with latent variables when attention is restricted to point estimates of model parameters. The CPO approach is shown to be the most robust against nonnormality. It is also robust to randomness of the exogenous variables, but not to the existence of measurement errors in them. The PO approach lacks robustness against nonnormality. The PE approach lacks robustness against t...

Choosing the Number of Categories in Agree–Disagree Scales

Although agree–disagree (AD) rating scales suffer from acquiescence response bias, entail enhanced cognitive burden, and yield data of lower quality, these scales remain popular with researchers due to practical considerations (e.g., ease of item preparation, speed of administration, and reduced administration costs). This article shows that if researchers want to use AD scales, they should offer 5 answer categories rather than 7 or 11, because the latter yield data of lower quality. This is shown using data from four multitrait-multimethod experiments implemented in the third round of the European Social Survey. The quality of items with different rating scale lengths were computed and compared.

A New Approach to Evaluating the Quality of Measurement Instruments: The Split-Ballot MTMM Design

Two distinctly different quantitative approaches are used to evaluate measurement instruments: the split-ballot experiment and the multitrait-multimethod (MTMM) approach. The first approach is typically used to indicate whether variation in the method causes differences in the response distribution; the second approach evaluates the reliability and validity of different methods. The new approach, suggested in this paper, combines the more attractive features of both methods. The strength of the split-ballot experiment is its use of independent random samples from the same population to provide information about differences in response distributions. This is also possible with the new approach, but this approach provides more detailed information about the reasons

Estimating the reliability and validity of personal support measures: full information ML estimation with planned incomplete data

Egocentered networks are common in social science research. Here, the unit of analysis is a respondent (ego) together with his/her personal network (alters). Usually, several variables are used to describe the relationship between egos and alters. In this paper, the aim is to estimate the reliability and validity of the averages of these measures by the multitrait–multimethod (MTMM) approach. This approach usually requires at least three repeated measurements (methods) of the same variable (trait) for model identification. This places a considerable burden on the respondent and increases the cost of data collection. In this paper, we use a split ballot MTMM experimental design, proposed by Saris (1999), in which separate groups of respondents get different combinations of just two methods. The design can also be regarded as having a planned missing data structure. The maximum likelihood estimation is used in the manner suggested by Allison (1987) of a confirmatory factor analysis model for MTMM-designs specified in Saris and Andrews (1991). This procedure is applied to social support data collected in the city of Ljubljana (Slovenia) in the year 2000.

Causal direction in a model of life satisfaction: The top-down/bottom-up controversy

In social indicator studies, there is some controversy about the causal direction between subjective well-being and domain-specific satisfaction variables; a “top-down” approach is distinguished from a “bottom-up” approach. In this paper, the effects in both directions are estimated in a model with reciprocal relationships as a starting model. It can then be determined which of the two effects for each pair of variables is strongest and which effect can be ignored in the model. This procedure is applied to four different datasets collected in the Netherlands, and to models with different exogenous variables. Comparing the best solutions obtained for all different models and datasets, it is shown that the direction of the effects is not consistent across models and datasets. We have to conclude that it is impossible to obtain a stable solution for the model of subjective well-being in this study. As a consequence, we also have to conclude that the results from other studies cannot be trusted in which the causal order in a model of subjective well-being is tested.

The Relationship between Subjective Well-being and Domain Satisfactions in South Africa

This paper examines the relationship between subjective well-being and domain satisfactions. In the past different models have been specified. The most commonly applied model is the bottom-up model in which domain satisfactions affect subjective well-being. The more recent top-down model suggests a reversed relationship. Finally there is the supposition that the correlations between these variables can be spurious due to the effect of personality characteristics. Empirical research has shown that different models are found for different domains and in different countries. Focussing on the effects of the domain satisfactions of finances, housing and social contacts it has been found that subjective well-being is mainly affected by satisfaction with social contacts in Western developed countries and by satisfaction with finances in East European countries. The question we should like to answer in this study is whether a similar pattern obtains for the factors which influence subjective well-being among the different race groups in South Africa. Interestingly, coloured people and Asians did indeed show the expected effects but the groups with the most extreme living conditions did not. Evaluation of life circumstances by black and white South Africans was determined by expectations for the future rather than by current living conditions. This surprising result is discussed in the light of the political situation in South Africa.

Design, Evaluation, and Analysis of Questionnaires for Survey Research: Saris/Design, Evaluation, and Analysis of Questionnaires for Survey Research

Preface. Introduction. PART I. THE THREE STEPS PROCEDURE TO DESIGN REQUESTS FOR AN ANSWER. 1. Concepts-by-postulation and concepts-by-intuition. 2. From social science concepts-by-intuition to assertions. 3. The formulation of requests for an answer. PART II. CHOICES INVOLVED IN QUESTIONNAIRE DESIGN. 4. Specific survey research features of requests for an answer. 5. Response alternatives. 6. The structure of open ended and closed survey items. 7. Survey items in batteries. 8. Mode of data collection and other choices. PART III. THE EFFECTS OF SURVEY CHARACTERISTICS ON DATA QUALITY. 9. Criteria for the quality of survey measures. 10. The estimation of reliability, validity and method effects. 11. The split ballot MTMM designs. 12. The estimation of the effects of measurement characteristics on the quality of survey questions. PART IV. APPLICATIONS IN SOCIAL SCIENCE RESEARCH. 13. The prediction and improvement of survey requests by SQP. 14. The quality of measures for concepts-by-postulation. 15. Correction for measurement error in survey data analysis. 16. Coping with measurement error in cross-cultural research. References. Index.