Christof Schuster

A note on the interpretation of weighted kappa and its relations to other rater agreement statistics for metric scales

This article presents a formula for weighted kappa in terms of rater means, rater variances, and the rater covariance that is particularly helpful in emphasizing that weighted kappa is an absolute agreement measure in the sense that it is sensitive to differences in rater’s marginal distributions. Specifically, rater mean differences will decrease the value of weighted kappa relative to the value of the intraclass correlation that ignores mean differences. In addition, if rater variances also differ, then the value of weighted kappa will be decreased relative to the value of the product-moment correlation. Equality constraints on the rater means and variances are given to illustrate the relationships between weighted kappa, the intraclass correlation, and the product-moment correlation. In addition, the expression for weighted kappa shows that weighted kappa belongs to the Zegers-ten Berge family of chance-corrected association coefficients. More specifically, weighted kappa is equivalent to the chance-corrected identity coefficient.

ADOLESCENT MARIJUANA USE AND ADULT OCCUPATIONAL ATTAINMENT: A LONGITUDINAL STUDY FROM AGE 18 TO 28

This report uses data from a nationally representative longitudinal sample to examine the relationship of marijuana use by high school seniors to occupational attainment 10 years later. Analyses were conducted separately by gender, with and without controlling for other variables. Control variables, all measured when respondents were seniors, were: academic performance, educational aspirations, and occupational aspirations. Results indicate that the influence of marijuana use on occupational attainment is considerably different for males and females.

The odds of resilience.

This article presents sample research designs for the investigation of questions concerning resilience. Sample hypothesis from specific research designs are described in the form of odds ratios.

The Relationship of ANOVA Models with Random Effects and Repeated Measurement Designs.

Understanding ANOVA models is often difficult because of the large amount of different experimental designs presented in applied textbooks.This article shows how different experimental designs arise out of the variation of three basic distinctions: block versus treatment factors, fixed versus random factors, and crossed versus nested factors.Once it is understood how each distinction influences the statistical analysis, the amount of experimental designs can be considerably reduced, because sometimes seemingly different experimental designs are essentially equivalent.This is shown by an example comparing a two-way analysis of variance model to a three-factor partially nested design.Furthermore, the way each distinction influences the statistical analysis of an experimental design can simplify the computational effort of the analysis because virtually every basic ANOVA procedure implemented in common statistical software packages can be used to fit more complex ANOVA models that are usually analyzed using special computer modules.

Robust Estimation of Latent Ability in Item Response Models

Because of response disturbances such as guessing, cheating, or carelessness, item response models often can only approximate the “true” individual response probabilities. As a consequence, maximum-likelihood estimates of ability will be biased. Typically, the nature and extent to which response disturbances are present is unknown, and, therefore, accounting for them by altering the model is not possible. Even if the nature of the response disturbances were known, accounting for them by increasing model complexity could easily lead to sample size requirements for estimation purpose that would be difficult to achieve. An approach based on weighting the contributions of the item responses to the log likelihood function has been suggested by Mislevy and Bock. This estimation approach has been shown to effectively reduce bias of ability estimates in the presence of response disturbances. However, this approach is prone to produce infinite ability estimates for unexpected response patterns in which correct answers are sparse. An alternative robust estimator of ability is suggested that does not appear to produce infinite estimates. Limited simulation studies show that the two estimators are equivalent when evaluated in terms of mean squared error.

The road to freedom: Quantitative developmental methodology in the third millennium

In this article we ask three questions. First, what is the relationship between substantive psychological research and research in the area of applied quantitative, statistical methods? Second, what is the development that applied quantitative methodology will undergo in the foreseeable future? Third, will this development affect the future of psychological research? The answers to these questions are cast in terms of six trends and three anticipated problems. The trends are: (1) increase in the number of available statistical methods; (2) substantive thinking triggers the development of new statistical methods; (3) statistical methods become increasingly flexible; (4) computers do the thinking; (5) there will be new paradigms such as the person-oriented perspective; and (6) statistical methods will be custom-tailored to substantive questions. The general tenor of the discussion of these trends is that the number of possibilities will increase and the researcher will be freed from method-specific constraints. The three problem trends are: (1) the choice between methods of analysis becomes increasingly difficult; (2) cases of misuse of statistics will become more frequent; and (3) the role of the methodologist will change such that more substantive knowledge will be required. In addition, issues of training of methodologists and financing research are discussed. The general outlook is depicted as encouraging.

Models for ordinal agreement data

Statistical models can be used to describe the probabilistic structure underlying cross-classified agreement data. This article explains how models for ordinal agreement data can be understood in terms of an association component and an agreement component. The association component accounts for the positive association typically present in ordinal ratings of two observers. The agreement component specifies a model for the diagonal cells of the cross-classified ratings. Several models for ordinal agreement data proposed in the literature are special cases of this approach. A new log-linear model for agreement data that can also be understood in terms of the two components is presented and illustrated using data from a case-control study of coronary heart disease.

Kappa as a Parameter of a Symmetry Model for Rater Agreement

If two raters assign targets to categories, the ratings can be arranged in a two-dimensional contingency table. A model for the frequencies in such a contingency table is presented for which Cohen’s kappa is a parameter. The model is based on two assumptions. First, the joint classification probabilities for the raters satisfy symmetry; second, the ratio of observed agreement to chance agreement is constant across categories. The model is illustrated using data from a study of the psychobiology of depression.

Estimating with a Latent Class Model the Reliability of Nominal Judgments Upon Which Two Raters Agree

Because nominal-scale judgments cannot directly be aggregated into meaningful composites, the addition of a second rater is usually motivated by a desire to estimate the quality of a single raters classifications rather than to improve reliability. When raters agree, the aggregation problem does not arise. Nevertheless, a proportion of this agreement is likely to have occurred by chance, so reliability issues remain even if the focus is on nominal judgments upon which raters concur. In this article, the reliability of nominal judgments upon which raters agree is addressed in the framework of a latent class model that includes a systematic agreement parameter. It is shown that if attention is limited to agreement cases, the value of this parameter increases as expected. Circumstances under which agreement case reliability is important are discussed.

Partielle Korrelation undmultiple lineare Regression

Die multiple lineare Regression erweitert die in Kap. 11 besprochene, einfache lineare Regression, indem sie die Berucksichtigung von mehr als einer Pradiktorvariablen ermoglicht. Die multiple Regression sowie die partielle Korrelation konnen eingesetzt werden, um den Einfluss von Drittvariablen in der Analyse von Variablenbeziehungen zu „kontrollieren“, d. h., statistisch konstant zu halten.