Janis E. Johnston

Weighted Kappa for Multiple Raters

Five procedures to calculate the probability of weighted kappa with multiple raters under the null hypothesis of independence are described and compared in terms of accuracy, ease of use, generality, and limitations. The five procedures are (1) exact variance, (2) resampling contingency, (3) intraclass correlation, (4) randomized block, and (5) resampling block. While each procedure possesses strengths and limitations, the resampling contingency procedure is shown to be the most versatile and accurate of the five procedures, provided the number of raters is not too large. The resampling contingency procedure permits any weighting scheme, accommodates both symmetrical and asymmetrical weights, is suitable for both weighted and unweighted kappa, and makes no assumptions about either the data distribution or the probability distribution.

THE EXACT VARIANCE OF WEIGHTED KAPPA WITH MULTIPLE RATERS

Weighted kappa described by Cohen in 1968 is widely used in psychological research to measure agreement between two independent raters. Everitt then provided the exact variance for weighted kappa for two raters. In this paper, Everitts exact variance is extended to three or more raters.

Permutation Tests: Precision in Estimating Probability Values

The number of resamplings necessary to accurately estimate a probability value is an open question. One million resamplings is shown to be sufficient to ensure precision to three places under most conditions.

Asymptotic log-linear analysis: Some cautions concerning sparse frequency tables

Traditional asymptotic probability values resulting from log-linear analyses of sparse frequency tables are often much too large. Asymptotic probability values for chi-squared and likelihood-ratio statistics are compared to nonasymptotic and exact probability values for selected log-linear models. The asymptotic probability values are all too often substantially larger than the exact probability values for the analysis of sparse frequency tables. An exact nondirectional permutation method is presented to analyze combined independent multinomial distributions. Exact nondirectional permutation methods to analyze hypergeometric distributions associated with r-way frequency tables are confined to r = 2.

Measures of Effect Size for Chi-Squared and Likelihood-Ratio Goodness-of-Fit Tests

A fundamental shift in editorial policy for psychological journals was initiated when the fourth edition of the Publication Manual of the American Psychological Association (1994) placed emphasis on reporting measures of effect size. This paper presents measures of effect size for the chi-squared and the likelihood-ratio goodness-of-fit statistic tests.

Permutation Methods for the Analysis of Matched-Pairs Experimental Designs

The Fisher-Pitman and powers of ranks permutation tests are shown to provide substantial resistance to extreme values when compared with the conventional t test for analyzing matched-pairs experimental designs.

A FORTRAN program for computing the exact variance of weighted kappa.

An algorithm and associated FORTRAN program are provided for the exact variance of weighted kappa. Program VARKAP provides the weighted kappa test statistic, the exact variance of weighted kappa, a Z score, one-sided lower- and upper-tail N(0,1) probability values, and the two-tail N(0,1) probability value.

A measure of effect size for experimental designs with heterogeneous variances.

A recent trend in the psychological literature has been to include measures of effect size when reporting probability values. The several measures of effect size associated with the Student t test for two independent samples are appropriate only when the variances are homogeneous. In this paper, commonly used measures of effect size are considered and compared, using four data sets. A chance-corrected measure of effect size is provided for two or more treatment groups characterized by either homogeneous or heterogeneous variances.

An alternative measure of effect size for Cochran's q test for related proportions

Measures of effect size are increasingly important in psychological research. In this paper, a chance-corrected measure of effect size is introduced for Cochrans Q test.

Exact and resampling probability values for weighted Kappa

Permutation procedures to compute exact and resampling probability values for weighted kappa are described. Comparisons with asymptotic probability values demonstrate that exact permutation procedures are advantageous for sparse data sets, whereas resampling permutation procedures are appropriate for both sparse and nonsparse data sets.