Geoffrey R. Barker

Intraclass correlations: A two-facet case study and some comments on the concept of reliability

The decomposition of observed scores into true scores and measurement error provides the classical conceptual framework for the analysis of reliability. The intraclass correlation coefficient (ICC) provides a general reliability coefficient that may be applied in many situations, providing that care is taken to formulate the appropriate ANOVA model. An illustrative analysis is described of a complex design involving two nested sources of error. An appropriate ICC is derived. The value of the information provided by such a reliability study, especially of the variance components, for designing a decision study is discussed. Because ICC is a function of both subject variability and measurement error, difficulties of interpretation can arise. The test theorist’s definition of reliability is argued to conflict with intuitive notions of this concept. Sensitivity coefficient is proposed as a more appropriate term for the classical reliability coefficient.