Anthony Woods

Statistical Aspects of Reliability in Language Testing.

Reliability is a concept central to testing. Whatever measure of reliability is used, its value will always have to be estimated from a sample and will therefore be subject to sampling variation. The magnitude of sampling error involved in measuring reliability from a sample should be indicated by quoting the standard error of a reliability estimate or by giving a confidence interval for its true value. This paper considers several simple ANOVA models which can be used to define and estimate reliability coefficients. It discusses the statistical properties of commonly used measures of reliability and explains how to test hypotheses about their true values.

Statistics in Language Studies: Tables and graphs

When a linguistic study is carried out the investigator will be faced with the prospect of understanding, and then explaining to others, the meaning of the data which have been collected. An essential first step in this process is to look for ways of summarising the results which bring out their most obvious features. Indeed if this is done imaginatively and the trends in the data are clear enough, there may be no need for sophisticated analysis. In this chapter we describe the types of table and diagram most commonly employed for data summary. Let us begin by looking at typical examples of the kind of data which might be collected in language studies. We will consider how, by means of tables, diagrams and a few simple calculations, the data may be summarised so that their important features can be displayed concisely. The procedure is analogous to writing a precis of an article or essay and has similar attractions and drawbacks. The aim is to reduce detail to a minimum while retaining sufficient information to communicate the essential characteristics of the original. Remember always that the use of data ought to enrich and elucidate the linguistic argument, and this can often be done quite well by means of a simple table or diagram. Categorical data It quite commonly arises that we wish to classify a group of people or responses or linguistic elements, putting each unit into one of a set of mutually exclusive classes.

Analysis of variance – ANOVA

In the last chapter we explained how it was possible to test whether two sample means were sufficiently different to allow us to conclude that the samples were probably drawn from populations with different population means. When more than two different groups or experimental conditions are involved we have to be careful how we test whether there might be differences in the corresponding population means. If all possible pairs of samples are tested using the techniques suggested in the previous chapter, the probability of type i errors will be greater than we expect, i.e. the ‘significance’ of any differences will be exaggerated. In the present chapter we intend to develop techniques which will allow us to investigate possible differences between the mean results obtained from several (i.e. more than two) samples, each referring to a different population or collected under different circumstances. Comparing several means simultaneously: one-way ANOVA Imagine that an investigator is interested in the standard of English of students coming to Britain for graduate training. In particular he wishes to discover whether there is a difference in the level of English proficiency between groups of distinct geographical origins – Europe, South America, North Africa and the Far East. As part of a pilot study he administers a multiple choice test to 40 graduate students (10 from each area) drawn at random from the complete set of such students listed on a central file. The scores obtained by these students on the test are shown in table 12.1.