Kyle Perkins

An exploratory study into the construct validity of a reading comprehension test: triangulation of data sources

Recent research in reading comprehension has focused on the processes of reading, while recent thinking in language testing has recognized the impor tance of gathering information on test taking processes as part of construct validation. And while there is a growing body of research on test-taking strate gies in language testing, as well as research into the relationship between item content and item performance, no research to date has attempted to examine the relationships among all three - test taking strategies, item content and item performance. This study thus serves as a methodological exploration in the use of information from both think-aloud protocols and more commonly used types of information on test content and test performance in the investigation of construct validity.

Using Objective Methods of Attained Writing Proficiency to Discriminate among Holistic Evaluations.

Ten objective measures of writing proficiency were used to evaluate compositions written by advanced ESL students. Only those objective measures which take the absence of errors into account discriminated among holistic evaluations of the compositions by experienced ESL teachers. A standardized objective writing test failed to discriminate among the holistic evaluations as well as the error-free objective measures did.

First and Second Language Reading Comprehension.

Random parallel reading comprehension tests in Japanese and English were administered to a sample of native Japanese students enrolled in intensive English instruction at three different levels of English language proficiency as assessed by an independent measure. Evidence for a threshold competence ceiling at which first language reading abilities transferred to second language reading abilities was found. At the highest proficiency level, those readers who scored high on the first language reading test also systematically scored high on the second language reading test. Pedagogical implications of the study are discussed.

Predicting Item Difficulty in a Reading Comprehension Test with an Artificial Neural Network.

This article reports the results of using a three-layer backpropagation artificial neural network to predict item difficulty in a reading comprehension test. Two network structures were developed: one with the sigmoid function in the output processing unit and the other without the sigmoid function in the output processing unit. The dataset which consisted of a table of coded test items and corresponding item difficulties was partitioned into a training set and a test set in order to train and test the neural networks. To demonstrate the consistency of the neural networks in predicting item difficulty, the training and testing runs were repeated four times starting with a new set of initial weights. Additionally, the training and testing runs were repeated by switching the training set and the test set. The mean squared error values between the actual and predicted item difficulty demonstrated the consistency of the neural networks in predicting item difficulty for the multiple training and testing runs. Significant correlations were obtained between the actual and predicted item difficulties and the Kruskal-Wallis test indicated no significant difference in the ranks of actual and predicted values.

Measuring Passage Contribution in ESL Reading Comprehension

Reading comprehension items are valid to the extent that they measure what subjects have understood of the stimulus material. This article reports an empirical analysis of two administrations of two reading tests: the first time, without the reading passages, and the second time, with the passages. Data from the two administrations were used to calculate the passage dependency of each test, that is, the extent to which questions can be answered without reading the texts upon which the questions are based. The two tests in this research, Davis Reading Test (Davis and Davis 1956) and Cooperative English Tests (Educational Testing Service 1960), exhibited little passage dependency. The stability of item types across the two presentation conditions is discussed, and a hierarchy of item-type difficulty is established using latent trait measurement logits of difficulty. The results of this study suggest that classroom teachers should examine commercially available tests carefully for passage dependency. Furthermore, in constructing reading tests, teachers should avoid writing items that test general knowledge. Instead, teachers should strive to write items that test memory organization and that reveal whether or not inferences have been drawn.

From Conventional to Computer-Adaptive Testing of ESL Reading Comprehension.

Abstract In this paper we describe the development of a computer-adaptive test of ESL reading comprehension. The computer-adaptive test was based on items from a battery of four conventional fixed-form tests and was constructed using an adaptive test development system for the Macintosh HyperCAT. The application of computer-adaptive testing (CAT) to the assessment of reading comprehension and other abilities in a second language serves as an illustration in this paper. We discuss the constraints that apply to CAT and the substantial advantages that CAT has over conventional testing modalities. The constraints on CAT include: the unidimensionality of the test, the homogeneity of the test population, and a neutral test method effect.

The effect of passage topical structure types on ESL reading comprehension difficulty

This study investigated the effect of five types of topical structure on the item difficulty of reading comprehension questions. A topical structure includes the initial sentence element which is the initially placed discourse material in a sentence; a mood (surface structure) subject; and a topical subject which is a surface subject relating to the discourse topic of a text. The results indicated that there was a difference in the item difficulty of the questions according to the type of topical structure on which the questions were based. The questions derived from sentences in which given information occurred before new information and from sentences in which topical information occurred in subject position tended to be easier than questions derived from sentences manifesting other types of topical structure. The article concludes with a suggestion that topical structure types be included as a variable in future models of ESL reading comprehension difficulty.

An investigation of patterns of discontinuous learning: implications for ESL measurement

Research from cognitive psychology, L2 acquisition and psychometry sug gests that competence/proficiency is multidimensional, and that proficiency does not always develop at the same rate in all its domains. In particular, L2 acquisition research indicates that competence does not always develop in a linear, monotonic pattern at an equal rate in all its component parts owing to restructuring and reorganization. Rehearsal, automatic and controlled pro cessing, creative simplification, task constraints and attention to form have been adduced to explain restructuring and reorganization. Classical test theory and item response theory models are not consonant with findings from cognitive psychology and L2 acquisition, because these models pinpoint individuals on a single continuum of proficiency. The study reported here tested the experimental hypothesis that there would not be a monotonic relationship between increasing competence in different linguistic domains and growth in L2 proficiency. There was general support for the hypothesis. The article concludes with a brief introduction to some assessment models which can accommodate discontinuous patterns of growth. The models examined can be used to describe and model a students qualitative cognitive state and knowledge structures.

Risk-factor fusion for predicting multifactorial diseases

A generalized classification methodology is developed to predict the presence or absence of a multifactorial disease from a set of risk factors thought to be correlated with the disease. The methodology includes fusion to combine risk factors into a single feature vector, normalization to overcome the problems associated with fusing features which have different formats and ranges, discrete Karhunen-Loeve transform (DKLT)-based transformation to facilitate parametric classifier development, the selection of features with high interclass separations, and the design of parametric classifiers. The validity of the method is demonstrated by applying it to predict the occurrence of gout from 14 risk factors. Cross-validation evaluations on 96 patients, 48 clinically diagnosed to have gout and 48 diagnosed to not have gout, showed that an average classification accuracy of 75.7% can be obtained. Even more promising is that higher classification accuracies can be achieved through the careful selection of the DKLT transformation matrix which in turn involves selecting design sets that are good representatives of the gout and nongout classes. It is concluded that the generalized methodology developed in this paper is quite effective in predicting multifactorial diseases and can, therefore, assist/support a physician in diagnosing a multifactorial disease.

ESL Reading Comprehension Instruction.

Observational studies have shown that very little time is devoted to comprehension instruction in reading classes. In addition, there is a tendency for reading students to practice skills on their own and to determine for themselves how and where to use those skills. In this paper the authors present some techniques for comprehension instruc tion and some guidance on how to teach students to apply the reading skills they have learned.