Roy Freedle

Does the text matter in a multiple-choice test of comprehension? the case for the construct validity of TOEFL's minitalks

The current study addresses a specific construct validity issue regarding multiple-choice language-comprehension tests by focusing on TOEFL’s minitalk passages: Is there evidence that examinees attend to the text passages in answering the test items? To address this problem, we analysed a large sample (n = 337) of minitalk items. The content and structure of the items and their associated text passages were represented by a set of predictor variables that included a wide variety of text and item characteristics identified from the experimental language-comprehension literature. Stepwise and hierarchical regression techniques showed that at least 33% of the item difficulty variance could be accounted for primarily by variables that reflected the content and structure of the whole passage and/or selected portions of the passage; item characteristics, however, accounted for very little of the variance. The pattern of these results was interpreted, with qualifications, as favouring the construct validity of TOEFL’s minitalks. Our methodology also allowed a detailed comparison between TOEFL reading and listening (minitalk) items. Several criticisms concerning multiple-choice language-comprehension tests were addressed. Future work is suggested.

The prediction of TOEFL reading item difficulty: implications for construct validity

The purpose of this study is to predict the difficulty of a large sample (n = 213) of TOEFL reading comprehension items. A related purpose was to examine whether text and text-by-item interaction variables play a significant role in predicting item difficulty. It was argued that evidence favouring the construct validity of multiple-choice reading test formats requires signincant contributions from these particular predictor variables. Details of item predictability and construct validity were explored by evaluating two hypoth eses : 1) that multiple-choice reading comprehension tests are sensitive to 12 categories of sentential and/or discourse variables found to influence compre hension processes in the experimental literature; and 2) that many of these categories of variables identified in the first hypothesis contribute significant independent variance in predicting item difficulty. For the first hypothesis, correlational analyses confirmed the importance of 11 out of the 12 categor ies, while stepwise regression analyses, accounting for up to 58% of the variance, provided some support for the second hypothesis. The pattern of predictors showed that text and text-by-item variables accounted for most of the variance, thereby providing evidence favouring the construct validity of the TOEFL reading items.

MOTHER–INFANT DYAD: THE CRADLE OF MEANING

Publisher Summary The interaction of the infant and its mother reflects a finely tuned and potentially meaning-laden system wherein each allows the other to act. The infant-maternal vocalization data would seem to parallel that of two adults. Vocalization–vocalization pairing is the most common interaction; however, smiling, looking at and fret/cry are instrumental in eliciting and reinforcing a members vocalization. The communication matrix makes clear that a vocalization can be the response of or the elicitor of behaviors other than a vocalization; any model used to study the interactive quality of the vocalization between members of a dyad would need to consider it. The inclusion of all behaviors in any interactive model results in a complex picture. Linguistic competency grows out of the communication matrix and the study of language development must emphasize the issue of function rather than structure. It is believed that context is the prime carrier variable for shared meaning. A complex conditional probability matrix would go a long way in telling the important measures of a communication system, which can predict subsequent vocalization measures.

Predicting black and white differential item functioning in verbal analogy performance

Abstract Differential item functioning (DIF) is one technique for comparing ethnic populations that test makers employ to help ensure the fairness of their tests. The purpose of this ethnic comparison study is to investigate factors that may have a significant influence on DIF values associated with 217 SAT and 234 GRE analogy items obtained by comparing large samples of Black and White examinees matched for total verbal score. In one study, five significant regression predictors of ethnic differences were found to account for approximately 30% of the DIF variance. A second study replicated these findings. These significant ethnic comparisons are interpreted as consistent with a cultural/contextualist framework although competing explanations involving social-economic status and biological contributions could not be ruled out. Practical implications are discussed.

A Developmental Investigation of Standard and Nonstandard English among Black and White Children

Data are reported and interpreted involving language imitation, comprehension, and free production of two English dialects. The major subgroupings of 360 subjects involved two races (black and white), sex, socioeconomic level (low SES and middle SES), and age (5, 8, and 10 years). Rate of improvement measures indicated that blacks improve at the same rate as whites in responding to standard English sentences. According to correlational results, the two dialect systems function behaviorally as separate cognitive systems. In a communication task, blacks and whites produce and comprehend messages of about the same quality

Can Multiple-Choice Reading Tests Be Construct-Valid? A Reply to Katz, Lautenschlager, Blackburn, and Harris

There is a long and continuing history of criticisms of multiple-choice tests of reading (see Farr, Pritchard, & Smitten, 1990). Farr et ai. indicated that of all the criticisms, the most serious maintains that examinees do not or need not read and comprehend the reading selections accompanying the test items at all. A similar criticism appeared in this journal (Katz, Lautenschlager, Blackburn, & Harris, 1990). Katz et ai. presented evidence that examinees were able to perform at better than chance levels of correctness for 100 items taken from the Scholastic Aptitude Tests (SAT) reading sections even when the examinees had not read the passages accompanying those items. In one study, up to 72% of the items were answered correctly at greater than chance levels (20%) in the absence ofthe passage. This work suggests that either information present in the reading items themselves is strongly related to information in the missing passage (and hence the passage need not be present to get the item correct) or the students reasoning ability and possibly background knowledge suffice to guide responses. Because of their findings, Katz et ai. called into question the construct validity of multiple-choice reading tests, and in particular the validity of the SATs reading section; that is, they suggested the test does not measure what it is intended to measure, passage comprehension. Other critics (Royer, 1990) have pointed out that various studies of factors that influence item difficulty in multiple-choice reading tests appear strongly to implicate item features as opposed to text passage features. In this regard, an especially influential study by Drum, Calfee, and Cook (1981) is often cited (see Davey, 1988; Embretson & Wetzel, 1987; Just & Carpenter, 1987; Royer, 1990). Drum et ai. divided several predictor variables into two broad categories: item variables and text variables. The best predictor turned out to be what can be called item plausibility. Without further critical analysis, this finding has been widely cited as evidence against the construct validity of multiple-choice reading tests. That is, the fact that an item variable was by far the strongest predictor of difficulty came to be viewed as evidence against construct validity-reading tests were apparently not measuring passage comprehension. The results of Katz et ai. (1990) underscore this prevailing viewpoint.

The relevance of multiple‐choice reading test data in studying expository passage comprehension: The saga of a 15 year effort towards an experimental/correlational merger

This article argues for the usefulness of merging experimental and correlational approaches in the study of expository prose comprehension. Towards this end I review the studies my colleagues and I have completed in recent years. In the course of reviewing the effects of rhetorical structure, concreteness/abstractness of texts, and other variables, it is shown that the correlational approach yields results similar to the experimental literature. But more importantly, the vast data banks associated with multiple‐choice tests allow us to explore some of the variance that exists across passages and across student ability levels. The types of significant variables studied in explaining especially main idea identification raise issues of how these results might impinge on several prominent theories of prose comprehension. Several suggestions for future research are made.

Effects of mean depth and grammaticality on children's imitations of sentences

This study investigated whether childrens tendency to delete function words and inflections of nouns and verbs (telegraphic speech) in an imitation task can be attributed solely to such factors as relative lack of stress and low-information value. The results indicated that the telegraphic nature of childrens (mean age, 4 years, 4 months) imitations is a result not of their inability to perceive and process articles and inflections, but rather of their tendency to delete these at the time of sentence production. In addition, an application of Yngves sentence mean-depth measure failed to account for relative difficulty of sentence imitations.

Observations with self-embedded sentences using written aids

Self-embedded sentences of any degree beyond one are permitted by the syntactic rules of English, yet previous studies report that Ss typically reject these sentences as being ungrammatical and, in addition, often cannot recover the meaning of these sentences. The present paper investigated the possibility of introducing specially structured self-embedded sentences as “aids” to the discovery of the structure of more complex self-embedded sentences. The group of Ss who received these aids performed significantly better in discovering the subject-verb and subject-object assignments in the complex sentences (many achieved perfect scores) in contrast to Ss who did not initially receive these aids (these latter Ss obtained close to the minimum possible scores).

Dialogue and Inquiring Systems: The Development of a Social Logic

Two aspects of dialectical psychology are considered: (1) The synthesis of a larger cognitive system from two previously separate parent systems; this leads to a consideration of how each parent syste