Irwin Kirsch

Examining the relationship between computer familiarity and performance on computer-based language tasks

The planned introduction of a computer-based Test of English as a Foreign Language (TOEFL) test raises concerns that language proficiency will be confounded with computer proficiency, introducing construct-irrelevant variance to the measurement of examinees English-language abilities. We administered a questionnaire focusing on examinees computer familiarity to 90,000 TOEFL test takers. A group of 1,200 “low-computer-familiar” and “high-computer-familiar” examinees from 12 international sites worked through a computer tutorial and a set of 60 computer-based TOEFL test items. We found no meaningful relationship between level of computer familiarity and level of performance on the computerized language tasks after controlling for English language ability. We concluded that no evidence exists of an adverse relationship between computer familiarity and computer-based TOEFL test performance due to lack of prior computer experience.

The framework used in developing and Interpreting the International Adult Literacy Survey (IALS)

This paper offers a framework that has been used for both developing the tasks used to measure literacy in the International Adult Literacy Survey and for understanding the meaning of what has been reported with respect to the comparative literacy proficiencies of adults in participating countries. The framework consists of six parts that represent a logical sequence of steps from needing to define and represent a particular domain of interest, to identifying and operationalizing characteristics used to construct items, to providing an empirical basis for interpreting results. The various parts of the framework are seen as important in that they help to provide a deeper understanding of the construct of literacy and the various processes associated with it. A processing model is proposed and variables associated with performance on the literacy tasks are identified and verified through regression analyses. These variables are shown to account for between 79 and 89 percent of the variance in task difficulty. Collectively, these process variables provide a means for moving away from interpreting performance on large-scale surveys in terms of discrete tasks or a single number towards identifying levels of performance that have generalizability across pools of tasks and towards what Messick has called a higher level of measurement.RésuméL’article présente un cadre qui permet l’élaboration de tâches pour mesurer la compréhension de l’écrit ainsi qu’un dépistage plus précis du sens des données analogues dont nous disposons relatives à la compréhension de l’écrit parmi les adultes des pays participants. Le cadre consiste de six parties dont la séquence logique va de la nécessité de définir et de représenter un domaine d’intérêt spécifique à l’établissement d’une base empirique pour l’interprétation des résultats, en passant par l’identification et l’opérationnalisation des caractéristiques qui entrent dans la construction des items. L’importance des éléments de ce cadre se manifeste par leur potentiel de contribuer à la compréhension approfondie de la notion de compréhension de l’écrit et des processus divers qui lui sont associés. On propose un modèle de processus mental et procède à l’identification et à la vérification, moyennant des analyses de régression, des variables déterminant la performance dans les tâches de compréhension de l’écrit. On montre que ces variables expliquent de 79% à 89% de la variance relative à la difficulté des tâches. Dans leur totalité, ces variables de processus mental permettent de sortir du mesurage, courant dans les enquêtes à grande échelle, de la performance par tâches discrètes ou par simple échelle numérique pour arriver à une identification des niveaux de performance propres à être généralisés pour des groupes entiers de tâches et, par là, à ce que Messick a appelé un niveaux supérieur de measure.

The Dynamics of Opportunity in America

Opportunity has long been the bedrock of American society. Today, however, the solid foundation that once grounded the lives of millions is fracturing along economic and social lines. Human capital, encompassing a broad set of cognitive and interpersonal skills, has become increasingly important in determining labor market outcomes as the evolving economic landscape, shaped by the interplay of globalization and technology, as well as governmental and business policies, changes who is working and what they are paid. There is now also a tighter link between human and social capital, which is the set of networks, norms, and values that serve to foster development and success. The strengthening of this relationship has contributed to a polarization in the accumulation of human and social capital that translates into distinctly different life outcomes for individuals. This changing landscape also affects the intergenerational transmission of opportunity, with children’s circumstances at birth becoming more determinative of their prospects as adults. This introduction sets the stage for the chapters that follow, which offer perspectives on opportunity from fi elds ranging from education and demography to economics and political science. The authors of these chapters, national leaders in their fi elds, offer their insights into policies and practices that could help us move forward to improve equality of opportunity and better realize America’s values and ideals.

Large-Scale Assessments of Adult Literacy

In this chapter, ETS’s work in large-scale adult literacy assessments is described. This work has been designed to meet policy needs, both in the United States and internationally, based on the growing awareness of literacy as human capital. The impact of these assessments has grown as policy makers and other stakeholders have increasingly come to understand the critical role that foundational skills play in allowing individuals to maintain and enhance their ability to meet changing work conditions and societal demands. Findings from these surveys have provided a wealth of information about how the distribution of skills is related to social and economic outcomes. Of equal importance, the surveys and associated research activities have contributed to large-scale assessment methodology, the development of innovative item types and delivery systems, and methods for reporting survey data in ways that ensure its utility to a range of stakeholders and audiences.

Growth in Reading Comprehension and Verbal Ability From Grades 1 Through 9

We systematically assessed the relationships between growth of four components of verbal ability—Information, Similarities, Vocabulary, and Comprehension subtests of the Wechsler Intelligence Scale–Revised—and longitudinal growth from Grades 1 to 9 of the Woodcock–Johnson Psycho-Educational Battery Passage Comprehension subtest while controlling for Word Identification and Word Attack, using multilevel growth models on a sample of 414 children. Growth was assessed over all grades (1-9), and separately for early grades (1-5) and later grades (5-9). Over all grades, growth in Word Identification had a substantial standardized loading to Passage Comprehension, and all four verbal abilities had smaller, but significant standardized loadings to Passage Comprehension (p < .05), with Information and Vocabulary having slightly higher loadings than Similarities and Comprehension. For early grades, results were similar to the overall results, with the exception of Vocabulary, which had a nonsignificant loading to Passage Comprehension. For later grades, Word Identification again had the largest, but substantially smaller standardized loading on Passage Comprehension and standardized loadings of all four verbal abilities were statistically significant with Vocabulary and Wechsler Intelligence Scale for Children–Revised (WISC-R) Comprehension having appreciably higher loadings than in the previous analyses. Conversation- and interaction-based intervention and instruction in oral language in general, and vocabulary in particular throughout early childhood and continuing throughout the school years, combined with evidence-based instruction that systematically develops the skills of phonologic awareness, decoding, word reading, fluency, and comprehension in school, may provide a pathway to reducing the achievement gap in reading.

Introduction: Opportunity in America—Setting the Stage

Opportunity has long been the bedrock of American society. Today, however, the solid foundation that once grounded the lives of millions is fracturing along economic and social lines. Human capital, encompassing a broad set of cognitive and interpersonal skills, has become increasingly important in determining labor market outcomes as the evolving economic landscape, shaped by the interplay of globalization and technology, as well as governmental and business policies, changes who is working and what they are paid. There is now also a tighter link between human and social capital, which is the set of networks, norms, and values that serve to foster development and success. The strengthening of this relationship has contributed to a polarization in the accumulation of human and social capital that translates into distinctly different life outcomes for individuals. This changing landscape also affects the intergenerational transmission of opportunity, with children’s circumstances at birth becoming more determinative of their prospects as adults. This introduction sets the stage for the chapters that follow, which offer perspectives on opportunity from fields ranging from education and demography to economics and political science. The authors of these chapters, national leaders in their fields, offer their insights into policies and practices that could help us move forward to improve equality of opportunity and better realize America’s values and ideals.