Antony John Kunnan

An Introduction to Structural Equation Modelling for Language Assessment Research.

This article provides an introduction to structural equation modelling (SEM) for language assessment research in five sections. In Section I, the general objectives of SEM applications relevant to language assessment are presented. In Section II, a brief overview of SEM that considers the methodology and the statistical assumptions about data that have to be met. In Section III, the commonly used steps and concepts in SEM are presented. In Section IV, SEM application matters with example models are discussed. In Section V, recent critical discussions and some directions for future SEM applications in language assessment research are addressed.

Test fairness and Toulmin’s argument structure

Test fairness as a fundamental concept in the evaluation of tests has been in the forefront of discussions in the field of language assessment from the late 1990s. An early such plea was a tentative fairness research agenda that I presented at the Language Testing Research Colloquium, Tampere, in 1996. I wrote then: ‘Although validation studies are said to generally take on the role of investigating the fairness of tests and testing practices, an examination of about 100 validation studies shows that the themes addressed by the researchers are not particularly concerned with fairness’ (Kunnan, 1997, p. 85). My interest in this issue was further invigorated as the theme of the LTRC, Orlando, in 1997 was ‘Fairness in language testing.’ But many scholars criticized the concept of test fairness as over-reaching and ambiguous and that such investigations were already part of the validation research agenda. At about this time, the 1999 Standards (AERA, APA, NCME, 1999) was published with a section entitled ‘Fairness in testing’ with chapters on testing and test use, rights and responsibilities of test takers, test takers with linguistic diversity, and test takers with disabilities. This publication renewed interest in test fairness but as Xi rightly points out, test fairness was now seen as an additional test quality test developers were asked to pay attention to rather than as an integral part of test development and testing practice.

Testing for Citizenship: The U.S. Naturalization Test

Although the United States has had a language requirement for citizenship (through the naturalization process) since the first decade of the 20th century, very little research and discussion has taken place in academic circles about this requirement, which has been enforced for the past two decades through a test and testing practice to which millions of citizenship applicants have been subjected. Overall, the requirement (and the test) is said to promote “civic integration,” “political allegiance,” “social cohesion,” and/or “social harmony.” The main question addressed in this article is whether the requirement (and the test), its purpose, content, administration, and consequence provide the impetus for citizenship applicants to integrate or develop a civic nationalism. I will apply the Test Context Framework (Kunnan, 2008) to the U.S. Naturalization Test and testing practice. This framework argues for the examination of the wider context of testing including the political, economic, and legal contexts in which a test is deployed for a fuller understanding of the testing practice.

Differential Item Functioning in Terms of Age in the Certificate in Advanced English Examination.

When standardized English-language tests are administered to test takers worldwide, the test-taking population could be varied on a number of personal and educational characteristics such as age, gender, first language, and academic discipline. As test tasks and test items may not always be prepared keeping this diversity of characteristics in mind, it is essential for test developers to continuously monitor their tests in terms of whether all test takers are receiving a fair test. This study investigates whether the test items on the listening section of the Certificate in Advanced English examination functioned differently for test takers from three different age groups. The main results showed that although statistical and content analyses procedures detected differential item functioning in a few items, expert judges could not clearly identify the sources of differential item functioning for the items.

Modelling relationships among some test-taker characteristics and performance on EFL tests: an approach to construct validation

Construct validation has seen two proposals recently: the use of construct representation and nomothetic span variables through structural modelling and the concept of population generalizability. This study investigated the influence of two major test-taker characteristics (TTCs), social milieu or cultural background and exposure or previous instruction, on test perform ance (TP) in tests of English as a foreign language (EFL) for two native language groups, the non-Indo-European (NIE) and the Indo-European (IE). Data from the Cambridge-TOEFL comparability study (Bachman et al., 1991; N = 1448) from eight sites in eight countries was used. The instruments were 1) a 45-item Likert scale background questionnaire which captured the test-taker characteristics; and 2) the First Certificate in English, administered by the University of Cambridge Local Examinations Syndicate, the TOEFL and the SPEAK, administered by the Educational Testing Service, Princeton, and the Test of English Writing. Modelling of the TT...Construct validation has seen two proposals recently: the use of construct representation and nomothetic span variables through structural modelling and the concept of population generalizability. This study investigated the influence of two major test-taker characteristics (TTCs), social milieu or cultural background and exposure or previous instruction, on test perform ance (TP) in tests of English as a foreign language (EFL) for two native language groups, the non-Indo-European (NIE) and the Indo-European (IE). Data from the Cambridge-TOEFL comparability study (Bachman et al., 1991; N = 1448) from eight sites in eight countries was used. The instruments were 1) a 45-item Likert scale background questionnaire which captured the test-taker characteristics; and 2) the First Certificate in English, administered by the University of Cambridge Local Examinations Syndicate, the TOEFL and the SPEAK, administered by the Educational Testing Service, Princeton, and the Test of English Writing. Modelling of the TTCs and the TP factors generally supported an equal influence factors model (where the factors have equal status) and an intervening factors type model (where the factors are not equal in status and one factor is an intervening factor) for both the NIE and IE groups.

Analysis of Test Takers’ Metacognitive and Cognitive Strategy Use and EFL Reading Test Performance:A Multi-Sample SEM Approach

This study investigates the relationships between test takers’ metacognitive and cognitive strategy use through a questionnaire and their test performance on an English as a Foreign Language reading test. A total of 593 Chinese college test takers responded to a 38-item metacognitive and cognitive strategy questionnaire and a 50-item reading test. The data were randomly split into two samples (N = 296 and N = 297). Based on relevant literature, three models (i.e., unitary, higher order, and correlated) of strategy use and test performance were hypothesized and tested to identify the baseline model. Further, cross-validation analyses were conducted. The results supported the invariance of factor loadings, measurement error variances, structural regression coefficients, and factor variances for the unitary model. It was found that college test takers’ strategy use affected their lexico-grammatical reading ability significantly. Findings from this study provide empirical and validating evidence for Bachman and Palmers (2010) model of strategic competence.

Response to Henning

In his review of Bachman et al. (1988), Henning makes several useful methodological criticisms, but the overall thrust of his review is to raise questions about the credibility of this study and to adopt an attitude of partisan defensiveness in behalf of the TOEFL programme, including its tests and research. Henning also criticizes a larger study which has not yet been reported in its entirety, of which the Bachman et al. article was a precursor. We find it extraordinary that Henning would presume to citicize a study yet to be published and believe that his comments on this ’forthcoming study’ are premature. Furthermore, we feel that his attempts to discredit this study by questioning its sponsorship, along with his institutional bias and defensiveness, are out of place in a professional forum. We will first address the issues of the sponsorship of the study and of independence in research and then the specific methodological points that Henning raises with regard to the Bachman et al. article.

The Duolingo English Test

This article provides a critical conceptual review of the Duolingo English test (DET) and explore possible consequences of its use for university admissions purposes. Because the DET is very new, t...

Automated Essay Evaluation for English Language Learners:A Case Study of MY Access

ABSTRACT Computer technology made its way into writing instruction and assessment with spelling and grammar checkers decades ago, but more recently it has done so with automated essay evaluation (AEE) and diagnostic feedback. And although many programs and tools have been developed in the last decade, not enough research has been conducted to support or evaluate the claims of the developers. This study examined the effectiveness of automated writing instructional programs in consistent scoring of essays and appropriate feedback to student writers. It examined the scoring and instructional program called MY Access! Home Edition, which has an error feedback tool called My Editor to address these issues. The study combined a quantitative study of agreement and correlational analyses with an analysis of content and topic. Participants included 114 English language learners who wrote 147 essays to three writing prompts, which were graded by trained EFL raters and MY Access. From the sample, 15 randomly selected essays were also used for an error analysis comparing My Editor with human annotations to examine My Editor’s accuracy. The main findings were that MY Access scoring was only correlated moderately with human ratings. Furthermore, because MY Access scoring is limited to the recognition of content words, not how these words are organized at the discourse level, it did not detect slightly off-topic essays and plagiarism. Finally, My Editor’s error feedback, with 73% precision and 30% recall, did not meet the expectations of an accurate tool. In conclusion, the home edition of MY Access was not found to be useful as an independent instructional tool. These findings give us pause regarding the effectiveness of MY Access.

Examining the Inseparability of Content Knowledge from LSP Reading Ability: An Approach Combining Bifactor-Multidimensional Item Response Theory and Structural Equation Modeling.

ABSTRACT This study examined the separability of domain-general and domain-specific content knowledge from Language for Specific Purposes (LSP) reading ability. A pool of 1,491 nursing students in China participated by responding to a nursing English test and a nursing knowledge test. Primary data analysis involved four steps: (a) conducting a bifactor-multidimensional item response theory model (bifactor-MIRT) analysis to establish measurement validity for the assumed domain-general factor and domain-specific factors underlying each test and to compute bifactor-MIRT direct scores; (b) transforming the bifactor-MIRT scores into composite scores; (c) conducting a confirmatory factor analysis with the composite scores to reconstruct the orginal bifactor-MIRT models, and (d) conducting a structural equation modeling analysis to explore the relationship between nursing knowledge factors (domain-general and domain-specific) and the nursing English reading factors (domain-general and domain-specific). The results showed that the domain-specific passage factors were significantly correlated with their corresponding domain-specific nursing knowledge factors and that domain-general nursing knowledge significantly predicted the variance of the domain-general reading factor. Overall, we concluded that content knowledge is inseparable from LSP reading ability. The implications for understanding LSP ability and for LSP reading test scoring are discussed.