Stuart Shaw

Disagreement over the best way to use the word ‘validity’ and options for reaching consensus

The ability to convey shared meaning with minimal ambiguity is highly desirable for technical terms within disciplines and professions. Unfortunately, there is no widespread professional consensus over the meaning of the word ‘validity’ as it pertains to educational and psychological testing. After illustrating the nature and extent of disagreement, we consider three options for reaching consensus: to eliminate its ambiguity by agreeing a precise technical definition; to embrace its ambiguity by agreeing a catchall lay usage; and to retire ‘validity’ from the testing lexicon.

A Framework for Evidencing Assessment Validity in Large-Scale, High-Stakes International Examinations.

It is important for educational assessment bodies to demonstrate how they are seeking to meet the demands of validity. The approach to validity taken here assumes a ‘consequentialist’ view where the appropriacy of the inferences made on the basis of assessment results is seen as central. This paper describes the development of a systematic approach to the collection of evidence that can support claims about validity for general qualifications. An operational framework was developed drawing on Kane (2006). The framework involves a list of inferences to be justified as indicated by a number of linked validation questions. For each question various data would be gathered to provide ‘evidence for validity’ and to identify any ‘threats to validity’. The structure is designed to be accessible for operational users. This paper describes the development of the proposed framework and the types of methods to be used to gather relevant evidence.

Assessment of International Students through the Medium of English: Ensuring Validity and Fairness in Content-Based Examinations.

International assessments in a wide range of subjects are being prepared for and delivered through the medium of English in a variety of educational contexts. These assessments are taken by many candidates whose first language is not necessarily English. This raises important issues relating to assessment validity and fairness. This study describes an analysis of the linguistic demands in the Cambridge International General Certificate of Secondary Education (IGCSE)—a major high-stakes qualification for 16-year-olds. The focus of the study is IGCSE History, Biology, and Geography. Data for analysis includes syllabuses, question papers, mark schemes, and candidate performances. Findings suggest that each subject necessarily requires academic language (Cognitive Academic Language Proficiency) and some of the key academic language skills are identified. An average language proficiency level of B2 on the Common European Framework of References for Languages (CEFR) is useful to access typical IGCSE exams, and CEFR level of C1 could provide an added advantage of linguistic resources to be able to develop arguments needed for higher grades for Humanities subjects such as History and Geography. The outcomes have contributed to the question writing process and have informed the construction of a toolkit for content and language teachers.

Applying methods to evaluate construct validity in the context of A level assessment

Validity is a central principle of assessment relating to the appropriateness of the uses and interpretations of test results. Usually, one of the inferences that we wish to make is that the score reflects the extent of a student’s learning in a given domain. Thus, it is important to establish that the assessment tasks elicit performances that reflect the intended constructs. This research explored the use of three methods for evaluating whether there are threats to validity in relation to the constructs elicited in international A level geography examinations: (a) Rasch analysis; (b) analysis of processes expected and apparent when students answer questions; and (c) qualitative analysis of responses to items identified as potentially problematic. The results provided strong evidence to support validity with regard to the elicitation of constructs although one question part was identified as a threat to validity. Strengths and weaknesses of the methods can be identified.

Handbook of test development – review of section 1, (foundations) edited by Suzanne Lane, Mark R. Raymond, and Thomas M. Haladyna

According to Samuel Messick, ‘The primary measurement standards that must be met to legitimise a proposed test use are those of reliability, validity, and fairness’ (1988, p. 1). Value-laden and intertwined concepts, validity, reliability and fairness not only constitute the focus of the three opening chapters in the Standards for Educational and Psychological Testing (American Educational Research Association (AERA), American Psychological Association, & National Council on Measurement in Education, 2014) but they are also core principles in test design and development. Assessment design is intricately associated with the evidence needed to support the validity, reliability and fairness of intended interpretations of test scores for specified purposes and intended uses. Assessment providers – even those that have only limited resources – have a responsibility to demonstrate the quality and validity of their assessments; in particular, whether the inferences made on the basis of the outcomes of the assessments are appropriate. Grounded in the recently revised Standards for Educational and Psychological Testing (AERA et al., 2014), the six chapters fronting the second edition of the Handbook of Test Development are appropriately grouped under the over-arching title Foundations. The authors of these chapters consistently attest to the primacy of the three measurement standards and remind the would-be test developer of the need to attend to the fundamental concerns that drive all test development activities and inform test design and validation. The authors argue that successful tests, that is, ones that achieve validity and fulfil a useful purpose in the contexts in which they are to be used, cannot be developed without due consideration being given to the following test development criteria:

Assessment for Learning in International Contexts: exploring shared and divergent dimensions in teacher values and practices

The Assessment for Learning in International Contexts (ALIC) project sought to extend knowledge around teachers’ understandings of Assessment for Learning (AfL). Using a modified version of a survey item devised by James and Pedder for use with teachers in England, evidence was gathered about the assessment practices that were highly valued by teachers across international contexts. The extent of congruence between these values and teachers’ reported classroom practices was explored and dimensions of teachers’ assessment practices were derived through factor analysis. While there was considerable congruence across the ALIC cohort of teachers and data sets derived from English teachers, particularly with respect to the items that have positive values–practice gaps, there were some interesting differences. Two components were derived from factor analysis, rather than the three derived by James and Pedder. These components were ‘Making learning explicit and promoting learner autonomy’ and ‘Student control of assessment processes’.

Towards an operational framework for establishing and assessing collaborative interactions

Abstract Collaboration skills have been identified as an important educational outcome, not just as a means to enable learning mechanisms to be enacted, but also as a demonstrable skill of interest to assessment practitioners. In recent years, there has been an ambition to develop models of assessment that are underpinned by the collaborative process, and directly assess collaborative abilities of individuals – an ‘assessment imperative’. Defining collaboration, however, has important implications for how it is appropriately measured. In this paper, we present a new framework that educational practitioners (teachers, awarding organisations and test developers) can utilise when attempting to set optimal assessment tasks from which constructs of collaboration can be engendered (and thus observed). This pragmatic framework is presented with reference to three pertinent, interrelated issues in developing assessments that target collaboration: initial construct definition; enabling ‘true’ collaboration formation; and the assessment model for collaboration. This framework is a first attempt at reconciling tensions at the heart of the assessment of collaboration, namely the imperative for assessors to arrive at a judgement on individuals’ collaborative skills, and the related difficulties in optimising group dynamics.