Therese N. Hopfenbeck

Assessment and learning: fields apart?

Abstract Educational assessments define what aspects of learning will formally be given credit and therefore have a huge impact upon teaching and learning. Although the impact of high-stakes national and international assessments on teaching and learning is considered in the literature, remarkably, there is little research on the connection between theories of learning and educational assessments. Given the voluminous assessment that takes place annually in systematic ways in most many nations, it is surprising that more has not been gained from these assessments in the development of theories of learning and vice versa. In this article, we consider both theories of learning and assessment and draw the main message of the article, that if assessments are to serve the goals of education, then theories of learning and assessment should be developing more closely with each other. We consider fundamental aspects of assessment theory, such as constructs, unidimensionality, invariance and quantifiability, and in doing so, we distinguish between educational and psychological assessment. Second, we show how less traditionally considered cases of (a) international assessments and (b) Assessment for Learning affect student learning. Through these cases we illustrate the otherwise somewhat theoretical discussion in the article. We argue that if assessment is to serve the learning goals of education, then this discussion on the relationship between assessment and learning should be developed further and be at the forefront of high-stakes, large-scale educational assessments.

Balancing tensions in educational policy reforms: large-scale implementation of Assessment for Learning in Norway

This study investigates how different stakeholders in Norway experienced a government-initiated, large-scale policy implementation programme on Assessment for Learning (AfL). Data were collected through 58 interviews with stakeholders in charge of the policy; Ministers of Education and members of the Directorate of Education and Training in Norway, and the main actors such as municipality leaders, teachers, school leaders and students. Successful implementation of AfL processes was found in municipalities where there were dialogue and trust between the municipality level, school leaders, teachers and students and where the programme was adapted to the local context. Implementation was challenged when the policy was interpreted as a way of controlling the schools. Despite the successful implementation in some municipalities, the programme did not have any effect upon students’ learning outcome, as measured on national tests in reading and mathematics. The results are discussed in relation to how local assessment cultures with particular characteristics influence governing, accountability and trust.

On the supranational spell of PISA in policy

Abstract Background: PISA results appear to have a large impact upon government policy. The phenomenon is growing, with more countries taking part in PISA testing and politicians pointing to PISA results as reasons for their reforms. Purpose: The aims of this research were to depict the policy reactions to PISA across a number of jurisdictions, to see whether they exhibited similar patterns and whether the same reforms were evident. Sources of evidence: We investigated policy and media reactions to the 2009 and 2012 PISA results in six cases: Canada, China (Shanghai), England, France, Norway and Switzerland. Cases were selected to contrast high-performing jurisdictions (Canada, China) with average performers (England, France, Norway and Switzerland). Countries that had already been well reported on in the literature were excluded (Finland, Germany). Design and methods: Policy documents, media reports and academic articles in English, French, Mandarin and Norwegian relating to each of the cases were critically evaluated. Results: A policy reaction of ‘scandalisation’ was evident in four of the six cases; a technique used to motivate change. Five of the six cases showed ‘standards-based reforms’ and two had reforms in line with the ‘ideal-governance’ model. However, these are categorisations: the actual reforms had significant differences across countries. There are chronological problems with the notion that PISA results were causal with regard to policy in some instances. Countries with similar PISA results responded with different policies, reflecting their differing cultural and historical education system trajectories. Conclusions: The connection between PISA results and policy is not always obvious. The supranational spell of PISA in policy is in the way that PISA results are used as a magic wand in political rhetoric, as though they conjure particular policy choices. This serves as a distraction from the ideological basis for reforms. The same PISA results could motivate a range of different policy solutions.

Large-scale implementation of Assessment for Learning

The publication of Black and Wiliam’s (1998) ‘Assessment and classroom learning’ in this journal and their widely disseminated pamphlet ‘Inside the black box: raising standards through classroom assessment’ initiated an international implementation of what is now generally known as Assessment for Learning. Its take-up has been described as a ‘research epidemic’ which in a relatively few years has ‘feverishly spread into every discipline and professional field’ (Steiner-Khamsi, 2004, p. 2). This is reflected in the range of contributors to this issue in which large-scale initiatives from eight different counties are reported. A recent review of the state of the field (Baird, Hopfenbeck, Newton, Stobart, & Steen-Utheim, 2014) found 907 peer-reviewed articles and 481 conference proceedings on Assessment for Learning and formative assessment. Fewer than 10 of these studies could be described as large-scale, the vast majority being case studies in one or two schools, with relatively few students. However, Assessment for Learning has been taken up widely at national and regional levels though we have as yet relatively little evidence of how successful these policies have been. This Special Issue is therefore a timely attempt to draw together research that has looked at large-scale implementation, either in the form of national policy or involving relatively large numbers of schools and teachers. In looking at the implementation of Assessment for Learning policies at the national level, we become aware of the importance of the social and educational contexts into which they are being introduced. A common theme here is the impact of summative assessment and accountability measures on the implementation of Assessment for Learning. In the opening article, María Teresa Flórez Petour uses her case study of assessment policy to demonstrate how the complex interaction of systems, ideologies and history impacts on the take-up and implementation of Assessment for Learning in Chile. Given this complexity how do we implement Assessment for Learning practices on a large scale, given many schools and teachers may be conscripts rather than enthusiastic volunteers? It is not simply a question of making clear what Assessment for Learning involves, we need mechanisms for supporting its introduction. This raises questions about how assessment reform can be more effectively implemented. Several of the large-scale initiatives adopt an explicit model of implementation. The studies from Sweden (Jonsson, Holmgren and Lundahl), Canada (DeLuca, Klinger, Piper and Woods), the USA (Wylie and Lyon) and Scotland (Hayward) utilise a variety of implementation models drawn from research. In the case of Norway (Hopfenbeck, Tolo and Flórez Petour), the dissemination model, a four-year rolling programme of working from the policy centre with volunteer groups, is less theoretically explicit but relies on continuous adaptation. The study of a national initiative in Trinidad and Tobago (De Lisle) demonstrates how the

Examining Evidence for the Validity of PISA Learning Strategy Scales Based on Student Response Processes

The aim of this study was to investigate response-process based evidence for the validity of the Programme for International Student Assessments (PISA) self-report questionnaire scales as measures of specific psychological constructs, with a focus on scales meant to measure inclination toward specific learning strategies. Cognitive interviews (N = 22) and descriptive and statistical analysis of Norwegian questionnaire data (N = 4279) provide converging evidence that (a) a non-trivial portion of students may not be providing responses to items that can be considered valid, for a variety of reasons; (b) students who perform more poorly on PISAs science scales are less likely to provide valid responses to the questionnaire items; and (c) quantitative examinations of response patterns provide limited ways of identifying students who may not be providing valid responses, although changes in the questionnaire design could strengthen the utility of these methods. These findings are discussed in the context of the larger interpretive argument surrounding the PISA learning strategy scales, and implications for future research on the measurement of learning strategies are discussed.

Lessons Learned from PISA: A Systematic Review of Peer-Reviewed Articles on the Programme for International Student Assessment.

ABSTRACT International large-scale assessments are on the rise, with the Programme for International Student Assessment (PISA) seen by many as having strategic prominence in education policy debates. The present article reviews PISA-related English-language peer-reviewed articles from the programme’s first cycle in 2000 to its most current in 2015. Five literature bases were searched, and results were analysed with SPSS. Results map the frequency of publications according to journal, country, and scientific discipline. They also summarise major themes within three identified categories: secondary analysis, policy impact, and critiques. Findings indicated that studies based on the PISA dataset has led to progress in educational research while simultaneously pointing to the need for caution when using this research to inform educational policy.

Predictability in high-stakes examinations: students' perspectives on a perennial assessment dilemma

Abstract Key debates within educational assessment continuously encourage us to reflect on the design, delivery and implementation of examination systems as well as their relevance to students. In more recent times, such reflections have also required a rethinking of who is authoritative about assessment issues and whose views we seek in order to better understand these perennial assessment dilemmas. This paper considers one such dilemma, predictability in high-stakes assessment, and presents students’ perspectives on this issue. The context is the Irish Leaving Certificate (LC) taken by upper secondary students (aged between 16 and 18) in order (mainly) to enter tertiary-level education. The data come from 13 group interviews with 81 students across a range of schools in Ireland. Listening to students about complex, high-stakes examining problems has a limited history within the educational assessment literature. The findings from the study address this shortcoming and depict how students’ insightful reflections can improve our understanding of these dilemmas. Further, students are more than able to reflect on their own situations with regard to high stakes examining contexts and have important contributions to make to our fuller understanding of those elements that will promote high quality and fair assessment.

Students’ test motivation in PISA: the case of Norway

ABSTRACT Do students make their best effort in large-scale assessment studies such as the Programme for International Student Assessment (PISA)? Despite six cycles of PISA surveys from 2000 to 2015, empirical studies regarding students’ test motivation and experience of the tests are sparse. The present study examines students’ test motivation in PISA, and how closely students’ achievement correlates with students’ reported test motivation. A total of 40 students from eight schools that participated in the PISA study in 2006, 2009 and 2012 were interviewed. In addition, questionnaire data from a total of 9400 students who participated in PISA 2009 and 2012 were collected. The findings of this study indicate that students overall were motivated to do their best in the PISA study, despite the fact that these low-stakes tests have no impact on students’ grades or future school entrance. In contrast to claims in the media, where it has been suggested that Norwegian students are more relaxed towards the PISA test than students in other countries, students report that they are motivated to do their best, but girls report significantly higher test motivation than boys. Despite the policy influence of PISA and the large number of countries participating, very few studies have reported findings from students’ perspectives regarding their test motivation and experience of the test. The present study aims to fill that gap.

The power of PISA – limitations and possibilities for educational research

On 6 December 2016, the Programme for International Student Assessment (PISA) releases its report on the achievements of 15-year-olds from 72 countries and economies around the world. This triennial international survey aims to evaluate education systems across 72 contexts by testing skills in Mathematics, Science and Reading Literacy. This is the sixth cycle of PISA and the OECD suggests countries and economies now have the capability to compare the results over time to ‘assess the impact of education policy decisions’1. Compared to other education studies, the media coverage of PISA must be described as massive (Baird et al., 2016; Meyer & Benavot, 2013) and, as with previous years, it is expected that PISA will attract considerable discussion among policy-makers, educators and researchers (Wiseman, 2014). It is therefore timely to present a thematic issue of Assessment in Education, where we publish four articles that have analysed previous data-sets from the PISA studies each commenting upon the challenges, limitations and potential future assessment research on the PISA data. The articles touch upon issues regarding sampling, language, item difficulty and demands, as well as the secondary analyses of students’ reported experiences of formative assessment in the classroom. One important message from the authors in this thematic Special Issue is the need for a more complex discussion around the use and misuse of PISA data, and the importance of pointing to the limitations of how the results are presented to policy-makers and the public. In an area where the media produces narratives on schools and education systems based upon rankings in PISA, researchers in the field of large-scale assessment studies have a particularly important role in stepping up and advising on how to interpret and understand these studies, while warning against potential misuse. In 2014, Yasmine El Masri gave a keynote at the Association for Educational Assessment-Europe conference in Tallinn, Estonia, following her Kathleen Tattersall New Researcher Award. We are pleased to publish the paper based upon her DPhil research: Language effects in international testing: the case of PISA 2006 science items. Together with Jo-Anne Baird and Art Graesser, El Masri investigates the extent to which language versions of the PISA test in Arabic, English and French are comparable in terms of item difficulty and demand (El Masri et al., 2016). As there is an ongoing discussion on whether it is possible to assess in a fair manner and compare science, mathematics and reading performances across countries and cultures, this present study offers important findings for future research. Using released PISA items, El Masri et al. show how language demands vary when comparing Arabic, English and French versions of the same item, and hence could impose different cognitive demands on the students participating in the PISA test in different countries. With the expansion of PISA to other countries through PISA for Development and the need for fair comparisons across countries, El Masri et al. suggest that subsequent research could explore the possibility of investigating computational linguistics approaches in test transadaptation as an alternative to the use of expert judgement which is the current practice in international test development. The next article in this issue by Freitas, Nunes, Reis, Seabra, and Ferro (2016), Correcting for sample problems in PISA and the improvement in Portugese students’ performance, reports a

PISA, power and policy – the emergence of global educational governance (Oxford Studies in Comparative Education)

Denzin, N. K., and Y. S. Lincoln, eds. 2011. The SAGE Handbook of Qualitative Research. London: Sage. Jurgenson, N. 2011. “Digital Dualism Versus Augmented Reality.” Cybergology: The Society Pages 24. http://thesocietypages.org/cyborgology/?s=Digital+dualism+versus+ augmented+reality. St. Pierre E. A. 1997. “Methodology in the Fold and the Irruption of Transgressive data.” International Journal of Qualitative Studies in Education 10 (2): 175–189.