Florence Le Hebel

Which Effective Competencies Do Students Use in PISA Assessment of Scientific Literacy

In France, PISA Science 2006 indicated overall a score about the average, and a particularly high proportion (compared to OECD average) of students at or below level 1, meaning that students are not able to use scientific knowledge to understand and do PISA’s easiest tasks. The aims of this project are to investigate links between students’ performances in PISA Science 2006 (i.e. their scores), students’ understanding of the context of the unit, proposed in the unit stimulus or items, and competencies involved when the students effectively construct the answer to PISA items. For this reason, we focus our research on the representations and strategies that students develop while solving PISA items. We collected audiotaped and/or videotaped data from 15-year-old students from grade 9 in middle secondary school and grade 10 in upper secondary school in order to analyze students’ oral and written productions when they answer PISA questions which we had previously selected. After a review of the different research studies developed about PISA secondary analysis, we present our results showing that the representations and strategies which students construct when answering questions differ between high and low achievers. For the same PISA question, our data also indicate that the competencies which students use when they construct their answer may differ from the competencies that PISA claims to evaluate. Our results show that different interpretations can be made about the actual meaning of the PISA measurement of scientific literacy. In particular, for low achievers, competencies should be refined.

Sources of difficulty in assessment: example of PISA science items

ABSTRACT The understanding of what makes a question difficult is a crucial concern in assessment. To study the difficulty of test questions, we focus on the case of PISA, which assesses to what degree 15-year-old students have acquired knowledge and skills essential for full participation in society. Our research question is to identify PISA science item characteristics that could influence the item’s proficiency level. It is based on an a-priori item analysis and a statistical analysis. Results show that only the cognitive complexity and the format out of the different characteristics of PISA science items determined in our a-priori analysis have an explanatory power on an item’s proficiency levels. The proficiency level cannot be explained by the dependence/independence of the information provided in the unit and/or item introduction and the competence. We conclude that in PISA, it appears possible to anticipate a high proficiency level, that is, students’ low scores for items displaying a high cognitive complexity. In the case of a middle or low cognitive complexity level item, the cognitive complexity level is not sufficient to predict item difficulty. Other characteristics play a crucial role in item difficulty. We discuss anticipating the difficulties in assessment in a broader perspective.

Which Answering Strategies Do Low Achievers Use to Solve PISA Science Items

In France, PISA science 2006 shows a high proportion of students who are not able to understand and solve PISA’s easiest tasks. In this context, the aim of our study is to investigate answering strategies that low achievers develop while solving PISA items. We chose a case study methodology, collecting videotaped data from 15-year-old students from grade 9 in middle secondary school and grade 10 in upper secondary school in order to analyze their mental and behavioral processes. Our results show that low achievers often develop few specific answering strategies, and this mostly without constructing relevant and stable representations of PISA unit and items goals when they build their answer. They often transform the question in order to be able to answer it. In terms of the didactic contract, low achievers do not expect to solve some kinds of PISA tasks requiring creative reasoning. Contrary to high achievers, they are not able to identify the gap between the initial state and the expected final aim of the task. They are not aware of what they are expected to supply – new knowledge, for instance – to solve the task. Nevertheless, they can employ the beginning of relevant strategies, but currently PISA assessment is not able to identify them and does not allow the evaluation of competencies employed by low achievers.

Students’ Perspectives on Peer Assessment

This chapter reports the results of three research studies on peer assessment carried out in different countries where it is an unusual practice (France, Switzerland, and the Czech Republic). The three research studies focus on different competences and different disciplines (sciences and mathematics), but they all involve inquiry-based approaches at primary and secondary school level. In the French study, the data reported in this chapter explore relationships between success in task processing and the ability to mark a peer’s written artefact on the same task. The Swiss research study examines the type of peer feedback students offer their peers while assessing their models, based on a fine-grained analysis of peer feedback comments. In the Czech Republic, the study focuses on students’ reflection on peer assessment in inquiry lessons. The three studies conclude the necessity of allowing the sharing of “knowledge authority” in the classroom to evolve and to be integrated into usual classroom practice. However, researchers have a divergent view on the sharing of responsibility for validation of knowledge between the student and the teacher. Moreover, we can conclude that peer assessment can be a way to trigger metacognitive work on knowledge and competences in science and on assessment criteria and teacher expectations, at a class level and individual student level.