Claire Scoular

Assessment of Collaborative Problem Solving in Education Environments

ABSTRACT The Assessment and Teaching of 21st Century Skills (ATC21STM) project undertook a research and development plan that included conceptualization of 21st century skills and assessment task development. Conceptualization focused on the definition of 21st century skills. This article outlines the particular case of one of these skills, collaborative problem solving, from its definition and identification of subskills, to development of a method for its assessment. The theories contributing to construct definition are described. These have implications for the particular clusters of subskills that are of interest for educational assessment. An approach to assessment task creation is illustrated through the deconstruction of a well-known reasoning task, and its re-development to sample both the cognitive and social aspects of collaborative problem solving. The assessment tasks are designed to generate formative feedback for teachers in order to identify levels of ability within and between their students and support tailoring of instruction differentially for improvement.

Collaborative Problem Solving Tasks

This chapter outlines two distinct types of collaborative problem solving tasks – content-free and content-dependent – each allowing students to apply different strategies to solve problems collaboratively. Content-free tasks were developed to emphasise the enhancement of inductive and deductive thinking skills. Content-dependent tasks allow students to draw on knowledge gained through traditional learning areas or subjects within the curriculum. The collaborative problem solving framework emphasises communication for the purpose of information gathering, identification of available and required information, identification and analysis of patterns in the data, formulation of contingencies or rules, generalisation of rules, and test hypotheses. Characteristics of tasks which were identified as appropriate for eliciting collaborative problem solving processes are reported and illustrated by exemplar items.

Automatic Coding Procedures for Collaborative Problem Solving

This chapter examines the procedure followed in defi ning a scoring process to enable the reporting of individual student results for teachers to use in the classroom. The procedure begins with the identifi cation of task features that match elements of the skills frameworks, and is followed by the generation of simple rules to collect data points to represent these elements. The data points are extracted from log fi les generated by students engaged in the assessment tasks and consist of the documentation of each event, chat and action from each student. The chapter includes examples of the process for defi ning and generating global and local (task specifi c) indicators, and examples of how the indicators are coded, scored and interpreted. The development of coding and scoring of data generated when students engage in collaborative problem solving tasks is described. The data generated are captured in a process stream data fi le. Patterns of these data are coded as indicators of ele- ments defi ned in the conceptual framework outlined in Hesse et al. ( 2015 ; Chap. 2 ) and the relative complexity of indicators is used in a scoring process. The scored data are then used to calibrate the tasks. The calibrations form the basis of interpre- tation and these are used in forming reports for students and teachers. Figure 6.1 summarises the entire process from task development to the reporting of student ability based on a developmental framework.

The Role of Assessment in Improving Learning in a Context of High Accountability

In this chapter, we discuss the issue of accountability and its effects on teaching and learning. We examine different forms of accountability, and take the stance that accountability can improve learning outcomes if accompanied by curriculum support rather than financial penalties. We draw example data from a range of national and systemic high-stakes testing programs and the effects on achievement levels over time. We consider contrasting effects of formative and summative roles of assessment within a context of accountability frameworks. This is not to question the role of accountability, but to assess ways in which the potential damaging effects can be reduced and student performance can be identified, validated and enhanced.

Measuring Collaborative Problem Solving Using Mathematics-Based Tasks

This study describes an online method of measuring individual students’ collaborative problem-solving abilities using four interactive mathematics-based tasks, with students working in pairs. Process stream data were captured from 3,000 students who completed the tasks in the United States, Australia, Canada, Costa Rica, Singapore, and Finland. The data were transformed into indicators of collaborative problem-solving ability and were analyzed using item response modeling. The assessments employed in this study can be used as a teaching tool for introduction to algebraic concepts and as a measurement instrument for collaborative problem-solving ability. The paper describes the construction, calibration, and reliability of the tasks and considers validation issues, such as fairness between assessments for both partners and avoidance of cultural biases. Investigations into the dependencies between student scores provide evidence for convergent and discriminant validity.

Australia in the Context of the ATC21S Project

Schooling in Australia involves 1 year in preparatory school and 12 years of primary and secondary school, with compulsory education to 15 years of age (Australian Government, Department of Education, Employment and Workplace Relations. http://www.deewr.gov.au. Accessed 8 Nov 2012, 2012). Education can be undertaken in government or non-government (independent or Catholic) schools. The Australian Constitution allocates responsibility for the operation of schooling to the state and territory governments (Australian Government, Review of funding for schooling. Emerging issues paper. Commonwealth of Australia, Canberra. http://www.deewr.gov.au, 2010b). The Australian Government (Review of funding for schooling. Emerging issues paper. Commonwealth of Australia, Canberra. http://www.deewr.gov.au, 2010b) is responsible for providing national leadership in educational reforms, and investing substantial funding in their delivery. An example of such leadership was the development of a national curriculum for which the states and territories bear responsibility for implementation. All state and territory Ministers with responsibility for education, and the Federal Minister for Education, are brought together in a Ministerial Council approximately every 10 years to determine strategies for the future of Australia’s education systems. The strategies are published as a ‘declaration’ after each Ministerial Council, and the 2008 Melbourne Declaration on Educational Goals for Young Australians (MCEETYA, Melbourne declaration on educational goals for young Australians. MCEETYA, Melbourne. Retrieved from http://www.mceecdya.edu.au/verve/_resources/National_Declaration_on_the_Educational_Goals_for_Young_Australians.pdf, 2008) initiated the redesign of the curriculum to develop successful learners, who are expected to have the skills necessary to be “creative and productive users of technology, especially ICT, as a foundation for success in all learning areas” (MCEETYA, Melbourne declaration on educational goals for young Australians. MCEETYA, Melbourne. Retrieved from http://www.mceecdya.edu.au/verve/_resources/National_Declaration_on_the_Educational_Goals_for_Young_Australians.pdf, 2008, p. 8). Under the National Curriculum, the general capabilities to be taught across all learning areas include ICT competence and the ability to work collaboratively in teams, across cultures and disciplines.

Teaching Twenty-First Century Skills: Implications at System Levels in Australia

The question surrounding teaching of twenty-first century skills is no longer why, but instead how. In this regard, there are many unknowns including teacher training, resources, and impact. To address these unknowns, research needs to focus on: What is the skill set educators need to teach the skills? How do they teach the skills? And, what are the implications of doing so? A necessary step in this endeavour is to identify clear definitions of the skills, and develop assessment tools to measure them. Training for educators to interpret the demonstration of skills at varying stages of ability needs to occur. Once skills are better understood, then instructional materials and guided examples of successful implementation can be provided. This chapter presents three examples from Australia of implemention in this emerging area. These examples vary in their approach and their perspectives, ranging across three systems; a professional body, an educational institution, and industry. The goal for each is the same – to support educators to find and implement effective methods to teach the skills.

Generic log files and algorithms developed for educational multiplayer games

There is potential for online multiplayer educational games to play a vital role to measure cognitive and social abilities among students. Educational games can be developed to capture student responses or actions, both shared and unshared, within the game environment. This paper presents a data capturing process in online games with a focus on the structural components required to measure sufficiently. The first part of this paper introduces a suggested structure of the log files and the second part describes generic algorithms used to extract information from the log files to indicate any observable cognitive and non-cognitive processes of the students. The term generic algorithm refers the common applicability of the algorithms to log files across games and students to provide researchers with rich and meaningful data about student behaviours. A considerable number of generic algorithms were developed to extract salient student behaviour from student activity log files, including intent to achieve efficiency in both present and future development of such complex educational endeavours.