Kathleen Scalise

Perspectives on Methodological Issues

In this chapter the authors have surveyed the methodological perspectives seen as important for assessing twenty-first century skills. Some of those issues are specific to twenty-first century skills, but the majority would apply more generally to the assessment of other psychological and educational variables. The narrative of the paper initially follows the logic of assessment development, commencing by defining constructs to be assessed, designing tasks that can be used to generate informative student responses, coding/valuing of those responses, delivering the tasks and gathering the responses, and modeling the responses in accordance with the constructs. The paper continues with a survey of the strands of validity evidence that need to be established, and a discussion of specific issues that are prominent in this context, such as the need to resolve issues of generality versus contextual specificity; the relationships of classroom to large-scale assessments; and the possible roles for technological advances in assessing these skills. There is also a brief segment discussing some issues that arise with respect to specific types of variables involved in the assessment of twenty-first century skills. The chapter concludes with a listing of particular challenges that are regarded as being prominent at the time of writing. There is an annexure that describes specific approaches to assessment design that are useful in the development of new assessments.

Differentiated e-learning: five approaches through instructional technology

Differentiated instruction is an approach to teaching that acknowledges people have multiple paths for learning and for making sense of ideas. In e-learning, differentiated instruction has the same meaning as in traditional instruction, but different tools are available to help students learn and to provide information in ways most appropriate to them, including types of new media inclusion, levels of interactivity, response actions, and enhanced ability to collect data on the fly and to deliver custom content. This paper discusses what the tools of e-learning contribute to differentiated instruction and shares a framework for five common approaches to adaptive courseware.

Improving assessment evidence in e-learning products: some solutions for reliability

E-learning products such as cognitive diagnosers interact with learners and collect assessment data to build a picture of some aspect of a learners thinking. One concern for this rapidly emerging area of e-learning is whether the diagnostic conclusions of such products are based on sound evidence, including whether or not the diagnostics are reliable. In online settings, the information may be used for adaptive delivery of content, individualising learning materials, dynamic feedback, teacher feed-forward, cognitive mapping, score reporting and course placement. A reliability index quantifies the impact that measurement error at the individual level may have on the accuracy of the inference. This paper investigates some simple solutions that substantially improve reliability within one e-learning product. These solutions include providing questions of appropriate difficulty that help to maximise item information across the distribution.

Contrasting the expectations for student understanding of chemistry with levels achieved: a brief case-study of student nurses

This case study examines the understanding of a small sample of nursing students in some aspects of general chemistry. In the United States most nursing programs require college- level nursing courses, with expectations that students will master basics of first-year general chemistry. Anxiety to achieve passing grades in such courses is high for nurses, and the courses are sometimes seen as a gatekeeper for who has access to the profession. This study examines understanding achieved for a small sample of nursing students regarding aspects of matter — basic ideas regarding understanding of matter composition, structure, amounts and properties. Our intention is to highlight the contrast between what chemistry knowledge is expected of nurses and what level they actually achieve, and what this may mean for their future professional performance. Findings include that the nursing students in the sample had limited understanding of the university-level chemistry they were being asked to master, and exhibited less comprehension and more pervasive misconceptions than comparison groups, including first term high school students, in our sample. [Chem. Educ. Res. Pract., 2006, 7 (3), 170-184]

Assessment of Learning in Digital Networks

This chapter provides both conceptual and empirical information about the skillset of Learning in Digital Networks – Information Communications Technologies (LDN-ICT). Data are drawn from the pilot phase of the ATC21STM project research and development process, and were collected from August to November 2011 across Australia, Finland, Singapore and the U.S.A. (The acronym ATC21STM has been globally trademarked. For purposes of simplicity the acronym is presented throughout the chapter as ATC21S.) The paper concludes with a discussion of ideas about reporting and use of the consequent development progression which underlies the construct.

The Nature of Assessment Systems to Support Effective Use of Evidence through Technology

The National Educational Technology Plan 2010 (NETP) presents a model of twenty-first-century learning powered by technology, with goals and recommendations in five essential areas: learning, assessment, teaching, infrastructure, and productivity. This article connects NETP ideas in one of these areas – assessment – with those of the Assessment and Teaching of 21st Century Skills (ATC21S) project. Launched by Cisco, Intel and Microsoft, ATC21S released results of a methodological working-group study at the Learning and Technology World Forum 2010 in London. The ATC21S methodological report discusses how the development of a good assessment system is rooted in the inferences the system is intended to support. Assessment is a special kind of evidentiary reasoning with evidence used to support particular kinds of claims. This article will illustrate how key questions and answers for decision-making are influenced by a new era of educational assessment with technology. Areas discussed include the characterization of the constructs to be assessed, the kinds of instruments to be developed, the level of information gathered, and promising avenues for analytic approaches.

Next Wave for Integration of Educational Technology into the Classroom: Collaborative Technology Integration Planning Practices

Technology integration planning (TIP) practices are yielding many emerging examples of effective ICT practice in schools. This chapter will explore and evaluate some best practices on a recently emerging trend in schools: the use of digital collaboration that brings together groups of students into learning networks. This chapter exemplifies the approach with a case study analysis of a sample of collaborative science notebooks from the Assessment and Teaching of Twenty-First Century Skills (ATC21S) project. How, when, and why such technology is being infused into education as digital literacy moves forward will be explored, using the case study. One approach to a systematic process for the effective inclusion of technology is to identify skills that students need to master to support a virtual skill or practice, such as digital collaboration. Here the TIP case study example helps teachers answer key questions about how to assess and evaluate collaborative work online, and how to employ such techniques in the classroom.

Accommodations in Digital Interactive STEM Assessment Tasks: Current Accommodations and Promising Practices for Enhancing Accessibility for Students With Disabilities

In this article, we describe current research findings on assessment accommodations and universal design within the context of emerging interactive digital assessment tasks that employ simulations such as in science, technology, engineering, and mathematics (STEM). STEM education in many classrooms now includes digitally based activities such as science simulations and virtual laboratories that have been shown in some cases to promote learning gains. When such technologies are used in STEM assessments, a major challenge is to ensure assessments are accessible so all students can show what they know and can do. Federal laws and regulations including the Individuals with Disabilities Education Act, Elementary & Secondary Education Act, and Americans with Disabilities Act require that students with disabilities (SWD) be provided an opportunity to participate in educational programing and services available to nondisabled peers. In addition to implementing principles of universal design in assessment contexts, reasonable accommodations must be afforded to ensure accessibility. This article focuses on universal design and accommodations where the STEM construct is not adjusted or modified. Here, we employ synthesis of the research literature to document accessibility recommendations and practices around interactive assessment tasks, especially in STEM. We illustrate with an example and highlight directions that future development might take. The intention is to inform educators, school administrators, state and local policy makers, and assessment developers on the availability and use of accommodations in interactive assessment contexts such as simulation, and what is needed to ensure appropriate accessibility for SWD.

Dispositions for Collaborative Problem Solving

In this chapter we discuss student dispositions toward an emerging domain called “collaborative problem solving” (CPS), recently assessed by the Programme for International Student Assessment (PISA) in the field trial and in the main study. Here, “dispositions” refers to the attitudes to and experiences of collaboration seen emerging in the international PISA survey data in 15-year-olds. For the field trial’s noncognitive measures, nine CPS-related constructs were developed for the student and teacher questionnaires. Information was collected on the types of collaborative activities and the support that was available, in and out of the classroom, as well as on student experiences of and attitudes to collaboration. We provide a description of the constructs and demonstrate how their development was related to present and past PISA cognitive measures in problem solving and collaborative problem solving.

Learning in Digital Networks as a Modern Approach to ICT Literacy

This chapter starts from the perspective that the current conceptualization of educational assessment is out of date, but particularly with regard to conception of information and communication (ICT) literacy. It provides a summary of the conceptual changes in the idea of ICT literacy in four main steps: a concentration of knowledge about computers and their use; a transition to a view of ICT literacy as a broad set of skills that have links to many traditional and non-traditional school subjects, accompanied by the move to technology integration in education; a second transition where ICT Literacy is expressed as progress variables that are essential tools for the design of curriculum and assessments; and finally, the impact of the “social network” perspective on ICT literacy – the critical need for building the power of virtual skills through proficiency with networks of people, information, tools, and resources. The chapter then describes the ACT21S ICT Literacy Framework, which we see as embodying the last of these stages. This includes four “progress variables” that describe student growth in sophistication in ICT Literacy, each involving several increasing levels of development. An online assessment tool to assess the dimensions of the framework is described, and examples of items and responses are given. Results from an international study of students interacting with the assessments are then displayed, including empirical versions of the progress variables, and estimates of their correlation, reliability, and fit are given. The chapter concludes with a brief summary of the findings, and a discussion of its broader implications.