C. Paul Newhouse

A follow‐up study of students using portable computers at a secondary school

This paper reports on the findings of a 1999 study that set out to investigate the current perceptions of students and teachers towards the use of portable computers at a secondary school. The aim was to compare these with the findings of a 1995 study carried out by the researcher at the same school. Data were collected from 102 Year Twelve students (17 year old), 104 Year Eight students (13 year old) and 4() teachers. The results indicated that for the Year Twelve students the computers had been of limited value while the Year Eight students appeared to be divided with about a quarter indicating negative attitudes. For the younger students the computers appeared to be used more often and for a greater range of tasks. Many teachers indicated concerns about the management of computers in the classroom and linking computer use to learning outcomes. These perceptions underline the need for targeted professional development, systematic support for the development of student computer-related skills, and changes in the curriculum towards more learner-centred approaches.

Development and use of an Instrument for Computer-Supported Learning Environments

This article begins by constructing an argument from the literature to support the view that research into the use of computers in classrooms must consider the overall learning environment. This provides the rationale for the development and use of the New Classroom Environment Instrument (NCEI) in research conducted into the use of portable computers in a school. This study particularly focused on the impact of the computers on classroom learning environments. An example is given of the use of the instrument to investigate the use of the portable computers with classes of 13-year-old students. The intention of the article is not to present the results of the study but to demonstrate the use of the instrument and encourage others to use such instruments in educational computing research.

Using IT to assess IT: Towards greater authenticity in summative performance assessment

An applied Information Technology (IT) course that is assessed using pen and paper may sound incongruous but it is symptomatic of the state of high-stakes assessment in jurisdictions such as Western Australia. Whereas technology has permeated most aspects of modern life, including schooling, and more has been demanded of education systems in terms of outcomes and participation, methods of summative assessment have changed little and are seriously out of alignment with curriculum, pedagogy and the needs of individuals and society. This paper reports on an analysis of some of the data from a component of a study into the feasibility of using digital technologies to achieve greater authenticity in summative performance assessment in the Applied Information Technology (AIT) course in Western Australian secondary schools. In the first phase of the study a sample of 115 students completed a digital portfolio and a computer-based exam that were both externally assessed using online tools and by two methods of marking, with the results analysed using Rasch modelling software. A traditional analytical method and a comparative pairs method of marking were investigated. The study found that both the digital portfolio and computer-based exam were implemented without significant technical difficulty and were well accepted by the students and teachers. The work output in digital form was readily accessed from an online repository by external markers using a standard web browser. The two methods of marking provided highly reliable scores, with those from the comparative pairs method being the more reliable. A number of questions of validity and manageability were raised and the strengths and weaknesses of the two forms of assessment revealed. It was concluded that it was feasible to implement either form of assessment for high-stakes purposes, with a resulting improvement in alignment and authenticity.

Using digital representations of practical production work for summative assessment

This paper presents the findings of the first phase of a three-year study investigating the efficacy of the digitisation of creative practical work as digital portfolios for the purposes of high-stakes summative assessment. At the same time the paired comparisons method of scoring was tried as an alternative to analytical rubric-based marking because we believed that it was likely that a more holistic approach to scoring would be more appropriate. Researchers created digital representations of the practical submissions of 75 Visual Arts and 82 Design students graduating from secondary school in Western Australia. These digital portfolios were scored using the two methods, with the scores compared to those officially awarded to the physical forms. It was concluded that the digital representations of the Visual Arts submissions had adequate fidelity for the purpose of awarding high-stakes scores particularly using the paired comparisons method. However, the Visual Arts teachers and students were opposed to digitisation. For the Design portfolios teachers and students were supportive of digital submission, but the structure of the portfolios reduced the reliability and validity of scores, particularly from analytical marking.

Using digital technologies and contemporary psychometrics in the assessment of performance on complex practical tasks

There is a critical need for research into the use of digital technologies to support the assessment of performance on complex tasks in schools. This paper reports on a component of a pilot study aimed at investigating the use of digital forms of performance assessment, manageable within schools, with high levels of reliability and capable of being scaled up for statewide implementation. Student practical performances were represented in digital files, placed in an online repository and marked using both a standards‐reference and comparative pairs method. It was found that the short performance tasks exam was readily implemented and the work easily marked. The resulting scores were shown to be highly reliable and well correlated when compared with the other case studies and suitable for ranking students. However, the limitations of the tasks, time and scaffolding of the tasks reduced the study’s validity in assessing student capability in the course. Overall the results were encouraging and provide a basis for a larger and longer study that has commenced.

Using comparative judgement and online technologies in the assessment and measurement of creative performance and capability

In this paper we argue that comparative judgement delivered by online technologies is a viable, valid and highly reliable alternative to traditional analytical marking. In the past, comparative judgement has been underused in educational assessment and measurement, particularly in large-scale testing, mainly due to the lack of supporting technologies to facilitate the large number of judgements and judges. We describe the foundations of comparative judgement and dispel many of the old issues regarding its use in regards to time, cost and training for large-scale assessment. Studies in the use of comparative judgement and online technologies for the assessment and measurement of practical performance conducted by Edith Cowan University provide a context for further promoting its use in educational testing.

Computer-Based Practical Exams in an Applied Information Technology Course

AbstractWorldwide, fewer and fewer work tasks are done using paper and pen, yet most high-stakes assessment in schools continues to use this primitive technology. This paper reports on one component of a project investigating the use of digital technologies to facilitate assessment tasks for high-stakes summative purposes in senior secondary courses. It reports on how a computer-based production exam was implemented in 17 final-year secondary school classes for the Applied Information Technology course in Western Australia. Further, it compares the comparative pairs method of marking with a more traditional analytical method. The conclusion was that this digital form of assessment should replace the existing paper-based exam; however, it was less certain which method of marking should be applied.Abstract Worldwide, fewer and fewer work tasks are done using paper and pen, yet most high-stakes assessment in schools continues to use this primitive technology. This paper reports on one component of a project investigating the use of digital technologies to facilitate assessment tasks for high-stakes summative purposes in senior secondary courses. It reports on how a computer-based production exam was implemented in 17 final-year secondary school classes for the Applied Information Technology course in Western Australia. Further, it compares the comparative pairs method of marking with a more traditional analytical method. The conclusion was that this digital form of assessment should replace the existing paper-based exam; however, it was less certain which method of marking should be applied.

Using digital technologies to improve the authenticity of performance assessment for high-stakes purposes.

This paper reports on the outcomes of a three-year study investigating the use of digital technologies to increase the authenticity of high-stakes summative assessment in four Western Australian senior secondary courses. The study involved 82 teachers and 1015 students and a range of digital forms of assessment using computer-based exams, digital portfolios and audiovisual recordings. The results were analysed using a feasibility framework concerning manageability, technical facility, functional operation and pedagogic alignment. By the end of the study, each form of assessment that was implemented was found to be feasible once some obstacles were overcome. Two methods of marking were tried, analytical rubric-based marking and holistic comparative pairs marking, with the latter found to generate more reliable scores. With the increased use of digital technologies in schools and the expectation that children will achieve more complex performances, more use of digital forms of assessment will be required.

Bring Your Own Digital Device in Teacher Education

Abstract This article reports on an investigation to advise a teacher education institution on the feasibility of having a “Bring Your Own Digital Device” policy for students. The investigation built on components of two research projects while adding the comprehensive testing of representative potential hardware and software platforms. The initial emphasis was on identifying the digital technology demands of our teacher education courses. A representative range of potential devices was tested against these demands. At the time of testing, all laptop devices were found to be adequate but only the iPad was adequate in the mobile range. In a survey of a sample of students, nearly all owned an appropriate device but most made limited, if any, use of it at the university.

Measuring the meaningful use of ICT in schools: a learning environments attributes approach

Increasingly teachers and school leaders need to be sure that the investment in information and communications technologies ICT maximises the positive impact on learning. There has been much debate on how this impact on learning should be measured. As a result, a group of researchers developed a framework that can be used to describe and evaluate this impact in schools. This paper gives the rationale behind the key learning environment attributes dimension of this framework, and explains the rubric-based tool and statistical scale that were developed to facilitate the measurement of this dimension. Application of this approach, to measure the impact of ICT use on learning in a variety of studies has proved to be useful in providing data to guide the decisions of teachers and school leaders about the meaningful integration of ICT to support the learning of students.