Carmel McNaught

Evaluating e-learning : guiding research and practice

How can novice e-learning researchers and postgraduate learners develop rigorous plans to study the effectiveness of technology-enhanced learning environments? How can practitioners gather and portray evidence of the impact of e-learning? How can the average educator who teaches online, without experience in evaluating emerging technologies, build on what is successful and modify what is not? By unpacking the e-learning lifecycle and focusing on learning, not technology, Evaluating e-Learning attempts to resolve some of the complexity inherent in evaluating the effectiveness of e-learning. The book presents practical advice in the form of an evaluation framework and a scaffolded approach to an e-learning research study, using divide-and-conquer techniques to reduce complexity in both design and delivery. It adapts and builds on familiar research methodology to offer a robust and accessible approach that can ensure effective evaluation of a wide range of innovative initiatives, including those covered in other books in the Connecting with e-Learning series. Readers will find this jargon-free guide is a must-have resource that provides the proper tools for evaluating e-learning practices with ease.

How academics use technology in teaching and learning: understanding the relationship between beliefs and practice

This paper reports on a detailed investigation into the beliefs and practices of teachers in 22 computer-assisted learning projects in Australia in the mid-1990s. Detailed interview data were obtained, supported by the project software and other curriculum materials. The interview transcripts and documentary material were collated and condensed into rich descriptions; these were then coded on a number of belief and practice dimensions. The resulting profiles were clustered into five belief‐practice categories: thoughtful instructors, pre-emptive professionals, conversational constructivists, learning facilitators and situated knowledge negotiators. These complex, yet interpretable, patterns of relationships between beliefs and practices are useful in understanding teachers’ reluctance to change their teaching, one instance of which is the relatively limited uptake of technology in higher education.

A workshop activity to demonstrate that approaches to learning are influenced by the teaching and learning environment

It is important to demonstrate to those taking courses for new teachers that approaches to learning have a relational nature — that they are influenced by the teaching and learning context. This article describes a workshop activity, based on the Revised Study Process Questionnaire. Workshop participants recorded their approaches to learning in two contexts: how they currently studied as postgraduate students, and how they studied in their most disliked undergraduate course. Analysis of the results from this activity indicates that approaches to learning are markedly influenced by the teaching and learning environment. This provides a graphic demonstration to workshop participants of the importance of their teaching, as it will have a strong influence on the quality of learning of their students. The data from the activity give quantitative evidence of the relational nature of approaches to learning. Further, there appears to be a discipline effect operating with the nature of the typical teaching and learning environment in the arts, humanities and social sciences being more conducive to students cultivating a deep approach to learning.

Describing Computer-Facilitated Learning Environments in Higher Education.

The research reported here is part of a study of the ways in which university academics design and incorporate computer-facilitated learning (CFL) environments into their courses. This study was based on archive material only (the initial application and final report) for 36 externally-funded technology-based projects from a number of disciplines. Projects were sorted into categories based on educational assumptions and practices. Categories were then compared and refined so as to reveal their major sources of similarity and difference. The resulting framework is one in which the use of educational technology in higher education can be interpreted in terms of several key qualitative dimensions which reflect academics’ beliefs about the origin of knowledge, the learning framework, control of the direction of learning, and the nature of the knowledge and of the learning process.

Design and evaluation of online courses containing media‐enhanced learning materials

With the current state of web technology, multimedia materials are readily accessible by students. This paper reports on the design and evaluation of three online courses from a university in Hong Kong which incorporate media‐enhanced learning materials. These cases are at different positions with respect to the types of knowledge and levels of cognitive reasoning outlined in the revised Bloom’s taxonomy. Evaluation data give qualified support for media‐enhanced aspects of the courses being beneficial to student learning. The study has also highlighted factors that influence the success of the learning experience: attention to the quality and design of the media, considering student motivation and focusing on feedback on learning during the course. Media and learning design, thus, are inextricably intertwined in a complex relationship. La conception et l’évaluation des cours en ligne contenant du matériel d’apprentissage enrichi par les medias L’état actuel de la technologie en ligne offre aux étudiants un accès facile aux documents multimedia. L’article ci‐joint étudie dans une université de Hong Kong. La conception et l’évaluation de trois cours en ligne dans lesquels on a intégré des matériaux d’apprentissage enrichis par les medias. Ces trois cas se situent à des positions différentes par rapport aux types de connaissance et aux niveaux de raisonnement cognitif décrits dans les versions révisées de la taxonomie de Bloom. Les données de l’évaluation font apparaître de façon nette que les aspects du cours qui sont enrichis par les medias ont une influence bénéfique sur l’apprentissage des étudiants. L’étude a aussi mis en relief certains facteurs qui influent sur le succès de l’expérience d’apprentissage : l’attention portée à la qualité et à la conception des medias, la prise en considération de la motivation des étudiants et l’accent sur la rétroalimentation et l’apprentissage pendant le cours. Les medias et la conception des matériaux d’apprentissage sont donc inextricablement liés dans une relation complexe. Entwurf und Auswertung von Online‐Kursen, die mediengestützte Lernmaterialien enthalten Beim gegenwärtigen Stand der Webtechnologie ist Studenten der Zugriff auf Multimedia‐Materialien leicht möglich. Dieser Beitrag berichtet über den Entwurf und die Auswertung von drei Online‐Kursen einer Universität in Hongkong, in die medienverwendende Lernmaterialien integriert wurden. Diese sind an verschiedenen Stellen, abhängig von den Typen des Wissens und dem Niveau der kognitiver Überlegungen entsprechend der überarbeiteten Bloomschen Taxonomie, eingebaut. Auswertungsdaten unterstützen die medienverbesserten Aspekte der Kurse qualitativ in Bezug auf die Lernwirksamkeit für die Studenten. Die Studie nutzt auch besondere Faktoren, die den Lernerfolg beeinflussen: Berücksichtigung der Qualität und der Gestaltung der Medien, damit sie die Studenten als Motivation empfinden und sie auch Rückmeldung über ihr Lernen während des Kurses geben. Dadurch sind die Medien und die Gestaltung des Lehrgangs untrennbar in einer komplexen Beziehung miteinander verflochten. Sobre el diseño y evaluación de los cursos en línea incorporando materiales de aprendizaje enriquecidos por elementos multimedia El estado presente de la tecnología en línea ofrece a los alumnos un acceso fácil a materiales en formato multimedia. El presente artículo estudia en una universidad de Hong Kong el diseño y evaluación de tres cursos en linea que incorporan materiales de aprendizaje enriquecidos por elementos multimedia. Esos casos se encuentran en posiciones diferentes en relación con los tipos de conocimiento y niveles de razonamiento cognitivo descritos en la versión actualizada de la taxonomía de Bloom. Los datos de las evaluaciones confirman netamente el impacto positivo de los aspectos de los cursos enrequecidos por los multimedia sobre el aprendizaje de los estudiantes. El estudio destacó tambien algunos factores que nfluyen sobre el éxito de la experiencia de aprendizaje como el enfoque sobre la calidad y el diseño de los medios, la toma en consideración de de la motivación estudiantil y la insistencia sobre la retroalimentación del aprendizajeza lo largo del curso. Hay entonces una vinculación muy estrecha y compleja entre los medios y el diseño para el aprendizaje.

Building an evaluation culture and evidence base for e-learning in three hong kong universities

The paper presents a strategy for the systematic evaluation of course websites built under the project e3Learning that operates across three universities in Hong Kong. Tailor-made evaluation plans were produced for each website; the evaluation was conducted by an evaluation officer, and a report was provided to each teacher, thus supporting continuous development. To date, 58 cases of evaluation data have been collected. In this paper, a meta-analysis across these cases is presented in order to see which are the most useful web-assisted functions currently in use in Hong Kong universities. This study indicates the following four functions as being most useful: learning tools such as glossaries, notes and PowerPoints, assessment tasks associated with grades, and creation and exhibition of multimedia projects. The reasons for these rankings are postulated.

Adopting technology should mean adapting it to meet learning needs

The thrust of this essay is that there is an increasing emphasis on the capabilities graduates need to have. We live in a changing, complex world and so post‐secondary programs need to equip students with the capacity to adapt their learning into working in a variety of new situations. Planning for the unknown is always tricky and needs to be done with care. Using technology in designing post‐secondary programs provides new opportunities but also adds another layer of complexity. Four aspects considered essential are: defining what we are asking students to ultimately achieve (graduate capabilities); designing at both program and course levels; working together in teams (the complexity demands that); and obtaining evidence that we are on the right track.

Improving assessment methods in university science education with negotiated self‐ and peer‐assessment

The aim of this study was to investigate whether, in the Hong Kong context, self‐ and peer‐assessment promote students’ self‐reflection and enable students to understand their own strengths and weaknesses better. A three‐stage assessment strategy was employed in three Science courses at The Chinese University of Hong Kong: (1) students developing assessment criteria, (2) self‐assessment, and (3) peer‐assessment. These assessment strategies are not common at the University. Education in Hong Kong is often portrayed as having less experience of, and being somewhat resistant to, educational innovation. So, in addition to studying the learning potential of these strategies, the study also focused on the practicality of implementing the new strategies. The results confirm that there are challenges in the present context but also indicate that these innovations to assessment are well perceived by students if they are appropriately framed and implemented.

The dilemma of case‐based teaching and learning in science in Hong Kong: Students need it, want it, but may not value it

The paper reports a study for determining a suitable process for converting traditional surface science courses into case‐based learning ones in two universities in Hong Kong. In this preparative study, a set of baseline data was collected on the current level of students’ conceptual understanding and also students’ perceptions about the traditional courses. The student data (from 38 students) came from examination results, the Study Process Questionnaire, a course‐end survey and a focus group meeting; teachers also kept reflective journals. This set of baseline data revealed factors that both support and inhibit case‐based learning. On the one hand, students demonstrate a motivation to be able to solve practical problems in this field; on the other hand, they show an unwillingness to take up a personal responsibility for learning. This finding strengthens our resolve to take into account students’ expectations and beliefs in the implementation of case‐based teaching and learning.

Innovation and change in higher education: Managing multiple polarities

Professor of Learning Enhancement at The Chinese University of Hong Kong. Her previous appointment was as Head of Professional Development in Learning Technology Services at RMIT University, Melbourne, Australia. Carmel has worked in eight universities in Australasia and southern Africa in chemistry, science education, second language learning and higher education. Address for correspondence: Centre for Learning Enhancement and Research, The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong. Tel: +852 2609 6028; Fax: +852 2603 6804; email: carmel.mcnaught@cuhk.edu.hk perspective