Rosa Lanzilotti

An approach to usability evaluation of e-learning applications

Despite recent advances of electronic technologies in e-learning, a consolidated evaluation methodology for e-learning applications is not available. The evaluation of educational software must consider its usability and more in general its accessibility, as well as its didactic effectiveness. This work is a first step towards the definition of a methodology for evaluating e-learning applications. Specific usability attributes capturing the peculiar features of these applications are identified. A preliminary user study involving a group of e-students, observed during their interaction with an e-learning application in a real situation, is reported. Then, the proposal is put forward to adapt to the e-learning domain a methodology for systematic usability evaluation, called SUE. Specifically, evaluation patterns are proposed that are able to drive the evaluators in the analysis of an e-learning application.

Explore! possibilities and challenges of mobile learning

This paper reports the experimental studies we have performed to evaluate Explore!, an m-learning system that supports middle school students during a visit to an archaeological park. It exploits a learning technique called excursion-game, whose aim is to help students to acquire historical notions while playing and to make archaeological visits more effective and exciting. In order to understand the potentials and limitations of Explore!, our studies compare the experience of playing the excursion-game with and without technological support. The design and evaluation of Explore! have provided knowledge on the advantages and pitfalls of m-learning that may be instrumental in informing the current debate on e-learning.

On the Usability Evaluation of E-Learning Applications

Despite the advances of the electronic technologies in e-learning, a consolidated evaluation methodology for e-learning applications does not yet exist. The goal of e-learning is to offer the users the possibility to become skillful and acquire knowledge on a new domain. The evaluation of educational software must consider its pedagogic effectiveness as well as its usability. The design of its interface should take into account the way students learn and also provide good usability so that students interactions with the software are as natural and intuitive as possible. In this paper, we present the results obtained from a first phase of observation and analysis of the interactions of people with e-learning applications. The aim is to provide a methodology for evaluating such applications.

Investigating and promoting UX practice in industry: an experimental study

Abstract The efforts of addressing user experience (UX) in product development keep growing, as demonstrated by the proliferation of workshops and conferences bringing together academics and practitioners, who aim at creating interactive software able to satisfy their users. This special issue focuses on “Interplay between User Experience Evaluation and Software Development”, stating that the gap between human-computer interaction and software engineering with regard to usability has somewhat been narrowed. Unfortunately, our experience shows that software development organizations perform few usability engineering activities or none at all. Several authors acknowledge that, in order to understand the reasons of the limited impact of usability engineering and UX methods, and to try to modify this situation, it is fundamental to thoroughly analyze current software development practices, involving practitioners and possibly working from inside the companies. This article contributes to this research line by reporting an experimental study conducted with software companies. The study has confirmed that still too many companies either neglect usability and UX, or do not properly consider them. Interesting problems emerged. This article gives suggestions on how they may be properly addressed, since their solution is the starting point for reducing the gap between research and practice of usability and UX. It also provides further evidence on the value of the research method, called Cooperative Method Development, based on the collaboration of researchers and practitioners in carrying out empirical research; it has been used in a step of the performed study and has revealed to be instrumental for showing practitioners why to improve their development processes and how to do so.

Experiencing the Past through the Senses: An M-Learning Game at Archaeological Parks

M-learning the combination of e-learn- ling with mobile technologies captures the very nature of e-learning by providing users with independence from the constraints of time and location.1 To exploit the potential of mobile technologies for learning, researchers must define new teaching and learning techniques.2 The Explore! m-learning system implements an excursion-game technique to help middle school students (ages 11 through 13) acquire historic knowledge while playing in an archaeological park.

End users as co-designers of their own tools and products

In our Age of exponential technological advance, recent developments are determining an evolution of end users from passive information consumers into information producers. Users are increasingly willing and, indeed, determined to shape the software they use to tailor it to their own needs. Based on a brief review of research activities we performed in the last decade, this paper analyzes some challenges that software designers face to comply with the new roles of end users in the software life cycle, and discusses how to provide end users with software environments that empower them to become co-designers of their own tools and products. The examples reported in the paper show why and how end users are involved in design activities in various application domains.

Mobile games to foster the learning of history at archaeological sites

This paper presents a system designed to support young students learning history at an archaeological site, by exploiting mobile technology. The approach uses game-play, since it stimulates in young students an understanding of history that would otherwise be difficult to engender, helping players to acquire historical notions and making archaeological visits more effective and exciting. A strength of the system is that, by running on the visitors own cellular phones, it requires minimal investments and small changes to the existing site exhibition.

Systematic evaluation of e-learning systems: an experimental validation

The evaluation of e-learning applications deserves special attention and evaluators need effective methodologies and appropriate guidelines to perform their task. We have proposed a methodology, called eLSE (e-Learning Systematic Evaluation), which combines a specific inspection technique with user-testing. This inspection aims at allowing inspectors that may not have a wide experience in evaluating e-learning systems to perform accurate evaluations. It is based on the use of evaluation patterns, called Abstract Tasks (ATs), which precisely describe the activities to be performed during inspection. For this reason, it is called AT inspection. In this paper, we present an empirical validation of the AT inspection technique: three groups of novice inspectors evaluated a commercial e-learning system applying the AT inspection, the heuristic inspection, or user-testing. Results have shown an advantage of the AT inspection over the other two usability evaluation methods, demonstrating that Abstract Tasks are effective and efficient tools to drive evaluators and improve their performance. Important methodological considerations on the reliability of usability evaluation techniques are discussed.

Enriching Archaeological Parks with Contextual Sounds and Mobile Technology

The importance of cultural heritage in forging a sense of identity is becoming increasingly evident. Information and communication technologies have a great potential to promote a greater awareness and appreciation of cultural heritage. This article presents some findings on how mobile technology can be used to foster a better understanding of an archaeological site by reconstructing the ancient environment and life. Children aged 11--13 years old are the target of our research. To motivate and engage them, a pervasive educational game has been developed and implemented in Explore!, a system aimed at supporting children exploring sites of cultural interest. Special attention has been devoted to the design of a soundscape that may improve players’ navigation in degraded physical environments and enrich their overall experience. A field study indicated that children judged their experience both useful and entertaining: not only did they enjoy playing the game, but they also learned historical notions and facts related to ancient Roman life. Contextual sounds were found to have a facilitating effect on space navigation, reducing the need for map reading and improving spatial orientation. This work provides insights into the design of educational games for use with cultural heritage and a model to enrich historical sites through the creation of soundscapes which can help visitors to navigate a site and feel its historical atmosphere.

Do patterns help novice evaluators? A comparative study

Evaluating e-learning systems is a complex activity which requires considerations of several criteria addressing quality in use as well as educational quality. Heuristic evaluation is a widespread method for usability evaluation, yet its output is often prone to subjective variability, primarily due to the generality of many heuristics. This paper presents the pattern-based (PB) inspection, which aims at reducing this drawback by exploiting a set of evaluation patterns to systematically drive inspectors in their evaluation activities. The application of PB inspection to the evaluation of e-learning systems is reported in this paper together with a study that compares this method to heuristic evaluation and user testing. The study involved 73 novice evaluators and 25 end users, who evaluated an e-learning application using one of the three techniques. The comparison metric was defined along six major dimensions, covering concepts of classical test theory and pragmatic aspects of usability evaluation. The study showed that evaluation patterns, capitalizing on the reuse of expert evaluators know-how, provide a systematic framework which reduces reliance on individual skills, increases inter-rater reliability and output standardization, permits the discovery of a larger set of different problems and decreases evaluation cost. Results also indicated that evaluation in general is strongly dependent on the methodological apparatus as well as on judgement bias and individual preferences of evaluators, providing support to the conceptualisation of interactive quality as a subjective judgement, recently brought forward by the UX research agenda.