Hanne Albrechtsen

A logging scheme for comparative digital library evaluation

Evaluation of digital libraries assesses their effectiveness, quality and overall impact. To facilitate the comparison of different evaluations and to support the re-use of evaluation data, we are proposing a new logging schema. This schema will allow for logging and sharing of a wide array of data about users, systems and their interactions. We discuss the multi-level logging framework presented in [19] and describe how the community can add to and gain from using the framework. The main focus of this paper is the logging of events within digital libraries on a generalised, conceptual level, as well as the services based on it. These services will allow diverse digital libraries to store their log data in a common repository using a common format. In addition they provide means for analysis and comparison of search history data.

An experimental framework for comparative digital library evaluation: the logging scheme

Evaluation of digital libraries assesses their effectiveness, quality and overall impact. In this paper we present a novel, multi-level logging framework that will provide complete coverage of the different aspects of DL usage for user-system interactions. Based on this framework, we can analyse for various DL stakeholders the logging data according to their specific interests. In addition, analysis tools and a freely accessible log data repository will yield synergies and sustainability in DL evaluation and encourage a community for DL evaluation by providing for discussion on a common ground

A work-centred approach to evaluation of a digital library: annotation system for knowledge sharing in collaborative work

This paper presents an expert evaluation of a collaborative annotation system named DiLAS (Digital Library Annotation Service). It can be globally accessed by individuals as well as different user communities, and knowledge is created and shared within English-speaking communities. It contains a collection of textual documents on information science and software science, and it gives access to all kinds of related material such as authors’ home pages, photos, articles, and so on. An analytical evaluation was conducted as a participatory group evaluation, which involved presentation beyond the objectives and rationale papers and development of the prototype. The empirical evaluation of DiLAS consisted of two experiments. The first evaluation was a bottom up evaluation that began at the elementary level with an evaluation of the usability of the interface. A Cognitive Walkthrough approach was chosen using a qualitative approach. The next evaluation moved towards the broader work context with a user- and work-centred evaluation involving an entire collaborative task situation, which required knowledge sharing on a common real life work task. This evaluation resulted in a large number of user requirements for the next prototype.

Evaluation and requirements elicitation of a DL annotation system for collaborative information sharing

We describe an expert evaluation for user requirement elicitation of an annotation system - The Digital Library Annotation Service, DiLAS, that facilitates collaborative information access and sharing. An analytical evaluation was conducted as a Participatory Group Evaluation, which involved presentation beyond the written papers of the objectives and rationale behind the development of the prototype. The empirical evaluation of DiLAS consisted of two experiments. The first evaluation experiment was a bottom up evaluation of the usability of the interface using a qualitative approach. The second stage of our evaluation moved towards a broader work context with a User and Work Centred Evaluation involving an entire, collaborative task situation, which required knowledge sharing on a common real life work task. This paper describes a first evaluation stage in an iterative evaluation process, and the preliminary result is a set of requirements that will inform the next stage of the DiLAS.