Marianne Lykke

Developing a test collection for the evaluation of integrated search

The poster discusses the characteristics needed in an information retrieval (IR) test collection to facilitate the evaluation of integrated search, i.e. search across a range of different sources but with one search box and one ranked result list, and describes and analyses a new test collection constructed for this purpose. The test collection consists of approx. 18,000 monographic records, 160,000 papers and journal articles in PDF and 275,000 abstracts with a varied set of metadata and vocabularies from the physics domain, 65 topics based on real work tasks and corresponding graded relevance assessments. The test collection may be used for systems- as well as user-oriented evaluation.

Enhancing social tagging with automated keywords from the Dewey Decimal Classification

Purpose – The purpose of this paper is to explore the potential of applying the Dewey Decimal Classification (DDC) as an established knowledge organization system (KOS) for enhancing social tagging, with the ultimate purpose of improving subject indexing and information retrieval. Design/methodology/approach – Over 11,000 Intute metadata records in politics were used. Totally, 28 politics students were each given four tasks, in which a total of 60 resources were tagged in two different configurations, one with uncontrolled social tags only and another with uncontrolled social tags as well as suggestions from a controlled vocabulary. The controlled vocabulary was DDC comprising also mappings from the Library of Congress Subject Headings. Findings – The results demonstrate the importance of controlled vocabulary suggestions for indexing and retrieval: to help produce ideas of which tags to use, to make it easier to find focus for the tagging, to ensure consistency and to increase the number of access points...

Motivating programming students by problem based learning and LEGO robots

Retention of first year students in Computer Science is a concern for universities internationally. Especially programming courses are regarded as difficult, and often have the highest failure and dropout rates. The Informatics School at Universidad Nacional in Costa Rica is not an exception. For this reason the school is focusing on different teaching methods to help their students master these skills. This paper introduces an experimental, controlled comparison study of three learning designs, involving a problem based learning (PBL) approach in connection with the use of LEGO Mindstorms to improve students programming skills and motivation for learning in an introductory programming course. The paper reports the results related with one of the components of the study - the experiential qualities of the three learning designs. The data were collected through a questionnaire survey with 229 students from three groups exposed to different learning designs and through six qualitative walk-alongs collecting data from these groups by informal interviews and observations. Findings from the three studies were discussed in three focus group interviews with 10 students from the three experimental groups.

A framework for evaluating automatic indexing or classification in the context of retrieval

Tools for automatic subject assignment help deal with scale and sustainability in creating and enriching metadata, establishing more connections across and between resources and enhancing consistency. Although some software vendors and experimental researchers claim the tools can replace manual subject indexing, hard scientific evidence of their performance in operating information environments is scarce. A major reason for this is that research is usually conducted in laboratory conditions, excluding the complexities of real‐life systems and situations. The article reviews and discusses issues with existing evaluation approaches such as problems of aboutness and relevance assessments, implying the need to use more than a single “gold standard” method when evaluating indexing and retrieval, and proposes a comprehensive evaluation framework. The framework is informed by a systematic review of the literature on evaluation approaches: evaluating indexing quality directly through assessment by an evaluator or through comparison with a gold standard, evaluating the quality of computer‐assisted indexing directly in the context of an indexing workflow, and evaluating indexing quality indirectly through analyzing retrieval performance.

Automated classification of Web pages in hierarchical browsing

Purpose – The purpose of this study is twofold: to investigate whether it is meaningful to use the Engineering Index (Ei) classification scheme for browsing, and then, if proven useful, to investigate the performance of an automated classification algorithm based on the Ei classification scheme.Design/methodology/approach – A user study was conducted in which users solved four controlled searching tasks. The users browsed the Ei classification scheme in order to examine the suitability of the classification systems for browsing. The classification algorithm was evaluated by the users who judged the correctness of the automatically assigned classes.Findings – The study showed that the Ei classification scheme is suited for browsing. Automatically assigned classes were on average partly correct, with some classes working better than others. Success of browsing showed to be correlated and dependent on classification correctness.Research limitations/implications – Further research should address problems of d...

Design considerations for introducing PBL in computer engineering

Nowadays, universities are driven to adopt pedagogical approaches that enable them to better meet the needs of the new generation of students and industry. This article presents an ongoing experience in innovating the computer engineering curriculum through a problem-based and project-oriented learning approach to develop skills such as problem solving, effective communication, teamwork and lifelong learning, which are increasingly demanded by employers and society. The investigation determined an initial set of design principles that can be used to guide an implementation strategy of the approach. The methodology allows continuous evaluation and refinement of the design. The findings so far point to a number of obstacles, from student, teachers and management, which must be overcome to achieve an effective result.

CHAOS: User-driven Development of a Metadata Scheme for Radio Broadcast Archives

CHAOS 1 (Cultural Heritage Archive Open System) is a digital platform for Danish radio broadcasts. Radio broadcasts are an important and vibrant part of our cultural heritage, but providing efficient and effective access to such archives is challenging for lack of a solid digital infrastructure. The Danish LARM project aims to meet this challenge by making one million hours of radio programs available to humanities researchers through the digital platform CHAOS. CHAOS is being built in close cooperation with the researchers involved in LARM. In this paper, we present the user-driven development of the multi-tiered metadata scheme used in CHAOS.

Social tagging in support of cancer patients’ information interaction

Purpose – The purpose of this chapter is to explore whether and how social tagging can be useful in an information web site for cancer patients and their relatives. Methodology/approach – Three studies have been carried out in order to investigate the research questions. First, we reviewed and analyzed literature about cancer patients’ information needs and seeking behavior, and about social tagging and patient terminology. Second, we analyzed tags applied to blog postings at Blogomkraeft.dk, a blog site at the Danish information web site Cancer.dk. The tags were compared with the formal browsing structure of Cancer.dk. Results from the two studies were used to develop a prototype for social tagging at Cancer.dk. Thus third, we evaluated the prototype in a usability study. Findings – We found that tags have the potential to describe and provide access to web site content from the users’ perspective and language use. Social tags may be a means to bridge between scientific viewpoints and terminology and everyday problems and vocabulary. Tags at Blogomkraeft.dk are mainly factual, often detailed, and do not cover as many functions as tags in more general bookmarking systems. An important finding is that some tags seemed to add to and supplement the content instead of factually describing the content of a blog posting. The usability test showed that our test persons liked the tagging feature. Social implications – Tagging features give the public an opportunity to apply their own terms to documents, reflecting their own model of the current topic. Tags may furthermore function as colloquial lead-in terms from users’ search formulations at search engines such as Google to the domain-specific, tailored cancer web site. Originality/value – Unlike most research on social tagging so far, we investigate tagging in a domain-specific setting, how tags can improve the interaction and communication between layman users and domain experts in an information web site within health care.

Tags in domain-specific sites: new information?

If researchers use tags in retrieval applications they might assume, implicitly, that tags represent novel information, e.g., when they attribute performance improvement in their retrieval algorithm(s) to the use of tags. In this work, we investigate whether this assumption is true. We focus on the use of tags in domain-specific websites because such websites are more likely to have a coherent, discernible website structure and because the users that are searching for and tagging pages in such a site may have specific information needs (as opposed to the broad range of information needs that users have when browsing/searching the Internet at large). For this study, we assume that the application of the same tag to multiple pages provides an indication that those pages are related. To determine whether this indication of relatedness is contributing new information, we first measure whether pages with common tag(s) could have been deemed as related based on site structure as measured by shortest navigational distance between pages. Second, we measure whether or not tags could have been determined algorithmically based on standard tf-idf scores of terms on the page. Based on our analysis of two different sites, we found that tags contribute novel information that is not discernible from site structure or site/page content.

Do User (Browse and Click) Sessions Relate to Their Questions in a Domain-specific Collection?

We seek to improve information retrieval in a domain-specific collection by clustering user sessions as recorded in a click log and then classifying later user sessions in real-time. As a preliminary step, we explore the main assumption of this approach: whether user sessions in such a site relate to the question that they are answering. The contribution of this paper is the evaluation of the suitability of common machine learning measurements (measuring the distance between two sessions) to distinguish sessions of users searching for the answer to same or different questions. We found that sessions for people answering the same question are significantly different than those answering different questions, but results are dependent on the distance measure used. We explain why some distance metrics performed better than others.