Michael B. Eisenberg

A re-examination of relevance: toward a dynamic, situational definition

Abstract Although relevance judgments are fundamental to the design and evaluation of all information retrieval systems, information scientists have not reached a consensus in defining the central concept of relevance. In this paper we ask two questions: What is the meaning of relevance? and What role does relevance play in information behavior? We attempt to address these questions by reviewing literature over the last 30 years that presents various views of relevance as topical, user-oriented, multidimensional, cognitive, and dynamic. We then discuss traditional assumptions on which most research in the field has been based and begin building a case for an approach to the problem of definition based on alternative assumptions. The dynamic, situational approach we suggest views the user — regardless of system — as the central and active determinant of the dimensions of relevance. We believe that relevance is a multidimensional concept; that it is dependent on both internal (cognitive) and external (situational) factors; that it is based on a dynamic human judgment process; and that it is a complex but systematic and measurable phenomenon.

Order effects: A study of the possible influence of presentation order on user judgments of document relevance

Studies concerned with the evaluation of information systems have typically relied on judgments of relevance as the fundamental measure in determining system performance. In most cases, subjects are asked to assign a relevance score using some category rating scale (1–4, 1–11, or simply relevant/non-relevant) to each document in a set retrieved in response to some information need or query. While the extensive studies of relevance conducted in the 1960s indicated that relevance judgments are influenced by a range of variables, little attention has been paid to the possible effects of the order in which the stimuli are presented to judges. This effect of “stimulus order” has been found to exist in measuring variables in other fields (Stevens 1975, Gescheider 1985). Questioning possible “presentable order effects” is particularly appropriate in that systems are being developed and evaluated in information science which present documents in some systematic way (e.g., with the documents considered by the system to be most relevant presented first). This article describes an effort to study whether the order of document presentation to judges influences the relevance scores assigned to those documents. A query and set of documents with relevance judgments were available from a previous study. Subjects were randomly assigned one of two orders (one ranked high to low, the other low to high) of fifteen document descriptions. They were then asked to assign a score to each document description to match their judgment of relevance in relation to the stated information query. Both a category rating (1–7) and open-ended, magnitude estimation scaling procedure were tested, and it was found that the judgments were influenced by the order of document presentation.

Measuring relevance judgements

Abstract Accurate, reliable measurement of relevance is fundamental to research in information science and to the design, development, and evaluation of information systems. This article describes a study focusing on the measurement of relevance and the application of magnitude estimation, an open-ended scaling technique developed in the field of sensory psychophysics, to the task of measuring relevance judgments. The study found that magnitude estimation is highly appropriate for the measurement of relevance judgments and less influenced by potential biases than commonly used category rating scaling procedures. One biasing factor—a range context effect—was found for both magnitude estimation and category rating scales.

Information-based education: an investigation of the nature and role of information attributes in education

Abstract Education is fundamentally affected by a series of events generally referred to as “the information explosion.” The enormous increase in available information presents major concerns related to educational content, teaching processes, and basic information skills needed by learners. Researchers need to probe into serious questions about the nature, role, and impact of specific attributes of information (e.g., information systems, resources, skills, and processes) in education. Although most educators would agree that all education is information-based to some degree, there is no unifying framework, model, theory, methodology, or even vocabulary for considering the nature and impact of specific information attributes. This paper seeks to describe the concept of information-based education (IBE), provide a theoretical basis for conducting a broad and systematic investigation into the role and use of information in educational situations, and commence the development of a classification scheme for documenting and distinguishing among various information bases. This classification scheme will form the necessary foundation for conducting field studies about information use in education, as well as foster consideration of information attributes when planning and delivering instruction, with implications for the development and implementation of computer-based information systems in educational situations.

Information seeking for instructional planning: An exploratory study

AbstractAlthough the amount and range of educational resources on the Internet is vast, finding appropriate materials for a given instructional situation is often problematic. This research study explored the information-seeking and Internet-searching patterns of prekindergarten through 12th-grade educators as part of the Gateway to Educational Materials project funded by the U.S. Department of Education. Some results indicate that educators tend to search for lesson plans more than other types of instructional resources and use parts of an instructional resource or adapt a resource to their specific situations rather than use the resource “as is.” Some differences in information-seeking patterns among various educator groups were also found.

Mapping Educational Standards to the Big6

Information literacy is arguably the essential 21st century skill set for all students, and as such, it is important that students have a solid grounding in information problem-solving, the application of information literacy skills. A wide variety of authorities have called for these skills to be incorporated into educational standards both in the United States and in Europe. This paper compares the American Association of School Librarians (AASL) information literacy standards to the Big6 model of information problem-solving to determine the extent that these standards engage with all aspects of the information problem-solving process—especially the stage of Evaluation. Findings indicate that Evaluation and Task Definition seem to be under-emphasized in the AASL standards and missing entirely from the CCSS and should be addressed in research, policy, and practice.

Challenges and Opportunities in Virtual Worlds: Informed Consent and Data Collection

This paper introduces innovative techniques for conducting research in virtual worlds. We analyze two unique methods: 1) informed consent via a ‘consent bot’ and 2) data collection via a ‘Heads-Up Display’. Such methods represent novel solutions that can be extended to other online research settings.

Avatar transparency and the establishment of trust in virtual information eco-systems

This poster addresses the issue of avatar transparency in virtual worlds, and its relationship to the establishment of trustworthiness. Preliminary analysis of interviews with Second Life® users offers insights on this topic. The authors describe the tension between anonymity and trust in participant contexts, and suggest a framework for describing avatar transparency and trust issues.

Lessons learned from a lifetime of work in information literacy

This paper is the full keynote address written for the 2014 ECIL Conference by Michael Eisenberg. Key information literacy milestones in his career that are representative of significant developments in information literacy, as well as education, information and library science, and information technology are presented in the paper.

Co-designing with the app generation

For information and technology, youth lead in defining the present and driving the future. Since early 2000 there has been an increasing interest across ASIST subfields in better understanding how youth engage with information and technology in formal and informal settings and how they can be better supported. This panel continues dialogue in the digital youth research and design community about work in-progress, upcoming events, collaboration opportunities and more. The session begins with an overview of the 2014 Digital Youth iLab held in Seattle and focuses on the trajectory of work pertaining to digital youth, its significance and the future of research, design, policy and practice. Digital youth, Digital media, design, technology, information behavior. INTRODUCTION A research community around digital youth has emerged across major professional meetings in information science—ASIST 2011 in New Orleans, LA (Fisher & Dresang, 2011), iConference 2013 in Fort Worth, TX (Fisher, Dresang, Davis, Yardi, & Druin, 2013), and iConference 2014 in Berlin, Germany (Juncker, Marrtens, Fisher, Dresang, Davis, Yardi, Druin, & Anderson, 2014). These sessions have attracted researchers and professionals across varied subfields, who share an interest in digital youth, in learning from each other, in collaborating, and in connecting the field with other disciplines, as well as industry and affecting policy. The 2014 ASIST panel aims to continue these synergies and launch conversation for establishing community research goals for 2015 and beyond. The panel will report on a national leadership forum—the Digital Youth iLab held in Seattle, share latest findings from their work as faculty in the University of Washington Information School’s Digital Youth Research Group, and share thoughts on opportunities for research programs, collaborations and impact. Like past events, the session will be highly interactive and engaging for all participants. BACKGROUND ON DIGITAL YOUTH Over the last few years there has been an increased desire to understand more about digital natives (Palfrey, 2013) and digital youth (Erstad, 2011). Notions of digital youth have been conflated with oversimplified statements about generational change promoted by the idea of digital natives and immigrants (Prensky, 2006) and amplified by notions like Generation X and Y, or Generation Me (Twenge, 2006). Other research (e.g. Buckingham, 2003) informs us that conceptions of digital youth are much more complex. This has resulted in several research initiatives (Erstad, 2012; Itō, M, 2009; Livingstone & Bulger 2013; Ahn, J. et. al, 2014) and publications (Boyd, 2014; Erstad, 2011; Gardner & Davis, 2013) dedicated to exploring how, when, and why youth use digital tools. Methodologies for digital youth research have also undergone change, becoming more collaborative and inclusive of children’s perspectives (Walsh, Foss, Yip, & Druin, 2013). In 2013 the University of Washington Information School made a commitment to helping researchers and professionals understand more about how youth engage with technology and how youth can be better supported that was based on bringing together experts from diverse, cognate fields. The iSchool with sponsorship from the Institute of Museum & library Services (IMLS), Facebook, Microsoft Research, the King County Library System, and the Seattle Public Library is organizing the 2014 Digital Youth iLab—where thought leaders from industry, research policy and practice are invited to engage and help shape a digital youth agenda to guide future work. The six areas for discussion include: digital and information literacies, formal and informal learning, games and learning, mind and brain research, policy initiatives, and social and mobile media. {This is the space reserved for copyright notices.] ASIST 2014,November 1-4, 2014, Seattle, WA, USA. [Author Retains Copyright. Insert personal or institutional copyright