Joseph Janes

Other people's judgments: a comparison of users' and others' judgments of document relevance, topicality, and utility

The emerging user-centric model of relevance proposes that the only valid measure of relevance of a document to a users information need is the one made by that user. If we accept this proposition, it raises an interesting question: how well do other people, especially those involved in information work who make such judgments as part of their training and work, perform as judges of documents for information needs they did not originate? This question was empirically tested, using three groups of subjects: incoming students to a school of information/library science, continuing students in that school, and academic librarians (holders of the MLS degree). These subjects made judgments of either “relevance,” “utility,” or “topicality” of two document sets to the original users stated information need. These judgments were then compared to those of the users to see what patterns emerged, and to see what can be learned not only about secondary judgments in general, but also the ways in which information and library professionals make such judgments. These results are interesting in their own right (subjects judgments compared reasonably well to those of users, looked more like users after more training and experience in library work, and fall into interesting patterns), but they also lead to some provocative questions about the nature of judgment and evaluation of information items.

Relevance judgments and the incremental presentation of document representations

Abstract A new approach to the solicitation and measurement of relevance judgments is presented, which attempts to resolve some of the difficulties inherent in the nature of relevance and human judgment, and which further seeks to examine how users judgments of document representations change as more information about documents is revealed to them. Subjects (university faculty and doctoral students) viewed three incremental versions of documents, and recorded ratio-level relevance judgments for each version. These judgments were analyzed by a variety of methods, including graphical inspection and examination of the number and degree of changes of judgments as new information is seen. A post questionnaire was also administered to obtain subjects perceptions of the process and the individual fields of information presented. A consistent pattern of perception and importance of these fields is seen: Abstracts are by far the most important field and have the greatest impact, followed by titles, bibliographic information, and indexing.

Relevance Judgments of Actual Users and Secondary Judges: A Comparative Study

This exploratory study sought to examine judgments of relevance of document representations to query statements made by people other than the originators of the queries. A small group of graduate students in the School of Information and Library Studies and undergraduates at the University of Michigan judged sets of documents that had been retrieved for and judged by real users for a previous study. The secondary judges assessments of relevance were analyzed by themselves and in comparison to the users assessments. The judges performed reasonably well, but some important differences were identified. Secondary judges use the various fields of document records differently than users, and they have a higher threshold of relevance. There are other interesting differences and similarities between the groups. Implications of these findings for designing and testing document retrieval systems are discussed.

Networked information retrieval and organization: issues and questions

The creation of guides to resources on the Internet specific to particular subjects have raised a number of interesting questions and issues. These issues fall into two broad categories: The Internet environment (its culture, virtual communities and netiquette ; agency and authority ; the nature of publication ; the importance and lack of standards ; and searching tools and processes) and the process of guide construction (the importance of people ; the nature of resources on the Net ; intellectual property ; levels of connectivity ; and time). Some conclusions are drawn regarding the future and evolution of information professions in this environment.

The Internet public library: an intellectual history

The ideas and work behind the Internet Public Library are described, from the original conception and initial set of projects through to current endeavors. Emphasis is given to questions asked about the nature of librarianship in a digital environment and projects and services that attempt to answer those questions.