Kelly L. Maglaughlin

User Perspectives on Relevance Criteria: A Comparison among Relevant, Partially Relevant, and Not-Relevant Judgments.

This study investigates the use of criteria to assess relevant, partially relevant, and not-relevant documents. Study participants identified passages within 20 document representations that they used to make relevance judgments; judged each document representation as a whole to be relevant, partially relevant, or not relevant to their information need; and explained their decisions in an interview. Analysis revealed 29 criteria, discussed positively and negatively, that were used by the participants when selecting passages that contributed or detracted from a documents relevance. These criteria can be grouped into six categories: abstract (e.g., citability, informativeness), author (e.g., novelty, discipline, affiliation, perceived status), content (e.g., accuracy/validity, background, novelty, contrast, depth/scope, domain, citations, links, relevant to other interests, rarity, subject matter, thought catalyst), full text (e.g., audience, novelty, type, possible content, utility), journal/publisher (e.g., novelty, main focus, perceived quality), and personal (e.g., competition, time requirements). Results further indicate that multiple criteria are used when making relevant, partially relevant, and not-relevant judgments, and that most criteria can have either a positive or negative contribution to the relevance of a document. The criteria most frequently mentioned by study participants were content, followed by criteria characterizing the full text document. These findings may have implications for relevance feedback in information retrieval systems, suggesting that systems accept and utilize multiple positive and negative relevance criteria from users. Systems designers may want to focus on supporting content criteria followed by full text criteria as these may provide the greatest cost benefit.

Designing to support situation awareness across distances: an example from a scientific collaboratory

When collaborating, individuals rely on situation awareness (the gathering, incorporation and utilization of environmental information) to help them combine their unique knowledge and skills and achieve their goals. When collaborating across distances, situation awareness is mediated by technology. There are few guidelines to help system analysts design systems or applications that support the creation and maintenance of situation awareness for teams or groups. We propose a framework to guide design decisions to enhance computer-mediated situation awareness during scientific research collaboration. The foundation for this framework is previous research in situation awareness and virtual reality, combined with our analysis of interviews with and observations of collaborating scientists. The framework suggests that situation awareness is comprised of contextual, task and process, and socio-emotional information. Research in virtual reality systems suggests control, sensory, distraction and realism attributes of technology contribute to a sense of presence [Presence 7 (1998) 225]. We suggest that consideration of these attributes with respect to contextual, task and process, and socio-emotional information provides insights to guide design decisions. We used the framework when designing a scientific collaboratory system. Results from a controlled experimental evaluation of the collaboratory system help illustrate the frameworks utility.

Designing to Support Collaborative Scientific Research Across Distances: The nanoManipulator Environment

Collaboration, and increasingly multidisciplinary collaboration across distances, is a fundamental and strategic component of the scientific research process. The importance of collaboration in the scientific research process has long been recognized by the National Institutes of Health (NIH) in the USA. Its current policy is to fund the development or purchase of specialized scientific instruments in (not-for- profit) research labs across the USA and to help fund scientists to travel to those labs to collaborate and conduct scientific experiments using these specialized scientific instruments.

Passage feedback with IRIS

We compare a user-defined passage feedback (pf) system to a document feedback (df) system. Df employed the adaptive linear model for retrieval, while pf used weighted query expansion based on positive and negative feedback. Twenty-four searchers performed the same six tasks in varying search and system-order per TREC-8 guidelines. We hypothesized that pf, which featured interactive query expansion, would outperform df, which relied on automatic query expansion. Initial analysis appeared to reject this hypothesis, as df showed slightly higher overall performance than pf. However, analysis by system-order groups indicates only the first pf use had lower performance. These data suggest that pf was more difficult to learn than df, though the second pf use yielded competitive performance. If performance of pf is indeed affected by learning, an improved pf system with usability enhancements may prove to be an effective mechanism for interactive information retrieval.

Enabling distributed collaborative science

CSCW’00, December 2-6, 2000, Philadelphia, PA. ACM 1-58113-222-0/00/0012. Enabling Distributed Collaborative Science Tom Hudson1, Diane Sonnenwald2, Kelly Maglaughlin2, Mary Whitton1, and Ronald Bergquist2 1Department of Computer Science 2School of Information and Library Science University of North Carolina, Chapel Hill University of North Carolina, Chapel Hill Campus Box 3175, Sitterson Hall Campus Box 3360, Manning Hall Chapel Hill, NC 27599-3175 Chapel Hill, NC 27599-3360 {hudson, whitton}@cs.unc.edu {dhs, maglk, bergr}@ils.unc.edu

Scientific collaboratories: evaluating their potential

to scientific data, specialized instruments and documents, as well as group work tools and information and communications tools, to support collaboration among scientists. While a number of collaboratories have been developed, few have been formally evaluated. Fundamental questions have yet to be answered: Can distributed scientific research produce high quality results? Do the capabilities afforded by collaboratories outweigh their disadvantages from scientists’ perspectives? Our goal was to address these questions by evaluating a specific scientific collaboratory.