Dennis C. Neale

Notification and awareness: synchronizing task-oriented collaborative activity

People working collaboratively must establish and maintain awareness of one anothers intentions, actions and results. Notification systems typically support awareness of the presence, tasks and actions of collaborators, but they do not adequately support awareness of persistent and complex activities. We analysed awareness breakdowns in use of our Virtual School system--stemming from problems related to the collaborative situation, group, task and tool support--to motivate the concept of activity awareness. Activity awareness builds on prior conceptions of social and action awareness, but emphasizes the importance of activity context factors like planning and coordination. This work suggests design strategies for notification systems to better support collaborative activity.

Evaluating computer-supported cooperative work: models and frameworks

Evaluating distributed CSCW applications is a difficult endeavor. Frameworks and methodologies for structuring this type of evaluation have become a central concern for CSCW researchers. In this paper we describe the problems involved in evaluating remote collaborations, and we review some of the more prominent conceptual frameworks of group interaction that have driven CSCW evaluation in the past. A multifaceted evaluation framework is presented that approaches the problem from the relationships underlying joint awareness, communication, collaboration, coordination, and work coupling. Finally, recommendations for carrying out multifaceted evaluations of remote interaction are provided.

The development of cooperation: five years of participatory design in the virtual school

During the past five years, our research group worked with a group of public school teachers to define, develop, and assess network-based support for collaborative learning in middle school physical science and high school physics. From the outset, we committed to a participatory design approach. This design collaboration has now existed far longer than is typical of participatory design endeavors, particularly in North America. The nature of our interactions, and in particular the nature of the role played by the teachers has changed significantly through the course of the project. We suggest that there may be a long-term developmental unfolding of roles and relationships in participatory design.

Classroom BRIDGE: using collaborative public and desktop timelines to support activity awareness

Classroom BRIDGE supports activity awareness by facilitating planning and goal revision in collaborative, project-based middle school science. It integrates large-screen and desktop views of project times to support incidental creation of awareness information through routine document transactions, integrated presentation of awareness information as part of workspace views, and public access to subgroup activity. It demonstrates and develops an object replication approach to integrating synchronous and asynchronous distributed work for a platform incorporating both desktop and large-screen devices. This paper describes an implementation of these concepts with preliminary evaluation data, using timeline-based user interfaces.

Multi-faceted evaluation for complex, distributed activities

Computer-supported cooperative learning presents challenges for evaluation methodology: Learning events and learning outcomes are dispersed in time and space, making causal relationships difficult to identify. We are developing techniques to address these challenges including systematic sampling, collation of multiple evaluation methods and data, and the use of collaborative critical incidents. In this paper we overview and discuss this emerging methodology.

Collaborative Critical Incident Development

We have been using critical incidents to research the design and evaluation of computer-mediated collaborative and communication technologies for use in K-12 education. A variety of methods have generated a number of critical incidents identified during classroom evaluations. To enhance our analysis and understanding of these incidents, we developed a Web-based forum called the collaborative critical incident tool (CCIT). Users (teachers) and researchers collaboratively post, rate incident criticality, and elaborate on critical incidents through sustained dialog, contributing to the understanding of underlying conditions, causes, and implications. In this paper we describe the tool and its impact on evaluation.

Data Logging: Higher-Level Capturing and Multi-Level Abstracting of User Activities

Data logging has been a standard, but under utilized, software evaluation technique for single-user systems. Large volumes of objective data can be collected automatically and unobtrusively. This data, however, is usually in the form of low-level system events, making it difficult to analyze and interpret meaningfully. In this paper we extend traditional logging approaches to collaborative multi-user (groupware) systems. We also show how data captured at a higher level of abstraction can characterize user-system interaction more meaningfully. Lastly, we show how higher-level data abstracted from logging can be more effectively combined with data from other usability methods.

Mentoring in a school environment

We describe how the integration of multiple computer-mediated communication and collaboration technologies can support educational activity through mentoring in a local community context. We discuss the student and mentor expectations, their ongoing relationships, and the role that technology has played throughout their interactions.

Laboratory Simulation Methods for Studying Complex Collaborative Tasks

This paper describes a new laboratory model developed for studying complex CSCW phenomena. In our prior work a more ecological laboratory approach was developed to study activity awareness issues derived directly from earlier fieldwork. In the first study participants worked on a simulated long-term project with a confederate who introduced collaborative breakdown scenarios. Although the study produced many realistic behaviors, findings indicated that there was room for improving the model by making the simulation more ecological and engaging for participants. This paper examines the results of a follow-up study conducted with the goal of improving the laboratory model. The follow-up study varied two important elements of simulation from the first study. A real long-term project was used, and the confederate was replaced with another participant. Differences in results from the two studies indicate how the changes made to the model impacted its effectiveness.

Knowledge Reuse through Categorical Breakdown Analysis: A Method for Collaborative Systems Evaluation

Designing CSCW systems that support the widely varying needs of targeted users is difficult. There is no silver bullet technology that enables users to effectively collaborate with one another in different contexts. We propose a method of collaborative systems evaluation that enables novice evaluators to make insightful observations about the systems they evaluate at a level comparable to experts in certain situations. These observations come in the form of a categorical breakdown analysis of a laboratory study. The quantity and type of breakdowns can then be connected to recommended CSCW tools and features developed and described in the related literature. We conducted a study to explore the results generated when the method was applied by both experts and novices in the field of CSCW. We observed that experts found the method to be usable, and that novices capitalized on the knowledge embodied in the breakdown categories to make categorizations similar to those of experts.