Richard C. Thomas

Studying long-term system use

here have been many studies of computer users but few that extend over very long periods. The reasons for this include costs, the quality of the data that can be obtained and satisfactorily analyzed , and the requirement to obtain results in the short term. We were interested in building long-term user models for teaching systems and wished to track the total development of a guru as well as other types of users. Such goals require tracking large numbers of people over a very long period, because guru-level experience takes years to acquire, and it cannot be predicted in advance who will turn out to develop this level of skill. Therefore, to find even one guru may require the tracking of around 20 to 100 users for a year or two. Finding 10 gurus might require 1,000 users. It is possible to observe such numbers over years only by using a method that is very careful about costs. The objective of this article is to present a methodology for monitoring many users in their natural settings over the long term and to give an example of its application to the domain of students learning an editor for three years. We show how data obtained from this methodology permits the construction of individual user models. Monitoring Methodology We developed a monitoring system to study the use of the sam text editor [13]. It is essentially a modeless editor, with most basic commands invoked using the mouse. It also has powerful keyboard commands, issued through the command window, including the usual Unix regular expressions and an elegant interface to the Unix shell. We had a number of design goals for our system, which strongly influenced our methods. First, we were restricted to low system-building costs followed by very low operational costs. Second, the system needed to be highly reliable; it would not be possible to make changes once the monitoring was under way, nor could the immediate availability of people to fix problems during operation be guaranteed. Third, the impact on the computing resources had to be negligible-a degradation in performance during the day was not an option. Fourth, we wanted to give the users some control over their own data, they should be able to read it and delete it if they really wanted. Finally, and most demandingly, we wanted to measure the time of first use of each command, but …

Learning and Transfer

All users have to start off by learning some aspects of the software they wish to use. Transfer theory suggests that those with the most prior experience will often learn fastest, but this need not always be so.

Comprehension strategies and difficulties in maintaining object-oriented systems: An explorative study

Program comprehension is a major time-consuming activity in software maintenance. Understanding the underlying mechanisms of program comprehension is therefore necessary for improving software maintenance. It has been argued that acquiring knowledge of how a program works before modifying it (the systematic strategy) is unrealistic in larger programs. The goal of the experiment presented in this paper is to explore this claim. The experiment examines strategies for program comprehension and cognitive difficulties of developers who maintain an unfamiliar object-oriented system. The subjects were 38 students in their third or fourth year of study in computer science. They used a professional Java tool to perform several maintenance tasks on a medium-size Java application system in a 6-h long experiment. The results showed that the subjects who applied the systematic strategy were more likely to produce correct solutions. Two major groups of difficulties were related to the comprehension of the application structure, namely to the understanding of GUI implementation and OO comprehension and programming. Acquisition of strategic knowledge might improve program comprehension in software maintenance.

A toolkit for appraising the long term usability of a text editor

We describe a large-scale, low-cost, project that has examined the way people develop their skill in using fundamental software tools. The study involved over 2000 users during a three-year period of use of the sam text editor. The work took place while the editor was being employed in normal day to day work — it was not a laboratory experiment. Our main contributions are first to demonstrate very long-term, low-cost monitoring with collections of simple analysis tools. Second, we have started to develop an understanding of how usability changes in the long term. Third, studies of usability often concentrate on assessment before a system is released for widespread use, whereas ours can help inform the long-term design of new tools — a different dimension of usability. In addition we have mixed snap-shot studies with descriptions of long-term, gradual change. We can track the full development of the user, even though the quality of the data is lower than that normally associated with usability studies.

The pervasiveness of evolution in GRUMPS software

This paper describes the evolution of the design and implementation of a distributed run‐time system that itself is designed to support the evolution of the topology and implementation of an executing, distributed system. The three different versions of the run‐time architecture that have been designed and implemented are presented, together with how each architecture addresses the problems of topological and functional evolution. In addition, the reasons for the rapid evolution of the design and implementation of the architecture are also described.

Comparing of feedback-collection and think-aloud methods in program comprehension studies

This paper reports an explorative experimental comparison of (i) an experience-sampling method called feedback collection and (ii) the think-aloud methods with respect to their usefulness in studies on program comprehension. Think-aloud methods are widely used in studies of cognitive processes, including program comprehension. Alternatively, as in the feedback-collection method (FCM), cognitive processes can be traced by collecting written feedback from the subjects at regular intervals. We compare FCM with concurrent think-aloud (CTA) and retrospective think-aloud (RTA) regarding type and usefulness of the collected information, costs related to analysis of the collected information and effects of the data collection methods on the subjects performance. FCM allowed us to identify a greater number of comprehension problems that prevented progress or caused significant delay (FCM: 30 problems; CTA: 5; RTA: 15). It was less precise in identifying strategies for comprehension than CTA (92% correctness for FCM; 100% for CTA). FCM was less expensive in analysis (transcription and coding) than the other two methods (FCM: 0.7 h of analysis per protocol; CTA: 31 h; RTA: 7.9 h). The results indicate that all three methods of data collection were intrusive and affected the performance of the subjects with respect to time and correctness (small to medium effect size). This research confirms that FCM can be used beneficially in studies that trace the cognitive processes involved in, and identify problems related to, the comprehension of software applications. On the basis of our experience, we recommend that FCM be used in studies that have a large number of subjects and as a complement to other methods for tracing cognitive processes, such as user log files. We recommend a design with two groups (verbalisation and silent control) and a pretest task to be used in studies with FCM or CTA that focus on performances.

Use of large databases for group projects at the nexus of teaching and research

Final year, group (capstone) projects in computing disciplines are often expected to fill multiple roles: in addition to allowing students to learn important domain-specific knowledge, they should reinforce computing and software engineering concepts and provide for the acquisition of transferable skills. For motivational and pedagogical reasons, it is clearly preferable that such projects respond to real needs, be those in research or industry. We describe two student projects based on a large repository of usage data and integrated into a course in Professional Computing. These projects fulfilled the objectives outlined above and were closely linked to the research of the first author. We suggest that similar projects based on large databases may offer a transferable paradigm for others to follow. Finally, we outline some important elements for a successful group project based on a large database.

A fast trapezoidation technique for planar polygons

Abstract We present a simple and efficient algorithm for decomposing a planar polygon with holes into trapezoids. We show that our trapezoidation algorithm takes O(n log n) time, where n is the total number of vertices in the polygon and holes. The previous best algorithm for this problem by Žalik and Clapworthy (Computers and Graphics 23 (1999) 353) runs in O(n 2 log n) time. Our algorithm is extremely simple and we use only simple data structures. Our experimental results show that our algorithm is very fast in practice.

A Distributed Usage Monitoring System

We are developing a distributed computer system that supports usability and interaction studies, by handling the collection, storage and analysis of usage data, such as that generated by user-computer interaction and associated sensing devices (e.g., cameras). Data sources may be distributed as may be the data repositories and data consumers (other computer processes and human investigators). The system supports dynamic configuration of the entire process, including changes in the goals of the investigation itself. In this paper we describe the system’s key features, including a generic and evolvable data transport and processing network, a set of tools for capturing and cleaning usage data, a tool for instrumenting software for data capture, and a system for managing the entire process. We also report on several trials of the system, identifying successes, failures, lessons learned and areas for future development.

A Fresh Look at Vocabulary

The term vocabulary is used to represent the set of commands about which the user has knowledge and practical experience. In many cases only some of the commands known will be actually used: these are sometimes said to form a working set. No longer is an extensive, or even complete, knowledge of a command set considered the passport to good practice that it once was. On the other hand it tends to be believed the vocabulary of some learners will grow over a long period as they become skilled.