Susan Elliott Sim

Using benchmarking to advance research: a challenge to software engineering

Benchmarks have been used in computer science to compare the performance of computer systems, information retrieval algorithms, databases, and many other technologies. The creation and widespread use of a benchmark within a research area is frequently accompanied by rapid technical progress and community building. These observations have led us to formulate a theory of benchmarking within scientific disciplines. Based on this theory, we challenge software engineering research to become more scientific and cohesive by working as a community to define benchmarks. In support of this challenge, we present a case study of the reverse engineering community, where we have successfully used benchmarks to advance the state of research.

The ramp-up problem in software projects: a case study of how software immigrants naturalize

Joining a software development team is like moving to a new country to start employment; the immigrant has a lot to learn about the job, the local customs, and sometimes a new language. In an exploratory case study, we interviewed four software immigrants, in order to characterize their naturalization process. Seven patterns in four major categories were found. In this paper, these patterns are substantiated, and their implications discussed. The lessons learned from this study can be applied equally to improving the naturalization process, and to the formulation of further research questions.

Archetypal source code searches: a survey of software developers and maintainers

We have conducted a survey to generate archetypes of source code searching by programmers across maintenance tasks. Using a questionnaire on a web page, we obtained 69 responses from readers of 7 newsgroups. Respondents were asked about their source code searching habits: what tools they used, why they searched, and what they searched for. The four most common search targets were function definitions, all uses of a function, variable definitions, and all uses of a variable. The most common search motivations were defect repair, code reuse, program understanding, feature addition, and impact analysis. Eleven archetypes were generated from the anecdotes and results. The implications and practical applications of these findings and method are discussed.

GXL: a graph-based standard exchange format for reengineering

GXL (Graph eXchange Language) is an XML-based standard exchange format for sharing data between tools. Formally, GXL represents typed, attributed, directed, ordered graphs which are extended to represent hypergraphs and hierarchical graphs. This flexible data model can be used for object-relational data and a wide variety of graphs. An advantage of GXL is that it can be used to exchange instance graphs together with their corresponding schema information in a uniform format, i.e. using a common document type specification. This paper describes GXL and shows how GXL is used to provide interoperability of graph-based tools. GXL has been ratified by reengineering and graph transformation research communities and is being considered for adoption by other communities.

Towards a standard schema for C/C++

Developing a standard schema at the abstract syntax tree (AST) level for C/C++ to be used by reverse engineering and reengineering tools is a complex and difficult problem. In this paper we present a catalogue of issues that need to be considered in order to design a solution. Three categories of issues are discussed. Lexical structure is the first category and pertains to characteristics of the source code, such as spaces and comments. The second category, syntax, includes both the mundane and hard problems in the C++ programming language. The final category is semantics and covers aspects such as naming and reference resolution. Example solutions to these challenges are provided from the Datrix schema from Bell Canada and the Columbus schema from University of Szeged. The paper concludes with a discussion of lessons learnt and plans for future work on a C/C++AST standard schema.

On using a benchmark to evaluate C++ extractors

In this paper, we take the concept of benchmarking, as used extensively in computing, and apply it to the evaluation of C++ fact extractors. We demonstrate the efficacy of this approach by developing a prototype benchmark, CppETS 1.0 (C++ Extractor Test Suite, pronounced see-pets) and collecting feedback in a workshop setting. The CppETS benchmark characterises C++ extractors along two dimensions: accuracy and robustness. It consists of a series of test buckets that contain small C++ programs and related questions that pose different challenges to the extractors. As with other research areas, benchmarks are best developed through technical work and consultation with a community, so we invited researchers to apply CppETS to their extractors and report on their results in a workshop. Four teams participated in this effort, evaluating the four extractors Ccia, cppx, the Rigi C++ parser and TkSee/SN. They found that CppETS gave results that were consistent with their experience with these tools and therefore had good external validity. Workshop participants agreed that CppETS was an important contribution to fact extractor development and testing. Further efforts to make CppETS a widely-accepted benchmark will involve technical improvements and collaboration with the broader community.

A structured demonstration of program comprehension tools

This paper describes a structured tool demonstration, a hybrid evaluation technique that combines elements from experiments, case studies and technology demonstrations. Developers of program understanding tools were invited to bring their tools to a common location to participate in a scenario with a common subject system. Working simultaneously the tool teams were given reverse engineering tasks and maintenance tasks to complete on an unfamiliar subject system. Observers were assigned to each team to find out how useful the observed program comprehension tool would be in an industrial setting. The demonstration was followed by a workshop panel where the development teams and the observers presented their results and findings from this experience.

A collaborative demonstration of reverse engineering tools

This paper describes a collaborative structured demonstration of reverse engineering tools that was presented at a working session at WCRE 2001 in Stuttgart, Germany. A structured demonstration is a hybrid tool evaluation technique that combines elements from experiments, case studies, technology demonstrations, and benchmarking. The essence of the technique is to facilitate learning about software engineering tools using a common set of tasks. The collaborative experience discussed at WCRE involved several peer and complementary technologies that were applied in concert to solve a real life reverse engineering problem. For the most part, the tool developers themselves applied their own tools to this problem. Preliminary results have shown to the research community that we still have much to learn about our tools and how they can be applied as part of a reverse engineering and reengineering process. Consequently, the participants agreed to continue participation in this demonstration beyond the WCRE event.

WoSEF: workshop on standard exchange format

A workshop was held at ICSE 2000 in Limerick, Ireland to further efforts in the development of a standard exchange format (SEF) for data extracted from and about source code. WoSEF (Workshop on Standard Exchange Format) brought together people with expertise in a variety of formats, such as RSF, TA, GraX, FAMIX, XML, and XMI, from across the software engineering discipline. We had five sessions consisting of a presentation and discussion period and a working session with three subgroups. The five sessions were: 1) Survey and Overview, 2) Language-level schemas and APIs, 3) High-level schemas, 4) MOF/XMI/UML and CDIF, and 5) Meta schemas and Typed Graphs. During that time we reviewed previous work and debated a number of important issues. This report includes descriptions of the presentations made during these sessions. The main result of the workshop is the agreement of the majority of participants to work on refining GXL (Graph eXchange Language) to be the SEF. GXL is an XML-based notation that uses attributed, typed graphs as a conceptual data model. It is currently a work in progress with contributors from reverse engineering and graph transformation communities in multiple countries. There is a great deal of work to be done to finalise the syntax and to establish reference models for schemas. Anyone interested is welcome to join the effort and instructions on how to get involved are found at the end of the workshop report. Three papers from the workshop have been reprinted here to promote reflection and encourage participation in the work to develop an SEF.

Next generation data interchange: tool-to-tool application program interfaces

Data interchange in the form of a standard exchange format (SEF) is only a first step rewards tool interoperability. Inter-tool communication using files is slow and cumbersome; a better approach would be an application program interface, or API, that allowed tools to communicate with each other directly. The paper argues that such an API is a logical next step that builds on the current drive towards an SEF. It presents high-level descriptions of three approaches to tool-to-tool APIs and illustrates how requirements for the SEF also apply to the API.