Andrew J. Malton

Source transformation in software engineering using the TXL transformation system

Abstract Many tasks in software engineering can be characterized as source to source transformations. Design recovery, software restructuring, forward engineering, language translation, platform migration, and code reuse can all be understood as transformations from one source text to another. The tree transformation language, TXL, is a programming language and rapid prototyping system specifically designed to support rule-based source to source transformation. Originally conceived as a tool for exploring programming language dialects, TXL has evolved into a general purpose source transformation system that has proven well suited to a wide range of software maintenance and reengineering tasks, including the design recovery, analysis and automated reprogramming of billions of lines of commercial Cobol, PL/I, and RPG code for the Year 2000. In this paper, we introduce the basic features of modern TXL and its use in a range of software engineering applications, with an emphasis on how each task can be achieved by source transformation.

Agile Parsing in TXL

Syntactic analysis forms a foundation of many source analysis and reverse engineering tools. However, a single standard grammar is not always appropriate for all source analysis and manipulation tasks. Small custom modifications to the grammar can make the programs used to implement these tasks simpler, clearer and more efficient. This leads to a new paradigm for programming these tools: agile parsing. In agile parsing the effective grammar used by a particular tool is a combination of two parts: the standard base grammar for the input language, and a set of explicit grammar overrides that modify the parse to support the task at hand. This paper introduces the basic techniques of agile parsing in TXL and discusses several industry proven techniques for exploiting agile parsing in software source analysis and transformation.

Union schemas as a basis for a C++ extractor

An extractor is a program which processes source code and outputs facts about the code in a software exchange format (SEF). An SEF can be further specified by a schema, analogous to a schema for a database. This paper explains how two such schemas can be combined into a union schema as the basis for creating an extractor. We describe CPPX (C Plus Plus eXtractor), which transforms a schema designed as the internals of a compiler (GCC) to a schema designed for software exchange (Datrix). CPPX performs this transformation as a pipelined sequence of sub-transformations. At each stage in the pipeline, the intermediate data conforms to the union of the two schemas.

Using design recovery techniques to transform legacy systems

The year 2000 problem posed a difficult problem for many IT shops world wide. The most difficult part of the problem was not the actual changes to ensure compliance, but finding and classifying the data fields that represent dates. This is a problem well suited to design recovery. The paper presents an overview of LS/2000, a system that used design recovery to analyze source code for year 2000 risks and guide a source transformation that was able to automatically remediate over 99% of the year 2000 risks in over three billion lines of production IT source.

Grammar programming in TXL

Syntactic analysis forms a foundation of many source analysis and reverse engineering tools. However, a single grammar is not always appropriate for all source analysis and manipulation tasks. Small changes to the grammar can make the programs used to accomplish these tasks simpler and more straightforward. This leads to a new paradigm of programming these tools: grammar programming. This paper discusses several industry proven techniques that can be used when designing base grammars and when making task specific changes to grammars.

The denotational semantics of a functional tree-manipulation language

The programming language TXL combines features of functional languages with a pattern-matching and failure-handling paradigm, designed for expressing tree-manipulation algorithms. The semantic problems, caused by the interaction of evaluation and pattern matching, are solved by choosing a suitable denotation for patterns and failure. There is no need for general continuations.

Correlating Social Interactions to Release History during Software Evolution

In this paper, we propose a method to reason about the nature of software changes by mining and correlating discussion archives. We employ an information retrieval approach to find correlation between source code change history and history of social interactions surrounding these changes. We apply our correlation method on two software systems, LSEdit and Apache Ant. The results of these exploratory case studies demonstrate the evidence of similarity between the content of free-form text emails among developers and the actual modifications in the code. We identify a set of correlation patterns between discussion and changed code vocabularies and discover that some releases referred to as minor should instead fall under the major category. These patterns can be used to give estimations about the type of a change and time needed to implement it.

HSML: design directed source code hot spots

HSML, the Hot Spot Markup Language, is an ultra-high level executable specification language designed for concisely specifying source code hot spots of all kinds. Each HSML rule specifies the abstract syntactic class of the items to be marked as hot using a nonterminal of the target language grammar, and the semantic conditions under which such items are to be marked using an algebraic expression on the design properties of the item. Conditions can include restrictions on abstract syntactic structure (patterns), design recovered semantic properties (queries on the design database), and semantic properties induced by other markup rules. HSML has been used in industrial practice to specify source code hot spots for the Year 2000 and a wide range of other application maintenance tasks on systems implemented in Cobol, PL/I and RPG. We introduce the basic concepts of HSML and demonstrate its use in real software maintenance tasks.

Software engineering by source transformation - experience with TXL

Many tasks in software engineering can be characterized as source to source transformations. Design recovery, software restructuring, forward engineering, language translation, platform migration and code reuse can all be understood as transformations from one source text to another. TXL, the Tree Transformation Language, is a programming language specifically designed to support rule-based source to source transformation. Originally conceived as a tool for exploring programming language dialects, TXL has evolved into a general purpose software transformation system that has proven well suited to a wide range of software maintenance and reengineering tasks, including the design recovery, analysis and automated reprogramming of billions of lines of commercial Cobol, PL/I and RPG code for the Year 2000. The authors introduce the basic features of modern TXL and its use in a range of software engineering applications, with an emphasis on how each task can be achieved by source transformation.

Completeness of a fact extractor

The process of software reverse engineering commonlyuses an extractor, which parses source code and extractsfacts about the code. The level of detail in these factsvaries from extractor to extractor. This paper describesfour levels of increasingly detailed completeness of thesefacts: (semantic completeness, compiler completeness,syntax completeness and source completeness) andintroduces the concept of relative completeness ofextractors. Validating that an extractor correctlyproduces facts at a given level of completeness is ingeneral very challenging. This paper gives a method forvalidating the semantic completeness of an extractor, anddescribes the application of this method to CPPX, anextractor for C or C++ based on GCC.