Adrian Johnstone

GLL Parsing

Recursive Descent (RD) parsers are popular because their control flow follows the structure of the grammar and hence they are easy to write and to debug. However, the class of grammars which admit RD parsers is very limited. Backtracking techniques may be used to extend this class, but can have explosive runtimes and cannot deal with grammars with left recursion. Tomita-style RNGLR parsers are fully general but are based on LR techniques and do not have the direct relationship with the grammar that an RD parser has. We develop the fully general GLL parsing technique which is recursive descent-like, and has the property that the parse follows closely the structure of the grammar rules, but uses RNGLR-like machinery to handle non-determinism. The resulting recognisers run in worst-case cubic time and can be built even for left recursive grammars.

Right nulled GLR parsers

The right nulled generalized LR parsing algorithm is a new generalization of LR parsing which provides an elegant correction to, and extension of, Tomitas GLR methods whereby we extend the notion of a reduction in a shift-reduce parser to include right nulled items. The result is a parsing technique which runs in linear time on LR(1) grammars and whose performance degrades gracefully to a polynomial bound in the presence of nonLR(1) rules. Compared to other GLR-based techniques, our algorithm is simpler and faster.

Generalised Parsing: Some Costs

We discuss generalisations of bottom up parsing, emphasising the relative costs for real programming languages. Our goal is to provide a roadmap of the available approaches in terms of their space and time performance for programming language applications, focusing mainly on GLR style algorithms. It is well known that the original Tomita GLR algorithm fails to terminate on hidden left recursion: here we analyse two approaches to correct GLR parsing (i) the modification due to Farshi that is incorporated into Visser’s work and (ii) our own right-nullable GLR (RNGLR) algorithm, showing that Farshi’s approach can be expensive. We also present results from our new Binary RNGLR algorithm which is asymptotically the fastest parser in this family and show that the recently reported reduction incorporated parsers can require automata that are too large to be practical on current machines.

GLL parse-tree generation

Backtracking techniques which are often used to extend recursive descent (RD) parsers can have explosive run-times and cannot deal with grammars with left recursion. GLL parsers are fully general, worst-case cubic parsers which have the recursive descent-like property that they are easy to write and to use for grammar debugging. They have the direct relationship with the grammar that an RD parser has. In this paper we give an algorithm for generating GLL parsers which build an SPPF representation of the derivations of the input, complementing our existing GLL recognition algorithm, and we show that such parsers and recognisers are worst-case cubic.

BRNGLR: a cubic Tomita-style GLR parsing algorithm

Tomita-style generalised LR (GLR) algorithms extend the standard LR algorithm to non-deterministic grammars by performing all possible choices of action. Cubic complexity is achieved if all rules are of length at most two. In this paper we shall show how to achieve cubic time bounds for all grammars by binarising the search performed whilst executing reduce actions in a GLR-style parser. We call the resulting algorithm Binary Right Nulled GLR (BRNGLR) parsing. The binarisation process generates run-time behaviour that is related to that shown by a parser which pre-processes its grammar or parse table into a binary form, but without the increase in table size and with a reduced run-time space overhead. BRNGLR parsers have worst-case cubic run time on all grammars, linear behaviour on LR(1) grammars and produce, in worst-case cubic time, a cubic size binary SPPF representation of all the derivations of a given sentence.

Generalised Recursive Descent parsing and Fellow-Determinism

This paper presents a construct for mapping arbitrary non-left recursive context-free grammars into recursive descent parsers that: handle ambiguous grammars correctly; perform with LL(1) efficiency on LL(1) grammars; allow straightforward implementation of both inherited and synthesized attributes; and allow semantic actions to be added at any point in the grammar. We describe both the basic algorithm and a tool, GRDP, which generates parsers which use this technique. Modifications of the basic algorithm to improve efficiency lead to a discussion of follow-determinism, a fundamental property that gives insights into the behaviour of both LL and LR parsers.

The Grammar Tool Box

The Grammar Tool Box is a toolset for manipulating Context Free Grammars and objects associated with them such as parsers, languages and derivations. GTB has three main roles: as a pedagogic tool; as an experimental platform for novel algorithms and representations; and as a production tool for translator front end generation. In this paper we give an overview of GTB and its companion Java-based animator tool PAT. We illustrate the use of the toolset in the construction of a comparative study of three variants of the Tomita-style GLR parsing algorithm running on LR(0), SLR(1) and LR(1) tables for ANSI-C, ISO-Pascal and IBM VS-COBOL, and give results showing the size of the structures constructed by these parsers and the amount of searching required during the parse, which abstracts their runtime.

Generalised reduction modified LR parsing for domain specific language prototyping

Domain specific languages should support syntax that is comfortable,for specialist users. We discuss the impact of the standard deterministic parsing techniques such as LALR(1) and LL(1) on the design of programming languages and the desirability of more flexible parsers in a development environment. We present a new bottom-up nondeterministic parsing algorithm (GRMLR) that combines a modified notion of reduction with a Tomita-style breadth-first search of parallel parsing stacks. We give experimental results,for standard programming language grammars and LR(0), SLR(1) and LR(1) tables; the weaker tables generate significant amounts of nondeterminism. We show that GRMLR parsing corrects errors in the standard Tomita algorithm without incurring the performance overheads associated with other published solutions. We also demonstrate that the performance of GRMLR is upper-bounded by the performance of Tomitas algorithm, and that,for once realistic language grammar GRMLR only needs to search around 74% of the nodes. Our heavily instrumented development version of the algorithm achieves parsing rates of around 4000-10000 tokens per second on a 400 MHz Pentium II processor. Proof of correctness and details of our implementation are omitted here for space reasons but are available in an accompanying technical report.

Modelling GLL parser implementations

We describe the development of space-efficient implementations of GLL parsers, and the process by which we refine a set-theoretic model of the algorithm into a practical parser generator that creates practical parsers. GLL parsers are recursive descent-like, in that the structure of the parsers code closely mirrors the grammar rules, and so grammars (and their parsers) may be debugged by tracing the running parser in a debugger. While GLL recognisers are straightforward to describe, full GLL parsers present technical traps and challenges for the unwary. In particular, naive implementations based closely on the theoretical description of GLL can result in data structures that are not practical for grammars for real programming language grammars such as ANSI-C. We develop an equivalent formulation of the algorithm as a high-level set-theoretic model supported by table-based indices, in order to then explore a set of alternative implementations which trade space for time in ways which preserve the cubic bound.

Evaluating GLR parsing algorithms

We describe the behaviour of three variants of GLR parsing: (i) Farshis original Correction to Tomitas non-general algorithm; (ii) the Right Nulled GLR algorithm which provides a more efficient generalisation of Tomita and (iii) the Binary Right Nulled GLR algorithm, on three types of LR table. We present a guide to the parse-time behaviour of these algorithms which illustrates the inefficiencies in conventional Farshi-style GLR parsing. We also describe the tool GTB (Grammar Tool Box) which provides a platform for comparative studies of parsing algorithms; and use GTB to exercise the three GLR algorithms running with LR(0), SLR(1) and LR(1) tables for ANSI-C, ISO-Pascal and IBM VS-COBOL. We give results showing the size of the structures constructed by these parsers and the amount of searching reguired during the parse, which abstracts their runtime.