Fabio Mascarenhas

On the relation between context-free grammars and parsing expression grammars

Abstract Context-Free Grammars (CFGs) and Parsing Expression Grammars (PEGs) have several similarities and a few differences in both their syntax and semantics, but they are usually presented through formalisms that hinder a proper comparison. In this paper we present a new formalism for CFGs that highlights the similarities and differences between them. The new formalism borrows from PEGs the use of parsing expressions and the recognition-based semantics. We show how one way of removing non-determinism from this formalism yields a formalism with the semantics of PEGs. We also prove, based on these new formalisms, how LL ( 1 ) grammars define the same language whether interpreted as CFGs or as PEGs, and also show how strong- LL ( k ) , right-linear, and LL-regular grammars have simple language-preserving translations from CFGs to PEGs. Once these classes of CFGs can be automatically translated to equivalent PEGs, we can reuse classic top-down grammars in PEG-based tools.

Left recursion in Parsing Expression Grammars

Parsing Expression Grammars (PEGs) are a formalism that can describe all deterministic context-free languages through a set of rules that specify a top-down parser for some language. PEGs are easy to use, and there are efficient implementations of PEG libraries in several programming languages.A frequently missed feature of PEGs is left recursion, which is commonly used in Context-Free Grammars (CFGs) to encode left-associative operations. We present a simple conservative extension to the semantics of PEGs that gives useful meaning to direct and indirect left-recursive rules, and show that our extensions make it easy to express left-recursive idioms from CFGs in PEGs, with similar results. We prove the conservativeness of these extensions, and also prove that they work with any left-recursive PEG.PEGs can also be compiled to programs in a low-level parsing machine. We present an extension to the semantics of the operations of this parsing machine that let it interpret left-recursive PEGs, and prove that this extension is correct with regard to our semantics for left-recursive PEGs. We present a semantics for left-recursive Parsing Expression Grammars.A small extension adds precedence/associativity declarations to operator grammars.We give a semantics for compilation of left-recursive PEGs to a parsing machine.Our semantics are conservative: non-left-recursive PEGs are unaffected.

Typed Lua: An Optional Type System for Lua

Dynamically typed languages trade flexibility and ease of use for safety, while statically typed languages prioritize the early detection of bugs, and provide a better framework for structure large programs. The idea of optional typing is to combine the two approaches in the same language: the programmer can begin development with dynamic types, and migrate to static types as the program matures. The challenge is designing a type system that feels natural to the programmer that is used to programming in a dynamic language. This paper presents the initial design of Typed Lua, an optionally-typed extension of the Lua scripting language. Lua is an imperative scripting language with first class functions and lightweight metaprogramming mechanisms. The design of Typed Luas type system has a novel combination of features that preserves some of the idioms that Lua programmers are used to, while bringing static type safety to them. We show how the major features of the type system type these idioms with some examples, and discuss some of the design issues we faced.

From regexes to parsing expression grammars

Most scripting languages nowadays use regex pattern-matching libraries. These regex libraries borrow the syntax of regular expressions, but have an informal semantics that is different from the semantics of regular expressions, removing the commutativity of alternation and adding ad-hoc extensions that cannot be expressed by formalisms for efficient recognition of regular languages, such as deterministic finite automata. Parsing expression grammars are a formalism that can describe all deterministic context-free languages and have a simple computational model. In this paper, we present a formalization of regexes via transformation to parsing expression grammars. The proposed transformation easily accommodates several common regex extensions, giving a formal meaning to them. It also provides a clear computational model that helps to estimate the efficiency of regex-based matchers, and a basis for specifying provably correct optimizations for them.

A formalization of typed lua

Programmers often migrate from a dynamically typed to a statically typed language when their simple scripts evolve into complex programs. Optional type systems are one way of having both static and dynamic typing in the same language, while keeping its dynamically typed semantics. This makes evolving a program from dynamic to static typing a matter of describing the implied types that it is using and adding annotations to make those types explicit. Designing an optional type system for an existing dynamically typed language is challenging, as its types should feel natural to programmers that are already familiar with this language. In this work, we give a formal description of Typed Lua, an optional type system for Lua, with a focus on two of its novel type system features: incremental evolution of imperative record and object types that is both lightweight and type-safe, and projection types, a combination of flow typing, functions that return multiple values, and multiple assignment. While our type system is tailored to the features and idioms of Lua, its features can be adapted to other imperative scripting languages.

Error reporting in Parsing Expression Grammars

We discuss strategies for error reporting in Parsing Expression Grammars.We show that we can use a popular strategy in any implementation with semantic actions.We show a novel error reporting strategy based on labeled failures.We compare the quality of error reporting of the two strategies.We show that labeled failures can encode other error reporting strategies. Parsing Expression Grammars (PEGs) describe top-down parsers. Unfortunately, the error-reporting techniques used in conventional top-down parsers do not directly apply to parsers based on Parsing Expression Grammars (PEGs), so they have to be somehow simulated. While the PEG formalism has no account of semantic actions, actual PEG implementations add them, and we show how to simulate an error-reporting heuristic through these semantic actions.We also propose a complementary error reporting strategy that may lead to better error messages: labeled failures. This approach is inspired by exception handling of programming languages, and lets a PEG define different kinds of failure, with each ordered choice operator specifying which kinds it catches. Labeled failures give a way to annotate grammars for better error reporting, to express some of the error reporting strategies used by deterministic parser combinators, and to encode predictive top-down parsing in a PEG.

LuaRocks - A Declarative and Extensible Package Management System for Lua

While sometimes dismissed as an operating systems issue, or even a matter of systems administration, module management is deeply linked to programming language design. The main issues are how to instruct the build and runtime environments to find modules and handle their dependencies; how to package modules into redistributable units; how to manage interaction of code written in different languages; and how to map modules to files. These issues are either handled by the language itself or delegated to external tools. Language-specific package managers have risen as a solution to these problems, as they can perform module management portably and in a manner suited to the overall design of the language. This paper presents LuaRocks, a package manager for Lua modules. LuaRocks adopts a declarative approach for specifications using Lua itself as a description language and features an extensible build system that copes with the heterogeneity of the Lua ecosystem.

Exception Handling for Error Reporting in Parsing Expression Grammars

Parsing Expression Grammars (PEGs) are a new formalism to describe a top-down parser of a language. However, error handling techniques that are often applied to top-down parsers are not directly applicable to PEGs. This problem is usually solved in PEGs using a heuristic that helps to simulate the error reporting technique from top-down parsers, but the error messages are generic. We propose the introduction of labeled failures to PEGs for error reporting, as labels help to produce more meaningful error messages. The labeled failures approach is close to that of generating and handling exceptions often used in programming languages, being useful to annotate and label grammar pieces that should not fail. Moreover, our approach is an extension to the PEGs formalism that is expressive enough to implement some previous work on parser combinators. Finally, labeled failures are also useful to compose grammars preserving the error messages of each separate grammar.

Efficient compilation of Lua for the CLR

Microsofts Common Language Runtime offers a target environment for compiler writers that provides a managed execution environment and type system, garbage collection, access to OS services, multithreading, and a Just-In-Time compiler. But the CLR uses a statically typed intermediate language, which is a problem for efficient compilation of dynamically typed languages in general, and the Lua language in particular. This paper presents a way to implement a Lua compiler for the CLR that generates efficient code. The code this compiler generates outperforms the same code executed by the Lua interpreter and similar code generated by Microsofts IronPython compiler. It approaches the performance of Lua code compiled to native code by LuaJIT.

Left recursion in parsing expression grammars

Parsing Expression Grammars (PEGs) are a formalism that can describe all deterministic context-free languages through a set of rules that specify a top-down parser for some language. PEGs are easy to use, and there are efficient implementations of PEG libraries in several programming languages. A frequently missed feature of PEGs is left recursion, which is commonly used in Context-Free Grammars (CFGs) to encode left-associative operations. We present a simple conservative extension to the semantics of PEGs that gives useful meaning to direct and indirect left-recursive rules, and show that our extensions make it easy to express left-recursive idioms from CFGs in PEGs, with similar results. We prove the conservativeness of these extensions, and also prove that they work with any left-recursive PEG.