Adding Interactive Visual Syntax to Textual Code
2222Adding Interactive Visual Syntax to Textual Code
LEIF ANDERSEN, MICHAEL BALLANTYNE, MATTHIAS FELLEISEN,
PLT and NortheasternUniversity, United States of AmericaMany programming problems call for turning geometrical thoughts into code: tables, hierarchical structures,nests of objects, trees, forests, graphs, and so on. Linear text does not do justice to such thoughts. But, it hasbeen the dominant programming medium for the past and will remain so for the foreseeable future.This paper proposes a novel mechanism for conveniently extending textual programming languages withproblem-specific visual syntax. It argues the necessity of this language feature, demonstrates the feasibilitywith a robust prototype, and sketches a design plan for adapting the idea to other languages.CCS Concepts: •
Software and its engineering → Visual languages ; Macro languages ; Domain spe-cific languages ; Graphical user interface languages ; Integrated and visual development environments ;• Human-centered computing → Human computer interaction (HCI) ; Accessibility technologies.Additional Key Words and Phrases: Domain-Specific Language
ACM Reference Format:
Leif Andersen, Michael Ballantyne, Matthias Felleisen. 2020. Adding Interactive Visual Syntax to Textual Code.
Proc. ACM Program. Lang.
4, OOPSLA, Article 222 (November 2020), 28 pages. https://doi.org/10.1145/3428290
Code is a message from a developer in the present to a developer in the future, possibly the sameperson but aged. This future developer must comprehend the code and reconstruct the thoughtsthat went into it. Hence, writing code means articulating thoughts as precisely as possible.Often these thoughts involve geometrical relationships: tables, nests of objects, graphs, etc.Furthermore, the geometry differs from problem domain to problem domain. To this day, though,programmers articulate their thoughts as linear text. Unsurprisingly, code maintainers have a hardtime reconstructing geometrical thoughts from linear text, which reduces their productivity.At first glance, visual languages [Boshernitsan and Downes 2004] eliminate this problem, butthey actually don’t. They too offer only a fixed set of constructs, though visual ones—meaning avisual language fails to address the problem-specific nature of geometric thought. And, as historyshows, developers clearly prefer textual languages over visual ones—meaning graphical syntaxshould aim to supplement , not displace, textual syntax.This paper presents a mechanism for extending textual languages with interactive and visual programming constructs tailored to specific problem domains. It demonstrates the feasibility ofthis idea with a prototype implementation. The design space description suggests the architecturecould be adapted to a broad spectrum of programming languages.
Author’s address: Leif Andersen, Michael Ballantyne, Matthias Felleisen, PLT, Khoury College of Computer Sciences,Northeastern University, 440 Huntington Ave., Boston, Massachusetts, 02115, United States of America, [email protected] to make digital or hard copies of part or all of this work for personal or classroom use is granted without feeprovided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice andthe full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses,contact the owner/author(s).© 2020 Copyright held by the owner/author(s).2475-1421/2020/11-ART222https://doi.org/10.1145/3428290Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 222. Publication date: November 2020. a r X i v : . [ c s . P L ] O c t To make this idea precise, consider a software system that implements a game such as Tsuro. Inthis game, players take turns growing a graph from square tiles, each of which displays four pathsegments. A player places one avatar at an entry point on the periphery of the grid-shaped board.New tiles are added next to a player’s avatar, and all avatars bordering this new tile are moved asfar as possible along the newly extended paths until they face an empty place again. If an avatarexits the board, its owner-player is eliminated. The last surviving player wins.Now imagine a programmer wishing to articulate unit testsin the context of a Tsuro implementation. When a mechanismfor creating interactive and visual syntax is a available, the testermay add a Tsuro tile as a new language construct. This developercreates an instance of this syntax via UI actions, i.e., key strokesor menu selection, and simultaneously inserts it into the IDE. Aninstance is referred to as editor . Consider the image on the right,which shows a tile editor. It displays a graphical representation ofthe tile (left) and manual entry text fields (right). These text fields and graphical representationare linked. The graphic updates whenever a user updates the text; the text fields update when theprogrammer connects two nodes graphically via GUI actions. What is shown here is the state ofthis syntax just as the developer is about to connect the nodes labeled "C" and "D" .Every tile editor compiles to code that evaluates to a bidirectional hash-table representationof its connections. For the above example, the hash-table connects the "A" node with "G" , as thefollowing lookup operations confirm: > (hash-ref 'A)'G > (hash-ref 'G)'A
Note how the editor itself assumes the role of a hash-table here.A Tsuro developer will also create interactive syntax for Tsuro boards.The board syntax supplies a grid of slots. Each slot is initially empty, butthe programmer may place Tsuro tiles there to mimic players’ actions.Take a look at the nearby example. In this image, three players haveclearly placed one tile each (the bottom row extreme left and extremeright plus the third slot in the top row), and each tile is occupied by anavatar. As far as run time is concerned, the Tsuro board editor evaluatesto a matrix of tiles.Once a programmer has extended the language with these two Tsuro-specific language constructs,a unit test using these graphical editors looks as follows in code: (check-equal? (send addTile player0) )
This one-line unit test checks whether the addTile method works properly. The method expectsan initial board state, a tile, and a player. Its result is a new board state with the tile placed in the https://en.wikipedia.org/wiki/TsuroProc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 222. Publication date: November 2020. dding Interactive Visual Syntax to Textual Code 222:3 (check-equal? (send (new tsuro-board% [tiles (hash '(2 0) '(H D E B C G F A)'(0 3) '(E F H G A B D C)'(3 3) '(E C B H A G F D))] [players (hash player1 '(2 0 A)player2 '(0 3 H)player3 '(3 3 E))]) addTile (new tsuro-tile% [connections '(G F D C H B A E)]) player1) (new tsuro-board% [tiles (hash '(2 0) '(H D E B C G F A)'(0 3) '(E F H G A B D C)'(3 3) '(E C B H A G F D)'(1 0) '(G F D C H B A E))] [players (hash player1 '(2 0 A)player2 '(0 3 H)player3 '(3 3 E))])) Fig. 1 . Textual Test Case slot that the player’s avatar faces in the given board state and with the avatar moved as far aspossible so that it again faces an empty spot on the grid.For comparison, figure 1 articulates the same unit test with plain textual code. As with thegraphical unit test, addTile expects a board, tile, and player. The board is constructed with tilesand player start locations. Each tile is a list of eight letters, each representing its connecting node.The authors invite the reader to improve on this notational choice and compare their inventionwith the visual syntax above.Interactive visual syntax is just syntax, and syntax composes. An editor may appear within textualsyntax, as shown above. And textual syntax may appear within interactive syntax. Let’s return to ourTsuro developer who may wish to write helper functions for unit tests that produce lists of board con-figurations for exploring moves. Here is such a function, again extracted from the authors’ code base. ; Tile -> [Listof Board](define (all-possible-configurations t)(for/list ([d DEGREES])... (send t rotate d) ...))
As the type signature says, this functionconsumes a tile and generates a list ofboards. Specifically, it ( for ) loops over alist of
DEGREES , with each iteration gen-erating an element of the resulting list(hence for/list ). Each iteration generates a board by rotating the given tile t by d degreesand placing it in a fixed board context. The dots surrounding the method call are supposed tosuggest this fixed context.Once again, the developer can either express this context as a map like that of figure 1 or use aninstance of interactive visual syntax. Figure 2 shows the second scenario. The spot on the boardwhere the tile is to be inserted is a piece of interactive syntax for editing code. The zoomed image onthe right indicates how a developer manipulates this code. Clicking on this tile pops up a separatetext editor. The developer manipulates code in this editor and closes the editor when the code iscompleted. Creating such an interactive Tsuro board is only slightly more work than creating theone used for the unit test above—but, in the opinion of the authors, the message it sends to thefuture maintainer of the code is infinitely clearer than plain text could ever be. Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 222. Publication date: November 2020. ; Tile -> [Listof Board](define (all-possible-configurations t)(for/list ([d DEGREES]))) (send t rotate d)
Fig. 2 . Code Inside Interactive Visual Syntax
The preceding examples are code from the authors’ code base, using a prototype implementationof the interactive-syntax extension mechanism. In fact, this paper itself is written using thisprototype for live rendering and figure manipulation. While the prototype is implemented inRacket [Flatt and PLT 2010], the next section discusses a general design space, which should helplanguage implementers to adapt this idea to their world. Following that, the paper demonstratesthe expressive power of interactive syntax, explains the Racket implementation, contrasts it withother attempts at combining code with images, and concludes with a concise explanation of howthe prototype falls short and what is needed to explore this idea from a user-facing perspective.
The project starts from an acknowledgment of the dominance of linear text. Its goal is to enabledevelopers to supplement linear text with visual syntax as soon as they are tempted to documentany code with some form of diagram. Furthermore, the transition from linear text to visual andinteractive syntax must become as smooth as possible. Two prior projects [Eisenberg and Kiczales2007; French et al. 2014] share the fundamental assumption and the ultimate goal of this project.Their shortcomings, however, expose the serious challenges of this undertaking.In order to gain wide spread adoption, interactive syntax must satisfy three primary criteria.First, it is imperative to demonstrate the feasibility of mixing textual and interactive syntax in thecontext of an ordinary and existing programming language. Ordinary here refers to the kind oflanguages developers already use: with classes, objects, functions, and side effects. Existing does notmean “widely used,” though in use by a non-trivial community. Adding a visual-syntax mechanismto such a language is the simplest ways to tempt programmers into its use and to convince them ofits usefulness as well as its usability. Second, more than one specific interactive development environment (IDE) must accommodateinteractive syntax. Indeed, programs in the revised language must not even preclude editing inpurely textual IDEs ones such as Vim. Developers frequently have strong preferences concerningthe IDE they use. They are much less likely to allow teammates to use interactive syntax if it meanseveryone on the team must migrate from their favorite IDEs to a specific one. The popularity of notebooks, REPLs, prompts, and virtual machines demonstrate the usefulness of visual organizationsfor experimentation and data analysis. But, they also show how quickly such an approach becomes unwieldy as softwaresystems grow. Techniques such as snapshots and cafés [Dybvig 2006] reduce the problem, but do not eliminate it.Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 222. Publication date: November 2020. dding Interactive Visual Syntax to Textual Code 222:5
Finally, developers must be able to amortize the investment in graphical user interfaces. Theconstruction of interactive-syntax extensions demands code that implements simple GUIs, a poten-tially time-consuming task compared to, say, drawing ASCII diagrams. If these GUIs can share codewith the actual interface of the software system, though, the cost of creating interactive syntaxextensions may look quite reasonable. The implication is that an interactive-syntax extensionmechanism must use the existing GUI libraries of the chosen language as much as possible.With these criteria in mind, we can now state specific design desiderata:(1)
An interactive visual syntax is just syntax. It merely articulates an idea better than textualsyntax.
If the underlying grammar distinguishes among definitions, expressions, patterns,and other syntactic categories, it should be possible to use visual syntax in all of them.(2)
Interactive syntax is persistent.
The point of interactive syntax is that it permits developers tosend a visual message across time. In contrast to wizards and code generators, it is not a GUIthat pops up so that a developer can create textual code. Hence an editor must continuouslyserialize and save its state. One developer can then quit the IDE, and another can open thissame file later, at which point the interactive-syntax editor can render itself after deserializingthe saved state.(3)
Interactive visual syntax constructs must compose with textual syntax according to the gram-matical productions.
As already demonstrated, this implies that textual syntax may containinteractive visual syntax and vice versa. In principle, developers should be able to nest visualand textual syntax arbitrarily deep.(4)
The ideal mechanism implements a low-friction model for the definition and use of interactivevisual syntax.
A developer should be able to define and use an interactive syntax extensionin the same file. Indeed, this principle can be further extended to lexical scope. As withtraditional syntax extension mechanisms, a developer should be able to define and use aninteractive-syntax extension within a function, method, class, module, or any other form ofcode block that sets up a lexical scope.(5)
The creator of interactive visual syntax must be able to exploit the entire language, includingthe extension mechanism itself.
If the underlying language permits abstraction over syntax,say like Rust, then a developer must be able to abstract over definitions of interactive visualsyntax; in the same vein, an instance of interactive visual syntax may create new forms ofvisual and textual syntax abstractions.(6)
Interactive visual syntax demands sandboxing for the IDE . The instantiation of interactivevisual syntax into an editor runs code. When developers manipulate editors, code is run again.In an ordinary language, such code may have side effects. Hence, the extension mechanismmust ensure that the code does not adversely affect the functioning of the IDE.(7)
Interactive visual syntax demands sandboxing for code composition . Experience with syntaxextension mechanisms [Flatt 2002] suggests that it is also desirable to isolate the executionof the edit-time code from other phases, say, the compilation phase and the runtime phase.This sandboxing greatly facilitates co-mingling code from different phases.The Racket prototype realizes the design principles, and its use is illustrated next.
Writing interactive-syntax extensions parallels the process of writing traditional syntax extensions.Both traditional and interactive syntax extensions allow compile-time code and run-time code toco-exist, in the same program, the same file, and even the same expression. Interactive syntax addsthe additional notion of an edit-time phase, code that runs while the programmer edits the code.
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 222. Publication date: November 2020.
This section describes interactive-syntax extensions, its parallels to traditional syntax exten-sions [Felleisen et al. 2018], and how these extensions can implement the Tsuro tiles from theprevious section. The key novelty is define-interactive-syntax , a construct for creating newinteractive-syntax forms.
Racket comes with a highly expressive sub-language of macros that enable programmers to extendthe language. To process a program, Racket’s reader creates a syntax tree, stored as a compile-time object. Next the macro expander traverses this tree and discovers Racket’s core forms whilerewriting instances of macros into new syntax sub-trees. In order to realize this rewriting process,the expander partially expands all syntax trees in a module and then adds macro definitions to atable of macro rewriting rules for the second, full expansion pass.Macros are functions from syntax trees to syntax trees. Instead of (define (f x) _ _ _ elided _ _ _ ) a programmer writes (define-syntax (m x) _ _ _ elided _ _ _ ) to define the syntax transformer m . When the expander encounters (m _ _ _ elided _ _ _ ) , it appliesthe transformer to this entire syntax tree. The transformer is expected to return a new syntax tree,and once this happens, the expander starts over with the traversal for this new one. While macrosare often used to produce expressions and definitions, (m _ _ _ elided _ _ _ ) may also expandto define-syntax and thus introduce new syntax definitions. The “on-the-fly” definitions arewhy the macro expander takes the one-and-a-half pass approach, described above, to elaboratingmodules into the Racket core language.A programmer may specify a macro either as a declarative rewriting rule or as a proceduralprocess. For the second variant, the macro may wish to rely on functions that are available atcompile time. In Racket a module may import ordinary libraries for-syntax or it may locallydefine functions to be available at compile time with begin-for-syntax . Thus, (begin-for-syntax(define (g a b c) _ _ _ elided _ _ _ )) makes the ternary function g available to procedural macros.Racketeers speak of the compile-time phase and the run-time phase. Here a phase is a syntacticseparation that determines when code runs and provides a semantic separation of its effects [Flatt2002]. Naturally, function definitions for the compile-time phase may call macros defined for thecompile-time phase, which are in turned defined in the compile-time phase’s compile-time phase.Programmatically a module may thus look as follows: (define-syntax (m x) _ _ _ (f a b) _ _ _) (begin-for-syntax (define (f y z) _ _ _ (k c d e) _ _ _) (define-syntax (k w) _ _ _ (g) _ _ _) (begin-for-syntax (define (g) _ _ _ elided _ _ _))) Phases in Racket programs are nested arbitrarily deep, because programmers appreciate this power.
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 222. Publication date: November 2020. dding Interactive Visual Syntax to Textual Code 222:7
Our technical idea is to support the editing phase , linguistically. The extension to Racket consists oftwo linguistic constructs, analogous to define-syntax and begin-for-syntax : • define-interactive-syntax creates and names an interactive-syntax extension. Roughlyspeaking, an interactive-syntax extension is a specialized graphical user interface defined asa class-like entity. It comes with one method of rendering itself in a graphical context, suchas an IDE; a second for reacting to events (keyboard, mouse, etc.); a third for tracking localstate; and a final one for turning the state into running code. • begin-for-interactive-syntax specifies code that exists for the editing phase in aninteractive-syntax extension. It can only appear at the module top level.That is, an interactive-syntax extension executes during edit time yet denotes compile-time andrun-time code.The code in a begin-for-interactive-syntax block runs when the editor for this module isopened, or its content is modified. Like begin-for-syntax , begin-for-interactive-syntax ismostly used as a mechanism for definitions that are needed to define interactive-syntax.In Tsuro, for example, a trace-player function calculates the path of a player’s token from thestart position to the end position. The Tsuro-board editors use it to calculate where they shoulddraw the tokens after the placement of a tile, meaning the function is needed at edit-time, too:interactive syntax module (begin-for-interactive-syntax(require "Model/player.rkt")) "Model/player.rkt" module (define (trace-player board player) _ _ _ elided _ _ _ ) Requiring an ordinary module at edit time imports its definitions into the desired scope at edit time.
The define-interactive-syntax form bridges the gap between run-time and edit-time code; i.e.,it is analogous to define-syntax . While define-syntax allows run-time code and compile-timecode to interact—the latter referencing and generating the former— define-interactive-syntax connects edit-time code with run-time code, generating the latter from the former.A Racket syntax extension consists of a new grammar production and a translation into existingsyntax. By contrast, interactive-syntax extensions consist of four different pieces:(1) a presentation , meaning a method for rendering its current state, runs at edit-time;(2) an interaction , that is, a method for reacting to direct manipulation actions, runs at edit-time;(3) a semantics , which is compile-time code that generates run-time code; and(4) persistent storage , i.e., a specification of persisted data and its external representation, whichexists at both edit-time and compile-time.Once an interactive-syntax definition exists, a developer may insert an instance into an IDEbuffer by a UI action, such as a mouse click or button press. DrRacket [Findler and PLT 2010],the predominant Racket IDE, implicitly places this editor within a begin-for-interactive-syntax block so that its edit-time code runs continuously during program development. When theprogrammer requests the execution of the module, the extension’s semantics turns the currentstate into run-time code, properly integrated into the lexical scope of its location.
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 222. Publication date: November 2020.
Concretely the define-interactive-syntax form consists of four sub-forms: (define-interactive-syntax name$ base$ (define/public (draw ctx) _ _ _ visually rendering _ _ _ ) (define/public (on-event event) _ _ _ interactions code _ _ _ ) (define-elaborator _ _ _ generating run-time code _ _ _ ) ( _ _ _ persistent data _ _ _ )) The first line specifies the name of the interactive syntax and from which base class it is derived.Like classes, interactive-syntax definitions benefit from implementation inheritance. The draw and on-event methods (lines 2–5) make up the forms’s user interface, code that heavily relieson Racket’s platform-independent graphical-user interfaces [Flatt et al. 2010] and tools for theinteractive development environment [Cooper and Krishnamurthi 2004; Findler and PLT 2010].For specifying the semantics of the new construct, a developer uses the meta-DSL for specifyingtext-based language extensions (lines 6 and 7). The remaining pieces (lines 8 and below) make upthe persistent storage of the syntax form.To support the specification and management of persistent state, our design also supplies acustom language extension. With this extension, a developer can articulate how to react to changesof the data, how to serialize it for future use, and how to deserialize data (if it exists) to resume theuse of an interactive-syntax. The extension is called define-state and its syntax is as follows: (define-state name default state-properties ...) The properties describe these aspects of the state variables. First, they provide traditional get-ter/setter methods. Second, they provide an optional mechanism for users to marshal seeminglyunserializable values into serializable ones. Finally, they describe how long a value must persist.Consider the state for an extension that adds word processors to the language. That state mightinclude the prose the user typed as well as the cursor’s position. The document’s getter/settermethods, as well as its serialization would be fairly pedestrian and the define-state form wouldhandle this automatically. However, persistence is less trivial. The user may expect the text to besaved in the file, but not the current cursor position. However, the user does expect the cursor toremain in place while the document is open.Semantically, an interactive-syntax definition is a class that runs code both during edit time andcompile time. Definitions using define/public are methods and capitalize on object inheritance.The define-interactive-syntax form ensures that draw and on-event are among the definedmethods. The define-state form is mapped to the class fields. Finally, the define-elaborator form acts as a macro that generates run-time code. See section 5 for details.
Implementing a dedicated state, renderer, event handler, and semantics for each interactive-syntaxextensions is somewhat labor intensive. To reduce the work load, our prototype implementationsupports standard GUI creation techniques available in Racket. These techniques fit into threecategories: inheritance, container editors, and graphical editors.Developers use inheritance and mixins [Flatt et al. 2006] to write only absolutely necessary draw and on-event methods. Inheritance works just like in Java. For example, every editor extendsa base$ class, which supplies basic drawing and event handling. Mixin functions abstract over
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 222. Publication date: November 2020. dding Interactive Visual Syntax to Textual Code 222:9 (begin-for-interactive-syntax (define TILE-NODES (list "A" "B" "C" "D" "E" "F" "G" "H"))) (define-interactive-syntax tsuro-tile$ horizontal-block$ (super-new) ;; STATE (define-state pairs (hash) ;; Char Char -> Void ;; EFFECT connects letter to other and vice versa in pairs (define/public (connect! letter other) (send (hash-ref field-gui letter) set-text! other) (send (hash-ref field-gui other) set-text! letter) (set! pairs (hash-set* pairs letter other other letter)) (send picture set-tile! (draw-tile pairs))) ;; VIEW : two horizontally aligned elements (define picture (new tsuro-picture$ [parent this] [connections pairs])) (define fields (new vertical-block$ [parent this])) ;; Char -> Void ;; EFFECT creates a text field as a child of fields (define (add-tsuro-field! letter) ;; TextField Event -> Void ;; EFFECT connect the specified char in f with this letter (define (letter-callback f e) (connect! (send f get-text) letter)) ;; Container -> Void ;; EFFECT create an option field as a child of p (define (option-maker p) (new text-field$ [parent p] [callback letter-callback])) (new labeled-option$ [parent fields] [label (format " ∼ a: " letter)] [option option-maker])) (define field-gui ;; create all text entry fields (for/hash ([a TILE-NODES]) (values a (send (add-tsuro-field! a) get-option)))) ;; CODE GENERATION (define-elaborator this ` ' Fig. 3 . Example Editor for Tsuro Tile
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 222. Publication date: November 2020. inheritance. The define-interactive-syntax-mixin form creates new mixin types. These mixinsare added to an editor’s code by applying them to the editor’s base class.Container editors facilitate editor composition, which almost completely eliminates the need formanually creating methods for the resulting product. The three most predominant container blocksare vertical-block$ for vertical alignment, horizontal-block$ for horizontal alignment, and pasteboard$ for free-flowing editors. Each of these containers work with the widget$ editor type.Each child has a super class that supplies its drawing and event-handling methods.To illustrate these abstraction and composition mechanisms, figure 3 presents the implementationof the Tsuro tile extension. The purpose of this interactive syntax is to permit the programmerto insert a graphical image of the tile where an expression is expected. The construction andmaintenance of this tile demands a capability for connecting and re-connecting the entry points ofthe tile, as well as for displaying the current state of the connections both graphically and as text.Rather than implementing the draw and on-event methods directly, the Tsuro syntax relies oncontainer classes (lines 4, 18–38) to manage layout and events. These labels and fields (lines 22–42)are provided by the standard library and can draw themselves. The tsuro-picture$ editor (lines18–20) works with trace-player to provide drawing and event handling functionality.The field and label sub-editors serve similar purposes; they both render text. The field editor,however, also handles user interaction, while the label one does not. These editors use the text$$ and focus$$ interactive-syntax mixins from the standard library; see figure 4. The text$$ mixin(lines 1–5) handles both the text portion of the editor’s state and drawing directly. The focus$$ mixin (lines 7–10) handles user interaction. Finally, the text-field$ editor (lines 12–14) combinesthe two mixins and applies them to the widget$ base.The code elaborator (lines 44–45) in figure 3 turns pairs , the state of the extension, into a hashtable for the run-time phase, using the traditional syntax extension mechanism.Importantly, developers may compose interactive-syntax extensions. Thus, for example, aninstance of tsuro-tile$ works with the interactive-syntax extension for the full Tsuro board.Each tile in the board is stored directly in the board editor, renders itself in the board’s GUI context,and reacts to events flowing down from this container. ;; Mixin for drawing text in an editor (define-interactive-syntax-mixin text$$ (super-new) (define-state text "") (define/augment (draw dc) ...)) ;; Mixin for basic user interaction (define-interactive-syntax-mixin focus$$ (super-new) (define/augment (on-event event) ...)) ;; A text field widget (define-interactive-syntax text-field$ (focus$$ (text$$ widget$)) (super-new)) Fig. 4 . Field editor using mixins
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 222. Publication date: November 2020. dding Interactive Visual Syntax to Textual Code 222:11
The Tsuro-specific syntax extensions illustrate two aspects of programming with interactive visualextensions. First, the interactive composition of visual and textual code can obviously express ideasbetter than just text (or just pictures). In a sense, this first insight is not surprising. Like English,many natural languages come with the idiom that “a picture is worth a thousand words.” Whatmight surprise readers (as it did the authors) is that there is barely any support for this idea in theworld of programming languages. Perhaps language designers could not imagine how widely thisidea is applicable or how to make this idea work easily.Second, the implementation sketch demonstrates the ease of developing such interactive exten-sions. The effort looks eminently reasonable in the context of a prototype, especially since theessential code of this particular example can be shared between the GUI interface to Tsuro and theunit test suites. A continued development of this prototype is likely to reduce the developmentburden even more, just like research on syntactic extensions has reduced the work of macro writers.Naturally, a single example cannot serve as the basis of an evaluation. A truly proper evaluationof this new language feature must demonstrate its expressive power with a number of distinct cases.Additionally, it must show that the effort remains reasonable across this spectrum of examples. Thissection starts with a list of inspirational sources: numerous text book illustrations of algorithms withdiagrams, pictorial illustrations in standards such as RFCs, and ASCII diagrams in code repositories.The second subsection surveys a range of uses and our implementation of those uses, with anemphasis on where and how interactive syntax can be deployed. The final two subsections presenttwo cases in some depth.
Text books, documentation, source code inspection, and practical experience all motivate the ideaof interactive-syntax extensions.
Tree Algorithms.
Every standard algorithms book and every tree automata monograph [Comon et al.2007; Cormen et al. 2009] comes with many diagrams to describe tree manipulations. Programmersoften include ASCII diagrams of trees in comments to document complex code. These diagramscontain concrete trees and depict abstract tree transformations.
Matrix.
Astute programmers format matrix-manipulation code to reflect literal matrices whenpossible. Mathematical programming books depict matrices as rectangles in otherwise lineartext [Fourer et al. 2002].
File System Data Structures.
Any systems course that covers the inode file representation describesit with box-and-pointer diagrams. Likewise, source code for these data structures frequentlyinclude ASCII sketches of these diagrams.
TCP.
RFC-793 [Postel 1981] for TCP lays out the format of messages via a table-shaped diagram.Each row represents a 32-bit word that is split into four 8-bit octets.
Pictures as Bindings.
Many visual programming environments, such as Game Maker [Overmars2004], allow developers to lay out their programs as actors placed on a spatial grid. Actors aredepicted as pictorial avatars and the code defining each actor’s behavior refers to other actors usingavatars. In other words, pictures act as the variable names referencing objects in this environment. https://git.musl-libc.org/cgit/musl/tree/src/search/tsearch.c?id=v1.1.21 Name Elements LOCTree Algorithms Data Literal, Pattern Matching, Templates 353Matrix Data Literal, Template 175File System Data Data Literal, Other Binding 178TCP Parser Data Literal, Pattern Matching, Templates 98Pictures As Bindings Other Binding 88Video Editor Data Literal, Template 80Circuit Editor Data Literal 307Tsuro Data Literal, Template 408Form Builder Data Literal, Template, Meta Binding 119
Fig. 5 . Attributes of the Worked Language Extensions
Video Editors.
Video editing is predominantly done via non-linear, graphical editors. Such purelygraphical editors are prone to force people to perform manually repetitive tasks.
Circuits.
Circuits are naturally described graphically. Reviewers might be familiar with Tikz andCircuitTikz, two LaTeX libraries for drawing diagrams and specifically circuit diagrams. Codingdiagrams in these languages is rather painful, though; manipulating them afterwards to put theminto the proper place within a paper can also pose challenges.Electrical engineers code circuits in the domain-specific SPICE [Vogt et al. 2019] simulationlanguage or hardware description languages such as Xilinx ISE. While both come with tools to editcircuits graphically, engineers cannot mix and match textual and graphical part definitions.
We have implemented the examples from the previous section with interactive syntax. Doing soyields easily readable code and several insights on the expressive power of mixing visual and textualsyntax. Here we present a classification of the linguistic roles that these extensions play withincode. Figure 5 provides a concise overview. The first column lists the name of the example, thesecond the role that interactive syntax plays. The third column reports the number of lines of codeneeded for these extensions.
Data Literal.
The simplest role is that of a data literal. In this role, developers interact with thesyntax only to enter plain textual data; the elaborator tends to translate these editors into structuresor objects. As the Tsuro examples in the introduction point out, data-literal forms of interactivesyntax can be replaced by a lot of text—at the cost of reduced readability.
Template.
The template’s role generalizes data literals. Instead of entering plain data into an editor,a developer inserts code into text fields of these instances. The Tsuro board in the introductionand the tree in figure 7 are examples of such templates. The templates build a board and tree,respectively, using pattern variables embedded in an editor.
Pattern Matching.
Languages such as Scala emphasize pattern matching, and interactive syntax cangreatly enhance the message that a pattern expresses. In this context, a developer fills an editorwith pattern variables, and the code generator synthesizes a pattern from the visual parts and thesepattern variables. In this role, pattern-matching editors serve as binding constructs.
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 222. Publication date: November 2020. dding Interactive Visual Syntax to Textual Code 222:13 ; Balance (- black) (= red) ; ; / -z- | -z- | -x- | -x- \ ; | / \ | / \ | / \ | / \ | =y= ; | =y= D | =x= D | A =z= | A =y= | / \ ; | / \ | / \ | / \ | / \ | --> -x- -z- ; | =x= C | A =y= | =y= D | B =z= | /| |\ ; | / \ | / \ | / \ | / \ | A B C D ; \ A B | B C | B C | C D / (define/match (balance t) [(or (tree z 'black (tree y 'red (tree x 'red A B) C) D) (tree z 'black (tree x 'red A (tree y 'red B C)) D) (tree x 'black A (tree z 'red (tree y 'red B C) D)) (tree x 'black A (tree y 'red B (tree z 'red C D)))) (tree y 'red (tree x 'black A B) (tree z 'black C D))] [else t]) Fig. 6 . A Textual Balance function for a Red-Black Tree
Meta Binding.
Since syntax extension is a form of meta programming, interactive syntax naturallyplays a meta-programming role, too. We refer to this role as meta-binding in figure 5. Here editorsare used to construct new types of syntax, and because the prototype is in Racket, they can generateboth graphical and textual syntax extensions. Section 4.4 demonstrates this idea with form builders.
Other Binding.
Finally, editors can play the role of a binding form. This role allows multiple editorsto “talk” to each other. The inode data structure supplies an example of this kind.
When programmers explain tree algorithms, they frequently describe the essential ideas withdiagrams. Often these diagrams make it into the library documentation and the programmerwho maintains the code has to go back and forth between the code and the diagrams in thedocumentation. A poor man’s fix is to render such diagrams as ASCII art. The balancing algorithm for red-black trees [Bayer 1972] illustrates this kind of work particularlywell. Figure 6 shows a code snippet from a tree-manipulation library in Racket. The snippet depictsa function for balancing red-black trees using pattern matching. The comment block (lines 1–9)makes up the internal ASCII-art documentation of the functionality, while the code itself (lines10–16) is written with Racket’s expressive pattern-matching construct.An interactive-syntax extension empowers the developers to express the algorithm directly as adiagram, which guarantees that the diagram and the code are always in sync. The key point is thatinteractive syntax can show up in the pattern part of a match expression as well as in the template part, both situated within ordinary program text.A look at figure 7 makes this point for the red-black tree balance algorithm. The match expressionconsists of two clauses, but only the first one matters for the current discussion.The or pattern combines several sub-patterns; if any one of them matches, the pattern matchesthe given tree t . This example uses four sub-patterns, each expressed with editors; each representsone of the four possible situations in which a balance must take place. The situation should remind https://blog.regehr.org/archives/1653Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 222. Publication date: November 2020. (define (balance t)(match t[(or ); => ][else t])) Fig. 7 . Visual Red-Black Tree Balance the reader of the diagram in Okasaki’s functional implementation [Okasaki 1999], which uses thesame four trees on the second page of his paper. The four sub-patterns name nodes— x , y , and z —and subtrees— A , B , C , and D —with consistent sets of pattern variables.The template—on the second line—refers to these pieces of the pattern. It is also an editor andshows how the nodes and sub-trees are put into a different position. The resulting tree is clearlybalanced relative to the matched subtrees.In sum, the code consists of four input patterns that map to the same output pattern. Anyprogrammer who opens this file in the future will immediately understand the relationship betweenthe input tree shapes and the output tree—plus the connection to the published paper. The presented examples generate only run-time code, and they exclusively focus on patterns anddata structures. Interactive-syntax extensions, however, can also generate compile-time and evenedit-time code. This means that interactive-syntax extensions can impose both domain-specificvalidation requirements and provide the tools to enforce them.As an example, consider embedding table-like forms into code. To make this idea truly concrete,imagine managing a large introductory course, with several hundred students and a staff of a fewdozen teaching assistants. The course coordinator must log information for each student and staffmember. Furthermore, each role requires different information, e.g., each student gets a grade, whilestaff members have grading assignments. To manage all this information, the course coordinatorcan use interactive syntax to create information forms. Rather than making forms directly, thecoordinator creates a form builder with interactive syntax to generate each type of form. Moregenerally, in this scenario interactive syntax is used to make new types of interactive-syntax.
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 222. Publication date: November 2020. dding Interactive Visual Syntax to Textual Code 222:15
Normally, forms act like the Tsuro editor and elaborate to tables containing their contents. Thesetables can act as dictionaries used directly in source programs, or as SQL statements that insertdata into a form-specific database.Programmers can add additional constraints to fields in those forms. Forms that do not meet theseconstraints are considered syntax errors. For example, here is a small program with an incorrectlyfilled grade form: > (dict-ref 'score)
Bad syntax in student-form$: Invalid "Student ID", expected an: id?, got: "Bob Smith"
The “Student ID” field expects an identification number, but the user put in a name rather than anID. Because forms may be created with checks that ensure correct data is entered, feedback happensduring edit time. As an aside, note the “Total Score” pseudo-field in this form, which displays aresult, but is not a text field. In addition to traditional (text) fields, forms can contain any arbitraryinteractive-syntax extension. For example, the “Total Score” area sums the result of "Problem 1" and "Problem 2" . Rather than creating form editors textually, the coordinator canuse a graphical form builder from a library of interactive syntaxconstructs to create new types of forms. The editor on the left is anexample of this meta-form. Using an IDE action, the coordinatorinstantiated this form editor and used the text field labeled define-form to give a name to all its instances. As displayed, this meta-editor already specifies a list of fields: "Student ID" , "Total Score" , "Problem 1" , "Problem1 Comments" , "Problem 2" , and "Problem 2 Comments" . Each field comes with a button,with which the coordinator can remove the field from the form. Additionally, each field containsa reference to either an editor or a predicate. If an editor is provided, say total$ , than the formuses it instead of the default text field. This enables custom sub-editors to display informationbased on other fields. Likewise, if a predicate is provided, it handles the validation of entries in thecorresponding text field.At the bottom is a text field plus a button; it permits the coordinator to add a new field to student-assign2$ . Indeed, once a field is deleted or added, every instance of this student-form% is updated to match its meta-definition. For example, if the coordinator were to add a "Problem 3" label, a correspondingly labeled, blank text field would show up in both of the editors in the list onthe next line.In addition to defining new form types in the program, the define-interactive-syntax meta-editor introduces a companion SQL table to the program. The table’s schema is specific to theintroduced form type, which allows users to add form instances into the table. The student-assign2$ table, for example, expects five fields: "Student ID" , "Problem 1" , "Problem 1 Com-ments" , "Problem 2" , and "Problem 2 Comments" . The "Total Score" field is absent becausethe total$ editor does not provide an SQL schema.Figure 8 presents the implementation of the form builder syntax. This interactive-syntax extensionis like a graphical GUI editor in that it generates another interactive-syntax extension. The edit-time code, such as the state and view, is similar to the Tsuro editor and hence of little interest. Itselaborator contains compile-time code that generates more code that runs at compile time, runtime, and edit time. First, at compile time it generates identifiers used throughout the elaborator Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 222. Publication date: November 2020. ` (begin(query-exec db (create-table name/sql Edit Time Compile Time Run Time
Fig. 8 . Example Editor for a Form Builder (lines 4-9, compile time). Second, the elaborator defines an SQL schema for form instances at runtime (lines 11-12, run time). Finally it synthesizes the edit-time code for the form itself (lines 13-17,edit time).The generated interactive-syntax extension looks deceivingly simple: • The new interactive-syntax class, named name as specified in the state of form-builder$ ,inherits from the table-base$ superclass (lines 13). This use of inheritance illustrates acommon use of generated interactive-syntax definitions. Rather than putting the entireimplementation for an editor in the template of the elaborator, we factor most of it outinto an external class. Besides separating common code from elaborator-specific one, italso generates significantly smaller code than a full-fledged implementation class, whichsignificantly improves compile-time and edit-time performance. The table-base$ class itselfis rather pedestrian GUI code. • The elaborator of the generated interactive-syntax simply expands to a dictionary with theform’s field names and values.While generating interactive-syntax extensions from interactive-syntax extensions sounds compli-cated, the decades-old precedent of syntax extension design should help the reader understand theincredible power in such meta-capabilities.In short, interactive visual syntax can be used to introduce new interactive visual syntax. Herea general form-generating interactive syntax construct is instantiated for creating assignmentspecific forms. This meta-editor adds another editor type to the program. At some point, of course,the process has to switch back to text, which is where the form extension itself is defined.
The implementation of an interactive-syntax extension mechanism poses four challenges: editorsyntax, edit-time semantics, editor elaboration, and IDE integration. This section first describes a
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 222. Publication date: November 2020. dding Interactive Visual Syntax to Textual Code 222:17 general implementation approach that realizes our design desiderata and suggests abstract solutionsto these four challenges. It then explains how the application of this approach to Racket yields areasonably robust prototype including a connection to the DrRacket IDE. We conjecture that, withsignificantly more labor, this approach would also work for other languages with macro systemssuch as Clojure, Elixir, Julia, Rust, Scala, and even C++, as well as alternative IDEs. As mentioned above, four efforts are required to implement support for interactive-syntax extensionsto a programming language. First, the language’s syntax must be augmented with a textual editor form to house these extensions. Second, the language’s semantics must be extended to support theexecution of code at edit time. Third, the language must include some mechanism for interactive-syntax elaboration. Finally, it demands the construction of plugins for graphical IDEs so thatthey use these extended semantics to interpret the textual editor form as a visual and interactivegraphical user interface. Any language tool chain that can accommodate these four componentscan also support interactive-syntax extensions.
The Editor Form.
For a language to support interactive-syntax extensions, implementers of thatlanguage must extend its syntax to include a textual representation for editors. This textual rep-resentation is how an editor is stored on the developer’s file system. Additionally, using a purelytextual representation allows developers to edit their code in any environment, such as plain texteditors, not just interactive-syntax specific ones.This textual extension must contain three parts: an editor’s state, a pointer to a means of convert-ing that state into a graphical editor, and a pointer to a means of converting that state into elaboratedcode. It can be added either to the language’s core implementation or as an external languageextension. Many common languages, such as C, already allow for small language extensions likethis. Languages that don’t, such as Java, may still have tools that can emulate language extensionsto a limited degree.As an example, a form builder from the previous section may have this textual representation: $editor {binding: ["lib/form-builder.rkt", "form-builder$"],state: {name: "person$",keys: ["Name" ,"Age"]}}
Here, the binding tag refers to the module and name of an interactive-syntax binding, which servesthe dual purpose of converting state into a graphical editor and into elaborated code. The rest ofthe syntax contains the editor’s state as a hash table. Due to this design choice, a plain text editor,such as Emacs or Vim, simply displays this text when a developer opens a module that containsinteractive syntax. IDEs with support for interactive syntax can display the editors as mini GUIsembedded in program text, using the information stored in the binding field.The interpretation of editor syntax is analogous to closures. Like a closure, an editor combinesa code pointer and its current state into a new kind of value. The code pointer, found in the binding keyword in the example above, refers to the define-interactive-syntax definitionthat the editor instantiates. The state component records those aspects of the editor’s state that https://github.com/videolang/interactive-syntaxProc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 222. Publication date: November 2020. this definition specifies as persistent. Together these two pieces suffice to fully re-instantiate theeditor as a graphical element in IDEs that can interpret these “closures” at edit time. Edit-time semantics.
The second component required for interactive-syntax extensions is semanticsfor edit-time code. This semantics is distinct from already existing semantics for compile-time andrun-time code. Furthermore, it must be possible to interleave edit-time code with other forms ofcode in a program’s body and to formulate such edit-time code in the same language as run-timecode or compile-time code. Finally, while not strictly necessary, the language should enforce a levelof isolation on effects among these phases to allow a clean separate compilation of modules in thepresence of co-mingled elements [Flatt 2002].Two syntactic forms must link run-time code and edit-time code. The first form simply allowsedit-time code to be spliced into run time code. The second form must create bindings in run-timecode, that refer to extensions defined at edit-time.As with the syntax extensions, this semantics can be either built into a language’s core, or itcan be added with an external preprocessor. If done externally, only one preprocessor is needed tosupport all interactive-syntax extensions.
Editor Elaboration.
The language also needs an expander to turn each editor form into its elaboratedcode. This expander must additionally be able to evaluate user code including the definition of newtypes of interactive-syntax. An expander can use its host language’s normal meta-programmingfacilities. However, these facilities must implement the one-and-a-half pass approach discussed insection 3.Expanding an editor into code happens in five steps. First, the expander must recognize editorsyntax. Second, it must deserialize an editor’s state. Third, the expander must locate the editor’selaborator. Fourth, the expander invokes the elaborator with the editor’s state. Finally, it splices thegenerated code into the program body.
Cooperating With an IDE.
An IDE must cooperate with the language to run the edit-time code ofinteractive-syntax extensions and to connect this code to its own graphical context. An implemen-tation can either modify the IDE directly or via a plugin. If an IDE supports interactive-syntaxextensions through a plugin, only one plugin is required to support all extensions.Enabling an IDE to interpret editors demands two kinds of extensions. First, the IDE mustrecognize interactive-syntax extensions so that it supports UI actions for the insertion of instancesinto code and for updating existing editors if their underlying definition changes. Second, the IDEmust supply a graphical context to editors so that they can render themselves and receive relevantUI events.The code that comprises an interactive-syntax extension is fundamentally user code. As such,the IDE must sandbox these extensions. This sandbox is similar to the traditional environment thatan operating system may use or a web browser for its tabs. For example, a sandbox may preventan extension from creating or modifying files without some file system permission. This sandboxenvironment must also allow an editor to gracefully fail, reverting back to some default renderingor even a variant of its textual representation.
As a proof of concept, we have constructed a prototype of interactive-syntax extensions for theRacket language and DrRacket IDE [Findler et al. 2002].
The Editor Form.
First, an editor’s textual representation is composed of binding information andstate syntax. Concretely, an editor is represented in text like the form in the proceeding subsection.
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 222. Publication date: November 2020. dding Interactive Visual Syntax to Textual Code 222:19
Making this form valid syntax requires a change to Racket’s reader. The extended reader generatesa valid syntax object with a known (macro) interpretation. Racket’s reader is extensible, meaningthat interactive-syntax can, in principle, work with any Racket-based language [Felleisen et al.2018] that also uses Racket’s reader extensions.
Edit-time semantics.
Second, while Racket already supports a hierarchy of compile-time phases (forsyntax extensions that generate syntax extensions), it has no mechanism for adding a new phase.Since we wish to demonstrate that interactive-syntax extensions can be added to a language withoutchanging the underlying virtual machine or interpreter, the prototype employs a surprisingly robustwork-around. We conjecture that such work-arounds exist for other languages too, though it isalso likely that in many cases, an implementer would have to explicitly add an edit phase.Interactive-syntax extensions are implemented as syntax extensions. They elaborate constructssuch as define-interactive-syntax , editors, and begin-for-interactive-syntax into a mixof further (plain) syntax extensions and submodules.Figure 9 shows an example of such an elaboration. The module in the top half consists of threepieces: the definition of an interactive-syntax extension, an editor of this extension (the denotes“locally defined,” the simple$ points to the definition; the editor has no state), and a simplisticedit-time test of this definition. The module at the bottom is (approximately) the transformation ofthe module at the top into Racket code.In the expanded program, all edit-time code is placed into a single edit submodule [Flatt 2013]. This new submodule is inserted at the bottom of the expansion module. The definition of theinteractive-syntax extension is separated into two pieces (as indicated with code highlightingand indicies): the elaborator called simple$:elaborator , which exists at compile time, and theinteractive-syntax class called simple$ , which exists at edit time. Recall that the elaborator trans-lates the state of an editor into run-time code; the interactive-syntax class inherits and implementsthe edit-time interaction functionality for the syntax extension. As for the textual editor form, itsreference to the simple$ interactive-syntax extension is refined to a reference to the elaborator;for edit time execution, the IDE plugin performs a separate name resolution. Finally, the test codein the begin-for-interactive-syntax block is also moved into the edit submodule.Placing the edit-time code into a separate submodule permits the runtime system to distinguishbetween editor-specific code and general-purpose program code. In particular, it ensures that theruntime system can execute the editor portion of an interactive-syntax extension independentlyof its host module. Indeed, the runtime system realizes this goal by merely requiring the edit submodule and thus obtaining the provided simple$ interactive-syntax class, which implementsthe GUI interactions. By contrast, the generated run-time code of an editor must remain subject tothe host module’s scope.
Editor Elaboration.
Third, editor elaboration is a straightforward use of Racket’s macro expander.The Racket parser converts the form in the source into traditional Racket syntaxcontaining a macro. The macro (figure 10) finds the elaborator (lines 3–5),deserializes the state (lines 6–7), and expands to the elaborator with the deserialized state (line 8).From here, the macro expander places the residual code back into the program body.
Cooperating With an IDE.
Finally, the prototype exploits DrRacket’s plug-in API for the eventhandling [Findler and PLT 2010] and its Cairo-based drawing API for editor rendering. A speciallydesigned plug-in connects interactive-syntax extensions to the IDE. It inserts menu entries thatprogrammers can use to instantiate interactive-syntax extensions and insert them into specificpoints into code (figure 11). The plug-in also assists with saving and retrieving modules that contain Appendix A contains a brief introduction to submodules in Racket.Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 222. Publication date: November 2020. (define-elaborate this )) elaborates to )) ))(require editor/test)(test-window (new simple$))) Fig. 9 . Elaboration of an Interactive-Syntax Extension editors. When a developer saves a file, the plugin serializes all instances of interactive syntaxextensions into blocks; conversely, when a developer opens such a module, it usesthe language’s parser to scan the file for blocks and informs the IDE about them.Technically, the prototype relies on Racket’s GUI toolbox and sandboxes mechanism [Flatt et al.2010]. Specifically, the Racket evaluator provides controlled channels for sandboxed namespacesto connect to the rest of the Racket runtime system. The Racket GUI toolbox already supportsgraphics within textual programs via the snip API. DrRacket supplies a drawing context to snipsand passes user events to them. However, snips are extra-linguistic. They are thus IDE-specific
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 222. Publication date: November 2020. dding Interactive Visual Syntax to Textual Code 222:21
Fig. 10 . Implementation of the macroFig. 11 . Inserting an editor inside of DrRacket elements and not elements of the language the programmer edits. They were DrRacket’s firstattempt at mixing graphical and textual programming, but made any file that used them unreadableoutside of DrRacket. The prototype bridges the gap between these two GUI elements so that editorsremain language elements yet connect to the IDE smoothly.The prototype accommodates failures with a simple fallback editorGUI. If a developer were to inject a typo into an viaEmacs or were to move the file that contains an interactive-syntaxextension, the prototype does not crash. Instead it hands control toa form editor, which displays a small default GUI whose fields showthe editor’s binding and state information shown on the right. Herethe form builder is found in "lib/form-builder.rkt" , which provides form-builder$ as anidentifier. The editor has two state fields: name and keys . Clicking on the button tells theruntime system to make another attempt at re-initializing the editor with these values.
Three years ago at ICFP in Oxford, Andersen et al. [2017] presented the Video language. Videomixes text and editable video clips so that users can easily script productions, both textually andinteractively. While this work on Video is a bespoke production, it raises the question of how togeneralize this idea of graphical elements embedded in scripts and programs. Summarily speaking,this work empowers developers to • create their own interactive-syntax extensions • abstract with a plain syntax extension over interactive-syntax extensions • abstract with an interactive-syntax extension over interactive-syntax extensions Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 222. Publication date: November 2020.
In other words, interactive-syntax extensions are a proper part of the programming language incontrast to IDE-specific plugins, such as Video’s.In addition to drawing inspiration from the work of Andersen et al., this project also draws onideas from research on edit time code, programming systems, and non-standard forms of editing.
Two rather distinct pieces of work combine edit-time computation with a form of programming.The first is due to Erdweg in the context of the Spoofax language workbench project and is trulyabout general-purpose programming languages. The second is Microsoft’s mixing of textual andgraphical “programs” in the productivity suite.Like Racket, Spoofax [Kats and Visser 2010] is a framework for developing programming lan-guages. Erdweg et al. [2011] recognizes that when developers grow programming languages, theywould also like to grow the IDE support. For example, a new language feature may require a newstatic analysis or refactoring transformations, and these tools should cooperate with the language’sIDE. They therefore propose a framework for creating edit-time libraries. In essence such librarieswould connect the language implementation with the IDE and specifically the IDE tool suite. LikeVideo, these libraries are IDE plugins and thus extra-linguistic.Microsoft Office plugins, called VSTO Add-ins [Microsoft 2019], allow authors to create newtypes of documents and embed them into other documents. A developer might make a musictype-setting editor, which another might use to put music notation into a PowerPoint presentation.Even though this tool set lives in the .NET framework, it is an extra-linguistic idea and does notallow developers to build programming abstractions.
Several programming systems have enabled a mixture of some graphical and textual programmingfor decades. The four most prominent examples are Boxer, Hypercard, Scratch, and Smalltalk.Boxer [diSessa and Abelson 1986] allows developers to embed GUI elements within other GUIelements (“boxing”), to name such GUI elements, and to refer to these names in program code. Thatis, “programs” consist of graphical renderings of GUI objects and program text (inside the boxes).For example, a Boxer programmer could create a box that contains an image of a Tsuro tile, nameit, and refer to this name in a unit test in a surrounding box. Boxer does not satisfy any of the otherdesiderata in section 2. In particular, it has poor support for creating new abstractions with regardto the GUI elements.Scratch [Resnick et al. 2009], also an MIT product, is a fully graphical language system, withwide applications in education. In Scratch, users write their programs by snapping graphical blockstogether. These blocks resemble puzzle pieces and snapping them together creates syntacticallyvalid programs. Scratch offers limited, but growing [Harvey and Mönig 2010], capabilities for aprogrammer to make new block types.Hypercard [Goodman 1988] gives users a graphical interface to make interactive documents.Authors have used hypercard to create everything from user interfaces to adventure games. Whilehypercard has been used in a wide variety of domains, it is not a general-purpose language.Smalltalk [Bergel et al. 2013; Goldberg and Robson 1983; Ingalls et al. 2008; Klokmose et al.2015; Rädle et al. 2017] supports direct manipulation of GUI objects, often called live programming.Rather than separating code from objects, Smalltalk programs exist in a shared environment, theMorphic [Maloney et al. 2001] user interface. Programmers can visualize GUI objects, inspect andmodify their code component, and re-connect them to the program. No Smalltalk systems trulyaccommodate general-purpose graphical-oriented programming as a primary mode, however.
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 222. Publication date: November 2020. dding Interactive Visual Syntax to Textual Code 222:23
GRAIL [Ellis et al. 1969a,b] is possibly one of the oldest examples of graphical syntax. It allowsusers to create and program with graphical flow diagrams. Despite the apparent limitations of thisdomain, GRAIL was powerful enough to be implemented using itself.Notebooks [Ashkenas 2019; Bernardin et al. 2012; Perez and Granger 2007; Wolfram 1988] andWebstrates [Klokmose et al. 2015; Rädle et al. 2017] are essentially a modern reincarnation of thismodel, except that they use a read-eval-print loop approach to object manipulation rather thanthe GUI-based one, made so attractive by the Morphic framework. These systems do not permitdomain-specific syntax extensions.
Bidirectional editors attempt to present two editable views for a program that developers canmanipulate in lockstep. Sketch-n-Sketch [Chugh et al. 2016; Hempel et al. 2018], for example,allows programmers to create SVG-like pictures both programmatically with text, and by directlymanipulating the picture. Another example is Dreamweaver [Adobe 2019], which allows authorsto create web pages directly, and drop down to HTML when needed. Changes made in one viewpropagate back to the other, keeping them in sync. We conjecture that an interactive-syntaxmechanism like ours could be used to implement such a bidirectional editing system. Likewise, abidirectional editing capability would improve the process of creating interactive-syntax extensions.Wizards and code completion tools, such as Graphite [Omar et al. 2012], preform this task inone direction. A small graphical UI can generate textual code for a programmer. However, oncefinished, the programmer cannot return to the graphical UI from text.Projectional editing aims to give programmers the ability to edit programs visually. Indeed, inthis world, there are no programs per se, only graphically presented abstract syntax trees (AST),which a developer can edit and manipulate. The system can then render the ASTs as conventionalprogram text. The most well-known system is MPS [Pech et al. 2013; Voelter and Lisson 2014].It has been used to create large non-textual programming systems [Voelter et al. 2012]. Unlikeinteractive-syntax extensions, projectional editors must be modified in their host editors and alwaysdemand separated edit-time and run-time modules. Such a separation means all editors must beattached to a program project, they cannot be constructed locally within a file. It therefore is ratherdifficult to abstract over them.Barista [Ko and Myers 2006] is a framework that lets programmers mix textual and visualprograms. The graphical extensions, however, are tied to the Barista framework, rather than theprograms themselves. Like MPS, Barista saves the ASTs for a program, rather than the raw text.The Hazel project and Livelits [Omar et al. 2019] are also closely related to interactive-syntaxextensions. Like editors, the Livelits proposal aims to let programmers embed graphical syntaxinto their code. In contrast to interactive-syntax extensions, which use phases to support editorinstantiation and manipulation, the proposed Livelits will employ typed-hole editing. Finally, whilethe Livelits proposal is just a two-page blueprint, we conjecture that these constructs will not bedeployed in the same range of linguistic contexts as interactive-syntax extensions (see section 4).
The prototype falls short of the plan laid out in section 2, though the shortcomings are of anon-essential technical nature, not principled ones:(1) The prototype partially re-uses Racket’s GUI library in the context of interactive-syntaxdefinitions. At the moment, the prototype relies on a Racket-coded GUI tailored to the Intentional Software [Simonyi et al. 2006] has similar goals, but there is almost no concrete information in the literatureabout this project. Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 222. Publication date: November 2020. interactive-syntax system, meaning developers cannot use all GUIs for both syntax extensionsand the application itself. The shortcoming is due to two intertwined technical reasons: thesetup of Racket’s GUI library and DrRacket’s use of an editor canvas, which cannot embedcontrols and other window areas.(2) The prototype does not validate the usability of the construction across different visual IDEs.It does accommodate the use of DrRacket and plain text editors such as Emacs or Vim. Whilethe textual rendering of editors is syntactically constrained, a developer who prefers a texteditor can still work on code and even read embedded interactive-syntax. Due to the designchoice of relying on a Racket-based GUI for editors, though, interactive syntax will work inany IDE that can run Racket code and grant access to a drawing context such as a canvas.For details on how to engineer a general solution, see section 5.1.(3) A minor shortcoming concerns editors that contain text fields into which developers entercode. In the current prototype, these text fields are just widgets that permit plain text editing.With some amount of labor, an interactive-syntax extension could use a miniature version ofDrRacket so that developers would not just edit plain text, but code, in these places.(4) Finally, the use of Racket’s sub-modules to implement an edit-time phase falls short of thelanguage’s standard meta-programming ideals. While they mostly work correctly for mappingeditors to code, the solution exhibits hygiene problems in some corner cases. Furthermore,while some meta-programming extensions in conventional languages, e.g., Rust, do imple-ment hygienic expansion, others completely fail in this regard, e.g., Scala, which may causeadditional problems in adapting this idea to different language contexts.All of these limitations naturally point to future investigations.Besides these technical investigations, the idea also demands a user-facing evaluation in additionto the expressiveness evaluation presented in section 4. Here are some questions for such a study: • How quickly do developers identify situations where the use of interactive syntax mightbenefit their successors? • How much more difficult is it to articulate code as interactive syntax than text? • Is it easier to comprehend code formulated with interactive syntax instead of text?Answers may simultaneously confirm the conjecture behind the design of interactive syntax andpoint to technical problems in existing systems. The authors will attempt to answer these questionafter hardening the prototype into an easily usable system.
Linear text is the most widely embraced means for writing down programs. But, we also know thatin many contexts a picture is worth a thousand words. Developers know this, which is why ASCIIdiagrams accompany many programs in comments and why type-set documentation comes withelaborate diagrams and graphics. Developers and their support staff create these comments anddocuments because they accept the idea that code is a message to some future programmer whowill have to understand and modify it.If we wish to combine the productivity of text-oriented programming with the power of pictures,we must extend our textual programming languages with graphical syntax. A fixed set of graphicalsyntaxes or static images do not suffice, however. We must equip developers with the expressivepower to create interactive graphics for the problems that they are working on and integrate thesegraphical pieces of program directly into the code. Concisely put, turning comments into executablecode is the only way to keep comments in sync with code.When a developer invests energy into interactive GUI code, this effort must pay off. Hencea developer should be able to exploit elements of the user-interface code in interactive-syntax
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 222. Publication date: November 2020. dding Interactive Visual Syntax to Textual Code 222:25 extensions. Conversely, any investment into GUI elements for an interactive-syntax extension mustcarry over to the actual user-interface code for a software system.Finally, good developers build reusable abstractions. In this spirit, an interactive-syntax extensionmechanism must come with the power to abstract over interactive-syntax extensions with aninteractive-syntax extension. If this is available to developers, they may soon offer completelibraries of interactive-syntax building blocks.Our paper presents the design, implementation, and evaluation of the first interactive-syntaxextensions mechanism that mostly satisfies all of these criteria. While the implementation is aprototype, it is robust enough to demonstrate the broad applicability of the idea with examples fromalgorithms, compilers, file systems, networking, as well as some narrow domains such as circuitsimulation and game program development. In terms of linguistics, the prototype can alreadyaccommodate interactive syntax for visual data objects, complex patterns, sophisticated templates,and meta forms. We consider it a promising step towards a true synthesis of text and “moving”pictures.
ACKNOWLEDGMENTS
This research was partially supported by NSF grants 1823244 and 20050550. We also thank BenjaminChung, Benjamin Greenman, Elizabeth Grimm, Jason Hemann, Shriram Krishnamurthi, and Ming-Ho Yee for useful discussions and feedback on early drafts of this paper.
A SUBMODULES IN RACKET, A BRIEF REVIEW
Racket’s notion of submodule [Flatt 2013] is the key to the implementation of an edit phase. InRacket, submodules exist to organize a single file-size module into separate entities. A submoduledoes not get evaluated unless it is explicitly required, which is possible locally as well as fromanother file.The introduction of submodules was motivated by two major desires: to designate some part ofthe code as main and to include exemplary unit tests right next to a function definition. Becausesubmodules are separated from the surrounding module, adding such tests has no impact on thesize or running time of the main module itself.Consider this example: ;; inc : [Box Integer] -> Void ;; increment the content of the given box by 1 (module+ test (let ([x (box 0)]) (check-equal? (unbox x) 0) (inc x) (check-equal? (unbox x) 1))) (define (inc counter) (set-box! counter (add1 (unbox counter)))) The unit-test module creates a fresh counter box, initialized to , increments it, and checks thevalue again. The programmer can evaluate these unit tests explicitly from another module or witha command-line tool: [linux] $ raco test inc.rktraco test: (submod "inc.rkt" test) Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 222. Publication date: November 2020.
When inc is imported from another module, the unit tests are neither loaded nor run.
REFERENCES
Adobe. Adobe Dreamweaver CC Help. Retrieved May, 2020, 2019. https://helpx.adobe.com/pdf/dreamweaver_reference.pdfLeif Andersen, Stephen Chang, and Matthias Felleisen. Super 8 Languages for Making Movies (FunctionalPearl).
Proceedings of the ACM on Programming Languages
Acta Informatica
Proc. Programming Languages Design and Implementation , pp. 341–354, 2016. https://doi.org/10.1145/2980983.2908103Hubert Comon, Max Dauchet, Remi Gilleron, Florent Jacquemard, Denis Lugiez, Christof Löding, SophieTison, and Marc Tommasi. Tree Automata Techniques and Applications. 2007. http://tata.gforge.inria.fr/Gregory Cooper and Shriram Krishnamurthi. Embedding Dynamic Dataflow in a Call-by-Value Language. In
Proc. European Symposium on Programming , pp. 294–308, 2004. https://doi.org/10.1007/11693024_20Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and Clifford Stein.
Introduction to Algorithms,Third Edition . MIT Press, 2009.Andrea A. diSessa and Harold Abelson. Boxer: A Reconstructible Computational Medium.
Communications ofthe ACM
Proc. International Conference on Functional Programming ,pp. 1–12, 2006. https://doi.org/10.1145/1160074.1159805Andrew D. Eisenberg and Gregor Kiczales. Expressive Programs Through Presentation Extension. In
Proc.International Conference on Aspect-Oriented Software Development
Proc. Generative Programming and ComponentEngineering , pp. 167–176, 2011. https://doi.org/10.1145/2189751.2047891Matthias Felleisen, Robert Bruce Findler, Matthew Flatt, Shriram Krishnamurthi, Eli Barzilay, Jay McCarthy,and Sam Tobin-Hochstadt. A Programmable Programming Language.
Communications of the ACM
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 222. Publication date: November 2020. dding Interactive Visual Syntax to Textual Code 222:27
Robert Bruce Findler, John Clements, Cormac Flanagan, Matthew Flatt, Shriram Krishnamurthi, Paul Steckler,and Matthias Felleisen. DrScheme: A Programming Environment for Scheme.
Journal of FunctionalProgramming
Proc. International Conference onFunctional Programming , pp. 72–83, 2002.Matthew Flatt. Submodules in Racket, You Want it When, Again? In
Proc. Generative Programming: Concepts& Experiences , pp. 13–22, 2013. https://doi.org/10.1145/2517208.2517211Matthew Flatt, Robert Bruce Findler, and John Clements. GUI: Racket Graphics Toolkit. PLT Design Inc.,PLT-TR-2010-3, 2010. https://racket-lang.org/tr3/Matthew Flatt, Robert Bruce Findler, and Matthias Felleisen. Scheme with Classes, Mixins, and Traits. In
Proc.Asian Symposium Programming Languages and Systems , pp. 270–289, 2006.Matthew Flatt and PLT. Reference: Racket. PLT Design Inc., PLT-TR-2010-1, 2010. https://racket-lang.org/tr1/Robert Fourer, David M. Gay, and Brian W. Kernighan. AMPL: A Modeling Language for MathematicalProgramming. 2nd edition. Cengage Learning, 2002. https://ampl.com/resources/the-ampl-book/G. W. French, J. R. Kennaway, and A. M. Day. Programs as Visual, Interactive Documents.
Journal of Software:Practice and Experience
Proc. Constructionism , pp. 1–10, 2010.Brian Hempel, Justin Lubin, Grace Lu, and Ravi Chugh. Deuce: A Lightweight User Interface for StructuredEditing. In
Proc. International Conference on Software Engineering , pp. 654–664, 2018. https://doi.org/10.1145/3180155.3180165Daniel Ingalls, Krzysztof Palacz, Stephen Uhler, Antero Taivalsaari, and Tommi Mikkonen. The Lively KernelA Self-supporting System on a Web Page. In
Proc. Self-Sustaining Systems , pp. 31–50, 2008. https://doi.org/10.1007/978-3-540-89275-5_2Lennart C. L. Kats and Eelco Visser. The Spoofax Language Workbench. In
Proc. Object-Oriented Programming,Systems, Languages & Applications , pp. 444–463, 2010. https://doi.org/10.1145/1932682.1869497Clemens N. Klokmose, James R. Eagan, Siemen Baader, Wendy Mackay, and Michel Beaudouin-Lafon. Web-strates: Shareable Dynamic Media. In
Proc. ACM Symposium on User Interface Software and Technology ,pp. 280–290, 2015. https://doi.org/10.1145/2807442.2807446Amy Ko and Brad A. Myers. Barista: An Implementation Framework for Enabling New Tools, InteractionTechniques and Views in Code Editors. In
Proc. Conference on Human Factors in Computing Systems , pp.387–396, 2006. https://doi.org/10.1145/1124772.1124831John Maloney, Kimberly M. Rose, and Walt Disney Imagineering. An Introduction to Morphic: The SqueakUser Interface Framework. In
Squeak: Open Personal Computing and Multimedia , pp. 39–77 Pearson, 2001.Microsoft. Office and SharePoint Development in Visual Studio. Retrieved January, 2019, 2019.https://docs.microsoft.com/en-us/visualstudio/vsto/office-and-sharepoint-development-in-visual-studio?view=vs-2017Chris Okasaki. Red-black Trees in a Functional Setting.
Journal of Functional Programming
Proc. Workshop on Type-driven Development , 2019.
Proc. ACM Program. Lang., Vol. 4, No. OOPSLA, Article 222. Publication date: November 2020.
Cyrus Omar, YoungSeok Yoon, Thomas D. LaToza, and Brad A. Myers. Active Code Completion. In
Proc.International Conference on Software Engineering , pp. 859–869, 2012.Mark Overmars. Teaching Computer Science Through Game Design.
Computer
Proc. Principlesand Practice of Programming in Java , pp. 165–168, 2013. https://doi.org/10.1145/2500828.2500846Fernando Perez and Brian E. Granger. IPython: A System for Interactive Scientific Computing.
Computing inScience and Engineering
Communications of the ACM
Proc. ACM Symposium on User Interface Software and Technology ,pp. 715–725, 2017. https://doi.org/10.1145/3126594.3126642Charles Simonyi, Magnus Christerson, and Shane Clifford. Intentional Software.
ACM SIGPLAN Notices
Proc. Interna-tional Workshop on The Globalization of Modeling Languages , 2014.Markus Voelter, Daniel Ratiu, Bernhard Schaetz, and Bernd Kolb. mbeddr: an Extensible C-based ProgrammingLanguage and IDE for Embedded Systems. In
Proc. Conference on Systems, Programming, and Applications:Software for Humanity , pp. 121–140, 2012. https://doi.org/10.1145/2384716.2384767Holger Vogt, Marcel Hendrix, and Paolo Nenzi. Ngspice Users Manual. NGSPICE, 30, 2019. http://ngspice.sourceforge.net/docs/ngspice-30-manual.pdfStephen Wolfram. The Mathematica Book. Fourth edition. Cambridge University Press, 1988., pp. 121–140, 2012. https://doi.org/10.1145/2384716.2384767Holger Vogt, Marcel Hendrix, and Paolo Nenzi. Ngspice Users Manual. NGSPICE, 30, 2019. http://ngspice.sourceforge.net/docs/ngspice-30-manual.pdfStephen Wolfram. The Mathematica Book. Fourth edition. Cambridge University Press, 1988.