Sharing and Learning Alloy on the Web
Nuno Macedo, Alcino Cunha, José Pereira, Renato Carvalho, Ricardo Silva, Ana C. R. Paiva, Miguel S. Ramalho, Daniel Silva
SSharing and Learning Alloy on the Web
Nuno Macedo , , Alcino Cunha , , Jos´e Pereira , Renato Carvalho , , RicardoSilva , Ana C. R. Paiva , , Miguel Sozinho Ramalho , , and Daniel Silva INESC TEC Universidade do Minho Universidade do Porto
Abstract.
We present Alloy4Fun, a web application that enables onlineediting and sharing of Alloy models and instances, to be used mainly inan educational context. By introducing the notion of secret paragraphsand commands in the models, it also allows the distribution and auto-matic evaluation of simple specification challenges, a useful mechanismthat enables students to learn relational logic at their own pace. Al-loy4Fun stores all versions of shared and analyzed models, as well asderivation trees that depict how those models evolved over time: thiswealth of information can be mined by researchers or tutors to identify,for example, learning breakdowns in the class or typical mistakes madeby students and other Alloy users. A beta version of Alloy4Fun was al-ready used in two formal methods courses, and we present some resultsof this preliminary evaluation.
Keywords:
Teaching formal methods · Alloy · Web application
Alloy [4] is a popular formal specification language and toolkit to describe andreason about a software design. It is also taught in several undergraduate andgraduate courses in formal methods , including our own graduate courses atUniversity of Minho and University of Porto, in Portugal. One of the reasonsfor this popularity is the support for automated analysis provided by the AlloyAnalyzer, an easy to download and install self-contained executable written inJava. The Analyzer also allows instances (either witness scenarios or counter-examples) to be graphically depicted using user-customized themes, a popularfeature both for experienced users and students.Despite such streamlined toolkit, over the many years we taught and re-searched with Alloy we identified some missing features and functionalities thatcould further ease its adoption, in particular in an educational context: – A straightforward mechanism to share simple Alloy models and instances. – Some auto-grading functionality or online judge system for students to au-tomatically check the correctness of their exercise resolutions. A (bit outdated) list of courses using Alloy and papers reporting on the experienceof teaching with Alloy can be found in its official website at . a r X i v : . [ c s . C Y ] J u l N. Macedo et al. – A mechanism to quickly obtain insight on how students use the language,namely identify typical mistakes or learning breakdowns in the class.To address these limitations we developed Alloy4Fun, a web application thatenables online editing and sharing of Alloy models and instances, including sim-ple specification challenges in the form of duels where students attempt to dis-cover a secret teacher’s specification.In this paper we present Alloy4Fun, starting with an overview of (and ratio-nale for) its current features in Section 2. Then, in Section 3 we briefly describeits implementation and some results from the evaluation of a beta version in twoformal methods courses. Finally, in Section 4 we conclude the paper and presentsome ideas for future work. Knowledge of Alloy is not required to understandthe paper, but can help better appreciate some of the features of Alloy4Fun.
The core of Alloy4Fun, mimics in a web application, the main features of thestandalone Alloy Analyzer. After accessing alloy4fun.inesctec.pt the usergets an empty online editor (with syntax highlighting) where Alloy models can bewritten. A button allows analysis commands in the model to be executed, either run commands to get witness scenarios or check commands to find counter-examples to properties. If found, these instances are depicted bellow the editoras graphs that, likewise in Analyzer, can be customized with user-defined themes(including signature projection). Besides these core functionalities, Alloy4funtries to improve some aspects of the Analyzer, which are described in the sequel.
Instance visualization and navigation.
Unlike in the Analyzer, some ofthe most frequent theme customizations (like changing colors or shapes) canbe performed quickly through a right-click menu on atoms or edges. Differentlayout algorithms can be selected to automatically organize nodes, which theuser can manually modify. Unlike in the Analyzer, these positions are preservedbetween the frames of projected instances. Figure 1 shows a screen capture ofAlloy4Fun where a counter-example is being depicted with a user-defined theme.In addition, besides navigating to the next instance, the user can also re-visitprevious instances, unlike in the Analyzer.
Sharing models and instances.
Currently, when someone wants to shareor reference a model or instance, it is necessary to copy the model or screencaptures, or setting up a web-site or repository. This rapidly becomes tediousand time consuming, in particular for tutors of large classes. Alloy4Fun providesthe ability to easily share models and instances. After pressing the “share model”button a permalink is generated, that can later be used to access the model. Anytheme defined by the user is also preserved when sharing, thus allowing instancesof shared models to be depicted as intended. Instances can also be shared via permalinks . The theme and layout of the depicted atoms and relations at thetime of sharing are preserved. This is a very handy feature since, likewise in theAnalyzer, the positioning of atoms by the automatic layout mechanism is oftennot ideal, requiring manually rearrangement for better comprehension. haring and Learning Alloy on the Web 3
Fig. 1.
A failed attempt to solve a challenge
Anonymous interaction.
In Alloy4Fun there are no user accounts normeans to recover the permalinks of previously shared models and instances.The user is responsible for keeping track of his permalinks using some externalmechanism, and Alloy4Fun provides a “copy to clipboard” button that helpswith this task. The anonymity, namely the absence of user accounts, was adesign choice in order to keep the interaction with the web application as simpleas possible, to maximize user exposure, and also to avoid dealing with privacyand security issues, namely the hassle of storing and managing user credentials.
Auto-grading.
Although Alloy’s specification language has very neat andsimple syntax and semantics, many students struggle with its declarative nature,in particular those used to procedural programming [2]. One way to overcomethis difficulty is by independently solving exercises proposed by tutors, but evenwith automated analysis it is often difficult for students to assess whether theyreached the correct answer, and tutors are required to inspect the solutions(something not scalable for large classes). These problems could be mitigatedwith an auto-grader, allowing students to solve exercises at their own pace andwithout the constant need for face-to-face time with tutors. In recent years,auto-graders and online judges have become widely popular for learning how toprogram [6], and we believe this success could be replicated in the learning offormal methods in general, and Alloy in particular.With this in mind, the user in Alloy4Fun has the ability to mark any para-graph of a model as secret . In Alloy a paragraph is either a signature (and itsfields) declaration, a fact with a constraint that is assumed to hold, an assertion
N. Macedo et al. to be checked, an auxiliary predicate or function, or a run or check command.To mark a paragraph as secret, the special comment //SECRET must be addedimmediately before. When sharing a model with secret paragraphs two perma-links are generated: a private one that, when accessed, reveals the full model,including secrets; and a public one that, when accessed, only shows public para-graphs, but internally still considers the secret in analyses and still allows theexecution of secret commands (whose names are public). Using a comment in-stead of a new keyword to mark secret paragraphs ensures compatibility withAlloy’s default syntax, allowing users to copy and paste models from Alloy4Funto the stand-alone Analyzer, and vice versa.This feature can be used to create simple specification challenges in the formof duels, where the user tries to reach a secret specification. Such models canhave a public predicate that the user must fill-in, together with a secret check command that asserts (for a given scope) that such predicate is equivalent to thedesired (hidden) specification. The model shown in Fig. 1 was obtained preciselyby accessing the public permalink of a model with 4 challenges (in this case,simple exercises where a natural language description of the desired specificationis given for each of them) . After filling the empty predicate (e.g., Inv2 ), a usercan check whether it is a valid solution (e.g., secret command
Inv2OK for
Inv2 ),which will either return a “no counter-example found” message, meaning thechallenge is solved, or a counter-example (as is the case in Fig. 1, where thecounter-example shows that the specification
Inv2 is still not correct).
Mining derivation trees.
A possible way to gain insight about the stu-dents’ learning process is to have access to their attempts at solving the pro-posed exercises, and tool support to mine this corpus for useful data [5]. Again,such feature would also be useful in a research context, and was one of thereasons that led Microsoft to develop the web service, thatallows researchers to easily deploy their tools on the web and collect human-tool interactions for posterior mining [1] (besides other advantages of web tools,like increased exposure, since the need for downloading and installing is elimi-nated, and promoting reliability given the large amount of test cases that canbe collected for free). One of the most popular examples available via Rise4Fun,and the inspiration for developing Alloy4Fun, is , a web-basededucational gaming environment for learning programming, where students canengage in coding duels where they attempt to write code equivalent to a teacher’ssecret implementation [7]. Pex [8], an advanced white box test-generation tool,is used on the background to find inputs that show discrepancies between thestudent’s code and the secret implementation. However, the interaction with theoutcome of the tools is limited in Rise4Fun, which would prevent the implemen-tation of key Alloy features like instance iteration and customization. This hasled us to implement our own solution rather than integrate Alloy in this service.Every shared model and instance is stored by Alloy4Fun in its database.However, to enable the proponents of challenges to mine the submissions for The permalink of this model and others can be found in a curated list of examplesat https://github.com/haslab/Alloy4Fun/wiki/Examples .haring and Learning Alloy on the Web 5 useful information, every command that was executed (and the respective result)and associated model are also stored. Moreover, for each model, the identifier ofthe model from which it derives and a time-stamp are also stored. This meansthat all the models that developed after accessing a shared permalink end upforming a derivation tree . In the case of a permalink with challenges, a branchin this tree typically corresponds to an interactive session where one user /student is trying to solve the different challenges inside, and can be analyzedto determine, for example, how many challenges where solved or how manyattempts where needed to solve each one. Subsequent branches represent pointswhere the a permalink was generated for a model, and accessed multiple times.Alloy4Fun allows anyone in possession of the secret permalink of a model todownload the respective derivation tree in an easy to process JSON format.
We developed Alloy4Fun with Meteor, a full-stack isomorphic JavaScript frame-work for developing web applications based on Node.js. The client uses CodeMir-ror as text editor and the Cytoscape.js graph visualization library to depictinstances. Models and instances are stored in a MongoDB document-orienteddatabase at the server. To execute commands, we encapsulated the Alloy An-alyzer in a RESTful web service implemented in Java. Seamless deployment ofboth the application and the service in a server is performed using Docker. All theAlloy4Fun code is open-source and available at github.com/haslab/Alloy4Fun .We performed a preliminary evaluation of Alloy4Fun in the first semester ofthe 2018/19 academic year in two graduate formal methods courses at Universityof Minho (UM) and University of Porto (UP), in Portugal. The former teachesAlloy for 6 weeks and had 22 students enrolled, and the latter for 4 weeks andhad 156 students enrolled. Both courses have one weakly lecture and one weaklylab session. The main goals of this evaluation were: 1) to test a beta version ofthe application in a medium-sized audience, to identify bugs and possible designimprovements; 2) to determine whether students found Alloy4Fun useful as anauto-grader while learning Alloy. Notice that the labs still relied mainly on theAlloy Analyzer for hands-on practice. Alloy4Fun use was not mandatory andwas introduced to the students via a list of challenges with exercises for them toindependently practice Alloy outside of the classes.Regarding goal 1, we identified many bugs during this evaluation, includinga major issue that caused a slowdown in the application that, unfortunately,rendered it almost unusable after a few weeks. All these bugs were fixed in thecurrent version. We also identified several potential design improvements, mostof them already implemented. For example, the beta version had a special “lock”comment to prevent the accidental edition of certain paragraphs. However, wenoticed students rarely tried to change the model outside of the challenge pred-icates, and opted to remove this feature for simplicity and efficiency.Regarding goal 2, we developed 19 different exercises containing challengesand provided them as shared models to students. These ranged from trivial exer-
N. Macedo et al.
Fig. 2.
Results of the preliminary evaluation cises (e.g., asking to enforce simple inclusion dependencies or multiplicities), tomore complex ones requiring the use of nested quantifiers or closures. Figure 2shows, for each shared model: the number of challenges inside (between paren-thesis), the number of students that had access to the challenge, how manytimes permalinks were generated by sharing, and how many solving sessionswere recorded (in a cumulative stacked bar discriminating the number of ses-sions where all, some or no challenges were solved). As an example, exercise permalink several times, instead of generatinga new permalink of a partial resolution for later resuming the work. Even withthis uncertainty, it is safe to say that almost all of the students used Alloy4Funat least once and most of them used it several times, a very positive result inparticular considering the aforementioned performance bug.
We briefly presented Alloy4Fun, a web application for online editing and sharingAlloy models and instances, that also allows the distribution and auto-gradingof simple specification challenges, by just marking some paragraphs (namely check commands) as secret. Its main intended use is in an educational context,and a preliminary evaluation of a beta version in two courses provided evidencethat students found the auto-grading feature useful for learning Alloy. In thefuture, we intend to develop tools to simplify the mining of useful data fromthe derivation trees, possibly to be run server-side at the click of a button (withresults visualized in the browser), and also incorporate an alternative instancevisualizer more amenable for dynamic systems [3]. We also intend to continueusing Alloy4Fun in our formal methods courses in the upcoming years, andcollect relevant data to conduct a detailed study on how students learn and useAlloy, namely identify which features of the language they find more challenging. haring and Learning Alloy on the Web 7
Acknowledgements
We would like to thank Daniel Jackson for the helpful comments and suggestionsabout the design of Alloy4Fun, and also all the students that were beta testers.This work is financed by the ERDF – European Regional Development Fundthrough the Operational Programme for Competitiveness and Internationalisa-tion - COMPETE 2020 Programme and by National Funds through the Por-tuguese funding agency, FCT - Funda¸c˜ao para a Ciˆencia e a Tecnologia, withinproject POCI-01-0145-FEDER-016826. The second author was also supportedby the FCT sabbatical grant with reference SFRH/BSAB/143106/2018.