MMerly.jl: Web Framework in Julia
JOSUÉ ACEVEDO MALDONADO, @neomatrixcode, México
Merly.jl is a package for creating web applications in Julia. It presents featuressuch as the creation of endpoints with function notation and with macronotation, handling of static files, use of path parameters, processing of datasent by a web client in the body in a personalized way, handling of CORSand compatibility with the use of middleware. It presents a familiar syntaxwith the rest of the most popular web frameworks without neglecting theexecution performance. This manuscript mentions the operation and mainfeatures of Merly.jlCCS Concepts: •
Software and its engineering → Software librariesand repositories ; Software design engineering .Additional Key Words and Phrases: software tools, open-source software,julialang, metaprogramming, microservices, cors
At present, information systems have gone from running on a per-sonal computer to working on the Internet, this change of envi-ronment has occurred mainly thanks to the evolution of protocolssuch as the Hypertext Transfer Protocol (HTTP) [1] , the creationof exchange formats of information such as XML[5] or JSON[6],software architectures such as Microservices[3] as well as Backendweb development libraries. Such has been the impact that web de-velopment libraries exist in practically all programming languages,the best known being Flask, Sinatra or Express.Julia is a programming language that has the main characteristicof using multiple dispatches and metaprogramming to provide flex-ibility in the syntax, such as execution efficiency on the machine[2], offers the possibility of developing web applications, for thethat requires the use of libraries that facilitate development andavoid reinventing the wheel. The most popular frameworks thatJulia currently presents are: Mux.jl, Bukdu.jl and Dance.jl.Merly.jl was born with the purpose of having a web frameworkthat is easy for the developer to write; taking up syntax conceptsfrom the most popular frameworks both from other languages andfrom Julia, as well as taking into account the performance of thecode’s execution.
Merly.jl creates an association using a path, an HTTP verb and ananonymous function AKA lambda [7]. When you request a URLwith a certain HTTP verb in a web browser this function will runand display the result.To obtain a good execution performance, the search process wasprioritized so it was decided to use a hash table instead of a self-balancing binary tree or a single path array, which among otherthings present problems when the number of stored data is veryhigh [4, 8].
Author’s address: Josué Acevedo Maldonado, @neomatrixcode, Oaxaca, México,[email protected].
Fig. 1. Create an endpoint in Merly.jl. A hash table is used due to theefficiency in the search process.
As the key of the hash table we have a number value, which iscreated from a value associated with each HTTP verb and whichcan be consulted in a table, which in practical terms is also a hashtable, concatenated to the number of elements present in the path.This technique makes it possible to reduce the number of collisionswhen carrying out a search .The value stored in the hash table will be an array of namedstructures, which will store the lambda function to be executed,such as the path and the parameters present in it. Being an array,this allows one or more routes to be added to it that have the samenumber of elements and the same HTTP verb.In such a way that, when performing the search, the hash tableallows us to discard a large number of possible options with fewcomputational calculations. On the resulting array of routes, wewill perform a search with the match between the searched routeand the previously stored route in parallel using the Iterators.filtermethod; which will finally obtain the function to be executed andwhose result will be sent to the web client.
Fig. 2. Search process and execution of an endpoint. The routes are obtainedfrom the hash table and are analyzed in parallel to finally obtain the functionto execute.
If a key is not found in the hash table that matches the client’srequest or, failing that, a path, a function stored with key 0 will beexecuted by default. This default function is known as “not found” a r X i v : . [ c s . S E ] F e b • Acevedo and it is possible to replace it with other that executes our owncustom code. Three elements are used to create route routes: an HTTP verb, aroute, and an anonymous or lambda function. The easiest way toachieve this is by using functions, which respond to the name of eachof the HTTP verbs. These functions receive as parameters a stringthat will define the path of the endpoint and an unnamed function,this lambda function always receives, when executed, a structurethat will contain all the data sent by the client, called request, and astructure called HTTP ( defined in the HTTP .jl package) that willhold the HTTP response code, body, and headers.
Get("/data", (request,HTTP)->beginHTTP.response(200, "test text Get")end)Post("/data", (request,HTTP)-> beginHTTP.response(200, "test text Post")end)Put("/data", (request,HTTP) -> beginHTTP.response(200, "test text Put")end)Delete("/data", (request,HTTP) -> beginHTTP.response(200, "test text Delete")end)Connect("/data", (request,HTTP) -> beginHTTP.response(200, "test text Connect")end)Trace("/data", (request,HTTP) -> beginHTTP.response(200, "test text Trace")end)Head("/data", (request,HTTP) -> beginHTTP.response(200, "test text head")end)Patch("/data", (request,HTTP) -> beginHTTP.response(200, "test text Patch")end)
There is a shorter syntax with which route paths can be defined,that is, through Macros, which assign a tuple of arguments to anexpression which is compiled directly. The macro @page implicitlyassigns the GET method to a certain route and instead of defining afunction, only its body is specified. @page "/" HTTP.Response(200,"Hello World!")
The @route macro, on the other hand, requires in addition to allthe elements mentioned above the name of the HTTP verb, which allows the use of the operator "|" to define more than one verb tothe same path and function body more easily. @route POST|PUT|DELETE "/route" beginprintln("query: ",request.query)println("body: ",request.body)HTTP.Response(200, HTTP.mkheaders(["Content-Type" => "text/plain"]), body="I did something!")end
When macros receive only the body of the function as a parameter,it is necessary to define a lambda function using metaprogramming,so the context of this function will be different to the one of theprogram where the macro was used.
It is possible to define, in the path of the endpoint, segments orareas that will be occupied by data sent by the web client, for whichthe colon character ( : ) is used, thanks to which a name will beassigned to said parameter, this characteristic is compatible withboth function notation and the two path definition syntaxes thatuse macros. If you want to use a regular expression to custom filteryour path parameters, you can do it using a pair of parentheses andthe backslash escape character ( \). @page "/hola/:usr" beginHTTP.Response(200,string("Hello ",request.params["usr"],"!"))end@route GET "/get/:data1" beginHTTP.Response(200, string(u ,request.params["data1"]))endGet("/test1/:usr",(request, HTTP) -> beginHTTP.Response(200,string("test1 ",request.params["usr"],"!"))end)@route GET "/regex/(\\w+\\d+)" beginreturn HTTP.Response(200,string("datos ",request.params["2"]))end
The data sent in the route will be accessible from the dictionarycalled params stored in the request structure, through the name erly.jl: Web Framework in Julia • 3 defined when creating the route; by using a regular expression, thedata can be accessed by referring to its position within the path.
Taking into account that the function associated with an endpointwill be executed or defined in a different context than the programwhere the route was established, variables can be passed to thecontext of the function to be executed, both in the function andMacro syntax.The data of these variables will be passed to the function eitherby reference or by value, depending on whether the data type ismutable or not, in the first case it will be passed by reference and inthe second by value. u=1@page "/get1" (;u=u) HTTP.Response(200,string("Get1 ",u," !"))@route GET "/get/:data1" (;u=u) beginu = u +1HTTP.Response(200, string(u ,request.params["data1"]))end
To process the data from the web client, it is necessary that thereis a process that listens for requests. To configure the executionoptions of this process such as the host, the port and the ssl keys,named variables are used. These are received by the function called start .The host parameter receives a text string with the ip address,which can be version 4 or version 6. To manage the configurationof the cryptographic keys to use an encrypted HTTP connection,the MbedTLS package is used, specifically the
SSLConfig functionwhich will process the files of both the certificate and the key. using MbedTLSstart(host = "127.0.0.1", port = 8086, sslconfig= MbedTLS.SSLConfig("localhost.crt","localhost.key") )
The values of the headers can be specified at the time of returningthe HTTP structure using the mkheaders function. However, if youwant to constantly return one or more specific headers you can relyon the headersalways function which receives an array of string2-tuples. This way code redundancy is avoided. headersalways(["X-PINGOTHER" => "pingpong", "Y-PINGOTHER" => "text"])
If you ever need to use CORS, Merly has you covered, there isa dedicated function to configure the CORS headers in addition to enabling the necessary OPTION routes for the verified CORSrequests to work without too much complexity. useCORS(AllowOrigins = "*", AllowHeaders = "Origin, Content-Type, Accept", AllowMethods = "GET,POST,PUT,DELETE", MaxAge = "178000")
The useCORS function receives a set of named variables whichare: • AllowOrigins: Specifies a URI that can access the resource. • AllowHeaders: Indicates which HTTP header can be usedwhen requesting the resource. • AllowMethods: Specifies the method or methods allowedwhen a resource is allocated. • MaxAge: This header indicates for how long the results ofthe verified request can be captured, and therefore it is nolonger necessary to carry out the process again.
If we need to associate a url with a file stored on the computer,allowing the access to its content through the http protocol we canuse the
File and webserverfiles functions.
File is a function that receives a text string that includes thename of a file with its perspective extension and returns the contentof that file in a text string format. By default, this file must be inthe same location where the program is executed, considered asthe root or current directory, otherwise the relative path from thiscurrent location must be included.
File("index.html")
The webserverfiles function will take all the existing files in theroot or current directory and expose them, although there is a param-eter that allows controlling this behavior; if the webserverfiles function receives a text string equal to ( * ) it will expose all fileswithout exception, but if you want to discriminate files by extension,then a text string with the extension or extensions separated bypipe will be passed to the function ( | ) of the files to expose. webserverfiles("*")
There is the possibility of modifying the directory considered asroot, with the webserverpath function that receives a text stringwith the name of the folder or its full path, in such a way that whenexecuting either the
File or webserverfiles function, you will beworking with the existing files in the new location. webserverpath("folder") The data sent by the client in the body field can be of any type,by default Merly.jl processes the data as text strings, so within thefunction this data can be found in the body field of the structurerequest.If at any time, you want to work with julia’s native dictionariesto process the data, the conversion process would have to be done • Acevedo manually each time is required. So an easy way to make Merly.jldeliver this data already processed is with a custom function that re-ceives a text string by default and returns the data already processed,with an existing library or with its own algorithm. function tojson(data::String)return JSON.parse(data)endformats["application/json"] = tojson
Subsequently, the name of this function is stored in a dictionarycalled formats , provided by the package; the key associated withthis function being a text string with a specific Content-Type. Thereis the possibility of processing each format or data type sent by theclient in a different way, for example, XML, JSON, etc.
Post("/data", (request,HTTP)-> beginHTTP.Response(200, HTTP.mkheaders(["Content-Type" => "text/plain"]), body=string("I did something! ", request.body["query"]))end)
At the moment, this feature is only available in function notation.However, it extends the functionalities of the functions to be exe-cuted by allowing them to concatenate with other functions thatperform tasks either before or after the main function. function authenticate(request, HTTP)isAuthenticated = falseif (request.params["status"] === "authenticated")isAuthenticated = trueendreturn request, HTTP, isAuthenticatedendGet("/verify/:status",(result(;middleware=authenticate) = (request, HTTP)->beginmyfunction = (request, HTTP, isAuthenticated)-> beginif (isAuthenticated == false )return HTTP.Response(403,"Unauthenticated. Please signup!")endreturn HTTP.Response(200,string("verify !")) endreturn myfunction(middleware(request,HTTP)...)end)())
In future versions of Merly.jl it is planned to add the possibility ofworking with websockets, make the middleware writing easier andoptimize memory use when creating routes.
The Project presents a set of tests hosted in the runtests.jl file, whichtests most of the Merly.jl features and runs automatically on theTravisCI platform on Windows, Linux and Mac OSX operatingsystems and on the Julia versions 1.5, 1.6 and 1.7. Also, the amount ofcode evaluated by the tests is monitored on the AppVoyer platform.Locally, these same tests can be executed with the Pkg.test ("Merly")command, so any user can verify that the package works correctly ontheir computer. If there is a problem with the operation of the pack-age or want to propose new features, users can go to the project’sissues page on Github and open a new one.
Merly works on Windows 7+, Mac OSX, Linux and FreeBSD
Julia v1.5+
The HTTP.jl package is used to manage communication through theHTTP protocol, as well as MbedTLS.jl to manage the cryptographickeys that allow the use of the HTTPS protocol. • Josué Acevedo Maldonado, lead developer of Merly.jl • Phelipe Wesley, contributed by rewriting a large part of thepackage to improve its compatibility with the 1.4 version ofJulia • File: ZenodoName: Merly.jlPersistent identifier: DOI: 10.5281/zenodo.4546005License: MITPublisher: Acevedo Maldonado JosuéVersion published: v1.0.0 • Code repository: GitHubName: neomatrixcode/Merly.jlPersistent identifier: https://github.com/neomatrixcode/Merly.jl erly.jl: Web Framework in Julia • 5
License: MITDate published: 07/02/2021Version published: v1.0.0Documentation Language: EnglishProgramming Language: Julia
Julia is a relatively recent programming language which has hadthe greatest boom so far in the field of data science and artificialintelligence. However, these are not the only areas where the lan-guage can address thanks to its features such as ease of use andexecution speed. Julians who are looking for a package that allowthem to expose resources on the web in a simple and efficient waywith periodic updates of features will find in Merly.jl an importantally to achieve their objectives.
ACKNOWLEDGMENTS
A thank you to the Julia community for their contributions to thepackage and their feedback on the features to add or modify in thispackage, also to my friend Vásquez Martínez Agustín (@maldad)for his help in reviewing the first versions of this document.
REFERENCES [1] Tim Berners-Lee. 2005. WWW at 15 Years: Looking Forward. In
Proceedings ofthe 14th International Conference on World Wide Web (Chiba, Japan) (WWW ’05) . Association for Computing Machinery, New York, NY, USA, 1. https://doi.org/10.1145/1060745.1060746[2] Jeff Bezanson, Alan Edelman, Stefan Karpinski, and Viral B. Shah. 2015. Julia: AFresh Approach to Numerical Computing. arXiv:1411.1607 [cs.MS][3] Robert Heinrich, André van Hoorn, Holger Knoche, Fei Li, Lucy Ellen Lwakatare,Claus Pahl, Stefan Schulte, and Johannes Wettinger. 2017. Performance Engi-neering for Microservices: Research Challenges and Directions (ICPE ’17 Com-panion) . Association for Computing Machinery, New York, NY, USA, 223–226.https://doi.org/10.1145/3053600.3053653[4] Donald E. Knuth. 1998.
The Art of Computer Programming, Volume 3: (2nd Ed.)Sorting and Searching . Addison Wesley Longman Publishing Co., Inc., USA.[5] Tova Milo, Serge Abiteboul, Bernd Amann, Omar Benjelloun, and Fred DangNgoc. 2003. Exchanging Intensional XML Data. In
Proceedings of the 2003 ACMSIGMOD International Conference on Management of Data (San Diego, California) (SIGMOD ’03) . Association for Computing Machinery, New York, NY, USA, 289–300.https://doi.org/10.1145/872757.872793[6] Felipe Pezoa, Juan L. Reutter, Fernando Suarez, Martín Ugarte, and Domagoj Vrgoč.2016. Foundations of JSON Schema. In
Proceedings of the 25th International Confer-ence on World Wide Web (Montréal, Québec, Canada) (WWW ’16) . InternationalWorld Wide Web Conferences Steering Committee, Republic and Canton of Geneva,CHE, 263–273. https://doi.org/10.1145/2872427.2883029[7] G. E. Revesz. 2009.
Lambda-Calculus, Combinators and Functional Programming (1st ed.). Cambridge University Press, USA.[8] Aaron M. Tenenbaum, Yedidyah Langsam, and Moshe J. Augenstein. 1990.
DataStructures Using C . Prentice-Hall, Inc., USA.