The development and deployment of formal methods in the UK
TThe development and deployment of formalmethods in the UK
Cliff B. Jones, Martyn ThomasJune 12, 2020
This work has been submitted to the IEEE forpossible publication. Copyright may be trans-ferred without notice, after which this versionmay no longer be accessible
Abstract
UK researchers have made major contributions to the technical ideasunderpinning formal approaches to the specification and development ofcomputer systems. Perhaps as a consequence of this, some of the signifi-cant attempts to deploy theoretical ideas into practical environments havetaken place in the UK. The authors of this paper have been involved informal methods for many years and both have tracked a significant pro-portion of the whole story. This paper both lists key ideas and indicateswhere attempts were made to use the ideas in practice. Not all of thesedeployment stories have been a complete success and an attempt is madeto tease out lessons that influence the probability of long-term impact.
The term “formal methods” covers the use of mathematically precise notationsto specify and reason about systems. This paper is mainly concerned withformal methods for software and emphasises deployments that have taken placein the UK. As well as mentioning some of the important scientific contributions,emphasis is placed on the application and deployment of technical ideas andtheir tool support.Both of the authors of this paper believe that the use of formal methods isessential for the creation of high-quality software. This paper is, however, nota sales pitch for such methods. What it sets out to do is to review some actualdeployments, to be frank about drawbacks as well as report successes and toidentify factors that have either aided or impeded the deployments.There is no claim that this paper reports on all (nor even the most important)applications of formal methods; [WLBF09] offers a broader survey. Given that Companion papers in this edition of
Annals cover other European countries. a r X i v : . [ c s . S E ] J un selection had to be made, it felt reasonable to choose applications where theauthors had personal knowledge or at least had direct access to colleagues withsuch knowledge.There is no attempt here to claim that formal approaches are a panacea inthe sense that they would solve all of the problems in the software industry. Forexample, the question of how to argue that a (formal or informal) specificationactually meets the needs of users is a huge challenge — the wise writings ofMichael Jackson such as [Jac00] are a good starting point here. Furthermore,there is the challenge of designing systems that are dependable when used inconjunction with faulty components (including hardware and human) — in thisarea [BGJ06, RT13] are useful starting points.Tony Hoare’s paper on the axiomatic basis of programming languages [Hoa69]is the foundation stone of a significant portion of the work on formal meth-ods. In tandem with notations that have been used to provide abstract mod-els of complex systems (e.g. VDM [Jon80], Z [Hay93], B [Abr96] and Event-B [Abr10a]) Hoare’s ideas permeate many of the applications in the body ofthis paper. Robin Milner’s LCF [GMW79] system provided a model for manyof the theorem proving assistants that have been developed. UK computerscientists have also made major contributions to concurrency research (e.g. Mil-ner’s CCS [Mil80] and Hoare’s CSP [Hoa85]).One technical area where UK researchers have made crucial contributionsis in the semantics of programming languages; although this plays a small partin the rest of the paper, Section 2 outlines this research and identifies someof its applications. We decided against trying to follow a strict timeline forthe remaining sections. This is partly because the various strands of the storyoverlap heavily in time but we also felt that the lessons could be more clearlyillustrated by looking at different modes of engagement. Beyond Section 2, thestructure of the paper is that Section 3 considers initiating academic/industrycollaboration using consultants, Section 4 reports on some projects where theexpertise was in-house, Section 5 identifies some factors that have an impact onany deployment; conclusions are summarised in Section 6.
Although the bulk of this paper is concerned with the development of programs,there are two reasons to address research on programming languages: this wasone of the first areas where is was realised that formalism could make a contri-bution and the UK played a significant part in the development of such research.The study of languages is sometimes referred to as “semiotics”. For pro-gramming languages, most interest is in their syntax (covering content and An excellent technical survey of 50 years of research on
Hoare-Logic is [AO19]. An attemptto outline the broader picture is [JPRW03]. Historical papers on these topics include [Gor00, PNW19, Moo19] and [Mac01] looks atwhat it means to claim that something has been proven. Charles Sanders Peirce 1839–1914 wrote extensively on philosophy, language and logic —in addition to his collected works, [Pei91] reprints an earlier book. + Backus Normal Form (often referred to as Backus Naur Form ). Variantsof BNF have been developed including
Extended BNF [Wir77] and, also fromNiklaus Wirth, a graphical representation referred to as
Railroad Diagrams . Allof these notations perform essentially the same function and there is little doubtas to their usefulness. One bonus from using such formal descriptions of the syn-tax of programming languages is that they can be employed to generate parsersfor the front-ends of language translators. This does bring some additional con-cerns about ambiguity and efficiency but, again, there is broad consensus onhow to resolve these more detailed points.The problems of finding ways to describe the semantics or meaning of pro-gramming languages has proved far more challenging. A landmark conferencewas held in Baden-bei-Wien in 1964. This, first ever, IFIP working confer-ence focussed on the subject of
Formal Language Description Languages andmost of the talks addressed proposals for ways to describe formally the seman-tics of programming languages. One paper that had considerable influence onthe work of the IBM Laboratory in Vienna was by the American John Mc-Carthy. In [McC66], he constructed an abstract interpreter for Micro-Algol .(Essentially, an interpreter takes a program and a starting state and iterativelycomputes its final state; McCarthy used the adjective “abstract” to indicatethat the metalanguage in which the interpreter was written was limited andmathematically tractable.) This technique provided an operational semantics for a very small language.The influence of work at the IBM Vienna Lab on UK research becomesimportant below. The phase in the 1960s saw the ideas of operational semanticsextended and applied to the huge PL/I programming language. Their techniquesbecame known as the
Vienna Definition Language (VDL) — see [LW69].The task of formally describing the evolving PL/I language was separatefrom the language design team. Of course, there was strong interaction andcommunication. But, whenever the formalisation detected problems in the in-herently ambiguous (and sometimes contradictory) natural language descrip-tion, the formalists had to communicate the problem (sometimes by writing anindicative program). The response was then an amendment to the text whichagain might not be crystal clear.
Lesson : Separation of designers from formalists is far less productive thanworking as an interdisciplinary team.Turning to key UK speakers to the 1964 working conference, both Christo-pher Strachey and Peter Landin spoke about a more abstract approach thanoperational semantics: rather than define an abstract interpreter, they were John Backus made the initial proposal. Peter Naur applied the notation to ALGOL and proposed extensions to the notation. The proceedings of the conference [Ste66] took some time to be published but are invalu-able to those who want to understand the development of ideas because the post-presentationdiscussions were captured. de-notational semantics , is a fascinating story of its own. Suffice it here to say herethat researchers at Oxford University made the seminal contributions. At the beginning of the 1970s, researchers at the IBM Lab in Vienna alsomade the step from operational to denotational semantics. The PL/I languagewas again a catalyst: the Lab had been invited to build the PL/I compiler fora radically new machine architecture. Unfortunately these machines were neverbuilt but the aspect of VDM (the
Vienna Development Method ) that relatedto describing language semantics was rescued by writing [BJ78, BJ82]. A moredetailed account of this work can be found in [JA16, Ast19] and the step fromVDL to VDM is discussed in [Jon01].
Lesson
Again there was a damaging wall between language designers andthe formalisers; moreover, the same mistake was repeated on a formal descrip-tion of the machine architecture itself. In both cases the separation provedwasteful and far less effective than if there had been a more tightly knit struc-ture.Returning to the Baden-bei-Wien conference, Tony Hoare expressed uneaseabout all of the proposed techniques because he saw the need to leave someaspects of a language undefined. Obvious cases include features of a program-ming language that relate to implementations on specific machines. But Hoare’sinterjection was prescient because concurrent programs can legitimately yielddifferent results depending on the rate of progress of their separate threads.This observation led Hoare to develop an axiomatic approach [Hoa69] which iskey to reasoning about programs satisfying formal specifications.The challenge of describing –in an operational approach– concurrent pro-gramming languages with their inherent non-determinism was overcome by Gor-don Plotkin’s Structural Operational Semantics (SOS) [Plo81]. The history of formal semantic descriptions is covered in a recent doctoralthesis [Ast19] and more technical details are given in [JA16, JA18]. Formalisa-tion of the Spark-Ada language is discussed below in Section 4.
One obvious mode of transferring ideas from academic originators into industrialpractice is for the originators to act as consultants to practitioners. This offers away of overcoming the inevitable lack of knowledge of novel ideas in the receiving The source language of the compiler was to have been that of the evolving ISO standardand the ISO PL/I standardisation committee were still evolving the language. These lecture notes from 1981 were widely circulated and, thankfully, reprintedas [Plo04b]; they are accompanied by a useful commentary [Plo04a].
European Laboratories IntegratedProfessional Training Program (ELIPT) technical courses. A course based onthe 1980 VDM book was offered and taught by Derek Andrews and Cliff Jones.Management at the IBM Laboratory in Germany at B¨obligen made the decisionto enrol most of its active programmers on this course and employees attendedin more-or-less coherent project groups. A typical course would begin at a hotelin the
Schwarzwald with two weeks of lectures and writing exercises followed bya one week intensive workshop that initiated a formal description of a (simpli-fied version of) the product of interest to that group of engineers. (Managersclaimed that they were too busy to commit to this length of course and a shorter Management Awareness course had to be prepared.) There were several signifi-cant success stories: one that is described in an external publication is [BHP83].In contrast to this organised enrolment, the IBM development laboratory inthe UK at Hursley (Hampshire) simply let individuals from any project enrolon the ELIPT course in a fairly random way. This meant that when needed(see below) there was no critical mass of engineers who were all up to speed onVDM. The lesson here is that education needs to establish a cohort of peoplein the receiving organisation.To emphasise this lesson it is worth comparing with Harlan Mills’ success inIBM’s
Federal Systems Division (FSD). Mills persuaded the director of FSDto attend the first course which ensured that no intermediate managers couldclaim they were too busy to take part. There was a notion of “passing” thecourse. Effectively, most development engineers in FSD attended.Jones was by this time doing his belated Doctorate under supervision ofTony Hoare at Oxford University. Jean-Raymond Abrial arrived at the
Pro-gramming Research Group at the same time (1979) as Jones. Hoare arrangedthat the two shared an office and many interesting discussions were conductedwith Jones writing specifications in the VDM notation from [Jon80] and Abrialexperimenting with what was to become the Z notation (see below). Jones was involved in teaching of the first eight ASD courses; the majority took place inGermany, several in the UK and one in Italy. This is reported from Jones’ memories of several personal discussions with Mills.
Customer Information Control System which had evolvedfrom a customer’s 1968 program to be a full-blown IBM product that supportedon-line transaction processing and was used by major financial and retail organ-isations. In 1988 it consisted of well-over half a million lines of code in at leasttwo languages. It was believed at one time to be one of IBM’s most profitableprogram products.Because of his contacts with IBM Hursley and his on-going courses, Joneswas asked about ways to help the CICS team adopt formal specification. Start-ing in 1980 there were informal discussions that led in early 1981 to the sugges-tion of a contract with a university (i.e. not with single consultants). Formalmeetings between Hursley managers and Prof Hoare in the middle of 1981 re-sulted in a contract between IBM Hursley and Oxford University being in placeby year end. Ib Sørensen and Tim Clement were to work on the project from theOxford end; Hursley people included Pete Collins, John Wordsworth and PeterLupton; Hoare had asked that Jones worked as a consultant on the project and was joined by Rod Burstall of Edinburgh University in the same role.As mentioned above, Abrial was developing the ideas that coalesced into theZ specification notation and it is clear that the challenges presented in describ-ing CICS had an influence of this evolution. Key discussion partners also in-cluded Bernard Sufrin and Carroll Morgan. Ian Hayes joined the project in Jan-uary 1983 which was perhaps the key time for the development of Z’s so-called“schema calculus”. Hayes also went on to edit the first book on Z [Hay87]. Other Z books from around this time include [Spi88, Spi92, Wor92, MP93] andMike Spivey also programmed the first tool that type-checked Z specifications.A useful intermediate report on “CICS Experience with Z” is the (unre-stricted) Technical Report [CNS87]. The summary includes:“From the industry point of view, this work has demonstrated that: • provided that there is adequate education and support, a math-ematical notation such as Z can be used for software develop-ment in an industrial environment; • the use of formal methods changes the development process andbrings greater precision to earlier stages; • communication between development groups can be improved.”Jim Woodcock worked on the Oxford end of the collaboration from 1985–93. Inparticular he sorted out the logic underlying Z [WB92]; he also co-authored a See https://en.wikipedia.org/wiki/CICS. Jones had worked in Hursley 1965–68, 1970–73; the gaps filled by assignments to the IBMLab Vienna and IBM’s
European Systems Research Institute in Belgium. Having submitted his thesis in June 1981 Jones moved to a chair in Manchester Universitystarting August. The frustration at not having a stable reference document for Z in 1981 led to the con-struction of a spoof document with Sufrin’s name shown as author but actually comprising apastiche of other papers put together by other researchers at PRG . Post-Covid-19, I’ll put a scan of this on my web site.
Case Studies book [Hay92, Part-IV]contains five chapters (by Hayes and King) on details of the description of partsof CICS.A joint paper (Steve King – Oxford University’s
Programming ResearchGroup (PRG) and Iain Houston – IBM) [HK91] provides a strongly positiveassessment of the exercise in formalisation. It includes the figure reproducedhere in Fig. 1. This clearly supports the oft-claimed benefit of formal meth-ods that more problems are detected early resulting in significantly reducedproblems overall.Furthermore, the collaboration was recognised with a Queens Award forTechnological Achievement in 1992 to the Oxford PRG group and IBM Hursley.Despite all of these positive indicators, an informal reunion of Hayes, Hoare,Jones and Sørensen (kindly arranged by Jonathan Lawrence) in September 2011found little trace of the continued use of Z in IBM Hursley. This prompts an Jonathan Lawrence drew Jones’ attention to [HDNS96]. lessons from what is widely viewed as one of the successstories of formal methods deployment. • PRG staff funded by the collaborative contract had to spend significantamounts of time working on details of the CICS code; • not only were general courses on Z designed and given to IBM engineers,but additional “readers’ courses” were needed; • the presence of “internal supporters” was crucial at the time of strongestinteraction; • there was a period when IBM engineers and management were pushinghard for a recognised standard for the Z notation; • (and linked to the previous point) there was a perceived need for toolsthat helped creation, maintenance and analysis of Z documents; • management support is crucial (see above on Mills and IBM FSD) — theattitude of IBM Hursley management ranged from supportive to antago-nistic; • King and Jones ascribe the drift away form the use of Z in Hursley downto people: there were probably not quite enough internal supporters, somemoved on to other roles; less positive managers became responsible for de-cisions which affected the selection of methods to be used in later releases.An extreme example of separation of both development from verificationand the respective teams concerns the software for the Sizewell-B nuclear reac-tor. Sizewell-B was the first nuclear reactor in the UK to have a programmableprimary protection system (PPS). The regulator (Nuclear Installations Inspec-torate) decided that it was necessary for the PPS software to be formally verified.The software had been developed by the US company Westinghouse without aformal specification. It was written in PL/M-86 with some ASM86 and smallamounts of PL/M-51 and ASM51; there were about 100,000 lines of uniqueexecutable code. Originally, there were two specification documents, a highlevel Software Design Requirements (SDR) and a more detailed Software De-sign Specification (SDS).The chosen route to (partial) formal verification was to manually translate(in the UK) the SDR and SDS into a mathematical specification and to write atranslator from PL/M-86 to IL, the Intermediate Language used as input to theMALPAS static analysis tool. This work started in January 1989 and completedin 1993. The MALPAS analysis project team grew to more than 80 and theproject cost GBP 7 million. A review by Nuclear Electric [Fen95] concluded An international standard for Z was approved in 2002. Fuzz . . . Jones’ notes made at the time record one fairly senior manager responding to “it’s justmathematical notation” with “I hate mathematics”. This clearly reinforces the lessons that a separation of developers and for-malists is damaging in general and that leaving verification to the late stages ofdevelopment is both lengthy and difficult.An extremely important vector of formal methods research and deploymentcentres around Jean-Raymond Abrial; this story links to the UK (and involve-ment of the current authors) but is actually more international. After layingthe groundwork of what became the Z notation, Abrial returned to France andacted as an independent consultant. He not only made a fundamental shift tocreate the B-method [Abr96] but he also developed a “B-tool” under contractto BP where Ib Sørensen had moved from Oxford after the CICS work. Subse-quently, Abrial developed a completely new “Atelier-B” which was subsequentlysupported by the French formal methods companies ClearSy and Systerel. Thenotation and tool were used in the important development of the software ofMetro line 14 for RATP [DM94]Abrial is a fascinating person who deserves a full biographical article. He isself-critical in the best possible way and has made several complete rethinks ofhis ideas. His next step was strongly influenced by books on Refinement Calculus by Carroll Morgan [Mor94] on the one hand and Ralph Back and Joakim vonWright [BvW98] on the other. Abrial’s
Event-B is described in [Abr10b]. Toolsupport was designed and built in an EU-funded project known as
Rodin whichwas led by Newcastle University. The central tool activity was undertaken byAbrial, Laurent Voison and Steffan Hallerstede at ETH Zurich in the chairof David Basin. A subsequent EU-funded project (
Deploy ) was again ledby Newcastle and this time involved four industrial companies. As its namesuggests, the emphasis here was on deployment of Event-B and the
Rodin Tools .The most accessible description is [RT13].Although neither of the current authors were directly involved, it wouldbe remiss in this section not to mention the work on hardware verificationled by Mike Gordon of Cambridge — Larry Paulson wrote the Royal Societyreport [Pau18]. Furthermore, the Oxford work on exploiting CSP by providingthe FDR tool is to be described in a paper being written by Bill Roscoe.
This section considers the use of formal methods in some organisations thatmade them a more integral part of their development process. Ward [War93] goes into detail on the MALPAS use of
Compliance Analysis and explainsthat there was no choice but to develop pre and post condition specifications of componentsbottom up from the code. This book contains a generous acknowledgement to the influence of VDM on B. Other partners included . . . and the Advisory Group was chaired by Thomas. An invaluable resource on theorem proving efforts is [Mac01]. using a formal hardware design and development system com-prising the Electronic Logic Language (ELLA), developed by Morison and laterreleased under a public license [MC93] supported by tools for design transfor-mation, symbolic execution and formal verification. The SWURCC team im-pressed RSRE sufficiently that in addition to Algol68, they were commissionedto support and market ELLA.SWURCC had built a reputation for software quality that led to them beinginvited to join a consortium ( Augusta ) funded by the UK Department of Tradeand Industry to investigate the use of the new Ada programming language.Augusta was led by Tim Denvir of STC’s telecommunications laboratory STL,who recalls that“Our report, delivered in September 1981, took a few example prob-lems, expressed a design following several different methods, anddeveloped implementations from each in Ada. We also did a litera-ture study of many more design methods and of developers. Amongthe mostly structured methods (such as JSD) we used and/or con-sidered CCS and VDM” [Den17].The SWURCC team became the kernel of Praxis, set up by Thomas andBean in 1983, transferring the ELLA and Algol68 compiler support and other See also [Mac01, Chap 7] on VIPER; Donald Mackenzie’s whole book contains a thoroughand deep analysis of the concept and limitations of formal proofs about software. When the management changed at STC, Tim Denvir and Mel Jacksonjoined Praxis, bringing with them a European Commission contract to spec-ify the
Portable Common Tools Environment (PCTE) in an extended versionof VDM (see [Mid89, Mid90] for VVSL). Praxis was a fertile environment forthe introduction of formal methods. They abandoned SADT and adoptedVDM [Jon80] and later Z [Spi89] as specification and design methods, par-ticularly for safety or security critical projects such as the protection system fora radiotherapy machine, the certification authority to support the MONDEXsmart card [HC02a], a system (SHOLIS) to support the landing of helicopterson naval ships [KHCP00], and a system (CDIS) [Hal96] to support the UKNational Air Traffic Services (NATS) controllers handling aircraft for the fiveLondon airports.Static data flow and program analysis tools were developed at RSRE basedon research by Bob Philips of RSRE. The work was declassified in the 1970s sothat it could be used on civilian safety critical projects, resulting in two com-mercially available products, the Malvern Program Analysis Suite (MALPAS)and the Southampton Program Analysis Development Environment (SPADE,which became SPARK, see below) developed by a team led by Bernard Carr´ewho were doing research in graph theory at Southampton University. BothMALPAS and SPARK have been used in the development and verification of arange of safety critical and security critical systems.SPADE was originally developed to analyse and verify programs written ina small subset of Pascal but Ada was then chosen as the foundation for futurework. An Ada subset (the SPADE Ada Kernel or SPARK) was defined infor-mally by Bernard Carr´e and Trevor Jennings in 1987 and more formally definedusing a variant of Z in
The Formal Semantics of SPARK [MO94]. Bernard Carr´e lessons from Praxis experience with formal methods: • Praxis developed its own courses to teach its software engineers to useVDM and Z. The course lasted four days, preceded by a single day teach-ing discrete mathematics. This was found to be enough for computerscience graduates to be able to read and start understanding a formalspecification, though the ability to write good Z developed over a periodof working on a project with access to experienced Z practitioners. • formal methods had to be fitted into an overall development process andcombined with other techniques, for example prototyping for user inter-faces, and Data Flow Diagrams for process design. Further, no single for-mal method covers every aspect so Praxis had to use different techniquesfor functionality and concurrency and find ways of integrating these. Theyalso had to write specifications at the system level and at the design leveland show that the design was consistent with the system specification. Us-ing high-level, set-theory based languages meant that there were almostno tools available which limited their ability to generate executable proto-types, or carry out proofs, model checking or automatic code generation. • The benefits of this approach were a systematic, rigorous and traceabledevelopment leading to systems with few defects in service [Ame02, Sys06,HC02b, KHCP00]; as Praxis improved their understanding of how to dothis, they steadily drove down defect levels [HC02a, HC02c, Hal05]. • Commercially, using formal methods had drawbacks as well as benefits: – Customers were nervous about formal methods; the US
National Se-curity Agency asked Praxis to demonstrate the practicality of formalmethods by developing an experimental system to control a secureenclave. The Tokeneer project was a success [BJW06] in that theNSA were unable to find any faults in the software, Praxis were ableto train two NSA interns to extend the system, and the NSA said thatthe productivity of the Praxis team was the highest they had everexperienced but somehow this did not lead to sales of the SPARKtoolset or to further NSA projects. The NSA later agreed that all12f the Tokeneer specification, design and development could be pub-licly released including all the tools and proofs, so that anyonecould download and study the project, experiment with the prooftechnology and see whether other tools might reveal defects that hadnot been found. The various follow-on projects and experiments havebeen summarised in a book chapter by Jim Woodcock [WGC10]. – on the CDIS project for NATS, Praxis developed a formal VDMspecification during the bid, to determine exactly what the customermeant by their requirements. This led to over 100 requests to NATSfor clarifications but it enabled Praxis to feel comfortable bidding fora large project whose cost exceeded Praxis’ annual turnover andto contract to repair at no charge any major faults that developedover the following five years. NATS later said that CDIS had beenby far the easiest system to integrate that they had experienced andits stability was so good that they had to ask Praxis after a few yearsto retrain their staff in how to restart CDIS because it had failed sorarely that they were unsure how to do it. – The functional specification for CDIS and subsystem specificationsfor the two component parts (server and workstation) were expressedin VVSL but the user interface used state transition diagrams and, formodelling concurrency used CSP and for the LAN design used Mil-ner’s CCS. Praxis’ technical architect, Anthony Hall, has describedand explained the choice of methods [Hal96]. With NATS agreement,Praxis made the CDIS code and project records available for analy-sis by two academic researchers. Their conclusions [PH97] were thatformal methods can contribute to achieving very reliable code (butwith many reservations, for which see the cited paper). – Formal methods projects of this sort need considerable work beforeany benefits are seen, which made it difficult to convince potentialcustomers to invest in such projects. – The lack of support tools can be a major problem for the use of for-mal methods that only address the specification stage. Some form ofprototyping is very important so that clients can validate the speci-fication and request changes while it is still relatively inexpensive tochange it. – System specifications need to be understood by domain experts aswell as by computer scientists. Praxis rewrote the informal specifi-cations using and including the formal specification so that it couldbe understood by clients. Preparing a formal specification facilitateswriting a better structured, clearer and shorter specification in nat-ural language. Thomas remarked to Jones over a dinner in Brussels that he had “bet the company onVDM”. Praxis offered some free warranties for projects that had a formalspecification agreed with the client, as this made it possible to dis-tinguish between errors and specification changes. This was less com-mercially risky than it might seem, because the warranty would bevoided if the client changed the software themselves or asked anothercompany to implement new features. – NATS subsequently contracted Altran UK to develop a new ATC sys-tem iFACTS that was successfully delivered using Z and SPARK.Thomas was present at one NATS meeting before this contract wasawarded where the objection was raised that if NATS presented theirregulator (The UK Civil Aviation Authority ) with SPARK code andformal proofs as part of the evidence for the safety of iFACTS, therewas a risk that the CAA would always require such strong evidencein future!
Apart from the lessons for those wishing to deploy formal methods, there areexogenous factors that have affected the growth and adoption of formal methods.This section covers some of these influences.
In the UK, funding from its research councils has been supportive to both theunderlying research and deployment of formal methods. In fact, it can be arguedthat such funding was pivotal in the 1970s: viewed from Manchester University(where Jones was at that time) it facilitated the creation of a world class formalmethods activity.It is also worth noting that BP’s “Future ventures” programme providedfunding for activities in Edinburgh Univerity’s Laboratory for Foundations ofComputer Science (and for Edsger Dijkstra).The body principally responsible for funding computer science research inU.K. Universities was then known as SERC. The Computer Science commit-tee, recognising the importance of distributed computing as a research area,appointed a panel in June 1976 under the chairmanship of Prof. I. Barron, toconsider what action was necessary to encourage, coordinate or direct researchin Distributed Computing. The Distributed Computing Systems programmestarted in the academic year 1977-78. DCS was the first attempt by SERC toestablish a long term, extensive, coordinated programme of research in Infor-mation Technology. The Technical Co-ordinators of DCS were Bob Hopgood(1977-79), Rob Witty (1979-1981) and David Duce (1981-1984). The primary https://nats.aero/blog/2013/07/how-technology-is-transforming-air-traffic-management/. Rob Witty (private communication) pointed out that specific support for formal methodswas an outgrowth of the funding for the
Distributed Computing Systems programme. • Theory and Languages: An adequate theoretical basis for DistributedComputing Systems. • Resource Management: Distribution of control, allocation, scheduling andorganization. • Architecture. • Operational Attributes: Particularly reliability and performance. • Design, Implementation and Application: Hardware and software tech-niques for development and implementation.A major theme in DCS was concerned with theories of parallel computa-tion and with the development of notations and techniques for specifying andverifying such systems. The UK Alvey Programme ran from 1983–87 and also invested in formalmethods. The focus of the Alvey programme [OO90] was pre-competitiveadvanced information technology research. It comprised four areas that seemedparticularly relevant at the time: • Software Engineering (led by David Talbot from ICL with Rob Witty fromInformatics as his Deputy, who brought the focus on formal methods fromDCS). • Intelligent Knowledge Based Systems • Man Machine Interaction • Advanced Microelectronics (VLSI Design)Research was a collaboration between academia, government and industry; itwas directed into important areas and coordinated and the funding was sub-stantial, GBP 350M at 1982 prices. The Programme put together 210 projectslasting on average 3 years and involving 2500 people at its peak. See and See for example Alvey News SE2/18 that contains a list of some of the projects fundedby the Software Engineering Programme.
Software Engineering funded Manchester Uni-versity and ICL to construct an integrated project support environment dubbed IPSE 2.5 . Researchers at Manchester and EPSRC’s own Rutherford Lab deliv-ered a theorem proving assistant
Mural [JJLM91] which was licensed to
WinfrithAtomic Energy Establishment .Another activity under the Alvey programme led to the creation of a hand-book of formal methods. The contents attempted to identify areas of applica-bility for notations such as VDM, Z, CSP and CCS.
Cliff says:
I hope to find some trace of this when, post-Covid, I can getback into the university!As recently as the 2010s, formal methods was still identified as an area forgrowth of EPSRC funding.Funding from the various European Union research framework programmeshas also been a significant aid to formal methods research and deployment.Again focussing on items where the authors have first-hand knowledge, one ofthe longest-lasting impacts started with funding of an activity called
VDM-Europe : meetings in Brussels of experts were supported for several years andled to the first symposium of
VDM-Europe [BJMN87]. These conferences mor-phed into FM-E which not only organises a highly-rated symposium at roughly18-monthly intervals but has also held two World Congresses [WWD99, tBMO19]and has widened its venues to North America, Singapore (and the 2021 eventis planned for China). The question of how much the adoption of formal methods is influenced by theavailability of tool support is interesting but is not universally agreed. Earlyon, large formal descriptions were constructed with minimal tool support. Asignificant example is the formal description of PL/I from the IBM ViennaLab. It is probably fair to say that the risks of introducing inconsistencies arefar higher when a document is revised than when it is first constructed. It ishowever clearly short-sighted not to at least syntax and type check and largeblock of formulae.The question is how much further one can go without the tool support be-coming an end in itself and possibly even distracting from the thought processthat is crucial to the construction of an abstract model. Jim Horning (privatecommunication) captured one of the reservations about tools with his phrase This project came close to non-submission when, having crafted a neat collaborationof three industrial organisations (STL, IDEC and ICL), Jones was informed that they weremerging and that the combination of their individual intended commitments was not defensibleto a single board of directors. Brian Warboys then of ICL steered a tense period of revisionsat the eleventh hour. Jim Woodcock tackled Jones about widening the remit of VDM-Europe to include otherspecification languages. Tokeneer study mentioned in Section 4 supports this view; Figure 1 providesevidence that formalism used to front-load thinking pays off; a similar resultcan be seen in the dual-track study reported in [BFL96].Many companies are willing to buy tools as they see this as a quick fix butfail to recognise that tools exist to support methods and the main investmenthas to be in adopting the methods.The view of “mental tools” in no way removes the need for tool supportbut it does moderate the extent to which the tool can be allowed to become themaster of the method. For example, a large formal text might be regarded as anobvious input to a theorem proving system. Unfortunately, there are few successstories of such efforts. The reason would appear to be that theorem provingsystems not only require learning another formalism but that their modes ofinteraction distract from thinking about the application in hand. There arenumerous stories of formal machine-checked proofs that do not actually capturewhat the user intended to establish.Examples of tools that offer fairly direct support for established specifica-tion languages include a VDM tool from Adelard, the IFAD Toolset (also forVDM) that has subsequently been developed into an open-source Overture tool.Probably the most significant set of tools come from Abrial with the Rodin toolsupport for Event-B being the most recent. One way in which the use of formalism could be encouraged is via standards. InMay 1989 the UK Ministry of Defence (MoD) issued Interim Defence Standard0055 Requirements for the procurement of safety critical software in defenceequipment for comment. The interim standard required that safety critical The US work starting with the Boyer-Moore prover through to ACL/2 is a notable achieve-ment — see [Moo19]. The work of the Adelard company would justify a paper of its own. for example theirdevelopment of the “Dust Expert” software is reported in [CCFJ99]. retaining the requirements for formalmethods and including several examples from the SHOLIS project. Def Stan00-55 issue 3 recommended the use of civil standards such as RTSA DO-178,ISO 61508 and RTSA DO-254.IEC 61508 is the international standard for functional safety of programmableelectronic systems. It requires that each safety function has a Safety integrityLevel that defines the allowable probability of failure: for continuous controlof the most safety critical function (SIL4) the allowable probability must notexceed 10 − /hour. The standard recommends the use of formal methods for aSIL4 software safety function but does not mandate their use. In successive re-visions of the standard, major European companies have repeatedly frustratedattempts to make formal methods mandatory for SIL 4 software.Returning to Harlan Mills and what he achieved in IBM’s Federal SystemsDivision (and again relying on Jones’ memory of personal discussions with Mills)perhaps he had the best approach to standards. At one point in time, a standardfor software developers stated that programmers should accompany loops withan indication of why they were claimed to achieve their aim; there was not amandated style for such annotations but the document did offer an example ofa style that would serve.There is of course also the question of standards for the formalism itself. Itwas mentioned in Section 3 that, during the CICS effort, there was a requestfrom IBM for a standard for the Z notation itself. This is an understandable wishin that it opens up the possibility of sourcing tools and expertise from differentorganisations. In fact, VDM was in 1996 the first formal method notation toget an ISO standard the Z standard followed in 2002. Our emphasis in this paper has been on lessons that can be derived from earlyattempts to apply research on formal methods in significant software devel-opment. This should in no way be seen as expressing reservations about thepotential of the ideas and Section 6.1 points to two important recent successstories. If we cannot learn from earlier difficulties, no progress is made. Ratherthan re-list all of the lessons noted earlier in the paper, Section 6.2 pinpoints afew key messages. .1 More recent work Recent attempts to use formal methods in industry include those spearheadedby Peter O’Hearn at Facebook and Byron Cook at
Amazon Web Services (AWS).O’Hearn made major contributions to
Concurrent Separation Logic (e.g. [O’H07])and went on to form, with colleagues,
Monoidics which was acquired by Face-book in 2013. The group has worked inside Facebook and [O’H15] reportsconsiderable success in creating tools (see [DFLO19]) that are used in the stan-dard development cycle by Facebook engineers. In a private conversation withJones, O’Hearn attributed the positive adoption by practicing engineers bothto the creation of apposite tools and the fact that the the general knowledgeof fundamental computer science ideas is much more widespread now than inthe attempts reported on earlier in this paper that mainly date from the lastcentury.Byron Cook’s [Coo18] is one of a sequence of papers reporting on applicationof formal methods at AWS; his recorded keynote talk at FLoC-18 in Oxford isinspirational and, related to the lessons about management commitment, evenmore impressive are the talks from senior managers at AWS. The contributions of UK researchers to the fundamental ideas that have shownhow formal concepts and notations can be used in the description and and devel-opment of software are significant. Rather than list and attribute the scientificsource material, we have in this paper identified some significant attempts todeploy the theory into practical environments. As indicated at the beginningof the paper, we have mainly reported on deployments of which we have firsthand knowledge. These close encounters have made it possible to analyse thedifficulties that were experienced.Probably the most important single difficulty that complicated early deploy-ments was the relatively small number of people available in receiving organi-sations who had acquaintance with a broad knowledge of theoretical concepts.A short course on one or another specific notation does not equip someone toapply that notation to abstract away from the details of potential implemen-tations; similarly, nor does being shown a few examples of proofs convey thefundamental idea of recording a convincing correctness argument. The morerecent experiences from major companies like Microsoft, Amazon and Facebooksuggest that the general educational environment now provides a far better basisthan was available last century.The attitude and commitment of management is clearly related and also ofmajor importance. The graph in Figure 1 indicates a challenge for managers whoonly feel comfortable when they can “weigh the code”: just as in all engineeringendeavours, care and thought early in development clearly pays off later but post hoc formalisation. This division appears to have beena source of problems in many of the early attempts to gain benefit from usingformal methods.Another issue is the choice of project — perhaps crucially the first project.A significant success story that lays about as far from our stated geographicfocus as can be is the work in Australia at CSIRO Data61 (previously known asNICTA). Gerwin Klein and his team recognised the importance of microkernelsbecause weaknesses here open any software built on top of them to subversion. Arecent papers on sel4 is [HKA20] and earlier publications can be traced from itsreferences. In today’s world where almost everything depends on software, it issometimes a financial aspect that identifies a development as “business critical”.There is, of course, the class of “safety critical” systems that comprised earlydeployments of formal methods. What the sel4 exercise is a reminder of is thatunderlying software can provide a Trojan Horse on which reliance should onlybe proportional to its demonstrated trustworthiness.One closing (and perhaps uncomfortable for the current authors) lesson wasmentioned by Jonathan Lawrence when he kindly reviewed the lessons from theIBM CICS project: he suggested that one should “not to be too ambitious”.It would be legitimate to ask whether some of the early deployments witheredon the vine precisely because researchers wanted to see the full extent of theirresearch put into practice even if the receiving organisation was unready.
Acknowledgements
The authors are indebted for input on aspects of this paper from: • IBM/CICS: Tim Clement, Ian Hayes, Steve King, Jonathan Lawrence andCarroll Morgan • STL: Tim Denvir • Praxis and Altran UK: David Bean, Roderick Chapman, Anthony Halland Martyn Ould • UK Research Councils: David Duce and Rob WittyTroy Astarte offered extremely valuable comments on a draft of the paper. Allremaining errors and opinions are however the responsibility of the authors.Jones gratefully acknowledges the support for his research of grant RPG-2019-020 from the Leverhulme Foundation.20 eferences [Abr96] J.-R. Abrial.
The B-Book: Assigning programs to meanings . Cam-bridge University Press, 1996.[Abr10a] J.-R. Abrial.
Modeling in Event-B: System and Software Engineer-ing . Cambridge University Press, New York, NY, USA, 2010.[Abr10b] J.-R. Abrial.
The Event-B Book . Cambridge University Press, Cam-bridge, UK, 2010.[Ame02] Peter Amey. Correctness by construction: Better can also becheaper.
CrossTalk: the Journal of Defense Software Engineering ,2:24–28, 2002.[AO19] Krzysztof R. Apt and Ernst-R¨udiger Olderog. Fifty years of Hoare’slogic.
Formal Aspects of Computing , 31(6):751–807, 2019.[Ast19] Troy K. Astarte.
Formalising Meaning: a History of ProgrammingLanguage Semantics . PhD thesis, Newcastle University, June 2019.[BBG +
60] John W. Backus, Friedrich L. Bauer, Julien Green, Charles Katz,John McCarthy, Peter Naur, Alan J. Perlis, Heinz Rutishauser,Klaus Samelson, Bernard Vauquois, et al. Report on the algorithmiclanguage ALGOL 60.
Numerische Mathematik , 2(1):106–136, 1960.[BFL96] TM Brookes, John S Fitzgerald, and Peter Gorm Larsen. Formaland informal specifications of a secure system component: final re-sults in a comparative study. In
International Symposium of FormalMethods Europe , pages 214–227. Springer, 1996.[BGJ06] D. Besnard, C. Gacek, and C. B. Jones, editors.
Structure forDependability: Computer-Based Systems from an InterdisciplinaryPerspective . Springer, 2006.[BHP83] F Beichter, Otthein Herzog, and Heiko Petzsch. Slan-4: a languagefor the specification and design of large software systems.
IBM jour-nal of research and development , 27(6):558–576, 1983.[BJ78] D. Bjørner and C. B. Jones, editors.
The Vienna DevelopmentMethod: The Meta-Language , volume 61 of
LNCS . Springer-Verlag,1978.[BJ82] Dines Bjørner and Cliff B. Jones, editors.
Formal Specification andSoftware Development . Prentice Hall International, 1982.[BJMN87] Dines Bjørner, C. B. Jones, M. Mac an Airchinnigh, and E. J.Neuhold, editors.
VDM – A Formal Definition at Work , volume252 of
LNCS . Springer-Verlag, 1987.21BJW06] J. Barnes, R. Johnson, and J. C. Widmaier. Engineering the Toke-neer enclave protection software.
Proceedings of IEEE InternationalSymposium on Secure Software Engineering , 2006.[BvW98] R.-J. R. Back and J. von Wright.
Refinement Calculus: A SystematicIntroduction . Springer, New York, 1998.[CCFJ99] Tim Clement, Ian Cottam, Peter Froome, and Claire Jones. The de-velopment of a commercial “shrink-wrapped application” to safetyintegrity level 2: The dust-expert TM story. In International Confer-ence on Computer Safety, Reliability, and Security , pages 216–225.Springer, 1999.[CK85] Martin Campbell-Kelly. Christopher Strachey, 1916-1975: A bio-graphical note.
IEEE Annals of the History of Computing , 1(7):19–42, 1985.[CNS87] B.P. Collins, J.E. Nicholls, and I.H. Sorensen. Introducing formalmethods: The CICS experience with Z. Technical Report TR12.060,IBM Hursley Laboratory, December 1987.[Coh88] A. Cohn. A proof of correctness of the VIPER microprocessor: Thefirst level. In G. Birtwistle and P.A. Subrahmanyam, editors,
VLSISpecification, Verification and Synthesis , volume 35. Kluwer, 1988.[Coo18] Byron Cook. Formal reasoning about the security of amazon webservices. In
International Conference on Computer Aided Verifica-tion , pages 38–47. Springer, 2018.[CS14] R. Chapman and F. Schanda. Are we there yet? 20 years of in-dustrial theorem proving with SPARK. In
Proceedings of Interac-tive Theorem Proving (ITP) , volume 8558 of
LNCS , pages 17–26.Springer, 2014.[Den17] T. Denvir. Fifty years of formal methods in software engineering.
FACTS , 2017.[DFLO19] Dino Distefano, Manuel F¨ahndrich, Francesco Logozzo, and Pe-ter W. O’Hearn. Scaling static analyses at Facebook.
CACM ,62(8):62–70, 2019.[DM94] Babak Dehbonei and Fernando Mejia. Formal methods in the rail-ways signalling industry. In
International Symposium of FormalMethods Europe , volume 873 of lncs , pages 26–34. Springer, 1994.[Fen95] T Fenney. Sizewell B Primary Protection System. an assessment ofthe confirmatory review processes. Technical report, Nuclear Elec-tric plc, 1995. 22GMW79] M. Gordon, R. Milner, and C. Wadsworth.
Edinburgh LCF , vol-ume 78 of
Lecture Notes in Computer Science . Springer-Verlag,1979.[Gor00] Mike Gordon. From LCF to HOL: a short history. In
Proof, lan-guage, and interaction , pages 169–186, 2000.[Hal96] Anthony Hall. Using formal methods to develop an ATC informationsystem.
IEEE Software , 13(2):66–76, 1996. Hard copy.[Hal05] Anthony Hall. Realising the benefits of formal methods. In
FormalMethods and Software Engineering , volume 3785 of
LNCS , pages1–4. Springer, 2005.[Hay87] I. J. Hayes, editor.
Specification Case Studies . Prentice Hall Inter-national, 1987.[Hay92] I. J. Hayes, editor.
Specification Case Studies . Prentice Hall Inter-national, second edition, 1992.[Hay93] Ian Hayes, editor.
Specification Case Studies . Prentice Hall Inter-national, Englewood Cliffs, N.J., USA, second edition, 1993.[HC02a] A. Hall and R. Chapman. Correctness by construction: building acommercial secure system.
Software , 19(1):18–25, 2002.[HC02b] Anthony Hall and Roderick Chapman. Correctness by construction:Developing a commercial secure system.
IEEE software , 19(1):18–25, 2002.[HC02c] Anthony Hall and Roderick Chapman. Correctness by construction:Developing a commercial secure system.
IEEE Software , pages 18–25, 2002.[HDNS96] Jonathan Hoare, Jeremy Dick, Dave Neilson, and Ib Sørensen. Ap-plying the B technologies to CICS. In
International Symposium ofFormal Methods Europe , pages 74–84. Springer, 1996.[HK91] Iain Houston and Steve King. CICS project report experiences andresults from the use of z in ibm. In
International Symposium ofVDM Europe , pages 588–596. Springer, 1991.[HKA20] Gernot Heiser, Gerwin Klein, and June Andronick. sel4 in australia:from research to real-world trustworthy systems. cacm , 63(4):72–75,2020.[Hoa69] C. A. R. Hoare. An axiomatic basis for computer programming.
Communications of the ACM , 12(10):576–580, 1969.[Hoa85] C. A. R. Hoare.
Communicating Sequential Processes . Prentice-Hall,1985. 23JA16] Cliff B. Jones and Troy K. Astarte. An Exegesis of Four Formal De-scriptions of ALGOL 60. Technical Report CS-TR-1498, NewcastleUniversity School of Computer Science, September 2016.[JA18] Cliff B. Jones and Troy K. Astarte. Challenges for semantic descrip-tion: comparing responses from the main approaches. In Jonathan P.Bowen and Zhiming Liu, editors,
Proceedings of the 3rd School onEngineering Trustworthy Software Systems , volume 11174 of
LectureNotes in Computer Science , pages 176–217, 2018.[Jac75] Michael Jackson.
Principles of Program Design . Academic Press,1975.[Jac00] Michael Jackson.
Problem Frames: Analyzing and structuring soft-ware development problems . Addison-Wesley, 2000.[JJLM91] C. B. Jones, K. D. Jones, P. A. Lindsay, and R. Moore. mural: AFormal Development Support System . Springer-Verlag, 1991.[Jon71] C. B. Jones. Development of correct programs: An example basedon Earley’s recogniser. Technical Report TN 9000, IBM Laboratory,Hursley, April 1971.[Jon73] C. B. Jones. Formal development of programs. Technical Report12.117, IBM Laboratory Hursley, June 1973.[Jon80] C. B. Jones.
Software Development: A Rigorous Approach . PrenticeHall International, Englewood Cliffs, N.J., USA, 1980.[Jon01] C. B. Jones. The transition from VDL to VDM.
Journal of UniversalComputer Science , 7(8):631–640, 2001.[JPRW03] Cliff Jones, Panos Periorellis, Alexander Romanovsky, and IanWelch. Structured handling of on-line interface upgrades in inte-grating dependable systems of systems. In N. Guelfi, E. Astesiano,and G. Reggio, editors,
FIDGI 2002 , volume 2604 of
LNCS , pages73–86. Springer Verlag, 2003.[KHCP00] Steve King, Jonathan Hammond, Rod Chapman, and Andy Pryor.Is proof more cost-effective than testing?
IEEE Transactions onsoftware Engineering , 26(8):675–686, 2000.[LW69] Peter Lucas and Kurt Walk. On the formal description of PL/I.
Annual Review in Automatic Programming , 6:105–182, 1969.[Mac01] Donald MacKenzie.
Mechanizing Proof: Computing, Risk, andTrust . MIT Press, 2001.[MC93] J.D. Morison and A.S. Clarke.
Ella 2000: A Language for ElectronicSystem Design . McGraw Hill, 1993.24McC66] John McCarthy. A formal description of a subset of ALGOL. In
For-mal Language Description Languages for Computer Programming ,pages 1–12. North-Holland, 1966.[Mid89] Cornelis Adam Middelburg. Vvsl: A language for structured vdmspecifications.
Formal aspects of computing , 1(1):115–135, 1989.[Mid90] C. A. Middelburg.
Syntax and Semantics of VVSL: A Languagefor Structured VDM Specifications . PhD thesis, PTT Research, Lei-dschendam, Department of Applied Computer Science, September1990.[Mil80] R. Milner.
A Calculus of Communicating Systems , volume 92 of
Lecture Notes in Computer Science . Springer-Verlag, 1980.[MO94] Marsh and O’Neil. The formal semantics of SPARK, 1994.[Moo19] J. Strother Moore. Milestones from the Pure Lisp theorem proverto ACL2.
Formal Aspects of Computing , 31(6):699–732, 2019.[Mor94] C. C. Morgan.
Programming from Specifications . Prentice Hall,second edition, 1994.[MP93] Mike McMorran and Steve Powell.
Z Guide for Beginners . AlfredWaller Limited, 1993.[O’H07] P. W. O’Hearn. Resources, concurrency and local reasoning.
Theo-retical Computer Science , 375(1-3):271–307, May 2007.[O’H15] Peter O’Hearn. From categorical logic to Facebook engineering. In
Logic in Computer Science (LICS), 2015 30th Annual ACM/IEEESymposium on , pages 17–20. IEEE, 2015.[OO90] Brian Oakley and Kenneth Owen.
Alvey: Britain’s strategic com-puting initiative . MIT press, 1990.[Pau18] Lawrence C Paulson. Michael John Caldwell Gordon (FRS 1994), 28February 1948–22 August 2017. arXiv preprint arXiv:1806.04002 ,2018. Royal Society obituary.[Pei91] Charles Sanders Peirce.
Peirce on signs: Writings on semiotic . UNCPress Books, 1991.[PH97] S.L. Pfleeger and L. Hatton. Investigating the influence of formalmethods.
Computer , pages 33–43, 1997.[Plo81] G. D. Plotkin. A structural approach to operational semantics. Tech-nical Report DAIMI FN-19, Aarhus University, 1981.[Plo04a] Gordon D. Plotkin. The origins of structural operational seman-tics.
Journal of Logic and Algebraic Programming , 60–61:3–15, July–December 2004. 25Plo04b] Gordon D. Plotkin. A structural approach to operational semantics.
Journal of Logic and Algebraic Programming , 60–61:17–139, July–December 2004.[PNW19] Lawrence C. Paulson, Tobias Nipkow, and Makarius Wenzel. FromLCF to Isabelle/HOL.
Formal Aspects of Computing , 31(6):675–698,2019.[Ros77] D.T. Ross. Structured analysis (sa): A language for communicatingideas.
IEEE Transactions on Software Engineering , SE-3(1):16–34,1977.[RT13] Alexander Romanovsky and Martyn Thomas.
Industrial deploymentof system engineering methods . Springer, 2013.[Spi88] J. M. Spivey.
Understanding Z—A Specification Language and itsFormal Semantics . Cambridge Tracts in Computer Science 3. Cam-bridge University Press, 1988.[Spi89] J. M. Spivey.
The Z Notation: A Reference Manual . Prentice HallInternational, 1989.[Spi92] J.M. Spivey.
The Z Notation: A Reference Manual . Prentice HallInternational, second edition, 1992.[Ste66] T. B. Steel.
Formal Language Description Languages for ComputerProgramming . North-Holland, 1966.[Sys06] Praxis Critical Systems. Spark: A successful contribution to theLockheed C130-J Hercules II. Praxis report, December 2006.[tBMO19] Maurice H. ter Beek, Annabelle McIver, and Jos´e N. Oliveira, edi-tors.
Formal Methods - The Next 30 Years - Third World Congress,FM 2019, Porto, Portugal, October 7-11, 2019, Proceedings , volume11800 of lncs . Springer, 2019.[Tho93] Martyn Thomas. The industrial use of formal methods.
Micropro-cessors and Microsystems , 17(1):31–36, 1993.[War93] NJ Ward. The rigorous retrospective static analysis of the Sizewell‘B’ Primary Protection System software. In
SAFECOMP’93 , pages171–181. Springer, 1993.[WB83] P.M. Woodward and S.G. Bond.
Guide to Algol 68 for users of RSSystems . Edward Arnold, 1983.[WB92] J.C.P. Woodcock and S.M. Brien. W: a logic for Z. In
Z UserWorkshop, York 1991 , pages 77–96. Springer, 1992. Hard copy.[WD96] Jim Woodcock and Jim Davies.
Using Z: Specification Refinementand Proof . Prentice Hall International, 1996.26WGC10] J. Woodcock, E. G¨okce Aydal, and R. Chapman. The Tokeneerexperiments. In C. B. Jones et al, editor,
Reflections on the Workon C. A. R. Hoare , pages 405–430. Springer, 2010.[Wir77] Niklaus Wirth. What can we do about the unnecessary diversity ofnotation for syntactic definitions?
Commun. ACM , 20(11):822–823,1977.[WLBF09] J. Woodcock, P. G. Larsen, J. Bicarregui, and J. Fitzgerald. Formalmethods: Practice and experience.
ACM Computing Surveys , 41(4),Oct 2009.[WMC17] N. White, S. Matthews, and R. Chapman. Formal verification: willthe seedling ever flower?
Phil Trans R Soc A , 375(2104), 2017.[Wor92] J. B. Wordsworth.
Software Development with Z . Addison-Wesley,1992.[WWD99] J.M. Wing, J. Woodcock, and J. Davies, editors.