Sarvani S. Vakkalanka

Formal verification of practical MPI programs

This paper considers the problem of formal verification of MPI programs operating under a fixed test harness for safety properties without building verification models. In our approach, we directly model-check the MPI/C source code, executing its interleavings with the help of a verification scheduler. Unfortunately, the total feasible number of interleavings is exponential, and impractical to examine even for our modest goals. Our earlier publications formalized and implemented a partial order reduction approach that avoided exploring equivalent interleavings, and presented a verification tool called ISP. This paper presents algorithmic and engineering innovations to ISP, including the use of OpenMP parallelization, that now enables it to handle practical MPI programs, including:(i)~ParMETIS - a widely used hypergraph partitioner, and (ii)~MADRE - a Memory Aware Data Re-distribution Engine, both developed outside our group. Over these benchmarks, ISP has automatically verified up to 14K lines of MPI/C code, producing error traces of deadlocks and assertion violations within seconds.

ISP: a tool for model checking MPI programs

A significant body of parallel programs are written using MPI [1]. Bugs in MPI programs arise due to many reasons, for example: (i) during manual optimizations that turn blocking sends/receives into their non-blocking counterparts, (ii) in the context of using wild-card communications (and the resulting non-determinism), and (iii) in programs that use one-sided communication [2]. Programmers are acutely aware of the need to test parallel programs over all their interleavings. They also known that this is impossible, thus confining testing to interleavings that are guessed to be adequate – very often incorrectly, as it turns out. Model checking [4] has helped debug many concurrent systems by relying on an exhaustive search of all possible interleavings – albeit on a suitably abstracted system model. For example, SPIN [5] recently won ACM’s prestigious software award, and has been used to verify many real-world protocols. For model checking to be successful in practice: (i) Designers must have modestly expensive techniques to create models of the protocols to be verified. (ii) They must be able to examine a fraction of the interleavings in a concurrent system, and be able to claim that the remaining interleavings are equivalent and hence need not be examined, using a suitable state space reduction algorithm [4, 5]. This paper presents a tool called ISP (in-situ partial order) which aspires to grow into a practical MPI model checker. In the only other active research effort on developing a model checker (called MPI SPIN) for MPI programs [6], the following approach is taken: (i) they model MPI library functions in Promela (SPIN’s modeling language), and uses SPIN for verification; (ii) they use the C extension features of SPIN to model advanced features such as non-blocking communication commands. (iii) they require users to manually express their intended MPI C program in MPI SPIN

Reduced Execution Semantics of MPI: From Theory to Practice

There is growing need to develop formal verification tools for Message Passing Interface (MPI) programs to eliminate bugs such as deadlocks and local assertion violations. Of all approaches, dynamic verification is most practical for MPI. Since the number of interleavings of concurrent programs grow exponentially, we devise a dynamic interleaving reduction algorithm (dynamic partial order reduction , DPOR) tailor-made for MPI, called POE. The key contributions of this paper are: (i) a formal semantics that elucidates the complex dynamic semantics of MPI, and played an essential role in the design of the POE algorithm and the construction of the ISP tool, and (ii) a formal specification of our POE algorithm. We discuss how these ideas may help us build dynamic verifiers for other APIs, and summarize a dynamic verifier being designed for applications written using a recently proposed API for multi-core communication.

Implementing Efficient Dynamic Formal Verification Methods for MPI Programs

We examine the problem of formally verifying MPI programs for safety properties through an efficient dynamic (runtime) method in which the processes of a given MPI program are executed under the control of an interleaving scheduler. To ensure full coverage for given input test data, the algorithm must take into consideration MPIs out-of-order completion semantics. The algorithm must also ensure that nondeterministic constructs (e.g., MPI wildcard receive matches) are executed in all possible ways. Our new algorithm rewrites wildcard receives to specific receives, one for eachsender that can potentially match with the receive. It then recursively explores each case of the specific receives. The list of potential senders matching a receive is determined through a runtime algorithm that exploits MPIs operation ordering semantics. Our verification tool ISP that incorporates this algorithm efficiently verifies several programs and finds bugs missed by existing informal verification tools.

A Formal Approach to Detect Functionally Irrelevant Barriers in MPI Programs

We examine the unsolved problem of automatically and efficiently detecting functionally irrelevant barriers in MPI programs. A functionally irrelevant barrier is a set of MPI_Barrier calls, one per MPI process, such that their removal does not alter the overall MPI communication structure of the program. Static analysis methods are incapable of solving this problem, as MPI programs can compute many quantities at runtime, including send targets, receive sources, tags, and communicators, and also can have data-dependent control flows. We offer an algorithm called Fib to solve this problem based on dynamic (runtime) analysis. Fib applies to MPI programs that employ 24 widely used two-sided MPI operations. We show that it is sufficient to detect barrier calls whose removal causes a wildcard receive statementplaced before or after a barrier to now begin matching a send statement with which it did not match before. Fib determines whether a barrier becomes relevant in anyinterleaving of the MPI processes of a given MPI program. Since the number of interleavings can grow exponentially with the number of processes, Fib employs a sound method to drastically reduce this number, by computing only the relevant interleavings. We show that many MPI programs do not have data dependent control flows, thus making the results of Fib applicable to all the input data the program can accept.

Scheduling considerations for building dynamic verification tools for MPI

Dynamic verification methods are the natural choice for formally verifying real world programs when model extraction and maintenance are expensive. Message passing programs written using the MPI library often fall under this category, especially when these programs have to be verified after being ported to new platforms and iteratively optimized. However, implementing dynamic verification tools for MPI requires solving many problems pertaining to the scheduling realities of the MPI runtime. In this paper, we describe the progression of our ideas during the development of our dynamic verification tool for MPI programs, called in-situ partial order (ISP). Each idea developed in this progression relates to our dual goals of ensuring full coverage of all representative interleavings (in the sense of partial order reduction) among MPI processes, and forcing the MPI runtime to affect these interleavings. We briefly examine similar issued faced by other builders of dynamic verification tools. We conclude with observations backing the growing importance of addressing scheduling issues in future dynamic verification tools for various concurrency APIs.

Precise dynamic analysis for slack elasticity: adding buffering without adding bugs

Increasing the amount of buffering for MPI sends is an effective way to improve the performance of MPI programs. However, for programs containing non-deterministic operations, this can result in new deadlocks or other safety assertion violations. Previous work did not provide any characterization of the space of slack elastic programs: those for which buffering can be safely added. In this paper, we offer a precise characterization of slack elasticity based on our formulation of MPIs happens before relation. We show how to efficiently locate potential culprit sends in such programs: MPI sends for which adding buffering can increase overall program non-determinism and cause new bugs. We present a procedure to minimally enumerate potential culprit sends and efficiently check for slack elasticity. Our results demonstrate that our new algorithm called POEMSE which is incorporated into our dynamic verifier ISP can efficiently run this new analysis on large MPI programs.

Sound and Efficient Dynamic Verification of MPI Programs with Probe Non-determinism

We consider the problem of verifying MPI programs that use MPI_Probe and MPI_Iprobe . Conventional testing tools, known to be inadequate in general, are even more so for testing MPI programs containing MPI probes. A few reasons are: (i) use of the MPI_ANY_SOURCE argument can make MPI probes non-deterministic, allowing them to match multiple senders, (ii) an MPI_Recv that follows an MPI probe need not match the MPI_Send that was successfully probed, and (iii) simply re-running the MPI program, even with schedule perturbations, is insufficient to bring out all behaviors of an MPI program using probes. We develop several key insights that help develop an elegant solution: prioritizing MPI processes during dynamic verification, handling non-determinism, and safe handling of probe loops. These solutions are incorporated into a new version of our dynamic verification tool ISP. ISP is now able to efficiently and soundly verify larger MPI examples, including MPI-BLAST and ADLB.

Some resources for teaching concurrency

With the increasing emphasis on exploiting concurrency efficiently and correctly, the lack of suitable pedagogical material for teaching concurrency is a growing problem. In this paper, we summarize a recently concluded class as well as some independent projects in the area of concurrency and multi-core computing that offer some insights to address this problem. We examine background papers, the teaching of low level concurrency, and the teaching of threading and message passing. The use of dynamic formal verification tools in a class setting is discussed in some detail. We conclude with a summary of pedagogical material being assembled, including exercises from a popular textbook on MPI solved using our dynamic verifier ISP. Our observation is that the teaching of concurrency is greatly facilitated by the use of dynamic push-button formal verification tools that can handle non-trivial concurrent programs. Given the growing number of publications on how to teach concurrency as well as employ new programming approaches, our work addresses the somewhat neglected topic of using modern dynamic formal verification methods within the context of widely used concurrency approaches and libraries.

Static-Analysis Assisted Dynamic Verification of MPI Waitany Programs (Poster Abstract)

It is well known that the number of schedules (interleavings) of a concurrent program grows exponentially with the number of processes. Our previous work has demonstrated the advantages of an MPI-specific dynamic partial order reduction (DPOR, [5]) algorithm called POE in a tool called ISP [1,2,3] in dramatically reducing the number of interleavings during formal dynamic verification. Higher degrees of interleaving reduction were achieved when the programs were deterministic . In this work, we consider the problem of verifying MPI using MPI_Waitany (and related operations wait/test some/all). Such programs potentially have a higher degree of non-determinism. For such programs, POE can become ineffective, as shown momentarily. To solve this problem, we employ static analysis (supported by ROSE [4]) in a supporting role to POE to determine the extent to which the out parameters of MPI_Waitany can affect subsequent control flow statements. This informs ISPs scheduler to exert even more intelligent backtrack/replay control.