Guy Katz

Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks

Deep neural networks have emerged as a widely used and effective means for tackling complex, real-world problems. However, a major obstacle in applying them to safety-critical systems is the great difficulty in providing formal guarantees about their behavior. We present a novel, scalable, and efficient technique for verifying properties of deep neural networks (or providing counter-examples). The technique is based on the simplex method, extended to handle the non-convex Rectified Linear Unit (ReLU) activation function, which is a crucial ingredient in many modern neural networks. The verification procedure tackles neural networks as a whole, without making any simplifying assumptions. We evaluated our technique on a prototype deep neural network implementation of the next-generation airborne collision avoidance system for unmanned aircraft (ACAS Xu). Results show that our technique can successfully prove properties of networks that are an order of magnitude larger than the largest networks verified using existing methods.

Non-intrusive Repair of Reactive Programs

We show how, under certain conditions, programs written in the behavioral programming approach can be modified (e.g., as result of new requirements or discovered bugs) using automatically-generated code modules. Given a trace of undesired behavior, one can generate a relatively small piece of code, whose execution is interwoven at run time with the rest of the system and brings about the desired changes without modifying existing code, and without introducing new bugs. At the core of our approach is the ability of a thread of behavior to prevent the triggering of events from other threads. Our repair algorithms apply model checking to the program and transform the counterexamples produced by the model-checker into corrective modules. Our work is supported by a proof-of-concept tool, which creates understandable modules that can be further manually managed as part of ongoing incremental system development.

Towards Proving the Adversarial Robustness of Deep Neural Networks.

Autonomous vehicles are highly complex systems, required to function reliably in a wide variety of situations. Manually crafting software controllers for these vehicles is difficult, but there has been some success in using deep neural networks generated using machine-learning. However, deep neural networks are opaque to human engineers, rendering their correctness very difficult to prove manually; and existing automated techniques, which were not designed to operate on neural networks, fail to scale to large systems. This paper focuses on proving the adversarial robustness of deep neural networks, i.e. proving that small perturbations to a correctly-classified input to the network cannot cause it to be misclassified. We describe some of our recent and ongoing work on verifying the adversarial robustness of networks, and discuss some of the open questions we have encountered and how they might be addressed.

On composing and proving the correctness of reactive behavior

We present a method and a tool for composing a reactive system and for accompanying the development and documentation process with a proof of its correctness. The approach is based on behavioral programming (BP) and the Z3 SMT solver. We show how program verification can be automated and streamlined by combining properties of individual modules, specified and verified separately, with application-independent specifications both of the BP semantics and of general theories. The method may yield an exponential acceleration of the verification process when compared with model-checking the composite application. We show that formalization of properties of independent modules in preparation for the correctness proofs can be useful as documentation for future development. We view this work as a further step towards making formal correctness proofs standard practice in the development of reactive systems, and carried out by programmers at large.

On Module-Based Abstraction and Repair of Behavioral Programs

The number of states a program has tends to grow exponentially in the size of the code. This phenomenon, known as state explosion, hinders the verification and repair of large programs. A key technique for coping with state explosion is using abstractions, where one substitutes a program’s state graph with smaller over-approximations thereof. We show how module-based abstraction-refinement strategies can be applied to the verification of programs written in the recently proposed framework of Behavioral Programming. Further, we demonstrate how — by using a sought-after repair as a means of refining existing abstractions — these techniques can improve the scalability of existing program repair algorithms. Our findings are supported by a proof-of-concept tool.

Scaling-Up Behavioral Programming: Steps from Basic Principles to Application Architectures

Behavioral programming (BP) is a decentralized scenario-based paradigm for the programming of reactive software, geared towards incremental and intuitive development. In this work we apply the principles of BP to a large, real-world case-study: a web-server. We discuss the conclusions learned from our attempt and propose several extension idioms to BP, aimed at improving the frameworks scalability. Specifically, we propose extending BP with a timeout idiom for handling various time constraints, program-specific execution strategies, dynamic thread creation for efficiently allocating system resources, and support for parameterized events to handle inputs with infinite domains. Our extensions and case-study are implemented in a new framework for behavioral programming in C++.

Relaxing Synchronization Constraints in Behavioral Programs

In behavioral programming, a program consists of separate modules called behavior threads, each representing a part of the system’s allowed, necessary or forbidden behavior. An execution of the program is a series of synchronizations between these threads, where at each synchronization point an event is selected to be carried out. As a result, the execution speed is dictated by the slowest thread. We propose an eager execution mechanism for such programs, which builds upon the realization that it is often possible to predict the outcome of a synchronization point even without waiting for slower threads to synchronize. This allows faster threads to continue running uninterrupted, whereas slower ones catch up at a later time. Consequently, eager execution brings about increased system performance, better support for the modular design of programs, and the ability to distribute programs across several machines. It also allows to apply behavioral programming to a variety of problems that were previously outside its scope. We illustrate the method by concrete examples, implemented in a behavioral programming framework in C + + .

An initial wise development environment for behavioral models

We present a development environment that proactively and interactively assists the software engineer in modeling complex reactive systems. Our framework repeatedly analyzes models of the system under development at various levels of abstraction, and then reasons about these models in order to detect possible errors and to derive emergent properties of interest. Upon request, the environment can then augment the system model in order to repair or avoid detected behavior that is undesired, or instrument it in order to monitor the execution for certain behaviors. Specialized automated and human-assisted techniques are incorporated to direct and prioritize the analysis and related tasks, based on the relevance of the observed properties and the expected impact of actions to be taken. Our development environment is an initial step in the direction of the very recent Wise Computing vision, which calls for turning the computer (namely, the development environment) into an equal member of the development team: knowledgeable, independent, concerned and proactively involved in the development process. Our tool is implemented within the context of behavioral programming (BP), a scenario-based modeling approach, where components are aligned with how humans often describe desired system behavior. Thus, our work further enhances the naturalness and incrementality of developing in BP.

The Effect of Concurrent Programming Idioms on Verification

In recent years formal verification techniques have become an important part of the development cycle of concurrent software. In order to tackle the state explosion problem and verify larger systems, a great deal of work has been put into improving the scalability of verification tools. In this work, we seek to draw attention to an alternative/complementary approach to improving scalability, which sometimes receives less notice: the effect the concurrent programming model itself has on ones ability to verify programs encoded within it. Recent work suggests that a suitable choice of model, tailored to the problem at hand, may render the produced software more amenable to verification techniques. We recapitulate some recent and new results demonstrating this effect in programming models for discrete, synchronous reactive systems, and outline some directions for future work. We hope that the paper will trigger additional research on this important topic.

Theory-aided model checking of concurrent transition systems

We present a method for the automatic compositional verification of certain classes of concurrent programs. Our approach is based on the casting of the model checking problem into a theory of transition systems within CVC4, a DPLL(T) based SMT solver. Our transition system theory then cooperates with other theories supported by the solver (e.g., arithmetic, arrays), which can help accelerate the verification process. More specifically, our theory solver looks for known patterns within the input programs and uses them to generate lemmas in the languages of other theories. When applicable, these lemmas can often steer the search away from safe parts of the search space, reducing the number of states to be explored and expediting the model checking procedure. We demonstrate the potential of our technique on a number of broad classes of programs.