Automatic Verification of Message-Based Device Drivers
Sidney Amani, Peter Chubb, Alastair F. Donaldson, Alexander Legg, Leonid Ryzhyk, Yanjin Zhu
FF. Cassez, R. Huuck, G. Klein and B. Schlich (eds.):Systems Software Verification Conference 2012 (SSV 2012)EPTCS 102, 2012, pp. 4–17, doi:10.4204/EPTCS.102.3 c (cid:13)
Sidney Amani et al.This work is licensed under theCreative Commons Attribution License.
Automatic Verification of Message-Based Device Drivers
Sidney Amani ‡§ Peter Chubb ‡§ Alastair F. Donaldson ¶ Alexander Legg ‡ Leonid Ryzhyk ‡§ Yanjin Zhu ‡§‡
NICTA § University of New South Wales ¶ Imperial College [email protected]
We develop a practical solution to the problem of automatic verification of the interface betweendevice drivers and the OS. Our solution relies on a combination of improved driver architecture andverification tools. It supports drivers written in C and can be implemented in any existing OS, whichsets it apart from previous proposals for verification-friendly drivers. Our Linux-based evaluationshows that this methodology amplifies the power of existing verification tools in detecting driverbugs, making it possible to verify properties beyond the reach of traditional techniques.
Faulty device drivers are a major source of operating system (OS) failures [14, 7]. Recent studies ofWindows and Linux drivers show that over a third of driver bugs result from the complex interfacebetween driver and OS [21, 3]. The OS defines numerous rules on the ordering and arguments of driverinvocations, rules that often are neither well documented nor are stable across OS releases. Worse, the OScan invoke driver functions from multiple concurrent threads, and so driver developers must implementcomplex synchronisation logic to avoid races and deadlocks.In addition to causing numerous programming errors, these problems complicate formal analysis ofdevice driver code. While automatic verification has proved useful in detecting OS interface violations indevice drivers, driver verification tools remain limited in the complexity of properties that can be verifiedefficiently [3, 9, 8, 16, 11].One way to address the problem is through an improved device driver architecture that simplifiesdriver development and makes them more amenable to automatic verification [12, 4]. In this architectureeach driver has its own thread and communicates with the OS using message passing, which makes thedriver control flow and its interactions with the OS easier to understand and analyse. We refer to suchdrivers as active drivers , in contrast to conventional, passive , drivers that are structured as collections ofentry points invoked by OS threads.Previous implementations of active drivers in Singularity [12] and RMoX [4] OSs rely on OS andlanguage support for improved verifiability. As such, they do not help address the driver reliabilityproblem in mainstream operating systems written in C.In this paper we show that the benefits of active drivers can be achieved while writing drivers in C fora conventional OS. To this end, we present an implementation of an active driver framework for Linuxalong with a new verification method that enables efficient, automatic checking of active driver protocols.Our method leverages existing verification tools for C, extended with several novel optimisations gearedtowards making active driver verification tractable. Like other existing automatic verification techniques,the method is not complete—it helps to find bugs, but does not guarantee their absence.Through experiments involving verification of several complex drivers for Linux, we demonstratethat our driver design and verification methodology amplifies the power of verification tools in findingidney Amaniet al. 5driver bugs. In particular, many properties that are hard or impossible to verify in conventional driverscan be easily checked on active drivers.In this paper we focus on verification of active device drivers. A detailed account of the design andimplementation of the active driver framework for Linux and its peformance evaluation can be found inthe accompanying technical report [2].The rest of the paper is structured as follows. Section 2 introduces the active driver architecture.Section 3 presents our visual formalism for specifying active driver protocols. Section 4 describes ourverification methodology. Section 5 outlines the design and implementation of the active driver frame-work for Linux. Section 6 presents experimental results. Section 7 surveys related work on device driververification. Section 8 concludes the paper.
In this section we discuss the shortcomings of the conventional driver architecture and show how activedrivers address these shortcomings.
Passive drivers
The passive driver architecture supported by all mainstream OSs suffers from twoproblems that complicate verification of the driver-OS interface: stack ripping and concurrency .A passive device driver comprises a collection of entry points invoked by the OS. When writing thedriver, the programmer makes assumptions about possible orders in which its entry points are going tobe activated; however these assumptions remain implicit in the implementation. As a result, the controlflow of the driver is scattered across multiple entry points and cannot be reconstructed from its sourcecode. This phenomenon is known as stack ripping [1].To complicate things further, the OS can invoke driver entry points from multiple concurrent threads,forcing driver developers to implement intricate synchronisation logic to avoid races and deadlocks.Multithreading further complicates automatic verification of device drivers, as thread interleaving leadsto dramatic state explosion.Previous research [21] has shown that the vast majority of device drivers do not get any performancebenefits from multithreading. The performance of most drivers is bound by I/O bandwidth rather thanCPU speed, therefore they do not require true multiprocessor parallelism. Device drivers are multi-threaded simply by virtue of executing within the multithreaded kernel environment and not becausethey require multithreading for performance or functionality.
Active drivers
In contrast to passive drivers, an active driver runs in the context of its own thread.Communication between the driver and the OS occurs via message passing. The OS sends I/O requestsand interrupt notifications to the driver using messages. The driver notifies the OS about completedrequests via reply messages.In an active device driver, the order in which the driver handles and responds to OS requests isdefined explicitly in its source code and can be readily analysed automatically. Since the driver handlesI/O requests sequentially, such analysis can be performed without running into state explosion due tothread interleaving.We present our instantiation of the active driver architecture for Linux. Our design is based on theDingo active driver framework [21], improving upon it in two ways. First, Dingo’s message passingprimitives are implemented as C language extensions. In contrast, our framework supports drivers inpure C. Second, Dingo does not support automatic driver protocol verification.In our framework, the driver-OS interface consists of a set of mailboxes , where each mailbox isused for a particular type of message. The driver exchanges messages with the OS via
EMIT and
AWAIT
Automatic Verification of Message-Based Device Drivers mb = AWAIT ( suspend , unplug ,..); if ( mb == s u s p e n d) { d e v _ s u s p e n d (); EMIT ( s u s p e n d _ c o mpl ete , msg ); // Bug ! U n c o m m e n t to fix mb = AWAIT ( resume /* , unplug */ ); ... } else if ( mb == unplug ) { ... } (a) Faulty code (b) Protocol Figure 1: Fragment of active driver code and the matching protocol specification.primitives, that operate on messages and mailboxes. The
EMIT function takes a pointer to a mailbox, amessage structure, and a list of message arguments. It places the message in the mailbox and returnscontrol to the caller without blocking. The
AWAIT function takes references to one or more mailboxesand blocks until a message arrives in one of them. It returns a reference to the mailbox containing themessage. A mailbox can queue multiple messages.
AWAIT always dequeues the first message in themailbox. This message is accessible via a pointer in the returned mailbox.Figure 1(a) shows a fragment of an active driver. In line 1 the driver waits on suspend and unplug mailboxes. After receiving a suspend request (checked by the condition at line 2) the driver suspendsthe device (line 3) and notifies the OS about completion of the request by sending a message to the suspend complete mailbox (line 4). It then waits for a resume request at line 7. As can be seen fromthis example, requests that the driver accepts in each state are explicitly listed in the driver source code,which simplifies the analysis of driver behaviour and in particular its interaction with the OS.
This section presents our visual formalism for specifying active driver protocols. The formalism issimilar to protocol state machines of Dingo [21] and Singularity [12], extended with additional means tocapture liveness and fairness constraints, which enable the detection of additional types of driver bugs.The active driver framework associates a protocol with each driver interface. The protocol specifieslegal sequences of messages exchanged by the driver and the OS. Protocols are defined by the driverframework designer and are generic in the sense that every driver that implements the given interfacemust comply with the associated protocol. In the case when the active driver framework is implementedwithin an existing OS, the framework includes wrapper components that perform the translation betweenthe native function-based interface and message-based active driver protocols.We specify driver protocols using deterministic finite state machines (FSMs). The protocol statemachine conceptually runs in parallel with the driver: whenever the driver sends or receives a messagethat belongs to the given protocol, this triggers a matching state transition in the protocol state machine.Figure 1(b) shows a state machine for the protocol used by the example driver, describing the handlingof power management and hot unplug requests. Each protocol state transition is labelled with the nameof the mailbox through which the driver sends (‘!’) or receives (‘?’) a message. We represent complexprotocol state machines compactly using Statecharts [15], which organise states into a hierarchy so thatseveral primitive states can be clustered into a super-state.idney Amaniet al. 7In some protocol states the OS is waiting for the driver to complete a request. The driver cannotremain in such a state indefinitely, but must eventually leave the state by sending a response message tothe OS. Such states are called timed states and are labelled with the clock symbol in Figure 1(b).In order to ensure that the driver does not deadlock in an
AWAIT statement, the developer must relyon an additional assumption that if the driver waits for all incoming OS messages enabled in the currentstate, then one of them will eventually arrive. This is a form of weak fairness constraint [18] on the OSbehaviour, which means that if some event (in this case, arrival of a message) is continuously enabled, itwill finally occur. Not all protocol states have the weak fairness property. In the protocol state machine,we show fair states with dashed border. For example, the
SUSPENDED state in Figure 1b is fair, whichguarantees that at least one of resume and unplug messages will eventually arrive in this state.A protocol-compliant device driver must obey the following 5 rules.
Rule 1. (EMIT) The driver is allowed to emit a message to a mailbox iff this message triggers a validstate transition in the protocol state machine.
Rule 2. (AWAIT1) When in a state where there is an enabled incoming message, the driver must even-tually issue an
AWAIT on the corresponding mailbox or transition into a state where this message is notenabled.
Rule 3. (AWAIT2) All
AWAIT operations eventually terminate. Equivalently, whenever the driver per-forms an
AWAIT operation, at least one of its protocols must be in a fair state and the
AWAIT must waitfor all enabled messages of this protocol.
Rule 4. (Timed) The driver must not remain in a timed state forever.
Rule 5. (Termination) When the main driver function returns, the protocol state machine must be in afinal state. Note that this rule does not require that every driver run terminates, merely that if it doesterminate then all protocols must be in their final states.
Rules 1, 3 and 5 describe safety properties, whose violation can be demonstrated by a finite executiontrace. Rules 2 and 4 are liveness rules, for which counterexamples are infinite runs.Going back to the example in Figure 1, we can see that the
AWAIT statement in line 6 violates Rule 3.This line corresponds to the
SUSPENDED state of the protocol, where the driver can receive unplug and resume messages. By waiting for only one of these messages, the driver can potentially deadlock.
The goal of driver protocol verification is to check whether the driver meets all safety and livenessrequirements assuming fair OS behaviour. We use two tools to this end: S AT A BS [8], geared towardssafety analysis, and G OANNA [13], geared towards liveness analysis. These tools provide complementarycapabilities that, when combined, enable full verification of many driver protocols. We use S AT A BS tocheck safety rules 1, 3, and 5 and G OANNA to check liveness rules 2 and 4. This combination workswell in practice, yielding a low overall false positive rate. Our methodology is compatible with othersimilar tools. We use S AT A BS and G OANNA because our team is familiar with their internals and hasthe expertise required to implement novel performance optimisations for these tools. S AT A BS is an abstraction-refinement based model checker for C and C++ for checking safety properties.It is designed to perform best when checking control-flow dominated properties with a small number ofdata dependencies. Active driver protocol-compliance safety checks fall into this category. Automatic Verification of Message-Based Device DriversGiven a program to verify, S AT A BS iteratively computes and verifies its finite-state abstraction withrespect to a set of predicates over program variables. At each iteration it either terminates (by discoveringa bug or proving that the program is correct) or generates a spurious counterexample. In the latter case,the counterexample is analysed by the tool to discover new predicates, used to construct a refined programabstraction. Abstraction and refinement are both fully automatic.S AT A BS verifies program properties expressed as source code assertions. We encode rules 1 and 3 asassertions embedded in modified versions of AWAIT and
EMIT functions. These functions keep track ofthe protocol state using a global state variable. The
AWAIT function simulates the receiving of a messageby randomly selecting one of incoming mailboxes enabled in the current state and updating the statevariable based on the current state and the message selected. Similarly, the
EMIT function updates thestate variable based on the current state and the message being sent. It contains an assertion that triggersan error when the driver is trying to send a message that is not allowed in the current state. To verifyrule 5, we append to the driver’s main function a check to ensure that, if the driver does terminate, theprotocol state machine is in a final state.Our preliminary experiments show that straightforward application of S AT A BS to active drivers re-sults in very long verification times. This is in part due to the complexity of driver protocols beingverified and in part because predicate selection heuristics implemented in S AT A BS introduce large num-bers of unnecessary predicates, leading to overly complex abstractions. The problem is not unique toS AT A BS . Our preliminary experiments with SLAM [3], another state-of-the-art abstraction-refinementtool, produced similar results. We describe several novel strategies that exploit the properties of activedrivers to make their safety verification feasible. We believe that these techniques will also be useful inother software protocol verification tasks. Protocol decomposition
The abstraction-refinement technique is highly sensitive to the size of theproperty being checked. Complex properties require many predicates. Since verification time growsexponentially with the number of predicates, it is beneficial to decompose complex properties into simpleones that can be verified independenly.We decompose each driver protocol state machine into a set of much simpler subprotocols as a pre-processing step. The decomposition is constructed in such a way that the driver satisfies safety constraintsof the original protocol if and only if it does so for each protocol in the decomposition. The followingproposition (stated informally) gives a sufficient condition for correctness of decomposition.
Proposition 1.
Consider a protocol P and its decomposition into protocols P , . . . , P n . If the followingconditions hold then a driver satisfies P if and only if it satisfies each of P , . . . , P n :1. The regular language generated by the protocol state machine of P is equivalent to the intersectionof languages generated by P , . . . , P n .2. There exists a bijection between fair states of P and the union of fair states of P , . . . , P n , such thatfor each fair state s of P and the corresponding fair state s ′ of P i , the set of incoming messagesenabled in s is equal to the set of incoming messages in s ′ . Figure 2 shows one possible decomposition of the protocol in Figure 1(b). Every subprotocol inthe decomposition captures a simple rule related to a single type of message, shown in bold italics inthe diagram. For instance, the third protocol from the left describes the occurrence of the suspend message: suspend can arrive in the initial state, is reenabled by the resume complete message, and ispermanently disabled by the unplug message. Messages that do not participate in the subprotocol areallowed in any state (as they are constrained by separate subprotocols) and are omitted in the diagram.In our experience, even complex driver protocols allow decomposition into simple subprotocols withno more than four states and only a few transitions. Verifying each subprotocol requires a small subsetof predicates involved in checking the monolithic protocol, leading to exponentially faster verification.idney Amaniet al. 9 ?unplug !unplug complete ?suspend !suspend complete ?resume !resume complete
Figure 2: Decomposition of the protocol in Figure 1(b).Correctness of a decomposition can be automatically checked based on Proposition 1. Furthermore,we found construction of protocol decompositions to be a largely mechanical task. As part of futurework on the project we will investigate approaches to automating this task.
Automatically provide key predicates
One way to speed-up the abstraction-refinement algorithm is toseed it with a small set of key predicates that avoid large families of spurious counterexamples. Guessingsuch key predicates in general is extremely difficult. In case of active driver verification, an importantclass of key predicates can be provided to S AT A BS automatically.As mentioned above, when checking a driver protocol, we introduce a global variable that keeps trackof protocol state. During verification, S AT A BS eventually discovers predicates over this variable of theform (state==1) , (state==2) , . . . , one for each state of the protocol. These predicates are importantto establishing the correspondence between the driver control flow and the protocol state machine. Wetherefore provide these predicates to S AT A BS on startup, which accelerates verification significantly. Control-flow transformations
We found that it often takes S AT A BS many iterations to correlatedependent program branches. This problem frequently occurs in active drivers when the driver AWAIT s onmultiple mailboxes and then checks the returned value (e.g., line 2 in Figure 1(a)). If the driver executesthe same comparison later in the execution, then both checks must produce the same outcome. S AT A BS does not know about this correlation initially, leading to a spurious counterexample trace that takesinconsistent branches, potentially leading to spurious counteraexample traces. These counterexamplescan be refuted using predicate p ↔ ( mb == suspend ) . In practice, however, S AT A BS may introducemany predicates that only refute a subset of these counterexamples before discovering p , which allowsrefuting all of them.To remedy the problem, we have implemented a novel control-flow graph transformation that usesstatic analysis to identify correlated branches, and merges them. The analysis identifies, through inspect-ing the use of the AWAIT function, where to apply the transformation. Then infeasible paths througheach candidate region are identified by generating Boolean satisfiability queries which are discharged toa SAT solver. The CFG region is then rewritten to eliminate infeasible paths. The effect of the rewritingon the CFG is shown in Figure 3.This technique effectively avoids the expensive search for additional predicates using much cheaperstatic program analysis. In our experiments, S AT A BS performs orders of magnitude more effectivelyover the new program structure, being able to quickly infer key predicates that could previously only beinferred after many abstraction refinement iterations and the inference of many redundant predicates.0 Automatic Verification of Message-Based Device DriversFigure 3: CFG transformation example. As S AT A BS is restricted to analysis of safety properties, the G OANNA tool comes into play for analysis ofliveness properties. G
OANNA is a C and C++ bug finding tool that supports user-defined rules written inthe CTL temporal logic [10], which allows natural specification of safety and liveness properties. UnlikeS AT A BS , G OANNA is intended as a fast compile-time checker and therefore does not perform data-flowanalysis.Properties to be checked for each protocol are extracted from the protocol specification. In particular,we apply the
AWAIT1 rule to every incoming mailbox and the
Timed rule to every timed state of theprotocol.Describing a temporal property using the G
OANNA specification language involves two steps. First,we identify a set of important program events related to the property being verified, such as sending andreceiving of messages. We use syntactic pattern matching to label program locations that correspond tothese events. Second, we encode the property to be checked as a temporal logic formula in a dialect ofCTL, defined over events identified at the previous step. Due to limited space, we omit the details of thisencoding.
We implemented the active driver framework along with three active device drivers in Linux 2.6.38. Theframework consists of loadable kernel modules and does not require any changes to other kernel compo-nents. The framework provides services required by all active drivers, including cooperative scheduling,message passing, and message-based interrupt delivery. In addition it defines protocols for supportedclasses of drivers and provides wrappers to perform the translation between the Linux driver interfaceand message-based active driver protocols. Wrappers enable conventional and active drivers to co-existwithin the kernel.The generic part of the framework shared by all active drivers provides support for scheduling andmessage passing. It implements the cooperative domain abstraction, which constitutes a collection ofcooperatively scheduled kernel threads hosting an active driver. Threads inside the domain communicatewith the kernel via a shared message queue. The framework guarantees that at most one thread in thedomain is runnable at any time. The thread keeps executing until it blocks in the
AWAIT function.
AWAIT checks whether there is a message available in one of the mailboxes specified by the caller and, if so,returns without blocking. Otherwise it calls the thread dispatcher function, which finds a thread for whicha message has arrived. The dispatcher uses the kernel scheduler interface to suspend the current threadidney Amaniet al. 11 driver protocol
PCI bus 13 41 11Ethernet 17 36 6Serial ATA (SATA) 39 70 22Digital Audio Interface (DAI) 8 20 6
Table 1: Implemented active driver protocols.and make the new thread runnable. In the future this design can be optimised by implementing nativesupport for light-weight threads in the kernel.
EMIT and
AWAIT functions do not perform memory allocation and therefore never fail. This simplifiesdriver development, as the driver does not need to implement error handling logic for each invocationof these ubiquitous operations. On the other hand this means that the driver is responsible for allocatingmessages sent to the OS and deallocating messages received from the OS. By design of driver protocols,most mailboxes can contain at most one message, since the sender can only emit a new message tothe mailbox after receiving a completion notification for the previous request. Such messages can bepre-allocated statically.Interrupt handling in active drivers is separated into top and bottom halves. The driver registers withthe framework a top-half function that is invoked by the kernel in the primary interrupt context (outsidethe cooperative domain). A typical top-half handler reads the interrupt status register, acknowledges theinterrupt in the device, and sends an IRQ message to the driver. The actual interrupt handling happensinside the cooperative domain in the context of the driver thread that receives the IRQ message. IRQdelivery latency can be minimised by queueing interrupt messages at the head of the message queue;alternatively interrupts can be queued as normal messages, which avoids interrupt livelock an ensuresfair scheduling of interrupts with respect to other driver tasks.In addition to the generic functionality described above, the active driver framework defines protocolsfor supported classes of drivers and provides wrappers to perform the translation between the Linuxdriver interface and message-based active driver protocols. Wrappers enable conventional and activedrivers to co-exist within the kernel.Active driver protocols are derived from the corresponding Linux interfaces by replacing every inter-face function with a message or a pair of request/response messages. While multiple function calls canoccur concurrently, messages are serialised by the wrapper.Since Linux lacks a formal or informal specification of driver interfaces, deriving protocol statemachines often required tedious inspection of the kernel source. On the positive side, we found that,compared to building an OS model as a C program, state machines provide a natural way to captureprotocol constraints and are useful not only for automatic verification, but also as documentation fordriver developers.Table 1 lists protocols we have specified and implemented wrappers for. For each protocol, it givesthe number of protocol states and transitions, and the number of subprotocols in its decomposition (seeSection 4.1). Table 2 lists active device drivers we have implemented along with protocols that eachdriver supports. All three drivers control common types of devices found in virtually every computersystem. These drivers were obtained by porting native Linux drivers to the active architecture, whichallows direct comparison of their performance and verifiability against conventional drivers.2 Automatic Verification of Message-Based Device Drivers driver supportedprotocols LOC(native) LOC(active) avg(max)time(minutes) avg(max)refinements avg(max)predicates
RTL8169 1Gb Ethernet PCI, Ethernet 4,220 4,317 29 (103) 3 (7) 3 (8)AHCI SATA controller PCI, SATA 2,268 2,487 123 (335) 2 (6) 2 (19)OMAP DAI audio DAI 583 705 5 (13) 2 (5) 2 (0)
Table 2: Active device driver case studies, protocols that each driver implements, size of the nativeLinux and active versions of the driver in lines of code (LOC) (measured using sloccount ), along withstatistics for checking safety properties using S AT A BS for each driver. We applied the verification methodology described in Section 4 to RTL8169, AHCI, and OMAP DAIdrivers. Verification was performed on machines with 2GHz quad-core Intel Xeon CPUs.
Verification using S AT A BS and G OANNA
For each of the three drivers we were able to verify all safetyproperties defined by their protocols using S AT A BS with zero false positives. The last three columns ofTable 2 show statistics for verifying safety properties using S AT A BS for each driver: average and maxi-mum time, the number of abstraction refinement loop iterations and the number of predicates required forverification to succeed, across all subprotocols of the driver. The number of predicates reflects predicatesdiscovered dynamically by the abstraction refinement loop and does not include candidate predicateswith which S AT A BS is initialised (see Section 4.1).The small number of predicates involved in checking these properties indicates that the control skele-ton of an active driver responsible for interaction with the OS has few data dependencies. This confirmsthat the active driver architecture achieves its goal of making the driver-OS interface amenable to effi-cient automatic verification. At the same time, the fact that several refinements are required in most casesindicates that the power of the abstraction refinement method is necessary to avoid false positives whenchecking safety.Despite the small number of predicates required, verification times are relatively high for our bench-marks. This is due to the large size of our drivers, and the fact that SMV [20], the model checker usedby S AT A BS , was not designed primarily for model checking boolean programs. We experimented withthe BOOM model checker [5], which is geared towards boolean program verification. While in manycases verification using BOOM was several times faster than with SMV, we did not use it in our finalexperiments due to stability issues.All optimisations described in Section 4.1 proved essential to making verification tractable. Disablingany one of them led to overly large abstractions that could not be analysed within reasonable time.We used G OANNA to verify liveness properties of drivers as explained in Section 4.2. G
OANNA performs a less precise analysis than S AT A BS and is therefore much faster. It verified all drivers in lessthan 1 minute while generating 8 false positives due to imprecise data flow analysis.These results demonstrate that active drivers’ protocol compliance can be verified using existingtools. At the same time they suggest that an optimal combination of accuracy and verification timerequires a trade-off between full-blown predicate abstraction of S AT A BS and purely syntactic analysis ofG OANNA . Comparison with conventional driver verification
In order to compare the effectiveness of ourverification methodology against conventional verification techniques for passive drivers, we carried outidney Amaniet al. 13a case study using the native Linux version of the RTL8169 Ethernet controller driver. We analysed thehistory of bug fixes made to this driver, and identified those fixes that address OS interface violation bugs.A typical example involves the driver attempting to use an OS resource such as timer after it has beendestroyed by a racing thread. We found 12 such bugs. We apply S AT A BS to detect these bugs. S AT A BS has been successfully applied to Linux drivers in the past [22]. Using S AT A BS as a model checker forboth active and traditional drivers provides a fair comparison.Detecting OS interface bugs in a passive driver requires a model of the OS. We built a series ofsuch models of increasing complexity so that each new model reveals additional errors but introducesadditional execution traces and is therefore harder to verify. This way we explore the best-case scenariofor the passive driver verification methodology: using our knowledge of the error we tune the model forthis exact error. In practice more general and hence less efficient models are used in driver verification.By gradually improving the OS model, we were able to find 8 out of 12 bugs. However, when beingprovided a model accurate enough to trigger the remaining 4 errors, S AT A BS was not able to find thebugs before being interrupted after 12 hours.We carried out an equivalent case study on the active version of the RTL8169 driver. To this end,we simulated the 12 OS protocol violations found in the native Linux driver in the active driver. Wewere able to detect each of the 12 protocol violation bugs within 3 minutes per bug. This result confirmsthat the active driver architecture along with the verification methodology presented above lead to devicedrivers that are more amenable to automatic verification than passive drivers. Microbenchmarks
The performance of active drivers depends on the overhead introduced by threadswitching and message passing. We measure this overhead on a machine with 2 quad-core 1.5GHz XeonCPUs.In the first set of experiments, we measure the communication throughput by sending a stream ofmessages from a normal kernel thread to a thread inside a cooperative domain. Messages are buffered inthe message queue and delivered in batches when the cooperative domain is activated by the scheduler.This setup simulates streaming of network packets through an Ethernet driver. The achieved throughput is2 · messages/s (500 ns/message) with both threads running on the same core and 1 . · messages/s(800 ns/message) with the two threads assigned to different cores on the same chip.Second, we run the same experiment with varying number of kernel threads distributed across avail-able CPU cores (without enforcing CPU affinity), with each Linux thread communicating with the co-operative thread through a separate mailbox. As shown in Figure 4, we do not observe any noticeabledegradation of the throughput or CPU utilisation as the number of clients contending to communicatewith the single server thread increases (the drop between one and two client threads is due to the highercost of inter-CPU communication). This shows that our implementation of message queueing scales wellwith the number of clients.Third, we measure the communication latency between a Linux thread and an active driver threadrunning on the same CPU by bouncing a message between them in a ping-pong fashion. The averagemeasured roundtrip latency is 1.8 m s. For comparison, the roundtrip latency of a Gigabit network link isat least 55 m s [19]. Macrobenchmarks
We compare the performance of the active RTL8169 Ethernet controller driveragainst equivalent native Linux driver using the Netperf benchmark suite on a 2.9GHz quad-core IntelCore i7 machine. Results of the comparison are shown in Figure 5. In the first set of experiments4 Automatic Verification of Message-Based Device Drivers t h r o u gp u t ( ^ m s g / s ) C P U u t ili s a t i o n ( % ) throughputCPU utilisation Figure 4: Message throughput and aggregate CPU utilisation over 8 CPUs for varying number of clients. transfer size (bytes) t h r o u g h p u t ( M b i t / s ) active drivernative Linux driver number of clients t h r o u g h p u t ( M b i t / s ) active drivernative Linux driver transfer size a v g l a t e n c y ( u s ) active drivernative Linux driver transfer size (bytes) C P U u t ili s a t i o n ( % ) active drivernative Linux driver number of clients a v g C P U u t ili s a t i o n ( % ) active drivernative Linux driver transfer size (bytes) C P U u t ili s a t i o n ( % ) active drivernative Linux driver (a) UDP throughput for varyingpacket sizes for a single client. Thetop graph shows achieved throughput;the bottom graph shows CPU utilisa-tion. (b) UDP throughput for multipleclients (packet size=64 bytes). Thetop graph shows aggregate through-put; the bottom graph shows averageCPU utilisation across 8 cores. (c) UDP latency for varying packetsizes for a single client. The top graphshows average round-trip latency; thebottom graph show CPU utilisation. Figure 5: Performance of the RTL8169 Ethernet driver measured with Netperf.we send a stream of UDP packets from the client to the host machine, measuring achieved throughput(using Netperf) and CPU utilisation (using oprofile ) for different payload sizes. The client machineis equipped with a 2GHz AMD Opteron CPU and a Broadcom NetXtreme BCM5704 NIC. The activedriver achieved the same throughput as the native Linux driver on all packet sizes, while using 20% moreCPU in the worst case (Figure 5(a)).In the second set of experiments, we fix payload size to 64 bytes and vary the number of clientsgenerating UDP traffic to the host between 1 and 8. The clients are distributed across four 2GHz IntelCeleron machines with an Intel PRO/1000 MT NIC. The results (Figure 5(b)) show that the active driversustains up to 10% higher throughput while using proportionally more CPU. Further analysis revealedthat the throughput improvement is due to slightly higher IRQ latency, which allows the driver to handlemore packets per interrupt, leading to lower packet loss rate.The third set of experiments measures the round trip communication latency between the host and aremote client with 2GHz AMD Opteron and NetXtreme BCM5704 NIC. Figure 5(c) shows that the la-tency introduced by message passing is completely masked by the network latency in these experiments.idney Amaniet al. 15 w r i t e / r e - w r i t e r e a d / r e - r e a d r a n d o m r / w r e v e r s e r e a d s t r i d e d r e a d m i x e d p w r i t e p r e a d C P U u t ili s a t i o n ( % ) Native Linux driversActive drivers
Figure 6: Native vs. active AHCI and ATA framework driver performance on the iozone benchmark.We evaluate the performance of the AHCI SATA controller driver using the iozone benchmarksuite running on a system with a 2.33GHz Intel Core 2 Duo CPU, Marvell 88SE9123 PCIe 2.0 SATAcontroller, and WD Caviar SATA-II 7200 RPM hard disk. We run the benchmark with working set of500MB on top of the raw disk.We benchmark the driver against equivalent Linux driver. Both drivers achieved the same I/Othroughput on all tests, while the active driver’s CPU utilisation was slightly higher (Figure 6). Thisoverhead can be reduced through improved protocol design. Our SATA driver protocol, based on theequivalent Linux interface requires 10 messages for each I/O operation. A clean-slate redesign of thisprotocol would involve much fewer messages.We did not benchmark the DAI driver, as it has trivial performance requirements and uses less than5% of CPU.
Active drivers
Singularity [12] is a research OS written in the Sing coord objects that expose well-defined sequential protocols to the user.
Verification tools
Automatic verification tools for C [3, 9, 8, 16]is an active area of research, which iscomplementary to our work on making drivers amenable to formal analysis using such tools. Several ver-ification tools, including SPIN [18], focus on checking message-based protocols in distributed systems.These tools work on an abstract model of the system that is either written by the user or extracted fromthe program source code [17]. Such a model constitutes a fixed abstraction of the system that cannot be6 Automatic Verification of Message-Based Device Driversautomatically refined if it proves too coarse to verify the property in question. Our experiments showthat abstraction refinement is essential to avoiding false positives in active driver verification; thereforewe do not expect these tools to perform well on active driver verification tasks.
Improvements in automatic device driver verification cannot rely solely on smarter verification tools andrequire an improved driver architecture. Previous proposals for verification-friendly drivers were basedon specialised language and OS support and were not compatible with existing systems. Based on ideasfrom this earlier research, we developed a driver architecture and verification methodology that can beimplemented in any existing OS. Our experiments confirm that this methodology enables more thoroughverification of the driver-OS interface than what is possible for conventional drivers.
We would like to thank Michael Tautschnig for his help in troubleshooting S AT A BS issues. We thank theG OANNA team, in particular Mark Bradley and Ansgar Fehnker, for explaining G
OANNA internals andproviding us with numerous ideas and examples of verifying active driver properties using G
OANNA .We thank Toby Murray for his feedback on a draft of the paper.NICTA is funded by the Australian Government as represented by the Department of Broadband,Communications and the Digital Economy and the Australian Research Council through the ICT Centreof Excellence program.
References [1] A. Adya, J. Howell, M. Theimer, W. Bolosky & J. Douceur (2002):
Cooperative task management withoutmanual stack management . In: 2002USENIX, Monterey, CA, USA, pp. 289–302.[2] Sidney Amani, Peter Chubb, Alastair Donaldson, Alexander Legg, Leonid Ryzhyk & Yanjin Zhu (2012):
Active Device Drivers . NICTA Technical Report, NICTA. .[3] Thomas Ball, Ella Bounimova, Byron Cook, Vladimir Levin, Jakob Lichtenberg, Con McGarvey, BohusOndrusek, Sriram K. Rajamani & Abdullah Ustuner (2006):
Thorough Static Analysis of Device Drivers . In:1stEuroSysConf., Leuven, Belgium, pp. 73–85, doi: .[4] Fred Barnes & Carl Ritson (2009):
Checking Process-Oriented Operating System Behaviour Using CSP andRefinement . Operat.Syst.Rev. 43(4), pp. 45–49, doi: .[5] G´erard Basler, Matthew Hague, Daniel Kroening, C.-H. Luke Ong, Thomas Wahl & Haoxian Zhao (2010):
Boom: Taking Boolean Program Model Checking One Step Further . In: TACAS, LectureNotesinComputerScience 6015, Springer, pp. 145–149, doi: .[6] Prakash Chandrasekaran, Christopher L. Conway, Joseph M. Joy & Sriram K. Rajamani (2007):
Program-ming asynchronous layers with CLARITY . In: 6thESEC, Dubrovnik, Croatia, pp. 65–74.[7] Andy Chou, Jun-Feng Yang, Benjamin Chelf, Seth Hallem & Dawson Engler (2001):
An Empirical Study ofOperating Systems Errors . In: 18thSOSP, Lake Louise, Alta, Canada, pp. 73–88.[8] Edmund M. Clarke, Daniel Kroening, Natasha Sharygina & Karen Yorav (2004):
Predicate Abstraction ofANSI-C Programs Using SAT . Formal Methods in System Design 25(2-3), pp. 105–127, doi: . idney Amaniet al. 17 [9] Byron Cook, Andreas Podelski & Andrey Rybalchenko (2006): Termination proofs for systems code . In:2006PLDI, Ottawa, Ontario, Canada, pp. 415–426, doi: .[10] Doron Peled Edmund M. Clarke, Orna Grumberg (1999):
Model Checking . MIT Press.[11] Dawson R. Engler, Benjamin Chelf, Andy Chou & Seth Hallem (2000):
Checking System Rules UsingSystem-Specific, Programmer-Written Compiler Extensions . In: 4thOSDI, pp. 1–16.[12] Manuel F¨ahndrich, Mark Aiken, Chris Hawblitzel, Orion Hodson, Galen C. Hunt, James R. Larus & StevenLevi (2006):
Language Support for Fast and Reliable Message-Based Communication in Singularity OS . In:1stEuroSysConf., Leuven, Belgium, pp. 177–190, doi: .[13] Ansgar Fehnker, Ralf Huuck, Patrick Jayet, Michel Lussenburg & Felix Rauch (2006):
Goanna — A StaticModel Checker . In: 11thFMICS, Bonn, Germany, pp. 297–300.[14] Archana Ganapathi, Viji Ganapathi & David Patterson (2006):
Windows XP Kernel Crash Analysis . In: 20thLISA, Washington, DC, USA, pp. 101–111.[15] David Harel (1987):
Statecharts: A Visual Formalism for Complex Systems . ScienceofComputerProgram-ming 8(3), pp. 231–274, doi: .[16] Thomas A. Henzinger, Ranjit Jhala, Rupak Majumdar, George C. Necula, Gr´egoire Sutre & Westley Weimer(2002):
Temporal-Safety Proofs for Systems Code . In: 14thCAV, Copenhagen, Denmark, pp. 526–538.[17] Gerard J. Holzmann (2000):
Logic Verification of ANSI-C Code with SPIN . In: 7thSPIN, pp. 131–147.[18] Gerard J. Holzmann (2003):
The SPIN Model Checker: Primer and Reference Manual , 1st edition. Addison-Wesley Professional.[19] Richard Hughes-Jones, Peter Clarke & Steven Dallison (2005):
Performance of 1 and 10 Gigabit Ethernetcards with server quality motherboards . FutureGenerationComputerSystems 21(4), pp. 469–488, doi: .[20] Kenneth McMillan (1993):
Symbolic Model Checking: An Approach to the State Explosion Problem . KluwerAcademic.[21] Leonid Ryzhyk, Peter Chubb, Ihor Kuz & Gernot Heiser (2009):
Dingo: Taming Device Drivers . In: 4thEuroSysConf., Nuremberg, Germany.[22] Thomas Witkowski, Nicolas Blanc, Daniel Kroening & Georg Weissenbacher (2007):