Automatically Checking Conformance on Asynchronous Reactive Systems
AAutomatically Checking Conformance on AsynchronousReactive Systems
Camila Sonada Gomes ∗ Adilson Luiz Bonifacio † Abstract
Software testing is one of the most important issues in the software development process toensure the quality of the products. Formal methods have been promising on testing reactivesystems, specially critical systems, where accuracy is mandatory and any fault can cause severedamage. Systems of this nature are characterized by receiving messages from the environmentand producing outputs in response. One of the biggest challenges in model-based testing isthe conformance checking of asynchronous reactive systems modeled by IOLTSs. The aim isto verify if an implementation is in compliance with its respective specification. In this work,we develop a practical tool to check the conformance relation between reactive models usinga more general theory based on regular languages. Our tool also offers the ioco conformancechecking and an intuitive interface to apply practical experiments. In addition, we presentsome testing scenarios in practical applications and compare to other tools from the literatureusing both notions of conformance.
Keywords: model-based testing conformance testing automatic verification reactive sys-tems.
Automatic testing tools have been proposed to support the development process of reactive systemsthat are characterized by continuous interaction with the environment. In this setting, systemsreceive external stimuli and produce outputs, asynchronously, in response. In addition, systems ofthis nature are usually critical and require more accuracy in their development process, especiallyin the testing activity, where appropriate formalisms must be used as the basis [8, 12, 23]. InputOutput Labeled Transition Systems (IOLTSs) [12, 23, 4, 22] are traditional formalisms usuallyapplied to model and test reactive systems.In model-based testing, an IOLTS specification can model desirable and undesirable behaviorsof an implementation under test (IUT). The aim is to find faults in an IUT according to a certainfault model [21, 8, 5] in order to show if requirements are satisfied regarding its respective systemspecification. A well-established conformance relation, called ioco and proposed by Tretmans [21],requires that the outputs produced by an IUT must also be produced by its respective specifica-tion, following the same behavior. A more recent and general conformance relation, proposed byBonifacio and Moura [8], specifies desirable and undesirable behaviors using regular languages todefine the testing fault model. ∗ Computing Department, University of Londrina, Londrina, Brazil. † Computing Department, University of Londrina, Londrina, Brazil. a r X i v : . [ c s . S E ] A ug n this work, we address the development of an automatic tool for conformance verification ofasynchronous reactive systems modeled by IOLTSs. Here we introduce in a practical tool bothnotions of conformance in order to provide a wider application range compare to other tools.JTorx [2], for instance, is a tool from the literature that also implements a conformance testingverification process, but only based on the classical ioco relation. Our tool comprises both theclassical ioco relation and also the more general conformance based on regular languages. We alsorun some practical sceneries to evaluate aspects related to the effectiveness and usability of bothconformance theories and these tools.We organize this paper as follows. Section 2 describes the conformance verification methodsusing regular languages and the ioco relation. The practical tool which implements the more generalmethod of conformance checking is presented in Section 3. Section 4 describes the comparativeanalysis of tools. Some applications and a comparative study are given in Section 5. Section 6offers some concluding remarks. The conformance checking task can determine if an IUT is in compliance with its specificationwhen both are modeled by appropriate formalisms. The classical ioco conformance relation [22, 23]establishes the compliance between IUTs and specifications when they are specified by IOLTS [22,12, 23, 4]. An IOLTS is a variation of the Labeled Transition Systems (LTS) [20, 13, 24, 11] withthe partitioning of input and output labels.
Definition 1
An IOLTS is given by S = ( S, s , L I , L U , T ) where: • S is the set of states; • s ∈ S is the initial state; • L I is a set of input labels; • L U is a set of output labels; • L = L I ∪ L U and L I ∩ L U = ∅ ; • T ⊆ S × ( L ∪ { τ } ) × S is a finite set of transitions, where the internal action τ / ∈ L ; and • ( S, s , L, T ) is the underlying LTS associated with S . A transition ( s, l, s (cid:48) ) ∈ T indicates that from the state s ∈ S with the label l ∈ ( L ∪ { τ } ) the state s (cid:48) ∈ S is reached in an LTS/IOLTS model. When we have a transition ( s, τ, s (cid:48) ) ∈ T with an internalaction it means that an external observer can not see the movement from state s to state s (cid:48) in themodel.We may also have the notion of quiescent states in IOLTS models. If a state s of an IOLTShas no output x ∈ L U and internal action τ defined on it, we say that s is quiescent [23]. When astate s is quiescent we then add a transition ( s, δ, s ), where δ / ∈ L τ . In a real scenario of black-boxtesting where an IUT sends messages to a tester and receives back responses, quiescence indicatesthat an IUT could no longer respond to the tester, or it has timed out, or even it is simply slow.We will need the semantics of LTS/IOLTS models, but first we introduce the notion of paths. In order to ease the notation we denote L ∪ { τ } by L τ . efinition 2 ([8]) Let S = ( S, s , L, T ) be a LTS and p, q ∈ S . Let σ = l , · · · , l n be a word in L (cid:63)τ .We say that σ is a path from p to q in S if there are states r i ∈ S , and labels l i ∈ L τ , ≤ i ≤ n ,such that ( r i − , l i , r i ) ∈ T , ≤ i ≤ n , with r = p and r n = q . We also say that α is an observablepath from p to q in S if we remove the internal actions τ from σ . A path can also be denoted by s σ −→ s (cid:48) , where the behavior σ ∈ L (cid:63)τ starts in the state s ∈ S andreaches the state s (cid:48) ∈ S . An observable path σ , from s to s (cid:48) , is denoted by s σ = ⇒ s (cid:48) . We can alsowrite s σ −→ or s σ = ⇒ when the final state is not important.Paths that start from state s are called paths of s , and the semantics of an LTS model is givenby the paths that start from their initial state. The semantics of the LTS model are given below. Definition 3 ([8]). Let S = ( S, s , L, T ) be a LTS and s ∈ S :1. The set of paths of s is given by tr ( s ) = { σ | s σ −→} and the set of observable paths of s is otr ( s ) = { σ | s σ = ⇒} .2. The semantics of S is tr ( s ) or tr ( S ) and the observable semantics of S is otr ( s ) or otr ( S ) . The semantics of an IOLTS is defined by the semantics of the underlying LTS.Conformance checking can be established between IOLTS models over the ioco relation. Whenwe apply input stimuli to both a specification and an IUT, if the IUT produces outputs that arealso defined in the specification, we say that the IUT conforms to the specification. Otherwise, wesay that they do not conform [23].
Definition 4 ([23, 8]). Let S = ( S, s , L I , L U , T ) be a specification and I = ( Q, q , L I , L U , R ) be an IUT. We say that I ioco S if, and only if, out ( q af ter σ ) ⊆ out ( s af ter σ ) for all σ ∈ otr ( S ) , where s af ter σ = { q | s σ = ⇒ q } for all s ∈ S and all σ ∈ otr ( S ) , and the function out ( V ) = (cid:83) s ∈ V { l ∈ L U | s l = ⇒} . On the other hand, a more general conformance relation has been proposed by Bonifacio andMoura [8], where the fault model is defined by regular languages. This approach provides a widerfault coverage for both LTS and IOLTS models. Basically, desirable and undesirable behaviors arespecified by regular expressions, D and F , respectively. Given an implementation I , a specification S , and regular languages D and F , I is in compliance with S according ( D, F ), i.e, I conf D,F S if,and only if, no undesirable behavior of F is observed in I and is specified in S , and all desirablebehaviors of D are observed in I and also are specified in S . Definition 5 ([8]) Let a set of symbols L , and the languages D , F ⊆ L (cid:63) over L . Let S and I , LTSmodels, with L as their set of labels, I conf D,F S if, and only if:1. σ ∈ otr ( I ) ∩ F , then σ / ∈ otr ( S ) ; and2. σ ∈ otr ( I ) ∩ D , then σ ∈ otr ( S ) . We remark that ordinary LTS models can be checked using the language-based conformanceapproach using only the notion of desirable and undesirable behaviors. In this case, we do not needto partition the alphabet into input and output labels, as required by IOLTS models and crucialfor ioco relation. 3 roposition 1 ([8]). Let the specification S and the IUT I be LTS models over L , and the lan-guages D, F ⊆ L (cid:63) over L . We say that I conf D,F S if, and only if, otr ( I ) ∩ [( D ∩ otr ( S )) ∩ ( F ∩ otr ( S ))] = ∅ , where otr ( S ) is the complement of otr ( S ) given by otr ( S ) = L (cid:63) − otr ( S ) . Lemma 1 shows that the more general notion of conformance relation given in Definition 5restrains the classical ioco conformance relation.
Lemma 1 ([8]). Let a specification S = ( S, s , L I , L U , T ) and an IUT I = ( Q, q , L I , L U , R ) beIOLTS models, we have that I ioco S if, and only if, I conf D,F S when D = otr ( S ) L U e F = ∅ . Clearly, the ioco relation can be given by the more general conformance relation using regularlanguages.In what follows we will see that the conformance checking can be obtained using the automatatheory. But first, LTS/IOLTS models must be transformed into Finite State Automatons (FSAs) asproposed by Bonifacio and Moura [8]. Hence, the union, intersection, and complement operationsover regular languages can be used in the conformance verification process.We start by formally defining FSA models [19].
Definition 6
A FSA M is defined as ( S, s , L, T, F ) where: • S is the set of states; • s is the initial state; • L is the alphabet; • T ⊆ S × ( L ∪ { (cid:15) } ) × S is the finite set of transitions, (cid:15) / ∈ L represents empty string; • F ⊆ S is the set of final states or acceptance states. Any LTS/IOLTS S can be transformed into a FSA M by mapping internal actions τ of S into (cid:15) -transitions in M , and all states of S are defined as final states in M . Definition 7 ([8]) Let S = ( S, s , L, T ) be a LTS. The FSA induced by S is A S = ( S, s , L, ρ, S ) where, for all p, q ∈ S and all l ∈ L , we have: • ( p, l, q ) ∈ ρ if and only if ( p, l, q ) ∈ T ; • ( p, (cid:15), q ) ∈ ρ if and only if ( p, τ, q ) ∈ T . The semantics of a FSA is given by the language it accepts. A language R ⊆ L (cid:63) is regular ifthere is a FSA M such that L ( M ) = R where L is an alphabet [19]. Since D and F are regularlanguages, we can effectively construct the automatons A D and A F such that D = L ( A D ) and F = L ( A F ).The following proposition says how to obtain FSAs that capture the union, the intersection andthe complement of regular languages accepted by their respective automatons. Proposition 2 ([8]) Let L be a set of symbols, and let A = ( S A , a , L, ρ A , F A ) and B = ( S B , b , L, ρ B , F B ) be FSAs. Then we can effectively construct a FSA T = ( S T , t , L, ρ T , F T ) such that1. L ( T ) = L ( A ) ∩ L ( B ) ; . L ( T ) = L ( A ) ∪ L ( B ) ;3. L ( T ) = L ( A ) = L ∗ − L ( A ) . Next, we introduce the notions of the test case and test suite according to formal languages.
Definition 8 ([8]). Let a set of symbols L , the test suite T over L is a language, where T ⊆ L (cid:63) ,so that each σ ∈ T is a test case. If the test suite is a regular language, then there is a FSA A that accepts it, such that the finalstates of A are fault states. The set of undesirable behaviors, defined by these fault states, is calledby fault model of S [8].Lemma 2 says that a complete test suite can be obtained from an IOLTS specification S and apair of languages ( D, F ) using the Proposition 1. The test suite is able to identify the absence ofdesirable behaviors specified by D and the presence of undesirable behaviors specified by F in thespecification S . Lemma 2 ([8]). Let an alphabet L , a specification LTS S and regular languages D, F ⊆ L (cid:63) . Theonly complete test suite for S using ( D, F ) is given by: T = [( D ∩ otr ( S )) ∪ ( F ∩ otr ( S ))] . We declare that an implementation I is in compliance with a specification S if there is no testcase of the test suite T that is also a behavior of I [8]. Definition 9 ([8]). Let the test suite T and a IUT I . I adheres to T if, and only if, for all σ ∈ otr ( I ) have to σ / ∈ T . We also need the notion of determinism over LTS/IOLTS models.
Definition 10
Let S = ( S, s , L, T ) be a LTS. We say that S is deterministic if S has no τ -labeledtransitions and, s σ = ⇒ s and s σ = ⇒ s imply s = s for all s , s ∈ S and all σ ∈ L ∗ . Let S be a deterministic IOLTS, so the automaton A induced by S is also deterministic,and we write L ( A ) = otr ( S ). Hence, we can effectively obtain a FSA A such that L ( A ) = L ( A F ) ∩ L ( A ) = F ∩ otr ( S ) by Proposition 2.Consider the FSA B obtained from A by reversing its set of final states, that is, a state s isa final state in B if and only if s is not a final state in A . Clearly, L ( B ) = L ( A ) = otr ( S ). Wecan now effectively get a FSA B such that L ( B ) = L ( A D ) ∩ L ( B ) = D ∩ otr ( S ).Since A and B are FSAs, we can construct a FSA C such that L ( C ) = L ( A ) ∪ L ( B ), where L ( C ) = T . We can conclude that when D and F are regular languages and S is a deterministicspecification, then a complete FSA T can be constructed such that L ( T ) = T . Proposition 3 ([8]) Let S and I be the deterministic specification and implementation IOLTSsover L with n S and n I states, respectively. Let also | L | = n L . Let A D and A F be deterministicFSAs over L with n D and n F states, respectively, and such that L ( A D ) = D and L ( A F ) = F .Then, we can effectively construct a complete FSA T with ( n S + 1) n D n F states, and such that L ( T ) is a complete test suite for S and ( D, F ) . Moreover, there is an algorithm, with polynomialtime complexity Θ( n S n I n D n F n L ) that effectively checks whether I conf D,F S holds. We obtain a similar result for the ioco conformance relation using Lemma 1.5 heorem 1 ([8]) Let S and I be deterministic specification and implementation IOLTSs over L with n S and n I states, respectively. Let L = L I ∪ L U , and | L | = n L . Then, we can effectivelyconstruct an algorithm with polynomial time complexity Θ( n S n I n L ) that checks whether I ioco S holds. In this work, we have developed the automatic checking conformance tool
Everest for reactivesystems modeled by LTS/IOLTS. Our tool supports the more general notion of conformance basedon regular languages and also the classical notion of ioco relation. Everest has been developed inJava [17] using the
Swing library [18], providing a yielding and friendly usability experience througha graphical interface.Some features provided by the Everest tool are: (i) check conformance based on regular languagesand ioco relation; (ii) describe desirable and undesirable behaviors using regular expressions; (iii)specify formal models in Aldebaran format [1]; (iv) generate test suites when non-conformanceverdicts are obtained; (v) provide state paths, i.e, the sequence of states induced by a test case overthe IUT and specification; and (vi) allow the graphical representation of the formal models.The tool’s architecture is organized in four modules as depicted in Figure 1. The modules aregiven by rectangles and the data flow between them is denoted by the arrows. The input data andthe output results are represented by ellipses.The
View module implements an intuitive graphical interface with three different views: con-figuration; ioco conformance; and language-based conformance. In the configuration view we setthe specification and implementation models (in Aldebaran format [9]), the model type (LTS orIOLTS) under test, and the partition into input and output labels for IOLTS models, if it is thecase. In the ioco and language-based conformance views, we run both verification processes andalso graphically inspect the models to ascertain the configuration information. After finishing thetesting process, if the IUT does not conform to the specification then a negative verdict is displayedtogether with the associated test cases. Otherwise, the tool displays a positive verdict of confor-mance. Note that we can provide regular expressions to represent the desirable behaviors and thefault properties in the language-based conformance view.The IUT and specification models are validated by the
Parser module where data structuresare constructed to internally represent them. The
Automaton Construction module transforms theLTS/IOLTS models into their respective finite automatons which, in turn, are used to constructthe fault model together with the automatons obtained by means of regular languages.The
Conformance Verification module provides all necessary operations over regular languagessuch as union, intersection, and complement [19]. This module also constructs the finite automatonthat represents the complete test suite and comprises both conformance verification techniques.Next, we describe the conformance checking processes and their essential algorithms. Someother basic algorithms are not described here since they are given by classical constructions on theautomata theory, such as intersection, complement, and union operations over regular languages.In the same vein, we omit the conversion algorithm from regular languages into the automatonsthat accept them, the conversion algorithm from LTS/IOLTS models into the associated FSAs, andthe automaton determinization algorithm. EVEREST – conformancE Verification on tEsting ReactivE SysTems
Available in https://everest-tool.github.io/everest-site
The Algorithm 1 performs the ioco verification given the IOLTS models of the specification S andthe IUT I . The line 2 of the algorithm indicates the fault model construction for the specification S represented by the automaton A T . At line 3 the IUT I is transformed into the automaton A I .Lines 4 and 5 build the automaton A B which recognizes the language obtained from the intersectionof the languages accepted by the automatons A T and A I . Note that A T represents the fault modelthat captures the behaviors that should not be found in the IUT. When this intersection is notempty it means that some behavior is present in both the fault model and the IUT. In this case, thebehavior is a word accepted by the automaton A B thus its set of acceptance states E it not empty.Then, from line 6 to line 10 whenever the set E of final states is empty the IUT I is declared incompliance to the specification S . Otherwise, a fault is detected and I is declared non-conformingto the specification S . Algoritmo 1
Verify ioco
Conformance input:
IOLTSs S , I output: I ioco conforms to S , or not function verifyIOCOConformance ( S , I ) A T = f aultM odelIoco ( S ) A I = ioltsT oAutomaton ( I ) A B = ( S B , s , L, δ, E ) A B = intersection ( A T , A I ) if E = ∅ then return I ioco S else return I ioco S end if end function The Algorithm 2 constructs the fault model automaton A T based on specification S . The fault7odel is the set of behaviors that should not be found in the implementation. The construction ofthis model is based on Lemma 2, where the sets of behaviors D and F are defined as D = otr ( S ) L U and F = ∅ .At line 2 of the algorithm is constructed the automaton that recognizes the behaviors in thespecification that ends with the production of an output. At line 3 the automaton A S is obtainedinduced by the specification S . At line 4 the automaton A compS is constructed to accept the be-haviors that are not present in the language accepted by the specification automaton. The faultmodel is then obtained at line 5, where automaton A T is constructed in order to capture the com-mon behaviors of the languages accepted by the automaton A D e A compS . Thus, the fault modelrepresents the behaviors considered as faults in candidate implementations. Algoritmo 2
Fault Model iocoinput:
IOLTS S output: The fault model automaton function faultModelIoco ( S ) A D = modelD ( S ) A S = ioltsT oAutomaton ( S ) A compS = complement ( A S ) A T = intersection ( A D , A compS ) return A T end function The construction of an automaton that accepts the specification behaviors that end with an outputis obtained by Algorithm 3. The automaton A D is constructed to capture the behaviors defined bythe set D = otr ( Q ) L U .The states set of A D is defined at line 3, by the states set of Q and the special state d . Thestate d then must be the only acceptance state (line 6) of A D . The transitions of δ are created asfollows:1. Lines 7 to 13 transform transitions τ of IOLTS Q in transitions (cid:15) of automaton A D ; or intransitions with the same label.2. Lines 14 to 18 create new transitions in δ with output labels L U from every state of q ∈ Q tothe special state d .The resulting automaton A D can be non-deterministic, then line 19 returns a deterministic au-tomaton using the function convertToDeterministicAutomaton . Algoritmo 3
Model D input:
IOLTS Q = ( Q, q , L I , L U , T ) output: A D automaton function modelD ( Q ) A D = ( Q, q , L, δ, F ) Q = Q ∪ d : L = L I ∪ L U δ = ∅ F = d for ( p, x, r ) ∈ T do if x = τ then δ = δ ∪ ( p, (cid:15), r ) else δ = δ ∪ ( p, x, r ) end if end for for q ∈ Q do for l ∈ L U do δ = δ ∪ ( q, l, d ) end for end for return convertT oDeterministicAutomaton ( A D ) end function The language-based conformance verification is performed by Algorithm 4. Let an IUT I , a speci-fication S and the pair of languages ( D, F ), the algorithm checks whether I conforms to S , basedon the desirable and undesirable behaviors defined by regular expressions D e F , respectively,according to Proposition 1.The fault model is represented by automaton A T constructed at line 2, based on the specifica-tion S and the languages ( D, F ). At line 3 the automaton A I is obtained induced by I . At lines4 and 5, the automaton A B is constructed, which recognizes the common behaviors of the regularlanguages accepted by the automatons A T and A I . At lines 6 to 10, if the set of acceptance states E of the intersection is empty then there is no common behavior between the fault model andthe implementation. Therefore, no fault is detected in the IUT and the algorithm declares that I conforms to S based on ( D, F ). Otherwise, when at least one fault is detected in I , the algorithmreturns that I does not conform to S using ( D, F ). Algoritmo 4
Verify Language-Based Conformance input:
LTS S , I and regex ( D, F ) output: I conforms to S , or not function verifyLanguageConformance ( S , I , D, F ) A T = f aultM odelLanguage ( S , D, F ) A I = ltsT oAutomaton ( I ) A B = ( S B , s , L, δ, E ) A B = intersection ( A T , A I ) if E = ∅ then return I conforms to S else return I does not conform to S end if end function The language-based fault model is obtained by Algorithm 5 based on Lemma 2. The constructionof the automaton that represents the fault model begins with the transformation of the regularexpressions D and F into the corresponding automatons that capture their respective languages,at lines 2 and 3. The function regexT oAutomaton gets the regular expressions D and F andconstructs the automatons A D and A F . Line 4 constructs the automaton A S induced by the LTS S . At line 5 the automaton A compS is constructed to accept the language complement accepted by A S .At line 6 the automaton A failD is constructed, representing the common behaviors between A D and A compS . The intersection contains desirable behaviors that are not present in the specification A S . At line 7 the automaton A failF is constructed representing the common behaviors acceptedby A F and A S . The intersection should include the fault behaviors present in the specification.Finally, at line 8 the automaton A T which corresponds to the union of all fault behaviors is con-structed, either by the absence of some desirable behavior that is not in S or by the presence ofsome undesirable behavior that is present in S . Algoritmo 5
Language-Based Fault Model input:
LTS S e the regex ( D, F ) output: The fault model automaton function faultModelLanguage ( S , D, F ) A D = regexT oAutomaton ( D ) A F = regexT oAutomaton ( F ) A S = ltsT oAutomaton ( S ) A compS = complement ( A S ) A failD = intersection ( A D , A compS ) A failF = intersection ( A F , A S ) A T = union ( A failD , A failF ) return A T end function The conformance verification process using Everest needs of a standard representation of the models.So LTS/IOLTS models should be modeled in the Aldebaran format [9] as a set of transitions.
Definition 11 [10] Models in Aldebaran format are described by:des(¡initial-state¿, ¡number-of-transitions¿, ¡number-of-states¿)(¡ini-state¿, ¡label¿, ¡end-state¿)
The initial state (¡initial-state¿), the number of transitions (¡number-of-transitions¿) and the num-ber of states (¡number-of-states¿) are defined in the file header. Then, the set of model transitions10s defined by the name of the source state (¡ini-state¿), by the transition label (¡label¿) and thename of the target state (¡end-state¿).Figure 2a presents an example of Aldebaran file ( .aut ) and Figure 2b shows the IOLTS diagramunderlying to the Aldebaran model. des (s0,9,4)(s0,?a,s1)(s0,?b,s3)(s1,?b,s2)(s1,!x,s2)(s1,?a,s3)(s2,?b,s2)(s2,!x,s3)(s3,?b,s0)(s3,?a,s3) (a) IOLTS S in Aldebaranformat. s s s s ab a b, xx bb a (b) IOLTS specification S . Figure 2: An example of Aldebaran file format.As mentioned, the Everest tool interface consists of three views: configuration; ioco conformanceand language-based conformance. Figure 3 presents the configuration interface, where specificationand the implementation models are selected, in the Aldebaran format. When the model type(defined in
M odel type ) is an LTS, fields
Label , Input labels , and
Output labels are hidden. IfIOLTS models are given then we need to inform how the input/output labels are distinguished(field
Label ), informing in the fields
Input labels and
Output labels , or in the file .aut of thespecification/implementation by markers “!”(output label) e “?”(input label).Figure 3: Interface: Configuration.Figure 4 presents the interface for ioco conformance verification. In the interface header, thefields fill in the configuration view is kept, as the items
M odel and
Implementation which display11he names of the files used as specification and implementation, in addition to the fields
Input Label and
Output Label , which shows the input and output labels of the models. The buttons view model and view IU T allow the graphical visualization of the implementation and specification models.The ioco conformance verification does not require any additional information, therefore, byclicking the button
V erif y , the verdict is textually displayed. In case of non-conformance, the toolpresents (in the text field below the button), for each test case that detects a fault, a set of pathsinduced in such a way as to obtain a transition coverage of the specification. The item
W arnings informs the incorrectly filled fields in the configuration view, which are essential for the verificationto be performed. Figure 4: Interface: ioco verification.Figure 5 displays the interface for language-based conformance verification. Regular expres-sions must be informed in the fields
Desirable behavior and
Undesirable behavior , which specifydesirable and undesirable behaviors, respectively. When no regular expression is provided, it isassumed Kleene closure [19] over the alphabet, in order to identify faults when models are not iso-morphic. After, clicking on button
V erif y the verdict is displayed similarly to the ioco verificationconformance.
In this section, we perform a comparative analysis between the Everest tool and JTorx. Both toolsimplement the ioco theory [23] and JTorx supports both the white-box testing, when the IUTstructure is known, and the black-box testing, when the internal structure of the IUT is unknown.In the black-box testing, JTorx uses the
Adapter module to implement the communication betweenthe IUT and the specification through the TCP or UDP protocol. The conformance verification isaccomplished by comparing the outputs expected by the specification and those produced by theimplementation.Everest tool implements the conformance verification on white-box implementations, althoughthe generation of complete test suites for black-box implementations is currently in development.In addition to ioco verification, the Everest tool implements a more general conformance relationthat allows specifying fault models and properties through regular languages. These languages mayexpress desirable and/or undesirable behaviors over implementations differently from JTorx that isbased only on ioco conformance. 12igure 5: Interface: language-based verification.In the classical ioco conformance relation restrictions and properties are imposed on the models,such as underspecified models [7] not being allowed, that is, models that are not input-enabled . Suchrestrictions are not imposed on models in the language-based conformance verification. Both JTorxand Everest tools support underspecified models, but the states where input labels are not specified,JTorx adds self-loops . On the other hand, underspecified models are treated with no change by theEverest tool.The Everest test suite generation using the language-based method [8] is complete over language-based and ioco relation for white-box scenarios. The generation algorithms for complete test suitesin a black-box testing are under development. The JTorx test suite generation is exhaustive withrespect to ioco relation, which implies that it generates very large (infinite) sets of tests, with thesame effectiveness on detecting faults using ioco conformance verification [14].
This section describes some practical testing scenarios applied to the Everest tool and briefly com-pares with the JTorx tool. Let S be the IOLTS specification of Figure 2b and let R and Q beimplementations candidates as depicted in Figures 6a and 6b. Also let L I = { a, b } and L U = { x } be the input and output alphabets, respectively. All models here are deterministic [15, 16] but weremark that our tool also deals with nondeterministic models.In the first scenario, we check if IUT R conforms to specification S . Everest tool has returned anon-conformance verdict using ioco relation and generated the test suite { b, aa, ba, aaa, ab, ax, abb,axb } . The subset of test cases { b, aa, ba, aaa } induces state paths from s to s in S and from q to q in R , where the output x is produced by R but S does not. Note that s in S is a quiescentstate whence no output is defined on it. The subset { ab, ax, abb, axb } induces state paths to state s in S and q in R . In this case, the output δ (quiescent label) is produced by IUT R whereas S produces x . That is, a fault is detected according to ioco relation. Note that both tools modifythe formal models by adding self-loops labeled with δ [21] on quiescent states.The same scenario has been also applied to JTorx tool, resulting in the same verdict, as expected,13 q q q ab a b, xa bb,x a (a) Implementation R q q q q ab a b, xa,x bb a (b) Implementation Q Figure 6: IOLTS Modelsbut it generates the test suite { b, ax, ab } . Notice that the test suite generated by JTorx is a subsetof the test suite generated by Everest. That is, Everest shows all test cases and associated statepaths related to each fault according to a transition cover criteria over the specification, whereasJTorx returns only one test case per fault. All this information may be useful and aid the tester inthe fault mitigation process.In the second scenario, checking the IUT Q against the specification S , the language-basedconformance verification was able to detect a fault that was not detected by the ioco conformancerelation. We have obtained the fault model using the regular expressions D = ( a | b ) ∗ ax and F = ∅ .Language D clearly expresses behaviors that finish with a stimulus a followed by an output x produced in response. Since the only complete test suite is given by [( D ∩ otr ( S )) ∪ ( F ∩ otr ( S ))]and F ∩ otr ( S ) = ∅ , so we check the condition D ∩ otr ( S ) (cid:54) = ∅ , i.e., a fault is detected when behaviorsof D are not present in S . Everest then results in a verdict of non-conformance and produces thetest suite { ababax, abaabax } reaching a fault that is not detected by JTorx using the ioco relation.Following, the details of how the verdicts of ioco and language-based conformance were obtainedfrom the second scenario according to Everest tool are shown.The specification S (Figure 2b) and the candidate implementation Q (Figure 6b) are IOLTSmodels which, after being converted into underlying automatons respectively, A S and A Q , have alltheir states defined as final states. Figure 7 displays the complement automaton of the specification.The ioco conformance verification (Algorithm 1) first obtains the underlying automata, A S and A Q , from the IOLTS models. The automaton D (Figure 8a) is constructed according tothe Algorithm 3, in order to obtain the fault model of Figure 8b through the intersection of theautomaton D and the complement of the specification (Figure 7).The automaton that represents the test suite (Figure 8c) is obtained by Algorithm 2 with theintersection between the fault model and the given implementation. Since the resulting automatonhas no final state, the verdict between models is that I ioco conforms to S .In language-based conformance verification, as well as in the ioco verification, the underlyingautomatons A S and A Q , are obtained from the IOLTS models, according to Figure 6 and Figure 2.From regular expression ( a | b ) ∗ ax the automaton (Figure 9a) which accepts the respective languageis also obtained. Since the fault model is given by [ D ∩ otr ( S )] ∪ [( F ∩ otr ( S ))] (Algorithm 5) andno undesirable behavior F is informed, therefore F ∩ otr ( S ) = ∅ , so fault behaviors are reduced to14 s s s c ab xa b, xx b ab xa a, b, xFigure 7: Automaton A S : complement of specification A S D ∩ otr ( S ). The automaton that represents the fault model is illustrated in Figure 9b.Finally, the automaton that represents the test suite is illustrated in Figure 9c and obtained byAlgorithm 4. Note that this automaton contains a final state, indicating that the words accepted bythe automaton are part of the test suite that reveals the faults and, consequently, the non-conformitybetween the models. The test suite generated by the Everest tool are { ababax, abaabax } . Testing of reactive systems is an important and complex activity in the development process forsystems of this nature. The complexity of such systems and, consequently, the complexity of thetesting task require high costs and resources in software development. Therefore, automation ofthe testing activity has become essential and several studies have addressed the testing of reactivesystems [8, 12, 23, 3, 6]. More precisely, many works have focused on the conformance checkingbetween IUTs and specifications to guarantee more reliability.In this work, we have developed an automatic tool for checking conformance on asynchronousreactive systems. We have implemented not only the more general relation based on regular lan-guages but also the classical ioco theory. We could observe that the Everest conformance verifi-cation process based on regular languages can find faults that JTorx could not detect using theclassical ioco relation. In the practical applications, we could see that JTorx yielded a verdict ofconformance whereas Everest could find a fault for the same scenario.In spite of the studies over these distinct approaches of conformance theory, we remark thatthe main contribution of this work is the tool development together with the designed algorithms,providing an intuitive graphical interface either for experts in the research area or for beginnerswith no specific knowledge over the conformance theories.A new module of Everest tool is already being developed to provide the test suite generationin a black-box setting. We also intend to perform more experiments and comparative studies withsimilar tools from the literature in order to give a more precise analysis related to the conformancechecking issue, specially, with regard the usability and performance of these tools.15 s s s f abx a b, xx bb ax (a) Automaton D = otr ( S ) L u . s s s s s s s s cf abx a b, xx bb ax (b) Fault model automaton. q s s q s s q s s q s s ab a b, xbxb a (c) Test suite automaton. Figure 8: Automatons: ioco conformance verification
References [1]
AUT manual page . Acessado em: 2019-02-28.[2]
JTorX: A Tool for Model-Based Testing . Acessado em: 2018-03-23.[3] Bernhard K. Aichernig, Elisabeth Jbstl, and Stefan Tiran. Model-based mutation testing viasymbolic refinement checking.
Science of Computer Programming , 97:383 – 404, 2015. SpecialIssue: Selected Papers from the 12th International Conference on Quality Software (QSIC2012).[4] Bernhard K. Aichernig and Martin Tappler. Symbolic input-output conformance checking formodel-based mutation testing.
Electronic Notes in Theoretical Computer Science , 320:3 – 19,2016. The 1st workshop on Uses of Symbolic Execution (USE).[5] Bernhard K. Aichernig, Martin Weiglhofer, and Franz Wotawa. Improving fault-based con-formance testing.
Electronic Notes in Theoretical Computer Science , 220(1):63 – 77, 2008.Proceedings of the Fourth Workshop on Model Based Testing (MBT 2008).[6] Saswat Anand, Edmund K. Burke, Tsong Yueh Chen, John Clark, Myra B. Cohen, WolfgangGrieskamp, Mark Harman, Mary Jean Harrold, and Phil McMinn. An orchestrated survey of16 d d b a axb (a) Automaton D. s d s d s d s d s d s d cd cd cd a b abx ab ab xba ab xab (b) Automaton fault model D ∩ otr ( S ). s d q s d q s d q s d q s d q cd q s d q cd q cd q cd q cd q cd q ab a bxb a ab a ba ba babaxb ab (c) Automaton test suite. Figure 9: Automaton: language-based conformance verification.methodologies for automated software test case generation.
Journal of Systems and Software ,86(8):1978 – 2001, 2013.[7] Axel Belinfante. Jtorx: Exploring model-based testing, 9 2014. IPA Dissertation series no.2014-09.[8] Adilson Luiz Bonifacio and Arnaldo Vieira Moura. Complete test suites for input/outputsystems, 2019.[9] cadp. Aut manual page. https://cadp.inria.fr/man/aut.html . Accessed: 2019-02-05.[10] J.R. Calam. Specification-based test generation with tgv.
Software Engineering Notes , 2005.1711] E. G. Cartaxo, F. G. O. Neto, and P. D. L. Machado. Test case generation by means of umlsequence diagrams and labeled transition systems. In , pages 1292–1297, Oct 2007.[12] Adenilso da Silva Sim˜ao and Alexandre Petrenko. Generating complete and finite test suitefor ioco: Is it possible? In
Proceedings Ninth Workshop on Model-Based Testing, MBT 2014,Grenoble, France, 6 April 2014. , pages 56–70, 2014.[13] P. Daca, T. A. Henzinger, W. Krenn, and D. Nickovic. Compositional specifications for iocotesting. In , pages 373–382, March 2014.[14] N. Goga. Comparing torx, autolink, tgv and uio test algorithms. In Rick Reed and JeanneReed, editors,
SDL 2001: Meeting UML , pages 379–402, Berlin, Heidelberg, 2001. SpringerBerlin Heidelberg.[15] John E. Hopcroft, Rajeev Motwani, and Jeffrey D. Ullman.
Introduction to Automata Theory,Languages, and Computation (3rd Edition) . Addison-Wesley Longman Publishing Co., Inc.,Boston, MA, USA, 2006.[16] Bruno Legeard Mark Utting. practical model-based testing a tools approach . Elsevier, 1ndedition, 2007.[17] Oracle. Java se development kit 8. .Accessed: 2019-01-30.[18] Oracle. Package javax swing. https://docs.oracle.com/javase/7/docs/api/javax/swing/package-summary.html . Accessed: 2019-01-31.[19] Michael Sipser.
Introduction to the Theory of Computation . Course Technology, second edition,2006.[20] G.J. Tretmans.
A Formal Approach to Conformance Testing . PhD thesis, University of Twente,1992.[21] G.J. Tretmans.
Test Generation with Inputs, Outputs and Repetitive Quiescence . Number TR-CTIT-96-26 in CTIT technical report series. Centre for Telematics and Information Technology(CTIT), Netherlands, 1996. CTIT Tecnnical Report Series 96-26.[22] Jan Tretmans. Testing concurrent systems: A formal approach. In Jos C. M. Baeten andSjouke Mauw, editors,
CONCUR’99 Concurrency Theory , pages 46–65, Berlin, Heidelberg,1999. Springer Berlin Heidelberg.[23] Jan Tretmans.
Model Based Testing with Labelled Transition Systems , pages 1–38. SpringerBerlin Heidelberg, Berlin, Heidelberg, 2008.[24] F. Zeng, Z. Chen, Q. Cao, and L. Mao. Research on method of object-oriented test casesgeneration based on uml and lts. In2009 First International Conference on InformationScience and Engineering