Automated Verification and Synthesis of Stochastic Hybrid Systems: A Survey
Abolfazl Lavaei, Sadegh Soudjani, Alessandro Abate, Majid Zamani
aa r X i v : . [ c s . L O ] J a n AUTOMATED VERIFICATION AND SYNTHESIS OF STOCHASTIC HYBRIDSYSTEMS: A SURVEY
ABOLFAZL LAVAEI , SADEGH SOUDJANI , ALESSANDRO ABATE , AND MAJID ZAMANI , Abstract.
Stochastic hybrid systems have received significant attentions as a relevant modelling frameworkdescribing many systems, from engineering to the life sciences: they enable the study of numerous applica-tions, including transportation networks, biological systems and chemical reaction networks, smart energy andpower grids, and beyond. Automated verification and policy synthesis for stochastic hybrid systems can beinherently challenging: this is due to the heterogeneity of their dynamics (presence of continuous and discretecomponents), the presence of uncertainty, and in some applications the large dimension of state and inputsets. Over the past few years, a few hundred articles have investigated these models, and developed diverseand powerful approaches to mitigate difficulties encountered in the analysis and synthesis of such complexstochastic systems. In this survey, we overview the most recent results in the literature and discuss differentapproaches, including (in)finite abstractions , verification and synthesis for temporal logic specifications , sto-chastic similarity relations , (control) barrier certificates , compositional techniques , and a selection of resultson continuous-time stochastic systems ; we finally survey recently developed software tools that implement thediscussed approaches. Throughout the manuscript we discuss a few open topics to be considered as potentialfuture research directions: we hope that this survey will guide younger researchers through a comprehensiveunderstanding of the various challenges, tools, and solutions in this enticing and rich scientific area. Introduction
Motivations.
Stochastic hybrid systems (SHS) [BLE +
06, CL06] concern complex dynamical modelscombining both digital-computation elements and physical components, tightly interacting with each other infeedback interconnections. SHS models thus comprise discrete dynamics modelling computational componentsincluding hardware and software, and continuous dynamics that model the physical system. Due to their broadreal-world applications, over the past few years SHS have gained remarkable attention in the areas of controltheory, formal verification, applied mathematics, and performance evaluation, among others. Automatedverification and policy synthesis to achieve high-level specifications, e.g., those expressed as (linear) temporallogic formulae [Pnu77], can naturally be very challenging for complex SHS. In particular, the ability to handlethe interaction between continuous and discrete dynamics is a prerequisite for providing a rigorous formalframework for automated verification and synthesis of SHS.The complexity of SHS models, resulting from the aforementioned interaction of discrete and continuouscomponents and from the presence of uncertainty modelled via probabilistic terms, makes it difficult to obtainanalytical results. Hence verification and policy synthesis for SHS are generally addressed by approximationtechniques. These can be categorized as (i) discretization-based and (ii) discretization-free techniques.1.2.
Discretization-based Techniques.
In the analysis of SHS, it is often the case that quantities of in-terest, such as value functions, or the characterization of optimal policies, are in general not available in aclosed (explicit, analytical) form. Therefore, a suitable approach for analyzing SHS is to approximate given(“concrete”) SHS models by simpler ones endowed with finite state spaces, also known as “finite abstractions”.Finite abstractions of SHS are often in the form of Markov decision processes (MDP), where each discrete statecorresponds to a set of continuous states of the concrete SHS (similarly for actions). In practice, such finiteabstractions can be generated by partitioning state/action sets of the concrete models. The discrete dynamicsof the finite abstractions are similarly obtained from those of the concrete continuous models (cf. Figure 1). , SADEGH SOUDJANI , ALESSANDRO ABATE , AND MAJID ZAMANI , Since the obtained abstractions are finite, many algorithmic machineries from computer science [BK08] aredirectly applicable to perform analysis, model checking, or to synthesize controllers maximizing reward orenforcing complex logic properties, including those expressed as logical formulae. A crucial step related tothese discretization-based techniques is to provide formal guarantees on the obtained abstractions, so that theverification or synthesis results on abstract models can be carried over to the original SHS. Discretization-basedtechniques using finite abstractions are schematically illustrated in Figure 1.
Discrete ControllerHybrid Controller ν = ν ^ ν ( x; ^ x; ^ ν ) x x + Original SHS : :
05 0 :
03 0 : : : : : : Finite Abstraction x : Hybrid state^ x : Finite state ν : Continuous input^ ν : Finite input ^ x ^ x ^ x q q q Lookup Table ^ ν (^ x ) orFinite Memory Controller: Figure 1.
Illustration of the procedure underlying discretization-based techniques based onfinite abstractions. The discrete controller can be a static lookup table or a dynamic controller(with finite memory).1.3.
Discretization-free Techniques.
The techniques discussed in the setting of finite abstractions rely onthe discretization (that is, partitioning, or gridding) of state and input/action sets; consequently, they cansuffer from an issue known as the curse of dimensionality : the complexity of constructing the abstraction growsexponentially with the state/input dimension of the SHS. This critical challenge motivates the development ofdiscretization-free approaches, such as those based on (control) barrier certificates , on Lyapunov-like techniquesfor invariance analysis, or on notions of probabilistic simulations. These techniques have been introduced overthe last 15 years for verification and controller synthesis of complex SHS. Barrier certificates are Lyapunov-likefunctions defined over the state space of the system and satisfying a set of inequalities on both the functionitself and the one-step transition (or the infinitesimal generator along the flow) of the system. An appropriatelevel set of a barrier certificate can separate an unsafe region from all system trajectories starting from agiven set of initial conditions (cf. Figure 2). Consequently, the existence of such a function provides a formalprobabilistic certificate for system safety.In this survey paper, we discuss recent approaches grounded on both discretization-based and -free techniques.We should mention that the main focus of the survey is on discrete-time, continuous-space stochastic hybridsystems, whereas we dedicate only one section (Sec. 8) to the otherwise interesting framework of continuous-time, continuous-space models, where we overview the corresponding major theoretical results. We should
UTOMATED VERIFICATION AND SYNTHESIS OF STOCHASTIC HYBRID SYSTEMS: A SURVEY 3 X u X X B( x ) ≤ η B( x ) ≥ β X : State space X : Initial set X u : Unsafe setB : Barrier certi fi cate β > η Figure 2.
Discretization-free techniques can study probabilistic safety based on the synthesisof control barrier certificates. The (red) dashed line denotes the level set B( x ) = η .also stress that much of the presented work builds on the extensive theoretical and algorithmic background of finite-space Markov models, which is not overviewed here in view of length limitations: we refer the interestedreader to [BK08, KNP11] for informative overviews.1.4.
Different Types of Closeness Guarantees.
In this survey we discuss four different types of closenessguarantees (or error bounds) between original SHS and their finite abstractions, as introduced next. Theseguarantees allow to compute assertions over abstractions and formally refine them over the concrete models.
Definition 1.1.
Let Σ be a concrete SHS and b Σ be its abstraction. For a given specification, the probabilisticcloseness between Σ and b Σ is defined according to one of the following:(i) the difference between probabilities of satisfaction of specifications over the original system Σ andits corresponding abstraction b Σ (cf. equation (2.3) or (2.4) );(ii) the probability of the difference between the output trajectories of Σ and b Σ being less than a giventhreshold (cf. equation (2.6) );(iii) the expectation (moment) of the difference between output trajectories of original system Σ andthose of its abstraction b Σ (cf. equation (8.1) );(iv) the probability of satisfaction of logic properties over the abstract system b Σ is either lower- orupper-bounds the satisfaction probability over original system Σ (cf. equations (2.7) and (2.8) ). In the following subsections, we formalize the required definitions and present preliminaries and main notationsfrom control theory and computer science, which are widely employed in the paper.1.5.
Notations and Preliminaries.
The sets of non-negative and positive integers are denoted by N := { , , , . . . } and N ≥ := { , , , . . . } , respectively. Moreover, the symbols R , R > , and R ≥ denote, respec-tively, the sets of real, positive and nonnegative real numbers. For any set X we denote by 2 X the powerset of X that is the set of all subsets of X . Given N vectors x i ∈ R n i , n i ∈ N ≥ , and i ∈ { , . . . , N } , weuse x = [ x ; . . . ; x N ] to denote the corresponding column vector of dimension P i n i . We denote by k · k and k · k the infinity and Euclidean norms, respectively. Given any a ∈ R , | a | denotes the absolute valueof a . Symbols I n , n , and n denote the identity matrix in R n × n and the column vector in R n × with allelements equal to zero and one, respectively. Given a matrix P = { p ij } ∈ R n × n , we denote the trace of P by Tr ( P ), where Tr ( P ) = P ni =1 p ii . We denote the disjunction ( ∨ ) and conjunction ( ∧ ) of Boolean functions f : Γ → { , } over a (possibly infinite) index set Γ by ∨ α ∈ Γ f ( α ) and ∧ α ∈ Γ f ( α ), respectively. Given functions f i : X i → Y i , for any i ∈ { , . . . , N } , their Cartesian product Q Ni =1 f i : Q Ni =1 X i → Q Ni =1 Y i is defined as ABOLFAZL LAVAEI , SADEGH SOUDJANI , ALESSANDRO ABATE , AND MAJID ZAMANI , ( Q Ni =1 f i )( x , . . . , x N ) = [ f ( x ); . . . ; f N ( x N )]. Given sets X and Y , a relation R ⊆ X × Y is a subset of theCartesian product X × Y that relates x ∈ X with y ∈ Y if ( x, y ) ∈ R , which is equivalently denoted by x R y . A function γ : R ≥ → R ≥ , is said to be a class K function if it is continuous, strictly increasing, and γ (0) = 0. A class K function γ ( · ) is said to be a class K ∞ if γ ( r ) → ∞ as r → ∞ . A continuous function β : R ≥ × R ≥ → R ≥ is said to belong to class KL if, for each fixed t , the map β ( r, t ) belongs to class K withrespect to r , and for each fixed nonzero r , the map β ( r, t ) is decreasing with respect to t , and β ( r, t ) → t → ∞ .We consider a probability space (Ω , F Ω , P Ω ), where Ω is the sample space, F Ω is a sigma-algebra on Ωcomprising subsets of Ω as events, and P Ω is a probability measure that assigns probabilities to events. Weassume that random variables introduced in the article are measurable functions of the form X : (Ω , F Ω ) → ( S X , F X ). Any random variable X induces a probability measure on its space ( S X , F X ) as P rob { A } = P Ω { X − ( A ) } for any A ∈ F X . We often directly discuss the probability measure on ( S X , F X ) withoutexplicitly mentioning the underlying probability space and the function X itself.A topological space S is called a Borel space if it is homeomorphic to a Borel subset of a Polish space ( i.e., a separable and completely metrizable space). Examples of a Borel space are Euclidean spaces R n , its Borelsubsets endowed with a subspace topology as well as hybrid spaces. Any Borel space S is assumed to beendowed with a Borel sigma-algebra, which is denoted by B ( S ). We say that a map f : S → Y is measurablewhenever it is Borel measurable.1.6. Discrete-Time Stochastic Hybrid Systems.
In this survey, we consider stochastic hybrid systems indiscrete time (dt-SHS) defined formally as follows.
Definition 1.2.
A discrete-time stochastic hybrid system (dt-SHS) is characterized by the tuple
Σ = ( Q , n, X, U, T x , Y, h ) , where (1.1) • Q := { q , . . . , q p } for some p ∈ N ≥ , represents the discrete-state space; • n : Q → N ≥ assigns to each discrete state value q ∈ Q the dimension of the continuous-state space R n ( q ) ; • X ⊆ ∪ q ∈Q { q }× R n ( q ) is a Borel space as the hybrid-state space of the system. We denote by ( X, B ( X )) the measurable space, with B ( X ) being the Borel sigma-algebra over the state space; • U ⊆ R m is a Borel space as the input space of the system; • T x : B ( X ) × X × U → [0 , is a conditional stochastic kernel that assigns to any x ∈ X , and ν ∈ U ,a probability measure T x ( ·| x, ν ) on the measurable space ( X, B ( X )) . This stochastic kernel specifiesprobabilities over executions { x ( k ) , k ∈ N } of the hybrid system, such that for any set A ∈ B ( X ) andany k ∈ N , P ( x ( k + 1) ∈ A (cid:12)(cid:12) x ( k ) , ν ( k )) = Z A T x ( d x ( k + 1) (cid:12)(cid:12) x ( k ) , ν ( k )); • Y ⊆ R q is a Borel space as the output space of the system; • h : X → Y is a measurable function as the output map that takes a state x ∈ X to its output y = h ( x ) . An example of dt-SHS is discussed in the running example and equation (1.4).This definition is general and describes the dynamics of rich and diverse SHS models, in particular coveringmodels with hybrid state spaces that are very expressive and useful to describe numerous applications. Asthis general structure of the state space can be notationally heavy, for the scope of this survey and for thesake of an easier presentation, we will introduce definitions, algorithms, and theorems based on a specificclass of SHS with a single discrete mode ( i.e., X ⊆ R n ), called discrete-time stochastic control systems(dt-SCS) [MT93, HLL96, HSA17]. We emphasize that broadly the notions and approaches underlying theproposed results can be generalized to SHS endowed with hybrid state spaces. A schematic representation ofdt-SCS Σ is shown in Figure 3. UTOMATED VERIFICATION AND SYNTHESIS OF STOCHASTIC HYBRID SYSTEMS: A SURVEY 5 Σ x ( k + 1) ∼ T x ( · j x ( k ) ; ν ( k )) h ( · ) ν ( k ) x (0) x ( k ) y ( k ) Figure 3.
A schematic representation of a dt-SCS Σ.As argued in [Kal97], any dt-SCS endowed with a stochastic transition kernel T x as in Definition 1.2 canbe equivalently represented by a dt-SCS with a terms ( f, ς ), as formalized next. Note that this alternativerepresentation is more common in control theory. It is often easier to show specific results of this paper basedon the alternative representation. Definition 1.3.
A discrete-time stochastic control system (dt-SCS) is represented by the tuple
Σ = (
X, U, ς, f, Y, h ) , where (1.2) • X ⊆ R n is a Borel space as the state space of the system; • U ⊆ R m is a Borel space as the input space of the system; • ς is a sequence of independent and identically distributed (i.i.d.) random variables from a sample space Ω to the measurable space ( V ς , F ς ) ς := { ς ( k ) : (Ω , F Ω ) → ( V ς , F ς ) , k ∈ N } ; • f : X × U × V ς → X is a measurable function characterizing the state evolution of the system; • Y ⊆ R q is a Borel space as the output space of the system; • h : X → Y is a measurable function as the output map.For given initial state x (0) ∈ X and input sequence ν ( · ) : N → U , the evolution of the state of the dt-SCS Σ can be written, with k ∈ N , as Σ : (cid:26) x ( k + 1) = f ( x ( k ) , ν ( k ) , ς ( k )) ,y ( k ) = h ( x ( k )) . (1.3) We denote by U the collection of the input sequences { ν ( k ) : Ω → U, k ∈ N } , in which ν ( k ) is independentof ς ( z ) for any k, z ∈ N and z ≥ k . For any initial state a ∈ X , and input ν ( · ) ∈ U , the random sequences x aν : Ω × N → X , and y aν : Ω × N → Y that satisfy (1.3) are respectively called the solution process and the output trajectory of Σ under an input ν , and an initial state a . System Σ is said to be finite if X and U arefinite sets, and infinite otherwise. Running Example.
To help the reader gain a better understanding of the details in this survey paper,we present a simple yet interesting running example. We apply the results presented in this survey to atemperature regulation problem for a room equipped with a heater. The model of this case study is borrowedfrom [FI04, MGW17], but modified by including an additive noise, which is intended to capture the effectof uncertain weather- or user-dependent factors. The evolution of the temperature T ( · ) over time can bedescribed by the following dt-SCS:Σ : (cid:26) T ( k + 1) = a ( k ) T ( k ) + γT h ν ( k ) + θT e + Rς ( k ) ,y ( k ) = T ( k ) , (1.4)where the signal a ( k ) := (1 − θ − γν ( k )) depends on the input ν ( k ), R = 0 . θ = 0 .
4, and γ = 0 . ABOLFAZL LAVAEI , SADEGH SOUDJANI , ALESSANDRO ABATE , AND MAJID ZAMANI , and the room, and between the heater and the room. The parameter T e = − ◦ C is the outside temperature, T h = 50 ◦ C is the heater temperature, and finally y is the output of the system, which corresponds to thetemperature itself. The model in (1.4) can be alternatively (and equivalently) characterized via the tuplein (1.2): here X, U are subsets of the real numbers, f ( x ( k ) , ν ( k ) , ς ( k )) = a ( k ) T ( k ) + γT h ν ( k ) + θT e + Rς ( k ),and the output map h is identity (accordingly, the output space Y = X ). Note that this system is a veryspecial instance of SHS in (1.1) endowed with a single discrete mode, where the conditional stochastic kernel T x is a normal distribution with mean a ( k ) T ( k ) + γT h ν ( k ) + θT e and covariance RR T . Alternatively, if theinput ν ( k ) is assumed to be a finite-value function of the state (the temperature T ), e.g. a binary functionswitching upon hitting the boundaries of a temperature interval [AKLP10], then the model can be interpretedas a two-mode (autonomous) SHS. (cid:3) Given the dt-SCS model in (1.2), we introduce
Markov policies as follows.
Definition 1.4.
A Markov policy for the dt-SCS Σ in (1.2) is a sequence µ = ( µ , µ , µ , . . . ) of universallymeasurable stochastic kernels µ n [BS96] , each defined on the input space U given X and such that for all x ( n ) ∈ X , µ n ( U ( x ( n )) (cid:12)(cid:12) x ( n )) = 1 . The class of all Markov policies is denoted by M p . In words, Markov policies are history-independent and the control input taken at the current time instance isselected, possibly randomly, with a distribution that depends only on the current state.We now define the notion of incremental input-to-state stability for Σ, as a pivotal assumption that will allowsome of the results, in particular to provide closeness guarantees between output trajectories of Σ and b Σ, asper (iii) and (iv) in Definition 1.1.
Definition 1.5.
A dt-SCS
Σ = (
X, U, ς, f, Y, h ) is called incrementally input-to-state stable ( δ -ISS) if thereexists a function S : X × X → R ≥ such that ∀ x, x ′ ∈ X , ∀ ν, ν ′ ∈ U , the following two inequalities hold: α ( k x − x ′ k ) ≤ S ( x, x ′ ) ≤ α ( k x − x ′ k ) , (1.5) and E h S ( f ( x, ν, ς ) , f ( x ′ , ν ′ , ς )) (cid:12)(cid:12) x, x ′ , ν, ν ′ i − S ( x, x ′ ) ≤ − ¯ κ ( S ( x, x ′ )) + ρ ( k ν − ν ′ k ) , (1.6) for some α, α, ¯ κ ∈ K ∞ ,and ρ ∈ K ∞ ∪ { } . Later we will show how one can use the δ -ISS property to bound the distance between two solution processesstarting from different initial conditions and under different input trajectories.We now define the notion of stochastic simulation functions (SSF) between b Σ and Σ, which allows to providecloseness guarantees between the output trajectories of the two models, as per (ii) in Definition 1.1.
Definition 1.6.
Consider two dt-SCS
Σ = (
X, U, ς, f, Y, h ) and b Σ = ( ˆ X, ˆ U , ς, ˆ f , ˆ Y , ˆ h ) . A function V : X × ˆ X → R ≥ is called a stochastic simulation function (SSF) from b Σ to Σ if • ∃ α ∈ K ∞ such that ∀ x ∈ X, ∀ ˆ x ∈ ˆ X, α ( k h ( x ) − ˆ h (ˆ x ) k ) ≤ V ( x, ˆ x ) , • ∀ x := x ( k ) ∈ X, ˆ x =: ˆ x ( k ) ∈ ˆ X, ˆ ν =: ˆ ν ( k ) ∈ ˆ U , ∃ ν =: ν ( k ) ∈ U such that E h V ( x ( k + 1) , ˆ x ( k + 1)) (cid:12)(cid:12) x, ˆ x, ν, ˆ ν i ≤ − κV ( x, ˆ x ) + ρ ext ( k ˆ ν k ) + ψ, (1.7) for some < κ < , ρ ext ∈ K ∞ ∪ { } , and ψ ∈ R ≥ .We denote b Σ (cid:22) Σ if there exists an SSF V from b Σ to Σ , and call the system b Σ an abstraction of the concrete(original) system Σ . Note that b Σ may be finite or infinite, depending on the cardinality of the sets ˆ X and ˆ U . UTOMATED VERIFICATION AND SYNTHESIS OF STOCHASTIC HYBRID SYSTEMS: A SURVEY 7
Informally, stochastic simulation functions are Lyapunov-like functions defined over the Cartesian product ofthe state spaces of the two models, which relate their output trajectories and indeed guarantee that theirmismatch (namely their difference, or the abstraction error) remains within some guaranteed error bounds.
Remark 1.7.
The second condition in Definition 1.6 implies the existence of a function ν = ν ˆ ν ( x, ˆ x, ˆ ν ) for thesatisfaction of (1.7) . This function is called the “interface function” and will be used to refine a synthesizedpolicy ˆ ν for b Σ to a policy ν for Σ (cf. Figure 1), and will be discussed later in Sections 3 and 4. Temporal Logic Specifications.
Formal requirements provide a rigorous and unambiguous formalismto express formal requirements over models. A common way to describe such formal requirements is utiliz-ing automata-based or temporal logic-based specifications, e.g., formulae expressed in linear temporal logic(LTL) [Pnu77]. For instance, consider the dt-SCS in (1.2) and a measurable target set B ⊂ Y . We say thatan output trajectory { y ( k ) } k ≥ reaches a target set B within the discrete time interval [0 , T d ] ⊂ N , if thereexists a k ∈ [0 , T d ] such that y ( k ) ∈ B . This bounded reaching of B is denoted by ♦ ≤ T d { y ∈ B } or briefly ♦ ≤ T d B . For T d → ∞ , we denote the reachability property as ♦ B , i.e., eventually B . This is a basic propertythat can be reframed more broadly within LTL. We formally define syntax and semantics of linear temporallogic (LTL) as follows. Definition 1.8.
Consider a set of atomic propositions AP and the alphabet Σ a := 2 AP . Let ω = ω (0) , ω (1) , ω (2) ,. . . be an infinite word, that is, a string composed of letters from Σ a (i.e. ω ( i ) ∈ Σ a , ∀ i ∈ N ). We are interestedin those atomic propositions that are relevant to the dt-SCS via a measurable labeling function L from theoutput space to the alphabet as L : Y → Σ a . Output trajectories { y ( k ) } k ≥ ∈ Y N can be readily mapped to theset of infinite words Σ N a , as ω = L ( { y ( k ) } k ≥ ) := { ω ∈ Σ N a | ω ( k ) = L ( y ( k )) } . We define the LTL syntax [BK08] as ϕ ::= true | p | ¬ ϕ | ϕ ∧ ϕ | (cid:13) ϕ | ϕ U ϕ . Given a trace ω = ω (0) , ω (1) , ω (2) , . . . , let us denote the suffix of ω starting from ω ( i ) by ( ω, i ) = ω ( i ) , ω ( i + 1) , ω ( i + 2) , . . . . We denote by ( ω, i ) | = ϕ when the LTL formula ϕ is true on the suffix ( ω, i ) . This satisfaction is definedinductively as follows: • ( ω, i ) | = true ; • ( ω, i ) | = p , for p ∈ AP iff p ∈ ω ( i ) ; • ( ω, i ) | = ¬ ϕ iff ( ω, i ) = ϕ ; • ( ω, i ) | = ϕ ∧ ϕ iff ( ω, i ) | = ϕ and ( ω, i ) | = ϕ ; • ( ω, i ) | = (cid:13) ϕ iff ( ω, i + 1) | = ϕ ; • ( ω, i ) | = ϕ U ϕ iff for some j such that i ≤ j , we have ( ω, j ) | = ϕ , and for all k s.t. i ≤ k < j , wehave ( ω, k ) | = ϕ .Formula ϕ is true on ω , denoted by ω | = ϕ , if and only if ( ω, | = ϕ . Based on the above operators, we can also introduce others, obtained via propositional or temporal manipu-lations. For instance, ϕ ∨ ϕ , ♦ ϕ , and (cid:3) ϕ have semantics • ( ω, i ) | = ϕ ∨ ϕ iff ( ω, i ) | = ϕ or ( ω, i ) | = ϕ ; • ( ω, i ) | = ♦ ϕ iff for some j such that i ≤ j , we have ( ω, j ) | = ϕ ; • ( ω, i ) | = (cid:3) ϕ iff for all j such that i ≤ j , we have ( ω, j ) | = ϕ .Later, we shall be interested in reasoning about likelihoods to verify given LTL formulae by Markov models,such as MDPs or dt-SCS. Clearly, this requires reasoning about measurability of events associated to the LTLspecifications introduced above; however, we shall not delve into this issue in the present survey. ABOLFAZL LAVAEI , SADEGH SOUDJANI , ALESSANDRO ABATE , AND MAJID ZAMANI , We now define a fragment of LTL properties known as syntactically co-safe linear temporal logic (scLTL)[KV01].
Definition 1.9.
An scLTL over a set of atomic propositions AP is a fragment of LTL such that the negationoperator ( ¬ ) only occurs before atomic propositions, and it is characterized by the following grammar: ϕ ::= true | p | ¬ p | ϕ ∧ ϕ | ϕ ∨ ϕ | (cid:13) ϕ | ϕ U ϕ | ♦ ϕ, with p ∈ AP . scLTL properties are popular since their verification can be performed via a reachability property, which canbe expressed by means of a finite-state automaton [KV01, BYG17]. For this purpose, we introduce a class ofmodels known as deterministic finite-state automata (DFA). Definition 1.10.
A DFA is a tuple A = ( Q ℓ , q , Σ a , F a , t ) , where Q ℓ is a finite set of locations, q ∈ Q ℓ is theinitial location, Σ a is a finite set (a.k.a. alphabet), F a ⊆ Q ℓ is a set of accept locations, and t : Q ℓ × Σ a → Q ℓ is a transition function. Consider a set of atomic propositions AP and the alphabet Σ a := 2 AP . A finite word composed of lettersof the alphabet, i.e., ω f = ( ω f (0) , . . . , ω f ( n )) ∈ Σ n +1 a , is accepted by a DFA A if there exists a finite run q = ( q (0) , . . . , q ( n +1)) ∈ Q n +2 ℓ such that q (0) = q , q ( i +1) = t ( q ( i ) , ω f ( i )) for all 0 ≤ i ≤ n , and q ( n +1) ∈ F a .The accepted language of A , denoted L ( A ), is the set of all words accepted by A . For every scLTL property ϕ , cf. Definition 1.9, there exists a DFA A ϕ such that L f ( ϕ ) = L ( A ϕ ) . where L f denotes the set of all words associated to an scLTL formula ϕ . The following example borrowedfrom [LSZ19b] provides an automaton associated with a reach-avoid specification. Example 1.11.
Consider two measurable sets A , B ⊂ Y as the safe and target sets, respectively. We presentthe DFA for the specification ( A U B ) which requires the output trajectories to reach the target set B whileremaining in the safe set A . Note that we do not assume these two sets being disjoint. Consider the set ofatomic propositions AP = { A , B } and the alphabet Σ a = {∅ , { A } , { B } , { A , B }} . Define the labeling function as L ( y ) = { A } =: a if y ∈ A \ B , { B } =: b if y ∈ B , ∅ =: c if y / ∈ A ∪ B . As can be seen from the above definition of the labeling function L , it induces a partition over the output space Y as L − ( a ) = A \ B , L − ( b ) = B , L − ( c ) = Y \ ( A ∪ B ) . Note that we have indicated the elements of Σ a with lower-case letters for the ease of notation. The specification ( A U B ) can be equivalently written as ( a U b ) with the associated DFA depicted in Figure 4. This DFA has theset of locations Q ℓ = { q , q , q , q } , the initial location q , and accepting location F a = { q } . Thus outputtrajectories of a dt-SCS Σ satisfy the specification ( a U b ) if and only if their associated words are accepted bythis DFA. Generalizing beyond scLTL and DFAs, often we are interested in infinite paths through a system and thus ininfinite words and the ω -automata that accept them. Two of the most commonly used automata to express ω -regular properties are B¨uchi and
Rabin automata. Indeed, non-deterministic
B¨uchi automata can encompassthe entire LTL.
Definition 1.12.
An NBA is a tuple A = ( Q ℓ , q , Σ a , F a , t ) , where Q ℓ is a finite set of locations, q ⊆ Q ℓ isthe initial location, Σ a is the finite alphabet, F a ⊆ Q ℓ is a set of accept locations, and t : Q ℓ × Σ a → Q ℓ is atransition relation. Here ω denotes infinite repetitions, so for instance the string p ω corresponds to { p,p,p,. . . } . UTOMATED VERIFICATION AND SYNTHESIS OF STOCHASTIC HYBRID SYSTEMS: A SURVEY 9 q q q q a ba bcc f a; b; c gf a; b; c g Figure 4.
DFA A ϕ of the reach-avoid specification ( a U b ).A infinite word composed of letters of the alphabet, i.e., ω f = ( ω f (0) , ω f (1) . . . ) ∈ Σ ω a , is accepted by the NBA A if there exists an infinite run q = ( q (0) , q (1) , . . . ) ∈ Q ωℓ such that q (0) ∈ q , q ( i + 1) ∈ t ( q ( i ) , ω f ( i )) for all0 ≤ i , and q ( i ) ∈ F a infinitely often. Let us remark that the non-deterministic feature of NBA is necessaryto express ω -regular properties, and in particular LTL specifications. Alternatively, later in this survey, weshall mention limit-deterministic B¨uchi automata. Similarly, deterministic
Rabin automata can be utilized,which employ a different (and more involuted) acceptance semantics. However, for the sake of space we avoidto detail them and instead refer the reader to [BK08] for a comprehensive discussion.For probabilistic models, properties of interest can be expressible in a different logic, which encompasses aprobabilistic operator in its syntax. Probabilistic computation tree logic (PCTL) [CG04] can be introducedas follows.
Definition 1.13.
The syntax of (PCTL) formulae is defined recursively using the following operators: ϕ ::= true | p | ϕ ∧ ϕ | ¬ ϕ | P ∼ p [ ψ ] ,ψ ::= (cid:13) ϕ | ϕ U ϕ, where p ∈ AP and ∼∈ { <, ≤ , ≥ , > } , and p ∈ [0 , . The semantics for (cid:13) and for U are as before, and thesemantics for true , p , ∧ , and ¬ are also identical except for being defined using the state s = ω (0) instead ofthe whole path ω . The remaining semantics for the probabilistic operator | = P ∼ p [ ψ ] are defined as: Pr( { ω ∈ Y ω | ω (0) = s and ω, | = ψ } ) ∼ p, where Pr is the probability distribution over the infinite paths through Y induced by the stochastic dynamics. Discussion of characterization and computation of basic PCTL specifications, such as safety and reachability,for dt-SCS is introduced in [AAP +
06] and generalized in [APLS08], is applied to invariance in [PP06], and thenextended to reach-avoid [SL10]. [RCSL10] connects the characterization of PCTL to dynamic programming.Further, [TMKA13, TMKA17a] generalizes this to regular and ω -regular properties, respectively, leveragingthe DFA an NBA automata from above, and a product construction further discussed in Section 6. Finally,let us mention that other temporal logics, such as signal temporal logic (STL) [MNP08], are also meaningfullyused over stochastic models - however, we do not delve into further details for the sake of space, and in viewof the relevance and prevalence of LTL and of PCTL.1.8. Contributions and Organization.
This paper provides the first survey of literature on automatedformal verification and synthesis of stochastic hybrid systems (SHS). While trying to be comprehensive, wefocus on the most recent and sharpest results in the literature, and discuss related approaches in varioussections in coarser detail. Besides the selection of the most relevant articles, this survey is intended to helpresearchers to gain an overall understanding of the many challenges and solution strategies related to the formalverification and the control synthesis of SHS, as well as the associated software tools that have been developed , SADEGH SOUDJANI , ALESSANDRO ABATE , AND MAJID ZAMANI , to support the theory. For the sake of a clear and streamlined presentation, we present selected methods,applications, and results in detail, and instead briefly overview alternative approaches, for the following topics.We discuss approaches in relevant literatures via both discretization-based and -free techniques, categorizingthem over four different closeness guarantees between the concrete SHS and their abstractions, according toDefinition 1.1. We employ a running example and discuss approaches under the lens of (ii) time complexity ,and (iii) memory requirements . We also discuss many open problems throughout this survey paper.We remark that although the survey paper in [TSS14] also covers stochastic hybrid systems, its main focus ison stability analysis: different notions of stability are overviewed, including Lyapunov, Lagrange, asymptoticstability, and recurrence analysis. In contrast, here we focus on formal verification and synthesis goals,defined around complex properties including those expressed as temporal logic formulae (simple instancesare safety and reachability specifications), as well as more general properties expressed via omega-regularlanguages [BK08]. In addition, we zoom in on algorithmic solutions for verification and synthesis of SHSagainst temporal properties.This survey paper is structured as follows. In Section 2, we present one of the pivotal theorems of the article,elaborating on four different types of closeness guarantees between the a given SHS and its abstraction. Wediscuss in depth the required assumptions, and present tools to compute such guarantees. Correspondingresults on stochastic similarity relations, to connect the probabilistic behavior of concrete models to that oftheir abstractions are also presented in the same section. Work on the construction of infinite abstractionsfor SHS is discussed in Section 3, and corresponding results on the construction of finite abstractions arestudied in Section 4. We also discuss existing abstraction algorithms, together with the assumptions anddetails underpinning them.In Section 5, we first formally present the definition of control barrier certificates, as discretization-free ap-proaches for the analysis and synthesis of SHS. We then present another main theorem of this survey, whichallows to quantify an upper bound on the probability that the given system reaches an unsafe region over bothfinite and infinite time horizons. Temporal logic verification and synthesis of SHS are studied in Section 6.Section 7 is devoted to compositional techniques as a potential direction for mitigating the curse of dimension-ality. We present the definition of subsystems together with the formal definition of interconnected systems.We then propose the main compositionality results, based on two different techniques from the literature.The results for continuous-time SHS are presented in Section 8. Section 9 is dedicated to simulation-basedanalysis of SHS. Software tools on verification and synthesis of SHS are discussed in Section 10. In Section 11,we summarize the existing analysis methods and highlight some important directions for future research. Inparticular, we discuss a few open problems including “formal analysis of SHS via learning and data-drivenapproaches”, “formal synthesis of partially-observed SHS”, “secured-by-construction controller synthesis forSHS”, “(mixed)-monotonicity of SHS”, “compositional construction of interval Markov processes”, “composi-tional controller synthesis for SHS”, and “potential extensions of software tools”.2.
Stochastic Similarity Relations for Abstractions
Closed-form solutions of SHS or optimal policies for SHS are in general not available explicitly, and thusrequired being numerically computed. A suitable approach is to approximate a given SHS model by simplerabstract ones, with possibly lower dimensional or finite state spaces. In order to render this approximation“formal,” it is desirable to provide guarantees on this approximation step, so that the analysis and/or synthesison the abstract models can be translated back to the original system. Stochastic similarity relations are indeedemployed to relate the probabilistic behavior of a concrete model to that of its abstractions. They can bepresented in the context of stochastic simulation and bisimulation relations, in exact and approximate form.In the following, we present four theorems that summarize several results in relevant literature, to providefour different types of closeness guarantees between original SHS and that of their abstractions. First, wepresent the difference between probabilities of satisfaction of logic properties over original systems Σ and their
UTOMATED VERIFICATION AND SYNTHESIS OF STOCHASTIC HYBRID SYSTEMS: A SURVEY 11 corresponding finite abstractions b Σ [Sou14, TMKA13], as introduced in Sec. 1.2. This type of probabilisticcloseness requires a Lipschitz continuity assumption on the stochastic kernel of the dt-SCS, as in the following.
Assumption 1.
The dt-SCS in Definition 1.3 is Lipschitz continuous if the stochastic kernel T x admits adensity function t s (¯ x | x, ν ) satisfying the following inequality for a some constant ¯ H ≥ : | t s (¯ x | x, ν ) − t s (¯ x | x ′ , ν ′ ) | ≤ ¯ H ( k x − x ′ k + k ν − ν ′ k ) , (2.1) for all x, x ′ , ¯ x ∈ X and all ν, ν ′ ∈ U . If the policy for dt-SCS is given as ˆ ν : X → U , we define the Lipschitzconstant of the stochastic kernel by H , where | t s (¯ x | x, ˆ ν ( x )) − t s (¯ x | x ′ , ˆ ν ( x ′ )) | ≤ H k x − x ′ k , (2.2) for all x, x ′ , ¯ x ∈ X . Now, we have all the ingredients to introduce the first theorem, which is related to (i) in Definition 1.1. Notethat a finite abstraction b Σ is obtained from the original system Σ by first constructing finite partitions of stateand input sets, and then selecting arbitrary “representative points” as abstract states and inputs. Transitionprobabilities in the finite abstraction b Σ are computed accordingly (cf. Section 4, Algorithm 1).
Theorem 2.1.
Let
Σ = (
X, U, ς, f, Y, h ) be a continuous-space dt-SCS and b Σ = ( ˆ X, ˆ U , ς, ˆ f , ˆ Y , ˆ h ) be its finiteabstraction. Assume that the original system Σ is Lipschitz continuous, as per Assumption 1. For a givenlogic specification ϕ , and for any policy ˆ ν ( · ) ∈ ˆ U that preserves the Markov property for the closed-loop b Σ (i.e.,system b Σ fed by input ˆ ν ( · ) , which is denoted by b Σ ˆ ν ), the probabilistic closeness between two systems can bedefined as: | P (Σ ˆ ν (cid:15) ϕ ) − P ( b Σ ˆ ν (cid:15) ϕ ) | ≤ λ , (2.3) with λ := T d δ H L b , where T d is the finite-time horizon, δ is the state discretization parameter, H is theLipschitz constant of the stochastic kernel T x under policy ˆ ν as in (2.2) , and L b is the Lebesgue measure of thestate space. Moreover, the difference between the optimal probabilities of satisfying the given LTL specificationby the two models is bounded by (cid:12)(cid:12) sup ν P (Σ ν (cid:15) ϕ ) − sup ˆ ν P ( b Σ ˆ ν (cid:15) ϕ ) (cid:12)(cid:12) ≤ ¯ λ , (2.4) with ¯ λ := T d δ ¯ H L b , where ¯ H is the Lipschitz constant of the stochastic kernel T x over the state x and input ν as in (2.1) . Furthermore, for the optimal policy ˆ ν ∗ that maximizes the satisfaction probability of ϕ for theabstraction b Σ , we have (cid:12)(cid:12) P (Σ ˆ ν ∗ (cid:15) ϕ ) − P ( b Σ ˆ ν ∗ (cid:15) ϕ ) (cid:12)(cid:12) ≤ λ . (2.5) Remark 2.2.
Note that the Lebesgue measure L b (informally, the “volume”) of the state set appears in theerrors, as per (2.3) - (2.5) , which makes them meaningful over bounded domain and, possibly, quite conservative.There exist techniques based on an adaptive and sequential gridding scheme (e.g., [SA13] ) that mitigate bothshortcomings. In the next theorem, we present a condition such that the probabilistic distance between output trajectories ofΣ and b Σ being less than a given threshold, which is related to (ii) in Definition 1.1, as proposed in [LSMZ17].Note that it works with a finite abstraction with a finite state set or an infinite abstraction with a lowerdimension, which can be constructed via a linear transformation of the state space, obtained with a rectangularmatrix (cf. Theorem 3.2).
Theorem 2.3.
Let
Σ = (
X, U, ς, f, Y, h ) be a continuous-space dt-SCS and b Σ = ( ˆ X, ˆ U , ς, ˆ f , ˆ Y , ˆ h ) be itsabstraction, either with a lower dimension or a finite state set. Suppose there exists a SSF V : X × ˆ X → R ≥
02 ABOLFAZL LAVAEI , SADEGH SOUDJANI , ALESSANDRO ABATE , AND MAJID ZAMANI , from b Σ to Σ as in Definition 1.6. For any input trajectory ˆ ν ( · ) ∈ ˆ U that preserves Markov property for theclosed-loop b Σ , and for any random variables a and ˆ a as the initial states of dt-SCS Σ and b Σ , respectively, onecan construct an input trajectory ν ( · ) ∈ U for Σ through an interface function associated with V (cf. Def. 1.6)such that: P n sup ≤ k ≤ T k y aν ( k ) − ˆ y ˆ a ˆ ν ( k ) k ≥ ε | [ a ; ˆ a ] o ≤ λ , (2.6) where, λ := ( − (1 − V ( a, ˆ a ) α ( ε ) )(1 − ˆ ψα ( ε ) ) T , if α ( ε ) ≥ ˆ ψκ , ( V ( a, ˆ a ) α ( ε ) )(1 − κ ) T + ( ˆ ψκα ( ε ) )(1 − (1 − κ ) T ) , if α ( ε ) < ˆ ψκ , with ˆ ψ ≥ ρ ext ( k ˆ ν k ∞ ) + ψ , where α ∈ K ∞ , < κ < , ρ ext ∈ K ∞ ∪ { } , and ε, ψ ∈ R ≥ as introduced inDef. 1.6. In order to provide the presented closeness guarantee between output trajectories of Σ and b Σ (as per (2.6)),the work in ( e.g., [LSZ19b, ZRME17]) has introduced sufficient conditions, namely requiring that the originalsystem Σ should be incrementally input-to-state stable ( δ -ISS), as per Definition 1.5. In contrast, notice thatthe closeness guarantee in (2.3) is more general, since it does not require original systems to be δ -ISS: instead,only the Lipschitz continuity of the associated stochastic kernels is required for such guarantee [SAM15].On the other hand, the abstraction error presented in (2.3) depends on the Lipschitz constants of the stochastickernel, and the error grows to infinity when the standard deviation of the noise goes to zero, which is not thecase for (2.6). Thus, whilst different in nature, the bound in (2.6) can practically outperform that in (2.3) fornoises with a small standard deviation, as long as the δ -ISS assumption is satisfied by the original model. Inaddition, recent works [HSA17, HS18, LSZ20b] have proposed a closeness guarantee as a version of (2.3) byestablishing an approximate probabilistic relation between Σ and b Σ based on a new notion of so-called δ -lifting.The proposed framework is based on constructing an ε -expansion or ε -contraction of the set of interest (cf. ε in (2.6)) over the abstract system. Accordingly, the probability of satisfaction computed over the modifiedsets on the abstract system provides upper and lower bounds for the probability of satisfaction on the originalmodel.We now present a result, related to (iv) in Definition 1.1, where the probability of satisfaction of a temporallogic property over the abstract system b Σ is either a lower or upper bound for the probability of propertysatisfaction over the original system Σ.
Theorem 2.4.
Let
Σ = (
X, U, ς, f, Y, h ) be a continuous-space dt-SCS and b Σ = ( ˆ X, ˆ U , ς, ˆ f , ˆ Y , ˆ h ) be its finiteabstraction. For a given LTL specification ϕ , and for any policy ˆ ν ( · ) ∈ ˆ U that preserves the Markov propertyfor the closed-loop b Σ , one can construct a policy of ν ( · ) ∈ U for Σ such that: P ( b Σ ˆ ν (cid:15) ϕ ) ≤ P (Σ ν (cid:15) ϕ ) . (2.7) Remark 2.5. As (2.7) provides a lower bound for the probability of satisfaction over Σ , it is mainly usefulwhen one is interested in maximizing the satisfaction probability. Conversely, if the goal is to minimize theprobability of satisfaction, one would want to search for an upper bound of the satisfaction probability. Suchan upper bound can be quantified from (2.7) using the negation of the specification (i.e., ¬ ϕ ) as the following: P (Σ ν (cid:15) ϕ ) ≤ − P ( b Σ ˆ ν (cid:15) ¬ ϕ ) . (2.8)One can employ the same approach as in [LSZ19b, Section 6] and transfer the proposed closeness boundof (2.6) to (2.3) for any class of specifications that can be expressed as accepting languages of deterministicfinite automata (DFA) [KV01]. In particular, any LTL property ϕ over the concrete system can be seen as the UTOMATED VERIFICATION AND SYNTHESIS OF STOCHASTIC HYBRID SYSTEMS: A SURVEY 13 union of events over the product output space. For instance, for a given safe set S , the safety property over afinite-time horizon T is a subset of Y T +1 (with Y T +1 = Q Ti =0 Y ) indicated by the set S T +1 . For all measurableevents A ⊂ Y T +1 , one can construct an ǫ -expansion and ǫ -contraction of A over the abstract model within agiven finite-time horizon T , as A ǫ := {{ y ( k ) } T ∈ Y T +1 (cid:12)(cid:12) ∃{ ¯ y ( k ) } T ∈ A , s.t. max k ≤ T k ¯ y ( k ) − y ( k ) k ≤ ǫ } , A − ǫ := {{ y ( k ) } T ∈ Y T +1 (cid:12)(cid:12) ∀{ ¯ y ( k ) } T ∈ Y T +1 \ A , max k ≤ T k ¯ y ( k ) − y ( k ) k > ǫ } , where { y ( k ) } T = [ y (0); . . . ; y ( T )], whose probabilities of satisfaction give respectively upper and lower boundsfor the probability of satisfaction in the concrete domain with some quantified error bounds in the form of (2.3).There has been substantial work in the area of Formal Methods on different types of stochastic similarityrelations, which are employed to relate the probabilistic behavior of a concrete model to that of its abstrac-tion and have been more recently studied for continuous-space models [Pan09, Aba13]. Early on, similarityrelations over finite-state stochastic systems via exact notions of probabilistic bisimulation relations have beenintroduced in [LS91]. Leveraging probabilistic transition systems as the underlying semantic model, the articleshows how a testing algorithm can distinguish, with a probability arbitrarily close to one, between processesthat are not bisimilar. Similarity relations over finite-state probabilistic models via exact probabilistic simu-lation relations are also presented in [SL95]. In general, similarities are based on simulation or bisimulationrelations, and can be either exact or approximate. Whenever the relation between a concrete model and itsabstraction is symmetric, it is called “bisimulation relation.” Exact simulation relations require the outputsof related systems to be exactly the same, while approximate simulation relations relax this requirement byallowing the outputs to differ up to a given error term [BK08, Tab09, BYG17].Admittedly, exact bisimulation relations raise very strong requirements amongst models, and in practice verylimited classes of models can admit abstractions with those types of relations [DLT08, DAK12]. This is partic-ularly true for continuous-space models [Aba13]. Similarity relations of probabilistic models via approximateversions of probabilistic (bi)simulation relations are provided in [DLT08]. The proposed framework is based ontwo-player games: the existence of a winning strategy for one of the players induces the ǫ -(bi)simulation, andfurthermore letting ǫ = 0 gives back the exact notion. The paper also proposes a polynomial time algorithmto compute a derived metric, where the distance between states s and t is defined as the smallest ǫ such that s and t are ǫ -equivalent.An approximate probabilistic bisimulation relation for discrete-time Markov chains is proposed in [DAK12].The provided scheme exploits the structure and properties of the approximate probabilistic bisimulation andleverages the mathematical framework of Markov set-chains [Har06] (related to Interval MC in Section 4.3) inorder to provide a quantified upper bound on a metric over probabilistic realizations for labeled Markov chains.It is shown that the existence of an approximate probabilistic bisimulation relation implies the preservationof robust PCTL formulae.Similarity relations for models with general, uncountable state spaces have also been proposed in the morerecent literature [Pan09, Aba13]. These relations can depend on stability requirements, on model’s dynamicsvia martingale theory [HH14a], or on contractivity analysis [ZMEM + + δ -ISS), and for every given precision ε >
0, a finite-state transition system can be constructed that is ε -approximately bisimilar to the original stochastic control system. It also provides a closeness bound betweenthe δ -ISS stochastic control system and its bisimilar finite abstraction, related to the one in (8.1).Similarity relations of dt-SCS via approximate (bi)simulation relations are proposed in [DGJP04], in which therelations enforce structural abstractions of a model by exploiting continuity conditions on its probability laws.Approximation metrics of stochastic processes, in particular Markovian processes in discrete time evolving ongeneral state spaces (which are again domains with infinite cardinality and endowed with proper measurabilityand metric structures), are based in [Aba13] on the notion of probabilistic bisimulation. , SADEGH SOUDJANI , ALESSANDRO ABATE , AND MAJID ZAMANI , Σ c Σ & ^ & ν ^ ν ( x; ^ x; ^ ν ) ^ ν y ^ yx ^ x LR Figure 5.
Notion of lifting for specifying the similarity between a dt-SCS Σ and its abstrac-tion b Σ.Labelled Markov processes (LMP) as probabilistic versions of labelled transition systems with continuous-statespaces are widely discussed in [Pan09] and related to dt-SCS [AKNP14]. This book covers basic probabilityand measure theory on continuous-state spaces and then develops the theory of LMPs. The main topicscovered are bisimulation, the logical characterization of bisimulation, metrics and approximation theory.Probabilistic model checking of dt-SCS via finite approximate bisimulations is proposed in [AKNP14]. Thepaper considers notions of (exact and approximate) probabilistic bisimulation and proposes a technique tocompute an approximate probabilistic bisimulation of a dt-SCS, where the resulting abstraction is characterizedas a finite-state Markov chain.A notion of approximate similarity relation based on “lifted” probability measures is presented in [HSA17],which is inspired by notions of similarity relations proposed in [SL95] for finite-state systems. The providedrelation, underpinned by the use of metrics, allows in particular for a useful trade-off between deviations overprobability distributions on states, and metric-based distances between model outputs. This new relationis inspired by a notion of simulation developed for finite-state models, and can be effectively employed overdt-SCS for both verification and synthesis purposes. The work also quantifies the distance in probabilitybetween the original system and its abstraction as a version of the closeness guarantee proposed in (2.3). Thenotion of lifting for specifying the similarity between a dt-SCS Σ and its abstraction b Σ is schematically shownin Figure 5. The relation R connects states of the two dt-SCS, and L specifies the relation between the twonoises. The interface function ν ˆ ν ( x, ˆ x, ˆ ν ) is used for refining a policy from the abstract system to the concreteone.These notions and results are then generalized in [HSA18] to a larger class of temporal properties ((bounded)probabilistic reachability problems and co-safe LTL specifications) and in [HS18] to synthesize policies fora robust satisfaction of these properties, with applications in building automation systems [HCA17]. Anextension of these results to networks of dt-SCS is presented in [LSZ20b], and will be discussed in more detailin Section 7.An approach for computing probabilistic bisimilarity distances for finite-state probabilistic automata has beenproposed in [BBL + ω -regular properties ( i.e., (namely, the notionis specification-independent). As expected, since the proposed results should hold for any arbitrary ω -regularspecification, it can be much more conservative and difficult to be fulfilled or checked, compared to establishingthe previously mentioned guarantees for a given specification.A notion of approximate probabilistic trace equivalences for both finite-state Markov processes and dt-SCS,and its relation to approximate probabilistic bisimulation, is presented in [BA17]. The proposed frameworkinduces a tight upper bound on the approximation between finite-horizon traces, as expressed by a totalvariation distance. This bound can be employed to relate the closeness in satisfaction probabilities overbounded linear-time properties, much as in Equation (2.3), and allows for probabilistic model checking ofconcrete models via their abstractions. UTOMATED VERIFICATION AND SYNTHESIS OF STOCHASTIC HYBRID SYSTEMS: A SURVEY 15
We conclude by raising the following open challenge.
Open Problem 1.
Let Σ and Σ be two dt-SCSs. Develop an approach for computing the probabilisticbisimilarity distance between Σ and Σ , satisfying any ω -regular specification ϕ as follows: | P (Σ (cid:15) ϕ ) − P (Σ (cid:15) ϕ ) | ≤ λ , ∀ ϕ. Infinite Abstractions
The computational complexity associated to verifying or to synthesizing controllers for dt-SCS (and thusfor SHS) models can be alleviated leveraging abstractions in two consecutive stages. In the first phase, theoriginal complex systems can be abstracted by models either with simpler dynamics (e.g. linear, noiseless, oretc) or lower-dimensional state spaces (this is also known in the control literature as “model-order reduction”).Then one can employ those simpler models (a.k.a. infinite abstractions) as a replacement of original systems,perform analysis and synthesis over those models, and finally refine the results back (via an interface map)over the original models. Since the mismatch between outputs of original systems and those of their infiniteabstractions are formally quantified, one can guarantee that concrete systems also satisfy the specificationsas abstract ones with some guaranteed error bounds. An example of infinite abstractions is schematicallydepicted in Figure 6. In comparison with Figure 1, which focuses on discretization-based techniques to obtainfinite abstractions, Figure 6 focuses on infinite abstractions with lower-dimensional systems.
Remark 3.1.
Infinite abstractions can take numerous shapes and forms: they can for instance be linearizedversions of the original models, they can be obtained via polynomial truncation, they can be models with differentnoises (e.g., stochastic realizations [vS89] ), or even models without noise [ZMEM + . The main focus of thissection is on infinite abstractions with lower-dimensional state spaces. Note that one can construct finite abstractions directly, namely without going through infinite abstractionsfirst (e.g., those obtained via model-order reduction). However, constructing finite abstractions for high-dimensional systems can result in large finite state spaces, which might not be practically viable with limitedcomputational and memory resources. One of the main benefits of infinite abstractions is thus to help reducingdimensions or complexity of concrete systems, and then leveraging finite abstractions for the reduced-orderversions, while still providing the probabilistic closeness guarantees.Developed earlier for continuous-time models [JP09, Aba09, Aba10] and further discussed in Section 8, theconstruction of infinite abstractions for discrete-time stochastic control systems is proposed in [LSMZ17]and [LSZ20d]. The abstraction framework is based on notions of stochastic simulation functions (Def. 1.6).These functions relate output trajectories of an abstract system to those of the original one, such that the mis-match between the output trajectories of two systems remains within some guaranteed error bound. Throughthese stochastic simulation functions it is possible to quantify the probabilistic distance between the originalstochastic system and its abstraction, based on the closeness in (2.6). The aforementioned work also focuseson a class of discrete-time linear stochastic control systems, as in (3.1) and further detailed next, and pro-poses a computational scheme to construct infinite abstractions together with their corresponding stochasticsimulation functions.Consider the class of discrete-time linear stochastic control system (a special instance of dt-SCS), asΣ : (cid:26) x ( k + 1) = Ax ( k ) + Bν ( k ) + Rς ( k ) ,y ( k ) = Cx ( k ) , (3.1)where the additive noise ς ( k ) is a sequence of independent random vectors with multivariate standard normaldistributions. We use the tuple Σ = ( A, B, C, R ) to refer to the class of linear systems in (3.1). In thenext theorem, we establish a formal relation between Σ and its reduced-order model b Σ, by constructingcorresponding matrices ˆ A, ˆ B, ˆ C, ˆ R . , SADEGH SOUDJANI , ALESSANDRO ABATE , AND MAJID ZAMANI , Figure 6.
Infinite abstractions.
The original dt-SCS has a 3-dimensional state set whileits abstraction has a 2-dimensional state set. This model reduction can be performed via atransformation matrix P satisfying conditions (3.2) and (3.3). Theorem 3.2.
Let
Σ = (
A, B, C, R ) , b Σ = ( ˆ A, ˆ B, ˆ C, ˆ R ) be two linear dt-SCS with independent additive noises.Suppose there exist a matrix K and a positive-definite matrix M such that the following matrix inequalities C T C (cid:22) M, (cid:0) (1 + π )( A + BK ) T M ( A + BK ) − M (cid:1) (cid:22) − ˆ κM, hold for some constants < π and < ˆ κ < . If further AP = P ˆ A − BQ, (3.2) CP = ˆ C, (3.3) hold for some matrices Q and P of appropriate dimension, then there exists a quadratic SSF V ( x, ˆ x ) [LSMZ17] between Σ and b Σ as V ( x, ˆ x ) = ( x − P ˆ x ) T M ( x − P ˆ x ) , (3.4) where P ∈ R n × ˆ n is a matrix of an appropriate dimension with ˆ n being the dimension of the reduced-ordermodel b Σ . The stochastic simulation function V ( x, ˆ x ) in (3.4) gives a formal probabilistic closeness guarantee betweenthe original dt-SCS Σ and its infinite abstraction b Σ, as per (2.6).
Remark 3.3.
Condition (3.2) holds as long as condition (V.18) in [ZA18] is satisfied. In addition, notice thatthe results in Theorem 3.2 do not impose any condition on the matrix ˆ B , which thus can be chosen arbitrarily.As an example, one can select ˆ B = I ˆ n , which renders the abstract system b Σ fully actuated and, hence, canfacilitate a subsequent synthesis task. UTOMATED VERIFICATION AND SYNTHESIS OF STOCHASTIC HYBRID SYSTEMS: A SURVEY 17
Notice further that the matrix ˆ R can be also chosen arbitrarily. In this case, the probabilistic closeness betweentwo systems Σ and b Σ can be quantified as λ in (2.6) , where ψ = Tr (cid:0) R T M R + ˆ R T P T M P ˆ R (cid:1) . One can readily verify that selecting ˆ R = 0 results in a tighter relationship between the original system Σ andits infinite abstraction b Σ . However, observe that this is not the case when the noises of the concrete systemand of its infinite abstraction are the same, as assumed in [Zam14, ZRME17] (i.e., the abstraction has accessto the noise of the concrete system, which is a strong assumption in practice), where the developed probabilisticcloseness is similar to that in (2.6) . The construction of infinite abstractions for dt-SCS is also discussed in [LSZ19b]. The proposed approachemploys the notion of stochastic storage function (a variant of the stochastic simulation function in Definition1.6) between a concrete system and its abstraction, which allows to provide a closeness guarantee as in (2.6).This work also focuses on a specific class of discrete-time nonlinear stochastic systems, with a nonlinearity Υsatisfying a slope restriction as 0 ≤ Υ( c ) − Υ( d ) c − d ≤ b, (3.5)for any c, d ∈ R , c = d, for some b ∈ R > ∪ {∞} , and proposes a construction scheme for building infiniteabstractions together with their corresponding stochastic storage functions.It is worth mentioning that the contributions in [LSMZ17, LSZ19b] do not raise any restrictions on the sourcesof uncertainty in the concrete and abstract systems ( i.e., the noises of the abstraction can be completelyindependent of that of the concrete system). In particular, the results provided in [LSMZ17, LSZ19b] aremore general than [ZRME17], where the noises in the concrete and abstract systems are assumed to be thesame, which practically means the abstraction has access to the noise of the concrete system. The resultsin [LSMZ17, LSZ19b] provide a closeness guarantee between output trajectories of Σ and b Σ as in (2.6).4.
Finite Abstractions
In the second phase of the abstraction procedure (cf. Figure 1), one can construct finite abstractions usuallyin the form of finite Markov decision processes (MDPs). These abstractions are approximate descriptions of(reduced-order) systems in which each discrete state corresponds to a set of continuous states of the (reduced-order) systems. Since the obtained abstractions are finite, one can employ algorithmic machineries and existingsoftware tools to automatically synthesize controllers, which can then be applied (refined) over the concretemodels, thus enforcing complex properties, including specifications expressed as temporal logical formulae.A concrete model Σ is approximated by a finite b Σ using Algorithm 1. For the sake of an easier presentation,we present the construction algorithm just for dt-SCS, however we refer the interested reader to [ZTA17,TMKA17a, HSA17] for related, more general SHS, and to [ZA14b, ZAG15, LSZ20a] for the construction offinite MDPs for a class of SHS namely stochastic switched systems. To construct such a finite approximation,the state and input sets (over which one is interested to perform analysis and synthesis) of the dt-SCS Σare restricted to be compact. The rest of the state space can be considered as a single absorbing state.Algorithm 1 first constructs a finite partition of the state set X = ∪ i X i and the input set U = ∪ i U i . Thenarbitrary “representative points” ¯ x i ∈ X i and ¯ ν i ∈ U i are selected as abstract states and inputs. Transitionprobabilities in the finite MDP b Σ are computed according to (4.1). The output map ˆ h is the same as h withits domain restricted to the finite set ˆ X (cf. Step 7) and the output set ˆ Y is the image of ˆ X under h (cf. Step6). This compactness assumption can be relaxed, as in [SA14a, SA15]. , SADEGH SOUDJANI , ALESSANDRO ABATE , AND MAJID ZAMANI , Algorithm 1
Approximation of a dt-SCS Σ by a finite MDP b Σ Require:
Input dt-SCS Σ = (
X, U, T x , Y, h ) Select finite partitions of sets
X, U as X = ∪ n ¯ x i =1 X i , U = ∪ n ¯ ν i =1 U i For each X i , and U i , select single representative points ¯ x i ∈ X i , ¯ ν i ∈ U i Define ˆ X := { ¯ x i , i = 1 , ..., n ¯ x } as the finite state set of MDP b Σ with the finite input set ˆ U := { ¯ ν i , i =1 , ..., n ¯ ν } Define the map Ξ : X → X that assigns to any x ∈ X , the corresponding partition set it belongs to, i.e., Ξ( x ) = X i if x ∈ X i for some i ∈ { , , . . . , n x } Compute the discrete transition probability matrix ˆ T x for b Σ as:ˆ T x ( x ′ | x, ν ) = T x (Ξ( x ′ ) | x, ν ) , (4.1)for all x, x ′ ∈ ˆ X, ν ∈ ˆ U Define the output space ˆ Y := h ( ˆ X ) Define the output map ˆ h := h | ˆ X Ensure:
Output finite MDP b Σ = ( ˆ X, ˆ U , ˆ T x , ˆ Y , ˆ h )Given a dt-SCS Σ = ( X, U, ς, f, Y, h ), the finite MDP b Σ constructed in Algorithm 1 can be represented as b Σ = ( ˆ X, ˆ U , ς, ˆ f , ˆ Y , ˆ h ) , (4.2)where ˆ f : ˆ X × ˆ U × V ς → ˆ X is defined as ˆ f (ˆ x, ˆ ν, ς ) = Π x ( f (ˆ x, ˆ ν, ς )) , (4.3)and Π x : X → ˆ X is the map that assigns to any x ∈ X , the representative point ¯ x ∈ ˆ X of the correspondingpartition set containing x . The initial state of b Σ is also selected according to ˆ x := Π x ( x ), with x being theinitial state of Σ.The dynamical representation of the abstract finite MDP b Σ employs the map Π x : X → ˆ X , which satisfies theinequality k Π x ( x ) − x k ≤ δ, ∀ x ∈ X, (4.4)where δ := sup {k x − x ′ k , x, x ′ ∈ X i , i = 1 , , . . . , n x } is the state discretization parameter. Remark 4.1.
Observe that the state discretization parameter δ appears in the probabilistic closeness quantifiedin (2.3) - (2.6) : thus, one can decrease the error by reducing the state discretization parameter, namely byaptly refining the state partitions. Notice that there is no requirement on the shape of the partition elementsin constructing the finite MDPs. For the sake of an easier implementation, one can for instance considerpartition sets as hyper-boxes, and representative points as centers of each box (cf. Figure 7). The errors andguarantees derived above provide flexibility and have been embedded in a few software tools generating finiteabstractions, cf. [SGA15, LKSZ20] . Abstractions for Finite-Horizon Specifications.
The construction of finite abstractions of SHSs hasbeen initially proposed in [AKLP10] and used for formal verification and synthesis. This work investigatesprobabilistic safety and reachability over a finite-time horizon for a general class of discrete-time SHS withcontrol inputs. The proposed framework characterizes the set of initial conditions providing a certain proba-bilistic guarantee that the system will keep evolving within a desired ‘safe’ region of the state space in termsof a value function, and determines ‘maximally safe’ Markov policies via dynamic programming over the finiteabstract MDP. An improved gridding scheme, which is adaptive and sequential, for the abstraction and veri-fication of stochastic processes is proposed in [SA13] (cf. Remark 4.1). The abstract model is constructed asa Markov chain using an adaptive gridding algorithm that conforms to the underlying dynamics of the modeland thus mitigates the curse of dimensionality unavoidably related to the partitioning procedure. The workfocuses on the study of a particular specification (probabilistic safety or invariance, over a finite horizon) and
UTOMATED VERIFICATION AND SYNTHESIS OF STOCHASTIC HYBRID SYSTEMS: A SURVEY 19 . . . . . .. . . . .. . .. . . . . . ^ ν ; : ν ; : ν ; : δ . . . . . .. . . . .. . .. . . . . . δ .. . . Figure 7.
Construction of finite MDPs: A grid is first overlaid on the state and input sets.The center of each cell is considered as a representative point, and the transition probability( i.e., probability of jumping from each representative point in a cell to all other cells) for allpossible discrete inputs is computed. By repeating the process and storing the probabilitiesin a matrix called transition probability matrix , the corresponding finite MDP is accordinglyconstructed.the results are then extended to SHS models with hybrid state spaces. The closeness guarantee between theoriginal SHS and their finite abstractions is in the form of (2.3).The above results in general rely on Lipschitz continuity of the stochastic kernel associated with the system.The works [SA12b, SA14b] extend the method for systems with discontinuous stochastic kernels and provideerror bounds for inequalities of the form (2.3). On the other hand, if the kernel admits higher-order derivatives,refined computations are proposed in [SA12a], with errors that naturally depend on higher orders of thediscretization parameter δ and that can show faster convergence to zero.Among the many applications of these formal abstractions, the approach is employed to aggregate modelingand control of thermostatically controlled loads, as in [SGE + δ -ISS. On the other hand, theabstraction error in [SAM15, SAM17] depends on the Lipschitz constants of the stochastic kernels associatedwith the system, and accordingly, it grows to infinity as the standard deviation of the noise goes to zero, whichis not the case in [LSZ20d, LSZ18b]. Running example (continued).
We construct a finite MDP from the model in (1.4) according to Algo-rithm 1, with the state discretization parameter δ = 0 . b Σ to be equal to 20, and using the proposed bound in (2.6), one can guarantee that the distance betweenthe outputs of Σ and b Σ does not exceed ε = 0 . T d = 100, with a probability of at least98%, i.e., P n k y ( k ) − ˆ y ( k ) k ≤ . , ∀ k ∈ [0 , o ≥ . . (4.5) , SADEGH SOUDJANI , ALESSANDRO ABATE , AND MAJID ZAMANI , Figure 8.
Closed-loop state trajectories with 10 different noise realizations for the finite timehorizon T d = 100.
19 19.5 20 20.5 2100.10.20.30.40.50.6
Figure 9.
Synthesized policy as a function of state (room temperature).Let us now synthesize a controller for Σ via its finite abstraction b Σ, such that for Σ the temperature of the roomremains in the safe region [19 , AMYTISS [LKSZ20]. Thesynthesized policy of the room as a function of state is illustrated in Figure 9. Closed-loop state trajectoriesdescribing the dynamics of the room temperature over the finite-time horizon T d = 100, under 10 differentnoise realizations, are illustrated in Figure 8. We remark that the synthesized concrete policy for this exampleis simply chosen as ν = ˆ ν , which is a special case of the interface function discussed in Remark 1.7.In order to better understand the provided probabilistic bound in (4.5), we also run Monte Carlo simulationof 10000 runs. One expects that the distance between the outputs of Σ and b Σ is always less than or equalto 0 . (cid:3) Abstractions for Infinite-Horizon Specifications.
The construction of finite Markov chains fordiscrete-time stochastic models with continuous state spaces and their use to verify infinite-horizon properties( e.g., safety and reachability specifications) is proposed by a line of work in [TA11, TA12a, TA12b, TA14].The proposed approaches employ notions of stochastic bisimulation functions and provide a lower bound for
UTOMATED VERIFICATION AND SYNTHESIS OF STOCHASTIC HYBRID SYSTEMS: A SURVEY 21 infinite-time probabilistic invariance (cf. equations (2.7) and (2.8)) by decomposing this property into a finite-time reach-avoid together with an infinite-time invariance around absorbing sets (cf. Definition 4.2 below)over the state space of the model.A quantitative abstraction-based controller synthesis for SHS is discussed in [TMKA13] and later extendedin [TMKA17b]. The problem is reformulated as an optimization of a probabilistic reachability property over aproduct process (known as product automaton), which is obtained from the model of the specification and thatof the system. The work develops a discretization procedure, which results in a standard synthesis problemover Markov decision processes with history-independent Markov policies, with errors of the form (2.3) and(2.4)), respectively.The satisfaction probability of infinite-horizon properties is theoretically investigated in [TA14]. Extending tocontrol-dependent models in [TMKA17b], it is shown that the satisfaction probability depends on the existenceof absorbing sets , as defined next.
Definition 4.2.
The set
A ∈ B ( X ) is called (weakly) absorbing if there exists a randomized selector ¯ µ suchthat for all x ∈ A , it holds that ¯ µ ( U (cid:12)(cid:12) x ) = 1 and Z U T x ( A (cid:12)(cid:12) x, ν )¯ µ ( d ν (cid:12)(cid:12) x ) = 1 . We say that the set
A ∈ B ( X ) is simple if it does not have non-empty (weakly) absorbing subsets. Employing Definition 4.2, the following theorem is proposed in [TMKA17b].
Theorem 4.3.
The infinite-horizon safety probability for a continuous-space dt-SCS and a compactsafe set S is equal to zero over the entire set if and only if the safe set S does not contain any absorbingsets. If the underlying system is a finite MDP, the simple absorbing sets in Definition 4.2 are bottom stronglyconnected components (BSCCs) of the MDP, and computing these BSCCs is straightforwardly done by graphsearch algorithms. Conversely, no computational method is proposed in the literature for finding absorbingsets of continuous systems. Motivated by Definition 4.2 and Theorem 4.3, we present the following openproblem, which would allow to expand beyond the results in [TA14, TMKA17b].
Open Problem 2.
Given a dt-SCS with continuous-state space, compute its absorbing sets, or to computeover- and under-approximations of such absorbing sets within a-priori precision.
A method to generate finite Markovian abstractions for discrete-time linear stochastic systems is providedin [LAB12]. The proposed approach proceeds by approximating the transition probabilities from one partitionset to another by calculating the probability from a single representative point in the first region. The workemploys an adaptive refinement algorithm that takes advantage of the dynamics of the system to achieve adesired error value. The proposed approach is similar to that of [SA13] with a closeness guarantee in the formof (2.3), however here the transition probabilities are averaged over partition sets.The construction of finite abstractions for discrete-time stochastic systems is also pursued in [LSZ18b, LSZ18a].Focusing on a specific class of linear dt-SCS, these results employ notions of stochastic simulation (or storage)functions by providing a probabilistic distance between the interconnection of stochastic control subsystemsand that of their finite abstractions based on (2.6).The construction of finite MDPs for stochastic systems that are not necessarily stabilizable is presentedin [LSZ19c, LZ19, LSZ20c]. The proposed frameworks rely on a relation between a system and its finiteabstraction employing a new notion called finite-step stochastic simulation. In comparison with the existingnotions of simulation functions in which stability or stabilizability of each subsystem is required, a finite-step stochastic simulation function needs to decay only after some finite numbers of steps (rather than ateach time step). This results in a less conservative approach in the sense that one can compositionally , SADEGH SOUDJANI , ALESSANDRO ABATE , AND MAJID ZAMANI , construct finite MDPs such that stabilizability of each subsystem is not necessarily required. The workin [LSZ19c, LZ19, LSZ20c] provides a closeness guarantee between output trajectories of Σ and b Σ as per (2.6).The construction of finite abstractions for stochastic switched systems is presented in [LSZ20a, LZ20]. Thetransition map switches between a finite set of modes and the switched system accepts multiple Lyapunov (orstorage) functions with a dwell-time condition that puts a lower bound on the interval between two consecutiveswitching time instants. The dwell-time is deterministic and always met by the controller designed using thefinite MDP. In particular, switching signals in those works are control inputs and the main goal is to synthesizethem with a specific dwell-time, such that the output of original systems satisfies some high-level specifications,such as safety, reachability, etc. Those works utilize notions of stochastic simulation (or storage) functions andprovide a closeness guarantee in the form of (2.6) but adapted to the switched setup. These works also showthat under standard assumptions ensuring incremental input-to-state stability of switched systems similar toDefinition 1.5 ( i.e., existence of common incremental Lyapunov (or storage) functions, or multiple incrementalLyapunov (or storage) functions with some dwell-time conditions), one can construct finite MDPs for nonlinearstochastic switched systems. These results also propose an approach to construct finite MDPs together withtheir corresponding stochastic simulation (or storage) functions for a particular class of nonlinear stochasticswitched systems whose nonlinearity Υ satisfies either a slope restriction similar to (3.5) or an incrementalquadratic inequality as (cid:20) d − d Υ p ( k, d ) − Υ p ( k, d ) (cid:21) T ¯ Q p (cid:20) d − d Υ p ( k, d ) − Υ p ( k, d ) (cid:21) ≥ , (4.6)for all k ∈ N , d , d ∈ R , for all switching modes p ∈ P = { , . . . , m } , and for all ¯ Q p ∈ ¯ Q p , where ¯ Q p is the setof symmetric matrices referred to as “incremental multiplier” matrices. For this class of nonlinear systems,the aforementioned incremental stability property can be readily checked via matrix inequalities.Abstraction-based synthesis of general MDPs using approximate probabilistic relations is proposed in [LSZ19a,LSZ20b]. The abstraction framework is based on the notion of δ -lifted relations, which is similar to [HSA17],using which one can quantify the distance in probability between dt-SCS and that of their abstractions as aversion of closeness guarantee proposed in (2.3). The works focus on a class of stochastic nonlinear dynamicalsystems and construct their (in)finite abstractions using both model order reduction and space discretizationin a unified framework.An alternative approach to handle infinite-horizon specifications is to employ interval MCs or interval MDPs,as discussed in the next subsection.4.3. Abstractions as Interval Markov Models.
The classical finite-state Markov models seen in theprevious sections are not the only possible architecture for abstractions: uncertain Markov models can as wellbe employed for this task, and indeed they have been in particular employed to construct finite abstractionsthat are capable of satisfying infinite-time horizon properties in a more natural manner than standard Markovmodels. Uncertain Markov models have been studied under different but related perspectives and semantics:[Jun20] provides an overview of existing approaches, and in particular focuses on models where the uncertaintyis described parametrically, where probabilities are symbolic expressions rather than concrete values. [Jun20]discusses the parameter synthesis problem for the analysis of this class of Markov models. Alternatively, whenthe probabilities of transition between states belong to intervals, we can use interval Markov chains (IMC)and interval Markov decision processes (IMDP).The definition of IMDP is similar to finite MDP as in Algorithm 1 with a tuple b Σ I = ( ˆ X, ˆ U , ˆ T x , ˆ T x , ˆ Y , ˆ h )where the exact transition probabilities are not known but are bounded above and below as ˆ T x ≤ ˆ T x ≤ ˆ T x . An IMC is schematically depicted in Figure 10.It is worth mentioning that constructing IMCs/IMDPs can be more complicated compared to standardMCs/MDPs since one needs to provide both lower and upper bounds for probabilities of transitions among
UTOMATED VERIFICATION AND SYNTHESIS OF STOCHASTIC HYBRID SYSTEMS: A SURVEY 23 a ≤ p ≤ b a ≤ p ≤ b a ≤ p ≤ b a i ; b i [0 ; i ; ; g ^ x ^ x ^ x p p p Figure 10.
Example of an IMC.partition sets, by solving max-min optimization problems. However, one can mitigate the construction com-plexity if the system has a property called mixed-monotonicity and if the noise of the system has some niceproperties.
Definition 4.4.
A function f : X → X is called mixed monotone if there exists a decomposition function g : X × X → X satisfying [Smi08, CA15] • ∀ x ∈ X : f ( x ) = g ( x, x ) , • ∀ x , x , z ∈ X : x ≤ x implies g ( x , z ) ≤ g ( x , z ) , • ∀ x, z , z ∈ X : z ≤ z implies g ( x, z ) ≤ g ( x, z ) . Mixed monotonicity generalizes the notion of monotonicity in dynamical systems, which is recovered when g ( x, z ) = f ( x ) for all x, z .Consider the discrete-time stochastic system x ( k + 1) = f ( x ( k )) + ς ( k ), where f ( · ) is mixed monotone andentries of the noise ς ( · ) are independent with unimodal distributions. Then, an IMC as abstraction of thissystem can be computed without the need for the optimization required in the computation of ˆ T x , ˆ T x [DC19].Specification-guided verification and abstraction refinement for mixed-monotone stochastic systems againstomega-regular specifications are proposed in [DC19]. The article presents a procedure to compute a finite-state interval-valued Markov chain abstraction of discrete-time mixed-monotone stochastic systems subjectto additive noise, given a rectangular partition of the state space. An algorithm is proposed for performingverification against omega-regular properties in IMCs that aims to compute bounds on the probability ofsatisfying a specification from any initial state of the IMC, in the form of (2.7). This is achieved by solving areachability problem on sets of so-called “winning and losing” components in the Cartesian product betweenthe IMC and a Rabin automaton representing the original specification.The results of [DC19] have been recently extended to the controller synthesis problem for discrete-time,continuous-state stochastic systems, under omega-regular specifications, by means of finite-state abstrac-tions [DHC20]. The work presents a synthesis algorithm for optimizing the probability that a discrete-timestochastic switched system with a finite number of modes satisfies an omega-regular property. The approachrelies on a finite-state abstraction of the underlying dynamics in the form of a bounded-parameter Markovdecision process arising from a finite partition of the model’s domain, with errors in the form of (2.7). SuchMarkovian abstractions allow for a range of probabilities of transitions between states for each selected ac-tion representing a mode of the original system. The proposed framework decomposes the synthesis into aqualitative problem, where the so-called greatest permanent winning or losing components of the productautomaton are created, and a quantitative problem, which requires maximizing the probability of reachingthese components in the worst-case instantiation of the transition intervals. , SADEGH SOUDJANI , ALESSANDRO ABATE , AND MAJID ZAMANI , Remark 4.5.
The results in [DC19] and [DHC20] leverage the mixed-monotonicity property (cf. Defini-tion 4.4) of the deterministic part of the map f by assuming that (i) the stochasticity is additive, (ii) thedistribution of the noise is unimodal, and (iii) noises of different states are independent from each other. Thisobservation leads to the following open problem. Open Problem 3.
Provide a suitable definition of mixed-monotonocity for stochastic systems based ontheir stochastic kernels and investigate which classes of systems satisfy that property. An initial investiga-tion is provided in Subsection 11.4.
An abstraction framework for mapping a discrete-time stochastic system to an IMC and mapping a switched dt-SCS to a bounded-parameter Markov decision process (BMDP) is proposed in [LAB15]. The work constructsmodel checking algorithms for IMCs and BMDPs against PCTL formulae to find sets of initial states thatdefinitely, possibly, and never satisfy a given specification. It also develops an algorithm for BMDPs thatsynthesizes a policy maximizing the probability of satisfaction, and further proposes an adaptive refinementalgorithm that exploits the dynamics of the system and the geometry of the partition to increase the precisionof the solution. The work proposes a closeness guarantee in the form of (2.7).Approximate abstractions of dt-SCS with interval MDPs are proposed in [ZLWDA18]. The abstraction lever-ages the semantics of IMDPs and the standard notion of approximate probabilistic bisimulation. The resultingmodel presents a smaller one-step bisimulation error, in the form of Equation (2.3) or (2.4), when compared toa Markov chain abstraction. The work outlines a method to perform probabilistic model checking, and showsthat the computational complexity of the new method is comparable to that of standard abstractions basedon approximate probabilistic bisimulations.A constructive procedure for obtaining a finite abstraction of a discrete-time SHS is proposed in [ADB11],with errors as in Equation (2.7) but usable over infinite horizons. Similar to the finite abstractions discussedabove, the procedure consists of a partition of the state space of the system which depends on a controllableparameter. Given proper continuity assumptions on the model, the approximation errors introduced by theabstraction procedure are explicitly computed and it is shown that they can be tuned through the parameterof the partition.An efficient abstraction framework for formal analysis and control synthesis of a class of discrete-time SHSwith linear dynamics is developed in [CLL + Open Problem 4.
The discussed results in the setting of IMCs/IMDPs by and large provide a guaranteein the form of (2.7) . In particular, the satisfaction probability computed over the IMDPs gives a lowerbound for the probability of satisfaction over the original system. Quantify instead the distance between theprobability of satisfactions over the two systems in the form of equation (2.3) or (2.4) . Verification and Synthesis via Barrier Certificates
As discussed in the previous sections, discretization-free approaches can prevent the curse of dimensionalityarising in the construction of finite abstractions. In this section we discuss discretization-free approaches basedon (control) barrier certificates that have been proposed in recent years. Work in this domain follows seminalcontributions in [PJP07] and the following [JP09, Aba09, Aba10] (these are further discussed below), which
UTOMATED VERIFICATION AND SYNTHESIS OF STOCHASTIC HYBRID SYSTEMS: A SURVEY 25 however are developed for continuous-time models. We first formally define control barrier certificates in thecontext of this work.
Definition 5.1.
Consider a dt-SCS
Σ = (
X, U, ς, f, Y, h ) with sets X , X u ⊆ X that are respectively initialand unsafe sets of the system. A function B : X → R ≥ is called a control barrier certificate (CBC) for Σ ifthere are constants η, β ∈ R ≥ with β > η such that B ( x ) ≤ η, ∀ x ∈ X , (5.1) B ( x ) ≥ β, ∀ x ∈ X u , (5.2) and ∀ x ∈ X , ∃ ν ∈ U , such that E h B ( x ( k + 1)) (cid:12)(cid:12) x ( k ) , ν ( k ) i ≤ max n κ B ( x ( k )) , c o , (5.3) for constants < κ ≤ and c ∈ R ≥ . Remark 5.2.
Note that the existential quantifier for the condition in (5.3) implies the existence of a feedbackcontroller for a model satisfying the conditions.
Employing Definition 5.1, one can propose an upper bound on the probability that the dt-SCS in (1.3) reachesan unsafe region over a finite time horizon, as presented in the next theorem. Note that the requirement β > η is needed in order to propose meaningful probabilistic bounds. Corollary 5.5 and the subsequent remarkdiscuss the choice of the constant c in the statement above. Theorem 5.3.
Consider a dt-SCS
Σ = (
X, U, ς, f, Y, h ) and a CBC B for Σ . Then the probability that thesolution process of Σ starts from any initial state x (0) ∈ X and reaches X u under the policy ν ( · ) (associatedwith the CBC B ) within the time interval [0 , T d ] is bounded by ¯ δ , namely P n x ( k ) ∈ X u for some k ∈ [0 , T d ] (cid:12)(cid:12) x (0) ∈ X o ≤ ¯ δ, (5.4) where if < κ < : ¯ δ := ( − (1 − ηβ )(1 − cβ ) T d , if β ≥ cκ , ( ηβ )(1 − κ ) T d + ( cκβ )(1 − (1 − κ ) T d ) , if β < cκ , (5.5) whereas if < κ ≤ : ¯ δ := η + cT d β . (5.6) Remark 5.4.
Note that the upper bound proposed in (5.5) is less conservative than that of (5.6) in the sensethat (5.5) yields a tighter probabilistic bound. On the other hand, the proposed bound in (5.6) is more general,since there may not exists a κ strictly less than one satisfying condition (5.3) for many classes of models anddynamics. The results in Theorem 5.3 provide upper bounds on the probability that the models reach unsafe regionswithin a finite time horizon. One can generalize the proposed results to an infinite time horizon, providedthat the constant c = 0, as stated in the following corollary. Corollary 5.5.
Let
Σ = (
X, U, ς, f, Y, h ) be a dt-SCS and suppose B is a CBC for Σ with c = 0 in (5.3) .Then the probability that the trajectory of Σ starts from any initial state x (0) ∈ X and reaches X u under thepolicy ν ( · ) is bounded by P n x ( k ) ∈ X u for some k ≥ (cid:12)(cid:12) x (0) o ≤ ηβ . (5.7) , SADEGH SOUDJANI , ALESSANDRO ABATE , AND MAJID ZAMANI , Remark 5.6.
Note that a CBC B satisfying condition (5.3) with c = 0 is a non-negative supermartin-gale [Kus67, Chapter I] . Although the supermartingale property on B allows one to provide probabilistic guar-antees for infinite-time horizons via Corollary 5.5, it is restrictive in the sense that a supermartingale B maynot exist [ST12, JSZ20] . One may therefore employ a more general c -martingale type condition as in (5.3) thatdoes not require such an assumption at the cost of providing probabilistic guarantees for finite-time horizons. Note that the computation underlying CBC does not generate an abstract model, and accordingly it doesnot rely on any similarity relation and closeness error as presented in Definition 1.1. Instead, one can employDefinition 5.1 together with Theorem 5.5 and directly compute an upper bound on the probability that adt-SCS reaches an unsafe region in a finite-time horizon, much alike (2.7).
Remark 5.7.
Note that the verification problem is a special case of the synthesis one, in which the main goalis to verify that the property of interest is satisfied with some probability lower-bound. The statements abovecan be accordingly tailored by changing the quantifier ‘ ∃ ’ in (5.3) to ‘ ∀ ’, and the results follow. Computation of CBC and of Control Policies.
In this subsection, we discuss suitable methods tosearch for CBCs and to synthesize corresponding control policies. We study two different approaches based on(i) sum-of-squares (SOS) optimization and on (ii) counter-example guided inductive synthesis (CEGIS) [JSZ20].5.1.1.
Sum-of-Squares Optimization Problem.
We reformulate conditions (5.1)-(5.3) as an SOS optimizationproblem [Par03], where a CBC is restricted to be a non-negative polynomial that can be written as a sum ofsquares of different polynomials. To do so, the following assumption is required.
Assumption 2.
The dt-SCS Σ has a continuous state set X ⊆ R n , and continuous input set U ⊆ R m .Moreover, the vector field f : X × U × V ς → X is a polynomial function of the state x and of the input ν .Sets X and U are bounded semi-algebraic sets (i.e., they can be represented by the intersection of polynomialinequalities). Under Assumption 2, one can reformulate conditions (5.1)-(5.3) as an SOS optimization problem to search fora polynomial CBC B and a polynomial controller ν ( · ) for the dt-SCS Σ. The following lemma provides theSOS formulation. Lemma 5.8.
Suppose Assumption 2 holds and sets X , X u , X can be defined as X = { x ∈ R n | g ( x ) ≥ } , X u = { x ∈ R n | g u ( x ) ≥ } and X = { x ∈ R n | g ( x ) ≥ } , where g , g u and g are vectors of polynomials andinequalities are intended element-wise. Suppose for a given dt-SCS Σ , there exists a sum-of-squares polynomial B ( x ) , constants η, β, ¯ c ∈ R ≥ , with β > η , < ¯ κ < , vectors of sum-of-squares polynomials λ ( x ) , λ u ( x ) , λ ( x ) , and polynomials λ ν j ( x ) corresponding to the j th input in ν = ( ν , ν , ..., ν m ) ∈ U ⊆ R m of appropriatedimensions, such that the following expressions are sum-of-squares polynomials: − B ( x ) − λ T ( x ) g ( x ) + η B ( x ) − λ Tu ( x ) g u ( x ) − β − E h B ( f ( x, ν, ς )) | x, ν i + ¯ κ B ( x ) + ¯ c − m X j =1 ( ν j − λ ν j ( x )) − λ T ( x ) g ( x ) . Then B ( x ) is a CBC satisfying conditions (5.1) - (5.3) and ν = [ λ ν ( x ); . . . ; λ ν m ( x )] , is the correspondingcontroller of the dt-SCS Σ , where κ = 1 − (1 − π )(1 − ¯ κ ) , c = ¯ cπ (1 − ¯ κ ) , with < π < . For such computations, one can readily employ existing software tools available in the literature such as
SOSTOOLS [PAV + SeDuMi [Stu99].
UTOMATED VERIFICATION AND SYNTHESIS OF STOCHASTIC HYBRID SYSTEMS: A SURVEY 27
Counter-Example Guided Inductive Synthesis.
One can find a CBC with a given parametric form, e.g., a polynomial, by utilizing satisfiability modulo theories (SMT) solvers such as Z3 [DMB08], dReal [GAC12]or MathSat [CGSS13]. The counter-example guided inductive synthesis (CEGIS, [SLTB + Lemma 5.9.
Consider the dt-SCS Σ . Suppose there exists a function B ( x ) , constants η, β, c ∈ R ≥ , and < κ < such that ^ x ∈ X ( B ( x ) ≤ η ) ^ x ∈ X u ( B ( x ) ≥ β ) ^ x ∈ X _ ν ∈ U (cid:16) E h B ( f ( x, ν, ς )) (cid:12)(cid:12) x, ν i ≤ max n κ B ( x ) , c o(cid:17) , where the index sets of conjunctions and disjunctions are possibly infinite. Then B ( x ) is a CBC satisfyingconditions (5.1) - (5.3) . Remark 5.10.
The computational complexity in the construction of finite MDPs as in Algorithm 1 growsexponentially with the dimension of the state set. In contrast, in the case of sum-of-squares optimization, thecomputational complexity depends on both the degree of the polynomials and the number of state variables.It is shown that for fixed degree of the polynomials, the required computation grows polynomially with thedimension [WTL15] . Hence, we expect this technique to be more scalable than discretization-based approachesto study specific problems, such as safety analysis, that can be tackled via barrier certificates. The CEGISapproach [JSZ20] has a bottleneck that resides with the SMT solver, and it is difficult to provide any analysison the computational complexity due to its iterative nature and lack of completeness (termination) guarantees.
Related Work.
Within this line of work, the synthesis of invariants to study probabilistic safety ofinfinite-state models (probabilistic programs) is discussed in [CS13]. The proposed analysis employs concen-tration inequalities and martingales theory.Finite-time safety verification of stochastic nonlinear systems using barrier certificates is proposed in [ST12].The work considers the problem of bounding the probability of failure (defined as leaving a given boundedregion of the state space) over a finite-time horizon for stochastic nonlinear systems with continuous stateand in continuous time. The proposed approach searches for exponential barrier functions ( e.g., B ( x, k ) = e x T M ( k ) x −
1) that provide bounds using c -martingale type conditions as in (5.3), however in continuous time.Probabilistic safety verification of systems using barrier certificates is proposed in [HCL + i.e., allstates are required to be constrained in a bounded compact set) are discussed in [LLT + , SADEGH SOUDJANI , ALESSANDRO ABATE , AND MAJID ZAMANI , Figure 11.
Closed-loop state trajectories with 10 different noise realizations for the finite-time horizon T d = 10.access to the full system information, as well as incomplete information systems where the state must bereconstructed from noisy measurements. In the complete information case, it formulates barrier functionsthat leads to sufficient conditions for safety with probability 1. However, in order to provide infinite-timehorizon guarantees, this result requires that the control barrier functions exhibit supermartingale property,which presupposes stochastic stability and vanishing noise at the equilibrium point of the system. Thisapproach is only applicable to systems with unbounded input sets and it does not provide any probabilisticguarantee 1 when the input set is bounded. In the incomplete information case, it formulates barrier functionsthat take an estimate from an extended Kalman filter as input, and derives bounds on the probability of safetyas a function of the asymptotic error for the filter.A methodology for safety verification of non-stochastic systems using barrier certificates is proposed in [PR05].Using the concepts of convex duality and density functions, the paper presents a converse statement for barriercertificates, showing that the existence of a barrier certificate is also necessary for safety. The results are thenextended in [WS15] to more general classes of dynamical systems: in particular, [WS15] proves converse barriercertificate theorems for a class of structurally stable dynamical systems. Open Problem 5.
There is no guarantee in general for the existence of barrier certificates for SHS. Inparticular, Definition 5.1 provides a set of sufficient conditions for the existence of CBC: if there exists abarrier certificate, then the system is safe with the a lower bound on the safety probability determined bythe CBC. One interesting direction as a future work is to investigate necessary and sufficient conditionsfor the existence of control barrier certificates for SHS.
Open Problem 6.
Develop computational techniques to construct CBC for general, nonlinear dt-SCS (notonly polynomial-type models).
Running example (continued) . The regions of interest in this example are considered as X ∈ [1 , , X ∈ [19 . , , X u = [1 , ∪ [23 , , SOSTOOLS [PAV +
13] and the SDP solver
SeDuMi [Stu99] to compute CBC asdescribed in Definition 5.1. We compute CBC of an order 2 as B ( T ) = 0 . T − . T + 331 . ν ( T ) = − . T + 0 .
9. Furthermore, the corresponding parametersin Definition 5.1 satisfying conditions (5.1)-(5.3) are quantified as η = 0 . , β = 4 . , κ = 0 . , and c = 99 × − . UTOMATED VERIFICATION AND SYNTHESIS OF STOCHASTIC HYBRID SYSTEMS: A SURVEY 29
By employing Theorem 5.3, one can guarantee that the temperature of the room starting from the initial set X = [19 . ,
20] remains in the safe set X u = [17 ,
23] during the time horizon T d = 10 with a probability atleast 95%, i.e., P n x ( k ) ∈ X u for some k ∈ [0 , T d ] (cid:12)(cid:12) a o ≥ . . (5.8)Closed-loop state trajectories with 10 noise realizations are illustrated in Figure 11. (cid:3) Temporal Logic Verification and Synthesis
In this section, we discuss how one can perform verification and synthesis for stochastic hybrid systems overinteresting requirements, such as safety, reachability, or over more complex specifications encompassed bytemporal logic or omega-regular languages. In presenting work at the interface between control theory andformal methods, we mainly focus on LTL and PCTL properties in this survey for the sake of better readability.A quantitative, abstraction-based controller synthesis for SHS is proposed in [TMKA13], for specificationsexpressed over DFAs. The problem is first reformulated as an optimization of a probabilistic reachabilityproperty over a product process obtained from the model for the specification and the model of the system.The article develops a discretization procedure leading into standard dynamic programming problems overfinite MDPs with history-independent Markov policies. Errors are in the form of Equation (2.4). A similarcontroller design scheme for stochastic hybrid systems is also provided in [KSL13]. As a generalization, anoptimal control synthesis approach defined over general controlled discrete-time Markov processes is proposedin [TMKA17a] in which the probability of a given event is optimized: it is shown that the optimization over awide class of LTL and ω -regular properties can be reduced to the solution of one of two fundamental problems:reachability and repeated reachability.A policy refinement scheme for dt-SCS via approximate similarity relations based on δ -lifting is proposedin [HSA17] by providing a closenesses guarantee similar to (2.3). In particular, given safety properties overthe concrete system, the work constructs an epsilon-perturbed specification over the abstract model whoseprobability of satisfaction gives a lower bound for the probability of satisfaction in the concrete domain withsome quantified error bounds in the form of (2.3). The work is then generalized in [HSA18] to a larger classof temporal properties ((bounded) probabilistic reachability problem and temporal logic control problem) andin [HS18] to synthesize policies for a robust satisfaction of specifications. This framework is applied in buildingautomation systems [HCA17].Policy refinement over co-safe linear temporal logic for stochastic control systems is proposed in [LSZ19b]in which it is discussed how synthesized policies for abstract systems can be refined back to original modelswhile providing guarantees on the probability of satisfaction. All those works [HSA17, HSA18, HS18, LSZ19b]quantify a probabilistic distance between original systems and that of their epsilon-perturbed abstractions as aversion of closeness guarantee proposed in (2.3). We should highlight that, given the DFA A in Definition 1.10,the epsilon-perturbed specification in [HSA17, HSA18, HS18, LSZ19b] corresponds to a new DFA ˆ A ϕ =( ¯ Q ℓ , q , ¯Σ a , F a , ¯ t ) in which one absorbing location q abs and one letter φ ◦ are added as ¯ Q ℓ := Q ℓ ∪ { q abs } and ¯Σ a := Σ a ∪ { φ ◦ } . The initial and accept locations are the same with A φ . The transition relation is defined, ∀ q ∈ ¯ Q ℓ , ∀ a ∈ ¯Σ a , as ¯ t ( q, a ) := t ( q, a ) if q ∈ Q ℓ , a ∈ Σ a ,q abs if a = φ ◦ , q ∈ ¯ Q ℓ ,q abs if q = q abs , a ∈ ¯Σ a . In other words, an absorbing state q abs is added and all states will jump to this absorbing state with label φ ◦ .As an example, the modified DFA of the reach-avoid specification in Figure 4 is plotted in Figure 12.Temporal logic versification and control of stochastic systems via control barrier certificates against a fragmentof linear temporal logic, i.e., safe LTL, over finite traces are presented in [JSZ18, JSZ20]. , SADEGH SOUDJANI , ALESSANDRO ABATE , AND MAJID ZAMANI , q q q q a ba bcc f a; b; c g φ ◦ φ ◦ φ ◦ φ ◦ q abs f a; b; c g f a; b; c; φ ◦ g Figure 12.
Modified DFA ˆ A ϕ of the specification ( a U b ).Forward stochastic reachability analysis for uncontrolled linear systems with affine (bounded or unbounded)disturbance is presented in [VHO17]. The proposed method utilizes Fourier transforms to efficiently computethe forward stochastic reach probability measure (density) and the forward stochastic reach set. Underpinnedby the same technique, an under-approximation of the stochastic reach-avoid probability for high-dimensionallinear stochastic systems is presented in [VO17] while providing guarantees of the type is equations (2.7)and (2.8). The proposed framework exploits fixed control sequences parameterized by the initial condition(an open-loop control policy) to generate the under-approximation. For Gaussian disturbances, the under-approximation can be obtained using existing efficient algorithms by solving a convex optimization problem.The work in [VO18] proposes a scalable algorithm to construct a polytopic under-approximation of the ter-minal hitting time stochastic reach-avoid set, for the verification of high-dimensional linear stochastic sys-tems with arbitrary stochastic disturbance. The existence of a polytopic under-approximation is proved bycharacterizing sufficient conditions under which the stochastic reach-avoid set and the proposed open-loopunder-approximation are compact and convex.Under-approximation of finite-time horizon, stochastic reach-avoid sets for discrete-time stochastic nonlinearsystems is discussed in [GVO17] via Lagrangian methods. The article utilizes the concept of target-tubereachability to define robust reach-avoid sets that are parameterized by the target set, safe set, and the setwhich the disturbance is drawn from. The proposed framework unifies two existing Lagrangian approaches tocompute these sets, and establishes that there exists an optimal Markov control policy for the robust reach-avoid sets. The results characterize the subset of the disturbance space whose corresponding robust reach-avoidset for a given target and safe set is a guaranteed underapproximation of the stochastic reach-avoid level setof interest. Although the proposed method is conservative, it does not rely on a grid, implying scalabilityfeatures that now hinge on geometrical computations.A framework for analyzing probabilistic safety and reachability problems for discrete-time SHS, in scenarioswhere system dynamics are affected by competing agents, is proposed in [KDS + + lexico-graphic approach that priorities the safety as a constraint to be met prior to optimizing a given reward. The UTOMATED VERIFICATION AND SYNTHESIS OF STOCHASTIC HYBRID SYSTEMS: A SURVEY 31 q q q q a ba bcc f a; b; c gf a; b; c g scLTL Spesi fi cation UnknownContinuous-Space MDPsModel-FreeReinforcement LearnerInterpreter A c t i o n S t a t e δ -quantized Observations, Rewards Figure 13.
Model-free reinforcement learning is employed by a DFA corresponding to an scLTL objective. In particular, the δ -quantized observation set of the dt-SCS Σ is used byan interpreter process to compute a run of the DFA. When the run reaches a final state, theinterpreter gives the reinforcement learner a positive reward and the training episode termi-nates. Any converging reinforcement learning algorithm over such δ -quantized observation setis guaranteed to maximize the probability of satisfaction of the scLTL objective and convergeto an optimal strategy over the unknown dt-SCS Σ.paper in [HNS21] proposes an abstraction framework for computation of policies to satisfy multiple specifica-tions with different priorities by encoding them in a multi-objective framework.We discuss briefly the formal synthesis over SHS via learning and data-driven approaches in Subsection 11.1as an open research direction. Here, to conclude this section, we only present a few limited recent workswith a focus on temporal logic verification and synthesis via learning approaches. A reinforcement learningframework for controller synthesis of finite MDPs with unknown transition probabilities against LTL objec-tives with a proof of convergence is proposed in [HAK19a, HPS + ω -regular properties into limit deterministic B¨uchi automata (LDBA), instead of theRabin automata that are standard with MDPs. [HAK19a] exploits the structure of the LDBA and shapes asynchronous reward function on-the-fly, so that an RL algorithm can synthesize a policy resulting in tracesthat maximize the probability of satisfying the linear temporal property. If the dt-SCS is finite, theoreticalguarantees are provided on the convergence of the RL algorithm to an optimal policy, maximizing the satisfac-tion probability. The setup is extended to probabilistic labels in [HKA + +
19] presents a constructivereduction from the satisfaction of LTL objectives to a reachability problem, and extends this technique tolearning how to control an MDP with unknown transition probabilities so that the chance of satisfying theobjective is maximized.A model-free reinforcement learning scheme to synthesize policies for unknown continuous-space dt-SCS isproposed in [LSS + , SADEGH SOUDJANI , ALESSANDRO ABATE , AND MAJID ZAMANI , Figure 13, extensions to continuous spaces and ω -regular properties with formal guarantees are studied in[HAK19b, HKA20, KS20]. This line of works leads to the following problem. Open Problem 7.
Provide convergence guarantee for learning-based approaches that compute a controllerfor dt-SCS to satisfy any given LTL specification. Compositional Techniques
It is of interest to extend the techniques introduced in previous sections to networks of interacting models, orto models with specially structured dynamics or coupling between variables. Moreover, the construction of(in)finite abstractions for large-scale stochastic hybrid systems in a monolithic manner suffers severely fromthe curse of dimensionality. To mitigate this issue, one promising solution is to consider a large-scale complexmodel as an interconnected system composed of several smaller subsystems. Compositional techniques arespecifically suitable to tackle these problems, and are broadly studied in this section. We overview resultson compositional frameworks for the construction of (in)finite abstractions for interconnected systems usingabstractions of smaller subsystems.We first define stochastic control subsystems next. The term “internal” is employed for inputs and outputs ofsubsystems that are affecting each other in the interconnection: an internal output of a subsystem affects aninternal input of another subsystem downstream. The term “external” instead is utilized to denote (exogenous)inputs and outputs that are not employed for the construction of the interconnection.
Definition 7.1.
A discrete-time stochastic control subsystem (dt-SCSu) is described by the tuple
Σ = (
X, U, W, ς, f, Y , Y , h , h ) , (7.1) where • X ⊆ R n is a Borel space as the state space of the subsystem; • U ⊆ R m is a Borel space as the external input space of the subsystem; • W ⊆ R ¯ p is a Borel space as the internal input space of the subsystem; • ς is a sequence of i.i.d. random variables from a sample space Ω to the measurable space ( V ς , F ς ) ; ς := { ς ( k ) : (Ω , F Ω ) → ( V ς , F ς ) , k ∈ N } ; • f : X × U × W × V ς → X is the transition map; • Y ⊆ R q is a Borel space as the external output space of the subsystem; • Y ⊆ R q is a Borel space as the internal output space of the subsystem; • h : X → Y is a measurable function as the external output map that takes a state x ∈ X to its external output y = h ( x ) ; • h : X → Y is a measurable function as the internal output map that takes a state x ∈ X to its internal output y = h ( x ) . Properties of the interconnected system are specified over external outputs, as in Definition 7.1, and thesynthesis objective is to control external inputs in order to satisfy desired properties over external outputs;whereas internal signals are utilized for the sake of interconnections amongst subsystems. We are now wellequipped to define an interconnected SCS.
Definition 7.2.
Consider N ∈ N ≥ stochastic control subsystems Σ i = ( X i , U i , W i , ς i , f i , Y i , Y i , h i , h i ) , ∀ i ∈ { , . . . , N } , and a matrix M of appropriate dimensions, defining the coupling between these subsystems.The interconnection of Σ i for any i ∈ { , . . . , N } , is the SCS Σ = (
X, U, ς, f, Y, h ) , denoted by I (Σ , . . . , Σ N ) ,such that X := Q Ni =1 X i , U := Q Ni =1 U i , f := Q Ni =1 f i , Y := Q Ni =1 Y i , and h = Q Ni =1 h i , subjected to thefollowing interconnection constraint: [ w ; . . . ; w N ] = M [ h ( x ); . . . ; h N ( x N )] . UTOMATED VERIFICATION AND SYNTHESIS OF STOCHASTIC HYBRID SYSTEMS: A SURVEY 33 Σ ν y Σ N . . . M [ h ( x ); : : : ; h N ( x N )] [ w ; : : : ; w N ] Figure 14.
An interconnected SCS with external input and output signals ν and y , respec-tively. Note that in the small-gain setting the interconnection matrix M is a permutationmatrix that results in an element-wise interconnection constraint ( i.e., ∀ i, j ∈ { , . . . , N } , i = j : w ij = y ji , y ji ⊆ W ij ). x x x : : :
03 0 : : : : : : ^ ν Σ (cid:1) x x x : : : : : : :
05 0 : : ^ y ν y Σ N . . ... . ^ M M x x x : :
07 0 :
03 0 : : : : : : ^ ν i ^ w i ^ h i (^ x i )^ h i (^ x i ) ν i w i h i ( x i ) h i ( x i ) Σ i (cid:1) Figure 15.
Representation of compositionality results under external input and output sig-nals ν and y , respectively. We denote b Σ (cid:22) Σ if there exists an SSF V from b Σ to Σ (cf.Definition 1.6).An interconnected SCS based on Definition 7.2 is schematically depicted in Figure 14.In this section, we discuss two different compositional approaches based on small-gain and dissipativity con-ditions. Small-gain and dissipativity techniques have been traditionally employed in the context of stabilityanalysis for networks of interconnected systems [DRW10, AMP16]. Suppose Σ is an interconnected SCS with N stable subsystems Σ , . . . , Σ N . Under some small-gain or dissipativity conditions, one can ensure that thecomposed network Σ = I (Σ , . . . , Σ N ) is also stable. A similar idea can be utilized here in the setting ofsimilarity relations, using which one can establish a formal relation between an interconnected system Σ and , SADEGH SOUDJANI , ALESSANDRO ABATE , AND MAJID ZAMANI , its abstraction b Σ, based on relations between subsystems and their corresponding abstractions. We presentthe semantics of compositionality techniques in the following.
Semantics of compositionality techniques.
Consider the interconnected SCS Σ = I (Σ , . . . , Σ N ), andassume proper relations between subsystems Σ i and their corresponding abstractions b Σ i in the sense of Def-inition 1.6. Under some compositionality conditions, one can construct an overall relation between the twointerconnected systems b Σ = I ( b Σ , . . . , b Σ N ) and Σ, based on the relations between the subsystems and theirabstractions.Compositionality conditions based on dissipativity approaches are in the form of LMI that can be readilychecked via semidefinite programming (SDP) solvers such as SeDuMi [Stu99]. For small-gain reasoning, wedistinguish the corresponding compositionality conditions based on so-called sum-type and max-type small-gain approaches. In particular, in sum-type small-gain approach, the second condition of the stochasticsimulation function (SSF) is in the form of (1.7), and the overall SSF is a weighted sum of SSF of subsystems.Accordingly, one deals with a spectral radius of some matrix that needs to be strictly less than one as thecompositionality condition. In contrast, in max-type small-gain approach, the upper bound in (1.7) is in themax form and the overall SSF is based on maximum of SSF of subsystems. We refer the interested readers formore details on the compositionality conditions in (sum and max-type) small-gain and dissipativity approachesrespectively to [LSMZ17, LSZ20d] and [LSZ19b]. It will be further discussed in this section that the closenessguarantees are of the type in (2.6).Compositionality results have been schematically depicted in Figure 15. As illustrated, if there exists alocal stochastic simulation function between each original subsystem and its corresponding finite MDP, onecan construct an overall stochastic simulation function between the original interconnected system and itsinterconnected finite abstraction provided that some compositionality conditions are satisfied.
Remark 7.3.
Note that the proposed compositionality results based on sum-type small-gain approaches [LSMZ17] require linear growth on gains of subsystems (cf. [LSMZ17, Assumption 1] ) and provide an additive overallerror (i.e., the error of the interconnected abstraction is linear combination of errors of abstractions of subsys-tems). In contrast, the max -type small-gain approaches [LSZ20d] are more general, since they do not requireany linearity assumption on gains of subsystems and the overall error is the maximum error of abstractionsof subsystems. Both errors provide closeness guarantees of the type in (2.6) . On the other hand, checking thecompositional condition in the sum-type small-gain is much easier than the max -type one, since it is based onthe spectral radius of some matrix that can be easily checked.
Compositional techniques based on infinite and finite abstractions have been schematically illustrated in Fig-ure 16.Compositional construction of finite abstractions for stochastic control systems is presented in [SAM15,SAM17]. Those works investigate the finite-horizon probabilistic invariance for dt-SCS and provide a close-ness guarantee between two systems in the form of (2.3). The compositionality framework is based on finite dynamic Bayesian networks (DBNs) and the works exploit the structure of the underlying Markov process tocompute the abstraction separately for each dimension and discuss how factor graphs and the sum-productalgorithm for DBNs can be utilized to solve the finite-horizon probabilistic invariance problem.Compositional construction of infinite abstractions for interconnected SCS is proposed in [LSMZ17] via sum-type small-gain conditions. The abstraction framework is based on notions of SSF in Definition 1.6, usingwhich one can quantify the probabilistic distance between original interconnected stochastic control systemsand their abstractions based on the closeness type in (2.6). A compositional scheme for constructing infinite abstractions based on dissipativity approaches is presented in [LSZ19b]. The proposed scheme employs theinterconnection matrix and joint dissipativity-type properties of subsystems and their abstractions describedby a notion of stochastic storage functions [LSZ19b, Definition 3.1].
UTOMATED VERIFICATION AND SYNTHESIS OF STOCHASTIC HYBRID SYSTEMS: A SURVEY 35 x ( k + 1) = f ( x ( k ) ; ν ( k ) ; In fi nite system (original model)Finite system (MDP) Hybrid controller & ( k ))~ x ( k + 1) = ~ f (~ x ( k ) ; ~ ν ( k ) ; ~ & ( k ))In fi nite system (lower dimension) ^ x ^ x ^ x : : :
03 0 : : : : : : ^ ν ^ x ^ x ^ x : : : : : : :
05 0 : : ^ y ...^ M Σ ν y Σ N . . . M ~ ν ~ y ~ M ~ Σ ~ Σ N . . . Hybrid controller q q q Lookup Table ^ ν (^ x ) orFinite Memory Controller: ~ ν = ν ~ ν (~ x; ^ x; ^ ν ) ν = ν ^ ν ( x; ~ x; ~ ν ) Figure 16.
Compositional techniques based on abstractions.Compositional construction of both infinite and finite abstractions via max-type small-gain conditions is dis-cussed in [LSZ18a, LSZ20d]. The proposed overall error is computed based on maximum errors of subsystems.The articles in [LSZ18a, LSZ20d] employ a variant of notions of SSF in Definition 1.6) and provide the prob-abilistic distance between the interconnection of stochastic control subsystems and that of their (in)finiteabstractions based on (2.6). The proposed framework also leverages the δ -ISS property of original systemsas in Definition 1.5 and provides an approach to construct finite MDPs of the concrete models (or theirreduced-order versions).Compositional construction of finite abstractions for stochastic control systems is also presented in [LSZ18b]but using dissipativity conditions. This work provides a closeness in the form of (2.6) and proposes an approachto construct finite MDPs for the general setting of discrete-time nonlinear SCS satisfying a passivity-likeproperty, whereby one can construct finite MDPs by selecting a suitable discretization of the input and statesets. Moreover, for linear SCS, the aforementioned property boils down to a matrix inequality. , SADEGH SOUDJANI , ALESSANDRO ABATE , AND MAJID ZAMANI , Compositional construction of finite MDPs for networks of not necessarily stabilizable stochastic systemsis presented in [LSZ19c] and [LSZ20c] respectively via relaxed max small-gain and dissipativity approaches.The proposed framework relies on a relation between each subsystem and its finite abstraction employing anew notion of simulation functions, called finite-step stochastic simulation functions [LSZ19c, Definition 3.5].In comparison with the existing notions of simulation functions in which stability or stabilizability of eachsubsystem is required, a finite-step stochastic simulation function needs to decay only after some finite numbersof steps instead of at each time step. This relaxation results in a less conservative version of small-gain ordissipativity conditions.Compositional construction of finite abstractions for networks of stochastic switched systems is presentedin [LSZ20a, LZ20] via respectively small-gain and dissipativity approaches. These contributions utilize notionsof stochastic simulation (or storage) functions and provide a closeness guarantee in the form of (2.6), howeveradapted to switched models.Compositional abstraction-based synthesis of dt-SCS using approximate probabilistic relations is proposedin [LSZ19a, LSZ20b]. The abstraction framework is based on the notion of δ -lifted relations, using which onecan quantify the distance in probability between the interconnected dt-SCS and that of their abstractions asa version of closeness guarantee proposed in (2.3). Those results provide some matrix (in)equality conditionsfor simultaneous existence of relations incorporating the structure of the network. It is shown that theunified compositional scheme is less conservative than the two-step consecutive procedure that independentlyconstructs infinite and finite abstractions ( e.g., [LSZ18b, LSZ19b]). Compositional construction of infinite andfinite abstractions for large-scale discrete-time stochastic systems via different compositionality conditions iswidely discussed in [Lav19]. Open Problem 8.
Developing a compositional multi-objective synthesis framework for continuous-spaceSHS is an interesting open problem. Note that there have been some partial results on this problem fordiscrete-space models [KNPQ13] and for continuous-space monolithic models [HNS21] , which however arerespectively not applicable or in general computationally tractable for (large-scale) SHS.
Although the proposed compositional frameworks for constructing finite abstractions can mitigate the state-space explosion problem, the curse of dimensionality may still arise at the level of single subsystems. Asdiscussed in Section 5, an alternative direction is to employ control barrier functions as a discretization-freetechnique for controller synthesis of complex stochastic systems. However, searching for control barrier certifi-cates for large-scale systems can be also computationally expensive. Consequently, developing compositionaltechniques for constructing control barrier functions is a promising solution to alleviate this complexity. Com-positional construction of control barrier certificates for large-scale stochastic control systems is presentedin [ALZ20, ALZ21]. The proposed compositional methodology is based on a notion of control sub-barriercertificates , enabling one to construct control barrier certificates of interconnected systems by leveraging somemax-type small-gain conditions. Compositional construction of control barrier certificates for discrete-timestochastic switched systems accepting multiple barrier certificates with some dwell-time conditions is insteadproposed in [NSZ20a].
Running example (continued) . We present the dynamics of (1.4) based on a network of two rooms, asillustrated in Figure 17. The evolution of the temperatures T i can be described by (1.4) in which A is a matrixwith diagonal elements ¯ a ii = (1 − σ − θ − γν i ( k )), i ∈ { , } , and off-diagonal elements a , = a , = σ .Parameter σ = 0 . T ( k ) = [ T ( k ); T ( k )], ν ( k ) =[ ν ( k ); ν ( k )], ς ( k ) = [ ς ( k ); ς ( k )], T E = [ T e ; T e ], and R = 0 . I . By considering the individual rooms Σ i asΣ i : (cid:26) T i ( k + 1) = ¯ a ii T i ( k ) + γT h ν i ( k ) + D i w i ( k ) + θT ei + 0 . ς i ( k ) ,y i ( k ) = T i ( k ) , (7.2)one can readily verify that Σ = I (Σ , Σ ) where D i = σ , and w i ( k ) = y i − ( k ) for any i ∈ { , } (with y = y ). One can also establish a quadratic stochastic simulation function between Σ i and b Σ i in the form UTOMATED VERIFICATION AND SYNTHESIS OF STOCHASTIC HYBRID SYSTEMS: A SURVEY 37 I (Σ , Σ ) Σ : Room 1Σ : Room 2 y ν y ν y w y w Figure 17.
Interconnection of two rooms Σ and Σ . Figure 18.
Closed-loop state trajectories of Room 1 (top) and Room 2 (bottom) with 10different noise realizations for the finite-time horizon T d = 100.of S i ( T i , ˆ T i ) = ( T i − ˆ T i ) satisfying [LSZ20d, conditions (III.1), (III.2)] with α i ( s ) = s , κ i ( s ) = 0 . s , ρ int i ( s ) = 0 . s , ρ ext i ( s ) = 0, ∀ s ∈ R ≥ , and ψ i = 6 . δ i , for any i ∈ { , } .Now we need to check the small-gain condition for the interconnected system (1.4). The small-gain condi-tion [LSZ20d, equation (V.2)] is readily satisfied if κ .κ < , (7.3)where κ ij for any i, j ∈ { , } , i = j , is defined as κ ij ( s ) = ρ int i ( α − j ( s )). Since κ = κ = 0 .
97, the small-gain condition (7.3) is simply satisfied. Hence, V ( T, ˆ T ) = max i ( T i − ˆ T i ) for any i ∈ { , } is a stochasticsimulation function from the interconnected system b Σ to Σ. By taking the state discretization parameter δ = 0 . b Σ as 20 , and using [LSZ20d, ineqality (III.3)], one can guarantee that the distance between outputs of Σand b Σ will not exceed ε = 0 . T d = 100 with a probability at least 98%, in the formof (2.6), i.e., P n k y ( k ) − ˆ y ( k ) k ≤ . , ∀ k ∈ [0 , o ≥ . . (7.4)Let us now synthesize a controller for Σ via its finite abstraction b Σ such that the controller maintains thetemperature of any room in a comfort zone defined as the interval [19 , b Σ i , and then refine it back to the subsystem Σ i using an interface map. Consequently, theoverall controller for the interconnected system Σ would be a vector such that each of its components is the We should highlight that condition (7.3) is exactly similar to what proposed in [Zam66] in the context of stability verificationof feedback interconnection of two linear systems. , SADEGH SOUDJANI , ALESSANDRO ABATE , AND MAJID ZAMANI , Σ Σ Σ Σ Σ Σ Figure 19.
A circular interconnection for a network of 1000 rooms.controller for subsystems Σ i . We employ the software tool AMYTISS [LKSZ20] to synthesize controllers forΣ i . Closed-loop state trajectories of two rooms with 10 different noise realizations are illustrated in Figure 18.The simulations show that none of 10 trajectories violates the specification, which is in accordance with thetheoretical guarantee (7.4). As discussed in Section 4, if one employs our designed controllers and run MonteCarlo simulations of the closed-loop model, the distance between outputs of Σ and b Σ will likely be empiricallycloser than 0 . n = 1000 and interconnect them in a circular fashion, as depictedFigure 19). In this case, A in (1.4) is a matrix with diagonal elements ¯ a ii = (1 − σ − θ − γν i ( k )), i ∈ { , . . . , n } ,off-diagonal elements ¯ a i,i +1 = ¯ a i +1 ,i = ¯ a ,n = ¯ a n, = σ , i ∈ { , . . . , n − } , and all other elements are identicallyzero. Moreover, σ is a conduction factor between pairs of room i ± i , T ( k ) = [ T ( k ); . . . ; T n ( k )], ν ( k ) =[ ν ( k ); . . . ; ν n ( k )], ς ( k ) = [ ς ( k ); . . . ; ς n ( k )], T E = [ T e ; . . . ; T en ], R = 0 . I n . Considering the individual roomsΣ i as (7.2), one can readily verify that Σ = I (Σ , . . . , Σ N ) where D i = [ σ ; σ ] T , and w i ( k ) = [ y i − ( k ); y i +1 ( k )](with y = y n and y n +1 = y ).We set the state discretization parameter δ = 0 . b Σ as 20 . Using the proposed bound in [LSZ20d, ineqality (III.3)], one can guarantee that the distancebetween outputs of Σ and b Σ will not exceed ε = 0 . T d = 100 with the probability atleast 98%, i.e., P n k y ( k ) − ˆ y ( k ) k ≤ . , ∀ k ∈ [0 , o ≥ . . (7.5)We employ AMYTISS [LKSZ20] and synthesize a controller for Σ via the abstraction b Σ such that the controllermaintains the temperature of any room in the comfort zone [19 , T d = 100. (cid:3) Continuous-Time Stochastic Hybrid Systems
Foundations of continuous-time SHS can be traced back to the work on piecewise-deterministic Markov modelsin [Dav93], which was extended to diffusion processes in [HLS00]. An early survey of work can be found in[PBLB03]. In this survey, we present selected results for the continuous-time setting, categorized according tothe different topics discussed in the previous sections.
Notations.
We assume that for continuous-time processes, the triple (Ω , F Ω , P Ω ) denotes a probability spaceendowed with a filtration F = ( F s ) s ≥ satisfying the standard conditions of completeness and right-continuity.Let ( W s ) s ≥ be a b -dimensional F -Brownian motion, and ( P s ) s ≥ be an r -dimensional F -Poisson process. We UTOMATED VERIFICATION AND SYNTHESIS OF STOCHASTIC HYBRID SYSTEMS: A SURVEY 39
Figure 20.
Closed-loop state trajectories of a representative room with 10 different noiserealizations, for the network of 1000 rooms.assume that the Poisson process and the Brownian motion are independent of each other. The Poisson process P s = [ P s ; · · · ; P r s ] models r events whose occurrences are assumed to be independent of each other.There is a notion of SSF also for continuous-time stochastic systems (cf. [ZRME17, Definition 3.2]) that issimilar to that in Definition 1.6. The only difference is that in the left-hand side of (1.7), instead of the shownone-step expectation, there is a condition on the stochastic process that acts on a twice differentiable function V . This notion was initially developed in [JP09]. Stochastic Similarity Relations.
In the next theorem, we present a result on the closeness in expecta-tion (moment) of the difference between output trajectories of original continuous-time systems Σ and theircorresponding abstractions b Σ, as proposed in [ZMEM + Theorem 8.1.
Let Σ be a continuous-time SCS and b Σ be its abstraction. Suppose there exists a stochasticsimulation function V : X × ˆ X → R ≥ from b Σ to Σ similar to Definition 1.6 but for continuous-time stochasticsystems. For any input trajectory ˆ ν ( · ) ∈ ˆ U that preserves the Markov property for the closed-loop b Σ , and forany random variables a and ˆ a as the initial states of Σ and b Σ , respectively, one can construct an input trajectory ν ( · ) ∈ U for Σ through an interface function associated with V (cf. Def. 1.6), such that: E h k y aν ( t ) − ˆ y ˆ a ˆ ν ( t ) k ¯ q i ≤ λ , ∀ t ∈ R > , (8.1) where, λ := β ( E [ V ( a, ˆ a )] , t ) + ρ ext ( E [ k ˆ ν k ¯ q ∞ ]) + c, with β ∈ KL , ρ ext ∈ K ∞ , c ∈ R > , and ¯ q ∈ N ≥ denoting the moment of a random variable. Remark 8.2.
Note that one can leverage the bound in (8.1) together with the Markov inequality [Oks13] toprovide a lower bound on the probability of satisfaction of logic specifications for which satisfiability is onlyconcerned at single time instances (e.g., reachability). The new bound is similar to (2.6) , but the supremumappears outside of the probability operator [ZMEM + . Note that this is not the case for logic specificationssuch as always (operator (cid:3) ) or even until ( U ). Then, one can conclude that the bound in (8.1) implies (2.6) and accordingly (2.3) and (2.7) . The characterization and computation of probabilistic bisimulations of diffusion processes is discussed in [Aba09].This work proposes sufficient conditions for the existence of a bisimulation function based on the use of con-tractivity analysis and supermartingale inequalities for probabilistic systems. It is shown that the notion of , SADEGH SOUDJANI , ALESSANDRO ABATE , AND MAJID ZAMANI , stochastic contractivity is related to a probabilistic version of the concept of incremental stability. This rela-tionship leads to a procedure that constructs a discrete approximation of a diffusion process. The results arethen extended in [Aba10] to probabilistic bisimulations between two diffusion processes that are additionallyendowed with switching and resetting behaviors. Infinite abstractions.
The construction of infinite abstractions for a class of continuous-time SHS is initiallyproposed in [JP09]. The approximation framework is based on stochastic simulation functions and the workprovides a closeness guarantee in the form of (8.1) but for infinite-time horizons ( i.e., ≤ k < ∞ ). Forthe class of jump linear stochastic systems and linear stochastic hybrid automata, the article shows that thecomputation of stochastic simulation functions can be cast as a linear matrix inequality (LMI) problem. Finite abstractions.
Construction of symbolic models ( i.e., finite abstractions) for incrementally stablestochastic switched systems is proposed in [ZAG15]. The underlying switched systems have a probabilis-tic evolution over a continuous domain and control-dependent discrete dynamics over a finite set of modes.The paper constructively derives approximately equivalent (bisimilar) symbolic models of stochastic switchedsystems. The article provides two different symbolic abstraction techniques: one requires state space dis-cretization, but the other one does not require any space discretization which can be potentially more efficientthan the first one especially when dealing with higher dimensional stochastic switched systems. Both tech-niques provide finite symbolic models that are approximately bisimilar to stochastic switched systems undersome stability assumptions on the concrete models.Construction of symbolic models for randomly switched stochastic system is studied in [ZA14a]. The proposedframework is based on approximate bisimilar relations and leverages some incremental stability assumptionover randomly switched stochastic systems to establish the relation between original systems and their cor-responding finite symbolic models. Construction of finite bisimilar abstractions for incrementally stable sto-chastic control systems without discrete dynamics is presented in [ZMEM + ε >
0, a finite-state transition system can be constructed, which is ε -approximately bisimilar tothe original stochastic control system. It also provides a closeness bound between the δ -ISS stochastic controlsystem and its bisimilar finite abstraction based on the one proposed in (8.1).Abstraction-based synthesis of continuous-time stochastic control systems is also discussed in [NSZ19]. Theproposed framework leverages stochastic simulation functions to relate continuous-time stochastic systems withtheir discrete-time (in)finite counterparts. In order to propose the construction procedure for finite abstrac-tions, the paper first introduces infinite abstractions as time-discretized versions of ct-SCS (as a middle step)since the finite abstractions are constructed from the infinite discrete-time counterparts. The work quantifiesthe distance in probability between original continuous-time stochastic control systems and their discrete-time(finite or infinite) abstractions at sampling times in the form of (2.6). It also constructs finite abstractionstogether with their corresponding stochastic simulation functions for a particular class of stochastic affinesystems having some stability property. It should be mentioned that the original models in [ZAG15, ZA14a]are stochastic, and whereas the constructed abstractions are finite, non-stochastic labeled transition systems,the finite abstractions in [NSZ19] are finite MDPs. Control barrier certificates.
Discretization-free approaches based on barrier certificates for safety verifica-tion of continuous-time stochastic hybrid systems are initially proposed in [PJP07]. The article leverages thesupermartingale property and quantifies an upper bound on the probability that a trajectory of the systemever reaches a given unsafe set (over an infinite time horizon) as proposed in (5.7). For polynomial-typesystems, barrier certificates can be constructed using convex optimization, which is deemed computationallytractable.Verification and control for finite-time safety of stochastic systems via barrier functions are discussed in [SDC19].The proposed certificate condition includes a state-dependent bound on the infinitesimal generator, allowing
UTOMATED VERIFICATION AND SYNTHESIS OF STOCHASTIC HYBRID SYSTEMS: A SURVEY 41 for tighter probability bounds. Moreover, for stochastic systems where the drift dynamics are affine in the con-trol input, the paper proposes a method for synthesizing a polynomial state-feedback controller that achievesa specified safety probability.A verification approach for stochastic switched systems with random switching signals via barrier functionsis presented in [AJZ19]. The temporal properties required on the system are expressed as safe-LTL specifica-tions over finite traces. The proposed approach combines automata-based verification and the use of barriercertificates. It relies on decomposing the automaton associated with the negation of the specification intoa sequence of simpler reachability tasks and computes upper bounds for these reachability probabilities bymeans of common or multiple barrier certificates.Control barrier functions for stochastic systems under (Gaussian) process and measurement noise have beenproposed in [Cla20]. The article first considers the case where the system state is known at each time step,and presents a construction that guarantees almost surely safety. It then extends the approach to models withincomplete state information, where the state must be estimated: it is shown that the proposed certificatesensure safety with probability 1 when the state estimate is within a given bound of the true state, whichcan be achieved using an Extended Kalman Filter when the system is linear or the process and measurementnoises are sufficiently small.Synthesis for stochastic systems with partial state information via control barrier functions is discussedin [JJZ20b]. Given an estimator with a probabilistic guarantee on its accuracy, the paper proposes an approachto compute a controller providing a lower bound on the probability that the trajectories of the stochastic controlsystem remain safe over a finite time horizon (similar to (5.6)). This work does not require a supermartingaleproperty on the control barrier functions, and in particular it does not require any stability assumption on themodel. The results of this article are recently generalized in [JJZ20a] in which no a-priori knowledge aboutthe estimation accuracy is needed. Besides, the class of properties is extended to those expressed by nonde-terministic finite automata (NFA), and the dynamics are also generalized to partially-observed jump-diffusionsystems. Compositional techniques.
Compositional construction of infinite abstractions (in particular, reduced-order models) for a class of SHS is proposed in [ZRME17] using sum-type small-gain conditions. The classof systems includes both jump linear stochastic systems and linear stochastic hybrid automata. The workemploys stochastic simulation functions to quantify an error between the interconnection of stochastic hybridsubsystems and that of their approximations, in the form of (8.1) (but in the continuous-time setting). It alsofocuses on a specific class of SHS, namely jump linear stochastic systems, and proposes a constructive schemeto determine approximations together with their corresponding stochastic simulation functions.Compositional construction of finite abstractions for stochastic control systems is presented in [MSSM17]. Theproposed framework is based on a notion of (approximate) disturbance bisimulation relation, which results ina closeness guarantee with the form of (8.1) (but in the continuous-time setting). Given any SCS satisfyinga stochastic version of a δ -ISS property and a positive error bound, the article shows how to construct afinite-state transition system (whenever existing) that is disturbance-bisimilar to the given stochastic controlsystem.The results of [NSZ19] are extended in [NSZ21] to provide a compositional framework for the construction offinite MDPs for networks of continuous-time stochastic systems via max small-gain conditions. The class ofsystems is also extended to a class of continuous-time SHS by adding Poisson processes to the dynamics, andthe construction framework of finite MDPs is generalized for a particular class of nonlinear SHS. Compositionalconstruction of finite MDPs for continuous-time stochastic systems is recently presented in [NZ20] but usinga different compositionality approach based on dissipativity conditions.Compositional construction of infinite abstractions for networks of stochastic hybrid systems under randomlyswitched topologies are proposed in [AZ18]. The proposed framework leverages the interconnection topology,switching randomly between P different interconnection topologies (it is modelled by a Markov chain), and thejoint dissipativity-type properties of subsystems and their abstractions. The abstraction itself is a stochastic , SADEGH SOUDJANI , ALESSANDRO ABATE , AND MAJID ZAMANI , hybrid system (possibly with a lower dimension) and can be used as a substitute of the original system in thecontroller design process. The work provides a closeness guarantee based on k moments similar to (8.1) butin the continuous-time setting.Compositional construction of infinite abstractions of interconnected stochastic hybrid systems via dissipativitytheory is discussed in [AZ19]. The proposed results leverage a notion of stochastic simulation function in whichthe supply rate has its own dynamics. The stochastic noises and jumps in the concrete subsystem and itsabstraction do not need to be the same. For a class of nonlinear stochastic hybrid subsystems with anincremental quadratic inequality on the nonlinearity, a set of matrix (in)equalities is established to facilitatethe construction of their abstractions together with the corresponding stochastic storage functions. Thearticle quantifies a formal error between the output behaviors of the original system and the ones of its infiniteabstractions in the form of (8.1).Compositional construction of control barrier functions for networks of continuous-time stochastic systemsis presented in [NSZ20c]. The proposed scheme is based on notions of pseudo-barrier functions (similar toDefinition 5.1 but computed over subsystems), using which one can synthesize state-feedback controllers forinterconnected systems enforcing safety specifications over a finite time horizon. This work leverages sum-typesmall-gain conditions to compositionally construct control barrier functions for interconnected systems basedon the corresponding pseudo-barrier functions computed for subsystems. Then, using the constructed controlbarrier functions, it quantifies upper bounds on the probability that an interconnected system reaches certain unsafe regions in a finite time horizon, similar to Theorem 5.3 but in the continuous-time setting. The workalso employs a systematic technique based on the SOS optimization program to search for pseudo-barrierfunctions of subsystems, while synthesizing safe controllers. Those results are extended to SHS [NSZ20b]and the class of specifications is also generalized to those that can be expressed as languages accepted bya DFA. Besides, the provided results in [NSZ20c] are only applicable to systems with polynomial dynamicsand continuous controllers, while [NSZ20b] employs an additional approach to compute control pseudo-barrierfunctions for finite input sets using the CEGIS framework, which does not require any assumptions on theunderlying dynamics and relies on SMT solvers. Temporal logic verification and synthesis.
Early work on SHS is in [GLQ06] and [KR06], which focusedon probabilistic reachability analysis.A probabilistic approach for control of continuous-time linear stochastic systems subject to LTL formulae overa set of linear predicates in the state of the system is presented in [LAB09]. The article defines a polyhedralpartition of the state space and a finite collection of controllers, represented as symbols, and constructs a finiteMDP. By utilizing an algorithm resembling LTL model checking, it determines a run satisfying the formula ina corresponding Kripke structure. A sequence of control actions in the MDP is determined to maximize theprobability of following the run.Measurability and safety verification of a class of SHS are discussed in [FHH +
11] in which the continuous-timebehaviour is given by differential equations, but discrete jumps are chosen by probability distributions. In thiswork, non-determinism is also supported, and it is exploited in an abstraction and evaluation method thatestablishes safe upper bounds on reachability probabilities.Reachability analysis for continuous-time stochastic hybrid systems with no resets is proposed in [LAB +
17] inwhich continuous dynamics described by linear stochastic differential equations. For this class of models, thearticle studies reachability (and dually, safety) properties on an abstraction defined in terms of a discrete-timeand finite-space Markov chain, with provable error bounds. The paper provides a characterization of theuniform convergence of the time discretization of stochastic processes with respect to safety properties, andthis allows to provide a complete and sound numerical procedure for reachability and safety computation overthe stochastic systems.Stochastic reachability analysis of hybrid systems is also studied in [Buj12]. This line of work continues tothese days [WBS20] and is summarized in [BL03].
UTOMATED VERIFICATION AND SYNTHESIS OF STOCHASTIC HYBRID SYSTEMS: A SURVEY 43
A new approach for the automated synthesis of safe and robust proportional-integral-derivative (PID) con-trollers for SHS is proposed in [SPB + + + Stability and Optimal control.
In this survey we do not delve into the broad issues of stability analysisand optimal control synthesis for SHS, which have been widely investigated over the past two decades. Forstability, we point the reader to the survey in [TSS14], whereas for optimal control we refer the interestedreader to the books [BLE +
06, CL06] (which covers seminal work [Dav93, ABFG + optimalcontrol by expressing logic specifications as a part of constraints in the optimization problem [LA18, HNS21]. , SADEGH SOUDJANI , ALESSANDRO ABATE , AND MAJID ZAMANI , SHS Simulation and Statistical Model Checking
In this section, we study simulation-based analysis of stochastic hybrid systems, and also encompass work onstatistical model checking (SMC). Early work, grounded on a sequential Monte Carlo simulations and the useof Petri Nets, is proposed in [BKB + x i ) ¯ Ni =1 are ¯ N i.i.d. sampled data from a set Ω. Simulation-based guarantees are presented in two-layer probabilities [CC06],as follows: P n (ˆ x i ) ¯ Ni =1 ∈ Ω : P (cid:8) ¯ y aν = ϕ (cid:9) ≤ ¯ ε o ≥ ¯ β, (9.1)where ϕ is the property of interest, ¯ y aν is any given random trajectory, and ¯ ε, ¯ β ∈ [0 ,
1] are respectively athreshold and confidence level. As a comparison with the studied approaches in the previous sections, theformal guarantee there comes with only one layer probability similar to (5.4). If one pushes the confidence ¯ β to zero, the chance constrained problem (9.1) can be understood to be similar to (5.4).A statistical model checking (SMC) algorithm to verify stochastic properties with unbounded until is presentedin [SVA05]. The algorithm is based on Monte Carlo simulation of the model and hypothesis testing of thesamples, as opposed to sequential hypothesis testing. Statistical model checking for synthesizing policieson stochastic models including finite MDPs is presented in [HMZ + δ -complete SMT procedures, which enables formal reasoning for nonlinear systems up to auser-definable numeric precision. Monte Carlo approaches for probability estimation assume that sampling ispossible for the real system at hand, however when using δ -complete simulations, one instead samples from anover-approximation of the random quantities at hand. The article introduces a Monte Carlo-SMT approach forcomputing probabilistic reachability confidence intervals that are both statistically and numerically rigorous.A survey on statistical model checking is provided in [AP18], which covers SMC algorithms, techniques, andtools, while emphasizing limitations and tradeoffs between precision and scalability.A multilevel Monte Carlo method for statistical model checking of continuous-time stochastic hybrid systems isproposed in [SMN17]. The provided approach relies on a sequence of discrete-time stochastic processes whoseexecutions approximate and converge weakly to that of the original continuous-time SHS with respect to thesatisfaction of a property of interest. With focus on bounded-horizon reachability, the paper casts the modelchecking problem as the computation of the distribution of an exit time, which is in turn formulated as theexpectation of an indicator function. This latter computation involves estimating discontinuous functionals,which reduces the bound on the convergence rate of the Monte Carlo algorithm. The work then proposes asmoothing step with tuneable precision and formally quantifies the error in the mean-square sense, which iscomposed of smoothing error, bias, and variance.A statistical simulator for hybrid Petri nets with general transitions, called HYPEG, is presented in [PER17],which combines discrete and continuous components with a possibly large number of random variables, whosestochastic behavior follows arbitrary probability distributions. HYPEG employs time-bounded discrete-event UTOMATED VERIFICATION AND SYNTHESIS OF STOCHASTIC HYBRID SYSTEMS: A SURVEY 45 simulation and well-known statistical model checking techniques to verify complex properties, including time-bounded reachability.In conclusion, statistical model checking approaches appear to be suitable for verification goals, whereas theyappear to be less efficient when synthesis is in order. The latter objective could be considered as an openproblem for future research. In addition, most of the proposed SMC results are suitable for finite-time horizons.More precisely, in the setting of SMC approaches in infinite-time horizons, the proposed results require somestrong assumptions that are not in general satisfiable by SHS. More emphasis on infinite-horizon propertiesvia SMC can be cast as another future research direction.10.
Software Tools
In this section, we discuss software tools for verification and synthesis, as well as simulation, of stochastichybrid systems.10.1.
Modest Toolset . Modest Toolset [HH14b] performs modelling and analysis for hybrid, real-time, dis-tributed and stochastic systems. At its core are models of networks of stochastic hybrid automata (SHA),which combine nondeterministic choices, continuous system dynamics, stochastic decisions and timing, andreal-time behaviour, including nondeterministic delays. The
Modest Toolset is a modular framework, support-ing as input the high-level Modest modelling language and providing a variety of analysis backends for variousspecial cases of SHA. Many existing automata-based formalisms are special cases of SHA.10.2.
SReach . SReach [WZK +
15] solves probabilistic, bounded-time reachability problems for two classes ofmodels: (i) nonlinear hybrid automata with parametric uncertainty, and (ii) probabilistic hybrid automatawith additional randomness on both transition probabilities and variable resets. Standard approaches toreachability analysis for linear hybrid systems require numerical solutions of large optimization problems,which become practically infeasible for systems involving both nonlinear dynamics and stochasticity.
SReach instead encodes stochasticity by using a set of random variables, and combines δ -complete decision proceduresand statistical tests to solve δ -reachability problems. Compared to standard simulation-based methods, SReach supports non-deterministic branching and allows one to increase the coverage of performed simulations.10.3.
ProbReach . ProbReach [SZ15] studies bounded-time reachability and other quantitative properties instochastic hybrid systems. It handles SHS with random continuous initial parameters, which encompass modelparameters or initial conditions that are chosen within the initial state and remain unchanged throughoutthe system evolution. For continuous dynamics,
ProbReach can analyze any Lipschitz-continuous differentialequations with stochastic parameters. Given an SHS with random continuous parameters and an arbitrarilysmall ǫ > ProbReach returns an interval of size not larger than ǫ containing the exact bounded-reachabilityprobability. This result is guaranteed to be numerically sound, e.g., free from floating-point inaccuracies.The introduction of discrete random parameters to the system will not affect the guarantees provided by ProbReach , however if the model features only discrete random parameters, then these guarantees do nothold: this happens because probability distributions over discrete random parameters are not continuous, andhence, an arbitrary precision cannot be provided. Introducing nondeterministic continuous parameters affectsthe guarantees the tool provides, as well: this happens because nondeterministic parameters do not have anyprobability measure. In this case,
ProbReach computes an enclosure that is guaranteed to contain all thepossible reachability probabilities. In general, such an enclosure may have size larger than ǫ > ProbReach employs a validated integration procedure to obtain a partition over the random parameters in such a waythat the guarantees described above hold. This partition is then used to enclose the probability value bycomputing under- and over-approximations. , SADEGH SOUDJANI , ALESSANDRO ABATE , AND MAJID ZAMANI , SReachTools . SReachTools [VGO19] is an open-source MATLAB toolbox for performing stochasticreachability of linear, potentially time-varying, discrete-time systems that are perturbed by a stochastic dis-turbance. More precisely, this tool addresses the problem of stochastic reachability of a target tube, which alsoencompasses terminal-time (hitting) problems, reach-avoid problems, and viability problems. The stochasticreachability of a target tube problem maximizes the likelihood that the state of a stochastic system will remainwithin a collection of time-varying target sets for a given time horizon, while respecting system dynamics andwithin a bounded control domain.
SReachTools implements several algorithms based on convex optimization,computational geometry, and Fourier transforms, to efficiently compute over- and under-approximations of thestochastic reach set.
SReachTools can be employed to perform probabilistic verification of closed-loop systems,and can also perform controller synthesis via open-loop, affine, and state-feedback controllers.
Table 1.
Comparison between features of
FAUST , StocHy , and
AMYTISS . Aspect
FAUST StocHy AMYTISS
Impl. Lang. MATLAB C++ C++/OpenCLLicense GNU GNU MITPlatform CPU CPU All platforms ( e.g.,
CPU, GPU, HWA)Algorithms Serial on HPC Serial on HPC Parallel on HPCModel SCSs: nonlinear SCS: linear, bilinear SCSs: nonlinearSpecification Safety, reachability Safety, reachability Safety, reachability, reach-avoidStochasticity Additive noise Additive noise Additive & multiplicative noisesDistribution Normal, user-defined Normal, user-defined Normal, uniform, exponential, beta, user-definedNon-deterministicdisturbance Not supported Supported Supported
Mascot-SDS . Mascot-SDS [MMS20] (the suffix SDS stands for “Stochastic Dynamical Systems”) is anopen-source tool for synthesizing controllers with formal correctness guarantees for discrete-time dynamicalsystems in the presence of stochastic perturbations. Mascot-SDS is written in C++, and is an extension ofMascot [HMMS18]. The tool supports infinite-horizon control specifications for stochastic dynamical systemsand computes over- and under-approximations of the set of states that satisfy the specification with prob-ability one. The current version of the tool is developed for “always eventually” specifications, namely forspecifications dealing with infinitely often requirements. However an upcoming new version will handle allomega-regular specifications including Linear Temporal Logic properties.10.6.
FAUST . FAUST [SGA15] generates formal abstractions for continuous-space discrete-time Markovprocesses defined over uncountable (continuous) state spaces, and performs verification and synthesis forsafety and reachability specifications. The abstract model is formally put in a relationship with the con-crete model via a user-defined maximum threshold on the approximation error introduced by the abstractionprocedure. FAUST allows exporting the abstract model to well-known probabilistic model checkers, suchas PRISM [KNP02] or Storm [DJKV17]. Alternatively, it can handle internally the computation of PCTLproperties ( e.g., safety or reachability) over the abstract model. It also allows refining the outcomes of theverification procedures over the concrete model in view of the quantified and tuneable error, which dependson the concrete dynamics and on the given formula.10.7. StocHy . StocHy [CDA19] performs quantitative stochastic analysis of discrete-time stochastic hybridsystems. The tool allows to (i) simulate the SHS evolution over a given time horizon; and to automaticallyconstruct finite abstractions of the SHS. Abstractions are then employed for (ii) formal verification or (iii)control synthesis satisfying safety and reachability specifications. The tool is implemented in C++ and employsmanipulations based on vector calculus, using sparse matrices, the symbolic construction of probabilistickernels, and multi-threading.
StocHy allows for modular modelling, and has separate simulation, verification
UTOMATED VERIFICATION AND SYNTHESIS OF STOCHASTIC HYBRID SYSTEMS: A SURVEY 47 and synthesis engines which are implemented as independent libraries. This allows for libraries to be readilyused and for extensions to be easily built.10.8.
AMYTISS . AMYTISS [LKSZ20] is developed in C++/OpenCL for designing correct-by-constructioncontrollers of large-scale discrete-time stochastic systems.
AMYTISS natively supports both additive andmultiplicative noises with different practical distributions including normal, uniform, exponential, and beta.This software tool provides scalable parallel algorithms that allow to (i) construct finite MDPs from discrete-time stochastic control systems, and (ii) synthesize controllers satisfying complex logic properties includingsafety, reachability, and reach-avoid specifications.
AMYTISS employs high-performance computing platformsand cloud-computing services to alleviate the effects of the state-explosion problem. This tool improvesperformances over computation time and memory usage by parallel execution over different heterogeneouscomputing platforms including CPUs, GPUs and hardware accelerators ( e.g.,
FPGA).
AMYTISS significantlyreduces the memory usage by setting a cutting probability threshold γ ∈ [0 ,
1] to control how many partitionelements around the mean of the system should be stored. Such an approximation allows controlling thesparsity of the columns of ˆ T x . AMYTISS also proposes another technique that further reduces the requiredmemory for computing ˆ T x as on-the-fly abstractions (OFA). In OFA, computing and storing the probabilitytransition matrix ˆ T x are skipped. Instead the required entries of ˆ T x on-the-fly are computed as they are neededfor the synthesis part via the standard dynamic programming. This reduces the required memory for ˆ T x butat the cost of repeated computation of their entries in each time step from 1 to a finite-time horizon T d .10.9. Tool Comparison.
In this subsection, we provide a comparison of
AMYTISS , FAUST and StocHy ,since they all perform similar verification tasks.
AMYTISS differs from
FAUST and StocHy in two directions.First,
AMYTISS implements novel parallel algorithms and data structures targeting high-performance com-puting (HPC) platforms to reduce the undesirable effects of the state-explosion problem. Accordingly, it isable to perform parallel execution in different heterogeneous computing platforms including CPUs, GPUs andHWAs. Whereas,
FAUST and StocHy can only run serially on one CPU. Additionally,
AMYTISS can handlethe abstraction construction and controller synthesis for two and a half player games, whereas
FAUST and StocHy only handle one and a half player games ( e.g., disturbance-free systems).
AMYTISS offers distributedexecution of parallel algorithms utilizing all available processing elements (PEs) in any heterogeneous comput-ing platform. To the best of the authors’ knowledge,
AMYTISS is the only tool of its kind for continuous-spacestochastic systems that is able to utilize all types of compute units (CUs), simultaneously.
AMYTISS alsoprovides a script that converts the constructed finite MDPs into
PRISM -input-files. In particular,
AMYTISS can natively construct finite MDPs from continuous-space stochastic control systems.
PRISM can then beemployed to perform the controller synthesis for those classes of complex specifications that
AMYTISS doesnot support.A comparison between
AMYTISS , FAUST and StocHy is provided in Table 1 in detail in terms of differenttechnical aspects. There have been some efforts in
FAUST and StocHy for parallel implementations: specif-ically,
FAUST employs some parallelization techniques using parallel for-loops and sparse matrices insideMatlab, and StocHy uses
Armadillo , a multi-threaded library for scientific computing. However, these toolsare not designed for the parallel computation on HPC platforms. Consequently, they can only utilize CPUsand cannot run on GPUs or HWAs. In comparison,
AMYTISS is developed in OpenCL, a language speciallydesigned for data-parallel tasks, and supports heterogeneous computing platforms combining CPUs, GPUsand HWAs. Note that
FAUST and StocHy do not natively support reach-avoid specifications. Implementingthis type of properties requires some modifications inside those tools.Outcomes of benchmarking competitions on tools for formal verification and policy synthesis of stochasticmodels are provided in [ABC +
18, ABC +
19, ABC + , SADEGH SOUDJANI , ALESSANDRO ABATE , AND MAJID ZAMANI , Directions for Open Research
In this subsection, we summarize and further discuss a few open topics that can be taken up as future researchinitiatives.11.1.
Formal Analysis of SHS via Learning and Data-Driven Approaches.
We discuss a few limitedworks on formal synthesis of SHS via learning and data-driven approaches, which is still considered as an opendirection. A deterministic policy gradient algorithm for reinforcement learning with continuous actions ispresented in [S + ω -regular objectives for Markov decision processes is pro-posed in [HAK19a, HPS + ω -regular properties are compiled into limit-deterministic B¨uchi automata(LDBA) instead of the traditional Rabin automata; this choice sidesteps difficulties that have marred previousproposals. [HAK19a] exploits the structure of the LDBA and shapes a synchronous reward function on-the-fly,so that an RL algorithm can synthesize a policy resulting in traces that maximize the probability of satisfyingthe linear temporal property. [HPS +
19] presents a constructive reduction from the almost-sure satisfactionof ω -regular objectives to an almost-sure reachability problem, and extends this technique to learning howto control an unknown model so that the chance of satisfying the objective is maximized. The approach in[BWZP20] proposes a reward scheme associated to the given specification that requires two discounting factorsin the reinforcement learning algorithm and provides a condition on these discounting to guarantee conver-gence of the learned policy to the optimal policy. These three approaches can be applied with off-the-shelfreinforcement learning algorithms to compute optimal strategies from the sample paths of the MDP. Exten-sions to continuous-space (and -actions) models are investigated in [LSS +
20, KS20] and in [HAK19b, HKA20],as surveyed in Section 6.A data-driven verification approach under signal temporal logic constraints is proposed in [SSZ20], basedon a framework introduced in [HHA17] for deterministic models and LTL properties. As the dynamics areparameterized and partially unknown, the framework collects data from the system and employs Bayesianinference techniques to associate a confidence value to the satisfaction of the property. The results combineboth data-driven and model-based techniques in order to have a two-layer probabilistic reasoning over thebehavior of the system: one layer is related to the stochastic noise inside the system and the next layer isrelated to the noisy data collected from the system. Approximate algorithms are also provided for computingthe confidence for linear dynamical systems.A data-driven technique for satisfying temporal properties on unknown stochastic processes with continuousspaces is recently presented in [KS20]. The proposed framework is based on reinforcement learning that isused to compute sub-optimal policies that are finite-memory and deterministic. The work addresses prop-erties expressed by LTL and uses their automaton representation to give a path-dependent reward functionmaximized via the RL algorithm. It also develops theoretical foundations characterizing the convergence ofthe learned policy to the optimal policy in the continuous space. To improve the performance of the learningon the constructed sparse reward function, the paper proposes a learning procedure based on a sequence oflabelling functions obtained from the positive normal form of the LTL specification. This procedure is utilizedto guide the RL algorithm towards the optimal policy. It is shown that the proposed approach can provideguaranteed lower bounds for the optimal satisfaction probability.11.2.
Formal Synthesis of Partially-Observed SHS.
With a few mentioned exceptions, most of thesurveyed work on automated verification and synthesis of SHS assumes complete state information. However,in many real applications we do not have access to full information. There have been a limited work on formalsynthesis of partially-observed SHS. An early formulation is put forward in [DAT13], which characterizes the
UTOMATED VERIFICATION AND SYNTHESIS OF STOCHASTIC HYBRID SYSTEMS: A SURVEY 49 safety problem measure-theoretically and develops an application in air traffic management. Reachabilityanalysis of partially observable discrete-time SHS is proposed in [LO14]. The proposed framework includes achange of measure that simplifies the distribution of the sufficient statistic conditioned on its previous value.A dynamic programming recursion is also developed for the solution of the equivalent perfect informationproblem, proving that the recursion is valid, an optimal solution exists, and results in the same solution as tothe original problem.A finite-state approximation for safety verification and control of partially observable SHS is presentedin [LO15b, LO16]. The papers solve a dynamic program over the finite state approximation to generatea lower bound to the viability probability, using a point-based method that generates samples of the informa-tion state. The proposed approach produces approximate probabilistic viable sets and synthesizes a controllerto satisfy safety specifications. It also provides error bounds and convergence results, assuming additive Gauss-ian noise in the continuous-state dynamics and observations. Computing probabilistic viable sets for partiallyobservable systems using truncated Gaussians and adaptive gridding is presented in [LO15a].A perception-aware point-based value iteration for POMDPs is presented in [GT19]. The approach avoidscombinatorial expansion over the action space from the integration of planning and perception decisions,through a greedy strategy for observation selection that minimizes an information-theoretic measure of thestate uncertainty. The article develops a point-based value iteration algorithm that incorporates this greedystrategy to pick perception actions for each sampled belief point in each iteration. A sequential decision makingprocess using POMDPs is studied in [WABT19]. The work aims to find strategies that actively interact withthe dynamical system, and observe its reactions so that the true model is determined efficiently and with highconfidence.Verification of uncertain POMDPs using barrier certificates is discussed in [ACJT18]. A class of POMDPsis considered with uncertain transition and/or observation probabilities in which the uncertainty takes theform of probability intervals. Given an uncertain POMDP representation of the autonomous agent, the maingoal is to propose a method for checking whether the system will satisfy an optimal performance, while notviolating a safety requirement. A policy synthesis in multi-agent POMDPs via discrete-time barrier functionsto enforce safety is proposed in [ASBA19]. The method is implemented online by a sequence of one-step greedyalgorithms as a standalone safe controller or as a safety-filter given a nominal planning policy.Synthesis of stochastic systems with partial state information via control barrier functions is proposed in [JJZ20b,JJZ20a], as surveyed in Section 8.
Open Problem 9.
Formal synthesis of POMDPs (even with finite set of states) is a hard problem and theavailable methods are not scalable. Developing compositional techniques for POMDPs to make the synthesisproblem more scalable is a potential future research direction. Generalizing the available work on games [DKS + is an additional desideratum. Secure-by-Construction Controller Synthesis.
Security vulnerabilities related to information leaksin complex systems, such as SHS, can be difficult to discover and mitigate, as the interaction between theembedded control software and the physical environment exposes numerous attack surfaces for maliciousexploitation. An important class of security properties is called opacity : opacity is a confidentiality propertyin security analysis, which characterizes whether or not some “secret” information about the system ( e.g., a defender) can be inferred by outside observers with potentially malicious intentions ( e.g., intruders orattackers). There has been recent work with the goal to synthesize controllers that not only satisfy complexLTL specifications, but also fulfill some security properties. In this respect, probabilistic opacity for Markovdecision processes is proposed in [BCS15], where non-deterministic choices are combined with probabilistictransitions, and where related decidability problems with partial or complete observation hypotheses for theschedulers are studied. The article proves that all questions are decidable with complete observations and ω -regular secrets. On the other hand, with partial observations, it is shown that all quantitative questions , SADEGH SOUDJANI , ALESSANDRO ABATE , AND MAJID ZAMANI , are undecidable. However, the question whether a system is almost surely non-opaque becomes decidable fora restricted class of ω -regular secrets, as well as for all ω -regular secrets under finite-memory schedulers.Notions of infinite-step and K -step opacity for stochastic discrete-event systems are presented in [YLWL17,YLWL19]. These works propose a new notion of opacity, called almost infinite-step opacity (respectively, al-most K -step opacity), to capture whether or not the probability of violating infinite-step opacity (respectively, K -step opacity) is smaller than a given threshold. They also provide an effective algorithm for the verificationof almost infinite-step and K -step opacity.A notion of approximate initial-state opacity for discrete-time stochastic control systems is discussed in [LYZ20].The work first proposes a notion of approximate initial-state opacity for stochastic control systems in orderto quantitatively evaluate the security guarantee. It then introduces a notion of opacity-preserving stochasticsimulation functions to quantify the distance between two systems in a probabilistic setting based on (2.6),while preserving approximate initial-state opacity across them. Open Problem 10.
The results in [LYZ20] are the few dealing with opacity verification of continuous-space stochastic control systems. However, to the best of our knowledge, there is still no procedure onsecure-by-construction controller synthesis of SHS by considering privacy properties in addition to temporallogic ones in the design phase: this stands as an interesting open research direction. (Mix)-monotonicity of SHS.
As discussed in Subsection 4.3, the construction of IMC/IMDP can bemore complex than standard abstractions based on MC/MDP, since one needs to compute lower and upperbounds for the probabilities of transition between states, rather than computing just a single number as instandard MCs/MDPs. As an alternative, under some assumptions ( e.g., additive stochasticity, unimodal noisedistribution, independent noises affecting different states), contributions in [DC18, DC19, DHC20] utilize themix-monotonicity property of the deterministic part of the map f and propose an approach to compute thoselower and upper bounds in an efficient way. Here, we suggest an appropriate definition of mix-monotonicityin the stochastic setting that can be useful in constructing IMDPs. Definition 11.1.
A stochastic system Σ with ν ≡ (i.e., an autonomous model) is called mix-monotone ifthere exist functions g : X × X → X and g : X × X → X such that: x ( k ) ∈ [ a , b ] = ⇒ z ≤ P (cid:8) x ( k + 1) ∈ [ a , b ] (cid:12)(cid:12) x ( k ) (cid:9) ≤ z , where, z = P (cid:8) x ( k + 1) ∈ [ a , b ] (cid:12)(cid:12) x ( k ) = g ( a , b ) (cid:9) ,z = P (cid:8) x ( k + 1) ∈ [ a , b ] (cid:12)(cid:12) x ( k ) = g ( a , b ) (cid:9) . If the mix-monotonicity property in Definition 11.1 holds, one can actually construct IMCs/IMDPs moreefficiently. This leads to the following outstanding problem.
Open Problem 11.
It is desirable to investigate which classes of systems satisfy the property in Defini-tion 11.1, as well as alternative ways of describing monotonocity for stochastic systems.
Compositional Construction of IMCs/IMDPs.
Since constructing IMCs/IMDPs is more complexthan standard abstractions, as discussed in Subsection 4.3, a promising approach to mitigate the related com-putational complexity is to develop compositional techniques: these might allow constructing IMCs/IMDPsof high-dimensional systems based on IMCs/IMDPs for smaller subsystems.11.6.
Compositional Controller Synthesis for SHS.
In this survey, we mainly discussed different compo-sitional approaches for the construction of (in)finite abstractions for networks of stochastic systems. Potentialfuture work concerns the investigation of compositional controller synthesis for stochastic hybrid systems.In particular, given a specification over the interconnected system, it is of interest to find a formal relation
UTOMATED VERIFICATION AND SYNTHESIS OF STOCHASTIC HYBRID SYSTEMS: A SURVEY 51 between the satisfaction probabilities provided by local controllers for individual subsystems, as well as theoptimal satisfaction probability for the specification on the monolithic (global) system.11.7.
Extensions and Development of Software Tools.
Developing efficient software tools based on thetheoretical and algorithmic results is essential for automated verification and synthesis of SHS. Most of thetools discussed in Section 10 are developed for discrete-time SHS. As such, a potential direction of useful workis to develop software tools for continuous-time
SHS. Moreover, developing software tools to handle infinite-horizon specifications is another promising extension. In addition, there is no tool at the moment that handlesthe construction of finite MDPs in a compositional way : further developing software tools for compositionalpurposes is therefore of interest.A comprehensive and up-to-date discussion about different software tools together with their potential direc-tions of extension can be found in [ABC +
20] and available at https://bit.ly/3nGechr.12.
Closing Discussion
In this article, we have provided the first survey of work on automated formal verification and control synthesisof stochastic hybrid systems. We have focused on most recent and sharpest results, and for the sake of a clearand streamlined presentation we have presented selected analysis methods, applications, and results in detail,and instead briefly overviewed alternative approaches. We have distinguished approaches as discretization-based and -free, and have investigated four different closeness guarantees between a concrete SHS modeland its abstractions. We have discussed different problems including stochastic similarity relations, infiniteand finite abstractions, control barrier certificates, temporal logic verification and synthesis, compositionaltechniques, continuous-time stochastic models, and overviewed existing software tools that implement thediscussed approaches. Throughout this survey, we have also added the discussion of a few open problems.We hope that this survey article provides an introduction to the foundations of SHS, towards an easierunderstanding of many challenges and existing solutions related to formal verification and control synthesis ofthese models, together with the associated software tools.
References [AAP +
06] S. Amin, A. Abate, M. Prandini, J. Lygeros, and S. Sastry. Reachability analysis for controlled discrete timestochastic hybrid systems. In
Proceedings of HSCC06, LNCS 3927 , pages 49–63. Springer Verlag, 2006.[Aba09] A. Abate. A contractivity approach for probabilistic bisimulations of diffusion processes. In
Proceedings of the 48thIEEE Conference of Decision and Control , pages 2230–2235, 2009.[Aba10] A. Abate. Probabilistic bisimulations of switching and resetting diffusions. In
Proceedings of the 49th IEEE Con-ference of Decision and Control , pages 5918–5923, 2010.[Aba13] A. Abate. Approximation metrics based on probabilistic bisimulations for general state-space Markov processes: asurvey.
Electronic Notes in Theoretical Computer Science , 297:3–25, 2013.[ABC +
18] Alessandro Abate, Henk Blom, Nathalie Cauchi, Sofie Haesaert, Arnd Hartmanns, Kendra Lesser, Meeko Oishi, Vi-gnesh Sivaramakrishnan, Sadegh Soudjani, Cristian-Ioan Vasile, and Abraham P. Vinod. ARCH-COMP18 categoryreport: Stochastic modelling. In , volume 54 of
EPiC Series in Computing , pages 71–103, 2018.[ABC +
19] Alessandro Abate, Henk Blom, Nathalie Cauchi, Kurt Degiorgio, Martin Fraenzle, Ernst Moritz Hahn, Sofie Hae-saert, Hao Ma, Meeko Oishi, Carina Pilch, Anne Remke, Mahmoud Salamati, Sadegh Soudjani, Birgit van Hui-jgevoort, and Abraham Vinod. ARCH-COMP19 category report: Stochastic modelling. In , volume 61 of
EPiC Series in Computing , pages62–102, 2019.[ABC +
20] Alessandro Abate, Henk Blom, Nathalie Cauchi, Joanna Delicaris, Arnd Hartmanns, Mahmoud Khaled, AbolfazlLavaei, Carina Pilch, Anne Remke, Stefan Schupp, Fedor Shmarov, Sadegh Soudjani, Abraham Vinod, Ben Wood-ing, Majid Zamani, and Paolo Zuliani. ARCH-COMP20 category report: Stochastic models. In , volume 74 of
EPiC Series inComputing , pages 76–106, 2020. , SADEGH SOUDJANI , ALESSANDRO ABATE , AND MAJID ZAMANI , [ABFG +
93] Aristotle Arapostathis, Vivek S. Borkar, Emmanuel Fern´andez-Gaucherand, Mrinal K. Ghosh, and Steven I. Mar-cus. Discrete-time controlled Markov processes with average cost criterion: A survey.
SIAM J. Control Optim. ,31(2):282–344, 1993.[ACJT18] M. Ahmadi, M. Cubuktepe, N. Jansen, and U. Topcu. Verification of uncertain POMDPs using barrier certificates.In
Proceedings of the Annual Allerton Conference on Communication, Control, and Computing , pages 115–122,2018.[ADB11] A. Abate, A. D’Innocenzo, and M.D. Di Benedetto. Approximate abstractions of stochastic hybrid systems.
IEEETransactions on Automatic Control , 56(11):2688–2694, 2011.[AJZ19] M Anand, P Jagtap, and M Zamani. Verification of switched stochastic systems via barrier certificates. In
Proceedingsof the 58th IEEE Conference on Decision and Control, to appear , pages 4373–4378, 2019.[AKLP10] A. Abate, J.P. Katoen, J. Lygeros, and M. Prandini. Approximate model checking of stochastic hybrid systems.
European Journal of Control , 16(6):624–641, 2010.[AKNP14] Al. Abate, M. Kwiatkowska, G. Norman, and D. Parker. Probabilistic model checking of labelled Markov processesvia finite approximate bisimulations. In
Horizons of the Mind. A Tribute to Prakash Panangaden , pages 40–58.Springer, 2014.[ALZ20] M. Anand, A. Lavaei, and M. Zamani. Compositional construction of control barrier certificates for large-scaleinterconnected stochastic systems. In
Proceedings of the 21st IFAC World Congress, to appear , 2020.[ALZ21] M. Anand, A. Lavaei, and M. Zamani. From small-gain theory to compositional construction of barrier certificatesfor large-scale stochastic systems. arXiv:2101.06916 , 2021.[AMP16] M. Arcak, Ch. Meissen, and A. Packard.
Networks of dissipative systems: compositional certification of stability,performance, and safety . Springer, 2016.[AP18] G. Agha and K. Palmskog. A survey of statistical model checking.
ACM Transactions on Modeling and ComputerSimulation (TOMACS) , 28(1):1–39, 2018.[APLS08] A. Abate, M. Prandini, J. Lygeros, and S. Sastry. Probabilistic reachability and safety for controlled discrete-timestochastic hybrid systems.
Automatica , 44(11):2724–2734, 2008.[ASBA19] M. Ahmadi, A. Singletary, J. W. Burdick, and A. D. Ames. Safe policy synthesis in multi-agent POMDPs viadiscrete-time barrier functions. In
Proceedings of the 58th Conference on Decision and Control (CDC) , pages 4797–4803, 2019.[AZ18] A. U. Awan and M. Zamani. Compositional abstractions of networks of stochastic hybrid systems under randomlyswitched topologies. In
Proceedings of the American Control Conference (ACC) , pages 1586–1591, 2018.[AZ19] A. U. Awan and M. Zamani. From dissipativity theory to compositional abstractions of interconnected stochastichybrid systems.
IEEE Transactions on Control of Network Systems , 7(1):433–445, 2019.[BA17] G. Bian and A. Abate. On the relationship between bisimulation and trace equivalence in an approximate probabilis-tic context. In
Proceedings of the International Conference on Foundations of Software Science and ComputationStructures , pages 321–337, 2017.[BBL +
19] G. Bacci, G. Bacci, K. G. Larsen, R. Mardare, Q. Tang, and F. van Breugel. Computing probabilistic bisimilaritydistances for probabilistic automata. arXiv:1907.01768 , 2019.[BCS15] B. B´erard, K. Chatterjee, and N. Sznajder. Probabilistic opacity for Markov decision processes.
Information Pro-cessing Letters , 115(1):52–59, 2015.[BEOB14] Marc Bouissou, Hilding Elmqvist, Martin Otter, and Albert Benveniste. Efficient Monte Carlo simulation of sto-chastic hybrid systems. In
Proceedings of the 10th International Modelica Conference , 2014.[BK08] C. Baier and J.-P. Katoen.
Principles of model checking . MIT press, 2008.[BKB +
07] H. AP. Blom, J. Krystul, G.J. Bakker, M. B. Klompstra, and B. K. Obbink. Free flight collision risk estimation bysequential MC simulation. pages 249–281, 2007.[BL03] M. L. Bujorianu and J. Lygeros. Reachability questions in piecewise deterministic Markov processes. In
Proceedingsof the 6th International Workshop on Hybrid Systems: Computation and Control , volume 2623 of
Lecture Notes inComputer Science , pages 126–140. Springer, 2003.[BLE +
06] H. AP. Blom, J. Lygeros, M. Everdij, S. Loizou, and K. Kyriakopoulos.
Stochastic hybrid systems: theory and safetycritical applications , volume 337. Springer, 2006.[BS96] D. P. Bertsekas and S. E. Shreve.
Stochastic optimal control: The discrete-time case . Athena Scientific, 1996.[BSB13] H. AP. Blom, S. H. Stroeve, and T. Bosse. Modelling of potential hazards in agent-based safety risk analysis. In
Proceedings of the 10th USA/Europe Air Traffic Management Research and Development Seminar , 2013.[Buj12] L. M. Bujorianu.
Stochastic reachability analysis of hybrid systems . Springer Science & Business Media, 2012.[BWZP20] A. Bozkurt, Yu Wang, Michael M. Zavlanos, and M. Pajic. Control synthesis from linear temporal logic specificationsusing model-free reinforcement learning. ,pages 10349–10355, 2020.[BYG17] Calin Belta, Boyan Yordanov, and Ebru Aydin Gol.
Formal Methods for Discrete-Time Dynamical Systems , vol-ume 89 of
Studies in Systems, Decision and Control . Springer, 2017.[CA15] S. Coogan and M. Arcak. Efficient finite abstraction of mixed monotone systems. In
Proceedings of the 18th Inter-national Conference on Hybrid Systems: Computation and Control , pages 58–67, 2015.
UTOMATED VERIFICATION AND SYNTHESIS OF STOCHASTIC HYBRID SYSTEMS: A SURVEY 53 [CA18] N. Cauchi and A. Abate. Benchmarks for cyber-physical systems: A modular model library for building automationsystems. In
Proceedings of ADHS , pages 49–54, 2018.[CC06] G. C. Calafiore and M. C. Campi. The scenario approach to robust control design.
IEEE Transactions on AutomaticControl , 51(5):742–753, 2006.[CDA19] N. Cauchi, K. Degiorgio, and A. Abate.
StocHy : Automated verification and synthesis of stochastic processes. In
TACAS’19 , Lecture Notes in Computer Science, pages 247–264. Springer, 2019.[CG04] F. Ciesinski and M. Gr¨oßer. On probabilistic computation tree logic. In
Validation of Stochastic Systems , pages147–188. Springer, 2004.[CGSS13] A. Cimatti, A. Griggio, B. J. Schaafsma, and R. Sebastiani. The MathSAT5 SMT solver. In
Tools and Algorithmsfor the Construction and Analysis of Systems , Lecture Notes in Computer Science, pages 93–107, 2013.[CL06] C. G. Cassandras and J. Lygeros.
Stochastic hybrid systems . CRC Press, 2006.[Cla19] A. Clark. Control barrier functions for complete and incomplete information stochastic systems. In
Proceedings ofthe American Control Conference (ACC) , pages 2928–2935, 2019.[Cla20] A. Clark. Control barrier functions for stochastic systems. arXiv:2003.03498 , 2020.[CLL +
19] N. Cauchi, L. Laurenti, M. Lahijanian, A. Abate, M. Kwiatkowska, and L. Cardelli. Efficiency through uncertainty:Scalable formal synthesis for stochastic hybrid systems. In
Proceedings of the 22nd ACM International Conferenceon Hybrid Systems: Computation and Control , pages 240–251, 2019.[CS13] A. Chakarov and S. Sankaranarayanan. Probabilistic program analysis with martingales. In
Proceedings of theInternational Conference on Computer Aided Verification , pages 511–526, 2013.[DAK12] A. D’Innocenzo, A. Abate, and J.P. Katoen. Robust PCTL model checking. In
Proceedings of the 15th ACMinternational conference on Hybrid Systems: Computation and Control , pages 275–286, 2012.[DAT13] J. Ding, A. Abate, and C. Tomlin. Optimal control of partially observable discrete time stochastic hybrid systemsfor safety specifications. In
Proceedings of the 2013 American Control Conference , pages 6231–6236, 2013.[Dav93] M. H. A. Davis.
Markov models and optimization , volume 49 of
Monographs on Statistics and Applied Probability .Chapman & Hall, London, 1993.[DC18] Maxence Dutreix and Samuel Coogan. Efficient verification for stochastic mixed monotone systems. In
Proceedingsof the 9th International Conference on Cyber-Physical Systems (ICCPS) , pages 150–161, 2018.[DC19] M. Dutreix and S. Coogan. Specification-guided verification and abstraction refinement of mixed monotone stochasticsystems. arXiv:1903.02191 , 2019.[DGJP04] J. Desharnais, V. Gupta, R. Jagadeesan, and P. Panangaden. Metrics for labelled Markov processes.
Theoreticalcomputer science , 318(3):323–354, 2004.[DHC20] M. Dutreix, J. Huh, and S. Coogan. Abstraction-based synthesis for stochastic systems with omega-regular objec-tives. arXiv:2001.09236 , 2020.[DJKV17] C. Dehnert, S. Junges, J.-P. Katoen, and M. Volk. A storm is coming: A modern probabilistic model checker. In
Proceedings of the 29th International Conference on Computer Aided Verification (CAV) , volume 10427 of
LectureNotes in Computer Science , pages 592–600. Springer, 2017.[DKS +
13] J. Ding, M. Kamgarpour, S. Summers, A. Abate, J. Lygeros, and C. Tomlin. A stochastic games framework forverification and control of discrete time stochastic hybrid systems.
Automatica , 49(9):2665–2674, 2013.[DLT08] J. Desharnais, F. Laviolette, and M. Tracol. Approximate analysis of probabilistic processes: Logic, simulation andgames. In
Proceedings of the 5th international conference on quantitative evaluation of system , pages 264–273, 2008.[DMB08] L. De Moura and N. Bjørner. Z3: An efficient SMT solver. In
Proceedings of the International conference on Toolsand Algorithms for the Construction and Analysis of Systems , pages 337–340, 2008.[DRW10] S. N Dashkovskiy, B. S. R¨uffer, and F. R. Wirth. Small gain theorems for large scale systems and construction ofISS Lyapunov functions.
SIAM Journal on Control and Optimization , 48(6):4089–4118, 2010.[FHH +
11] M. Fr¨anzle, E. M. Hahn, H. Hermanns, N. Wolovick, and L. Zhang. Measurability and safety verification forstochastic hybrid systems. In
Proceedings of the 14th international conference on hybrid systems: computation andcontrol , pages 43–52, 2011.[FI04] A. Fehnker and F. Ivanˇci´c. Benchmarks for hybrid systems verification. In
International Workshop on HybridSystems: Computation and Control , pages 326–341. Springer, 2004.[FT15] J. Fu and U. Topcu. Computational methods for stochastic control with metric interval temporal logic specifications.In
Proceedings of the 54th IEEE Conference on Decision and Control (CDC) , pages 7440–7447, 2015.[GAC12] S. Gao, J. Avigad, and E. M. Clarke. δ -complete decision procedures for satisfiability over the reals. In AutomatedReasoning , Lecture Notes in Computer Science, pages 286–300, 2012.[GLQ06] Y. Gao, J. Lygeros, and M. Quincampoix. The reachability problem for uncertain hybrid systems revisited: Aviability theory perspective. In
Proceedings of the 9th International Workshop on Hybrid Systems: Computationand Control , volume 3927 of
Lecture Notes in Computer Science , pages 242–256. Springer, 2006.[GT19] M. Ghasemi and U. Topcu. Perception-aware point-based value iteration for partially observable Markov decisionprocesses. In
Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI) , pages 2371–2377, 2019. , SADEGH SOUDJANI , ALESSANDRO ABATE , AND MAJID ZAMANI , [GVO17] J. D. Gleason, A. P. Vinod, and M. MK. Oishi. Underapproximation of reach-avoid sets for discrete-time stochasticsystems via Lagrangian methods. In Proceedings of the 56th Conference on Decision and Control , pages 4283–4290,2017.[HAK19a] M. Hasanbeig, A. Abate, and D. Kroening. Certified reinforcement learning with logic guidance. arXiv:1902.00778 ,2019.[HAK19b] Mohammadhosein Hasanbeig, Alessandro Abate, and Daniel Kroening. Logically-Constrained Neural Fitted Q-Iteration. In
Proceedings of the 18th International Conference on Autonomous Agents and Multi-Agent Systems(AAMAS) , pages 2012–2014, 2019.[Har06] D. J. Hartfiel.
Markov set-chains . Springer, 2006.[HCA17] S. Haesaert, N. Cauchi, and A. Abate. Certified policy synthesis for general Markov decision processes: An applica-tion in building automation systems.
Performance Evaluation , 117:75–103, 2017.[HCL +
17] C. Huang, X. Chen, W. Lin, Z. Yang, and X. Li. Probabilistic safety verification of stochastic hybrid systems usingbarrier certificates.
ACM Transactions on Embedded Computing Systems (TECS) , 16(5s):186, 2017.[HH14a] P. Hall and C. C. Heyde.
Martingale limit theory and its application . Academic press, 2014.[HH14b] A. Hartmanns and H. Hermanns. The modest toolset: An integrated environment for quantitative modelling andverification. In
Proceedings of the International Conference on Tools and Algorithms for the Construction andAnalysis of Systems , pages 593–598, 2014.[HHA17] S. Haesaert, P.M.J. V.d. Hof, and A. Abate. Data-driven and model-based verification via Bayesian identificationand reachability analysis.
Automatica , 79(5):115–126, 2017.[HKA +
19] M. Hasanbeig, Y. Kantaros, A. Abate, D. Kroening, G. J. Pappas, and I. Lee. Reinforcement Learning for Tempo-ral Logic Control Synthesis with Probabilistic Satisfaction Guarantees. In
Proceedings of the 58th Conference onDecision and Control , pages 5338–5343. IEEE, 2019.[HKA20] Mohammadhosein Hasanbeig, Daniel Kroening, and Alessandro Abate. Deep reinforcement learning with temporallogics. In
FORMATS , pages 1–22. Springer LNCS 12288, 2020.[HLL96] On´esimo Hern´andez-Lerma and J. B. Lasserre.
Discrete-time Markov control processes . Appl. Math. 30, Springe,New York, 1996.[HLS00] Jianghai Hu, John Lygeros, and Shankar Sastry. Towars a theory of stochastic hybrid systems. In
Proceedings ofthe Third International Workshop on Hybrid Systems: Computation and Control , volume 1790 of
Lecture Notes inComputer Science , pages 160–173. Springer, 2000.[HMMS18] K. Hsu, R. Majumdar, K. Mallik, and A.-K. Schmuck. Multi-layered abstraction-based controller synthesis forcontinuous-time systems. In
Proceedings of the 21st International Conference on Hybrid Systems: Computationand Control , pages 120–129, 2018.[HMZ +
12] D. Henriques, J. G. Martins, P. Zuliani, A. Platzer, and E. M. Clarke. Statistical model checking for Markov decisionprocesses. In
Proceedings of the 9th international conference on quantitative evaluation of systems , pages 84–93,2012.[HNS21] S. Haesaert, P. Nilsson, and S. Soudjani. Formal multi-objective synthesis of continuous-state MDPs.
IEEE ControlSystems Letters , 5(5):1765–1770, 2021.[HPS +
19] E. M. Hahn, M. Perez, S. Schewe, F. Somenzi, A. Trivedi, and D. Wojtczak. Omega-regular objectives in model-freereinforcement learning. In
Proceedings of the International Conference on Tools and Algorithms for the Constructionand Analysis of Systems , pages 395–412, 2019.[HS18] S. Haesaert and S. Soudjani. Robust dynamic programming for temporal logic control of stochastic systems.
CoRR ,abs/1811.11445, 2018.[HSA17] Sofie Haesaert, Sadegh Soudjani, and Alessandro Abate. Verification of general Markov decision processes by ap-proximate similarity relations and policy refinement.
SIAM Journal on Control and Optimization , 55(4):2333–2367,2017.[HSA18] S. Haesaert, S. Soudjani, and A. Abate. Temporal logic control of general Markov decision processes by approximatepolicy refinement.
IFAC-PapersOnLine , 51(16):73–78, 2018.[JJZ20a] N. Jahanshahi, P. Jagtap, and M. Zamani. Synthesis of partially observed jump-diffusion systems via control barrierfunctions.
IEEE Control Systems Letters , 5(1):253–258, 2020.[JJZ20b] N. Jahanshahi, P. Jagtap, and M. Zamani. Synthesis of stochastic systems with partial information via controlbarrier functions. In
Proceedings of the 21st IFAC World Congress, to appear , 2020.[JP09] A. A. Julius and G. J. Pappas. Approximations of stochastic hybrid systems.
IEEE Transactions on AutomaticControl , 54(6):1193–1203, 2009.[JSZ18] P. Jagtap, S. Soudjani, and M. Zamani. Temporal logic verification of stochastic systems using barrier certificates. In
Proceedings of the International Symposium on Automated Technology for Verification and Analysis , pages 177–193,2018.[JSZ20] P. Jagtap, S. Soudjani, and M. Zamani. Formal synthesis of stochastic systems via control barrier certificates.
IEEETransactions on Automatic Control, DOI: 10.1109/TAC.2020.3013916 , 2020.[Jun20] S. Junges.
Parameter Synthesis in Markov Models . PhD thesis, RWTH Aachen University, Germany, 2020.[Kal97] O. Kallenberg.
Foundations of modern probability . Springer-Verlag, New York, 1997.
UTOMATED VERIFICATION AND SYNTHESIS OF STOCHASTIC HYBRID SYSTEMS: A SURVEY 55 [KDS +
11] M. Kamgarpour, J. Ding, S. Summers, A. Abate, J. Lygeros, and C. Tomlin. Discrete time stochastic hybriddynamical games: Verification & controller synthesis. In
Proceedings of the 50th IEEE Conference on Decision andControl and European Control Conference , pages 6122–6127, 2011.[KNP02] M. Kwiatkowska, G. Norman, and D. Parker. PRISM: Probabilistic symbolic model checker. In
Proceedings ofthe International Conference on Modelling Techniques and Tools for Computer Performance Evaluation , pages200–204, 2002.[KNP11] M. Kwiatkowska, G. Norman, and D. Parker. Prism 4.0: Verification of probabilistic real-time systems. In
Interna-tional conference on computer aided verification , pages 585–591. Springer, 2011.[KNPQ13] M. Kwiatkowska, G. Norman, D. Parker, and H. Qu. Compositional probabilistic verification through multi-objectivemodel checking.
Information and Computation , 232:38–65, 2013.[KR06] X. D. Koutsoukos and D. Riley. Computational methods for reachability analysis of stochastic hybrid systems. In
Proceedings of the 9th International Workshop on Hybrid Systems: Computation and Control , volume 3927 of
Lecture Notes in Computer Science , pages 377–391. Springer, 2006.[KS20] Milad Kazemi and Sadegh Soudjani. Formal policy synthesis for continuous-space systems via reinforcement learning.In
International Conference on integrated Formal Methods (iFM20) , 2020.[KSL13] M. Kamgarpour, S. Summers, and J. Lygeros. Control design for specifications on stochastic hybrid systems. In
Proceedings of the 16th international conference on hybrid systems: computation and control , pages 303–312, 2013.[Kus67] H. J. Kushner.
Stochastic Stability and Control . Mathematics in Science and Engineering. Elsevier Science, 1967.[KV01] O. Kupferman and M. Y. Vardi. Model checking of safety properties.
Formal Methods in System Design , 19(3):291–314, 2001.[LA18] K. Lesser and A. Abate. Multi-objective optimal control with safety as a priority.
IEEE Transactions on ControlSystems Technology , 26(3):1015–1027, 2018.[LAB09] M. Lahijanian, S. B. Andersson, and C. Belta. A probabilistic approach for control of a stochastic system from LTLspecifications. In
Proceedings of the 48h IEEE Conference on Decision and Control (CDC) held jointly with 28thChinese Control Conference , pages 2236–2241, 2009.[LAB12] M. Lahijanian, S. B. Andersson, and C. Belta. Approximate Markovian abstractions for linear stochastic systems.In
Proceedings of the 51st IEEE Conference on Decision and Control (CDC) , pages 5966–5971, 2012.[LAB15] M. Lahijanian, S. B. Andersson, and C. Belta. Formal verification and synthesis for discrete-time stochastic systems.
IEEE Transactions on Automatic Control , 60(8):2031–2045, 2015.[LAB +
17] L. Laurenti, A. Abate, L. Bortolussi, L. Cardelli, M. Ceska, and M. Kwiatkowska. Reachability computation forswitching diffusions: Finite abstractions with certifiable and tuneable precision. In
Proceedings of the 20th ACMInternational Conference on Hybrid Systems: Computation and Control , pages 55–64, 2017.[Lav19] A. Lavaei.
Automated Verification and Control of Large-Scale Stochastic Cyber-Physical Systems: CompositionalTechniques . PhD thesis, Department of Electrical Engineering, Technische Universit¨at M¨unchen, Germany, 2019.[LKSZ20] A. Lavaei, M. Khaled, S. Soudjani, and M. Zamani.
AMYTISS : Parallelized automated controller synthesis for large-scale stochastic systems. In
Proceedings of the 32nd International Conference on Computer-Aided Verification(CAV), Lecture Notes in Computer Science 12225 , pages 461–474, 2020.[LLA +
20] L. Laurenti, M. Lahijanian, A. Abate, L. Cardelli, and M. Kwiatkowska. Formal and efficient synthesis for continuous-time linear stochastic hybrid processes.
IEEE Transactions on Automatic Control , 2020.[LLT +
18] Y.-J. Liu, S. Lu, S. Tong, X. Chen, C. P. Chen, and D.-J. Li. Adaptive control-based barrier Lyapunov functionsfor a class of stochastic nonlinear systems with full state constraints.
Automatica , 87:83–93, 2018.[LO14] K. Lesser and M. Oishi. Reachability for partially observable discrete time stochastic hybrid systems.
Automatica ,50(8):1989–1998, 2014.[LO15a] K. Lesser and M. Oishi. Computing probabilistic viable sets for partially observable systems using truncated gaus-sians and adaptive gridding. In
Proceedings of the American Control Conference (ACC) , pages 1505–1512, 2015.[LO15b] K. Lesser and M. Oishi. Finite state approximation for verification of partially observable stochastic hybrid systems.In
Proceedings of the 18th International Conference on Hybrid Systems: Computation and Control , pages 159–168,2015.[LO16] K. Lesser and M. Oishi. Approximate safety verification and control of partially observable stochastic hybrid systems.
IEEE Transactions on Automatic Control , 62(1):81–96, 2016.[LS91] K. G. Larsen and A. Skou. Bisimulation through probabilistic testing.
Information and computation , 94(1):1–28,1991.[LSMZ17] A. Lavaei, S. Soudjani, R. Majumdar, and M. Zamani. Compositional abstractions of interconnected discrete-timestochastic control systems. In
Proceedings of the 56th IEEE Conference on Decision and Control , pages 3551–3556,2017.[LSS +
20] A. Lavaei, F. Somenzi, S. Soudjani, A. Trivedi, and M. Zamani. Formal controller synthesis for continuous-spacemdps via model-free reinforcement learning. In
Proceedings of the 11th ACM/IEEE International Conference onCyber-Physical Systems (ICCPS) , pages 98–107. IEEE, 2020.[LSZ18a] A. Lavaei, S. Soudjani, and M. Zamani. Compositional synthesis of finite abstractions for continuous-space stochasticcontrol systems: A small-gain approach.
Proceedings of the 6th IFAC Conference on Analysis and Design of HybridSystems , 51(16):265–270, 2018. , SADEGH SOUDJANI , ALESSANDRO ABATE , AND MAJID ZAMANI , [LSZ18b] A. Lavaei, S. Soudjani, and M. Zamani. From dissipativity theory to compositional construction of finite Markovdecision processes. In Proceedings of the 21st ACM International Conference on Hybrid Systems: Computation andControl , pages 21–30, 2018.[LSZ19a] A. Lavaei, S. Soudjani, and M. Zamani. Approximate probabilistic relations for compositional synthesis of stochasticsystems. In
Proceedings of the Numerical Software Verification , pages 101–109, 2019. Lecture Notes in ComputerScience 11652.[LSZ19b] A. Lavaei, S. Soudjani, and M. Zamani. Compositional construction of infinite abstractions for networks of stochasticcontrol systems.
Automatica , 107:125–137, 2019.[LSZ19c] A. Lavaei, S. Soudjani, and M. Zamani. Compositional synthesis of not necessarily stabilizable stochastic systemsvia finite abstractions. In
Proceedings of the 18th European Control Conference , pages 2802–2807, 2019.[LSZ20a] A. Lavaei, S. Soudjani, and M. Zamani. Compositional abstraction-based synthesis for networks of stochasticswitched systems.
Automatica , 114, 2020.[LSZ20b] A. Lavaei, S. Soudjani, and M. Zamani. Compositional abstraction-based synthesis of general MDPs via approximateprobabilistic relations.
Nonlinear Analysis: Hybrid Systems , 39, 2020.[LSZ20c] A. Lavaei, S. Soudjani, and M. Zamani. Compositional abstraction of large-scale stochastic systems: A relaxeddissipativity approach.
Nonlinear Analysis: Hybrid Systems , 36, 2020.[LSZ20d] A. Lavaei, S. Soudjani, and M. Zamani. Compositional (in)finite abstractions for large-scale interconnected stochasticsystems.
IEEE Transactions on Automatic Control , 65(12):5280–5295, 2020.[LYZ20] S. Liu, X. Yin, and M. Zamani. On a notion of approximate opacity for discrete-time stochastic control systems. In
Proceedings of the American Control Conference (ACC), to appear , 2020.[LZ19] A. Lavaei and M. Zamani. Compositional verification of large-scale stochastic systems via relaxed small-gain condi-tions. In
Proceedings of the 58th IEEE Conference on Decision and Control , pages 2574–2579, 2019.[LZ20] A. Lavaei and M. Zamani. From dissipativity theory to compositional synthesis of large-scale stochastic switchedsystems.
Submitted for publication , 2020.[MECL15] P. Mohajerin Esfahani, D. Chatterjee, and J. Lygeros. Motion planning for continuous-time stochastic processes: Adynamic programming approach.
IEEE Transactions on Automatic Control , 61(8):2155–2170, 2015.[MGW17] P. J. Meyer, A. Girard, and E. Witrant. Compositional abstraction and safety synthesis using overlapping symbolicmodels.
IEEE Transactions on Automatic Control , 2017.[MMS20] R. Majumdar, K. Mallik, and S. Soudjani. Symbolic controller synthesis for b¨uchi specifications on stochasticsystems. In
Proceedings of the 23rd International Conference on Hybrid Systems: Computation and Control , pages1–11, 2020.[MNP08] O. Maler, D. Nickovic, and A. Pnueli. Checking temporal properties of discrete, timed and continuous behaviors. In
Pillars of computer science , pages 475–505. Springer, 2008.[MSSM17] K. Mallik, S. Soudjani, A.-K. Schmuck, and R. Majumdar. Compositional construction of finite state abstractionsfor stochastic control systems. In
Proceedings of the 56th IEEE International Conference on Decision and Control(CDC) , pages 550–557, 2017.[MT93] S. P. Meyn and R. L. Tweedie.
Markov chains and stochastic stability . Comm. Control Engrg., Springer, London,1993.[NSZ19] A. Nejati, S. Soudjani, and M. Zamani. Abstraction-based synthesis of continuous-time stochastic control systems.In
Proceedings of the 18th European Control Conference , pages 3212–3217, 2019.[NSZ20a] A. Nejati, S. Soudjani, and M. Zamani. Compositional construction of control barrier certificates for large-scalestochastic switched systems.
IEEE Control Systems Letters , 4(4):845–850, 2020.[NSZ20b] A. Nejati, S. Soudjani, and M. Zamani. Compositional construction of control barrier functions for continuous-timestochastic hybrid systems. arXiv:2012.07296 , 2020.[NSZ20c] A. Nejati, S. Soudjani, and M. Zamani. Compositional construction of control barrier functions for networks ofcontinuous-time stochastic systems. In
Proceedings of the 21st IFAC World Congress, to appear , 2020.[NSZ21] A. Nejati, S. Soudjani, and M. Zamani. Compositional abstraction-based synthesis for continuous-time stochastichybrid systems.
European Journal of Control , 57:82–94, 2021.[NZ20] A. Nejati and M. Zamani. Compositional construction of finite MDPs for continuous-time stochastic systems: Adissipativity approach. In
Proceedings of the 21st IFAC World Congress, to appear , 2020.[Oks13] B. Oksendal.
Stochastic differential equations: an introduction with applications . Springer Science & Business Media,2013.[Pan09] Prakash Panangaden.
Labelled Markov Processes . IMPERIAL COLLEGE PRESS, 2009.[Par03] P. A. Parrilo. Semidefinite programming relaxations for semialgebraic problems.
Mathematical Programming ,96(2):293–320, 2003.[PAV +
13] A. Papachristodoulou, J. Anderson, G. Valmorbida, S. Prajna, P. Seiler, and P. Parrilo. SOSTOOLS version 3.00sum of squares optimization toolbox for MATLAB. arXiv:1310.4716 , 2013.[PBLB03] Giordano Pola, Manuela L. Bujorianu, John Lygeros, and Maria Domenica Di Benedetto. Stochastic hybrid models:An overview. In
IFAC Conference on Analysis and Design of Hybrid Systems , volume 36-6, pages 45–50, 2003.[PC16] Ali Pakniyat and Peter E. Caines. On the stochastic minimum principle for hybrid systems. In , pages 1139–1144, 2016.
UTOMATED VERIFICATION AND SYNTHESIS OF STOCHASTIC HYBRID SYSTEMS: A SURVEY 57 [PER17] C. Pilch, F. Edenfeld, and A. Remke. Hypeg: Statistical model checking for hybrid petri nets: Tool paper. In
Proceedings of the 11th EAI International Conference on Performance Evaluation Methodologies and Tools , pages186–191, 2017.[PJP07] S. Prajna, A. Jadbabaie, and G. J. Pappas. A framework for worst-case and stochastic safety verification usingbarrier certificates.
IEEE Transactions on Automatic Control , 52(8):1415–1428, 2007.[Pnu77] A. Pnueli. The temporal logic of programs. In
Proceedings of the 18th Annual Symposium on Foundations ofComputer Science , pages 46–57, 1977.[PP06] Gianni Pola and Giordano Pola. Optimal dynamic asset allocation: A stochastic invariance approach. In , pages 2589–2594, 2006.[PR05] S. Prajna and A. Rantzer. On the necessity of barrier certificates.
IFAC Proceedings Volumes , 38(1):526–531, 2005.[RCSL10] Federico Ramponi, Debasish Chatterjee, Sean Summers, and John Lygeros. On the connections between PCTL anddynamic programming. In
Proceedings of the 13th ACM International Conference on Hybrid Systems: Computationand Control , pages 253–262, 2010.[S +
14] D. Silver et al. Deterministic policy gradient algorithms. In
Proceedings of the 31st International Conference onInternational Conference on Machine Learning , pages 387–395, 2014.[SA12a] S. Soudjani and A. Abate. Higher-order approximations for verification of stochastic hybrid systems. In
AutomatedTechnology for Verification and Analysis , volume 7561 of
Lecture Notes in Computer Science , pages 416–434.Springer, 2012.[SA12b] S. Soudjani and A. Abate. Probabilistic invariance of mixed deterministic-stochastic dynamical systems. In
ACMProceedings of the 15th International Conference on Hybrid Systems: Computation and Control , pages 207–216,April 2012.[SA13] S. Soudjani and A. Abate. Adaptive and sequential gridding procedures for the abstraction and verification ofstochastic processes.
SIAM Journal on Applied Dynamical Systems , 12(2):921–956, 2013.[SA14a] S. Soudjani and A. Abate. Precise approximations of the probability distribution of a Markov process in time: anapplication to probabilistic invariance. In
Proceedings of TACAS14, LNCS 8413 , pages 547–561. Springer Verlag,2014.[SA14b] S. Soudjani and A. Abate. Probabilistic reach-avoid computation for partially-degenerate stochastic processes.
IEEETransactions on Automatic Control , 59(2):528–534, 2014.[SA15] S. Soudjani and A. Abate. Quantitative approximation of the probability distribution of a Markov process by formalabstractions.
Logical Methods in Computer Science , 11(3), 2015.[SAM15] S. Soudjani, A. Abate, and R. Majumdar. Dynamic Bayesian networks as formal abstractions of structured stochasticprocesses. In
Proceedings of the 26th International Conference on Concurrency Theory , pages 1–14, 2015.[SAM17] S. Soudjani, A. Abate, and R. Majumdar. Dynamic Bayesian networks for formal verification of structured stochasticprocesses.
Acta Informatica , 54(2):217–242, 2017.[SDC19] C. Santoyo, M. Dutreix, and S.l Coogan. Verification and control for finite-time safety of stochastic systems viabarrier functions. In
Proceedings of the IEEE Conference on Control Technology and Applications , pages 712–717,2019.[SGA15] S. Soudjani, C. Gevaerts, and A. Abate.
FAUST : Formal abstractions of uncountable-state stochastic processes. In TACAS’15 , volume 9035 of
Lecture Notes in Computer Science , pages 272–286. Springer, 2015.[SGE +
14] S. Soudjani, S. Gerwinn, C. Ellen, M. Fr¨anzle, and A. Abate. Formal synthesis and validation of inhomogeneousthermostatically controlled loads. In
Proceedings of the International Conference on Quantitative Evaluation ofSystems , pages 57–73, 2014.[SL95] R. Segala and N. Lynch. Probabilistic simulations for probabilistic processes.
Nordic Journal of Computing , 2(2):250–273, 1995.[SL10] Sean Summers and John Lygeros. Verification of discrete time stochastic hybrid systems: A stochastic reach-avoiddecision problem.
Autom. , 46(12):1951–1961, 2010.[SLTB +
06] Armando Solar-Lezama, Liviu Tancau, Rastislav Bodik, Sanjit Seshia, and Vijay Saraswat. Combinatorial sketchingfor finite programs.
ACM Sigplan Notices , 41(11):404–415, 2006.[Smi08] H. L. Smith. Global stability for mixed monotone systems.
Journal of Difference Equations and Applications ,14(10-11):1159–1164, 2008.[SMN17] S. Soudjani, R. Majumdar, and T. Nagapetyan. Multilevel Monte Carlo method for statistical model checking ofhybrid systems. In
International Conference on Quantitative Evaluation of Systems , pages 351–367, 2017.[Sou14] S. Soudjani.
Formal Abstractions for Automated Verification and Synthesis of Stochastic Systems . PhD thesis,Technische Universiteit Delft, The Netherlands, 2014.[SPB +
17] F. Shmarov, N. Paoletti, E. Bartocci, S. Lin, S. A. Smolka, and P. Zuliani. Automated synthesis of safe and robustPID controllers for stochastic hybrid systems. arXiv:1707.05229 , 2017.[SSP +
19] F. Shmarov, S. Soudjani, N. Paoletti, E. Bartocci, S. Lin, S. A. Smolka, and P. Zuliani. Automated synthesis of safedigital controllers for sampled-data stochastic nonlinear systems. arXiv:1901.03315 , 2019.[SSZ20] A. Salamati, S. Soudjani, and M. Zamani. Data-driven verification under signal temporal logic constraints. arXiv:2005.05040 , 2020. , SADEGH SOUDJANI , ALESSANDRO ABATE , AND MAJID ZAMANI , [ST12] J. Steinhardt and R. Tedrake. Finite-time regional verification of stochastic non-linear systems. The InternationalJournal of Robotics Research , 31(7):901–923, 2012.[Stu99] J. F. Sturm. Using SeDuMi 1.02, a MATLAB toolbox for optimization over symmetric cones.
Optimization methodsand software , 11(1-4):625–653, 1999.[SVA05] K. Sen, M. Viswanathan, and G. Agha. On statistical model checking of stochastic systems. In
International Con-ference on Computer Aided Verification , pages 266–280, 2005.[SZ15] F. Shmarov and P. Zuliani.
ProbReach : Verified probabilistic delta-reachability for stochastic hybrid systems. In
Proceedings of the 18th International Conference on Hybrid Systems: Computation and Control , pages 134–139,2015.[SZ16] F. Shmarov and P. Zuliani. Probabilistic hybrid systems verification via SMT and Monte Carlo techniques. In
Proceedings of the Haifa Verification Conference , pages 152–168, 2016.[TA11] I. Tkachev and A. Abate. On infinite-horizon probabilistic properties and stochastic bisimulation functions. In
Proceedings of the 50th IEEE Conference on Decision and Control and European Control Conference (CDC-ECC) ,pages 526–531, 2011.[TA12a] I. Tkachev and A. Abate. Regularization of Bellman equations for infinite-horizon probabilistic properties. In
Pro-ceedings of the 15th ACM international conference on Hybrid Systems: computation and control , pages 227–236,2012.[TA12b] I. Tkachev and A. Abate. Stability and attractivity of absorbing sets for discrete-time Markov processes. In
Pro-ceedings of the 51st IEEE Conference on Decision and Control , pages 7652–7657, 2012.[TA14] I. Tkachev and A. Abate. Characterization and computation of infinite horizon specifications over Markov processes.
Theoretical Computer Science , 515:1–18, 2014.[Tab09] P. Tabuada.
Verification and control of hybrid systems: A symbolic approach . Springer Science & Business Media,2009.[TMKA13] I. Tkachev, A. Mereacre, Joost-Pieter Katoen, and A. Abate. Quantitative automata-based controller synthesis fornon-autonomous stochastic hybrid systems. In
Proceedings of the 16th ACM International Conference on HybridSystems: Computation and Control , pages 293–302, 2013.[TMKA17a] I. Tkachev, A. Mereacre, J.-P. Katoen, and A. Abate. Quantitative model-checking of controlled discrete-timeMarkov processes.
Information and Computation , 253:1–35, 2017.[TMKA17b] Ilya Tkachev, Alexandru Mereacre, Joost-Pieter Katoen, and Alessandro Abate. Quantitative model-checking ofcontrolled discrete-time Markov processes.
Information and Computation , 253:1 – 35, 2017.[TSS14] A. R. Teel, A. Subbaraman, and A. Sferlazza. Stability analysis for stochastic hybrid systems: A survey.
Automatica ,50(10):2435–2456, 2014.[VGO19] A. P. Vinod, J. D. Gleason, and M. M. Oishi.
SReachTools : A MATLAB stochastic reachability toolbox. In
Pro-ceedings of the 22nd ACM International Conference on Hybrid Systems: Computation and Control , pages 33–38,2019.[VHO17] A. P. Vinod, B. HomChaudhuri, and M. MK. Oishi. Forward stochastic reachability analysis for uncontrolled linearsystems using fourier transforms. In
Proceedings of the 20th ACM International Conference on Hybrid Systems:Computation and Control , pages 35–44, 2017.[VO17] A. P. Vinod and M. MK. Oishi. Scalable underapproximation for the stochastic reach-avoid problem for high-dimensional LTI systems using fourier transforms.
IEEE control systems letters , 1(2):316–321, 2017.[VO18] A.P. Vinod and Meeko M.K. Oishi. Scalable underapproximative verification of stochastic LTI systems using con-vexity and compactness. In
Proceedings of the 21st International Conference on Hybrid Systems: Computation andControl , pages 1–10, 2018.[vS89] J. H. van Schuppen. Stochastic realization problems. In
Three decades of mathematical system theory , pages 480–523.Springer, 1989.[WABT19] B. Wu, M. Ahmadi, S. Bharadwaj, and U. Topcu. Cost-bounded active classification using partially observableMarkov decision processes. In
Proceedings of the American Control Conference (ACC) , pages 1216–1223, 2019.[WBS20] Rafael Wisniewski, Manuela L. Bujorianu, and Christoffer Sloth. p-safe analysis of stochastic hybrid processes.
IEEETrans. Autom. Control. , 65(12):5220–5235, 2020.[WS15] R. Wisniewski and C. Sloth. Converse barrier certificate theorems.
IEEE Transactions on Automatic Control ,61(5):1356–1361, 2015.[WTL15] T. Wongpiromsarn, U. Topcu, and A. Lamperski. Automata theory meets barrier certificates: Temporal logicverification of nonlinear systems.
IEEE Transactions on Automatic Control , 61(11):3344–3355, 2015.[WZK +
15] Q. Wang, P. Zuliani, S. Kong, S. Gao, and E. M. Clarke.
SReach : A probabilistic bounded delta-reachability analyzerfor stochastic hybrid systems. In
Proceedings of the International Conference on Computational Methods in SystemsBiology , pages 15–27, 2015.[YLWL17] X. Yin, Z. Li, W. Wang, and S. Li. Infinite-step opacity of stochastic discrete-event systems. In , pages 102–107, 2017.[YLWL19] X. Yin, Z. Li, W. Wang, and S. Li. Infinite-step opacity and k-step opacity of stochastic discrete-event systems.
Automatica , 99:266–274, 2019.
UTOMATED VERIFICATION AND SYNTHESIS OF STOCHASTIC HYBRID SYSTEMS: A SURVEY 59 [ZA14a] M. Zamani and A. Abate. Approximately bisimilar symbolic models for randomly switched stochastic systems.
IEEEControl Systems Letters , 69:38–46, 2014.[ZA14b] M. Zamani and A. Abate. Symbolic models for randomly switched stochastic systems.
Systems & Control Letters ,69:38–46, 2014.[ZA18] M. Zamani and M. Arcak. Compositional abstraction for networks of control systems: A dissipativity approach.
IEEE Transactions on Control of Network Systems , 5(3):1003–1015, 2018.[ZAG15] M. Zamani, A. Abate, and A. Girard. Symbolic models for stochastic switched systems: A discretization and adiscretization-free approach.
Automatica , 55(5):183–196, 2015.[Zam66] G. Zames. On the input-output stability of time-varying nonlinear feedback systems part one: Conditions derivedusing concepts of loop gain, conicity, and positivity.
IEEE Transactions on Automatic Control , 11(2):228–238, 1966.[Zam14] M. Zamani. Compositional approximations of interconnected stochastic hybrid systems. In
Proceedings of the 53rdIEEE Conference on Decision and Control (CDC) , pages 3395–3400, 2014.[ZLWDA18] Y. Zacchia Lun, J. Wheatley, A. D’Innocenzo, and A. Abate. Approximate abstractions of Markov chains withinterval decision processes.
Proceedings of the 6th IFAC Conference on Analysis and Design of Hybrid Systems ,51(16):91–96, 2018.[ZMEM +
14] M. Zamani, P. Mohajerin Esfahani, R. Majumdar, A. Abate, and J. Lygeros. Symbolic control of stochastic systemsvia approximately bisimilar finite abstractions.
IEEE Transactions on Automatic Control , 59(12):3135–3150, 2014.[ZRME17] M. Zamani, M. Rungger, and P. Mohajerin Esfahani. Approximations of stochastic hybrid systems: A compositionalapproach.
IEEE Transactions on Automatic Control , 62(6):2838–2853, 2017.[ZTA17] M. Zamani, I. Tkachev, and A. Abate. Towards scalable synthesis of stochastic control systems.
Discrete EventDynamic Systems , 27(2):341–369, 2017. Institute for Dynamic Systems and Control, ETH Zurich, Switzerland
Email address : [email protected] School of Computing, Newcastle University, United Kingdom
Email address : [email protected] Department of Computer Science, University of Oxford, United Kingdom
Email address : [email protected] Department of Computer Science, University of Colorado Boulder, USA Department of Computer Science, LMU Munich, Germany
Email address ::