Maximal switchability of centralized networks
MMaximal switchability of centralized networks
Sergei Vakulenko , , Ivan Morozov and Ovidiu Radulescu , Institute for Mech. Engineering Problems, Saint Petersburg, Russia Saint Petersburg National Research University of Information Technologies, Mechanics and Optics,Saint Petersburg, Russia University of Technology and Design, St.Petersburg, Russia, DIMNP UMR CNRS 5235, University of Montpellier 2, Montpellier, France.
November 9, 2018
Abstract
We consider continuous time Hopfield-like recurrent networks as dynamical models for generegulation and neural networks. We are interested in networks that contain n high-degree nodespreferably connected to a large number of N s weakly connected satellites, a property that we call n/N s -centrality. If the hub dynamics is slow, we obtain that the large time network dynamicsis completely defined by the hub dynamics. Moreover, such networks are maximally flexible andswitchable, in the sense that they can switch from a globally attractive rest state to any structurallystable dynamics when the response time of a special controller hub is changed. In particular, weshow that a decrease of the controller hub response time can lead to a sharp variation in the net-work attractor structure: we can obtain a set of new local attractors, whose number can increaseexponentially with N , the total number of nodes of the nework. These new attractors can be peri-odic or even chaotic. We provide an algorithm, which allows us to design networks with the desiredswitching properties, or to learn them from time series, by adjusting the interactions between hubsand satellites. Such switchable networks could be used as models for context dependent adaptationin functional genetics or as models for cognitive functions in neuroscience. Keywords
Networks, Attractors, Chaos, Bifurcations
Networks of dynamically coupled elements have imposed themselves as models of complex systemsin physics, chemistry, biology and engineering [36]. The most studied propriety of networks istheir topological structure. Structural features of networks are usually defined by the distributionof the number of direct connections a node has, or by various statistical properties of paths andcircuits in the network [36, 2]. An important structure related property of networks is their scale-freeness [23, 22, 2, 6] often invoked as a paradigm of self-organization and spontaneous emergenceof complex collective behaviour [9]. In scale-free networks the fraction P ( k ) of nodes in the networkhaving k connections to other nodes (i.e. having degree k ) can be estimated for large values of k as P ( k ) ∼ k − γ , where γ is a parameter whose value is typically in the range 2 < γ < a r X i v : . [ q - b i o . M N ] J un r bimodal degree networks can be called centralized. Centralized connectivity has been foundby functional imaging of brain activity in neuroscience [9], and also by large scale studies of theprotein-protein interactions or of the metabolic networks in functional genetics [23, 22].The centralized architecture was shown to be important for many emergent properties of net-works. For instance, there has been a lot of interest in the resilience of networks with respectto attacks that remove some of their components [3]. It was shown that networks with bimodaldegree connectivity are resilient to simultaneous targeted and random attacks [51], whereas scale-free networks are robust with respect to random attacks, but sensitive to targeted attacks that aredirected against hubs [10, 4]. For this reason, the term ”robust-yet-fragile” was coined in relationto scale-free networks [7].From a more dynamical perspective, a centralized architecture facilitates communication be-tween hubs, stabilizes hubs by making them insensitive to noise [55, 54] and allows for hub synchro-nization even in the absence of satellite synchronization [42, 41, 48]. Another important questionconcerning networks is how to push their dynamics from one region of the phase space to an-other or from one type of behaviour to another, briefly how to control the network dynamics[30, 49, 35, 12, 46, 40, 24, 39, 16, 61]. Several authors used Kalman’s results for linear systems tounderstand how network structure influences network dynamics controllability, and in particularhow to choose the control nodes [30, 35, 12]. As pointed out by [34, 27] several difficulties occurwhen one tries to apply these general results to real networks. Even for linear networks, the controlof trajectories is nonlocal [49] and shortcuts are rarely allowed. As a result, even small changesof the network state may ask for control signals of large amplitude and energy [59]. The controlof nonlinear networks is even more difficult and in this case we have no general results. Nonlinearnetworks can have several co-existing attractors and it is interesting to find out how to push thestate of the network from one attractor basin to another. The ability of networks to change attrac-tor under the effect of targeted perturbations can be called switchability. In relation to this, thepaper [43] has introduced the terminology ”stable yet switchable” (SyS) meaning that the networkremains stable given a context and is able to reach another stable state when a stimulus indicatesa change of the context. It was shown, by numerical simulations, that centralized networks withbimodal degree distribution are more prone to SyS behavior than scale-free networks [43]. Switch-ability is important for practical reasons, for instance in drug design. In such applications, oneuses pharmaceutical action on nodes to push a network that functions in a pathological attractor(such pathological attractors were discussed in relation to cancer [21] or neurological disorders[47, 14]) to a healthy functioning mode, characterized by a different attractor. Numerical methodsto study switchability of linear [58] and nonlinear [11] networks were discussed in relation withdrug design in cancer research. In theoretical biology, network switchability can be important formathematical theories of genetic adaptation [37]. If one looks at organisms as complex systemsand model them by networks, then adaptation to changes in the environment can be described asswitching the network from one attractor to another one with a higher fitness [37]. An importantquestion that is often asked with respect to tuning network dynamics is how many driver nodesare needed to control that dynamics. For linear networks, it was shown that this number is largeif we aim to obtain a total control, which allows us to switch the network between any pair ofstates. This number can be as high as 80% for molecular regulatory networks [31]. This fact, asemphasized in [58], contradicts empirical results about cellular reprogramming and about adaptiveevolution. Much less nodes are needed if instead of full controlability one wants switching betweenspecific pairs of unexpected and desired states [58]. This concept, named “transittability” in [58],is very similar to our switchability, but was studied only for linear systems.In this paper, we study dynamical properties of large nonlinear networks with centralized archi-tecture. We consider continuous time versions of the Hopfield model of recurrent neural networks[19] with a large number N of neurons. The Hopfield model is based on the two-states McCulloghand Pitts formal neuron and uses symmetrical weight matrices to specify interactions between neu-rons. Like to the Hopfield version, we use a thresholding function to describe switching between2he two neuron states, active and inactive. However, contrary to the original Hopfield version, wedo not impose symmetrical interactions between neurons, in other words our weight matrix is notnecessarily symmetric. This model has been successfully used to describe associative memories [19],neural computation [20, 32], disordered systems in statistical physics [50], neural activity [29, 14]and also to investigate space-time dynamics of gene networks in molecular biology [33, 57]. Thechoice of such type of dynamics is motivated by the existence of universal approximation results formultilayered perceptrons (see, for example, [5]). In particular, we have shown elsewhere that net-works with Hopfield-type dynamics can approximate any structurally stable dynamics, includingreaction-diffusion biochemical networks also largely used in biology [54].Our aim is to study analytically the ability of a network with centralized architecture to beswitchable. We employ a special notion of centrality. Many biological networks exhibit so-calleddissortative mixing, i.e., high-degree nodes are preferably connected to low-degree nodes [25]. Wewill consider networks with n strongly connected hubs. We also assume that each hub is underthe action of at least N s weakly connected satellites, that on turn receive actions from all thehubs. For large networks, N s increases at least as fast as a power of N , N s > c N θ where c >
0, 0 < θ < N is the total number of nodes. We call this property n/N s -centrality. This network architecture ensures a large number of feed-back loops that producecomplex dynamics. Furthermore, the dissortative connectivity implies functional heterogeneity ofthe hubs and satellites. The hubs play the role of controllers and the satellites sustain the feedbackloops needed for attractor multiplicity. The large number of satellites guarantees a sufficientflexibility of the network dynamics and also buffer the perturbations transmitted to the hubs. Thisprinciple applies well to gene networks. The hubs in such networks can be the transcription factors,which are stabilized by numerous interactions with non-coding RNAs that represent the satellites[28]. In addition to structural conditions, we will consider a special correlation between time scalesand connectivity of the nodes: the hubs have slow response, whereas the satellites respond rapidly.This condition is natural for many real networks. The hubs have to cope with multiple tasks,therefore they must have more complex interaction than the satellites. Consequently, the hubsneed more resources to be produced, decomposed, and react with other nodes, therefore theirdynamics is slow. This property is obvious for gene networks, where transcription factors arecomplex proteins, much larger and more stable than the non-coding RNAs.Our first result is valid without conditions on the structure and depends only on the conditionon the timescales. We assume that there exist n << N slow nodes, whereas all the remainingones are fast. Then, the dynamics of the network can be reduced to n variables. We prove theexistence of an inertial manifold of dimension n , which completely captures all network dynamicsfor large times. We recall that the fundamental concept of inertial manifold was introduced forinfinite dimensional and multidimensional systems. The inertial manifolds are globally attractinginvariant ones [38]. The large time dynamics of a system possessing an inertial manifold, is definedby a smooth vector field F of relatively small dimension, so-called inertial form. All attractors lieon inertial manifold [38].The second result holds under the structural assumption that the network is n/N s -central.Under this condition, we show that the inertial forms F obtained from such networks are dense inthe set of all smooth vector fields of dimension n . This implies that given a certain combinationof attractors defined by vector fields Q i we can construct a centralized network that exhibits acombination of attractors that is topologically equivalent to the one given. Furthermore, we showthat n/N s -central networks can exhibit ”maximal switchability”. By changing a control parameter ξ , which determines the response time of a single network hub (”controller” hub), we can sharplychange the network attractor. For instance we can switch from a situation when the network has asingle rest point for ξ > ξ to a situation when the network has a complicated global attractor for ξ < ξ , including a number of local attractors, which may be periodic or chaotic. The network statetends to the corresponding local attractor depending on the initial state of the control hub. Thisresult shows in an analytical and rigorous way how nonlinear networks can be switched by only3ne control node. The possibility of switching nonlinear networks by a small number of nodes iscrucial in theories of genetic adaptation. Indeed, phenomenological theories predict and empiricaldata confirm that the main part of the adaptive evolution process consists in only a few mutationsproducing large fitness changes [37].Our third result proves, in an analytical way, that the number of rest point local attractors(and therefore the network capacity) of n/N s -central networks may be exponentially large in thenumber of nodes.We also describe a constructive algorithm, which allows us to obtain a centralized network thatperforms a prescribed inertial dynamics and the desired switching properties of the network. We consider the Hopfield-like networks [19] described by the ordinary differential equations du i dt = σ ( N (cid:88) j =1 W ij u j − h i ) − λ i u i , (2.1)where u i , h i and λ i > i = 1 , ..., N are node activities, activation thresholds and degradationcoefficients, respectively. The matrix entry W ij describes the action of the node j on the node i ,which is an activation if W ij > W ij <
0. Contrary to the original Hopfieldmodel, the interaction matrix W is not necessarily symmetric. The function σ is an increasing andsmooth (at least twice differentiable) ”sigmoidal” function such that σ ( −∞ ) = 0 , σ (+ ∞ ) = 1 , σ (cid:48) ( z ) > . (2.2)Typical examples can be given by σ ( h ) = 11 + exp( − h ) , σ ( h ) = 12 (cid:18) h √ h + 1 (cid:19) . (2.3)The structure of interactions in the model is defined by a weighted digraph ( V, E, W ) with theset V of nodes, the edge set E and weights W ij . The nodes v j , j = 1 ..., N can be neurons or genes,depending on applications. Assumption 1.
Assume that if W ji (cid:54) = 0 , then ( i, j ) is an edge of the graph, ( i, j ) ∈ E . This means that the i -thnode can act on the j -th node only if it is prescribed by an edge of the digraph ( V, E, W ) . We alsosuppose that ( i, i ) / ∈ E , i.e., the nodes do not act on themselves .Assume that the digraph ( V, E, W ) satisfies a condition, which is a variant of the centralityproperty. This condition is a purely topological one and thus it is independent on the weights W ij .To formulate this condition, we introduce a special notation.Let us consider a node v j . Let us denote by S ∗ ( j ) the set of all nodes, which act on the neuron j : S ∗ ( j ) = { v i ∈ V : edge ( i, j ) ∈ E } . (2.4)For each set of nodes C ⊂ V we introduce the set S ( C ) of the nodes, which are under action of allnodes from C and which are not belonging to C : S ( C ) = { v i ∈ V : f or each j ∈ C edge ( j, i ) ∈ E and v i / ∈ C} . (2.5) n/N s -Centrality assumption. The graph ( V, E, W ) is connected and there exists a set ofnodes C such that C consists of n nodes; ii for each j ∈ C the intersection S ∗ ( j ) ∩ S ( C ) contains at least N s nodes, where N s > c N θ with constants c > , θ ∈ (0 , , which are independent of j and N . The nodes from C can be interpreted as hubs (centers) and the nodes from S ( C ) are the satellites.The condition ii implies that each center is under action of sufficiently many satellites. In turn, ifwe consider the union of these satellites, all the centers act on them (see Fig.1). Such an intensiveinteraction leads, as we will see below, to a very complicated large time behaviour. w w w v w w w v Figure 1: This image shows an n/N s -central network with n = 2 and N s = 3. The graph consistsof 8 nodes denoted by v , v , w , w , w , w , w , w . The set { v , v } is the set of centers C . The sets S ( C ) , S ∗ ( v ) and S ∗ ( v ) are as follows: S ( C ) = { w , w , w , w , w , w } , S ∗ ( v ) = { w , w , w } and S ∗ ( v ) = { w , w , w } . The sets S ∗ ( v ) ∩ S ( C ) = { w , w , w } and S ∗ ( v ) ∩ S ( C ) = { w , w , w } contain three nodes each. Our results can be outlined as follows. The result on the inertial dynamics existence describesa situation, when the interaction topology is quite arbitrary. We assume that there exist n slownodes, say, u , u , ..., u n with λ i = O (1) whereas all the rest ones u n +1 , ...u N are fast, i.e., thecorresponding λ i have order O ( κ − ), where κ is a small parameter. Then we show that thereexists an inertial manifold of dimension n . We obtain, under general conditions, that for times t >> κ log κ the dynamics of (2.1) is defined by the reduced equations du j dt = F j ( u , ..., u n , W, h, λ ) , (3.1) u k = U k ( u , ..., u n , W, h, λ ) , k = n + 1 , ..., N, (3.2)where F j and U k are some smooth functions of u , ..., u n , and h, λ denote the vector parameters( h , ..., h N ) and ( λ , ..., λ N ), respectively. So, F gives us the inertial form on an inertial manifold.The inertial form completely defines the dynamics for large times [38].More interestingly, we can show that the vector field F is, in a sense, maximally flexible.Roughly speaking, by the number of nodes N , the matrix W and h we can obtain all possiblefields F (up to a small accuracy (cid:15) , which can be done arbitrarily small as N goes to ∞ ), see section5 for a formal statement of this flexibility property. For the networks this flexibility property holdsunder n/N s -Centrality assumption . 5et us introduce a special control parameter ξ , which modulates the degradation coefficient λ i for a hub: λ i = ξ ¯ λ i for some i ∈ C . This hub is a ”controller”. When we vary the coefficient ξ ,the interaction topology and the entries of the interaction matrix do not change, but the responsetime of the controller hub changes.One can choose the network parameters N, W, λ in such a way that for ξ > ξ the globalattractor is trivial, it is a rest point, but for an open set of other values ξ the global attractor of(2.1) contains a number of local attractors.This result can be interpreted as “maximal switchability”. A similar effect was found in [13] bynumerical simulations for some models of neural networks. This effect describes a transition fromneural resting states (NRS) to complicated global attractors, which occur as a reaction on learningtasks. Note that in [13] attractors consist of a number of steady states. In our case the globalattractors can include many local attractors of all possible kinds including chaotic and periodicones.We end this section with a remark. Our method approximates vector fields by neural networks,but what can be said about the relationship between the trajectories of the simulated system andthe ones corresponding to the neural network?For chaotic and even for periodic attractors, direct comparison of trajectories is not a suitabletest for the accuracy of the approximation. General mathematical arguments allow us say only thatthese trajectories will be close for bounded times. For large times we can say nothing especiallyfor general chaotic attractors. Consider the case when the attractor A of the simulated system istransitive. This means the dynamics is ergodic and for smooth function φ the time averages S F,φ = lim T → + ∞ T − (cid:90) T φ ( v ( t )) dt (3.3)coincide with the averages (cid:82) A φ ( v ) dµ ( v ) over the attractor, where µ is an invariant measure on A .Then, a suitable criterion of approximation is that the averages S F,φ and the correspondingones generated by the approximating centralized neural network, are close for smooth φ : | S F,φ − S G anN , φ | = Err approx < δ ( (cid:15), φ ) (3.4)where G anN is the neural network approximation of F and δ → (cid:15) →
0. This “stochasticstability” property holds for hyperbolic (structurally stable) attractors [26, 60, 56].
Our first results do not use any assumptions on the network topology. However, we suppose thatthere are two types of network components that are distinguished by their time scales into slownodes and fast nodes. To take into account the two types of the nodes, we use distinct variables v j for slow variables, j = 1 , . . . , n and w i for the fast ones, i = 1 , . . . , N − n = N . The real matrixentry A ji defines the intensity of the action of the fast node i on the slow node j . Similarly, the n × N matrix B , N × N matrix C and n × n matrix D define the action of the slow nodes onthe fast ones, the interactions between the fast nodes and the interactions between the slow nodes,respectively. We denote by h i and λ i the threshold and degradation parameters of the fast nodesand by ˜ h i and ˜ λ i the same parameters for the slow nodes, respectively. To simplify formulas, weuse the notation n (cid:88) j =1 D ij v j = D i v, N (cid:88) k =1 C jk w k = C j w. dw i dt = σ (cid:16) B i v + C i w − ˜ h i (cid:17) − κ − ˜ λ i w i , (4.1) dv j dt = σ ( A j w + D j v − h j ) − λ j v j , (4.2)where i = 1 , ..., N , j = 1 , ..., n . Here unknown functions w i ( t ) , v j ( t ) are defined for times t ≥ κ is a positive parameter, therefore, the variables w i are fast.We set the initial conditions w i (0) = ˜ φ i ≥ , v j (0) = φ j ≥ . (4.3)It is natural to assume that all concentrations are non-negative at the initial moment. It is clearthat they stay non-negative for all times. Let us prove that the network dynamics is correctly defined for all t and solutions are non-negativeand bounded. For positive vectors r = ( r , ..., r n ) and R = ( R , ...., R N ), let us introduce the sets B defined by B ( r, R ) = { ( w, v ) : 0 ≤ v j ≤ r j , ≤ w i ≤ R j , j = 1 , ..., n, i = 1 , ..., N } . Note that dw i dt < − κ − ˜ λ i w i . Thus, w i ( t ) < X ( t ) for positive times t , where dXdt = 1 − κ − ˜ λ i X, X (0) = w i (0) . Therefore, resolving the last equation, and repeating the same estimates for v i ( t ), one finds0 ≤ w i ( x, t ) ≤ ˜ φ i exp( − ˜ κ − λ i t ) + κ ˜ λ − i (1 − exp( − κ − ˜ λ i t )) , ≤ v j ( x, t ) ≤ φ j exp( − λ j t ) + λ − j (1 − exp( − λ j t )) , (4.4)Let us take arbitrary a > r j ( a ) = aλ − j and R i ( a ) = aκ ˜ λ − i . Estimates (4.4) showthat solutions of (4.1), (4.2) exist for all times t and they enter the set B ( r ( a ) , R ( a )) at a timemoment t . The solutions stay in this set for all t > t , thus, this set is absorbing. This showsthat system (4.1),(4.2) defines a global dissipative semiflow S tH [17]. Moreover, this semiflow hasa global attractor contained in each B ( r ( a ) , R ( a )), where a > A simpler asymptotic description of system dynamics is possible under assumptions on networkcomponents timescales. We suppose here that the u -variables are fast and the v -ones are slow. Weshow then that the fast w variables are slaved, for large times, by the slow v modes. More precisely,one has w = κU ( v ) + ˜ w , where κU ( v ) is a correction and κ > P = { A , B , C , D , h, ˜ h, ˜ λ, λ } satisfy the following conditions: A = κ − ¯ A , (4.5)7 ¯ A | , | B | , | C | , | D | < c , (4.6)0 < c < ¯ λ i < c , < ˜ λ i < c . (4.7)Here all positive constants c k are independent of κ for small κ .The scaling assumption on A is needed because, as we will prove later, w = O ( κ ) for small κ .For the same reasons, C i w can be neglected with respect to B i v for small κ , meaning that theaction of centers on satellites is dominant with respect to satellites mutual interactions. In otherwords, these conditions describe a divide and rule control principle . Our goal is to show that the network dynamics can realize, in a sense, arbitrary structurally stabledynamics of the centers. To precise this assertion, let us describe the method of realization of thevector fields for dissipative systems (proposed in [44]). More precisely, we are interested in systemsenjoying the following properties: A These systems generate global semiflows S t P in an ambient Hilbert or Banach phase space H .These semiflows depend on some parameters P (which could be elements of another Banach space B ). They have global attractors and finite dimensional local attracting invariant C - manifolds M , at least for some P . B Dynamics of S t P reduced on these invariant manifolds can be, in a sense, almost completelytuned by variations of the parameter P .It can be described as follows. Assume the differential equations dqdt = Q ( q ) , Q ∈ C ( B n ) (5.1) define a global semiflow in a unit ball B n ⊂ R n .For any prescribed dynamics (5.1) and any (cid:15) > , we can choose suitable parameters P = P ( n, F, (cid:15) ) such that B1 The semiflow S t P has a C - smooth locally attracting invariant manifold M P diffeomorphicto B n ; B2 The reduced dynamics S t P | M P is defined by equations dqdt = ˜ Q ( q, P ) , ˜ Q ∈ C ( B n ) (5.2) where the estimate | Q − ˜ Q | C ( B n ) < (cid:15) (5.3) holds. In other words, one can say that, by P , the reduced dynamics on the invariant manifoldcan be specified to within an arbitrarily small error. Therefore, roughly speaking all robust dynamics (stable under small perturbations) can begenerated by the systems, which satisfy above formulated properties. Such systems can be named maximally flexible . In order to show that maximal flexibility covers also the case of chaotic dy-namics, let us recall some facts about chaos and hyperbolic sets.Let us consider dynamical systems (global semiflows) S t , ..., S tk , t >
0, defined on the n -dimensional closed ball B n ⊂ R n defined by finite dimensional vector fields F ( k ) ∈ C ( B n ) andhaving structurally stable attractors A l , l = 1 , ..., k . These attractors can have a complex form,since it is well known that structurally stable dynamics may be “chaotic”. There is a rather widevariation in different definitions of ”chaos”. In principle, one can use here any concept of chaos,8rovided that this is stable under small C -perturbations. To fix ideas, we shall use here, follow-ing [45], such a definition. We say that a finite dimensional dynamics is chaotic if it generates acompact invariant hyperbolic set Γ , which is not a periodic cycle or a rest point (for a definition ofhyperbolic sets see, for example, [45]). The hyperbolic sets give remarkable analytically tractableexamples, where chaotic dynamics can be studied. For example, the Smale horseshoe is a hyper-bolic set. If this set Γ is attracting we say that Γ is a chaotic (strange) attractor. In this paper, weuse only the following basic property of hyperbolic sets, so-called Persistence [45]. This means thatthe hyperbolic sets are, in a sense, stable(robust). This property can be described as follows. Let asystem of differential equations be defined by a C -smooth vector field Q on an open domain in R n with a smooth boundary or on a smooth compact finite dimensional manifold. Assume this systemdefines a dynamics having a compact invariant hyperbolic set Γ . Let us consider (cid:15) -perturbed thevector field Q + (cid:15) ˜ Q , where ˜ Q is bounded in C -norm. Then, if (cid:15) > Γ . Thecorresponding dynamics restricted to Γ and ˜ Γ respectively, are topologically orbitally equivalent( topological equivalency of two semiflows means that there exists a homeomorphism, which mapsthe trajectories of the first semiflows on the trajectories of the second one, see [45] for details).We recall that chaotic structurally stable ( persistent) attractors and invariant sets exist: thisfact is well known from the theory of hyperbolic dynamics [45].Thus, any kind of the chaotic hyperbolic sets can occur in the dynamics of the systems, forexample, the Smale horseshoes, Anosov flows, and the Ruelle-Takens-Newhouse chaos, see [45]. Ex-amples of systems satisfying these properties can be presented by some reaction-diffusion equationsand systems [44, 52, 53], and neural network models [53]. For vectors a = ( a , ..., a n ) and b = ( b , ..., b n ) such that a i < b i for each i let us denote by Π ( a, b ) = { v ∈ R n : a i ≤ v i ≤ b i } (6.1)a n -dimensional box in v -space. Moreover, let us define Π λ by Π λ = Π (0 , λ − ), where the vector λ − has components ( λ − , ..., λ − n ). Theorem 6.1
Under assumptions (2.2), (4.5), (4.6) and (4.7) for sufficiently small κ there existsa n -dimensional inertial manifold M n defined by w i = κ ˜ λ − i U i ( v, κ, P ) , v ∈ Π λ (6.2) where U i ∈ C r ( Π λ ) , and r ∈ (0 , . The functions U i admit the estimate | U i ( v, κ, P ) − σ (cid:16) B i v − ˜ h i (cid:17) | C ( Π λ ) < c κ, v ∈ Π λ . (6.3) The v dynamics for large times takes the form dv j dt = F j ( v, P ) + ˜ F j ( v, κ, P ) , (6.4) where ˜ F j satisfy | ˜ F j | C ( Π λ ) < c κ (6.5) with F j ( v, P ) = σ (cid:32) N − n (cid:88) i =1 ¯ A ji ˜ λ − i σ (cid:16) B i v − ˜ h i (cid:17) + D j v − h j (cid:33) − λ j v j . (6.6)9ote that the matrix C is not involved in relation (6.6), which defines the family of the vectorfields F ( inertial forms). This property holds due to the property that inter-satellite interactionsare dominated by the satellite-center ones. The next assertion means that this principle allows usto create a network dynamics with prescribed dynamics (if the network satisfies n/N s -centralityassumption and N is large enough). It is valid under the additional condition that the interactiongraph ( V, E ) verifies the centrality condition.
Theorem 6.2
Assume n/N s -centrality assumption is satisfied. Then the family of the vector fields F defined by (6.6) is dense in the set of all C vector fields Q defined on the unit ball B n ⊂ R n .In the other words, centralized Hopfield neural networks are maximally flexible. Let us choose some i C such that i C belongs to C . The corresponding node will be called acontroller hub. We introduce the control parameter ξ by λ i C = ξ ¯ λ i C , (6.7)where we fix a positive ¯ λ i C .Theorem 6.2 can be used to show the following Theorem 6.3 (Maximal switchability theorem)
Let us consider dynamical systems (globalsemiflows) S t , ..., S tk , t > , defined on the n -dimensional closed ball B n ⊂ R n defined by finitedimensional vector fields F ( k ) ∈ C ( B n ) and having structurally stable attractors A l , l = 1 , ..., k .For sufficiently large N and any graph ( V, E ) satisfying the n/N s - centrality condition thereexists a choice of interactions W ij and thresholds h i such that Assumption 1 holds and( i ) there exist a ξ such that for all ξ > ξ the dynamics of network (2.1) has a rest point,which is a global attractor;( ii ) for an open interval of values ξ the global semiflow S tH defined by (2.1) have local attractors B l such that the restrictions of the semiflow S tH to B l are orbitally topological equivalent to thesemiflows S tl restricted to A l . Finally, let us give an estimate on the maximal number of equilibria N eq of centralized networks.This number is a characteristics of the network capacity, flexibility and adaptivity. To proceed tothese estimates, let us define a procedure, which can be named decomposition into “distar” motifs.In the network interaction graph ( E, V ) we choose some nodes v , ..., v n , which we conditionallyconsider as hubs. By “distar” motif we understand a part of interaction graph consisting of thehub v j and the subset S j of the set S ∗ j (defined by (2.5)) consisting of the nodes connected in bothdirections to v j : S j = { v i ∈ V : ( i, j ) and ( j, i ) ∈ E } . This distar motif becomes an usual starif directions of the edges are ignored. Consider the union U n of all S j . Some nodes w ∈ U n maybelong to two different sets S j and S k , where k (cid:54) = j . We remove from the vertex set V all suchnodes. After such removing we obtain a part of graph G n = ( V (cid:48) , E (cid:48) ) of the initial graph ( E, V ),which is a union of n disjoint distars S , ..., S n , where each S k contains a single center { v k } and µ ( S k ) satellites connected with the center in both directions. Recall that the graph ( V (cid:48) , E (cid:48) ) is apart of graph ( V, E ) if V (cid:48) ⊂ V and E (cid:48) ⊂ E . These numbers µ ( S k ) depend on the choice of hubnodes { v , ..., v n } .We will prove the following theorem: Theorem 6.4
The maximal possible number N eq ( E, N ) of equilibria of a network with a giveninteraction graph ( E, V ) , where V consists of N nodes, satisfies N eq ≥ sup µ ( S ) µ ( S ) ...µ ( S n ) , (6.8) where the supremum is taken over all integers n > and all graphs G n ,which are parts of interactiongraph ( V, E ) and consist of n disjoint distars. Here µ ( S l ) is the number of the nodes in the distar S l . (cid:98) ( N − n ) /n (cid:99) . Then, the maximal possible number N eq of equilibria in such a centralizednetwork (2.1) with N nodes and n centers satisfies N eq ≥ (cid:98) ( N − n ) /n (cid:99) n , where (cid:98) x (cid:99) denotes thefloor of a real number x . Note that for a fixed N the maximum of ( N/n ) n over n = 1 , , ... is attained at n = (cid:98) N/ (cid:99) , when the distars contain 5 satellites each. Therefore we obtain theestimate N eq ≥ (cid:98) N/ (cid:99) . Let us start by proving a lemma
Lemma 7.1
Under assumptions (4.5), (4.6) and (4.7) for sufficiently small positive κ < κ solu-tions ( u, v ) of (4.1), (4.2) and (4.3) satisfy w i ( t ) = κU i ( v ( t ) , B , ˜ h ) + ˜ w i ( t ) , (7.1) where U = ( U , ..., U n ) is defined by U i ( v, B , ˜ h ) = ˜ λ − i σ (cid:16) B i v ( t ) − ˜ h i (cid:17) . (7.2) Then, for some T function ˜ w satisfies the estimates | ˜ w ( t ) | < c κ , t > T (7.3) where c does not depend on t and κ . The time moment T depends on initial data and the networkparameters. Proof . Let us introduce a new variables ˜ w i by (7.1). They satisfy the equations d ˜ w i dt = H i ( v, ˜ w ) − κ − ˜ λ i ˜ w i , (7.4)where H i ( v, ˜ w ) = κZ i ( v ) + W i ( v, ˜ w ) ,Z i ( v ) = n (cid:88) j =1 ∂U i ( v ) ∂v j ( σ (cid:0) ¯ A j U + D j v − h j (cid:1) − ξ ¯ λ j v j ) , and W i ( v, ˜ w ) = σ (cid:16) B i v + C i w − ˜ h i (cid:17) − σ (cid:16) B i v − ˜ h i (cid:17) . Let us estimate H i ( v, ˜ w ) for sufficiently large t . According to (4.4), for such times we can usethat ( w, v ) ∈ B ( r ( a ) , R ( a )), where a >
1. In this domain B ( r ( a ) , R ( a )) one has sup | Z i | < c andsup | W i | < c κ , where c , c are independent of κ . Therefore, H i ( v ( t ) , ˜ w ( t )) < c κ, t > T ( κ, P ) . Now, as above in subsection 4.1, equation (7.4) entails estimate (7.3). The assertion is proved.
Proof of Theorem 6.1.
The rest part of the proof of Theorem 6.1 uses the well knowntechnique of invariant manifold theory, see, for example, [45, 38, 18]. Let us consider the domain D κ = { w : | w | < c κ } . Theorem 6.1.7 [18] shows that for d ∈ (0 ,
1) there is a locally attractive C d - smooth invariant manifold M n . Relation (6.3) follows from (7.3). The global attractivityof this manifold also follows from (7.3). The theorem is proved.11 Proof of Theorems 6.2, 6.3 and 6.4
The main idea of the subsequent statement is to study the dependence of the fields F j definedby Eq.(6.6) on the parameters P . To this end, we apply a special method stated in the nextsubsection.Let us formulate a lemma, that gives us a key tool and which implies Theorem 6.2. Lemma 8.1
Assume a i > δ/λ i , b i < (1 − δ ) /λ i i = 1 , ..., n. (8.1) Let Q = ( Q ( v ) , ..., Q n ( v )) be a C smooth vector field on Π ( a, b ) and δ > verify − δ < Q i ( v ) < δ, v ∈ Π ( a, b ) , i = 1 , ..., n. (8.2) Then there are parameters P of the neural network such that the field F defined by (6.6) satisfiesthe estimates sup v ∈ Π ( a,b ) | F ( v, P ) − Q ( v ) | < (cid:15), (8.3)sup v ∈ Π ( a,b ) |∇ F ( v, P ) − ∇ Q ( v ) | < (cid:15). (8.4) In other words, the fields F are dense in the vector space of all C smooth vector fields satisfyingto (8.2). Proof . The proof uses the standard results of the multilayered network theory.
Step 1 . The first preliminary step is as follows. Let us solve the system of equations σ ( R j ) = Q j ( v ) + λ j v j , v ∈ Π ( a, b ) (8.5)with unknown R j . Here R j are the regulatory inputs of the sigmoidal functions. These equationshave a unique solution due to conditions (2.2), (8.1) and (8.2): the right hand sides V j + λ j v j rangein (0 , R i ( v ) are C -smooth vector fields. Step 2 . Consider relation (6.6). We choose entries A ji and B il in a special way. First, let usset A ji = 0 if i / ∈ S ∗ ( j ), where the set S ∗ ( j ) is defined in the n/N s -centrality assumption, seecondition ii . Recall that S ∗ ( j ) is the set of the satellites acting on the center j . Note that thensum (6.6) can be rewritten as F j ( v, P ) = σ (cid:88) i ∈S ∗ ( j ) ¯ A ji ˜ λ − i σ (cid:16) B i v − ˜ h i (cid:17) + D j v − h j − λ j v j . (8.6)Using the result of step 1 and this relation, we see that our problem is reduced to the following:to approximate R j ( v ) in C norm with a small accuracy O ( (cid:15) ) by H j ( v, P ) = (cid:88) i ∈S ∗ ( j ) ¯ A ji ˜ λ − i σ (cid:16) B i v − ˜ h i (cid:17) + D j v − h j . (8.7)Note that, according to the centrality assumption, the set S ∗ ( j ) contains N s > CN θ elements.Moreover, due to this assumption, the sum B i = (cid:80) k B ik v k involves all k, k = 1 , ..., n . Therefore,since n is fixed and N can be taken arbitrarily large, the theorem on the universal approximationby multilayered perceptrons (see, for example, [5]) implies that the fields H = ( H , ..., H n ) aredense in the Banach space of all the vector fields on Π ( a, b ) (with C - norm). Therefore, H j approximate R j with O ( (cid:15) )-accuracy in C - norm. This finishes the proof.12 w w v w w w v ˜ w ˜ w ˜ w zFigure 2: Modular architecture. The switching module consists of the center z and the satellites˜ w , ˜ w , ˜ w . The generating module consists of the centers v , v and the satellites w , ..., w . Ideas behind proof . Before stating a formal proof, we present a brief outline, which describes mainideas of the proof and the architecture of the switchable network. The network consists of twomodules. The first module is a generating one and it is a centralized neural network with n centers v , ..., v n and satellites w , ..., w N . The second module consists of a center v n +1 = z and m satellites˜ w , ..., ˜ w m . The satellites from this module interact only with the module center z , i.e., in thismodule the interactions can be described by a distar graph. Only the center of the second moduleinteracts with the neurons of the first (generating) module. We refer to the second module as aswitching one. This architecture is shown on Fig. 2.For the switching module the correspoding equations have the following form. Let us considera distar interaction motif, where a node z is connected in both directions with m nodes ˜ w , ..., ˜ w m .We set n = 1 and N = m , ˜ λ i = 1, D = , C = , λ = 1, and A j = κ − ¯ a j in eqs. (4.1) and(4.2). By such notation the equations for the switching module can be rewritten in the form d ˜ w i dt = σ (cid:16) ˜ b i z − ˜ h i (cid:17) − κ − ˜ w i , (8.8) dzdt = σ κ − m (cid:88) j =1 ˜ a j ˜ w j − h − ξ ¯ λz, (8.9)where i = 1 , ..., m and ˜ b i , ˜ a j , ¯ λ >
0. 13nder above assumptions on the network interactions, equations for generating module can berepresented as follows: dw i dt = σ (cid:0) B i v + C i w − d i z − ¯ h i (cid:1) − κ − ˜ λ i w i , (8.10) dv j dt = σ (cid:16) A j w + D j v − ˜ d j z − h j (cid:17) − λ j v j , (8.11)where i = 1 , ..., N, j = 1 , ..., m and d i , ˜ d j are coefficients.These equations involve z as a parameter. This fact can be used in such a way. Consider thesystem of the differential equations dv/dt = Q ( v, z ) , v = ( v , . . . , v n ) (8.12)where z is a real control parameter. Let z , ..., z m +1 be some values of this parameter. We finda vector field Q such that for z = z l , where l = 1 , ..., m , the dynamics defined by (8.12) has theprescribed structurally stable invariant sets Γ l . Furthermore, according to theorem 6.2, for eachpositive (cid:15) we can choose the parameters N, B i , C i , ˜ b i , ˜ a i , ¯ h i , A j , D j , d i , ˜ d j , h j , λ j , ˜ λ i of the system(8.10) and (8.11) such that the dynamics of this system will have structurally stable invariant sets˜ Γ l topologically equivalent to Γ l .For the switching module we adjust the center-satellite interactions and the center responsetime parameter ξ in such a way that for a set of values ξ the switching module has the dynamicsof system (8.8),(8.9) with m different stable hyperbolic equilibria z = z , z , ..., z m +1 and forsufficiently large ξ system (8.8) and (8.9) has a single equilibrium close to z = 0. Existence ofsuch a choice will be shown in coming lemma 8.2. Then the both modules form a network havingneed dynamical properties formulated in the assertion of Theorem 6.3. Proof . Let us formulate some auxiliary assertions. First we consider the switching module.
Lemma 8.2
Let m be a positive integer and β ∈ (0 , . For sufficiently small κ > there exist ¯ a j , b i , ˜ h i , h such that i for an open interval of values ξ system (8.8),(8.9) has m stable hyperbolic rest points z j ∈ ( j − β, j + β ) , where j = 1 , ..., m ; ii for ξ > ξ > system (8.8),(8.9) has a single stable hyperbolic rest point. Proof . Let h = 0. To find equilibria z , we set d ˜ w i /dt = 0, and express ˜ w i via z . Then weobtain the following equation for the rest points z : ξz = σ m (cid:88) j =1 ˜ a j σ (cid:16) ˜ b j z − ˜ h j (cid:17) . (8.13)For especially adjusted parameters eq. (8.13) has at least m solutions, which give stable equilibriaof system (8.8),(8.9). To show it, we assume that 0 < κ <<
1, ˜ b j = ˜ b = κ − / and ˜ h j = ˜ bµ j , where µ j = j − β . We obtain then V ( ξz ) = m (cid:88) j =1 σ (˜ b ( z − µ j )) + O ( κ ) = F m ( z, β, κ ) , (8.14)where V ( z ) is a function inverse to σ ( z ) defined on (0 , b >> κ , the plot of thefunction F m is close to a stairway (see Fig. 3). Let ξ = 1 , ˜ a = V ( µ ) + κ, ˜ a j = V ( µ j ) − V ( µ j − ) , j = 2 , ..., m. V ( z ) with the almost horizontal pieces of the plot of F m give us m stable equilibria of system (8.8),(8.9). These equilibria z j lie in the corresponding intervals( j − β, j + β ). For sufficiently large ξ we have a single rest stable point z at 0. The lemma isproved.Consider compact invariant hyperbolic sets Γ , ..., Γ m of semiflows defined by arbitrarily chosen C smooth vector fields Q ( l ) on the unit ball B n ⊂ R n , where l = 1 , ..., m . Lemma 8.3
Let Π ( a, b ) be a box in R n and m > be a positive integer. There is a C -smoothvector field Q on Π ( a, b ) × [0 , m + 1] such that equation (5.1) defines a semiflow having hyperbolicsets Γ , ..., Γ m and the restriction of this field on Π ( a, b ) × [0 , has an attractor consisting of asingle hyperbolic rest point. Figure 3:
The intersections of the curve F m ( z, β, κ ) and the curve V ( z ) give equilibria of system (8.8),(8.9)for ξ = 1. Stable equilibria correspond to the intersections of V with almost horizontal pieces of the graphof F m . Proof . The proof uses the following idea. For k ∈ { , ..., m + 1 } let Q ( k ) ( v ) be a vector fieldon Π ( a, b ) having Γ k − as an invariant compact hyperbolic set. Moreover, suppose that Q (1) hasa single globally attracting rest point in Π ( a, b ), z j ∈ ( j − β, j + β ), where j = 1 , ..., m and β ∈ (0 , χ k ( z ) be smooth functions of z ∈ R such that χ k ( z l ) = δ lk , l ∈ { , ..., m } , k = 1 , ..., m where δ lk stands for the Kronecker delta. Let Q ( v, z ) be the vector field on Π ( a, b ) × [0 , m + β ]defined by Q i ( v, z ) = m (cid:88) k =1 Q ( k ) i χ k ( z ) , i ∈ { , ..., n } , (8.15)for first n components and n + 1-th component of this field (denoted by z ) is defined by Q n +1 ( v, z ) = F m ( z, β, κ ) , (8.16)where F m is defined by (8.14). For β ∈ (0 ,
1) the function F m has stable roots at the points z = 1 , , ..., m . We observe that the equation for z -component dz/dt = F m ( z, β, κ ) does notinvolve v . By applying Lemma 8.2 we note that solutions z ( t, z (0)) of the Cauchy problem for this15ifferential equation verify | z ( t ) − z j | < exp( − c t ) , if z (0) lies in an open neighbourhood of z j . Toconclude the proof, we consider the system dv i /dt = Q i ( v, z ) , i = 1 , ..., n,dz/dt = F m ( z, β, κ ) − ξ ¯ λz = Q n +1 ( z ) . The right hand sides of this system define the field Q of dimension n + 1 from the assertion ofLemma 8.3. To check this fact, we apply Lemmas 8.1 and 8.2 that completes the proof.Next, to finish the proof of Theorem 6.3, let us take a box Π ( a, b ), where 0 < a i < b i . Thesemiflows defined by differential equations dv/dt = δQ ( v ) are orbitally topologically equivalent forall δ >
0. We approximate the first n components of the field Q by our neural network usingLemma 8.3 . We multiply here Q on an appropriate positive δ to have a field with componentsbounded by sufficiently small number in order to apply Lemma 8.1. Namely, we take δ such that a i > δ/ ( ξ ¯ λ i ) and b i < (1 − δ ) / ( ξ ¯ λ i ) and apply Lemma 8.1. Note that this appoximation doesnot involve the control parameter ξ . Indeed, this parameter is involved only in the approximationof Q n +1 , which can be done independently, see the distar graph lemma 8.2. This concludes theproof of Theorem 6.3. Remark . In Theorem 6.3, we assume that the vector field Q ( v ) is given. However, by centralizednetworks we can solve the problem of identification of dynamical systems supposing that thetrajectories v ( t ) are given on a sufficiently large time interval whereas Q is unknown or we know thisfield only up to unknown parameters. An example, where we consider an identification constructionfor a modified noisy Lorenz system, can be found in section 9. Let us refer to the distar centers as hubs and to periphery nodes as satellites. We suppose thatsatellites do not interact each with others and a satellite interacts only with the correspondinghub. Therefore the interaction graph resulting from the ”hub disconnecting” construction consistsof n disconnected distar motifs. Step 1.
Let n = 1. We apply lemma 8.2 to the distar graphs, see the proof of the previoustheorem. Then we have m stable equilibria, where m is the number of satellites in the distarmotif. Step 2 . In the case n > n distarmotifs, where the j -th distar motif contains m j nodes. One has m + m + ... + m n = N − n andtotally the graph consists of N nodes. For each distar we adjust the parameters as above (see step1). We obtain thus m m ...m n of equilibria and the theorem is proven. The proof of Theorem 6.3 can be used to construct practically feasible algorithms, which solve theproblem of construction of a switchable network with prescribed dynamical properties. As a matterof fact, we can address two different, but related problems. The first problem is the synthesis of aneural network with prescribed attractors and switchability properties. The second problem is the identification of a neural network from time series. First we state the solution of the first problemand after we describe how to resolve the second one by analogous methods.The prescribed network properties for the synthesis problem are stated in Theorem 6.3. Wedescribe here a step by step algorithm, allowing to construct a network with these properties.16onsider structurally stable dynamical systems defined by the equations dv/dt = Q ( l ) ( v ) v = ( v , ..., v n ) ∈ Π ( a, b ) ⊂ R n , (9.1)where l = 1 , ..., m and Π ( a, b ) is a defined by (6.1). We suppose that the fields Q ( l ) ( v ) aresufficiently smooth, for example, Q ( l ) ∈ C ∞ ( Π ( a, b )). Without any loss of generality we canassume that 1 < a i < b i , (9.2)(otherwise we can shift variables v i setting v i = ˜ v i − c i ). Step 1 . Find a sufficiently small (cid:15) such that perturbations of vector fields Q ( v ) ( l ) , which are (cid:15) small in C norm, do not change topologies of semiflows defined by 9.1. Actually, it is hard tocompute such a value of (cid:15) , so, in practice we simply choose a small (cid:15) by the trial and error method. Step 2.
We find a vector field Q ( v, z ) with n +1 components, where z = v n +1 ∈ [ a n +1 , b n +1 ] ⊂ R such that the first n components of Q ( v, z ) are defined by relations (8.15) and the n + 1 componentis defined by (8.16). Let D = Π ( a, b ) × [ a n +1 , b n +1 ].To describe the next steps, first let us introduce the functions G j (¯ v, P ) = N (cid:88) i =1 ¯ A ji σ ( B i ¯ v − h i ) , (9.3)where the parameter P = { N, ¯ A ji , B ik , h j , j = 1 , ..., n + 1 , i, k = 1 , ..., N } and ¯ v = ( v , ..., v n , z ).Let us observe that dynamical systems dq/dt = Q ( q ) and dq/dt = γQ ( q ) with γ > Q we can use γQ . We choosea γ > δ < − δ < γQ i (¯ v ) < δ, ¯ v ∈ D, i = 1 , ..., n + 1 (9.4)and a i > δ/λ i , b i < (1 − δ ) /λ i i = 1 , ..., n + 1 (9.5)for λ i > < γQ j (¯ v ) + λ j ¯ v j < , ¯ v ∈ D, j = 1 , ..., n + 1 . (9.6)Let σ − be the function inverse to σ . Due to (9.6) the functions R j (¯ v ) = σ − ( γQ j (¯ v ) + λ j ¯ v j ) (9.7)are correctly defined and smooth on D .Now we solve the following approximation problem. To find the number N , the matrices ¯ A , B and vector h such that | R j (¯ v ) − G j (¯ v, P ) | + | D ¯ v ( R j (¯ v ) − G j (¯ v, P )) | ≤ (cid:15)/ , j = 1 , ..., n + 1 . (9.8)This problem can be resolved by standard algorithms, which perform approximations of func-tions by multilayered perceptrons [5]. Note that these standard methods are based on iterationprocedures, which can use a large running time.We describe here a new variant of the algorithm for this approximation problem, which uses awavelet-like approach. This approach does not exploit any iteration procedures or linear systemsolving. All the procedure reduces to a computation of the Fourier and wavelet coefficients. How-ever, this algorithm is numerically effective only for sufficiently smooth R j with fast decreasingFourier coefficients and for not too large dimensions n .17he solution of the approximation problem (9.8) proceeds in the two steps. Step 3 . We reduce the n + 1-dimensional problem (9.8) to a set of one-dimensional ones asfollows. Let us approximate the functions R j by the Fourier expansion:sup ¯ v ∈ D ( | R j (¯ v ) − ˆ R j (¯ v ) | + |∇ ¯ v ( R j (¯ v ) − ˆ R j (¯ v )) | ) < (cid:15)/ , (9.9)where ˆ R j (¯ v ) = (cid:88) k ∈ K D ˆ R j ( k ) exp( i ( k, ¯ v )) , (9.10)( k, ¯ v ) = k ¯ v + k v + ... + k n +1 ¯ v n +1 and the set K D of vectors k is a finite subset of the ( n + 1)-dimensional lattice L D K D ⊂ L D = { k = ( k , ..., k n +1 ) : k i = ( a i − b i ) − πm i f or some m i ∈ Z } . (9.11)The Fourier coefficients ˆ R j ( k ) can be computed byˆ R j ( k ) = ( volume ( D )) − (cid:90) D R j (¯ v ) exp( − i ( k, ¯ v )) d ¯ v. In order to satisfy (9.9), we take a sequence of extending sets K D . For some K D relation (9.9) willbe satisfied because the Fourier coefficients ˆ R j ( k ) fastly decrease in | k | . Step 4 . We exploit the fact that the problem (9.8) is linear with respect to the coefficients ¯ A ij .For each k ∈ K D we resolve the following one-dimensional problem. Let g ( q, M, a, β, ¯ h ) = M (cid:88) i =1 a i σ ( β i ( q − ¯ h i )) . (9.12)We are seeking for integer M > a = ( a , ..., a M ), β = ( β , ..., β M ) and¯ h = (¯ h , ..., ¯ h M ) such thatsup q ∈ I k | W j,k ( q ) − g ( q, M, a, β, ¯ h ) | < (cid:15) (10 | K D | ) − , (9.13)sup q ∈ I k | dW j,k ( q ) /dq − g (cid:48) ( q, M, a, β, ¯ h ) | < (cid:15) ≤ (cid:15) (10 | K D | ) − , (9.14)where | K D | is the number of the elements k in the set K D , W j,k ( q ) = ˆ R j ( k ) exp( iq ) ,g (cid:48) ( q, M, a, β, ¯ h ) = M (cid:88) i =1 a i σ (cid:48) ( β i ( q − ¯ h i )) , (9.15)and q = ( k, ¯ v ) ∈ I k , where I k is the interval [ q − ( k ) , q + ( k )] with q − ( k ) = min ¯ v ∈ D ( k, ¯ v ) , q + ( k ) = min ¯ v ∈ D ( k, ¯ v ) . These approximation problems are indexed by ( j, k ), where j = 1 , ..., n + 1 and k ∈ K D (wetemporarily omit dependence on ( j, k ) in a, β, ¯ h, M to simplify notation).To resolve these one-dimensional approximation problems, we apply a method based on thewavelet theory. Notice that this method is numerically effective. First we observe that if (9.14) isfulfilled with a sufficiently small (cid:15) , then, to satisfy (9.13), it is sufficient to add a constant termof the form a M +1 σ ( b M +1 q ) with b M +1 = 0 to the sum in the right hand side of (9.12).18et us define the function ψ by ψ ( q ) = σ (cid:48) ( q ) − σ (cid:48) ( q − . (9.16)We observe that (cid:90) ∞−∞ ψ ( q ) dq = 0 (9.17)and ψ ( q ) → | q | → ∞ , therefore, ψ is a wavelet-like function.Let us introduce the following family of functions indexed by the real parameters r, h : ψ r,ξ ( q ) = | r | − / ψ ( r − ( q − ξ )) . (9.18)For any f ∈ L ( R ) we define the wavelet coefficients T f ( r, ξ ) of the function f by T f ( r, ξ ) = (cid:104) f, ψ r,ξ (cid:105) = (cid:90) ∞−∞ dqf ( q ) ψ r,ξ ( q ) . (9.19)For any smooth function f with a finite support I R = ( − R, R ) one has the following fundamentalrelation: f = c ψ (cid:90) ∞ (cid:90) ∞−∞ r − drdξT f ( r, ξ ) ψ r,ξ = f wav . (9.20)for some constant c ψ . This equality holds in a weak sense: the left hand side and the right handside define the same linear functionals on L ( R ), i.e., for each smooth, well localized g one has (cid:104) f, g (cid:105) = (cid:104) f wav , g (cid:105) . Let δ ( (cid:15) ) << (cid:15) be a small positive number. According to (9.20) we can find positive integers p , p , points r , ..., r p , ξ , ..., ξ p and a constant ¯ c ψ such that the integral in the right hand side of(9.20) can be approximated by a finite sum:sup | f ( q ) − ¯ f wav ( q ) | < δ, (9.21)where ¯ f wav = ¯ c ψ p (cid:88) l =1 p (cid:88) l =1 r − l T f ( r l , ξ l ) ψ r l ,ξ l . In our case for each ( j, k ) we set f = W j,k ( q ) for q ∈ I k and f = 0 for q / ∈ I k . We can take r l = r + l /p , where r + is large enough, and ξ l = q min + ( q max − q min ) l /p , where q min < q − ( k ), q max > q + ( k ) are sufficiently large and l = 1 , ..., p , l = 1 , ..., p . We can renumerate the points( r l , ξ l ) by a single index l = 1 , ..., p , where p = p p , that gives us r l , ξ l and the wavelet coefficients T l = ¯ c ψ T f ( r l , ξ l ).Having p , r l , ξ l and the wavelet coefficients T l , we obtain the following solution of the approx-imation problem (9.12): M ( j, k ) = p, ¯ h l − ( j, k ) = r − l ξ l , ¯ h l ( j, k ) = r − l ( ξ l + 1) ,β l − ( j, k ) = β l ( j, k ) = r − l , a l − ( j, k ) = − a l ( j, k ) = T l , where we have introduced the index ( j, k ) in notation for the solution ( M, a, β, ¯ h ) to emphasizethat problem (9.12) depends on this index.Finally, in the end of this step we obtain the coefficients M ( j, k ) , a ( j, k ) , ..., a M ( j,k ) ( j, k ) , β ( j, k ) , ..., β M ( j,k ) ( j, k ) , ¯ h ( j, k ) , ..., ¯ h M ( j,k ) ( j, k ) . (9.22)19 tep 5. We construct a network with n + 1 centers ¯ v , ..., ¯ v n +1 and N satellites as follows. Let C = 0 and D = 0, i.e., we assume that the satellites don’t interact among themselves and thereare no direct interactions between the centers. The number of satellites is defined by N = n +1 (cid:88) j =1 (cid:88) k ∈ K D M ( j, k ) . Each satellite can be equipped with a triple index ( i, j, k ), where j = 1 , ...n + 1, k ∈ K D and i ∈ { , ...M ( j, k ) } . We set that all h j = 0, ˜ λ i = 1, and λ j are chosen as above. The threshold h i,j,k for the satellite with the index ( i, j, k ) is defined by h i,j,k = ¯ h i ( j, k )where ¯ h i ( j, k ) are obtained at the Step 4 (see (9.22)).Furthermore, we define the matrices ¯ A and B as follows. One has B ( i,j,k ) ,l = β i ( j, k ) k l , (this relation describes an action of the l -th center on the satellite with index ( i, j, k )) and¯ A l, ( i,j,k ) = a l ( j, k )(this relation describes an action of the l -th center on the satellite with index ( i, j, k )). Here i ∈ { , ...M ( j, k ) } , j, l = 1 , ..., n + 1 and k ∈ K D . Remark . This algorithm can be simplified if instead networks (4.1), (4.2) we use analogousnetworks where satellites act on centers in a linear way: dw i dt = σ (cid:16) B i v + C i w − ˜ h i (cid:17) − κ − ˜ λ i w i , (9.23) dv j dt = ( A j w − h j ) − λ j v j , (9.24)where i = 1 , ..., N , j = 1 , ..., n , and the fields Q ( l ) are defined by polynomials (note that Jackson’stheorems [1] guarantee that any Q can be approximated by a polynomial field on Π ( a, b ) in C -norm). Then we can simplify Step 3 and Step 4 of the algorithm as follows. We observe that wecan set γ = 1 and in this case the functions R j have the form R j (¯ v ) = Q j (¯ v ) + λ j ¯ v j . (9.25)On Step 3 for polynomial functions R j ( v ) we can also use simple algebraic transformations, in-stead of the Fourier decomposition, to reduce the multidimensional approximation problem to onedimensional ones. On step 4 the function ψ defined by (9.16) is well localized and therefore alter-natively step 4 can be realized by standard programs using radial basic functions and the methodof least squares (see an example on the Lorenz system below).Let us turn now to the problem of identification of a neural network from time series producedby a dynamical system dv/dt = Q ( v, P ), v ∈ R n with unknown parameters P . Assume that weobserve a time series v ( t ) , v ( t ) , ..., v ( t K ) and the time interval between observations is small: t i +1 − t i = ∆t <<
1. We want to construct a network with n centers, which produces, in a sense,analogous time series. According to (3.4), a suitable criterion of trajectory similarity is as follows.We can approximate the averages S Q,φ from (3.3) by the time series S Q, P ,φ ≈ K − ∆T K (cid:88) k =1 φ ( v ( t k )) = S ( K ) Q, P ,φ . (9.26)20hen, if the network identification is correct, the averages defined by time series and the cor-responding ones generated by the approximating centralized neural network, should be close forsmooth weight functions φ : | S ( K ) Q, P ,φ − S ( K ) G anN ,φ | = Err approx < δ ( φ ) << , (9.27)where G anN is the approximation of Q by the neural network.As a first step, we can approximate the unknown field Q ( v ) by finite differences, for example,using the relation Q (˜ v i , P ) = ( v ( t i +1 ) − v ( t i )) ∆t − , ˜ v i = ( v ( t i +1 ) + v ( t i )) / . (9.28)For other values v the field Q can be reconstructed, for example, by a linear interpolation. Theneural network approximation of Q can be obtained by applying the steps 2-5 of the synthesisalgorithm described above.We end this section with an illustration of the simplified variant of the identification andsynthesis algorithm, see the preceding Remark.As an example, we describe a solution of the following identification problem. Consider timeseries generated by the Lorenz system perturbed by noise. The Lorenz system involves a controller parameter. Adjusting the values of this parameter, we can obtain chaotic dynamics, time periodicone or dynamics with convergent trajectories. We are going to find a centralized network, whichalso has a controller parameter and can generate all this rich variety of trajectories. For chaoticand periodic trajectories this neural approximation should exhibit dynamics with analogous ergodicproperties (in the sense of (9.27).Recall that the Lorenz system has the form dx/dt = α ( y − x ) , dy/dt = x ( ρ − z ) − y, dz/dt = xy − βz. (9.29)This system shows a chaotic behaviour for α = 10 , β = 8 / ρ = 28. For α = 10 , β = 8 / ρ ∈ (0 ,
1) this system has a globally attracting rest point.We introduce new variables v = x, v = y, v = z and v = ρ and consider a more complicatedmodified Lorenz system with a controller parameter: (compare with the proof of Theorem 6.3): dv /dt = α ( v − v ) = f , dv /dt = r v ( v − v ) − r v = f , (9.30) dv /dt = r v v − βz = f , dv /dt = σ H ( v , b , h ) − ξv = f , (9.31)where σ H is a regularized step function defined by H ( w ) = (1 + exp( − b ( w − h )) − with b >> h = 1. We set ξ = 0 . r = 14 , r = 1 , r = 1. The initial data for the fourth component v = v (0) is a controller parameter. For large b the differential equation for v has two stableequilibria: v − ≈ v +4 ≈
2. Therefore, for v ∈ (0 ,
1) system (9.30), (9.31) has a globallyattracting rest point and for v > P = ( α, β, r , r , r ).Suppose we observe trajectories v ( t ), t ∈ [0 , T ] of system (9.30) at some time moments t =0 , t = dt, ..., t p = p∆t . In order to simulate experimental errors we have perturbed the systemwith additive noise. We are going to find a centralized network, which has an attractor with,in a sense, similar statistical characteristics. More precisely, we aim to minimize Err approx fromrelation (9.27). For identification procedure we use a centralized network with 4 centers v , v , v and v . In this case steps 3, 4 can be simplified if we use this specific form of the modified Lorenzsystem. The last center v serves as a controller.We state the algorithm for the modified Lorenz system, however, the method is general and fea-sible for identification by trajectories generated by all low-dimensional dynamical systems definedby polynomial vector fields. 21irst we set C = D = 0 . (9.32)This means that only satellites act on centers and vice versa. To find the matrices A , B and thethresholds h i , we solve the following approximation problems: R ( A , B , h ) → min, R = (cid:88) i =1 p (cid:88) j =1 ( Q i ( t j ) − S i ( v ( t j , A , B , h )) (9.33)where Q i ( t j ) = ( v i ( t j + ∆t ) − v i ( t j )) /∆t, S i ( v, A , B , h ) = N i (cid:88) k =1 A ik σ ( (cid:88) j =1 B kj v j − h ik ) . (9.34)This approximation problem is nonlinear with respect to B and h . We can simplify this problemby the following heuristic method. Each function f i ( v ) defined on a open bounded domain canbe represented as a linear combination of functions g l ( v · k li ), where vectors k li belong to a finiteset of vectors K i . For example, for system (9.30), (9.31) the components f j for j = 1 , , f j ( v ) = g j ( v ) − λ j v j , g j ( v ) = (cid:88) l =1 C ( j, l ) T l ( v ) (9.35)where T l = v l , l = 1 , , , T l +1 = ( v + v l ) , T l +2 = ( v − v l ) , l = 2 , , , T = 1 . and λ = α, λ = 1 , λ = β. Therefore, K = { k = (1 , , , } , K = { k = (1 , , , , k =(1 , , − , , k = (1 , , , , k = (1 , , , − } , K = { k = (1 , , , , k = (1 , − , , } , K = { k = (1 , , , n i be the number of the vectors contained in the set K i , n = 1 , n = 4, n = 2 and n = 1. In this case of the modified Lorenz system, the set K D from (9.11) is the unionof sets K i , i = 1 , ..., N L , a large b and define the auxiliary thresholds ¯ h k li ,j , where j = 1 , ..., N L , by¯ h k li ,j = min s =1 ,...,p,l ∈ K i v ( t s ) · k li + j ( max s =1 ,...,p,l ∈ K i v ( t s ) · k li − min s =1 ,...,p,l ∈ K i v ( t s ) · k li ) /N L. We seek coefficients ¯ A il, k li and C i , which minimize R i ( ¯ A , C i ) for i = 1 , , , R i ( ¯ A , C i ) → min, R i = p (cid:88) j =1 ( Q i ( t J ) − ˜ S i ( v ( t j ) , ¯ A , C i )) (9.36)where ˜ S i ( v, ¯ A , C ) = C i + n i (cid:88) l =1 N L (cid:88) j =1 ¯ A ij, k li σ ( b ( k li · v − ¯ h k li ,j )) . (9.37)Note that since ˜ S i are linear functions of ¯ A il, k li and C i , problems (9.36) can be solved by the leastsquare method. The important advantage of this approach is that approximations can be doneindependently for different components i .This approximation produces a centralized network involving 4 centers and N = 8 N L + 8satellites. Indeed, each vector k li associated with a quadratic term T l , gives us N L sattellites to22igure 4: This plot shows trajectories of v -component of the Lorenz system perturbed by noise (the solidcurve) and its neural approximation with N = 20 satellites (the dotted curve). The curves are not closebut they exhibit almost identical statistical properties ( Err approx = 0 .
008 (the white noise level is 0 . .
001 on the interval [0 , approximate this term. Moreover, we use 4 satellites for approximations of the linear terms and 4satellites are necessary for constants C i in the right hand sides of (9.37).The numerical simulations give the following results. The trajectories to identify are producedby the Euler method applied to the system (9.30), (9.31) perturbed by noise, where the time step0 .
005 on the interval [0 , (cid:15) N ω ( t i ), where ω ( t ) is the standard white noiseand (cid:15) N = 0 .
05. As a result of minimization procedure, we have obtained the errors R i of the order0 . − .
1. The trajectories of the system (9.30), (9.31) perturbed by noise and the correspondingneural networks are not close but they have a similar form and statistical characteristics that isconfirmed by the value
Err approx (defined by (9.27)), which is 0 . φ is φ ( v ) = v + v / − v . These results are illustrated by Fig. 4.
10 Conclusion and discussion
In this paper, we have proposed a complete analytic theory of maximally flexible and switchableHopfield networks. We shown that dynamics of a network with n slow components v , ..., v n canbe reduced to a system of n differential equations defined by a smooth n dimensional vector field F ( v ). If these slow components are hubs, i.e., they are connected with a number of other weaklyconnected nodes (satellites) and center-satellite interactions dominate inter-satellite forces, thenthe network becomes maximally flexible. Namely, by adjusting only center-satellite interactionswe can obtain smooth F of arbitrary forms.These networks are also maximally switchable. We describe networks of a special architecture,which contains a controller hub. By changing the state of this hub and the hub response timeparameter ξ one can completely change the network dynamics from an unique global attractivesteady state to any combination of periodic or chaotic attractors.Our results provide a rigorous framework for the idea that centralized networks are flexible.We also propose mechanisms for switching between attractors of these networks with controllerhubs. In functional genomics there are numerous examples when transitions between attractors ofgene regulatory networks can be triggered by controller proteins having multiple states sometimesresulting from interactions with micro-RNA satellites [8]. Similarly, neurons having multiple inter-23al states can trigger phase transitions of brain networks suggesting that single neuron activationcould be used for neural network control [15].The proofs of our results are constructive and are based on an algorithm allowing the networkreconstruction. This algorithm has several potential applications in biology. Identified networks canbe used to study emergent network properties such as robustness, controllability and switchability.Gene networks with the desired switchability properties could be build by synthetic biology toolsfor various applications in biotechnology. Furthermore, maximal switchable network models canbe used in neuroscience to relate structure and function in the brain activity, or in genetics toexplain how a minimal number of mutations can induce large phenotypic changes from one typeof adaptive behavior to another one. Acknowledgements
S.V. was financially supported by Government of Russian Federation, Grant 074-U01, alsosupported in part by grant RO1 OD010936 (formerly RR07801) from the US NIH and by grant -of Russian Fund of Basic Research. O.R. was supported by the Labex EPIGENMED (ANR-10-LABX-12-01). The authors are grateful to the anonymous referees for their useful remarks, thathelped improve the text.
References [1] Naum I Achieser.
Theory of approximation . Courier Corporation, 2013.[2] R´eka Albert and Albert-L´aszl´o Barab´asi. Statistical mechanics of complex networks.
Reviewsof modern physics , 74(1):47, 2002.[3] R´eka Albert, Hawoong Jeong, and Albert-L´aszl´o Barab´asi. Error and attack tolerance ofcomplex networks.
Nature , 406(6794):378–382, 2000.[4] Yaneer Bar-Yam and Irving R Epstein. Response of complex networks to stimuli.
Proceedingsof the National Academy of Sciences of the United States of America , 101(13):4341–4345,2004.[5] Andrew R Barron. Universal approximation bounds for superpositions of a sigmoidal function.
Information Theory, IEEE Transactions on , 39(3):930–945, 1993.[6] Jordi Bascompte. Networks in ecology.
Basic and Applied Ecology , 8(6):485–490, 2007.[7] Jean M Carlson and John Doyle. Complexity and robustness.
Proceedings of the NationalAcademy of Sciences , 99(suppl 1):2538–2545, 2002.[8] Richard W Carthew. Gene regulation by microRNAs.
Current opinion in genetics & devel-opment , 16(2):203–208, 2006.[9] Dante R Chialvo. Emergent complex neural dynamics.
Nature physics , 6(10):744–750, 2010.[10] Reuven Cohen, Keren Erez, Daniel Ben-Avraham, and Shlomo Havlin. Breakdown of theInternet under intentional attack.
Physical review letters , 86(16):3682, 2001.[11] Sean P Cornelius, William L Kath, and Adilson E Motter. Realistic control of networkdynamics.
Nature communications , 4, 2013.[12] Noah J Cowan, Erick J Chastain, Daril A Vilhena, James S Freudenberg, and Carl TBergstrom. Nodal dynamics, not degree distributions, determine the structural controllabilityof complex networks.
PloS one , 7(6):e38398, 2012.2413] Gustavo Deco and Viktor K Jirsa. Ongoing cortical activity at rest: criticality, multistability,and ghost attractors.
The Journal of Neuroscience , 32(10):3366–3375, 2012.[14] Roderick Edwards, Anne Beuter, and Leon Glass. Parkinsonian tremor and simplification innetwork dynamics.
Bulletin of mathematical biology , 61(1):157–177, 1999.[15] Shigeyoshi Fujisawa, Norio Matsuki, and Yuji Ikegaya. Single neurons can induce phase tran-sitions of cortical recurrent networks with multiple internal states.
Cerebral Cortex , 16(5):639–654, 2006.[16] Jianxi Gao, Yang-Yu Liu, Raissa M D’Souza, and Albert-L´aszl´o Barab´asi. Target control ofcomplex networks.
Nature communications , 5, 2014.[17] Jack K Hale.
Asymptotic behavior of dissipative systems , volume 25. American MathematicalSoc., 2010.[18] Dan Henry.
Geometric theory of semilinear parabolic equations , volume 840. Springer-Verlag,Berlin, 1981.[19] John J Hopfield. Neural networks and physical systems with emergent collective computationalabilities.
Proceedings of the National Academy of Sciences , 79(8):2554–2558, 1982.[20] John J Hopfield, David W Tank, et al. Computing with neural circuits- a model.
Science ,233(4764):625–633, 1986.[21] Sui Huang, Ingemar Ernberg, and Stuart Kauffman. Cancer attractors: A systems view oftumors from a gene network dynamics and developmental perspective.
Seminars in Cell & Developmental Biology , 20(7):869 – 876, 2009.[22] Hawoong Jeong, Sean P Mason, A-L Barab´asi, and Zoltan N Oltvai. Lethality and centralityin protein networks.
Nature , 411(6833):41–42, 2001.[23] Hawoong Jeong, B´alint Tombor, R´eka Albert, Zoltan N Oltvai, and A-L Barab´asi. The large-scale organization of metabolic networks.
Nature , 407(6804):651–654, 2000.[24] Tao Jia and Albert-L´aszl´o Barab´asi. Control capacity and a random sampling method inexploring controllability of complex networks.
Scientific reports , 3, 2013.[25] Angelo Valleriani J¨org Menche and Reinhard Lipowsky. Dynamical processes on dissortativescale-free networks.
EPL (Europhysics Letters) , 89(1):18002, 2010.[26] Yuri Kifer. General random perturbations of hyperbolic and expanding transformations.
Jour-nal dAnalyse Math´ematique , 47(1):111–150, 1986.[27] Ying-Cheng Lai. Controlling complex, non-linear dynamical networks.
National Science Re-view , 1(3):339–341, 2014.[28] Xin Li, Justin J Cassidy, Catherine A Reinke, Stephen Fischboeck, and Richard W Carthew.A microRNA imparts robustness against environmental fluctuation during development.
Cell ,137(2):273–282, 2009.[29] Zhaoping Li and JJ Hopfield. Modeling the olfactory bulb and its neural oscillatory process-ings.
Biological cybernetics , 61(5):379–392, 1989.[30] Ching Tai Lin. Structural controllability.
Automatic Control, IEEE Transactions on ,19(3):201–208, 1974. 2531] Yang-Yu Liu, Jean-Jacques Slotine, and Albert-L´aszl´o Barab´asi. Controllability of complexnetworks.
Nature , 473(7346):167–173, 2011.[32] Wolfgang Maass, Georg Schnitger, and Eduardo D Sontag. On the computational powerof sigmoid versus Boolean threshold circuits. In
Foundations of Computer Science, 1991.Proceedings., 32nd Annual Symposium on , pages 767–776. IEEE, 1991.[33] Eric Mjolsness, David H Sharp, and John Reinitz. A connectionist model of development.
Journal of theoretical Biology , 152(4):429–453, 1991.[34] Adilson E Motter. Networkcontrology.
Chaos: An Interdisciplinary Journal of NonlinearScience , 25(9):097621, 2015.[35] Tam´as Nepusz and Tam´as Vicsek. Controlling edge dynamics in complex networks.
NaturePhysics , 8(7):568–573, 2012.[36] Mark EJ Newman. The structure and function of complex networks.
SIAM review , 45(2):167–256, 2003.[37] H Allen Orr. The genetic theory of adaptation: a brief history.
Nature Reviews Genetics ,6(2):119–127, 2005.[38] B. Nicolaenko P. Constantin, C. Foias and R. Temam. Integral Manifolds and Inertial Man-ifolds for Dissipative Partial Differential Equations.
Springer-Verlag, Applies MathematicalSciences Series, , 70.[39] Yujian Pan and Xiang Li. Structural controllability and controlling centrality of temporalnetworks.
PloS one , 9(4):e94998, 2014.[40] Fabio Pasqualetti, Sandro Zampieri, and Francesco Bullo. Controllability metrics, limitationsand algorithms for complex networks.
Control of Network Systems, IEEE Transactions on ,1(1):40–52, 2014.[41] Tiago Pereira. Hub synchronization in scale-free networks.
Physical Review E , 82(3):036201,2010.[42] Tiago Pereira, Deniz Eroglu, G Baris Bagci, Ugur Tirnakli, and Henrik Jeldtoft Jensen.Connectivity-driven coherence in complex networks.
Physical review letters , 110(23):234103,2013.[43] Subramoniam Perumal and Ali A Minai. Stable-yet-switchable (sys) attractor networks. In
Neural Networks , pages 2509–2516, 2009.[44] Peter Pol´aˇcik. Complicated dynamics in scalar semilinear parabolic equations in higher spacedimension.
Journal of differential equations , 89(2):244–271, 1991.[45] David Ruelle.
Elements of differentiable dynamics and bifurcation theory . Elsevier, 2014.[46] Justin Ruths and Derek Ruths. Control profiles of complex networks.
Science , 343(6177):1373–1376, 2014.[47] CJ Stam, B Jelles, HAM Achtereekte, SARB Rombouts, JPJ Slaets, and RWM Keunen. In-vestigation of EEG non-linearity in dementia and Parkinson’s disease.
Electroencephalographyand clinical neurophysiology , 95(5):309–317, 1995.2648] Jake Stroud, Mauricio Barahona, and Tiago Pereira. Dynamics of Cluster Synchronisation inModular Networks: Implications for Structural and Functional Networks. In
Applications ofChaos and Nonlinear Dynamics in Science and Engineering-Vol. 4 , pages 107–130. Springer,2015.[49] Jie Sun and Adilson E Motter. Controllability transition and nonlocality in network control.
Physical Review Letters , 110(20):208701, 2013.[50] Michel Talagrand. Rigorous results for the hopfield model with many patterns.
Probabilitytheory and related fields , 110(2):177–275, 1998.[51] Toshi Tanizawa, Gerald Paul, Reuven Cohen, Shlomo Havlin, and H Eugene Stanley. Opti-mization of network robustness to waves of targeted and random attacks.
Physical review E ,71(4):047101, 2005.[52] SA Vakulenko. A system of coupled oscillators can have arbitrary prescribed attractors.
Journal of Physics A: Mathematical and General , 27(7):2335, 1994.[53] SA Vakulenko. Dissipative systems generating any structurally stable chaos.
Advances inDifferential Equations , 5(7-9):1139–1178, 2000.[54] Sergei Vakulenko and Ovidiu Radulescu. Flexible and robust patterning by centralized genenetworks.
Fundamenta Informaticae , 118(4):345–369, 2012.[55] Sergey A Vakulenko and Ovidiu Radulescu. Flexible and robust networks.
Journal of bioin-formatics and computational biology , 10(02):1241011, 2012.[56] M Viana. Dynamics : A Probabilistic and Geometric Perspective.
Documenta Mathematica ,Extra Volume ICM:557–578, 1998.[57] JI ˇR´I Vohradsk´y. Neural network model of gene expression.
The FASEB Journal , 15(3):846–854, 2001.[58] Fang-Xiang Wu, Lin Wu, Jianxin Wang, Juan Liu, and Luonan Chen. Transittability ofcomplex networks and its applications to regulatory biomolecular networks.
Scientific reports ,4, 2014.[59] Gang Yan, Jie Ren, Ying-Cheng Lai, Choy-Heng Lai, and Baowen Li. Controlling complexnetworks: How much energy is needed?
Physical review letters , 108(21):218703, 2012.[60] Lai-Sang Young. Stochastic stability of hyperbolic attractors.
Ergodic Theory and DynamicalSystems , 6(02):311–319, 1986.[61] Zhengzhong Yuan, Chen Zhao, Zengru Di, Wen-Xu Wang, and Ying-Cheng Lai. Exact con-trollability of complex networks.