Yin Baoqun

Performance optimization of continuous-time Markov control processes based on performance potentials

Average-cost optimization problems for a class of continuous-time Markov control processes with a compact action set have been studied. The definition of a generalized average-cost Poisson equation, which can be viewed as an extension to the standard one is first given. Markov performance potentials are defined as its unique solution. Based on the formula of performance potentials, an average-cost optimality equation is derived and the existence of its solution is established. Then, policy iteration and value iteration algorithms are proposed and their convergence discussed. A numerical example for controlled closed queuing networks illustrates the application of the proposed value iteration algorithm.

Error bounds of optimization algorithms for semi-Markov decision processes

Caos work shows that, by defining an α-dependent equivalent infinitesimal generator A α, a semi-Markov decision process (SMDP) with both average- and discounted-cost criteria can be treated as an α-equivalent Markov decision process (MDP), and the performance potential theory can also be developed for SMDPs. In this work, we focus on establishing error bounds for potential and A α-based iterative optimization methods. First, we introduce an α-uniformized Markov chain (UMC) for a SMDP via A α and a uniformized parameter, and show their relations. Especially, we obtain that their performance potentials, as solutions of corresponding Poisson equations, are proportional, so that the studies of a SMDP and the α-UMC based on potentials are unified. Using these relations, we derive the error bounds for a potential-based policy-iteration algorithm and a value-iteration algorithm, respectively, when there exist various calculation errors. The obtained results can be applied directly to the special models, i.e., continuous-time MDPs and Markov chains, and can be extended to some simulation-based optimization methods such as reinforcement learning and neuro-dynamic programming, where estimation errors or approximation errors are common cases. Finally, we give an application example on the look-ahead control of a conveyor-serviced production station (CSPS), and show the corresponding error bounds.

The optimal robust control policy for uncertain semi-Markov control processes

The optimization problems of Markov control processes (MCPs) with exact knowledge of system parameters, in the form of transition probabilities or infinitesimal transition rates, can be solved by using the concept of Markov performance potential which plays an important role in the sensitivity analysis of MCPs. In this paper, by using an equivalent infinitesimal generator, we first introduce a definition of discounted Poisson equations for semi-Markov control processes (SMCPs), which is similar to that for MCPs, and the performance potentials of SMCPs are defined as solution of the equation. Some related optimization techniques based on performance potentials for MCPs may be extended to the optimization of SMCPs if the system parameters are known with certainty. Unfortunately, exact values of the distributions of the sojourn times at some states or the transition probabilities of the embedded Markov chain for a large-scale SMCP are generally difficult or impossible to obtain, which leads to the uncertainty of the semi-Markov kernel, and thereby to the uncertainty of equivalent infinitesimal transition rates. Similar to the optimization of uncertain MCPs, a potential-based policy iteration method is proposed in this work to search for the optimal robust control policy for SMCPs with uncertain infinitesimal transition rates that are represented as compact sets. In addition, convergence of the algorithm is discussed.The optimization problems of Markov control processes (MCPs) with exact knowledge of system parameters, in the form of transition probabilities or infinitesimal transition rates, can be solved by using the concept of Markov performance potential which plays an important role in the sensitivity analysis of MCPs. In this paper, by using an equivalent infinitesimal generator, we first introduce a definition of discounted Poisson equations for semi-Markov control processes (SMCPs), which is similar to that for MCPs, and the performance potentials of SMCPs are defined as solution of the equation. Some related optimization techniques based on performance potentials for MCPs may be extended to the optimization of SMCPs if the system parameters are known with certainty. Unfortunately, exact values of the distributions of the sojourn times at some states or the transition probabilities of the embedded Markov chain for a large-scale SMCP are generally difficult or impossible to obtain, which leads to the uncertainty of the semi-Markov kernel, and thereby to the uncertainty of equivalent infinitesimal transition rates. Similar to the optimization of uncertain MCPs, a potential-based policy iteration method is proposed in this work to search for the optimal robust control policy for SMCPs with uncertain infinitesimal transition rates that are represented as compact sets. In addition, convergence of the algorithm is discussed.

Estimation of System Power Consumption on Mobile Computing Devices

The relationship between power consumption and parameters of system state on mobile computing devices is studied in this paper, using genetic algorithm and artificial neural network. Then based on this relationship, a run-time power consumption model is proposed to estimate the energy used on a per process basis. This result can help us to design an intrusion detection system for battery exhaustion attacks or give some advice on how to design a less power consumption program on mobile computing devices.

Synchronization of nonlinear systems with stair-step signal

From the viewpoint of computer control, the synchronization of nonlinear systems is investigated. The controlled synchronization problem with stair-step signal is presented. The synchronization error controlled by the stair-step signal is analyzed and it is proved with Lyapunov method that if the Euclid norm of the control signal is no more than the norm of the synchronization error times a constant, and the signal makes the Euler approximate model of the error exponentially stable, then the stair-step signal will make the slave synchronized with the master.

Continuous-time Hidden Markov models in Network Simulation

The use of continuous-time hidden Markov models for network protocol and application performance evaluation has been validated to simulate network environments. In this paper, we develop a better algorithm to infer the continuous-time hidden Markov model from a series of end-to-end delay and loss observation of probing packets. We prove the algorithms feasibility by theory deduction and realize numerable validation by comparing the probability of the observed sequence produced by the model inferred by different methods. The algorithm complexity is lower.

An event-driven dynamic load balancing strategy for streaming media clustered server systems

Based on stochastic switching model, an event-driven dynamic load balancing strategy is presented for the streaming media clustered server systems. This strategy increases the server cluster availability by balancing the workloads among the servers within a cluster. Additionally, it improves the access hit ratio of cached files in delivery servers to alleviate the limitation of I/O bandwidth of storage node. First, the load balancing problem is formulated as a two-layer semi-Markov switching state-space control process. Then, an online policy iteration algorithm is proposed to optimize the file grouping policy. By utilizing the features of the event-driven policy, the proposed optimization algorithm is adaptive and with less computational cost. Simulation results demonstrate the effectiveness of the proposed approach.

Online Adaptive Optimization Algorithm for Semi-Markov Control Processes

Semi-Markov control problems with unknown kernel are considered, a reinforcement learning based online adaptive optimization algorithm is proposed. First an event-driven stochastic switching model is introduced to formulate the semi-Markov control problems. Then by utilizing the features of event-driven policy an optimization algorithm that combines policy gradient estimation and stochastic approximation is derived. This algorithm can converge to global optimization without the explicit knowledge of the semi-Markov kernel. Moreover, this algorithm does not require the computation of performance potentials or other related quantities (e.g. Q-factors) and therefore saves computational cost significantly. Simulation results demonstrate the effectiveness of the proposed algorithm.

Optimization of Semi-Markov Switching State-space Control Processes for Network Communication Systems

Motivated by optimization of network communication systems, this paper presents an event-driven semi-Markov switching state-space control process with hierarchical dynamic architectures. First, the semi-Markov kernel of the switching control process is constructed, and the sensitivity formula for performance derivatives under average criterion is derived. Then, an online optimization algorithm that combines policy gradient estimation and stochastic approximation is proposed. This analytic model is with constructional flexibility and scalability, and the proposed optimization algorithm is adaptive and with less computational cost. Finally, as an illustrative example, the load balancing problem in a streaming media server cluster is formulated and addressed.

Stochastic Optimization for a Class of Hierarchical Unstructured P2P System

For a class of hierarchical unstructured P2P systems, a Markov Switching-Space model is introduced to describe their behavior of dynamic grouping. It is also formulated as a optimization problem based on Markov Decision Processes. And parameterized gradient algorithms are provided to optimize the performance, where two parameter reductions are mentioned. Following them are the corresponding simulations and discussions.