Kalyanam Krishnamoorthy

State aggregation based linear programming approach to approximate dynamic programming

One often encounters the curse of dimensionality in the application of dynamic programming to determine optimal policies for controlled Markov chains. In this paper, we provide a method to construct sub-optimal policies along with a bound for the deviation of such a policy from the optimum through the use of restricted linear programming. The novelty of this approach lies in circumventing the need for a value iteration or a linear program defined on the entire state-space. Instead, the state-space is partitioned based on the reward structure and the optimal cost-to-go or value function is approximated by a constant over each partition. We associate a meta-state with each partition, where the transition probabilities between these meta-states can be derived from the original Markov chain specification. The state aggregation approach results in a significant reduction in the computational burden and lends itself to a restricted linear program defined on the aggregated state-space. Finally, the proposed method is bench marked on a perimeter surveillance stochastic control problem.

UAV search & capture of a moving ground target under delayed information

The optimal control of a UAV searching for a slower moving ground target on a road network is considered. To aid the UAV, the road network has been instrumented with Unattended Ground Sensors (UGSs). The target/intruder heads towards a protected goal region, oblivious of being tracked by the UAV. Whenever the intruder reaches an intersection in the road network, we assume that he randomly chooses his future course of action. The UGSs are placed at all the road junctions and are triggered when the intruder passes by. When the UAV arrives at an UGS, the UGS informs the UAV if and when the intruder passed by. The UAV does not have on-board capability to detect and identify the intruder and therefore must investigate the UGSs to learn the intruders location. When the intruder and the UAV are at an UGS location at the same time, the UGS is triggered and this information is instantly relayed to the UAV, which snaps a picture, thereby “capturing” the intruder. Sufficient conditions for guaranteed capture of the intruder, before he reaches the goal, and the corresponding UAV control policy, that leads to capture, are provided.

Optimal minimax pursuit evasion on a Manhattan grid

The optimal control of a pursuer searching for a slower moving evader on a Manhattan grid road network is considered. The pursuer does not have on-board capability to detect the evader and relies instead on Unattended Ground Sensors (UGSs) to locate the evader. We assume that all the intersections in the road network have been instrumented with UGSs. When an evader passes by an UGS location, it triggers the UGS and this time-stamped information is stored by the UGS. When the pursuer arrives at an UGS location, the UGS informs the pursuer if and when the evader passed by. When the evader and the pursuer arrive at an UGS location simultaneously, the UGS is triggered and this information is instantly relayed to the pursuer, thereby enabling “capture”.We derive exact values for the optimal worst case time to capture the evader on the Manhattan grid and the corresponding pursuit policy.

Bounding procedure for stochastic dynamic programs with application to the perimeter patrol problem

One often encounters the curse of dimensionality in the application of dynamic programming to determine optimal policies for controlled Markov chains. In this paper, we provide a method to construct sub-optimal policies along with a bound for the deviation of such a policy from the optimum via a linear programming approach. The state-space is partitioned and the optimal cost-to-go or value function is approximated by a constant over each partition. By minimizing a positive cost function defined on the partitions, one can construct an approximate value function which also happens to be an upper bound for the optimal value function of the original Markov Decision Process (MDP). As a key result, we show that this approximate value function is independent of the positive cost function (or state dependent weights; as it is referred to, in the literature) and moreover, this is the least upper bound that one can obtain; once the partitions are specified. We apply the linear programming approach to a perimeter surveillance stochastic optimal control problem; whose structure enables efficient computation of the upper bound.

Sub-Optimal Stationary Policies for a Class of Stochastic Optimization Problems Arising in Robotic Surveillance Applications

This paper deals with the development of sub-optimal decision making algorithms for a collection of robots in order to aid a remotely located human operator in the task of classification of incursions across a perimeter in a surveillance application. The operator is tasked with the classification of incursion as either a nuisance or a threat. Whenever there is an incursion into the perimeter, Unattended Ground Sensors (UGS) raise an alert and the robots service the alerts by visiting the alert location and collecting evidence in the form of video and other images and transmit them to the operator. There are two competing needs for a robot: it needs to spend more time at an alert location for aiding the operator in accurate classification and it needs to service the alerts as quickly as possible so that the evidence collected is relevant. A natural problem is to determine the optimal amount of time a robot must spend servicing an alert. In this paper, we discretize the problem spatially and temporally and recast the optimization problem as follows: Is it better for a robot to spend the next time interval at the alert location in terms of maximizing the expected, discounted payoff? The payoff associated with a state is an increasing function of the time spent by a robot servicing an alert and a decreasing function of the number of unserviced alerts. This problem can be easily be cast as a Markov Decision Process (MDP). However, the number of states runs into billions even for a modest size problem. We consider Approximate Dynamic Programming via linear programming as this approach provides an upper (and lower) bound on the optimal expected discounted payoff and enables the construction of a suboptimal policy. The bounds may then be used to provide an estimate of the quality of sub-optimal policy employed. We also provide a computationally tractable way of computing the lower bound using linear programming. Finally, numerical results supporting our method are provided.Copyright

UAV perimeter patrol operations optimization using efficient Dynamic Programming

A reduced order Dynamic Programming (DP) method that efficiently computes the optimal policy and value function for a class of controlled Markov chains is developed. We assume that the Markov chains exhibit the property that a subset of the states have a single (default) control action asso-ciated with them. Furthermore, we assume that the transition probabilities between the remaining (decision) states can be derived from the original Markov chain specification. Under these assumptions, the suggested reduced order DP method yields significant savings in computation time and also leads to faster convergence to the optimal solution. Most importantly, the reduced order DP has been shown analytically to give the exact same solution that one would obtain via performing DP on the original full state space Markov chain. The method is illustrated via a multi UAV perimeter patrol stochastic optimal control problem.

State partitioning based linear program for stochastic dynamic programs: An invariance property

Abstract A common approximate dynamic programming method entails state partitioning and the use of linear programming, i.e., the state-space is partitioned and the optimal value function is approximated by a constant over each partition. By minimizing a positive cost function defined on the partitions, one can construct an upper bound for the optimal value function. We show that this approximate value function is independent of the positive cost function and that it is the least upper bound, given the partitions.

Maximizing the throughput of a patrolling UAV by dynamic programming

This paper addresses the following base defense scenario: an Unmanned Aerial Vehicle (UAV) performs the task of perimeter alert patrol. There are m alert stations/sites located on the perimeter where a nearby breaching of the perimeter by an intruder can be sensed and is flagged by an Unattended Ground Sensor (UGS). We assume that the alert arrival process is Poisson. In order to determine whether an incursion flagged by a UGS is a false alarm or a real threat, a patrolling UAV flies to the alert site to investigate the alert. The decision problem for a UAV is to determine, in real-time, which station to head toward next, upon completion of a service, so as to maximize the system throughput or equivalently minimize the mean waiting time of an alert in the system. The throughput is defined as the number of messages/alerts serviced on average in unit time.

Pursuit on a graph using partial information

The optimal control of a “blind” pursuer searching for an evader moving on a road network and heading at a known speed toward a set of goal vertices is considered. To aid the pursuer, certain roads in the network have been instrumented with Unattended Ground Sensors (UGSs) that detect the evaders passage. When the pursuer arrives at an instrumented node, the UGS therein informs the pursuer if and when the evader visited the node. The pursuers motion is not restricted to the road network. In addition, the pursuer can choose to wait/loiter for an arbitrary time at any UGS location/node. At time 0, the evader passes by an entry node on his way towards one of the exit nodes. The pursuer also arrives at this entry node after some delay and is thus informed about the presence of the intruder/evader in the network, whereupon the chase is on - the pursuer is tasked with capturing the evader. Because the pursuer is blind, capture entails the pursuer and evader being collocated at an UGS location. If this happens, the UGS is triggered and this information is instantaneously relayed to the pursuer, thereby enabling capture. On the other hand, if the evader reaches one of the exit nodes without being captured, he is deemed to have escaped. We provide an algorithm that computes the maximum initial delay at the entry node for which capture is guaranteed. The algorithm also returns the corresponding optimal pursuit policy.

Moving ground target isolation by a UAV using predicted observations

An interesting variant of the cops and robbers pursuit-evasion problem is presented. The motivation for the problem comes from the need to monitor a network of roads leading to a protected area. In this paper, an unmanned aerial vehicle (UAV) is searching for a slower moving ground vehicle on a road network that has been out-fitted with unattended ground sensors (UGSs). The UGSs are communication constrained and can only relay observations of the ground vehicle when the UAV visits them. Furthermore, the UAV does not have on-board capability to detect and identify the evader and therefore must investigate the UGSs to learn the evaders location. This scenario leads the UAVs control actions to depend on partial information. A sufficient condition for guaranteed isolation of the intruder is provided along with the corresponding pursuit policy. Since the policy is non-unique, we select a policy that provides the least time to capture among all policies with the guarantee.