Praveen Paruchuri

Deployed ARMOR protection: the application of a game theoretic model for security at the Los Angeles International Airport

Security at major locations of economic or political importance is a key concern around the world, particularly given the threat of terrorism. Limited security resources prevent full security coverage at all times, which allows adversaries to observe and exploit patterns in selective patrolling or monitoring, e.g. they can plan an attack avoiding existing patrols. Hence, randomized patrolling or monitoring is important, but randomization must provide distinct weights to different actions based on their complex costs and benefits. To this end, this paper describes a promising transition of the latest in multi-agent algorithms -- in fact, an algorithm that represents a culmination of research presented at AAMAS - into a deployed application. In particular, it describes a software assistant agent called ARMOR (Assistant for Randomized Monitoring over Routes) that casts this patrolling/monitoring problem as a Bayesian Stackelberg game, allowing the agent to appropriately weigh the different actions in randomization, as well as uncertainty over adversary types. ARMOR combines three key features: (i) It uses the fastest known solver for Bayesian Stackelberg games called DOBSS, where the dominant mixed strategies enable randomization; (ii) Its mixed-initiative based interface allows users to occasionally adjust or override the automated schedule based on their local constraints; (iii) It alerts the users if mixed-initiative overrides appear to degrade the overall desired randomization. ARMOR has been successfully deployed since August 2007 at the Los Angeles International Airport (LAX) to randomize checkpoints on the roadways entering the airport and canine patrol routes within the airport terminals. This paper examines the information, design choices, challenges, and evaluation that went into designing ARMOR.

An efficient heuristic approach for security against multiple adversaries

In adversarial multiagent domains, security, commonly defined as the ability to deal with intentional threats from other agents, is a critical issue. This paper focuses on domains where these threats come from unknown adversaries. These domains can be modeled as Bayesian games; much work has been done on finding equilibria for such games. However, it is often the case in multiagent security domains that one agent can commit to a mixed strategy which its adversaries observe before choosing their own strategies. In this case, the agent can maximize reward by finding an optimal strategy, without requiring equilibrium. Previous work has shown this problem of optimal strategy selection to be NP-hard. Therefore, we present a heuristic called ASAP, with three key advantages to address the problem. First, ASAP searches for the highest-reward strategy, rather than a Bayes-Nash equilibrium, allowing it to find feasible strategies that exploit the natural first-mover advantage of the game. Second, it provides strategies which are simple to understand, represent, and implement. Third, it operates directly on the compact, Bayesian game representation, without requiring conversion to normal form. We provide an efficient Mixed Integer Linear Program (MILP) implementation for ASAP, along with experimental results illustrating significant speedups and higher rewards over other approaches.

Conflicts in teamwork: hybrids to the rescue

Today within the AAMAS community, we see at least four competing approaches to building multiagent systems: belief-desire-intention (BDI), distributed constraint optimization (DCOP), distributed POMDPs, and auctions or game-theoretic approaches. While there is exciting progress within each approach, there is a lack of cross-cutting research. This paper highlights hybrid approaches for multiagent teamwork. In particular, for the past decade, the TEAMCORE research group has focused on building agent teams in complex, dynamic domains. While our early work was inspired by BDI, we will present an overview of recent research that uses DCOPs and distributed POMDPs in building agent teams. While DCOP and distributed POMDP algorithms provide promising results, hybrid approaches help us address problems of scalability and expressiveness. For example, in the BDI-POMDP hybrid approach, BDI team plans are exploited to improve POMDP tractability, and POMDPs improve BDI team plan performance. We present some recent results from applying this approach in a Disaster Rescue simulation domain being developed with help from the Los Angeles Fire Department.

Towards a Formalization of Teamwork with Resource Constraints

Despite the recent advances in distributed MDP frameworks for reasoning about multiagent teams, these frameworks mostly do not reason about resource constraints, a crucial issue in teams. To address this shortcoming, we provide four key contributions. First, we introduce EMTDP, a distributed MDP framework where agents must not only maximize expected team reward, but must simultaneously bound expected resource consumption. While there exist single-agent constrained MDP (CMDP) frameworks that reason about resource constraints, EMTDP is not just a CMDP with multiple agents. Instead, EMTDP must resolve the miscoordination that arises due to policy randomization. Thus, our second contribution is an algorithm for EMTDP transformation, so that resulting policies, even if randomized, avoid such miscoordination. Third, we prove equivalence of different techniques of EMTDP transformation. Finally, we present solution algorithms for these EMTDPs and show through experiments their efficiency in solving application-sized problems.

Bayesian stackelberg games and their application for security at Los Angeles international airport

Many multiagent settings are appropriately modeled as Stackelberg games [Fudenberg and Tirole 1991; Paruchuri et al. 2007], where a leader commits to a strategy first, and then a follower selfishly optimizes its own reward, considering the strategy chosen by the leader. Stackelberg games are commonly used to model attacker-defender scenarios in security domains [Brown et al. 2006] as well as in patrolling [Paruchuri et al. 2007; Paruchuri et al. 2008]. For example, security personnel patrolling an infrastructure commit to a patrolling strategy first, before their adversaries act taking this committed strategy into account. Indeed, Stackelberg games are being used at the Los Angeles International Airport to schedule security checkpoints and canine patrols [Murr 2007; Paruchuri et al. 2008; Pita et al. 2008a]. They could potentially be used in network routing, pricing in transportation systems and many other situations [Korilis et al. 1997; Cardinal et al. 2005]. Although the follower in a Stackelberg game is allowed to observe the leader’s strategy before choosing its own strategy, there is often an advantage for the leader over the case where both players must choose their moves simultaneously. To see the advantage of being the leader in a Stackelberg game, consider the game with the payoff as shown in Table I. The leader is the row player and the follower is the column player. The only pure-strategy Nash equilibrium for this game is when the leader plays a and the follower plays c which gives the leader a payoff of 2. However, if the leader commits to a mixed strategy of playing a and b with equal (0.5) probability, then the follower will play d, leading to an expected payoff for the leader of 3.5.

Coordinating randomized policies for increasing security of agent systems

We consider the problem of providing decision support to a patrolling or security service in an adversarial domain. The idea is to create patrols that can achieve a high level of coverage or reward while taking into account the presence of an adversary. We assume that the adversary can learn or observe the patrolling strategy and use this to its advantage. We follow two different approaches depending on what is known about the adversary. If there is no information about the adversary we use a Markov Decision Process (MDP) to represent patrols and identify randomized solutions that minimize the information available to the adversary. This lead to the development of algorithms CRLP and BRLP, for policy randomization of MDPs. Second, when there is partial information about the adversary we decide on efficient patrols by solving a Bayesian–Stackelberg games. Here, the leader decides first on a patrolling strategy and then an adversary, of possibly many adversary types, selects its best response for the given patrol. We provide two efficient MIP formulations named DOBSS and ASAP to solve this NP-hard problem. Our experimental results show the efficiency of these algorithms and illustrate how these techniques provide optimal and secure patrolling policies. We note that these models have been applied in practice, with DOBSS being at the heart of the ARMOR system that is currently deployed at the Los Angeles International airport (LAX) for randomizing checkpoints on the roadways entering the airport and canine patrol routes within the airport terminals.

Self-Organized Criticality of Belief Propagation in Large Heterogeneous Teams

Large, heterogeneous teams will often be faced with situations where there is a large volume of incoming, conflicting data about some important fact. Not every team member will have access to the same data and team members will be influenced most by the teammates with whom they communicate directly. In this paper, we use an abstract model to investigate the dynamics and emergent behaviors of a large team trying to decide whether some fact is true. Simulation results show that the belief dynamics of a large team have the properties of a Self-Organizing Critical system. A key property of such systems is that they regularly enter critical states, where one additional input can cause dramatic, system wide changes. In the belief sharing case, this criticality corresponds to a situation where one additional sensor input causes many agents to change their beliefs. This can include the entire team coming to a “wrong” conclusion despite the majority of the evidence suggesting the right conclusion. Self-organizing criticality is not dependent on carefully tuned parameters, hence the observed phenomena are likely to occur in the real world.

Multiagent Communication Security in Adversarial Settings

In many exciting multiagent applications -- including future battlefields, law enforcement, and commerce -- agents must communicate in inherently or potentially hostile environments in which an adversaries disrupt or intercept the communication between agents for malicious purposes, but the wireless ad hoc networks often proposed for these applications are particularly susceptible to attack. Intelligent agents must balance network performance with possible harm suffered from an adversarys attack, while accounting for the broadcast nature of their communication and heterogenous vulnerabilities of communication links. Furthermore, they must do so when the adversary is also actively and rationally attempting to counter their efforts. We address this challenge in this paper by representing the problem as a game between a sender agent choosing communication paths through a network and an adversary choosing nodes and links to attack. We introduce a network-flow-based approach for compactly representing the competing objectives of network performance and security from adversary attack, and provide a polynomial-time algorithm for finding the equilibrium strategy for the sender. Through empirical evaluation we show how this technique improves upon existing approaches.

Effect of Humans on Belief Propagation in Large Heterogeneous Teams

Members of large, heterogeneous teams often need to interact with different kinds of teammates to accomplish their tasks, teammates with dramatically different capabilities to their own. While the role of humans in teams has progressively decreased with the deployment of increasingly intelligent systems, they still have a major role to play. In this chapter, we focus on the role of humans in large, heterogeneous teams that are faced with situations, where there is a large volume of incoming, conflicting data about some important fact. We use an abstract model of both humans and agents to investigate the dynamics and emergent behaviors of large teams trying to decide whether some fact is true. In particular, we focus on the role of humans in handling noisy information and their role in convergence of beliefs in large heterogeneous teams. Our simulation results show that systems involving humans exhibit an enabler-impeder effect, where if humans are present in low percentages, they aid in propagating information; however when the percentage of humans increase beyond a certain threshold, they seem to impede the information propagation.

Inter-cultural opponent behavior modeling in a POMDP based Automated Negotiating Agent

As the world gets increasingly networked, business and political negotiations take place between people of different cultures. Cross-cultural negotiations have been mainly studied empirically and there is a dearth of computational models of negotiation that incorporate the culture of the negotiators. In this chapter, we take the first steps towards building a partially observable Markov decision process (POMDP) based automated negotiation (PAN) agent, that takes the culture of the negotiators into account. We move away from the offer-counteroffer paradigm that is usually used in computational modeling of negotiation. We assume that apart from making offers, the agents can take other actions for seeking/providing information during negotiation. Our approach is motivated by the experimental findings that (a) during negotiation, people communicate their preferences and justification of their preferences apart from making direct offers and (b) cultural distinctions can be made between negotiating agents based on an abstract coding of their conversation. We show that in accordance with an existing cognitive theory of inter-cultural negotiation from behavioral psychology literature, we can construct a POMDP model of negotiation. A key challenge in developing the PAN agent is in obtaining the state transition function for the POMDP. We demonstrate that the state transition function can be built from transcripts of actual negotiations between people.