Faruk Polat

Multiagent reinforcement learning using function approximation

Learning in a partially observable and nonstationary environment is still one of the challenging problems in the area of multiagent (MA) learning. Reinforcement learning is a generic method that suits the needs of MA learning in many aspects. This paper presents two new multiagent based domain independent coordination mechanisms for reinforcement learning; multiple agents do not require explicit communication among themselves to learn coordinated behavior. The first coordination mechanism is the perceptual coordination mechanism, where other agents are included in state descriptions and coordination information is learned from state transitions. The second is the observing coordination mechanism, which also includes other agents in state descriptions and additionally the rewards of nearby agents are observed from the environment. The observed rewards and agents own reward are used to construct an optimal policy. This way, the latter mechanism tends to increase region-wide joint rewards. The selected experimented domain is adversarial food-collecting world (AFCW), which can be configured both as single and multiagent environments. Function approximation and generalization techniques are used because of the huge state space. Experimental results show the effectiveness of these mechanisms.

Efficient Automated Mining of Fuzzy Association Rules

Mining association rules is one of the important research problems in data mining. So, many algorithms have been proposed to find association rules in databases with either binary or quantitative attributes. One of these approaches is fuzzy association rules mining. However, most of the earlier algorithms proposed for mining fuzzy association rules assume that fuzzy sets are given. In this paper, we propose an automated method for autonomous mining of both fuzzy sets and fuzzy association rules. For this purpose, we first find fuzzy sets by using an efficient clustering algorithm, namely CURE, and then determine their membership functions. Finally, we decide on interesting fuzzy association rules. Experimental results show the efficiency of the presented approach for synthetic transactions.

Multi-agent real-time pursuit

In this paper, we address the problem of multi-agent pursuit in dynamic and partially observable environments, modeled as grid worlds; and present an algorithm called Multi-Agent Real-Time Pursuit (MAPS) for multiple predators to capture a moving prey cooperatively. MAPS introduces two new coordination strategies namely Blocking Escape Directions and Using Alternative Proposals, which help the predators waylay the possible escape directions of the prey in coordination. We compared our coordination strategies with the uncoordinated one against a prey controlled by Prey A*, and observed an impressive reduction in the number of moves to catch the prey.

Influence of Prior Knowledge in Constraint-Based Learning of Gene Regulatory Networks

Constraint-based structure learning algorithms generally perform well on sparse graphs. Although sparsity is not uncommon, there are some domains where the underlying graph can have some dense regions; one of these domains is gene regulatory networks, which is the main motivation to undertake the study described in this paper. We propose a new constraint-based algorithm that can both increase the quality of output and decrease the computational requirements for learning the structure of gene regulatory networks. The algorithm is based on and extends the PC algorithm. Two different types of information are derived from the prior knowledge; one is the probability of existence of edges, and the other is the nodes that seem to be dependent on a large number of nodes compared to other nodes in the graph. Also a new method based on Gene Ontology for gene regulatory network validation is proposed. We demonstrate the applicability and effectiveness of the proposed algorithms on both synthetic and real data sets.

Reengineering relational databases to object-oriented: constructing the class hierarchy and migrating the data

The object-oriented data model is predicted to be the heart of the next generation of database systems. Users want to move from old legacy databases into applying this new technology that provides extensibility and flexibility in maintenance. However, a major limitation on the wide acceptance of object-oriented databases is the amount of time and money invested on existing database applications, which are based on conventional legacy systems. Users do not want to loose the huge amounts of data present in conventional databases. This paper presents a novel approach to transform a given conventional database into an object-oriented database. It is assumed that the necessary characteristics of the conventional database to be re-engineered are known and available. The source of these characteristics might be the data dictionary and/or an expert in the given conventional database. We implemented a system that builds an understanding of a given conventional database by taking these characteristics as input and produces the corresponding object-oriented database as output. The system derives a graph that summarizes the conceptual model. Links in the graph are classified into inheritance links and aggregation links. This classification leads to the class hierarchy. Finally, we handle the migration of data from the conventional database to the constructed object-oriented database.

Closure maintenance in an object-oriented query model

An object-algebra is presented as a formal query model for object-oriented data models. The algebra serves not only to access and manipulate the structure and behavior of objects, but it also supports the creation of new objects and the introduction of new relationships into the schema. It provides a more powerful and flexible tool than messages for effectively dealing with complex situations and meeting associative access requirements. Operands as well as the results of operations in the proposed algebra are formally characterized as a pair of sets—a set of objects capturing the states and a set of message expressions comprised of sequences of messages modeling the object behavior. The closure property is achieved in a natural way by letting the results of operations possess the same characteristics as the operands in an algebra expression. Some operators of the algebra resemble those of the relational algebra but with different syntax and semantics. Additional operators are introduced to complement them. A class is shown to posses the properties of an operand by defining a set of objects and deriving a set of message expressions for it. Furthermore, the result of an object algebra expression is shown to have the characteristics of a class whose superclass/subclass relationships with its operand class(es) can be established providing a mechanism to properly and persistently place it in the class lattice (schema).

Limited-Damage A*: A path search algorithm that considers damage as a feasibility criterion

Pathfinding algorithms used in todays computer games consider the path length or a similar criterion as the only measure of optimality. However, these games usually involve opposing parties, whose agents can inflict damage on those of the others. Therefore, the shortest path in such games may not always be the safest one. Consequently, a new suboptimal offline path search algorithm based on the A^* algorithm was developed, which takes the threat zones in the game map into consideration. Given an upper limit as the tolerable amount of damage for an agent, this algorithm searches for the shortest path from a starting location to a destination, where the agent may suffer damage less than or equal to the specified limit. Due to its behavior, the algorithm is called Limited-Damage A^* (LDA^*). Performance of LDA^* was tested in randomly-generated maze-like grid-based environments of varying sizes, and in hand-crafted fully-observable environments, in which 8-way movement is utilized. Results obtained from LDA^* are compared with those obtained from Multiobjective A^* (MOA^*), which is a complete and optimal algorithm that yields exact (best) solutions for every case. LDA^* was found to perform much faster than MOA^*, yielding acceptable sub-optimality in path length.

A layered approach to learning coordination knowledge in multiagent environments

Multiagent learning involves acquisition of cooperative behavior among intelligent agents in order to satisfy the joint goals. Reinforcement Learning (RL) is a promising unsupervised machine learning technique inspired from the earlier studies in animal learning. In this paper, we propose a new RL technique called the Two Level Reinforcement Learning with Communication (2LRL) method to provide cooperative action selection in a multiagent environment. In 2LRL, learning takes place in two hierarchical levels; in the first level agents learn to select their target and then they select the action directed to their target in the second level. The agents communicate their perception to their neighbors and use the communication information in their decision-making. We applied 2LRL method in a hunter-prey environment and observed a satisfactory cooperative behavior.

Real-Time Edge Follow: A Real-Time Path Search Approach

Real-time path search is the problem of searching a path from a starting point to a goal point in real-time. In dynamic and partially observable environments, agents need to observe the environment to track changes, explore to learn unknowns, and search suitable routes to reach the goal rapidly. These tasks frequently require real-time search. In this paper, we address the problem of real-time path search for grid-type environments; we propose an effective heuristic method, namely a real-time edge follow alternative reduction method (RTEF-ARM), which makes use of perceptual information in a real-time search. We developed several heuristics powered by the proposed method. Finally, we generated various grids (random-, maze-, and U-type), and compared our proposal with real-time A*, and its extended version real-time A* with n-look-ahead depth; we obtained very significant improvements in the solution quality.

A Conflict Resolution-Based Decentralized Multi-Agent Problem Solving Model

Distributed Artificial Intelligence (DAI) is a subfield of artificial intelligence which is concerned with solving problems by using both AI techniques and distributed processing capabilities. Due to different goals, knowledge and viewpoints of agents, conflicts might arise at any phase of the problem-solving process. Managing diverse knowledge requires well-organized models of conflict resolution. In this paper, we present a computational model in which a set of knowledge-based agents cooperate for solving problems in the domain of engineering design. The model is based on the insights that each agent has its own conflict management knowledge which is separated from its domain level knowledge. Each agent has its own conflict management knowledge which is not accessible, or visible to others. In addition, there are no globally known conflict resolution strategies. The problem-solving environment allows a new problem solver to be added, or an existing one to be removed without requiring any modification on the rest of the system. The model is described by using an example in the domain of office design and it is compared with other systems.