Dávid Vincze

Incremental Rule Base Creation with Fuzzy Rule Interpolation-Based Q-Learning

Reinforcement Learning (RL) is a widely known topic in computational intelligence. In the RL concept the problem needed to be solved is hidden in the feedback of the environment, called rewards. Using these rewards the system can learn which action is considered to be the best choice in a given state. One of the most frequently used RL method is the Q-learning, which was originally introduced for discrete states and actions. Applying fuzzy reasoning, the method can be adapted for continuous environments, called Fuzzy Q-learning. An extension of the Fuzzy Q-learning method with the capability of handling sparse fuzzy rule bases is already introduced by the authors. The latter suggests a Fuzzy Rule Interpolation (FRI) method to be the reasoning method applied with Q-learning, called FRIQ-learning. The main goal of this paper is to introduce a method which can construct the requested FRI fuzzy model from scratch in a reduced size. The reduction is achieved by incremental creation of an intentionally sparse fuzzy rule base. Moreover an application example (cart-pole problem simulation) shows the promising results of the proposed rule base reduction method.

Interpolation based fuzzy automaton for human-robot interaction

Abstract One way of handling Human-Robot Interaction (HRI) is based on the concept, that the robot acts like an animal companion to human. According to this paradigm the Robot should not be molded to mimic the human being, and form human-to-human like communication, but to follow the existing biological examples and form inter-species interaction. The 20.000 year old human-dog relationship is a good example for this paradigm of the HRI, as interaction of different species. One good reason of this approach in HRI is the lack of the “uncanny valley” effect i.e. increasing similarity of robots to humans will actually increase the chances that humans refuse interaction (will be frightened). In this paper, for ethologically inspired HRI model implementation, a fuzzy model structure built upon the framework of low computational demand Fuzzy Rule Interpolation (FRI) methods and fuzzy automaton is suggested. The application of FRI methods fits well the conceptually “spare rule-based” structure of the existing descriptive verbal ethological models. (In case of the descriptive verbal ethological models, the “completeness” of the rule-base is not required). The main benefit of the FRI method adaptation in ethological model implementation is the fact, that it has a simple rule-based knowledge representation format. Because of this, even after numerical optimization of the model, the rules are still “human readable”, and helps the formal validation of the model by the ethological experts. On the other side due to the FRI base, the model has still low computational demand and fits directly the requirements of the embedded implementations. For demonstrating the applicability of the proposed structure, some components of a human-dog interaction FRI model, which also suitable for HRI, will be briefly introduced in this paper.

Fuzzy Rule Interpolation-based Q-learning

Reinforcement learning is a well known topic in computational intelligence. It can be used to solve control problems in unknown environments without defining an exact method on how to solve problems in various situations. Instead the goal is defined and all the actions done in the different states are given feedback, called reward or punishment (positive or negative reward). Based on these rewards the system can learn which action is considered the best in a given state. A method called Q-learning can be used for building up the state-action-value function. This method uses discrete states. With the application of fuzzy reasoning the method can be extended to be used in continuous environment, called Fuzzy Q-learning (FQ-Learning). Traditional Fuzzy Q-learning uses 0-order Takagi-Sugeno fuzzy inference. The main goal of this paper is to introduce Fuzzy Rule Interpolation (FRI), namely the FIVE (Fuzzy rule Interpolation based on Vague Environment) to be the model applied with Q-learning (FRIQ-learning). The paper also includes an application example: the well known cart pole (reversed pendulum) problem is used for demonstrating the applicability of the FIVE model in Q-learning.

Ethologically inspired robot behavior implementation

For implementing ethologically inspired robot behavior in this paper a platform based on fuzzy automaton (fuzzy state-machine) is suggested. It can react the human intervention as a function of the robot state and the human action. This platform is suitable for implementing quite complicated action-reaction sequences, like the interaction of human and an animal, e.g. a behavior of an animal companion to the human. The suggested fuzzy model structure built upon the framework of low computational demand Fuzzy Rule Interpolation (FRI) methods and fuzzy automaton. For demonstrating the applicability of the proposed structure, some components of an action-reaction FRI model, will be briefly introduced in this paper.

Fuzzy automaton based Human-Robot Interaction

A novel aspect of human-robot interaction (HRI) can be put on the basis, that the robot side is implemented on a state-machine (fuzzy automaton), which reacts the human intervention as a function of the robot state and the human action. This platform is suitable for implementing quite complicated action-reaction sequences, like the interaction of human and an animal, e.g. a behaviour of an animal companion to the human. According to this paradigm the robot can follow the existing biological examples and form inter-species interaction. The 20,000 year old human-dog relationship is a good example for this paradigm of the HRI, as interaction of different species. In this paper, for ethologically inspired HRI model implementation, a fuzzy model structure built upon the framework of low computational demand fuzzy rule interpolation (FRI) methods and fuzzy automaton is suggested. For demonstrating the applicability of the proposed structure, some components of a human-dog interaction FRI model, which also suitable for HRI, will be briefly introduced in this paper.

Ethologically inspired human-robot interaction interfaces

This paper presents human-robot interaction interfaces based on ethological studies. An ethological test procedure was modeled with the application of a fuzzy rule interpolation based fuzzy automaton. This fuzzy automaton was loaded with rules formed from the extracted ethological knowledge. Using the behaviours supplied by the fuzzy automaton as conclusions, different interfaces can be defined for the incarnation of the model. The ethological test procedure and its modeling technique based on the fuzzy automaton will be shortly introduced in the paper, and then the various human-robot interfaces based on the former will be presented. These include interfaces of simulated environments and also interfaces as real robot hardware with their supplemental devices (sensors, cameras, etc.).

Performance Optimization of the Fuzzy Rule Interpolation Method “FIVE”

Fuzzy Rule Interpolation (FRI) methods are efficient structures for knowledge-representation with relatively few rules. In spite of their good knowledge representation efficiency, their high computational demand makes the FRI methods hardly suitable for embedded real-time applications, for which short reasoning time has a high importance. On the other hand, the fact that currently available devices have increased computational power gives the FRI methods an opportunity to appear in real-time embedded applications. Therefore, the need for a low-computation and lowresource-demand FRI method is emerging. The goal of this paper is to introduce some implementation details of such an FRI method, together with its brief time and space complexity analysis. The paper also gives some hints for further performance optimization possibilities.

Using fuzzy rule interpolation based automata for controlling navigation and collision avoidance behaviour of a robot

Relatively few Fuzzy Rule Interpolation (FRI) techniques can be found among the practical fuzzy rule based applications. Many of them have limitations from the direct application point of view, for example they can be applied only in one dimensional case, or defined based on the two closest surrounding rules of the actual observation. Additionally the FRI methods can dramatically simplify the building of fuzzy rule bases by enabling the application of sparse rule bases. FRI methods can provide reasonable (interpolated) conclusions even if none of the existing rules fires under the current observation. These methods can help the expert to concentrate on the cardinal actions only. Compared to the classical fuzzy CRI, by omitting the derivable rules, the number of the fuzzy rules needed to be handled during the design process could be dramatically reduced. This paper provides a brief overview of several FRI methods and in more detailed an application oriented simple and quick FRI method FIVE will be introduced. For the demonstration of the benefits of the interpolation-based fuzzy reasoning as systematic approach, a robot navigation application is presented, where the robot is able to cycle through waypoints while avoiding collision with obstacles and walls. All the controlling parts were accomplished with fuzzy rule bases of the FIVE FRI method.

Declarative Language for Behaviour Description

The Fuzzy Rule Interpolation (FRI)-based Fuzzy Automaton is an efficient structure for describing complex behaviour models in a relatively simple manner. The goal of this paper is to introduce a novel declarative behaviour description language which is created for supporting special needs of ethologically inspired behaviour model definition. For the sake of simplicity, the grammar is created with as few keywords as possible, keeping the ability to describe complex behavioural patterns as well. The language is a declarative language mainly supporting the behaviour models built upon structures of interpolative fuzzy automata. The paper firstly presents the formal structure of the behaviour description language itself, then gives an overview of the interpreting and processing engine designed for the language. Finally, an application example, a definition of a set of behaviours and a simulated environment is also presented.

Fuzzy rule interpolation and reinforcement learning

Reinforcement Learning (RL) methods became popular decades ago and still maintain to be one of the mainstream topics in computational intelligence. Countless different RL methods and variants can be found in the literature, each one having its own advantages and disadvantages in a specific application domain. Representation of the revealed knowledge can be realized in several ways depending on the exact RL method, including e.g. simple discrete Q-tables, fuzzy rule-bases, artificial neural networks. Introducing interpolation within the knowledge-base allows the omission of less important, redundant information, while still keeping the system functional. A Fuzzy Rule Interpolation-based (FRI) RL method called FRIQ-learning is a method which possesses this feature. By omitting the unimportant, dependent fuzzy rules — emphasizing the cardinal entries of the knowledge representation — FRIQ-learning is also suitable for knowledge extraction. In this paper the fundamental concepts of FRIQ-learning and associated extensions of the method along with benchmarks will be discussed.