Chih Jui Lin

Backward Q-learning: The combination of Sarsa algorithm and Q-learning

Reinforcement learning (RL) has been applied to many fields and applications, but there are still some dilemmas between exploration and exploitation strategy for action selection policy. The well-known areas of reinforcement learning are the Q-learning and the Sarsa algorithms, but they possess different characteristics. Generally speaking, the Sarsa algorithm has faster convergence characteristics, while the Q-learning algorithm has a better final performance. However, Sarsa algorithm is easily stuck in the local minimum and Q-learning needs longer time to learn. Most literatures investigated the action selection policy. Instead of studying an action selection strategy, this paper focuses on how to combine Q-learning with the Sarsa algorithm, and presents a new method, called backward Q-learning, which can be implemented in the Sarsa algorithm and Q-learning. The backward Q-learning algorithm directly tunes the Q-values, and then the Q-values will indirectly affect the action selection policy. Therefore, the proposed RL algorithms can enhance learning speed and improve final performance. Finally, three experimental results including cliff walk, mountain car, and cart-pole balancing control system are utilized to verify the feasibility and effectiveness of the proposed scheme. All the simulations illustrate that the backward Q-learning based RL algorithm outperforms the well-known Q-learning and the Sarsa algorithm.

Integrated particle swarm optimization algorithm based obstacle avoidance control design for home service robot

Display Omitted A new adaptive PSO method is proposed and verified by simulations and a real robot.Our proposed method has been successful applied to three-dimensional obstacle avoidance with manipulator for the home service robot.Both the free-space and obstacle avoidance states are established for evaluations in computer simulations and real-time experiments. Our PSO-IAC algorithm has achieved outstanding performance compared to other methods in these experiments. This paper presents a new particle swarm optimization (PSO) algorithm, called the PSO-IAC algorithm, to resolve the goal of reaching with the obstacle avoidance problem for a 6-DOF manipulator of the home service robot. The proposed PSO-IAC algorithm integrates the improved adaptive inertia weight and the constriction factor with the standard PSO. Both the free-space and obstacle avoidance states are established for evaluations in computer simulations and real-time experiments. The performance comparisons of the PSO-IAC algorithm with respect to the existing inertia weighted PSO (PSO-W), constriction factor based PSO (PSO-C), constriction factor and inertia weighted PSO (PSO-CW), and adaptive inertia weighted PSO (PSO-A) algorithms are examined. Simulation results indicate that the PSO-IAC algorithm provides the fastest convergence capability. Finally, the proposed control scheme can make the manipulator of the home service robot arrive at the goal position with and without obstacles in all real-time experiments.

Development of an Automatic Emotional Music Accompaniment System by Fuzzy Logic and Adaptive Partition Evolutionary Genetic Algorithm

Music is everywhere in the world, and its applications in commerce are extremely versatile. Generally speaking, in order to create some music for background music, it is necessary to engage sound recordists and instrumental performers. However, the process is very time-consuming and costly. In this paper, a real-time emotion-based music accompaniment system is proposed to solve this issue. For different emotions, a fuzzy logic controller is designed to adjust the tempo of the music, and an adaptive partition evolutionary genetic algorithm is developed to create corresponding melodies. The chord progressions are generated via music theory, and the instrumentation is disposed by the conception of the probability. What is noteworthy is that all the processes can be output by Virtual Studio Technology in real time so that users can listen directly to the composing results from any emotions. From the experimental results, the proposed adaptive partition evolutionary genetic algorithm performs better than other optimal algorithms in such topics.

A Fast Color Information Setup Using EP-Like PSO for Manipulator Grasping Color Objects

A fast color information setup based on evolutionary programming (EP) like particles swarm optimization (EPSO) for the manipulator control system is examined in this paper. The first step for a manipulator to grasp and place color objects into the correct location is to correctly identify the RGB or the corresponding hue, saturation, value (HSV) color model. The commonly used method to determine the thresholds of HSV range is manual tuning, but it is time-consuming to find the best boundary to segment the color image. This paper proposes a new method to learn color information, which is executed by semiautomatic learning. At first, the watershed algorithm incorporates user interactions to segment the color image and obtain the target image. Then, the comparison between the target image and the original image is utilized to build a lookup table (LUT) of color information, where three HSV thresholds are learned by PSO methods. Because the convergence speed of well-known PSO algorithms is slow and may be stuck in the local minimum, we present the EPSO method realized by applying EP to the PSO method. Moreover, a novel approach is investigated to escape the local minimum supposing the particles are stuck in the local minimum. Finally, both the numerical and experimental results demonstrate that the developed approach can not only rapidly learn the thresholds to segment a color image but can also jump out the local minimum.

Design and implementation of a 4WS4WD mobile robot and its control applications

This paper mainly develops of a four-wheel steering and four-wheel drive (4WS4WD) mobile robot and examines its control applications. The 4WS4WD possesses the benefits of the 4WD structure and the advantages of a 4WS system, and has the better performance of lateral dynamics in comparison with traditional mobile robots. By handling the 4WS4WD mobile robot, the motion control, obstacle avoidance and control strategy are addressed in this paper. The robotic maneuver control for 4WS4WD mobile robot is firstly investigated. For obstacle avoidance, the dynamic window approach (DWA) is used to safely control the mobile robot. The A* algorithm is adopted to implement the path planning and mobile robot navigation system. The experimental results demonstrate that the realized mobile robot can successfully conquer many kinds of terrains and carry out all the tasks in the SKS Intelligent Security Robot Competition held in Taipei International Robot Show (TIROS).

Grasping Posture Control Design for a Home Service Robot using an ABC-based Adaptive PSO Algorithm

In this paper, a grasping posture control for a robotic arm is developed based on novel adaptive particle swarm optimization (PSO) for the home service robot. To grasp an object using the robotic arm of the home-service robot, both the spatial coordinates of the target and the appropriate collocation of the grasping posture should be examined. In this paper, we present another method for dealing with this problem, which integrates the artificial bee colony (ABC) algorithm into the adaptive particle swarm optimization (APSO) algorithm, where the mutation concept of the scout bee in the ABC algorithm is used to increase the diversity of the particles. In addition, adaptive acceleration coefficients and adaptive inertia weight are presented to ameliorate the convergence rate of the PSO algorithm. We name this control scheme AIWCPSO-S, which represents Adaptive Inertia Weight and acceleration Coefficients PSO with the aid of the Scout bee. Performance comparisons of existing ABC, global ABC, adaptive inertia weight PSO, low-discrepancy sequence initialized PSO algorithm with high-order nonlinear time-varying inertia weight (LHNPSO), oscillating triangular inertia weight PSO (OTIWPSO) and AIWCPSO-S algorithms are conducted by computer simulations. The experiment results show that the presented algorithm gives the most correct and fastest convergence capability.

Implementation of real-time object recognition system for home-service robot by integrating SURF and BRISK

This paper proposes a real-time object recognition systems for recognizing known objects and searching unknown objects for home service robot. The object recognition system is mainly used in recognizing known objects, which combined Compute Unified Device Architecture (CUDA), Speeded Up Robust Features (SURF) detector and Binary Robust Invariant Scalable Keypoints (BRISK) descriptor for improving the computation speed and decreasing the consumption on memory. On the other side, the visual perception system is usually used for searching unknown objects, which calculated the depth differences and found the contours of objects. The experimental results in the laboratory and the competition in robot@home league at RoboCup Japan Open 2013 Tokyo illustrate that the robot can successfully real-time recognize the known objects and search the unknown objects.

A migrant-inspired path planning algorithm for obstacle run using particle swarm optimization, potential field navigation, and fuzzy logic controller

Obstacle avoidance is an important issue in robotics. In this paper, the particle swarm optimization (PSO) algorithm, which is inspired by the collective behaviors of birds, has been designed for solving the obstacle avoidance problem. Some animals that travel to the different places at a specific time of the year are called migrants. The migrants also represent the particles of PSO for defining the walking paths in this work. Migrants consider not only the collective behaviors, but also geomagnetic fields during their migration in nature. Therefore, in order to improve the performance and the convergence speed of the PSO algorithm, concepts from the migrant navigation method have been adopted for use in the proposed hybrid particle swarm optimization (H-PSO) algorithm. Moreover, the potential field navigation method and the designed fuzzy logic controller have been combined in H-PSO, which provided a good performance in the simulation and the experimental results. Finally, the Federation of International Robot-soccer Association (FIRA) HuroCup Obstacle Run Event has been chosen for validating the feasibility and the practicability of the proposed method in real time. The designed adult-sized humanoid robot also performed well in the 2015 FIRA HuroCup Obstacle Run Event through utilizing the proposed H-PSO.