Yoshiki Matsuo

Cognitively inspired reinforcement learning architecture and its application to giant-swing motion control

Many algorithms and methods in artificial intelligence or machine learning were inspired by human cognition. As a mechanism to handle the exploration-exploitation dilemma in reinforcement learning, the loosely symmetric (LS) value function that models causal intuition of humans was proposed (Shinohara et al., 2007). While LS shows the highest correlation with causal induction by humans, it has been reported that it effectively works in multi-armed bandit problems that form the simplest class of tasks representing the dilemma. However, the scope of application of LS was limited to the reinforcement learning problems that have K actions with only one state (K-armed bandit problems). This study proposes LS-Q learning architecture that can deal with general reinforcement learning tasks with multiple states and delayed reward. We tested the learning performance of the new architecture in giant-swing robot motion learning, where uncertainty and unknown-ness of the environment is huge. In the test, the help of ready-made internal models or functional approximation of the state space were not given. The simulations showed that while the ordinary Q-learning agent does not reach giant-swing motion because of stagnant loops (local optima with low rewards), LS-Q escapes such loops and acquires giant-swing. It is confirmed that the smaller number of states is, in other words, the more coarse-grained the division of states and the more incomplete the state observation is, the better LS-Q performs in comparison with Q-learning. We also showed that the high performance of LS-Q depends comparatively little on parameter tuning and learning time. This suggests that the proposed method inspired by human cognition works adaptively in real environments.

The efficacy of symmetric cognitive biases in robotic motion learning

We propose an application of human-like decision-making to robotic motion learning. Human is known to have illogical symmetric cognitive biases that induce “if p then q” and “if not q then not p” from “if q then p.” The loosely symmetric Shinohara model quantitatively represents the tendencies (Shinohara et al. 2007). Previous studies one of the authors have revealed that an agent with the model used as the action value function shows great performance in n-armed bandit problems, because of the illogical biases. In this study, we apply the model to reinforcement learning with Q-learning algorithm. Testing the model on a simulated giant-swing robot, we have confirmed its efficacy in convergence speed increase and avoidance of local optimum.

Management of a Lecture of Robot Contest for Many Students

Abstract This paper describes a method of management of a lecture on the robot contest for many students. A lecture on robot contest has many merits. For example, the student not only learns engineering knowledge, but also develops problem-solving. However, there is often a problem that a lot of faculty staffs are needed to handle many students. Therefore, we divide the students into the group. And we teach the group how to solve a question, after the member of the group discusses the question. In this paper, the schedule is shown first. Next, the point of the lecture and effectiveness that we expect are illustrated. Finally, the result is discussed through the robot contest.

A study on effect of two-arch structure of foot for biped robots

The arch structure of human foot plays important roles such as impact absorption for bipedal locomotion. In this paper, the two-arch structure is introduced into the foot model of biped robot. Using a flat foot model, a one-arch foot model and the two-arch foot model, simulations of robot motion are performed by ODE and effects of the arch structure are verified. When the inner arch was softer than the outer arch like human, the robot motion such that the COG returns to inner direction was observed. It was verified that the property of the returning motion was determined by the difference of elasticities between the inner arch and the outer arch, and the property of motion in the frontal direction was determined by the average of elasticities.

Improvement of start-up and transient characteristics of digital-controlled DC-DC converter using switching-timing predictive PWM control

This paper proposes a new method for the control of a digital-controlled DC-DC converter. By using this method, both start-up and transient characteristics are improved. The dynamic characteristics at sampling time can be matched to a first-order system. Accordingly, the start-up waveform monotonically increases and the output voltage variation at load transient is small. This method is implemented in a digital signal processor (DSP) and its effectiveness is confirmed in circuit experiments. As a result, the output voltage variation at load transient is as small as 230 mV, which is 7.0% of the rated voltage of 3.3 V, and that at input transient is 43 mV (1.3%). In the start-up characteristics, this method enabled the output voltage to rise to the rated voltage smoothly without overshoot under any conditions.