Amir Massoud Farahmand

Manifold-adaptive dimension estimation

Intuitively, learning should be easier when the data points lie on a low-dimensional submanifold of the input space. Recently there has been a growing interest in algorithms that aim to exploit such geometrical properties of the data. Oftentimes these algorithms require estimating the dimension of the manifold first. In this paper we propose an algorithm for dimension estimation and study its finite-sample behaviour. The algorithm estimates the dimension locally around the data points using nearest neighbor techniques and then combines these local estimates. We show that the rate of convergence of the resulting estimate is independent of the dimension of the input space and hence the algorithm is “manifold-adaptive”. Thus, when the manifold supporting the data is low dimensional, the algorithm can be exponentially more efficient than its counterparts that are not exploiting this property. Our computer experiments confirm the obtained theoretical results.

Regularized Fitted Q-Iteration for planning in continuous-space Markovian decision problems

Reinforcement learning with linear and non-linear function approximation has been studied extensively in the last decade. However, as opposed to other fields of machine learning such as supervised learning, the effect of finite sample has not been thoroughly addressed within the reinforcement learning framework. In this paper we propose to use L2 regularization to control the complexity of the value function in reinforcement learning and planning problems. We consider the Regularized Fitted Q-Iteration algorithm and provide generalization bounds that account for small sample sizes. Finally, a realistic visual-servoing problem is used to illustrate the benefits of using the regularization procedure.

Robust Jacobian estimation for uncalibrated visual servoing

This paper addresses robust estimation of the uncalibrated visual-motor Jacobian for an image-based visual servoing (IBVS) system. The proposed method does not require knowledge of model or system parameters and is robust to outliers caused by various visual tracking errors, such as occlusion or mis-tracking. Previous uncalibrated methods are not robust to outliers and assume that the visual-motor data belong to the underlying model. In unstructured environments, this assumption may not hold. Outliers to the visual-motor model may deteriorate the Jacobian, which can make the system unstable or drive the arm in the wrong direction. We propose to apply a statistically robust M-estimator to reject the outliers. We compare the quality of the robust Jacobian estimation with the least squares-based estimation. The effect of outliers on the estimation quality is studied through MATLAB simulations and eye-in-hand visual servoing experiments using a WAM arm. Experimental results show that the Jacobian estimated by robust M-estimation is robust when up to 40% of the visualmotor data are outliers.

Global visual-motor estimation for uncalibrated visual servoing

In this paper, we present two methods for the estimation of a globally valid visual-motor model of a robotic manipulator. In conventional uncalibrated visual servoing, the visuo-motor function is approximated locally with a Jacobian. However, for optimal task planning, or nonlinear controller design with global stability guarantee, one needs to know a model that provides some information about the behavior of the system over the whole workspace. Our presented methods remedy this drawback in uncalibrated visual servoing by incrementally building a global estimator based on the movement history. We implement two such methods. The first method is a K-nearest neighborhood regressor over Jacobian that uses previously estimated local models. The second method stores previous movements and computes an estimate of the Jacobian by solving a local least squares problem. Experimental results show that both methods provide better global estimation quality compared to the conventional local estimation method with much lower estimation variance.

Model selection in reinforcement learning

We consider the challenge of automating parameter tuning in reinforcement learning. More specifically, we consider the batch (off-line, non-interactive) reinforcement learning setting and the problem of learning an action-value function with a small Bellman error. We propose a complexity regularization-based model selection algorithm and prove its adaptivity : the procedure is shown to perform almost as well as if the best parameter setting was known ahead of time. We also discuss other approaches to derive adaptive procedures in reinforcement learning.

Regularized Fitted Q-Iteration: Application to Planning

We consider planning in a Markovian decision problem, i.e., the problem of finding a good policy given access to a generative model of the environment. We propose to use fitted Q-iteration with penalized (or regularized) least-squares regression as the regression subroutine to address the problem of controlling model-complexity. The algorithm is presented in detail for the case when the function space is a reproducingkernel Hilbert space underlying a user-chosen kernel function. We derive bounds on the quality of the solution and argue that data-dependent penalties can lead to almost optimal performance. A simple example is used to illustrate the benefits of using a penalized procedure.

Model-based and model-free reinforcement learning for visual servoing

To address the difficulty of designing a controller for complex visual-servoing tasks, two learning-based uncalibrated approaches are introduced. The first method starts by building an estimated model for the visual-motor forward kinematic of the vision-robot system by a locally linear regression method. Afterwards, it uses a reinforcement learning method named Regularized Fitted Q-Iteration to find a controller (i.e. policy) for the system (model-based RL). The second method directly uses samples coming from the robot without building any intermediate model (model-free RL). The simulation results show that both methods perform comparably well despite not having any a priori knowledge about the robot.

Towards Learning Robotic Reaching and Pointing: An Uncalibrated Visual Servoing Approach

It is desirable for a robot to be able to operate in unstructured environments. In this paper, we demonstrate how a robot can learn primitive skills and we show how to augment them. We formalize 2D-decidable (pointing) and 3D-decidable (reaching) skills within an uncalibrated visual servoing framework. Skill decidability is defined in conjunction with an image-based controller, which has local asymptotic stability. In addition, we propose sequential composition of primitive skills to combine pointing and reaching skills in order to increase the accuracy of reaching skill. We use simple primitive tasks such as multi-point alignment and point-to-line alignment. We validate our results with real uncalibrated eye-in-hand experiments with a 4-DOF WAM from Barrett Technology Inc., alongside computer simulations.

Channel Assignment using Chaotic Simulated Annealing Enhanced Hopfield Neural Network

Channel assignment problem in cellular communication is a difficult combinatorial optimization problem. There is no exact polynomial-time solution for it and searching the whole solution space is infeasible for large problems. By defining the problem’s cost function as the energy function of a chaotic Hopfield neural network, we devise a framework for finding competitive suboptimal or even optimal solutions for combinatorial optimization problem in general, and channel assignment problem in particular. In our architecture, we inject chaotic noise in order to help the network escape from local minima of the energy function while we enforce problem constraints by external inputs of neurons. Experimental results show the superiority of our method to other methods.