Eduardo F. Morales

Multi-objective optimization of water-using systems

Industrial water systems often allow efficient water uses via water reuse and/or recirculation. The design of the network layout connecting water-using processes is a complex problem which involves several criteria to optimize. Most of the time, this design is achieved using Water Pinch technology, optimizing the freshwater flow rate entering the system. This paper describes an approach that considers two criteria: (i) the minimization of freshwater consumption and (ii) the minimization of the infrastructure cost required to build the network. The optimization model considers water reuse between operations and wastewater treatment as the main mechanisms to reduce freshwater consumption. The model is solved using multi-objective distributed Q-learning (MDQL), a heuristic approach based on the exploitation of knowledge acquired during the search process. MDQL has been previously tested on several multi-objective optimization benchmark problems with promising results [C. Mariano, Reinforcement learning in multi-objective optimization, Ph.D. thesis in Computer Science, Instituto Tecnologico y de Estudios Superiores de Monterrey, Campus Cuernavaca, March, 2002, Cuernavaca, Mor., Mexico, 2001]. In order to compare the quality of the results obtained with MDQL, the reduced gradient method was applied to solve a weighted combination of the two objective functions used in the model. The proposed approach was tested on three cases: (i) a single contaminant four unitary operations problem where freshwater consumption is reduced via water reuse, (ii) a four contaminants real-world case with ten unitary operations, also with water reuse, and (iii) the water distribution network operation of Cuernavaca, Mexico, considering reduction of water leaks, operation of existing treatment plants at their design capacity, and design and construction of new treatment infrastructure to treat 100% of the wastewater produced. It is shown that the proposed approach can solved highly constrained real-world multi-objective optimization problems.

SICIB: An Interactive Music Composition System Using Body Movements

Traditionally, music and dance have been comple-mentary arts. However, their integration has notalways been entirely satisfactory. In general, adancer must conform movements to a predefinedpiece of music, leaving very little room for impro-visational creativity. In this article, a system calledSICIB—capable of music composition, improvisa-tion, and performance using body movements—isdescribed. SICIB uses data from sensors attached todancers and “if-then” rules to couple choreo-graphic gestures with music. The article describesthe choreographic elements considered by the sys-tem (such as position, velocity, acceleration, curva-ture, and torsion of movements, jumps, etc.), aswell as the musical elements that can be affectedby them (e.g., intensity, tone, music sequences,etc.) through two different music composition sys-tems: Escamol and Aura. The choreographic infor-mation obtained from the sensors, the musicalcapabilities of the music composition systems, anda simple rule-based coupling mechanism offersgood opportunities for interaction between chore-ographers and composers.The architecture of SICIB, which allows real-time performance, is also described. SICIB hasbeen used by three different composers and a cho-reographer with very encouraging results. In par-ticular, the dancer has been involved in music dia-logues with live performance musicians. Ourexperiences with the development of SICIB andour own insights into the relationship that newtechnologies offer to choreographers and dancersare also discussed.

Learning Structure from Data and Its Application to Ozone Prediction

In this paper we propose an algorithm for structure learning in predictive expert systems based on a probabilistic network representation. The idea is to have the “simplest” structure (minimum number of links) with acceptable predictive capability. The algorithm starts by building a tree structure based on measuring mutual information between pairs of variables, and then it adds links as necessary to obtain certain predictive performance. We have applied this method for ozone prediction in México City, where the ozone level is used as a global indicator for the air quality in different parts of the city. It is important to predict the ozone level a day, or at least several hours in advance, to reduce the health hazards and industrial losses that occur when the ozone reaches emergency levels. We obtained as a first approximation a tree-structured dependency model for predicting ozone in one part of the city. We observe that even with only three parameters, its estimations are acceptable.A causal network representation and the structure learning techniques produced some very interesting results for the ozone prediction problem. Firstly, we got some insight into the dependence structure of the phenomena. Secondly, we got an indication of which are the important and not so important variables for ozone forecasting. Taking this into account, the measurement and computational costs for ozone prediction could be reduced. And thirdly, we have obtained satisfactory short term ozone predictions based on a small set of the most important parameters.

Learning to fly by combining reinforcement learning with behavioural cloning

Reinforcement learning deals with learning optimal or near optimal policies while interacting with the environment. Application domains with many continuous variables are difficult to solve with existing reinforcement learning methods due to the large search space. In this paper, we use a relational representation to define powerful abstractions that allow us to incorporate domain knowledge and re-use previously learned policies in other similar problems. We also describe how to learn useful actions from human traces using a behavioural cloning approach combined with an exploration phase. Since several conflicting actions may be induced for the same abstract state, reinforcement learning is used to learn an optimal policy over this reduced space. It is shown experimentally how a combination of behavioural cloning and reinforcement learning using a relational representation is powerful enough to learn how to fly an aircraft through different points in space and different turbulence conditions.

A New Distributed Reinforcement Learning Algorithm for Multiple Objective Optimization Problems

This paper describes a new algorithm, called MDQL, for the solution of multiple objective optimization problems. MDQL is based on a new distributed Q-learning algorithm, called DQL, which is also introduced in this paper. In DQL a family of independent agents, exploring different options, finds a common policy in a common environment. Information about action goodness is transmitted using traces over state-action pairs. MDQL extends this idea to multiple objectives, assigning a family of agents for each objective involved. A non-dominant criterion is used to construct Pareto fronts and by delaying adjustments on the rewards MDQL achieves better distributions of solutions. Furthermore, an extension for applying reinforcement learning to continuous functions is also given. Successful results of MDQL on several test-bed problems suggested in the literature are described.

PAL: A Pattern-Based First-Order Inductive System

It has been argued that much of human intelligence can be viewed as the process of matching stored patterns. In particular, it is believed that chess masters use a pattern–based knowledge to analyze a position, followed by a pattern–based controlled search to verify or correct the analysis. In this paper, a first–order system, called PAL, that can learn patterns in the form of Horn clauses from simple example descriptions and general purpose knowledge is described. The learning model is based on (i) a constrained least general generalization algorithm to structure the hypothesis space and guide the learning process, and (ii) a pattern–based representation knowledge to constrain the construction of hypothesis. It is shown how PAL can learn chess patterns which are beyond the learning capabilities of current inductive systems. The same pattern–based approach is used to learn qualitative models of simple dynamic systems and counterpoint rules for two–voice musical pieces. Limitations of PAL in particular, and first–order systems in general, are exposed in domains where a large number of background definitions may be required for induction. Conclusions and future research directions are given.

DQL: A New Updating Strategy for Reinforcement Learning Based on Q-Learning

In reinforcement learning an autonomous agent learns an optimal policy while interacting with the environment. In particular, in one-step Q-learning, with each action an agent updates its Q values considering immediate rewards. In this paper a new strategy for updating Q values is proposed. The strategy, implemented in an algorithm called DQL, uses a set of agents all searching the same goal in the same space to obtain the same optimal policy. Each agent leaves traces over a copy of the environment (copies of Q-values), while searching for a goal. These copies are used by the agents to decide which actions to take. Once all the agents reach a goal, the original Q-values of the best solution found by all the agents are updated using Watkins Q-learning formula. DQL has some similarities with Gambardellas Ant-Q algorithm [4], however it does not require the definition of a domain dependent heuristic and consequently the tuning of additional parameters. DQL also does not update the original Q-values with zero reward while the agents are searching, as Ant-Q does. It is shown how DQLs guided exploration of several agents with selected exploitation (updating only the best solution) produces faster convergence times than Q-learning and Ant-Q on several testbed problems under similar conditions.

A New Approach for the Solution of Multiple Objective Optimization Problems Based on Reinforcement Learning

Many problems can be characterized by several competing objectives. Multiple objective optimization problems have recently received considerable attention specially by the evolutionary algorithms community. Their proposals, however, require an adequate codification of the problem into strings, which is not always easy to do. This paper introduces a new algorithm, called MDQL, for multiple objective optimization problems which does not suffer from previous limitations. MDQL is based on a new distributed Q-learning algorithm, called DQL, which is also introduced in this paper. Furthermore, an extension for applying reinforcement learning to continuos functions is also given. Successful results of MDQL on a continuos non restricted problem whose Pareto front is convex and on a continuos non-convex problem with restrictions are described.

A Robust Exploration and Navigation Approach for Indoor Mobile Robots Merging Local and Global Strategies

A mobile robot must explore its workspace in order to learn a map of its environment. Given the perceptual limitations and accuracy of its sensors, the robot has to stay close to obstacles in order to track its position and never get lost. This paper describes a new method for exploring and navigating autonomously in indoor environments. It merges a local strategy, similar to a wall following strategy to keep the robot close to obstacles, within a global search frame, based on a dynamic programming algorithm. This hybrid approach takes advantages of local strategies that consider perceptual limitations of sensors without losing the completeness of a global search. These methods for exploring and navigating are tested using a mobile robot simulator with very good results.

Fast Feature Selection Method for Continuous Attributes with Nominal Class

Feature selection has become a relevant pre-processing problem on knowledge discovery in databases, because of very large databases or because some attributes are expensive to obtain. There is a large number of diverse feature selection methods for databases with pure nominal data (attributes and class), or pure continuous data, but little work has been done for the case of continuous attributes with nominal class. Normally what we can do is perform discretization, and then apply some traditional feature selection method; however the results can vary greatly depending on the discretization method used. We propose a direct method for feature selection on continuous data with nominal class, inspired in the Shannons entropy and an Information Gain measure. In the experiments that we realized, with synthetic and real databases, the proposed method has shown to be fast and to produce very competitive solutions with a small set of attributes