Walid Gomaa

Adaptive multi-objective reinforcement learning with hybrid exploration for traffic signal control based on cooperative multi-agent framework

In this paper, we focus on computing a consistent traffic signal configuration at each junction that optimizes multiple performance indices, i.e., multi-objective traffic signal control. The multi-objective function includes minimizing trip waiting time, total trip time, and junction waiting time. Moreover, the multi-objective function includes maximizing flow rate, satisfying green waves for platoons traveling in main roads, avoiding accidents especially in residential areas, and forcing vehicles to move within moderate speed range of minimum fuel consumption. In particular, we formulate our multi-objective traffic signal control as a multi-agent system (MAS). Traffic signal controllers have a distributed nature in which each traffic signal agent acts individually and possibly cooperatively in a MAS. In addition, agents act autonomously according to the current traffic situation without any human intervention. Thus, we develop a multi-agent multi-objective reinforcement learning (RL) traffic signal control framework that simulates the drivers behavior (acceleration/deceleration) continuously in space and time dimensions. The proposed framework is based on a multi-objective sequential decision making process whose parameters are estimated based on the Bayesian interpretation of probability. Using this interpretation together with a novel adaptive cooperative exploration technique, the proposed traffic signal controller can make real-time adaptation in the sense that it responds effectively to the changing road dynamics. These road dynamics are simulated by the Green Light District (GLD) vehicle traffic simulator that is the testbed of our traffic signal control. We have implemented the Intelligent Driver Model (IDM) acceleration model in the GLD traffic simulator. The change in road conditions is modeled by varying the traffic demand probability distribution and adapting the IDM parameters to the adverse weather conditions. Under the congested and free traffic situations, the proposed multi-objective controller significantly outperforms the underlying single objective controller which only minimizes the trip waiting time (i.e., the total waiting time in the whole vehicle trip rather than at a specific junction). For instance, the average trip and waiting times are ~8 and 6 times lower respectively when using the multi-objective controller.

Machine learning in computational docking

OBJECTIVE The objective of this paper is to highlight the state-of-the-art machine learning (ML) techniques in computational docking. The use of smart computational methods in the life cycle of drug design is relatively a recent development that has gained much popularity and interest over the last few years. Central to this methodology is the notion of computational docking which is the process of predicting the best pose (orientation + conformation) of a small molecule (drug candidate) when bound to a target larger receptor molecule (protein) in order to form a stable complex molecule. In computational docking, a large number of binding poses are evaluated and ranked using a scoring function. The scoring function is a mathematical predictive model that produces a score that represents the binding free energy, and hence the stability, of the resulting complex molecule. Generally, such a function should produce a set of plausible ligands ranked according to their binding stability along with their binding poses. In more practical terms, an effective scoring function should produce promising drug candidates which can then be synthesized and physically screened using high throughput screening process. Therefore, the key to computer-aided drug design is the design of an efficient highly accurate scoring function (using ML techniques). METHODS The methods presented in this paper are specifically based on ML techniques. Despite many traditional techniques have been proposed, the performance was generally poor. Only in the last few years started the application of the ML technology in the design of scoring functions; and the results have been very promising. MATERIAL The ML-based techniques are based on various molecular features extracted from the abundance of protein-ligand information in the public molecular databases, e.g., protein data bank bind (PDBbind). RESULTS In this paper, we present this paradigm shift elaborating on the main constituent elements of the ML approach to molecular docking along with the state-of-the-art research in this area. For instance, the best random forest (RF)-based scoring function on PDBbind v2007 achieves a Pearson correlation coefficient between the predicted and experimentally determined binding affinities of 0.803 while the best conventional scoring function achieves 0.644. The best RF-based ranking power ranks the ligands correctly based on their experimentally determined binding affinities with accuracy 62.5% and identifies the top binding ligand with accuracy 78.1%. CONCLUSIONS We conclude with open questions and potential future research directions that can be pursued in smart computational docking; using molecular features of different nature (geometrical, energy terms, pharmacophore), advanced ML techniques (e.g., deep learning), combining more than one ML models.

Tokyo Virtual Living Lab: Designing Smart Cities Based on the 3D Internet

The Tokyo Virtual Living Lab is an experimental space based on 3D Internet technology that lets researchers conduct controlled driving and travel studies, including those involving multiple users in the same shared space. This shared-use feature is crucial for analyzing interactive driving behaviors in future smart cities. The labs novelty is two-fold: it outputs a semantically enriched graphical navigation network using free map data as input, and it includes a navigation segment agent that coordinates a multiagent traffic simulator. This simulator, which is based on the navigation network, supports the integration of user-controlled vehicles. The labs approach can significantly reduce the effort of preparing realistic driving behavior studies. To demonstrate this, the authors built a 3D model of a part of Tokyo to perform experiments with human drivers in two conditions: normal traffic and ubiquitous eco-traffic.

Enhanced multiagent multi-objective reinforcement learning for urban traffic light control

Traffic light control is one of the major problems in urban areas. This is due to the increasing number of vehicles and the high dynamics of the traffic network. Ordinary methods for traffic light control cause high rate of accidents, waste in time, and affect the environment negatively due to the high rates of fuel consumption. In this paper, we develop an enhanced version of our multiagent multi-objective traffic light control system that is based on a Reinforcement Learning (RL) approach. As a testbed framework for our traffic light controller, we use the open source Green Light District (GLD) vehicle traffic simulator. We analyze and fix some implementation problems in GLD that emerged when applying a more realistic continuous time acceleration model. We propose a new cooperation method between the neighboring traffic light agent controllers using specific learning and exploration rates. Our enhanced traffic light controller minimizes the trip time in major arteries and increases safety in residential areas. In addition, our traffic light controller satisfies green waves for platoons traveling in major arteries and considers as well the traffic environmental impact by keeping the vehicles speeds within the desirable thresholds for lowest fuel consumption. In order to evaluate the enhancements and new methods proposed in this paper, we have added new performance indices to GLD.

Analytical properties of resource-bounded real functionals

Computable analysis is an extension of classical discrete computability by enhancing the normal Turing machine model. It investigates mathematical analysis from the computability perspective. Though it is well developed on the computability level, it is still under developed on the complexity perspective, that is, when bounding the available computational resources. Recently Kawamura and Cook developed a framework to define the computational complexity of operators arising in analysis. Our goal is to understand the effects of complexity restrictions on the analytical properties of the operator. We focus on the case of norms over C[0,1] and introduce the notion of dependence of a norm on a point and relate it to the query complexity of the norm. We show that the dependence of almost every point is of the order of the query complexity of the norm. A norm with small complexity depends on a few points but, as compensation, highly depends on them. We briefly show how to obtain similar results for non-deterministic time complexity. We characterize the functionals that are computable using one oracle call only and discuss the uniformity of that characterization. This paper is a significant revision and expansion of an earlier conference version.

Multi-objective traffic light control system based on Bayesian probability interpretation

Traffic light control is a challenging problem in modern societies. This is due to the huge number of vehicles and the high dynamics of the traffic system. Poor traffic management causes a high rate of accidents, time losses, and negative impact on the economy as well as the environment. In this paper, we develop a multiagent traffic light control system based on a multi-objective sequential decision making framework. In order to respond effectively to the changing environment and the non-stationarity of the road network, the proposed traffic light controller is based on the Bayesian interpretation of probability. We use the open source Green Light District (GLD) vehicle traffic simulator as a testbed framework. The change in road conditions is modeled by varying the vehicles generation probability distributions and adapting the Intelligent Driver Model (IDM) parameters to the adverse weather conditions. We have added a set of new performance indices in GLD based on collaborative learning to better evaluate the performance of the proposed multi-objective traffic light controller. The results show that the proposed multi-objective controller outperforms the single-objective controller.

Adaptive traffic control system based on Bayesian probability interpretation

Traffic control (TC) is a challenging problem in todays modern society. This is due to several factors including the huge number of vehicles, the high dynamics of the system, and the nonlinear behavior exhibited by the different components of the system. Poor traffic management inflicts considerable cost due to the high rate of accidents, time losses, and negative impact on the economy as well as the environment. In this paper, we develop a traffic control system based on the Bayesian interpretation of probability that is adaptive to the high dynamics and non-stationarity of the road network. In order to simulate the traffic non-stationarity, we extend the Green Light District (GLD) vehicle traffic simulator. The change in road conditions is modeled by varying vehicle spawning probability distributions. We also implement the acceleration and lane changing models in GLD based on the Intelligent Driver Model (IDM).

Multi-Agent Reinforcement Learning Control for Ramp Metering

Traffic congestion is a challenging problem faced in everyday life. It has multiple negative effects on average speed, overall total travel time, and fuel consumption; in addition, it is a primary cause of accidents and air pollution. Hence, comes the need for an intelligent reliable traffic control system. The objective of this paper is to optimize the overall traffic congestion in freeways via multiple ramps metering controls. An optimal freeway operation can be reached if we always keep the freeway density within a small margin of the critical ratio for maximum traffic flow. In this paper, we propose a multi-agent reinforcement learning control system for ramp metering. Our system introduces a new microscopic framework at the network level based on collaborative Markov Decision Process modeling and an associated cooperative Q-learning algorithm. The technique incorporates payoff propagation (max-plus algorithm) under the coordination graph framework, particularly suited for optimal control purposes. The proposed system provides three control designs: fully independent, fully distributed, and centralized; suited for different network architectures. Our framework was extensively tested in order to assess the proposed model of the joint payoff, as well as the global payoff. We conducted experiments with heavy traffic flow under the renowned VISSIM traffic simulator so as to evaluate the proposed framework. The experimental results show that we achieved a significant decrease in the total travel time and an increase in the average speed -when compared with the base case- while maintaining an optimal traffic flow.

Comparative assessment of machine-learning scoring functions on PDBbind 2013

Computational docking is the core process of computer-aided drug design (CADD); it aims at predicting the best orientation and conformation of a small molecule (drug ligand) when bound to a target large receptor molecule (protein) in order to form a stable complex molecule. The docking quality is typically measured by a scoring function: a mathematical predictive model that produces a score representing the binding free energy and hence the stability of the resulting complex molecule. An effective scoring function should produce promising drug candidates which can then be synthesized and physically screened using high throughput screening (HTS) process. Therefore, the key to CADD is the design of an efficient highly accurate scoring function. Many traditional techniques have been proposed, however, the performance was generally poor. Only in the last few years the application of the machine learning (ML) technology has been applied in the design of scoring functions; and the results have been very promising.In this paper, we propose 12 scoring functions based on a wide range of ML techniques. We analyze the performance of each on the scoring power (binding affinity prediction), ranking power (relative ranking prediction), docking power (identifying the native binding poses among computer-generated decoys), and screening power (classifying true binders versus negative binders) using the PDBbind 2013 database. We compare our results with the recently published comparative assessment of scoring functions (CASF-2013) of 20 classical scoring functions most of which are implemented in main-stream commercial software. For the scoring and ranking powers, the proposed ML scoring functions depend on a wide range of features (energy terms, pharmacophore, geometrical) that entirely characterize the protein-ligand complexes (about 108 features); these features are extracted from several docking software available in the literature; a feature-space reduction technique, namely, principal component analysis is then applied and the performance is studied accordingly. For the docking and screening powers, the proposed ML scoring functions depend on the geometrical features of the RF-Score (36 features) to utilize a larger number of training complexes (relative to the large number of decoys in the testing set). For the scoring power, the best ML scoring function (RF) achieves a Pearson correlation coefficient between the predicted and experimentally determined binding affinities of 0.704 versus 0.614 achieved by the best classical scoring function ( X-Score HM ). For the ranking power, the best ML scoring function (RF) achieves a Spearman correlation coefficient between the ranks based on the predicted and experimentally determined binding affinities of 0.697 versus 0.626 achieved by the best classical scoring function ( X-Score HM ). For the docking power, the best ML scoring function (BRT) has a success rate in identifying the top best-scored ligand binding pose within 2? root-mean-square deviation from the native pose of 13.8% versus 81.0% achieved by the best classical scoring function (ChemPLP@GOLD). For the screening power, the best ML scoring function (SVM) has an average enrichment factor and success rate at the top 1% level of 3.76 and 6.45% versus 19.54 and 60% respectively achieved by the best classical scoring function (GlideScore-SP).

Freeway ramp-metering control based on Reinforcement learning

Random occurrences of traffic congestion on freeways lead to system degradation over time. If no smart control measures are applied, this degradation can lead to accumulated congestion which can severely affect other parts of the traffic network. Consequently, the need for an optimal and reliable traffic control has become more critical. The aim of this research is to control the amount of vehicles entering the mainstream freeway from the ramp merging area, i.e., balance the demand and the capacity of the freeway . This keeps the freeway density below the critical density. Consequently, this leads to maximum utilization of the freeway without entering in congestion while maintaining the optimal freeway operation. The Reinforcement learning based density control agent (RLCA) is designed based on Markovion modeling with an associated Q-learning algorithm in order to address the stochastic nature of the traffic situation. Extensive analysis is conducted in order to assess the proposed definition of the (state, action) pairs, as well as the reward function. We experiment with two case studies with two different network structures and demands. The first case study, which is the benchmark network used in literature, is the network with dense demand. Whereas the other one is the network with light demand. RLCA shows a superior response with respect to a predetermined reference points especially in terms of freeway density, flow rate, and total travel time.