Hossam Abdelgawad

Multiagent Reinforcement Learning for Integrated Network of Adaptive Traffic Signal Controllers (MARLIN-ATSC): Methodology and Large-Scale Application on Downtown Toronto

Population is steadily increasing worldwide, resulting in intractable traffic congestion in dense urban areas. Adaptive traffic signal control (ATSC) has shown strong potential to effectively alleviate urban traffic congestion by adjusting signal timing plans in real time in response to traffic fluctuations to achieve desirable objectives (e.g., minimize delay). Efficient and robust ATSC can be designed using a multiagent reinforcement learning (MARL) approach in which each controller (agent) is responsible for the control of traffic lights around a single traffic junction. Applying MARL approaches to the ATSC problem is associated with a few challenges as agents typically react to changes in the environment at the individual level, but the overall behavior of all agents may not be optimal. This paper presents the development and evaluation of a novel system of multiagent reinforcement learning for integrated network of adaptive traffic signal controllers (MARLIN-ATSC). MARLIN-ATSC offers two possible modes: 1) independent mode, where each intersection controller works independently of other agents; and 2) integrated mode, where each controller coordinates signal control actions with neighboring intersections. MARLIN-ATSC is tested on a large-scale simulated network of 59 intersections in the lower downtown core of the City of Toronto, ON, Canada, for the morning rush hour. The results show unprecedented reduction in the average intersection delay ranging from 27% in mode 1 to 39% in mode 2 at the network level and travel-time savings of 15% in mode 1 and 26% in mode 2, along the busiest routes in Downtown Toronto.

Emergency evacuation planning as a network design problem: a critical review

Abstract Planning for emergency evacuation has evolved and matured substantially during the last two decades. The significant rise in natural and man-made disasters in recent years created a surge in the need for improved emergency evacuation planning. Voluminous studies, formulations, and control approaches have been presented in the literature with the common goal of improving the evacuation process to save precious time and lives. Proposed methods for improving the efficiency of emergency evacuation include reversing lanes in the direction of evacuation (contra-flow), staging the evacuation process, optimally controlling traffic, and providing route guidance to the evacuees. The objective of this paper is to review and compile the main strategies and approaches of emergency evacuation planning. Since most of these strategies can be generally viewed as a network design problem, particular attention has been given to briefly reviewing the network design problem (NDP) and the mutual mapping to the evacuation planning problem. In addition, this review touches upon the role of simulation in replicating the transportation network and the optimization in solving the evacuation problem. As we summarize the common schools of thought in evacuation planning, we critique the limitations, gaps, and challenges that hinder the development of an integrated optimal evacuation planning model. We also propose a framework that compiles evacuation strategies, network design problem formulation, traffic simulation and optimization tools in an attempt to fill some of the identified gaps/limitations.

Managing Large-Scale Multimodal Emergency Evacuations

This article presents the development of a novel framework that optimizes the evacuation of large cities using multiple modes including vehicular traffic, rapid transit, and mass-transit shuttle buses. A large-scale evacuation model is developed for the evacuation of the City of Toronto in case of emergency. A demand estimation model is first designed to accurately quantify the evacuation demand by mode (drivers vs. transit users), over time of the day when the crisis begins, and over space (location). The output of the demand estimation model is then fed into two optimization platforms: (1) an optimal spatio-temporal evacuation (OSTE) model that synergizes evacuation scheduling, route choice, and destination choice for vehicular traffic and (2) a model based on a new variant of the vehicle routing problem to optimize the routing and scheduling of mass-transit vehicles. The study concluded that OSTE can clear the City of Toronto 4 times faster than the do-nothing strategy. The OSTE average automobile evacuation time for the 1.21 million people using their cars is close to 2 h. The optimization of the routing and scheduling of the readily available Toronto Transit Commission fleet (4 Rapid Transit lines and 1320 transit buses used as shuttles) can efficiently evacuate the transit-dependent population (1.34 million) within 2 h.

Multiobjective Optimization for Multimodal Evacuation

This paper proposes a multimodal optimization framework that combines vehicular traffic and mass transit for emergency evacuation. The multi-objective approach optimizes the multimodal evacuation framework by investigating three objectives: minimizing in-vehicle travel time, minimizing at-origin waiting time, and minimizing fleet cost in the case of mass transit evacuation. For auto evacuees, an optimal spatiotemporal evacuation (OSTE) formulation is presented for generating optimal demand scheduling, destination choice, and route choice simultaneously. OSTE implements dynamic traffic assignment techniques coupled with genetic optimization to achieve the objective functions. For transit vehicles, a multiple-depot, time-constrained, pickup and delivery vehicle routing problem (MDTCPD-VRP) is formulated to model the use of public transit shuttle buses during evacuation. MDTCPD-VRP implements constraint programming and local search techniques to achieve the objective function and satisfy constraints. The OSTE and MDTCPD-VRP platforms are integrated in one framework to replicate the impact of congestion caused by traffic on transit vehicle travel times. This paper presents a prototype implementation of the conceptual framework for a hypothetical medium-size network in downtown Toronto, Ontario, Canada. The results show that including the waiting time and the in-vehicle travel time in the objective function reduced the network clearance time for auto-evacuees by 40% compared with including only the in-vehicle travel time. For mass transit, when considering fleet cost, an increase of 13% in network clearance time for transit evacuees was observed with a decrease of 12% in fleet size. Mass transit was shown to provide latent transportation capacity that is needed in evacuation situations.

Using Smartphones and Sensor Technologies to Automate Collection of Travel Data

This paper presents a data collection framework and its prototype application for personal activity–travel surveys through the use of smartphone sensors. The core components of the framework run on smartphones backed by cloud-based (online) services for data storage, information dissemination, and decision support. The framework employs machine-learning techniques to infer automatically activity types and travel modes with minimum interruption for the respondents. The three main components of the framework are (a) 24-h location data collection, (b) a dynamic land use database, and (c) a transportation mode identification component. The location logger is based on the smartphone network and can run for 24 h with minimal impact on smartphone battery life. The location logger is applicable equally in places where Global Positioning System signals are and are not available. The land use information is continuously updated from Internet location services such as Foursquare. The transportation mode identification module is able to distinguish six modes with 98.85% accuracy. The prototype application is conducted in the city of Toronto, Ontario, Canada, and the results clearly indicate the viability of this framework.

Design of Reinforcement Learning Parameters for Seamless Application of Adaptive Traffic Signal Control

Adaptive traffic signal control (ATSC) is a promising technique to alleviate traffic congestion. This article focuses on the development of an adaptive traffic signal control system using Reinforcement Learning (RL) as one of the efficient approaches to solve such stochastic closed loop optimal control problem. A generic RL control engine is developed and applied to a multi-phase traffic signal at an isolated intersection in Downtown Toronto in a simulation environment. Paramics, a microscopic simulation platform, is used to train and evaluate the adaptive traffic control system. This article investigates the following dimensions of the control problem: 1) RL learning methods, 2) traffic state representations, 3) action selection methods, 4) traffic signal phasing schemes, 5) reward definitions, and 6) variability of flow arrivals to the intersection. The system was tested on three networks (i.e., small, medium, large-scale) to ensure seamless transferability of the system design and results. The RL controller is benchmarked against optimized pretimed control and actuated control. The RL-based controller saves 48% average vehicle delay when compared to optimized pretimed controller and fully-actuated controller. In addition, the effect of the best design of RL-based ATSC system is tested on a large-scale application of 59 intersections in downtown Toronto and the results are compared versus the base case scenario of signal control systems in the field which are mix of pretimed and actuated controllers. The RL-based ATSC results in the following savings: average delay (27%), queue length (28%), and l CO2 emission factors (28%).

Towards a Multi-cluster Analytical Engine for Transportation Data

In the new digital age, the pace and volume of growing transportation related data is exceeding our ability to manage and analyze it. In this position paper, we present a data engine, Godzilla, to ingest real-time traffic data and support analytic and data mining over traffic data. Godzilla is a multi-cluster approach to handle large volumes of growing data, changing workloads and varying number of users. The data originates at multiple sources, and consists of multiple types. Meanwhile, the workloads belong to three camps, namely batch processing, interactive queries and graph analysis. Godzilla support multiple language abstractions from scripting to SQL-like language.

Self-Learning Adaptive Ramp Metering: Analysis of Design Parameters on a Test Case in Toronto, Canada

Ramp metering (RM) is the most effective dynamic traffic measure in response to growing congestion in urban freeway networks. Among the extensive RM methods available, those based on optimal control theory have shown strong potential in improving freeway performance. However, these algorithms require an accurate traffic model that limits their applicability in practice. Reinforcement learning (RL) provides the tools to achieve optimal RM control without reliance on any traffic model. In this paper, a guideline for designing RM control systems based on RL is presented by testing different states’ representations, learning methods, action selection, and reward definitions. A microscopic simulation test bed based on a portion of Highway 401 in Toronto, Canada, is developed to evaluate each of the above design parameters and quantify various RM control strategies. A comparison of the reinforcement learning ramp-metering (RLRM) algorithm with a modified version of ALINEA shows the potential of RLRM to improve freeway traffic conditions. When applied to the developed case study, the proposed RLRM algorithm and modified ALINEA reduce the total travel time by 40% and 20%, respectively, compared with the case with no RM.

Optimizing the number and locations of freeway roadside equipment units for travel time estimation in a connected vehicle environment

ABSTRACT This article introduces a methodology for determining the optimal number and locations of roadside equipment (RSE) units for travel time estimation in vehicle-to-infrastructure and vehicle-to-vehicle communication environments. The developed approach is a novel technique for modeling RSE placement to optimize the number and positions of RSE units while minimizing the travel time estimation error rate. A non-dominated sorting genetic algorithm (NSGA-II) was used to optimize this multi-objective problem. A microsimulation model of the highway 401 network in Toronto, Canada, was used as a testbed to evaluate the proposed approach. The NSGA-II approach produces an optimal Pareto front that minimizes the number, and hence cost, of RSE units while maximizing travel time estimation accuracy. Points on the Pareto front are equally optimal, dominate over all other points in the cost-accuracy search space, and offer the option to optimize the trade-off between infrastructure cost and estimation accuracy. This empirical study illustrates the impact of RSE placement on travel time accuracy in a connected vehicle environment. The optimization results indicate that the actual locations of the RSE units have a greater influence on the quality of the estimates than the number of RSE units. Thus, the accuracy of travel time estimates depends primarily on the locations of the RSE units and less on the total RSE density. Expanding RSE deployment might improve the accuracy of estimation; however, the associated costs will simultaneously increase.

Application of reinforcement learning with continuous state space to ramp metering in real-world conditions

In this paper we introduce a new approach to Freeway Ramp Metering (RM) based on Reinforcement Learning (RL) with focus on real-life experiments in a case study in the City of Toronto. Typical RL methods consider discrete state representation that lead to slow convergence in complex problems. Continuous representation of state space has the potential to significantly improve the learning speed and therefore enables tackling large-scale complex problems. A robust approach based on local regression, named k nearest neighbors temporal difference (kNN-TD), is employed to represent state space continuously in the RL environment. The performance of the new algorithm is compared against the ALINEA controller and typical RL methods using a micro-simulation testbed in Paramics. The results show that RM using the kNN-TD method can reduce total network travel time by 44% compared to the do-nothing case (without RM) and by 17% compared to ALINEA.