Scott Proper

Scaling model-based average-reward reinforcement learning for product delivery

Reinforcement learning in real-world domains suffers from three curses of dimensionality: explosions in state and action spaces, and high stochasticity. We present approaches that mitigate each of these curses. To handle the state-space explosion, we introduce “tabular linear functions” that generalize tile-coding and linear value functions. Action space complexity is reduced by replacing complete joint action space search with a form of hill climbing. To deal with high stochasticity, we introduce a new algorithm called ASH-learning, which is an afterstate version of H-Learning. Our extensions make it practical to apply reinforcement learning to a domain of product delivery – an optimization problem that combines inventory control and vehicle routing.

Transfer learning via relational templates

Transfer Learning refers to learning of knowledge in one domain that can be applied to a different domain. In this paper, we view transfer learning as generalization of knowledge in a richer representation language that includes multiple subdomains as parts of the same superdomain. We employ relational templates of different specificity to learn pieces of additive value functions. We show significant transfer of learned knowledge across different subdomains of a real-time strategy game by generalizing the value function using relational templates.

Simulation Based Machine Learning for Fault Detection in Complex Systems Using the Functional Failure Identification and Propagation Framework

Fault detection and identification in mechatronic systems with complex interdependencies between subsystems is a very active research area. Various alternative quantitative and qualitative methods have been proposed in the literature for fault identification on industrial processes, making it difficult for researchers and industrial practitioners to choose a method for their application. The Functional Failure Identification and Propagation (FFIP) framework has been proposed in past research for risk assessment of early complex system designs. FFIP is a versatile framework which has been extended in prior work to automatically evaluate sets of alternative system designs, perform sensitivity analysis, and event trees generation from critical event scenario simulation results. This paper’s contribution is an FFIP extension, used to generate the training and testing data sets needed to develop fault detection systems based on data driven machine learning methods. The methodology is illustrated with a case study of a generic nuclear power plant where a fault or the location of a fault within the system is identified. Two fault detection methods are compared, based on an artificial neural network and a decision tree. The case study results show that the decision tree was more meaningful as a model and had better detection accuracy (97% success in identification of fault location).Copyright

Coordinating actions in congestion games: impact of top–down and bottom–up utilities

Congestion games offer a perfect environment in which to study the impact of local decisions on global utilities in multiagent systems. What is particularly interesting in such problems is that no individual action is intrinsically “good” or “bad” but that combinations of actions lead to desirable or undesirable outcomes. As a consequence, agents need to learn how to coordinate their actions with those of other agents, rather than learn a particular set of “good” actions. A congestion game can be studied from two different perspectives: (i) from the top down, where a global utility (e.g., a system-centric view of congestion) specifies the task to be achieved; or (ii) from the bottom up, where each agent has its own intrinsic utility it wants to maximize. In many cases, these two approaches are at odds with one another, where agents aiming to maximize their intrinsic utilities lead to poor values of a system level utility. In this paper we extend results on difference utilities, a form of shaped utility that enables multiagent learning in congested, noisy conditions, to study the global behavior that arises from the agents’ choices in two types of congestion games. Our key result is that agents that aim to maximize a modified version of their own intrinsic utilities not only perform well in terms of the global utility, but also, on average perform better with respect to their own original utilities. In addition, we show that difference utilities are robust to agents “defecting” and using their own intrinsic utilities, and that performance degrades gracefully with the number of defectors.

DECISION MAKING FOR THE COLLABORATIVE ENERGY SUPPLY SYSTEM OF OREGON AND WASHINGTON

As demand for electricity in the United States continues to increase, it is necessary to explore the means through which the modern power supply system can accommodate both increasing affluence (which is accompanied by increased per-capita consumption) and the continually growing global population. Though there has been a great deal of research into the theoretical optimization of large-scale power systems, research into the use of an existing power system as a foundation for this growth has yet to be fully explored. Current successful and robust power generation systems that have significant renewable energy penetration — despite not having been optimized a priori — can be used to inform the advancement of modern power systems to accommodate the increasing demand for electricity. Leveraging ongoing research projects at Oregon State University and the National Energy Technology Laboratory, this work explores how an accurate and state-of-the-art computational model of the Oregon/Washington (OR/WA) energy system can be employed as part of an overarching power systems optimization scheme that looks to inform the decision making process for next generation power supply systems. Research scenarios that explore an introductory multi-objective power flow analysis for the OR/WA grid will be shown, along with a discussion of future research directions.Copyright

A Plant-Wide and Function-Specific Hierarchical Functional Fault Detection and Identification (HFFDI) System for Multiple Fault Scenarios on Complex Systems

The development of Fault Detection and Identification (FDI) systems for complex mechatronic systems is a challenging process. Many quantitative and qualitative fault detection methods have been proposed in past literature. Few methods address multiple faults, instead an emphasis is placed on accurately proving a single fault exists. The omission of multiple faults regulates the capability of most fault detection methods. The Functional Failure Identification and Propagation (FFIP) framework has been utilized in past research for various applications related to fault propagation in complex systems. In this paper a Hierarchical Functional Fault Detection and Identification (HFFDI) system is proposed. The development of the HFFDI system is based on machine learning techniques, commonly used as a basis for FDI systems, and the functional system decomposition of the FFIP framework. The HFFDI is composed of a plant-wide FDI system and function-specific FDI systems. The HFFDI aims at fault identification in multiple fault scenarios using single fault data sets, when faults happen in different system functions. The methodology is applied to a case study of a generic nuclear power plant with 17 system functions. Compared with a plant-wide FDI system, in multiple fault scenarios the HFFDI gave better results for identifying one fault and also was able to identify more than one faults. The case study results show that in two fault scenarios the HFFDI was able to identify one of the faults with 79% accuracy and both faults with 13% accuracy. In three fault scenarios the HFFDI was able to identify one of the faults with 69% accuracy, two faults with 22% accuracy and all three faults with 1% accuracy.Copyright

Robust Topology Design of Complex Infrastructure Systems

Optimizing the topology of complex infrastructure systems can minimize the impact of cascading failures due to an initiating failure event. This paper presents a novel approach for the concept-stage design of complex infrastructure systems by integrating model-based design with network analysis to increase system robustness. This approach focuses on system performance after cascading has occurred, and examines design trade-offs of the resultant (or degraded) system state. In this research, robustness is defined as the invariability of system performance due to uncertain failure events. Where a robust network has the ability to meet minimum performance requirements despite the impact of cascading failures. This research is motivated by catastrophic complex infrastructure system failures such as the August 13th Blackout of 2003, highlighting the vulnerability of systems such as the North American Power Grid (NAPG). A mathematical model was developed using an adjacency matrix, where removing a network connection simulates uncertain failure events. Performance degradation is iteratively calculated as failures cascade throughout the system, and robustness is measured by the lack of performance variability over multiple cascading failure scenarios. Two case studies are provided: an extrapolated IEEE 14 test bus, and the Oregon State University campus power network. The overarching goal of this research is to understand key system design trade-offs between robustness, performance objectives, and cost. In addition, optimizing network topologies to mitigate performance loss during concept-stage design will enable system robustness.Copyright