Theologos Bountourelis

Efficient PAC Learning for Episodic Tasks with Acyclic State Spaces

This paper considers the problem of computing an optimal policy for a Markov decision process, under lack of complete a priori knowledge of (1) the branching probability distributions determining the evolution of the process state upon the execution of the different actions, and (2) the probability distributions characterizing the immediate rewards returned by the environment as a result of the execution of these actions at different states of the process. In addition, it is assumed that the underlying process evolves in a repetitive, episodic manner, with each episode starting from a well-defined initial state and evolving over an acyclic state space. A novel efficient algorithm for this problem is proposed, and its convergence properties and computational complexity are rigorously characterized in the formal framework of computational learning theory. Furthermore, in the process of deriving the aforementioned results, the presented work generalizes Bechhofer’s “indifference-zone” approach for the ranking & selection problem, that arises in statistical inference theory, so that it applies to populations with bounded general distributions.

Development and validation of a large scale ICU simulation model with blocking

Intensive Care Units (ICUs) are specialized healthcare delivery units for patients that require the highest level of monitored care. ICUs are typically integrated into larger healthcare facilities and their operation is dependent on the operational status of other inpatient units and departments of the host facility. As patients transition between units, a lack of available beds in a requested unit may cause patients to stay in a level of care other than that which is clinically indicated, leading to unnecessary or unwarranted costs without improving medical outcomes. The simulation modeling work presented in this paper is part of a multidisciplinary research project aimed towards patient delays. We describe the design and validation of a large scale ICU simulation model that includes various inpatient units and departments of the hospital. We describe the (i) input data analysis, (ii) modeling of patient flow, and (iii) validation of the simulation model.

Optimal Node Visitation in Stochastic Digraphs

The optimal node visitation (ONV) problem addressed in this paper concerns the visitation of a subset of nodes in a stochastic graph a specified number of times, while minimizing the expected visits to another node in this graph. The presented results first provide a formulation of the ONV problem as a stochastic shortest path problem, and subsequently they develop a suboptimal policy that is computationally tractable and asymptotically optimal. In particular, it is established that the ratio of the expected performance of this policy to the expected performance of an optimal policy converges to one, as the underlying visitation requirements are scaled uniformly to infinity. Furthermore, it is shown that under some stronger assumptions, the divergence of the performance of this policy from the performance of the optimal policy remains uniformly bounded by a constant, as the visitation requirements are scaled to infinity. Finally, it is shown that, for certain problem structures, the considered policy admits a closed-form characterization of its performance, which subsequently enables its optimized parameterization and its efficient integration into adaptive control schemes of even higher efficiency.

Optimal node visitation in acyclic stochastic digraphs

Given a stochastic, acyclic, connected digraph with a single source node and a control agent that repetitively traverses this graph, each time starting from the source node, we want to define a control policy that will enable this agent to visit each of the graph terminal nodes a prespecified number of times, while minimizing the expected number of the graph traversals. We first formulate this problem as a specially structured discrete time Markov decision process, and we subsequently develop an asymptotically optimal randomized policy of polynomial complexity with respect to the problem size

The Modeling, Analysis, and Management of Intensive Care Units

Intensive care units (ICUs) are limited-capacity, resource-intensive wards in a hospital designed to provide continuously monitored, intensive care and temporary support to critically-ill patients with a broad range of health conditions. Therefore, their efficient operation and management are critical to providing quality care to the most severely ill patients and to reducing costs for healthcare providers. Computer-based simulation and analytical models have historically been used to analyze ICU operational outcomes such as patient waiting times, bed occupancy rates, denied admission rates, and daily operating costs. This chapter highlights the variety of models and techniques that are prevalent in the modeling, analysis, and management of ICUs. Additionally, we describe an ongoing, multidisciplinary research project whose aim is to develop an empirically-validated discrete-event simulation model to analyze the performance of multiple ICUs at a local Veterans Affairs (VA) hospital.

The case against utilization: deceptive performance measures in inpatient care capacity models

Health care capacity decisions are often based on average performance metrics such as utilization. However, such decisions can be misleading, as a large portion of the costs in service operations is due to the inability to provide service due to congestion. This paper will review sources of variation that affect inpatient care capacity and develop a series of models of patient flow in a health care facility. We demonstrate that even in settings where the patient population and services provided are fixed, models that do not account for natural variations in the arrival rate and correlation in patient lengths of stay in sequential units will show the same utilization, but underestimate congestion and the resulting costs. Therefore, we argue that utilization is an inappropriate measure for validating models and congestion metrics such as blocking and diversions should be used instead.

Sensitivity analysis of an ICU simulation model

The modeling and simulation of inpatient healthcare systems comprising of multiple interconnected units of monitored care is a challenging task given the nature of clinical practices and procedures that regulate patient flow. Therefore, any related study on the properties of patient flow should (i) explicitly consider the modeling of patient movement rules in face of congestion, and (ii) examine the sensitivity of simulation output, expressed by patient delays and diversions, over different patient movement modeling approaches. In this work, we use a high fidelity simulation model of a tertiary facility that can incorporate complex patient movement rules to investigate the challenges inherent in its employment for resource allocation tasks.

Customized learning algorithms for episodic tasks with acyclic state spaces

The work presented in this paper provides a practical, customized learning algorithm for reinforcement learning tasks that evolve episodically over acyclic state spaces. The presented results are motivated by the Optimal Disassembly Planning (ODP) problem described in [14], and they complement and enhance some earlier developments on this problem that were presented in [15]. In particular, the proposed algorithm is shown to be a substantial improvement of the original algorithm developed in [15], in terms of, both, the involved computational effort and the attained performance, where the latter is measured by the accumulated reward. The new algorithm also leads to a robust performance gain over the typical Q-learning implementations for the considered problem context.

Optimal Node Visitation in Acyclic Stochastic Digraphs with Multi-threaded Traversals and Internal Visitation Requirements

The original definition of the problem of optimal node visitation (ONV) in acyclic stochastic digraphs concerns the identification of a routing policy that will enable the visitation of each leaf node a requested number of times, while minimizing the expected number of the graph traversals. The original work of Bountourelis and Reveliotis (2006) formulated this problem as a Stochastic Shortest Path (SSP) problem, and since the state space of this SSP formulation is exponentially sized with respect to the number of the target nodes, it also proposed a suboptimal policy that is computationally tractable and asymptotically optimal. This paper extends the results of Bountourelis and Reveliotis (2006) to the cases where (i) the tokens traversing the graph can “split” during certain transitions to a number of (sub-)tokens, allowing, thus, the satisfaction of many visitation requirements during a single graph traversal, and (ii) there are additional visitation requirements attached to the internal graph nodes, which, however, can be served only when the visitation requirements of their successors have been fully met. In addition, the presented set of results establishes stronger convergence properties for the proposed suboptimal policies, and it provides a formal complexity analysis of the considered ONV formulations. From a practical standpoint, the extension of the original results performed in this paper enables their effective usage in the application domains that motivated the ONV problem, in the first place.

Efficient learning algorithms for episodic tasks with acyclic state spaces

This paper considers the problem of computing an optimal policy for a Markov decision process (MDP), under lack of complete a priori knowledge of (i) the branching probability distributions determining the evolution of the process state upon the execution of the different actions, and (ii) the probability distributions characterizing the immediate rewards returned by the environment as a result of the execution of these actions at different states of the process. In addition, it is assumed that the underlying task evolves in a repetitive, episodic manner, with each episode starting from a well-defined initial state and evolving over an acyclic state space. A novel efficient algorithm for this problem is proposed, and its convergence properties and computational complexity are rigorously characterized in the formal framework of computational learning theory